CASE STUDY:
WORD CLOUD

TL;DR

  • Word cloud was getting complaints that it wasn't useful or accurate

  • Analyzed feedback and conducted interviews to understand where the solution was falling short

  • Designed a solution to better match workers' needs and built a proof of concept

  • Tested the proof of concept with actual workers in their actual environment

  • Learned, iterated, built, and released a better solution

  • There was much rejoicing (usage went up, complaints went away)

BACKGROUND

The original concept for Traverse was for it to primarily be a "case discovery" tool. Traverse would OCR (optical character recognition) case documents, use NLP (natural language processing) to analyze the documents, find people and events, and produce a word cloud that summarizes the case. These features were largely using IBM Watson under the hood.

THE PROBLEMS

While the word cloud was a cool selling point to buyers, we began getting feedback almost immediately that there were problems. Specifically, there were 3 main issues with the word cloud:

  1. The word cloud was featured very prominently on the case overview (and other areas), but needed infrequently.

  2. Workers who knew the cases well, didn't feel the word cloud correctly represented the cases in many instances

  3. The word cloud used words that the case workers didn't use themselves to describe things.

FIRST, A QUICK FIX

First, we moved the word cloud off the case overview page. We added an "insights" area to the case and moved the word cloud there. Sales was still able to still have the word cloud to demo and our day-to-day users were glad to have it out of their way most of the time. We got more time and goodwill from our customers to find a better fix.

GETTING TO THE HEART OF THE PROBLEM

Just knowing that the word cloud didn't seem "right" to workers wasn't a whole lot to go on, so we dug deeper into the feedback and talked to customers. As we did, we began to uncover not only what was problematic with the current word cloud, but also what their true needs were. We were able to distill what we were learning down to the following issues:

  • The word cloud often included words that the workers themselves would rarely use to describe their cases. Sometimes words appeared that the workers would go out of their way to avoid using due to their harsh, polarizing, or traumatizing nature.

  • When clicking a word in the cloud, workers were taken to a list of content items that included content related to that word. However, the words or phrases in the document that would potentially give context as to why it was there weren't easy to find.

  • Workers usually had specific things they were looking for in a case - positive factors, negative factors, drugs, etc.

  • Workers were more interested in seeing how the case may be changing over time rather than a static, overall view of the entire life of the case.

A CONCEPT AND AN EXPERIMENT

As part of our research, we saw workers doing the same text searches over and over looking for specific words and phrases. Rather than using a black box NLP solution to analyze the content, we thought we might just need to automatically search for all the things they are looking for all at once and organize the results for workers to browse. It was also going to be important to show the actual instances of the words and phrases in context in the documents so the workers could decide if they were relevant or not.

Knowing that workers were often looking for drug references in documents, we decided to start by using a common drug database to seed the words we'd be searching for. We quickly built a proof of concept to test:

  1. The simple first solution was built so it could be turned on in the actual product so workers would be looking at their real case data.

  2. We went on-site to observe workers using the drug insights tool, get their feedback, and ask more questions.

  3. The concept was well received and workers were asking unprompted if we could add positive and negative factors to the insights - our planned future state!

MORE ITERATION

Once we knew the basic concept was solid, we started working out next steps. A good solution was going to need to be able to do a few important things:

  • The words and phrases being searched for the insights tab would need to be able to be updated and customized per customer

  • Ideally, the searches would happen "on-the-fly" as the worker chose the insights tab to accommodate new documents and updates to the words and phrases

We worked alongside our internal team of former caseworkers to gather an initial set of words and phrases the various categories of positive (protective) and negative (risk) factors we would be adding. We took the initial set and ran it through software to find related words and phrases and had our internal team pick through the additional data to weed out things that didn't make sense.

The next version was built and tested. More feedback. More refinement.

Once we got to an insights tab with protective factors, risk factors, and drug references, we coordinated with Customer Success, Training, Support, and Sales to release it to everyone!

THE RESULT

The updated Insights word cloud was well-received. Complaint calls went away (replaced by improvement suggestions). Usage went up - measured by clicks into works in the word cloud.

Designs were created to allow workers to constrain the word cloud by date ranges and new visualization concepts to better show changes over time - both of which were added to the future product roadmap.