How To Drive Data Driven Change In A Legacy Organization
- 1. How to drive data driven change in a legacy organization Subhasree Chatterjee Sr. Data Analyst LexisNexis
- 2. About me Lead Data Analyst Master’s in Business Analytics Long distance runner
- 3. Who we are • Innovating to support each customer's success • Combining information & analytics to help customers make more informed decisions & achieve better outcomes • Advancing the rule of law around the world What we do • Help lawyers with comprehensive, efficient research • Collaborate with universities to educate students, and governments and courts by making laws accessible and strengthening legal infrastructures • Partner with leading global associations to advance the rule of law
- 4. Typical product workflow Search Document View Document Retrieval
- 5. Data-Driven Product Design Identification & Prioritization Phase • Get involved at the beginning • Ask the right questions • Utilize different sources of data Hypotheses formation phase • Define specific and testable hypotheses • Create metrics to define success • Set up proper tracking mechanisms and dashboards Delivery Phase • A/B testing • Feature Performance Dashboard • User feedback • User satisfaction score How to shift from “gut feel”/received wisdom to data informed design change
- 6. Qual+Quant Data Triangulation Process WHAT? Why?
- 7. Case Study: Enhancement to Document Retrieval
- 8. Received wisdom around document retrieval TOP PREDICTOR OF USER SUCCESS HIGHER = BETTER
- 9. Identify & Prioritize the Customer Problem EVIDENCE • CX, NPS, and UX feedback indicate that users are having trouble with document retrieval PRIORITIZATION • Number of document retrieval is a top NPS driver (both positive and negative) • Document retrieval is a key indicator of overall task success CUSTOMER PROBLEM • Users are experiencing high document retrieval failure and cancellation rates
- 10. Form Hypotheses 1. Users have difficulty distinguishing between the document retrieval icons(print,email,download) in the toolbar, causing them to accidentally choose the wrong action 2. Users are drawn to the strong affordance of the “quick” action icon 3. Pop-up blockers are causing the retrieval to fail
- 11. Evaluate Hypotheses 1. Users have difficulty distinguishing between the retrieval icons in the toolbar, causing them to accidentally choose the wrong action 36% of users retrieve via more than one method on a single document “One of my biggest problems is printing. I typically have to print a document multiple times before I can get a full download that will print. And even when it works, it’s slow.” WHAT? WHY?
- 12. Evaluate Hypotheses 2. Users are drawn to the strong affordance of the “quick” action icon 80% of users retrieve via the “quick” action icon “I actually did not know you could do all of the items in this survey. I usually just press email and okay. It is interesting to know that I could customize how the document is emailed.” -- Associate WHAT? WHY?
- 13. Evaluate Hypotheses 3. Pop-up blockers are causing the document retrieval to fail Cancellation/failure rate were around 10% “How do I print a case? when I print, it does not go to my printer, nor does a printer dialog box pop up so I can choose a printer. and I can't find anywhere to set printer settings.” -- Associate WHAT? WHY?
- 14. Prototype and Deliver AMBIGUOUS BUTTONS WERE CLEANED UP DOCUMENT RETRIEVAL WITH SETTINGS WAS MADE THE DEFAULT OPTION A “PROCESSING BUDDY” WAS INTRODUCED
- 15. A/B Test: Document retrieval with settings won’t have any effect on cancellation rate
- 16. Design changes Launched. Panic!!
- 17. Right Metrics to look at New toolbar iconography “Ambiguous” icons removed New processing window
- 18. Lessons Learnt • Importance of defining the right metrics • More is always not better! • Importance of proper tracking of user actions • Importance of working together with other teams to rule out the opposing ways behavioral data can be interpreted
- 19. How to make this process work for you? 1. Get involved at the beginning. Data (both qualitative and quantitative) is very important at all stages of the product development lifecycle, but arguably the most important at the beginning, while assessing the feature itself 2. Ask the right questions. • Is there really a problem to solve? • How big is the problem? • What change are we trying to bring about and why? • How do we drive the necessary change in behavior? 3. Eyes on the prize. How do we measure success or failure? • Create metrics to define success. Just one composite metric or some vanity metrics won’t help. Have specific sets of metrics for a specific chunk of a problem. • Make sure to have proper tracking mechanisms to measure and have a visual setup to track the movement of those metrics 4. There is no such thing as no data, even when there is time pressure • Use usage data, customer support call data, nps data, heuristic reviews, etc. to gain a better understanding of how many users utilize the feature and what their pain points may be 5. Collaboration is everything! Having Product, UX and data science consistently talking to each other is important both to avoid wrinkles going forward and work efficiently
- 20. Key Questions to ask •What user problem are we trying to solve •What evidence do we have on the problem •What’s users’ reaction on the solution •How do we know we have solved it