Analytics Lessons Learnt
0 likes247 views
The document discusses the complexities and challenges of business analytics, emphasizing iterative processes, the need for trust in data, and the importance of collaborative team dynamics. It highlights that while analytics can drive real-world impact, the journey is often laborious and error-prone, requiring careful planning and empathetic communication. Key takeaways include the significance of data preparation, the continuous refinement of questions, and the necessity for decision-makers to embrace a culture of learning and skepticism.
Analytics Lessons Learnt
- 1. Analytics - Lessons Learnt Dr. Venkata Pingali April 1, 2016
- 3. Conceptual Process Biz Analytics Team Data Engg Qtns, Context Data Req Datasets Model Results Story Telling All three roles could be in a single team!
- 4. Process in Reality Biz Analytics Team Data Engg Qtns, Context Data Req Datasets Model Results Story Telling Iterative Uncertain Expensive Laborious
- 5. Process in Reality Biz Analytics Team Data Engg Qtns, Context Data Req Datasets Model Results Story Telling Iterative Uncertain Expensive Laborious http://fortune.com/2016/02/05/why-big-data-isnt-paying-off-for-companies-yet/ "80% of .. companies strategic decision go haywire.. “flawed” data
- 7. Sense-making with Purpose ● Goal is impact - real change in the real world ○ Not mathematical machoness ○ Not blogs, presentations, etc. ● Model + Delivery = Impact ● Model - an approximation to real world ○ Three levels - Question, Domain, Process ○ Realworld has (unknown) complexity ○ Not an end in itself ● Delivery - Facilitation of incremental change ○ Multiple levels - Mindsets, technology, processes Closing loop is a reality check
- 8. An imperfect search process ● Imperfect questions, data, and process ○ Complexity discovered over time ○ Iterative refinement ● Laborious, error prone, and always incomplete ○ Data preparation (60-80% of work) is error prone ○ Questions -> Answers -> Questions ● Initial framing is just the beginning ○ Story will reveal itself over time Design for uncertainty
- 9. Successful Analytics Shifts Power ● There are winners and losers ○ Change is always painful ○ Efficiencies have to come from somewhere ● Mostly through power to contradict ○ Upsets conventional wisdom ● Sometimes through new paths forward Analytics is serious business
- 10. Trust is #1 requirement ● Change require trust in output (evidence and path forward) ● Gaining trust is hard work ○ Delivered by what you do and how ○ All the time and everything you do ● Integrity required through the entire lifecycle ○ Data ○ Process ○ Interpretation Design for trust
- 11. Math is either correct or not ● Sense-making may be qualitative but data or transformations are not ○ Every step is mathematical step ● Correct math is the basis for trust ○ Process is laborious ○ Work should not be trusted by default! ● “Hidden” transformations are risky ○ Excel changes ○ Filtering rules Mathematical indiscipline will be punished
- 12. Efficiency is #2 requirement ● Data science getting out of the lab environment ● Decision makers have realized that they could be wrong (often?) ○ Need to be contradicted only once - happening frequently ○ Now they are asking for input in all areas ● Sea change in last 4 years ● Growing combinations - #decisions x #scope x #frequency x #depth ○ Growing much faster than people & process can cope Process efficiency is essential to scaling
- 13. Team Character determines Quality ● Fundamentally about collaborative reasoning under uncertainty ○ Need a creative group of people ● Balanced skill along multiple dimensions ○ Domain (technology, business, individual) ○ Approach (model, experiment, field work) ○ Engagement (presentation, tech delivery, ops) ● Balanced process ○ Increased curiosity bandwidth will give people mastery, purpose ● Sense of purpose Look to build a strong team
- 14. Surviving the Insight Ladder ● Step 1 Wranging - Get to facts at summary level ● Step 2 Discovery - Frame initial questions & iterate to get to real questions ● Step 3 Relevance - Meaningful imprecise answers ● Step 4 Accuracy - Meaning precise answers ● Step 5 Robustness - Meaningful, precise, robust answers Continuously increase curiosity bandwidth Time spent here = Curiousity bandwidth
- 15. Business
- 16. Has to be shared organizational experience ● Mistakes are frequent ○ Through the entire lifecycle ● Domain knowledge is discovered ○ More important than math Make analytics a collective experience
- 17. Costs are front-loaded ● Data preparation/wranging ○ Takes arbitrary amount of time ○ Time/Effort ~ #elements ^^ 2 ● Errors in model development and operation ● Data version updates ● Changes in narratives Budgeting and expectation setting should be realistic
- 18. Empathetic delivery ● Analytics has collateral damage ○ People get fired, budgets are cut, new responsibilities get added ● Empathetic positioning and language ○ Understand that everybody wants to do their job well ○ People are not dumb ● Incremental actionables ○ Show way forward in byte chunks Plan the delivery carefully
- 19. Analytics work is risky ● Over-hyped context ○ Bigger, better examples everywhere - real or imagined ● Burden of expectations/magic from customer ● Things go wrong ○ Underwhelming/no results, methodological issues, wrong data ● Crisis as a teaching moment ○ Culture of learning, understanding and continuous refinement Enable team to take risks and have honest conversations
- 20. Individual
- 21. Dont be pygmalion ● Dont fall in love with data ○ It is imperfect like everything else ● Even simple data is too rich ○ You see what you want to see ● Be deeply skeptical ● Explore without judgment, detached Develop non-judgment curiousity
- 22. Extra
- 23. Decision-maker Questions 1. Where did the numbers come from? (Correctness, Lineage) a. Assumption, models, datasets 2. Is this an accident? Does it hold now? (Reproducibility, Retargetability) a. Model, dataset, and question revisions 3. Can you get the results faster? (Efficiency) a. Time, effort, cost 4. Can you also analyze X? (Extensibility) a. Different dataset, question 5. Could we try X? (Dataset generation - synthetic and real) a. What if scenarios, field experiments