Scaling Analysis Responsibly
2 likes10,555 views
The document discusses challenges that arise when trying to scale analytics teams by building business intelligence (BI) tools, and proposes an alternative approach of hiring "Analysis Developers" to help analysts scale their work using R. Some key points made include: - Building BI tools often leads to dysfunction as product and engineering teams compete to build the simplest tools. - Analysis Developers would develop reusable R packages and help all analysts work more efficiently through skills training. - This avoids issues like static tools becoming unstable as requirements change, and allows for flexible, reproducible analyses. - Promoting skills acquisition rather than deliverables helps analysts progress in their careers.
![Close 2-way partnership with data engineers to optimize
the creation of datasets for certain common analyses.
The assembly line handoff from scientist to engineer creates [an
uncreative] environment. The trick is to create an environment
that allows for autonomy, ownership, and focus for everyone
involved. - Jeff Magnusson
http://multithreaded.stitchfix.com/blog/2016/03/16/engineers-shouldnt-write-etl/](https://image.slidesharecdn.com/hilaryparkerrstatsnyc2016-160511213442/85/Scaling-Analysis-Responsibly-23-320.jpg)
Scaling Analysis Responsibly
- 2. #rcatladies Not So Standard Deviations @keegsdur
- 3. “We just don’t have enough analysts!”
- 4. “Let’s scale by building the perfect BI tool!”
- 5. That sounds great! We should automate some of the things that are slowing you down PRODUCT TEAM DATA http://xkcd.com/
- 6. That seems perfectly reasonable! Let’s just enlist some folks from engineering to help you with it DATAPRODUCT TEAM
- 7. DATA ENG Sure thing! ...and finally can it add this last graph?
- 9. ENG Sure! File a ticket! Can we add these 132 extra metrics to the testing? PRODUCT TEAM
- 10. You can’t do that, your family-wise error rate will tend to 1!! ENG PRODUCT TEAM DATA
- 11. ENG That’s a reasonable expectation for an internal product. I’m on it! I’d really like this tool to be more stable. PRODUCT TEAM
- 12. Our test violates a subtle statistical assumption for this new application, and we need to gut this stable product! ENG PRODUCT TEAM DATA
- 13. Almost impossible to avoid 2-against-1 dysfunction as product teams become “self-service” with engineering support Invariably becomes a race to the bottom as internal competition for the simplest tool emerges Stability prioritized over flexibility
- 14. (In tech) Building = Owning
- 16. “Analysis Developer” Someone on the analyst team who develops reproducible, flexible analyses in R and helps all analysts scale their work
- 17. I’ll work with the analysis developer on my team! We should automate some of the things that are slowing you down PRODUCT TEAM DATA
- 18. Avoids common types of dysfunction Allows for flexible, accurate analysis Analysts acquire marketable skills!
- 19. Instead of creating dashboards or using static BI tools... http://dilbert.com/strip/2007-05-16
- 20. Series of R packages highly specified for business case, “mix and match” elements to rapidly create common reports. library(“internal_package”)
- 22. Instead of “assembly line” data processing…
- 23. Close 2-way partnership with data engineers to optimize the creation of datasets for certain common analyses. The assembly line handoff from scientist to engineer creates [an uncreative] environment. The trick is to create an environment that allows for autonomy, ownership, and focus for everyone involved. - Jeff Magnusson http://multithreaded.stitchfix.com/blog/2016/03/16/engineers-shouldnt-write-etl/
- 24. Instead of PM anxiously watching dashboards… https://www.youtube.com/watch?v=CCbWyYr82BM
- 25. Analysts can create shorter-lived, reproducible reports
- 26. Expectation manage the shorter lifespan of the report, but include that report will require less work from teams once created Productionize in the short-term with CRON jobs Can add in more stats this way! Y/Y turns into semiparametric models, etc.
- 27. “The Problem with Dashboards (And A Solution)” by Stephanie Evergreen http://stephanieevergreen. com/problem-with-dashboards/
- 28. http://dilbert.com/strip/2004-04-05 Instead of promotion based on deliverables…
- 29. Consider skill acquisition for analyst promotion For analysis developers, promoted based on whether or not they were able to help other analysts become more efficient Support for skill acquisition!
- 30. Education support for learning better analysis development methods for all analysts Internally created resources
- 31. Instead of PMs self-teaching analysis based on what’s presented in dashboarding tools.. https://xkcd.com/605/
- 32. PMs can use tools for education analysts if they want to “ramp up” on analytical skills like R This way you can bake in statistical education as well.
- 33. “Isn’t this just package development?”
- 34. “Isn’t this just package development?” No!
- 36. Ad-hoc spreadsheet work + scripting
- 37. Ad-hoc spreadsheet work R workflows + scripting
- 38. Ad-hoc spreadsheet work R workflows + scripting + reproducibility, some functions, “analysis testing”
- 39. Ad-hoc spreadsheet work R workflows Reproducible R analyses + scripting + reproducibility, some functions, “analysis testing”
- 40. Ad-hoc spreadsheet work R workflows Reproducible R analyses + scripting + reproducibility, some functions, “analysis testing” + workplace-wide audience, documentation, testing - problem-specific writeups and functions
- 41. Ad-hoc spreadsheet work R workflows Reproducible R analyses Internal package development + scripting + reproducibility, some functions, “analysis testing” + workplace-wide audience, documentation, testing - problem-specific writeups and functions
- 42. Ad-hoc spreadsheet work R workflows Reproducible R analyses Internal package development + scripting + reproducibility, some functions, “analysis testing” + workplace-wide audience, documentation, testing - problem-specific writeups and functions + industry-wide audience - company-specific code and functions
- 43. Ad-hoc spreadsheet work R workflows Reproducible R analyses Internal package development External package development + scripting + reproducibility, some functions, “analysis testing” + workplace-wide audience, documentation, testing - problem-specific writeups and functions + industry-wide audience - company-specific code and functions
- 44. Ad-hoc spreadsheet work R workflows Reproducible R analyses Internal package development External package development + reproducibility, some functions, “analysis testing” + scripting + workplace-wide audience, documentation, testing - problem-specific writeups and functions + industry-wide audience - company-specific code and functions Analysis Developer Open-Source Developer
- 45. Analysis Developer Stop trying to scale with static BI tools -- this will (almost) always lead to dysfunction Instead, scale by increasing analyst efficiency using R and education! Hire Analysis Developers to help with all this!