Data Science at The New York Times: what industry can learn from us; what we can learn from industry
0 likes248 views
The document discusses the evolution of data science and its applications at The New York Times, highlighting the importance of new analytical tools and collaboration across disciplines. It outlines various modeling techniques such as descriptive, predictive, and prescriptive modeling, along with insights from user engagement research. Additionally, it emphasizes the challenges and requirements in data science, including the need for skilled personnel and innovative ideas.
![Related/extensions: lessons from statistical physics:
- Variational Methods for complex algebraic models
- Urteaga & W. arXiv:1709.0316 / AISTATS 2018
- Monte Carlo methods
- nonparametric mixture models: Urteaga & W. arXiv:
1808.02932 [stat.ML] via MC)
- particle filtering for arbitrary computational
models: Urteaga & W. arXiv:1808.02933](https://image.slidesharecdn.com/wiggins-191104234215/85/Data-Science-at-The-New-York-Times-what-industry-can-learn-from-us-what-we-can-learn-from-industry-74-320.jpg)
Data Science at The New York Times: what industry can learn from us; what we can learn from industry
- 1. data science @ The New York Times: chris.wiggins@columbia.edu chris.wiggins@nytimes.com what industry can learn from us; what we can learn from industry
- 2. data science @ The New York Times
- 3. biology: 1892 vs. 1995
- 4. biology: 1892 vs. 1995 biology changed for good.
- 5. biology: 1892 vs. 1995 new toolset, new mindset
- 6. biology: 1892 vs. 1995 “These are indeed exciting times, not unlike the early days of recombinant DNA in the 1970s, in which a revolutionary new technology permitted entirely new questions about the nature of genes to be raised. This challenge is new to biology, and its resolution will require, in addition to existing paradigms of molecular biology, new sets of analytical tools... disciplines outside of biology will be required to collaborate on this problem.” new toolset, new mindset
- 7. genetics: 1837 vs. 2012 ML toolset; data science mindset
- 8. genetics: 1837 vs. 2012
- 9. genetics: 1837 vs. 2012 ML toolset; data science mindset arxiv.org/abs/1105.5821 ; github.com/rajanil/mkboost Raj, Dewar, Palacios, Rabadan, CW
- 10. data science: mindset & toolset
- 15. in tf in gcp
- 16. data science: mindset & toolset drew conway, 2010
- 17. data science: mindset & toolset develop + deploy machine learning solutions to newsroom + business problems
- 18. data science @ The New York Times:
- 19. data science @ The New York Timesdata science @ The New York Times
- 20. 1851
- 21. news: 20th century church state
- 22. news: 20th century church state
- 23. news: 20th century church state
- 25. news: 21st century church state data
- 26. learnings - descriptive modeling - predictive modeling - prescriptive modeling
- 27. (actually ML, shhhh…) - (unsupervised learning) - (supervised learning) - (reinforcement learning)
- 28. (actually ML, shhhh…) 2012; h/t michael littman
- 29. learnings - descriptive modeling - predictive modeling - prescriptive modeling - descriptive modeling - predictive modeling - prescriptive modeling
- 32. Chong & Blei, SIGKDD 2011 CTM generative model: r: clicks w: words u: user-topic association v: article-topic association
- 33. Chong & Blei. SIGKDD 2011 related: learning phenotypes from EHR
- 34. Chong & Blei. SIGKDD 2011 related: learning phenotypes from EHR
- 35. UPDATE COPYReaderscope In the course of our global expansion, we realized we needed to have much more sophisticated, real-time insight into what’s happening across our site. Who is reading what? And where?
- 36. LOCATIONS FAQAUDIENCE SEGMENTSTOPICS FAQ or Intro Information Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut quislabore et dolore magna aliqua. enim ad minim veniam, quis quis aliqua ullamconostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. aliquip ex ea commodo consequat.aliquip ex Privacy Policy Terms of Service © The New York Times CompanyNYTimes.com Send Us Feedback Searchex. High-tech Lifestyle, Parents, Media - Comedy Films Audience Segment Search by nytreaderscope Illustration by Clara Nguyen LOCATIONS FAQAUDIENCE SEGMENTSTOPICS FAQ or Intro Information Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut quislabore et dolore magna aliqua. enim ad minim veniam, quis quis aliqua ullamconostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. aliquip ex ea commodo consequat.aliquip ex Privacy Policy Terms of Service © The New York Times CompanyNYTimes.com Send Us Feedback Searchex. High-tech Lifestyle, Parents, Media - Comedy Films Audience Segment Search by nytreaderscope Illustration by Clara Nguyen f f f Tool: Readerscope AUDIENCE INSIGHTS ENGINE
- 37. LOCATIONS FAQAUDIENCE SEGMENTSTOPICS FAQ or Intro Information Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut quislabore et dolore magna aliqua. enim ad minim veniam, quis quis aliqua ullamconostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. aliquip ex ea commodo consequat.aliquip ex Privacy Policy Terms of Service © The New York Times CompanyNYTimes.com Send Us Feedback Searchex. High-tech Lifestyle, Parents, Media - Comedy Films Audience Segment Search by nytreaderscope Illustration by Clara Nguyen LOCATIONS FAQAUDIENCE SEGMENTSTOPICS FAQ or Intro Information Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut quislabore et dolore magna aliqua. enim ad minim veniam, quis quis aliqua ullamconostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. aliquip ex ea commodo consequat.aliquip ex Privacy Policy Terms of Service © The New York Times CompanyNYTimes.com Send Us Feedback Searchex. High-tech Lifestyle, Parents, Media - Comedy Films Audience Segment Search by nytreaderscope Illustration by Clara Nguyen f f f Tool: Readerscope AUDIENCE INSIGHTS ENGINE
- 38. LOCATIONS FAQAUDIENCE SEGMENTSTOPICS FAQ or Intro Information Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut quislabore et dolore magna aliqua. enim ad minim veniam, quis quis aliqua ullamconostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. aliquip ex ea commodo consequat.aliquip ex Privacy Policy Terms of Service © The New York Times CompanyNYTimes.com Send Us Feedback Searchex. High-tech Lifestyle, Parents, Media - Comedy Films Audience Segment Search by nytreaderscope Illustration by Clara Nguyen LOCATIONS FAQAUDIENCE SEGMENTSTOPICS FAQ or Intro Information Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut quislabore et dolore magna aliqua. enim ad minim veniam, quis quis aliqua ullamconostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. aliquip ex ea commodo consequat.aliquip ex Privacy Policy Terms of Service © The New York Times CompanyNYTimes.com Send Us Feedback Searchex. High-tech Lifestyle, Parents, Media - Comedy Films Audience Segment Search by nytreaderscope Illustration by Clara Nguyen f Tool: Readerscope AUDIENCE INSIGHTS ENGINE
- 39. LOCATIONS FAQAUDIENCE SEGMENTSTOPICS FAQ or Intro Information Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut quislabore et dolore magna aliqua. enim ad minim veniam, quis quis aliqua ullamconostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. aliquip ex ea commodo consequat.aliquip ex Privacy Policy Terms of Service © The New York Times CompanyNYTimes.com Send Us Feedback Searchex. High-tech Lifestyle, Parents, Media - Comedy Films Audience Segment Search by nytreaderscope Illustration by Clara Nguyen C-Suite C-Suite, Executives and BDMs - Entertainment C-Suite, Executives and BDMs - Media C-Suite| C-Suite C-Suite, Executives and BDMs - Entertainment C-Suite, Executives and BDMs - Media LOCATIONS FAQAUDIENCE SEGMENTSTOPICS FAQ or Intro Information Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut quislabore et dolore magna aliqua. enim ad minim veniam, quis quis aliqua ullamconostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. aliquip ex ea commodo consequat.aliquip ex Privacy Policy Terms of Service © The New York Times CompanyNYTimes.com Send Us Feedback Searchex. High-tech Lifestyle, Parents, Media - Comedy Films Audience Segment Search by nytreaderscope Illustration by Clara Nguyen f Tool: Readerscope AUDIENCE INSIGHTS ENGINE
- 40. learnings - descriptive modeling - predictive modeling - prescriptive modeling - descriptive modeling - predictive modeling - prescriptive modeling
- 41. predictive modeling, e.g., “the funnel”
- 42. predictive modeling, e.g., “the funnel”
- 44. interpretable predictive modeling supercoolstuff Middendorf, Kundaje, Shah, Freund, CW, Leslie arxiv.org/abs/q-bio/0701021
- 46. work w/Daeil Kim & Hiroko Tabuchi
- 47. work w/Daeil Kim & Hiroko Tabuchi
- 48. work w/Daeil Kim & Hiroko Tabuchi
- 49. work w/Daeil Kim & Hiroko Tabuchi driving question: which records should she investigate?
- 50. We conducted user research to gather millions of observations about how different articles made people feel. DATA COLLECTION When reading this article, did you feel… Anger Sadness Happiness Despair Hurt No Emotion Jealousy Frustration Anxiety Hope Hate Interest Guilt Contentment Contempt Love Compassion Shame Amusement Stress Irritation Fear Boredom Surprise Confusion Disgust Irony Pride Disappointment
- 51. *units based on 100th percentile
- 52. Adventur ous 98 Interest 42 Happiness 96 Self Confident 39 Love 97 *units based on 100th percentile Hate Inspired 100 Amused 100 Sadness 27
- 54. Sources: Google DFP, NYT Ad Performance Data, Sizmek April May June July August September October November December Perspective Targeting Impression Volume By Month Throughout the year, NYT began running more and more perspective targeting campaigns every month. And performance kept breaking boundaries and setting new benchmarks for success. A Record First Year
- 55. Perspective Targeting outperformed brands’ own first-party segments, demonstrating that we’ve managed to find a new way to deliver meaning to these audiences.
- 56. learnings - descriptive modeling - predictive modeling - prescriptive modeling - descriptive modeling - predictive modeling - prescriptive modeling
- 57. learnings - descriptive modeling - predictive modeling - prescriptive modeling - descriptive modeling - predictive modeling - prescriptive modeling …two examples
- 60. leverage methods which are predictive yet performant w/“audience development” team
- 61. driving question: which content should we promote, where and when?
- 62. NB: data informed, not data-driven
- 63. learnings - descriptive modeling - predictive modeling - prescriptive modeling - descriptive modeling - predictive modeling - prescriptive modeling … recommendation as prescription
- 64. 2018: algos for *highly editorially curated* content pools - smarter living - midterms - editors picks
- 65. 2019: all of the above, plus: - For You Tab - stay tuned…
- 66. 2019: all of the above, plus: - For You Tab - stay tuned…
- 69. slow: Randomized controlled trial
- 70. fast: bandits
- 71. old (1933) idea: do the best you can
- 72. Lihong Li (YHOO->MSFT->GOOG), 2011 thompson sampling & “bandits”
- 73. old (1933) idea: do the best you can
- 74. Related/extensions: lessons from statistical physics: - Variational Methods for complex algebraic models - Urteaga & W. arXiv:1709.0316 / AISTATS 2018 - Monte Carlo methods - nonparametric mixture models: Urteaga & W. arXiv: 1808.02932 [stat.ML] via MC) - particle filtering for arbitrary computational models: Urteaga & W. arXiv:1808.02933
- 75. approximate variational methods for bandits Urteaga & W. arXiv:1709.03163
- 76. cf. modelingsocialdata.org monte carlo filtering w/drift? -> ‘genetic’ bandit
- 77. dynamic environments Urteaga & W. arXiv:1808.02932
- 78. common requirements in data science:
- 79. common requirements in data science: 1. people 2. ideas 3. things cf. John Boyd, USAF
- 80. monica rogati, Aug 1 2017 hackernoon.com things: de>da>ds/ml/ai
- 84. Reporting Learning Test Optimizing Exploredescriptive: predictive: prescriptive: ML primitives: learning, scoring, testing; - speed? - scale? - cost?
- 85. watch this space: NYT+AI physics math/fin p chem app math cog sciEE
- 86. people.. so far (we’re hiring!!!!) astrophys math/fin pure mathapp math cog sciEE biophysseismology neuro physics
- 87. data science @ The New York Times: chris.wiggins@columbia.edu chris.wiggins@nytimes.com chris.wiggins@hackNY.org @chrishwiggins what industry can learn from us; what we can learn from industry
- 88. data science @ The New York Times: chris.wiggins@columbia.edu chris.wiggins@nytimes.com chris.wiggins@hackNY.org @chrishwiggins what industry can learn from us; what we can learn from industry also: we’re hiring!