Data Science Day New York: Data Science: A Personal History
Download as PPT, PDF24 likes3,648 views
The document outlines a personal history of data science from Jeff Hammerbacher's perspective, emphasizing the evolution and role of data scientists at Facebook. It details the philosophies and tools related to data management and analysis, including Cloudera's platforms and products that support data-driven decision-making. Additionally, it discusses the future of data science, highlighting potential workloads and the importance of data quality for improved outcomes.
Data Science Day New York: Data Science: A Personal History
- 1. Data Science: A Personal History Jeff Hammerbacher 1
- 3. Data Applications Scientist “I have only heard back from one person about that ‘Data Applications Scientist’ thing. I had anticipated more discussion” – me, February 29, 2008 3
- 4. “I guess I’m arguing for ‘Data’ to replace ‘Research’ in those titles (I am happy to drop the ‘Applications’) as the primary focus of our organization is not corporate research.” – me, March 1, 2008 4
- 5. Data Scientist “I’d like to avoid specialization at this early stage and I expect every member of our group to have a mix of research, engineering, and analysis in their workload.” – me, March 1, 2008 5
- 6. Facebook Data Team The Facebook Data Team built scalable platforms for the collection, management, and analysis of data. We used these platforms to drive informed decisions in areas critical to the success of the company and to build data-intensive products and services. 6
- 8. Introduction to Data Science 1. Data Preparation 2. Data Presentation 3. Experimentation 4. Observation 5. Data Products 8
- 10. Philosophy • Instrument everything • Put all of your data in one place • Data first, questions later • Store first, structure later • Keep raw data forever • Let everyone party on the data • Produce tools to support the whole research cycle • Modular and composable infrastructure 10
- 11. CDH • Storage • Append-only unstructured data • Append-only tabular data • Mutable tabular data 11
- 12. CDH • Compute • Resource management • Parallel frameworks • High-level interfaces • Libraries 12
- 13. CDH • Integration • File system API • Database API • Batch data import/export • Event data import • User interface 13
- 14. Cloudera Products • Subscription • Proprietary software • Support • Training and Certification • Services 14
- 15. Cloudera Deployment Application Data Database CD H Warehouse Business Analytics Intelligence 15
- 16. Cloudera Workloads (Batch) • Active archive • Data reservoir • ETL/ELT offload 16
- 17. Cloudera Workloads (Interactive) • Application data delivery 17
- 18. Cloudera Customer Survey • 67% use Hive • 54% use HBase • 51% load data every 90 minutes or less • 71% move data from Hadoop to RDBMS for interactive SQL • 62% would like to consolidate into single platform 18
- 19. Cloudera Impala • General-purpose SQL query engine • Should work both for analytic and transactional workloads • Will support queries that take from microseconds to hours 19
- 20. Cloudera Impala • Runs directly within Hadoop • Reads widely used Hadoop file formats • Talks to widely used Hadoop storage managers • Runs on same nodes that run Hadoop processes 20
- 21. Cloudera Impala • High performance • C++ instead of Java • Runtime code generation • Completely new execution engine—not MapReduce 21
- 22. Cloudera Impala • Validated Beta Partners • MicroStrategy • QlikView • Tableau • Pentaho • Karmasphere • Capgemini 22
- 23. New Cloudera Workloads (Interactive) • Operational reporting • Ad hoc query 23
- 24. Cloudera Deployment Application Data Database CD H Warehouse Business Analytics Intelligence 24
- 25. The Future 25
- 26. Potential Future Workloads • Search • MPI • Stream processing • Graph computations • Linear algebra • Optimization • Simulation 26
- 27. The Last Mile • Data libraries • Language • Libraries • IDE for Data Scientists • Mixed-initiative • Memory • Collaboration • Model and analysis path selection 27
- 28. Doing Data Science • More data sources • More rows • More columns (novel or derived) • Better data quality • Better outcomes • Better loss functions • Causal inference in observational studies • Effect size estimates • Meta-analysis • Model lifecycle 28
- 29. 29