Batch (Spark) and Streaming (Kafka) Data-Preprocessing
Download as PPTX, PDF0 likes453 views
The document provides an overview of a data science team's operations, highlighting their use of tools like Apache Spark and Kafka for data processing and machine learning. It mentions the team's strong growth, with over 150 employees across multiple countries, and emphasizes the importance of data preparation, which constitutes a significant portion of a data scientist's role. Additionally, it outlines the team's strategy for scaling their infrastructure and improving data processing efficiency.
Batch (Spark) and Streaming (Kafka) Data-Preprocessing
- 1. MEET OUR TEAM WRITE HERE SOMETHING #1 Data Science Club sli.do/exponea 17/18 Summer
- 2. MEET OUR TEAM WRITE HERE SOMETHING Batch (Spark) and Streaming (Kafka) Data Preprocessing Matus Cimerman
- 3. What do we do
- 5. SOME NUMBERS ● 150+ employees from 12 countries ● 29 average age ● 8 offices in 5 countries on 4 continents ● 1000%+ growth over the last 2 years
- 6. AI TEAM ● Recommendations, ● Propensity to buy during the session, ● Optimal time to send an email, ● Tech stack: Python, Gensim, Kubernetes, Spark, Go, TF, ...
- 7. How do we collect data
- 8. We don't scrape any websites
- 9. Arbitrary schema-less JSON objects Storage: IMF, MongoDB, HDFS Data Sources Events Columnar storage (CSV, SQL DB, …) Storage: Elastic, HDFS Products Static attributes (age, location, …) Dynamic attributes (# page visits, …) Storage: MongoDB, IMF, HDFS (parquet) Customers
- 10. Long-term storage (archive) ● Events: Newline delimited JSON ● Customers: Parquet file ● Products: CSV, soon BigTable
- 11. Why even this talk at all?
- 12. Data preparation & preprocessing is at least 80% of Data Scientist job
- 13. ML Algorithms don't eat raw data
- 14. We expect to growth 10-20x in the upcoming 1-2 years, so scale-out is critical
- 15. Batch (Spark)
- 16. Batch processing is done mostly by Apache Spark jobs, sadly written in Python
- 17. EASY, 12 LINES OF CODE + 500 LOC of boilerplate
- 18. Batch processing and Spark things to consider first 1. Try Kappa first. 2. If not possible, try to implement Lambda arch. 3. Combine them all!
- 20. Streaming ETL
- 21. Streaming ETL
- 22. Streaming ETL
- 23. EASY, 8 LINES OF CODE + 500 LOC of boilerplate
- 24. Infrastructure
- 26. Google Cloud Platform for the Win
- 27. Google Cloud Platform for the win
- 28. Google Cloud Platform more moving parts ● Inputs for ML algorithms from Storage. ● Training WORM1 to Storage. ● Kubernetes for orchestration. ● Large scale models deployment, K8s. ● Easy monitoring thanks to Stackdriver. 1Write once read many
- 29. Lessons learned
- 32. Future ● Spark Streaming + for all transformations. ● Batch in Spark for daily data consolidation. ● Kafka Streams for real-time use-cases.
- 33. That's all folks
- 35. PALO ALTO, CA 456 University Ave Palo Alto, CA 94301 +1 (650) 440-7297 PRAGUE, CZ Rohanské nábřeží 687/29, 186 00 Prague, Czechia +420 601 372 909 BRATISLAVA, SK Twin City B, Mlynské Nivy 12, 821 09, Bratislava, Slovakia +421 948 127 332 WARSAW, PL Postępu 14, 02-676 Warsaw, PL +48 603 663 766 MOSCOW, RU 10c1 Kozhevnicheskaya Street 115114, Moscow, RU +7 (495) 120 26 53 LONDON, UK 41 Corsham Street London N1 6DR, UK +44 (0) 203 086 8894 MANCHESTER, UK 1 Spinningfields, Quay Street Manchester M3 3JE +44 (0) 203 086 8894 EDINBURGH, UK 20/6 Fountainhall Road Edinburgh EH9 2NN +44 (0) 203 086 8894 www.exponea.com