Data Engineering at Udemy
Download as PPTX, PDF3 likes1,862 views
The document summarizes the typical evolution of data processing at a startup company and provides details about data engineering at Udemy. It describes how companies initially struggle with data before establishing scalable data infrastructure and workflows. At Udemy, they use AWS Redshift as their data warehouse, ingest data from various sources using Python ETL pipelines scheduled through Pinball, and use Hadoop/EMR for batch processing and AWS Kinesis for real-time processing. Lessons learned include starting with batch processing, considering the type of data, and storing data in a log format for debugging.
Data Engineering at Udemy
- 1. Data Engineering at Udemy Keeyong Han Principal Data Architect @Udemy Ankara Data Meetup - Bilkent Cyberpark, August 5, 2015
- 2. About Me • 20+ years of experience from 9 different companies • Currently manages Data team at Udemy • Prior to joining Udemy – Manager of data/search team at Polyvore – Director of Engineering at Yahoo Search – Started career from Samsung Electronics in Korea Ankara Data Meetup - Bilkent Cyberpark, August 5, 2015
- 3. Agenda • Typical Evolution of Data Processing • Data Engineering at Udemy • Lessons Learned Ankara Data Meetup - Bilkent Cyberpark, August 5, 2015
- 4. TYPICAL EVOLUTION OF DATA PROCESSING From a small start-up Ankara Data Meetup - Bilkent Cyberpark, August 5, 2015
- 5. In the beginning • You don’t have any data • So no data infrastructure or data science – The most important thing is to survive and to keep iterating Ankara Data Meetup - Bilkent Cyberpark, August 5, 2015
- 6. After a struggle you have some data • Now you survived and now you have some data to work with – Data analysts are hired – They want to analyze the data Ankara Data Meetup - Bilkent Cyberpark, August 5, 2015
- 7. Then … • You don’t know where the data is exactly • You find your data but – It is not clean and is missing key information – Data is likely not in the format you want • You store them in non-optimal storage – MySQL is likely used to store all kinds of data • But MySQL doesn’t scale – You ask analysts to query MySQL • They will kill the web site a few times Ankara Data Meetup - Bilkent Cyberpark, August 5, 2015
- 8. Ankara Data Meetup - Bilkent Cyberpark, August 5, 2015
- 9. Now what to do? (I) • You have to find a scalable and separate storage for data analysis – This is called Data Warehouse or Data Analytics – This will be the central storage for your important data – Udemy uses AWS Redshift • Migrate some data from MySQL – Key/Value data to NoSQL solution (Cassandra/Hbase, MongoDB, …) – Log type of data (use Nginx log for example) – MySQL should only have key data which is needed from Web service Ankara Data Meetup - Bilkent Cyberpark, August 5, 2015
- 10. Now what to do? (II) • The goal is to put every data into a single storage – This is the most important and the very first step toward becoming “true” data organization – This storage should be separated from runtime storage (MySQL for example) – This storage should be scalable – Being consistent is more important than being correct in the beginning Ankara Data Meetup - Bilkent Cyberpark, August 5, 2015
- 11. Now You Add More Data • Different Ways of Collecting Data – This is called ETL (Extract, Transform and Load) – Different Aspects to Consider • Size: 1KB to 20GB • Frequency: Hourly, Daily, Weekly, Monthly • How to collect data: – FTP, API, Webhook, S3, HTTP, mysql commandline • You will have multiple data collection workflows – Use cronjob (or some scheduler) to manage – Udemy uses Pinball (Open Source from Pinterest) Ankara Data Meetup - Bilkent Cyberpark, August 5, 2015
- 12. How It Will Look Like Your Cool Web Service Log Files MySQL Key/Value Data Warehouse External Data SourcesETL ETL Ankara Data Meetup - Bilkent Cyberpark, August 5, 2015
- 13. Simple Data import • Just use some script language – Many data sources are small and simple enough to use a script language • Udemy uses Python for this purpose – Implemented a set of Python classes to handle different types of data import – Plan to open source this in 1st half of 2016 Ankara Data Meetup - Bilkent Cyberpark, August 5, 2015
- 14. Large Data Batch Import • Large data import and processing will require more scalable solution • Hadoop can be used for this purpose – SQL on Hadoop: Hive, Tajo, Presto and so on – Pig, Java MapReduce • Spark is getting a lot of attention and we plan to evaluate Ankara Data Meetup - Bilkent Cyberpark, August 5, 2015
- 15. Realtime Data import • Some of data better be imported as it happens • This requires different type of technology – Realtime Data Message Queue: Kafka, Kinesis – Realtime Data Consumer: Storm, Samza, Spark Streaming Ankara Data Meetup - Bilkent Cyberpark, August 5, 2015
- 16. What’s Next? (I) • Build Summary Tables – Having raw data tables is good but it can be too detailed and too much information – Build these tables in your Data Warehouse • Track the performance of key metrics – This should be from summary tables above – You need dashboard tool (build one or use 3rd party solution – Birst, chartIO, Tableau and so on) Ankara Data Meetup - Bilkent Cyberpark, August 5, 2015
- 17. What’s Next? (II) • Provide this data to Data Science team – Draw insight and create feedback loop – Build machine learned models for recommendation, search ranking and so on – The topic for the next session (Thanks Larry!) • Supporting Data Science from Infrastructure – This will require scalable infrastructure – Example: Scoring every pairs of user/course in Udemy • 7M users X 30K courses = 210B pairs of computation – You need scalable Serving Layer (Cassandra, Hbase, …) Ankara Data Meetup - Bilkent Cyberpark, August 5, 2015
- 18. DATA ENGINEERING AT UDEMY Ankara Data Meetup - Bilkent Cyberpark, August 5, 2015
- 19. Data Warehouse/Analytics • We use AWS Redshift as our data warehouse (or data analytics backend) • What is AWS Redshift? – Scalable Postgresql Engine up to 1.6PB of data – Roughly it is 600 USD per TB per month – Mainly for offline batch processing – Supports bulk update (through AWS S3) – Two type of options: Compute vs. Storage Ankara Data Meetup - Bilkent Cyberpark, August 5, 2015
- 20. Kind of Data Stored in Redshift • 800+ tables with 2.4TB of data • Key tables from MySQL • Email Marketing Data • Ads Campaign Performance Data • SEO data from Google • Data from Web access log • Support Ticket Data • A/B Test Data (Mobile, Web) • Human curated data from Google Spreadsheets Ankara Data Meetup - Bilkent Cyberpark, August 5, 2015
- 21. Details on ETL Pipelines • All data pipelines are scheduled through Pinball – Every 5 minutes, hourly, daily, weekly and monthly • Most pipelines are purely in Python • Some uses Hadoop/Hive and Hadoop/Pig for Batch Processing • Start using Kinesis for Realtime Processing Ankara Data Meetup - Bilkent Cyberpark, August 5, 2015
- 22. Pinball Screenshot Ankara Data Meetup - Bilkent Cyberpark, August 5, 2015
- 23. Batching Processing Infrastructure • We use Hadoop 2.6 with Hive and Pig – CDH 5.4 (community version) • We use our own hadoop cluster and AWS EMR (ElasticMapReduce) at the same time – This is used to do ETL on massive data – This is also used to run massive model/scoring pipelines from Data Science team • Plan to evaluate Spark potentially as an alternative Ankara Data Meetup - Bilkent Cyberpark, August 5, 2015
- 24. Realtime Processing • Applications – The first application is to process web access log – Eventually we plan to use this to generate personalized recommendation on-the-fly • Plan to use AWS Kinesis – Evaluated Apache Kafka and AWS Kinesis • They are very similar but Kafka is an open source while Kinesis is a managed service from AWS • Decided to use AWS Kinesis • Plan to evaluate Realtime Consumer – Samza, Storm, Spark Streaming Ankara Data Meetup - Bilkent Cyberpark, August 5, 2015
- 25. What is Kinesis (Kafka)? • Realtime data processing service in AWS – Publisher-Subscriber message broker – Very similar to Kafka • It has two components – One is message queue where stream of data is stored • 24 hours of retention period • Pay hourly by the read/write rate of the queue – The other is KCL (Kinesis Client Library) • Using this, build Data Producer application or Data Consumer Application • This can be combined with Storm, Spark Streaming, … Ankara Data Meetup - Bilkent Cyberpark, August 5, 2015
- 26. Data Serving Layer • Redshift isn’t a good fit to read out the data in realtime fashion so you need something else • We are using (or plan to use) the followings: – Cassandra – Redis – ElasticSearch – MySQL Ankara Data Meetup - Bilkent Cyberpark, August 5, 2015
- 27. How It Looks Like Udemy Log Files (Nginx) MySQL Key/Value (Cassandra) Data Warehouse (Redshift) External Data Sources Data Science Pipeline ETL ETL Data Science Pipeline Ankara Data Meetup - Bilkent Cyberpark, August 5, 2015
- 28. LESSONS LEARNED Ankara Data Meetup - Bilkent Cyberpark, August 5, 2015
- 29. • As a small start-up survive first and then work on data • Starting point is to store all data in a single location (data warehouse) • Start with batch processing and then realtime • Consider the type of data you store – Log vs. Key/Value vs. Transactional Record • Store data in the form of log (change history) – So that you can always go back and debug/replay • Cloud is good unless you have really massive data Ankara Data Meetup - Bilkent Cyberpark, August 5, 2015
- 30. Q & A Udemy is Hiring Ankara Data Meetup - Bilkent Cyberpark, August 5, 2015
Editor's Notes
- #8: Logging format. Don’t try to take a snapshot and do aggregation
- #10: Add diagram MySQL was likely to be used to store all data and used by data analysts
- #11: Add diagram MySQL was likely to be used to store all data and used by data analysts What happens when you don’t have this – everyone does their own analysis and derive their own conclusion – waste of resource from a lot of one-off efforts
- #13: Add diagram MySQL was likely to be used to store all data and used by data analysts What happens when you don’t have this – everyone does their own analysis and derive their own conclusion – waste of resource from a lot of one-off efforts
- #16: Realtime recommendation