Log Events @Twitter
- 1. Event Log Pipeline @Twitter Challenges of handling Billions of events per minute Systems @Scale Fall 2021 Lohit VijayaRenu @lohitvijayarenu
- 2. # TAGHASHT AG#HASH #HASHTAG # TAG#HASH DataPlatform @Twitter #Startups #Yahoo! #Hadoop Lohit VijayaRenu @lohitvijayarenu 9:35AM · Oct 13, 2021
- 3. Example of slide ● User interactions generate events ● Events are grouped as Datasets ● Datasets for Data processing & Analytics Event Log Pipeline Event Log Pipeline
- 4. 1B 5B ~500 datasets ● Scale ● Resource requirement ~800 datasets Events per minute Events per minute Challenges
- 6. Example of slide Architecture Microservices >100k instances Clients Event Aggregation Scribe ~3K instances Events
- 7. Example of slide Architecture Microservices >100k instances Clients Event Aggregation Scribe ~3K instances MapReduce >10K jobs Event Processing Events
- 8. Example of slide Architecture Microservices >100k instances Clients Event Aggregation Scribe ~3K instances MapReduce >10K jobs HDFS, Presto, Scalding >100PBs of data Event Processing Event Log Management Events Datasets
- 9. Example of slide Example of Statement Slide. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Text size should be between 80 pt to 140 pt depending on paragraph length. ● Per aggregator scaling challenges ● No multi tenancy for datasets ● Eco system integration ● Difficult to maintain and improve Event Aggregation Problem
- 10. Example of slide Example of Statement Slide. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Text size should be between 80 pt to 140 pt depending on paragraph length. ● Moved to open source Flume ● Per aggregator improvements ● Microbatch, memory management,... ● Optimize resource usage ● Streaming at network speed Event Aggregation
- 11. Example of slide Example of Statement Slide. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Text size should be between 80 pt to 140 pt depending on paragraph length. ● Memory spikes before and after Event Aggregation
- 12. Example of slide Example of Statement Slide. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Text size should be between 80 pt to 140 pt depending on paragraph length. ● Data tiers with priorities ● Dataset groups ● Aggregator groups ● Dynamic scaling Event Aggregation
- 13. Example of slide Example of Statement Slide. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Text size should be between 80 pt to 140 pt depending on paragraph length. ● Scaling Aggregation Event Aggregation Dataset 1 Dataset 2 Dataset 3 Aggregator Group 1 Aggregator Group 2 Service Discovery Dataset Storage
- 14. Example of slide Example of Statement Slide. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Text size should be between 80 pt to 140 pt depending on paragraph length. Event Processing Problem ● Thousands of process jobs ● Multitenancy ● Long tail reducers ● Unpredictable run time (traffic surge)
- 15. Example of slide Event Processing ● Apache Tez based processor ● Data tier processors ● Dynamic Hash based partition
- 16. Example of slide Event Processing ● Apache Tez based processor ● Data tier processors ● Dynamic Hash based partition Raw Events Partition Partition Partition
- 17. Example of slide Example of Statement Slide. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Text size should be between 80 pt to 140 pt depending on paragraph length. Migration ● Transparent migration ● More than 5 Billion events per minute ● Per dataset scaling ● Reliable and fault tolerant framework
- 18. Example of slide Example of Statement Slide. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Text size should be between 80 pt to 140 pt depending on paragraph length. Migration ● Transparent migration ● More than 5 Billion events per minute ● Per dataset scaling ● Reliable and fault tolerant framework But……. Latency!!!
- 19. Example of slide Near real time Near real time event streaming ● Analytics on real time user interactions ● End to end latency of minutes ● Data storage with real time insertion
- 21. Example of slide Architecture Microservices >10k instances Clients Streaming Aggregation Flume, Google PubSub Events
- 22. Example of slide Architecture Microservices >10k instances Clients Streaming Aggregation Flume, Google PubSub Apache Beam, Google Dataflow Streaming Processor Events
- 23. Example of slide Architecture Microservices >10k instances Clients Streaming Aggregation Flume, Google PubSub Apache Beam, Google Dataflow BigQuery, Google Cloud Storage, Amplitude Streaming Processor Real time insertion Events Datasets
- 24. Example of slide Latency Dataset Type Event Latency p90 p95 p99 Dataset 1 (very small datasets) 0.42 sec 0.48 sec 0.59 sec Dataset 2 (Smaller datasets) 2.6 sec 2.8 sec 8.4 sec Dataset 3 (Large datasets) 22.1 sec 29.9 sec 67.7 sec Dataset 4 (Large datasets 5 min micro batch) 6.3 min 6.5 min 7.2 min ● Promising results ● Independent pipelines
- 25. Example of slide Example of Statement Slide. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Text size should be between 80 pt to 140 pt depending on paragraph length. New challenges and future ● Scale for volume and spikes ● Stream processing and ingestion at scale ● Faster catch up on failures ● Change Event Log Pipeline to Event Streaming Pipelines Follow @TwitterEng @TwitterCareers
- 26. Thank You! 26 Log Ingestion and Replication Data Platform Infrastructure Data Platform SRE
Editor's Notes
- #4: What are events and what does Event Log Pipeline look like? User interactions generate events. These flow throw our Event Log Pipeline which produce datasets These are available for data processing, data analytics
- #5: What was our challenge Used to see YoY growth. We had projected to grow from 1B events per minute to 5B events per minute
- #6: At the high level architecture Client emit event, More than 100K instance Event Aggregation framework at about 3K instances built on Scribe Once they generate raw dataset, these are further curated and optimized in processor Generating hourly batch
- #7: At the high level architecture Client emit event, More than 100K instance Event Aggregation framework at about 3K instances built on Scribe Once they generate raw dataset, these are further curated and optimized in processor Generating hourly batch
- #8: At the high level architecture Client emit event, More than 100K instance Event Aggregation framework at about 3K instances built on Scribe Once they generate raw dataset, these are further curated and optimized in processor Generating hourly batch
- #9: At the high level architecture Client emit event, More than 100K instance Event Aggregation framework at about 3K instances built on Scribe Once they generate raw dataset, these are further curated and optimized in processor Generating hourly batch
- #10: Per aggregator scaling challenge. Stress testing showed that there is a cap on how many events can be handled because of CPU bottleneck There was no multi tenancy. Bad dataset could cause problems to
- #13: Planned vs unplanned
- #14: Planned vs unplanned