Analysing high throughput data in real time
Download as PPTX, PDF0 likes68 views
The document discusses high throughput data analysis at Hotstar, focusing on real-time stream processing with Apache Kafka and key metrics for video delivery and social signals. It outlines the ingestion of vast amounts of data and the importance of real-time analytics for monitoring metrics such as playback failure rates and online user engagement. Key takeaways emphasize the utility of stream processing for immediate action and anomaly detection.
Analysing high throughput data in real time
- 1. Analysing high throughput data in Real Time
- 2. namit@hotstar.com $whoamit $whoami Namit Mahuvakar Data Engineering
- 3. t Scale @Hotstar 1 million events ingested per second ● Built on Apache Kafka ● HA and Durable Ingestion ● Project 50 billion events per day 14.0 tb data produced per day ● Data in S3 & HDFS ● Warehouse in hBase ● Project 30 tb of data per day 8.1 tb data directly consumed ● Single query interface ● Parity between streaming & stationary data ● No limit on amount of data scans Ingestion Patterns Storage Patterns Consumption Patterns
- 4. What are we covering? ● Stream processing @Hotstar ● Case Study : Video Delivery Metrics ● Crucial metrics in Video Delivery ● Case Study : Social Signals ● Solving Engagement with Signals ● Where do we fit all this ● Key Takeaways
- 5. Video Delivery Metrics ● How do we track surge in video start up failure ? ● How do we track sudden increase in video buffering rate ? ● How do you react on these metrics if they are batch processed periodically ?
- 6. Social Signals ● How do we find online users in real-time ? ● How do we track online user notifications in real-time ? ● How to notify users in real time if the data is batch processed ?
- 7. Real-time User Targeting ● How do we target online users during a live match?
- 8. Not on my batch! ● Why? 25.3M concurrent viewers! ● When can Data streams can be processed in a batch fashion ? ● Leveraging stream processing: ○ P1 (Crucial) metrics for Hotstar ○ Social Signals - User notification ○ Real time user segmentation and Targeting
- 9. Case Study : Video Delivery Metrics
- 10. Video Delivery Metrics Ingest continuous streams of events from 25.3 Mil concurrent users, ● Video Player P1 Metrics - ○ Playback Failure Rate ○ Average Bitrate ○ Buffer time per minute ● Ingest CDN logs to measure last mile delivery
- 11. ● P1 Performance Metric ○ Real Time - Multiple Streams ● Ratio - Failed Video/Started Video ● What did we use for stream processing ? ○ Hive Kafka SQL Integration How did we solve : Playback Failure Rate
- 12. Real Time Analytics Use Case : How did we use Hive Kafka SQL to get Playback Failure Rate ?
- 13. Case Study : Social Signals
- 14. ● Indicators or signals sent to the user’s device ● Increase engagement with Social Feed ● Discover presence and activities with their social Graph ● Common Signals - ○ Friend Online ○ Friend Earns Rewards What are Social Signals ?
- 15. ● Real time processing of a clickstream event Started Video ● Helps us approximate presence of user on stream ● Query friends for Online user via social graph ● Publish notification to users online friends Solving Engagement with Signals
- 16. How did we solve Social Signals
- 17. Where do we fit all this
- 18. Key Takeaways ● Stream Processing is useful for: ○ Monitoring and real time alerting for quick actionables ○ Quick processing of incoming streams of data to drive actionables ex. targeting users ○ Process and find anomalies in live log or time series data
- 19. Check us out at tech.hotstar.com Blogs @ blog.hotstar.com
- 20. Any Questions?
Editor's Notes
- #2: Welcome to the talk. I will be talking about Data Platform Patterns for Scale. I will be sharing my learnings based on building the platform for Hotstar scale
- #19: Untangle movement of data Single source of truth No duplicate writes Anyone can consume anything Decouples data generation from data computation Process, transform and react on the data as it happens Sub-second latencies Anomaly detection on bad stream quality Timely notification to users who dropped off in a live match
- #20: Newer technologies are added
- #21: Newer technologies are added