Druid Optimizations for Scaling Customer Facing Analytics
- 1. Druid Optimizations for Scaling Customer Facing Analytics at Conviva
- 2. Amir Youssefi Principal Software Engineer Conviva Presenters Pawas Ranjan Engineer Manager Conviva
- 3. Highlights ● Streaming Analytics at Conviva ● Before Druid ● Druid Usage and Challenges (Query Timeouts, Reliable Data Ingestion...) ● Solutions ● Outcomes (add a 9 to our reliability...)
- 5. Conviva: Video Streaming Analytics
- 6. Products
- 7. Products
- 8. Products
- 10. Conviva: Stats
- 11. Before Druid
- 12. Before (2019 and earlier) Data Pipeline ● Hadoop MR Batch Jobs (5m) with rollup (Hourly and Daily) ● Spark Streaming (mini batches) ● Serving from HBase using Phoenix (SQL) ● SQL Query Gateway Data Center Locations ● On-premise ● Cloud (AWS): Hot Backup
- 14. Druid since 2019 Data Pipeline ● Started at Druid 3.x ● Native Druid Query Gateway ● Hadoop and Spark for batch ingestion ● Spark Streaming (Real Time, mini batches) ● Akka (Scala) Streaming and Spark Streaming ● Elastic and Imply Clarity for query/log analysis Data Center Locations ● On Premise ● Amazon AWS ● Google GCP
- 15. Druid Challenges
- 16. Druid Challenges ● Reliability and Performance (Query Timeouts) ● High Cardinality Measures ● Cost (specially on Cloud) ● Multi Tenancy and Data Locality ● Wide Rows (Metadata Performance) ● Multi Data Center/Cloud ● Orchestration (Docker -> K8s ) ● Rapid Disaster Recovery
- 18. Analytics over Query Logs Query Start/End Timestamps Run Times Entity Distribution Time-out Distribution Detailed Study of Query Access Patterns, Timestamps, Time-outs, etc.
- 19. Challenges/Solutions (Reliability) Reliability Issues ● Query Timeouts ● Reliable Ingestion & Query Speed Balance ● Query Performance (High Avg Time) ● Query Runtime Fluctuations ● Random Ingestion Task Failures Solutions ● Druid Configuration & Tuning ● Data Locality, Dynamic Partitions, Multi Tenancy ● Tuning Brokers; Extra Tier 3 for recent Data and Queries ● Query/Context Updates ● Resource and Configuration Adjustment
- 20. Challenge/Solutions Solutions ● Resource Optimizations; On-prem+GCP ● Created in-house Dimensional index ● Using Native Query instead of Druid SQL ● K8s+Helm Chart Improvements on GCP ● Supervisor Optimizations and disabling Historicals Challenges ● Cost (specially on Cloud) ● Querying High Cardinality Measures ● SQL Metadata Performance due to Wide Rows ● Rapid Disaster Recovery ● Real Time Cluster
- 21. Outcomes
- 22. Before/After: Response Times & Query Timeout
- 23. Additional 9 added to reliability... Recent days at Query Gateway
- 24. Questions?
- 25. Thank you Contact: amir@conviva.com , pranjan@conviva.com Special thanks to Ashrut Tewari and Haijie Wu