Streaming Analytics in Uber
2 likes1,852 views
The document discusses stream processing and analytics using Flink for four kinds of analytics: on-demand aggregation and pattern detection, clustering, forecasting, and pattern detection on geo-temporal data. It describes building a unified stream processing architecture on Flink to support these analytics use cases by developing shared abstractions for multi-dimensional geo-temporal data and a geotemporal API. Examples of building forecasting applications on this architecture are provided. Key lessons learned include ensuring robust infrastructure, optimizing for single-node scaling, achieving exactly-once processing through proper data modeling, using external state storage, and standardizing monitoring.
Streaming Analytics in Uber
- 1. Stream Processing & Analytics with Flink Danny Yuan, Engineer @ Uber @g9yuayon
- 2. Four Kinds of Analytics - On demand aggregation and pattern detection - Clustering - Forecasting - Pattern detection on geo-temporal data
- 4. Real-time aggregation and pattern matching
- 5. Slide title Complex Event Processing
- 6. How many cars enter and exit a user defined area in past 5 minutes Examples
- 7. It doesn’t have to be within a time or area CEP with full historical context Notify me if a partner completed her 100th trip in a given area just now?
- 8. Patterns in the future How many first-time riders will be dropped off in a given area in the next 5 minutes?
- 9. Patterns in the future How many first-time riders will be dropped off in a given area in the next 5 minutes?
- 10. Geo: user flexibility is important
- 11. Geo: user flexibility is important
- 12. It needs to be scalable
- 13. It needs to be scalable
- 14. It needs to be scalable - Every hexagon - Every driver/rider
- 15. CEP Pipeline Built on Samza - No hard-coded CEP rules - Applying CEP rules per individual entity: topic, driver, rider, cohorts, and etc - Flexible checkpointing and statement management
- 16. Slide title
- 17. Slide title
- 18. Slide title
- 19. Slide title
- 20. Slide title
- 21. Slide title
- 22. Slide title
- 23. Slide title
- 24. We need to evolve our architecture for other analytics
- 25. Clustering
- 26. Slide title Manually Created Cluster
- 27. Slide titleCall for algorithmically created clusters - Clustering based on key performance metrics
- 28. Slide titleCall for algorithmically created cluster - Clustering based on key performance metrics - Continuously measure the clusters
- 29. Slide titleCall for algorithmically created clusters - Clustering based on key performance metrics - Continuously measure the clusters - Different clustering for different business needs
- 30. Slide titleCall for algorithmically created clusters - Clustering based on key performance metrics - Continuously measure the clusters - Different clustering for different business needs - Create clusters in minutes for all cities
- 31. Slide titleCall for algorithmically created clusters - Clustering based on key performance metrics - Continuously measure the clusters - Different clustering for different business needs - Create clusters in minutes for all cities - Foundation for other stream analytics
- 32. Slide title Home-grown Clustering Service
- 33. Slide title Home-grown Clustering Service - All cities under 3 minutes
- 34. Slide title Home-grown Clustering Service - All cities under 3 minutes - Pluggable algorithms and measurements
- 35. Slide title Home-grown Clustering Service - All cities under 3 minutes - Easily pluggable algorithms and measurements - Historical geo-temporal data for clustering
- 36. Slide title Home-grown Clustering Service - All cities under 3 minutes - Easily pluggable algorithms and measurements - Historical geo-temporal data for clustering - Real-time geo-temporal data for measurement
- 37. Slide title Home-grown Clustering Service - All cities under 3 minutes - Easily pluggable algorithms and measurements - Historical geo-temporal data for clustering - Real-time geo-temporal data for measurement - Shared optimizations
- 38. Slide title Home-grown Clustering Service - All cities under 3 minutes - Easily pluggable algorithms and measurements - Historical geo-temporal data for clustering - Real-time geo-temporal data for measurement - Shared optimizations. To put things in perspective: - 70,000 hexagons in SF - Naive distance function requires at least 70,000 x 70,000 = 4.9 billion pairs!
- 39. Slide title Home-grown Clustering Service - All cities under 3 minutes - Easily pluggable algorithms and measurements - Historical geo-temporal data for clustering - Real-time geo-temporal data for measurement - Shared optimizations - Incremental updates - Compact data representation - Memoization - Avoid anything more complex than O(nlog(n))
- 40. Forecasting - Every decision is based on forecasting
- 41. Forecasting - Forecasting based on both historical data and stream input
- 42. Forecasting - Forecasting based on both historical data and stream input
- 43. Forecasting - Forecasting based on both historical data and stream input Anomaly, or emerging demand?
- 44. Forecasting - Spatially granular forecasting - down to every hexagon
- 45. Forecasting - Spatially granular forecasting - down to every hexagon
- 46. Forecasting - Temporally granular forecasting - down to every minute
- 47. Forecasting - Temporally granular forecasting - down to every minute
- 48. Pattern Detection - Similarity of different metrics across geolocation and time - Metric outliers across geolocations and time - Frequent occurrences of certain patterns - Clustered behavior - Anomalies
- 49. Common Requirements in Pattern Detection - Not just traditional time series analysis - Incorporating insights on marketplace data - Required both historical data and real-time input - Spatially granular patterns - down to every hexagon - Temporally granular patterns - down to every minute
- 50. Example: Anomaly Detection - Simple time series analysis - For a single geo area - Can be noisy
- 51. A More Realistic Anomaly Detection
- 54. What’s the right architecture to support the analytics use cases?
- 55. Shared abstraction: multi-dimensional geo-temporal data
- 56. - Time series by event time Shared abstraction: multi-dimensional geo-temporal data
- 57. - Time series by event time Shared abstraction: multi-dimensional geo-temporal data https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101
- 58. - Time series by event time Shared abstraction: multi-dimensional geo-temporal data https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101
- 59. - Time series by event time Shared abstraction: multi-dimensional geo-temporal data https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101
- 60. - Time series by event time - Flexible windowing - tumbling, sliding, conditionally triggered Shared abstraction: multi-dimensional geo-temporal data
- 61. - Time series by event time - Flexible windowing - tumbling, sliding, conditionally triggered - e.g. event-based triggers Shared abstraction: multi-dimensional geo-temporal data
- 62. - Time series by event time - Flexible windowing - tumbling, sliding, conditionally triggered - e.g. event-based triggers - e.g., triggers of computation results Shared abstraction: multi-dimensional geo-temporal data
- 63. - Time series by event time - Flexible windowing - tumbling, sliding, conditionally triggered - Stateful processing Shared abstraction: multi-dimensional geo-temporal data
- 64. - Time series by event time - Flexible windowing - tumbling, sliding, conditionally triggered - Stateful processing. E.g., Shared abstraction: multi-dimensional geo-temporal data
- 65. - Time series by event time - Flexible windowing - tumbling, sliding, conditionally triggered - Stateful processing. E.g., Shared abstraction: multi-dimensional geo-temporal data
- 66. - Time series by event time - Flexible windowing - tumbling, sliding, conditionally triggered - Stateful processing. E.g., Shared abstraction: multi-dimensional geo-temporal data
- 67. - Time series by event time - Flexible windowing - tumbling, sliding, conditionally triggered - Stateful processing. E.g., Shared abstraction: multi-dimensional geo-temporal data
- 68. - Time series by event time - Flexible windowing - tumbling, sliding, conditionally triggered - Stateful processing. E.g., State Shared abstraction: multi-dimensional geo-temporal data
- 69. - Time series by event time - Flexible windowing - tumbling, sliding, conditionally triggered - Stateful processing. E.g., State per key Shared abstraction: multi-dimensional geo-temporal data
- 70. - Time series by event time - Flexible windowing - tumbling, sliding, conditionally triggered - Stateful processing - Unified stream Shared abstraction: multi-dimensional geo-temporal data
- 71. - Time series by event time - Flexible windowing - tumbling, sliding, conditionally triggered - Stateful processing - Unified stream - Real-time streams: unbounded streams Shared abstraction: multi-dimensional geo-temporal data
- 72. - Time series by event time - Flexible windowing - tumbling, sliding, conditionally triggered - Stateful processing - Unified stream - Real-time streams: unbounded streams - Batch: bounded streams Shared abstraction: multi-dimensional geo-temporal data
- 73. - Time series by event time - Flexible windowing - tumbling, sliding, conditionally triggered - Stateful processing - Unified stream - Real-time streams: unbounded streams - Batch: bounded streams - s/lambda/kappa Shared abstraction: multi-dimensional geo-temporal data
- 74. - Ordering by event time - Flexible windowing with watermark and triggers - Exactly-once semantics - Built-in state management and checkpointing - Nice data flow APIs Apache Flink
- 75. Mental Picture for Processing Geo-temporal Data
- 76. Mental Picture for Processing Geo-temporal Data
- 77. Mental Picture for Processing Geo-temporal Data
- 78. Mental Picture for Processing Geo-temporal Data
- 79. Mental Picture for Processing Geo-temporal Data
- 80. Mental Picture for Processing Geo-temporal Data
- 81. Mental Picture for Processing Geo-temporal Data
- 82. A Simple Example: simple prediction Sources .fromKafka() .config(config) .cluster(aCluster) .topics(topicList)
- 90. High Level Data Flow
- 91. High Level Data Flow
- 92. High Level Data Flow
- 93. High Level Data Flow
- 94. High Level Data Flow
- 95. High Level Data Flow
- 96. High Level Data Flow
- 97. High Level Data Flow
- 98. High Level Data Flow
- 99. High Level Data Flow
- 100. High Level Data Flow
- 101. Geotemporal API for efficiency
- 102. Geotemporal API for efficiency
- 103. Geotemporal API for productivity
- 104. Geotemporal API for productivity
- 105. Geotemporal API for productivity
- 106. Forecasting as an example
- 107. Forecasting as an example
- 108. Forecasting as an example
- 109. Forecasting as an example
- 110. Forecasting as an example
- 111. Forecasting as an example
- 112. Forecasting as an example
- 113. Forecasting as an example
- 114. Forecasting as an example
- 115. Forecasting as an example
- 116. Forecasting as an example
- 117. Forecasting as an example
- 118. Forecasting as an example
- 119. Forecasting as an example
- 120. Forecasting as an example
- 121. Forecasting as an example
- 122. Forecasting as an example
- 123. Lessons Learned - Make sure you have robust infrastructure support - Scaling up, namely single-node optimization matters - Ensure exactly-once by proper data modeling - Use external state store to avoid too much snapshotting - Standardize monitoring and data validation
- 124. Lessons Learned - Make sure you have robust infrastructure support
- 125. Lessons Learned - Make sure you have robust infrastructure support
- 126. Lessons Learned - Make sure you have robust infrastructure support
- 127. Lessons Learned - Make sure you have robust infrastructure support
- 128. Lessons Learned - Make sure you have robust infrastructure support - Scaling up, namely single-node optimization matters
- 129. Lessons Learned - Make sure you have robust infrastructure support - Scaling up, namely single-node optimization matters - Ensure exactly-once by proper data modeling
- 130. Lessons Learned - Make sure you have robust infrastructure support - Scaling up, namely single-node optimization matters - Ensure exactly-once by proper data modeling - Use external state store to avoid too much snapshotting - Standardize monitoring and data validation
- 131. Lessons Learned - Make sure you have robust infrastructure support - Scaling up, namely single-node optimization matters - Ensure exactly-once by proper data modeling - Standardize monitoring and data validation
- 132. Choose a Stream Processing Platform
- 133. Thank You