Real-Time Analytics: Going Beyond Stream Processing With Apache Pinot
0 likes121 views
Apache Pinot is a real-time analytics platform designed for user-facing applications. It provides excellent query performance on large, mutable datasets by using columnar storage and optimized indexing. Pinot handles real-time data ingestion from streaming and batch sources while supporting updates, deletes, and semi-structured data. It also offers a hybrid architecture that separates compute from storage for cost efficiency and scales from gigabytes to petabytes of data. Pinot aims to provide a full SQL interface and one platform for building various real-time analytics use cases with low latency and strong data consistency.
Real-Time Analytics: Going Beyond Stream Processing With Apache Pinot
- 1. Real-time Analytics Rong Rong Software Engineer - Going Beyond Stream Processing with Apache Pinot
- 2. Evolution of Real-Time Analytics
- 3. Dashboards / BI Tools
- 5. Total users 700 Million QPS 10000+ Latency SLA < 100 ms p99th Freshness Seconds Actionable Insights
- 6. Missed orders Inaccurate orders Downtime Top selling items Menu item Feedback Actionable Insights Restaurant Dashboard
- 8. ● Real-Time analytics landscape is rapidly changing. ● OnLine Analytical Processing (OLAP) system are evolving with these trends. Internal-Facing Analytics Structured Data Approximate Data / Query Consistency Slice-and-Dice Queries User-Facing Analytics Semi-Structured Data Strong Data / Query Consistency Full-SQL semantics GB-to-TB of Data TB-to-PB of Data Evolution of Real-Time Analytics
- 9. Better Ingestion Better Queries Better Scalability Requirements are changing!
- 10. Intro to Apache Pinot
- 11. What is Apache Pinot?
- 12. Table Pinot Data Segments Col1 .. ca jp us ca … ColN .. … … … … … . . . + Indexes Metadata Columnar Store
- 16. Tailored for User-Facing Analytics
- 17. Efficient Pruning
- 18. browser chrome firefox ie firefox ie … ... ... ... ... ... ... ... country browser ... us chrome ... ca firefox ... jp ie ... us firefox ... ca ie ... … … ... Raw Data Column Based country us ca jp us ca … 1 select count(*) from X where country = us Country Browser Star-Tree Index US CA IE C IE C Optimized Index Algorithm
- 19. Filtering Optimizations Inverted Index Sorted Index Range Index JSON Index Geo Index Text Index Aggregation Optimizations Theta Sketches StarTree Index HyperLogLog Optimized Index Algorithm
- 21. Filtering Optimizations Inverted Index Sorted Index Range Index JSON Index Geo Index Text Index Aggregation Optimizations Theta Sketches StarTree Index HyperLogLog Support Semi-Structured Data
- 22. OLAP Real-Time Insert OLAP Best Effort Queries (Double Counting) Accurate Queries Real-Time Insert + Updates + Deletes Support Strong Data Consistency
- 23. Support Strong Data Consistency UPSERT data ingestions Real-Time Stream K=1, V=5 K=1 ,V=1 S3 S4 S1 S2 Key Index Insert: (K=1, S=S1, Row=1) Key Index Update: (K=1, S=S1, Row=1) K=3, V=4 K=2, V=3 K=2, V=2 Key Index Insert: (K=2, S=S2, Row=2) Key Index Update: (K=2, S=S3, Row=2) Key Index Insert: (K=3, S=S4, Row=1)
- 24. OLAP OLAP Queries (Aggregations, Order By, Group By, UDFs, …) Full SQL Semantics ! (OLAP Queries + Joins + Nested Queries + Window functions + …) OLAP Towards full-SQL semantics
- 26. Local Storage Compute OLAP Data Nodes Cloud Storage $$$$ $$ TBs - PBs of Data Local Storage Compute OLAP Data Nodes $ GBs - TBs of Data Compute & Storage Decoupling
- 27. Pinot Servers Pinot Brokers Pinot Segments Compute & Storage Decoupling
- 28. Pinot Servers Cloud Storage Pinot Brokers Pinot Segments Compute & Storage Decoupling
- 29. Summary
- 30. ● Real-time analytics has evolved to power more user-facing, actionable applications. ● Apache Pinot is our solution for real-time analytics. Excellent Query Performance Handles real-time data mutation Ingest from Streaming, Batch, SQL sources Semi-Structured Data support Hybrid architecture for cost efficiency One platform for building all kinds of use cases Takeaway