Pinot: Enabling Real-time Analytics Applications @ LinkedIn's Scale
13 likes3,008 views
Pinot is an open source distributed OLAP data store designed for low latency analytics on large datasets. It is used at LinkedIn for various real-time analytics applications requiring sub-second latency on billions of events daily. Pinot uses a columnar data format, inverted indexes, encoding, and star tree indexes to enable fast filtering and aggregation. It also supports both batch and real-time ingestion from streaming data sources like Kafka.
![Site Facing Analytics
select sum(articleViewCount) from T
where articleId = x
...
and time >= y time < z
group by viewer[title|geo|industry]
• Pre-defined queries with different
filtering values
• Usually have a filter on the primary key
(e.g. articleId)
• High QPS (thousands), low latency
(< 100ms for 99%) requirements](https://image.slidesharecdn.com/apacheconpinot-190911001418/85/Pinot-Enabling-Real-time-Analytics-Applications-LinkedIn-s-Scale-33-320.jpg)
![Anomaly Detection
for d1 in [us, ca, ...]
for d2 in [chrome, firefox, ...]
...
select sum(pageViews) from T
where country = d1 and browser = d2…
group by time
Filter Aggregation
select …
where country = us …
Slow, scan 60-70% data
select …
where country = ireland …
Scan less than 1%
• Identifying issues requires monitoring
all possible combinations
• Data distribution can be skewed](https://image.slidesharecdn.com/apacheconpinot-190911001418/85/Pinot-Enabling-Real-time-Analytics-Applications-LinkedIn-s-Scale-34-320.jpg)
Pinot: Enabling Real-time Analytics Applications @ LinkedIn's Scale
- 1. Enabling Real-time Analytics Applications @ LinkedIn’s Scale Mayank Shrivastava Jackie Jiang Senior Software Engineer Seunghyun Lee Senior Software EngineerStaff Software Engineer Apache Pinot
- 2. 1 2 3 4 Agenda Introduction Pinot @ LinkedIn How to use Pinot Pinot Performance
- 3. How is data generated and used at LinkedIn Actor Verb Member Job Post Company Object Life Cycle Create Generate Analyze Product DataInsights 600+ million members Tens of million posts likes/shared per day 3+ million jobs posted per month 30 million companies Trillions of events per day
- 4. Real-time Analytics Applications at LinkedIn
- 5. How to build an online analytics application? • Real-time data ingestion • Millions of active users, 1000s of queries per sec • Super low latency (10s ms) • Highly available, always on
- 6. Approach 1. Join on the fly Event Stream Profile View Profile View Table Member Table Application Server Who viewed my profile • Real-time (depending on storage) • High latency due to join
- 7. Approach 2. Pre Join + Pre Aggregate • Near real-time ingestion • Latency varies with query selectivity Event Stream Profile View Profile View Table Member Table Application Server Who viewed my profile Stream Processing Engine Pre Join + Pre Aggr
- 8. Approach 3. Pre Join + Pre Aggregate + Pre Cube • Very fast • Batch ingestion (hourly / daily) • Storage explosion • Re-bootstrap on schema change Event Stream Profile View Profile View Table Member Table Application Server Who viewed my profile Batch Processing Engine Pre Join + Pre Aggr + Pre Cube
- 9. Latency vs. Flexibility Profile View Table Member Table Pre-Join Pre-Aggregation Pre-Cube Spark SQL Presto Hive Big Query Druid Elastic Search Pinot Kylin KV Store Latency Flexibility lowhigh lowhigh Pinot
- 10. Who Viewed My Profile @ LinkedIn Data Lake Stream Processing WVMP Dashboard Ad-hoc Queries Espresso Raw Tracking Data Pre-joined Data Pre Join + Pre Aggr
- 11. What is Apache Pinot? • OLAP Datastore • Columnar, indexed storage • Low latency analytics • Distributed – highly available, reliable, scalable • Lambda architecture ○ Offline data pushes + Real-time stream ingestion • Open Source
- 12. 1 2 3 4 Agenda Introduction Pinot @ LinkedIn How to use Pinot Pinot Performance
- 13. Pinot @ LinkedIn 70+ 2000+ 100K+ 1M+ Member Facing Use Cases Dashboards for Internal Business Metrics Queries Per Second Records Ingested Per Second
- 14. Pinot @ LinkedIn: Member Facing Analytics Report • Providing analytics reports for Linkedin member-facing applications • Very high QPS (Thousands) • Requires strict latency SLA (10s ms - sub-sec)
- 15. Pinot @ LinkedIn: Interactive Dashboard • Visualization tool for multi-dimensional metrics • Complex, explorative queries • 2000+ metrics, used by 1000+ employees
- 16. Pinot @ LinkedIn: Anomaly Detection • Efficiently detect and investigate anomalies in metrics • Third Eye: Part of Apache Pinot open source
- 17. Pinot Usage @ Other Companies
- 18. 1 2 3 4 Agenda Introduction Pinot @ LinkedIn How to use Pinot Pinot Performance
- 19. How to use Pinot Batch Data Ingestion Real-time Data Ingestion SQL-like Query Interface (PQL)
- 20. Let’s build something cool Event RSVP Data
- 21. How to use Pinot: Workflow Define Schema Define Table Configuration Create Table One Time Setup Raw Data Generate Pinot Segments Push Data Streaming Data Setup Stream Data Source Batch (Scheduled Job) Real-time (One Time Setup) Data Ingestion HDFS, S3, ADSL, NFS... Kafka, Event Hub...
- 22. How to use Pinot: Define Schema ● Schema name: meetupRsvp ● Dimension field specs ○ event_name (string) ○ event_time (long) ○ country (string) ○ city (string) ○ … ● Metrics field specs ○ rsvp_count (int) ● Time field spec ○ timestamp (long) ■ timetype: epoch / datetime ■ granularity: millisecond / second/hour/day • Dimension: an attribute of your data (filter, group by) • Metric: a number that is used to measure characteristics of a dimension (aggregation) • Time: a timestamp of an event (partitioning, retention management) SELECT event_name, sum(rsvp_count) FROM meetupRsvp WHERE country = “us” GROUP BY event_name TOP 10 Example Query - Top 10 events in US
- 23. How to use Pinot: Configure and Create Table Pinot Schema Table Config ● Table name: meetupRsvp ● Table type: batch / realtime / hybrid ● Replication factor: 2 ● Index Columns: ... ● Bloom filters: ... ● Retention: 30 days ● ... Pinot Admin Client
- 24. How to use Pinot: Batch Ingestion Raw DataRaw Data Raw Data Segment Generation Job (library) Json, CSV, Avro, Parquet, ORC... Pinot Schema Table Config Pinot Segment Pinot Segment Pinot Segment HDFS, S3, ADLS, NFS... HDFS, S3, ADLS, NFS...
- 25. How to use Pinot: Batch Ingestion Raw Data Segment Generation Job (library) Json, Avro, Parquet, ORC... Pinot Schema Table Config Pinot Segment Pinot Segment Pinot Segment Segment Push Job (library) HDFS, S3, ADLS, NFS... HDFS, S3, ADLS, NFS...
- 26. How to use Pinot: Segment Assignment Segment Push Job Controller Helix Zookeeper Server-0 Server-1 Server-2 Pinot • Assignment strategies ○ Uniform ○ Replica Group ○ Partition Aware Segment Store S0 S2S1 HDFS, S3, ADLS, NFS... ● S0: Sever-0, Server-1 ● S1: Server-1, Server-2 ● S2: Server-0, Server-2 S0 S2 S1 S0 S2 S1 1. Table name 2. Segment name 3. Segment URI path
- 27. How to use Pinot: Query Routing Segment Push Job Controller Helix • Routing Strategies ○ Uniform ○ Replica Group ○ Partition Aware Broker Queries Segment Store S0 S2S1 HDFS, S3, ADLS, NFS... Server-0 Server-1 Server-2 Pinot S0 S2 S1 S0 S2 S1
- 28. How to use Pinot: Batch + Realtime Segment Push Job Controller Helix Real-time Servers Offline Servers Broker Queries Pinot Streaming Data Kafka, Event Hub, Kinesis... Table Config ● Table name: meetupRsvp ● Table type: real-time ● Replication factor: 2 ● Kafka broker: ... ● Kafka topic name: ... ● Retention: 5 days ● ... • A single schema for both offline + real-time tables
- 29. How to use Pinot: Batch + Realtime Segment Push Job Controller Helix Real-time Servers Offline Servers Broker Queries Pinot Streaming Data Kafka, Event Hub, Kinesis... • Real-time servers keep consumed data in memory, periodically flush data to segment store. • Broker handles offline and real-time federation.
- 30. Quick Demo Event RSVP Data
- 31. 1 2 3 4 Agenda Introduction Pinot @ LinkedIn How to use Pinot Pinot Performance
- 32. Interactive Dashboard select sum(pageView) from T where country = us and browser = chrome ... group by time • Human-driven queries • Slice and dice over arbitrary dimensions 5000 Queries Pinot Druid Total Time 11 minutes 24 minutes P50 84ms 136ms P90 206ms 667ms
- 33. Site Facing Analytics select sum(articleViewCount) from T where articleId = x ... and time >= y time < z group by viewer[title|geo|industry] • Pre-defined queries with different filtering values • Usually have a filter on the primary key (e.g. articleId) • High QPS (thousands), low latency (< 100ms for 99%) requirements
- 34. Anomaly Detection for d1 in [us, ca, ...] for d2 in [chrome, firefox, ...] ... select sum(pageViews) from T where country = d1 and browser = d2… group by time Filter Aggregation select … where country = us … Slow, scan 60-70% data select … where country = ireland … Scan less than 1% • Identifying issues requires monitoring all possible combinations • Data distribution can be skewed
- 35. Secret behind Pinot Aggregation Filter Storage Scan Star-Tree Pre-aggregation Scan Inverted Index Columnar Store Encoding/Compression Sorted Index Star-Tree Index ❏ Common Techniques ❏ Pinot & Druid ❏ Pinot Only select sum(pageView) from T where country = us and browser = chrome
- 36. Columnar Store • Read relevant columns only country browser ... us chrome ... ca firefox ... jp ie ... us firefox ... ca ie ... … … ... Raw Data Row Based Column Based Aggregation Filter Storage select sum(pageView) from T where country = us and browser = chrome Columnar us chrome ... ca firefox ... jp ie ... country us ca jp us ca … browser chrome firefox ie firefox ie … ... ... ... ... ... ... ...
- 37. Encoding & Compression Dictionary Forward Index country ca jp us … browser chrome firefox ie … country 2 0 1 2 0 ... browser 0 1 2 1 2 ... • Storage compression ○ Dictionary encoding ○ Bit compression Aggregation Filter Storage Encoding/Compression select sum(pageView) from T where country = us and browser = chrome Column Based country us ca jp us ca … browser chrome firefox ie firefox ie … docId 0 1 2 3 4 … docId 0 1 2 3 4 ... dictId 0 1 2 …
- 38. Inverted Index docId country browser 0 us chrome 1 ca firefox 2 jp ie 3 us firefox 4 ca ie … … … Raw Data country docIds ca 1, 4... jp 2... us 0, 3... ... ... Inverted Index browser docIds chrome 0 ... firefox 1, 3... ie 2, 4... ... ...• Storing bitmap for each value • Fast filtering： ○ Constant time value lookup ○ Bit operations for AND/OR clause Aggregation Filter Storage Inverted Index select sum(pageView) from T where country = us and browser = chrome
- 39. Sorted Index • Better data compression: ○ Run length encoding ○ Can be accessed as forward/inverted index • Spatial locality country start docId end docId ca 0 80 jp 81 100 us 101 300 … … … docId country 0 ca ... … 100 jp 101 us … … 300 us … … sorted index inverted index Aggregation Filter Storage Sorted Index select sum(pageView) from T where country = us and browser = chrome
- 40. Latency vs. Space Trade-off latency space requirement scan pre-cubeStar-Tree select sum(pageView) from T where country = us and browser = chrome Aggregation Filter Storage Star-Tree Pre-aggregation Star-Tree Index
- 41. Star-Tree Index latency space requirement T=infinity T=1,000,000 T=10,000 T=100 T=1 • Configurable trade-off between latency and space by partial pre-aggregation technique • Be able to achieve a hard upper bound for query latencies
- 42. Star-Tree Index
- 43. Flexible Query Execution Plan Query Optimization select max(col) from T Use metadata instead of scanning select sum(metric) from T where country = us and accountId = x Reorder filter based on the available indexes (apply accountId before country predicate) Segment level physical query planner can intelligently choose the best way to solve the query based on the segment metadata and available indexes.
- 44. Global Optimizations Problem Solution Querying all segments Segment pruning to minimize the number of segments to query Querying all servers Smart segment assignment to reduce the fan-out to servers
- 45. Conclusion User Activity Data Member Facing Applications Interactive Dashboard Anomaly Detection
- 46. Contributing to Pinot • We are looking for contributions! • Apache Pinot (incubating) 0.1.0 is available at https://pinot.apache.org • Pinot Twitter Account https://twitter.com/ApachePinot • Pinot Meetup Page https://www.meetup.com/apache-pinot • Pinot Slack Channel https://tinyurl.com/pinotSlackChannel
- 47. Folks behind Pinot Mayank Shrivastava Subbu Subramaniam Jean-Francois Im Jackie Jiang Seunghyun Lee Jennifer Dai Neha Pawar Jialiang Li Sunitha Beeram Shraddha Sahay Kishore Gopalakrishna Xiang Fu James Shao Prasanna Ravi John Gutmann Dino Occhialini Walter Huf Xiaohui Sun Long Huynh Akshay Rai Alexander Pucher Jihao Zhang Felix Cheung Olivier Lamy Jim Jagielski Marcel Siegrist Roman Shaposhnik Anurag Shendge
- 48. Thank you