Exactly-Once Financial Data Processing at Scale with Flink and Pinot
Download as PPTX, PDF0 likes1,177 views
The document discusses the implementation of a near real-time, exactly-once financial data processing pipeline using Kafka, Flink, and Pinot at Stripe, addressing the challenges of processing large volumes of transactions without missing or duplicating them. Key requirements include low latency and operational efficiency, with focus on deduplication strategies and handling Kafka transactions carefully to prevent data loss. Additionally, it outlines lessons learned and offers best practices for managing the architecture effectively.
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
- 1. Exactly-Once Financial Data Processing at Scale with Flink and Pinot
- 2. Speakers 2 Xiang Zhang Stripe Pratyush Sharma Stripe Xiaoman Dong StarTree
- 3. Agenda Problem: near real-time end-to-end exactly once processing pipeline at scale The architecture: Kafka, Flink, Pinot and how to connect all together Operational challenges and learnings 3 1 2 3
- 4. The problem to solve—Ledger dataset Ledger is a data set that Stripe maintains to record all money movements 4
- 5. Requirements for the Ledger pipeline Near real-time processing to meet SLO targets (p99 in orders of minutes; p90 < 1 minute) 5 1
- 6. Requirements for the Ledger pipeline Near real-time processing to meet SLO targets (p99 in orders of minutes p90 < 1 minute) Be able to process events at scale 6 1 2
- 7. Requirements for the Ledger pipeline Near real-time processing to meet SLO targets (p99 in orders of minutes p90 < 1 minute) Be able to process events at scale No missing transactions: a single transaction can be of millions of dollars 7 1 2 3
- 8. Requirements for the Ledger pipeline Near real-time processing to meet SLO targets (p99 in orders of minutes p90 < 1 minute) Be able to process events at scale No missing transactions: a single transaction can be of millions of dollars No duplicate transactions across the entire history: â—Ź Duplicates are inevitable on the source sides (deployments, restarts, accidental duplicate job executions etc.) 8 1 2 3 4
- 9. Requirements for the Ledger pipeline Near real-time processing to meet SLO targets (p99 in orders of minutes p90 < 1 minute) Be able to process events at scale No missing transactions: a single transaction can be of millions of dollars No duplicate transactions across the entire history 9 1 2 3 4 Near real-time end-to-end exactly-once processing at scale!
- 10. Agenda Problem: near real-time end-to-end exactly once processing pipeline at scale The architecture: Kafka, Flink, Pinot and how to connect all together Operational challenges and learnings 10 1 2 3
- 13. The Deduplicator 13 In reality, we store transactions IDs in Flink state for deduplication
- 14. Flink End-to-End Exactly Once Processing - Flink Deduplicator (1/3) 14 Source: https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html
- 15. Flink End-to-End Exactly Once Processing - Flink Deduplicator (2/3) 15 Source: https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html
- 16. Flink End-to-End Exactly Once Processing - Flink Deduplicator (3/3) 16 Source: https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html
- 18. Pinot Exactly Once Ingestion (1/5) 18
- 19. Pinot Exactly Once Ingestion (2/5) 19 ● Pinot table rows are stored in immutable chunks/batches called segments ● Real time segments being indexed are mutable. Once they are full they will be “sealed” and become immutable. New mutable segments will be created to continue indexing.
- 20. Pinot Exactly Once Ingestion (3/5) 20 We can consider Pinot’s latest segment as one database transaction: ● Transaction begins at segment creation ● Transaction is committed when “sealed” ● Kafka offset stored atomically along with Pinot segment metadata ● If any exception happens, the whole transaction (segment) is aborted and restarted
- 21. Pinot Exactly Once Ingestion (4/5) 21 { "segment.crc": "3251475672", "segment.creation.time": "1648231912328", "segment.download.url": "s3://some/table/mytable__8__0__20220325T1811Z", "segment.end.time": "1388707200000", "segment.flush.threshold.size": "4166", "segment.index.version": "v3", "segment.realtime.endOffset": "10264", "segment.realtime.numReplicas": "2", "segment.realtime.startOffset": "10240", "segment.realtime.status": "DONE", "segment.start.time": "1388707200000", "segment.time.unit": "MILLISECONDS", "segment.total.docs": "24" } ● Each segment has one single Zookeeper node storing its metadata ● Kafka Offsets are stored inside segment metadata ● Atomicity ○ Zookeeper node update is atomic ○ Kafka offset is updated at the same time segment status updates (“DONE”)
- 22. Pinot Exactly Once Ingestion (5/5) 22 If Pinot server is restarted or crashed â—Ź Whole segment is discarded â—Ź Segment recreated starting from the next position of offset from Segment_0 â—Ź Kafka consumer seek() is called
- 23. Agenda Problem: near real-time end-to-end exactly once processing pipeline at scale The architecture: Kafka, Flink, Pinot and how to connect all together Operational challenges and learnings 23 1 2 3
- 24. Caveats of exactly-once - nothing is free! Exactly-once is not bulletproof. Data loss or duplicate can still happen. 24 1
- 25. Caveats of exactly-once - nothing is free! Exactly-once is not bulletproof. Data loss or duplicate can still happen. It might give users a false sense of security. 25 1 2
- 26. Caveats of exactly-once - nothing is free! Exactly-once is not bulletproof. Data loss or duplicate can still happen. It might give users a false sense of security. Hard to add additional layers to the architecture due to transactional guarantee. 26 1 2 3
- 27. Caveats of exactly-once - nothing is free! Exactly-once is not bulletproof. Data loss or duplicate can still happen. It might give users a false sense of security. Hard to add additional layers to the architecture due to transactional guarantee. Latency and SLO is impacted by checkpoint intervals. 27 1 2 3 4
- 28. Potential data loss in two-phase commit 28 Source: https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html
- 29. Potential data loss in two-phase commit 29 Source: https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html The transaction can be expired in Kafka!
- 30. Optimizing large state hydration at recovery time The ledger deduplicator app maintains tens of terabytes of states to do all time deduplication 30 1
- 31. Optimizing large state hydration at recovery time The ledger deduplicator app maintains tens of terabytes of states to do all time deduplication Task local recovery doesn’t work with multiple disks mounted (FLINK-10954) ● Need to hydrate the entire state everytime the job is rescheduled (job failure, host failure/restarts/recycle) ● Impacts end to end latency 31 1 2
- 32. Optimizing large state hydration at recovery time The ledger deduplicator app maintains tens of terabytes of states to do all time deduplication Task local recovery doesn’t work with multiple disks mounted (FLINK-10954) ● Need to hydrate the entire state everytime the job is rescheduled (job failure, host failure/restarts/recycle) ● Impacts end to end latency Even if we make local recovery work, Stripe recycles hosts periodically ● The pipeline is as slow as the slowest host to recover state 32 1 2 3
- 33. Optimizing large state hydration at recovery time â—Ź Task parallelism increase to the rescue: the more threads, the faster to download the state and to rebuild the local state DB! 33 â—‹ Increasing parallelism requires state redistribution. â—‹ Flink uses the concept of key-group as an atomic unit for state distribution.
- 34. Parallelism Increase from 180 to 270 doesn’t work 34 Each task gets 2 key groups assigned 1. 180 tasks have 1 key group 2. 90 tasks have 2 key groups
- 35. What we want is even distribution 35 Each task gets 2 key groups assigned Each task gets 1 key group assigned
- 36. Monitoring Large State Size â—Ź Flink can report native RocksDB metrics. â—Ź State backend latency tracking metrics can help debugging. â—Ź Large pending Rocks DB compactions can affect performance. 36
- 37. Linux OOM Kills Causing Job Restarts â—Ź Flink < 1.12 uses glibc to allocate memory, which leads to memory fragmentation. â—Ź Combined with large states required by the deduplicator app, it consistently causes OOM. â—Ź With large number of task managers and time it takes to rehydrate state, it impacts latency SLO. 37
- 38. jemalloc Everywhere â—Ź Flink switched to jemalloc for its default memory allocator in Docker images in Flink 1.12. 38 Pre jemalloc Post jemalloc
- 39. Data Quality Monitoring ● Pinot is an analytics platform that runs SQL blazingly fast, so… ○ Duplicate detection: ■SELECT primary_key, count(*) as cnt FROM mytable GROUP BY primary_key HAVING cnt > 1 ■Run query in REALTIME only to help query performance by using special table name like mytable_REALTIME ○ Missing entry detection: ■Bucket rows by time and count by bucket ■JOIN/Compare to source of truth (upstream metric in Data Warehouse) 39
- 40. How to repair data in Pinot? â—Ź If some range of data are corrupted (contains duplicate) â—‹ Find the duplicated data by SQL query. â—‹ Delete and rebuild the Pinot segments containing duplicates. â—‹ Pinot virtual column names like $segmentName helps locating segments. â—Ź Best Practices â—‹ A reliable exactly-once Kafka Archive (backup) will come in handy in a fire. â—‹ Build stable/reliable timestamp into primary key, use that timestamp as Pinot timestamp. 40
- 41. Lessons Learned â—Ź Flink â—‹ Set a Kafka transaction timeout large enough to account for any job downtime. â—‹ Set a parallelism to a number such that max parallelism is divisible by this number. â—‹ Use jemalloc in Flink. â—Ź Pinot â—‹ Higher Kafka transaction frequency and shorter Flink checkpoint intervals will improve end to end data freshness in Pinot. â—‹ Beware of bogus message counts: Many Kafka internal metrics include messages of failed transactions. â—‹ Duplicate monitoring is a must for critical apps. 41
Editor's Notes
- #19: Transitional slides from talking about deduplicator to pinot ingestion.
- #20: Transitional slides from talking about deduplicator to pinot ingestion.
- #21: Transitional slides from talking about deduplicator to pinot ingestion.
- #22: Transitional slides from talking about deduplicator to pinot ingestion.
- #23: Transitional slides from talking about deduplicator to pinot ingestion.