Mitigating the Impact of State Management in Cloud Stream Processing Systems
Download as PPTX, PDF0 likes115 views
The document discusses challenges in state management for cloud stream processing systems, highlighting RisingWave Labs and its distributed SQL streaming database. It explores existing solutions, proposes a tiered storage approach to optimize performance, and suggests strategies like remote compaction to alleviate latency issues. Ultimately, it emphasizes the importance of a decoupled compute-storage architecture and careful management of cloud resources to enhance scalability and efficiency.
Mitigating the Impact of State Management in Cloud Stream Processing Systems
- 1. HOSTED BY Mitigating the Impact of State Management in Cloud Stream Processing Systems Yingjun Wu Founder at RisingWave Labs
- 2. Yingjun Wu (he/him/his) Founder at RisingWave Labs ■ Chief Everything Officer ■ Ex-AWS Redshift ■ Ex-IBM Research Almaden ■ PhD in database systems and stream processing
- 3. What is RisingWave? ■ A distributed SQL streaming database ■ Open sourced in April 2022 under Apache 2.0 License >5KGitHub stars >130GitHub contributors >1KSlack members >100KK8s deployments
- 4. Background
- 5. Background
- 6. Background
- 7. Background
- 8. ■ Stream processing systems continuously process large volume of data Background
- 9. ■ Stream processing systems continuously process large volume of data ● Need to maintain internal states for stateful operators such as joins and aggregations Background
- 10. ■ Consider joining two data streams ● Impression stream ● Click stream State Management
- 11. ■ Consider joining two data streams ● Impression stream ● Click stream State Management
- 12. ■ Consider joining two data streams ● Impression stream ● Click stream State Management
- 13. State Management: Existing Solutions ■ MapReduce style ■ Compute-storage coupled
- 14. ■ MapReduce style ■ Compute-storage coupled State Management: Existing Solutions
- 15. State Management: Existing Solutions ■ MapReduce style ■ Compute-storage coupled
- 16. State Management: Existing Solutions ■ MapReduce style ■ Compute-storage coupled
- 17. State Management: the Cloud Era ■ Cloud-native style ■ Compute-storage decoupled
- 18. State Management: the Cloud Era ■ Cloud-native style ■ Compute-storage decoupled +
- 19. ■ Consider joining two data streams ● Impression stream ● Click stream State Management in the Cloud
- 20. ■ Consider joining two data streams ● Impression stream ● Click stream State Management in the Cloud
- 21. ■ Consider joining two data streams ● Impression stream ● Click stream State Management in the Cloud
- 22. ■ Consider joining two data streams ● Impression stream ● Click stream State Management in the Cloud
- 23. How to mitigate the issue?
- 24. Tiered Storage ■ EC2 local storage: the “cloud cache” ● Super fast! ● Data will get lost if machine crashes ■ EBS: the “cloud disk” ● Fast ● 99.999% durability (5 nines) ■ S3: the persistent storage ● Slow ● 99.999999999% durability (11 nines)
- 25. Tiered Storage ■ EC2 local storage: the “cloud cache” ● Super fast! ● Data will get lost if machine crashes ■ EBS: the “cloud disk” ● Fast ● 99.999% durability (5 nines) ■ S3: the persistent storage ● Slow ● 99.999999999% durability (11 nines)
- 26. Tiered Storage ■ Should we really leverage EBS?
- 27. Tiered Storage ■ Should we really leverage EBS? ● No, at least for now…
- 28. Tiered Storage ■ Should we really leverage EBS? ● No, at least for now… ■ EC2 local storage vs EBS ● EBS is more expensive than EC2 local storage ● EBS is slower than EC2 local storage ● EBS is persistent while EC2 local storage is not
- 29. Tiered Storage ■ How to manage data at different layers?
- 30. Log Structured Merge Tree ■ Use log structured merge tree! ● Recently accessed data will be cached in local machine ● Upper level runs will be periodically compacted to lower levels
- 31. Compaction ■ Compactions in LSM trees can cause huge latency spikes!
- 32. Compaction ■ Compactions in LSM trees can cause huge latency spikes! ● Move the compaction to remote machines
- 33. Compaction ■ Compactions in LSM trees can cause huge latency spikes! ● Move the compaction to remote machines
- 34. Compaction ■ Compactions in LSM trees can cause huge latency spikes! ● Move the compaction to remote machines Build SST Upload SST
- 35. Compaction ■ Compactions in LSM trees can cause huge latency spikes! ● Move the compaction to remote machines Build SST Upload SST
- 36. Compaction ■ Accessing S3 is always expensive!
- 37. Compaction ■ Accessing S3 is always expensive!
- 38. Compaction ■ Accessing S3 is always expensive! S3 buckets
- 39. Compaction ■ Accessing S3 is always expensive! ● Each bucket has performance limit! S3 buckets
- 40. Compaction ■ Accessing S3 is always expensive! ● Each bucket has performance limit! ● Using too many buckets can cause fragmentation issues! ■ We set #buckets to a magic number and scale based on workloads. S3 buckets
- 41. Conclusion ■ State management is a challenging problem in stream processing systems. ■ Decoupled compute-storage architecture helps achieve infinite and independent scalability. ■ Tiered storage helps optimize performance. ■ Using remote compaction and streaming compaction to optimize network. ■ Set number of buckets wisely to reduce S3 access bottleneck.
- 42. Yingjun Wu @YingjunWu risingwave.com/github risingwave.com/slack Thank you! Let’s connect.