How Spotify scales Apache Storm Pipelines
46 likes6,785 views
The document outlines the architecture and operational considerations for a real-time personalization pipeline using technologies like Apache Kafka, Apache Storm, and Apache Cassandra. It emphasizes the configuration of Cassandra for optimal performance, including strategies for connection management and parallelism tuning based on task characteristics. Additionally, it discusses the trade-offs between external and in-process caching in terms of latency, memory usage, and failure risks.
How Spotify scales Apache Storm Pipelines
- 5. ○ ○ ○ ○ ○
- 9. Log Events Apache Kafka Real-time Personalization Pipeline Apache Storm User Profile Store Apache Cassandra Entity Metadata Store Apache Cassandra
- 10. ○ ○ ○ ○ ○ ○ ○ ○
- 15. ○ ○ ○ ○ ○ ○ ○ ○
- 18. ○ ○ ○ ○ ○ ○ ○ ○
- 21. ○ ○ ○ ○ ○ ○ ○ ○
- 23. ○ ○ ○ ○ ○ ○ ○ ○
- 25. ○ ○ ○ ○ ○ ○ ○ ○
- 28. ○ ○ ○ ○ ○ ○ ○ ○
- 29. ○ ○ ○ ○
- 30. ○ ○ ○ ○ ○ ○ ○ ○
- 31. Log Events Apache Kafka Real-time Personalization Pipeline Apache Storm User Profile Store Apache Cassandra Entity Metadata Store Apache Cassandra
- 34. ● Different tables for different TTLs and set gc_grace_period=0. Read repairs are disabled. ● Used DateTieredCompactionStrategy for short lived data. ● Control the number of open connections from Storm topology to Cassandra ● Configure Snitch to ensure proper call routing
- 36. ● 1 worker per node per topology ● 1 executor per core for CPU bound tasks ● 1-10 executors per core for IO bound tasks ● Compute total parallelism possible and distribute it amongst slow and fast tasks. High parallelism for slow tasks, low for fast tasks. * Parallelism tuning inspired by P Taylor Goetz’s Strata 2014 talk
- 37. ○ ○ ○ ○ ○ ○ ○ ○
- 38. ● Think about constraints in external vs in-process caching ○ External Caching ■ Network IO ■ Latency ■ Another point of failure ○ In-process ■ Limited memory ■ No persistence