Storm at Spotify
27 likes10,571 views
This document summarizes Neville Li's work at Spotify developing real-time data streaming applications using Storm. It describes Spotify's large data volumes, how Storm is used to process streaming data at Spotify, details of a social listening topology, and lessons learned around development processes, language choices, and deployment.
![Metadata Decoration Bolt
•tuple.getStringByField(“trackId”)!
•Append fields in output tuple
•[input fields…, “artistId”, “albumId”]!
•Input fields as constructor argument
•Reuse!
monadic!](https://image.slidesharecdn.com/storm-140325191734-phpapp02/85/Storm-at-Spotify-29-320.jpg)
Storm at Spotify
- 1. March 25, 2014 Neville Li neville@spotify.com @sinisa_lyh Storm at Spotify
- 2. •@Spotify since 2011 •Recommendation Team •Data & Backend •Storm, Scalding, Spark, Scala… About Me
- 3. March 25, 2014 Spotify in numbers Started in 2006, available in 55 markets 20+ million songs, 20,000 added per day 24+ million active users, 6+ million subscribers 1.5 billion playlists !
- 4. Big Data @spotify 600 node cluster Every day •400GB service logs •4.5TB user data •5,000 Hadoop jobs •61TB generated
- 5. March 25, 2014 What is Storm? In data-layman’s terms • Real time stream processing • Like Hadoop without HDFS • Like Map/Reduce with many reducer steps • Fault tolerant & guaranteed message processing Photo © Blaine Courts http://www.flickr.com/photos/blainecourts/8417266909/
- 6. Storm @spotify •storm-0.8.0 •22 node cluster •15+ topologies •200,000+ tuples per second •recommendation, ads, monitoring, analytics, etc.
- 7. “Never Gonna Give You Up” Rick Astley Map ! First Storm Application @Spotify 7
- 8. RT Market Launch Stats
- 9. Other Uses •Trending tracks •Email campaign •App performance tracking •UX tracking
- 10. Anatomy of A Storm Topology From play to recommendation
- 11. Social Listening Take 1 •PUB/SUB •Almost real-time •Spammy •Hard to scale All characters appearing in this work are fictitious. Any resemblance to real persons, living or dead, is purely coincidental. this
- 12. guy
- 13. again!
- 14. Social Listening Take 2 •Hadoop daily batch •High latency •M/R aggregation •Easier to scale
- 15. Social Listening Revamped •Kafka → Storm → Backend •Soft real-time •Aggregate trigger bolt •Easy to scale
- 16. Getting Data accesspoint playlist search storage social kafka
- 17. What are we transferring? •TSV logs with version type (moving to Avro) •Centralized Schema Repository •Parsers in Python Java •Log parsing splitting by topic in Kafka EndSong 21 username:Str timestamp:Int trackId:Str msPlayed:Int reasonStart:Str reasonEnd:Str … ClientEvent 15 username:Str platform:Str timestamp:Int jsonData: Str …
- 18. March 25, 2014 Getting Data Across the Globe Photo © Riley Kaminer http://www.flickr.com/photos/rwkphotography/3282192071/ Ashburn London Stockholm San
- 22. filter kafka
- 23. spout metadata
- 25. trigger privacy
- 26. filterZMTP
- 28. EndSong Filter Bolt •Discard some tuples –Skipped –Too short •Keep some fields –Context –Reasons
- 29. Metadata Decoration Bolt •tuple.getStringByField(“trackId”)! •Append fields in output tuple •[input fields…, “artistId”, “albumId”]! •Input fields as constructor argument •Reuse! monadic!
- 30. Async Batch RPC metadata tuple batch callback queueupstream REQ REP emit ack bolt
- 32. thread
- 33. Listening Trigger Bolt •Rule based triggers –High intent –Repeats •In heap LRU cache –Repeat counter –Rate limiting
- 34. •Similar to metadata bolt •Async lookup •Ack all, emit some •In heap LRU cache •Cache private cases only Privacy Bolt
- 37. service •[uri, username, payload] •DNS SRV for discovery •ZMQ PUB socket on bolt •1+ subscribers •1+ redundant bolts •Tools for testing
- 39. March 25, 2014 Development Process One git repository One storm-shared sub project → jar → Artifactory Many storm-team/application subprojects Sampled log for local development Turnable params in config files
- 40. Problem Factory topology storm •lein uberjar (or mvn) •maven-shade-plugin •relocate com.google.* guava
- 41. 10guava
- 43. March 25, 2014 Language Choices Java for boring stuff - Cassandra, memcaced, RPC, etc. ! Clojure for fun stuff - algorithm heavy ! Scala - summingbird?
- 44. March 25, 2014 Deployment Puppet Shared cluster right now One per team in the future YARN? ! Monitoring from service side Photo © Ian Koppenbadger http://www.flickr.com/photos/runnerone/3391661946/
- 45. March 25, 2014 Thank You
- 46. March 25, 2014 Want to join the band? Check out http://www.spotify.com/jobs or @Spotifyjobs for more information. ! Neville Li neville@spotify.com @sinisa_lyh