Open analytics meetup alex poon (1)
0 likes574 views
Storm @ Visual Revenue provides concise summaries of key points about Visual Revenue's use of Storm for real-time data processing. They process 14 billion page views per month on a 50 node Storm cluster on AWS. They previously built their own system using ZMQ and batch processing but found Storm and Kafka to be more scalable, fault tolerant, and efficient. Their data aggregation topologies in Storm collect and transform customer data in real-time to power dashboards and algorithms.
Open analytics meetup alex poon (1)
- 1. Storm @ Visual Revenue (an Outbrain Company) Alex Poon VP of Engineering
- 2. Who are we?
- 3. What we do? Customer Traffic • 14B page views per month • At peak, 8000-10000 per sec Web Servers • Deployed Storm to production ~ 1 Ka=a month ago Data Transform/ Aggrega8on • Storm cluster of ~50 instances on Storm AWS Databases Dashboard Algo Automa8on
- 4. Before Storm • Built our own distributed data processing • ZMQ • Batch based process • Hashing processing by customers • Advantages • Simple in-house system built from very basic components • Well understood • Disadvantages • Hard to scale, constant battle for keeping up with pings • Machine management was clumsy • Uneven distribution of traffic • Multiple processes doing similar work, wasting resources
- 5. Why Kafka/Storm? • Kafka • open-sourced, distributed publish-subscribe messaging system • Storm • open-sourced, real-time computation system for continuous computation • They are awesome • Distributed, highly scalable, and fault tolerance • High throughput • Reliable • Real-time • Great at in-memory analytics, and real-time decision support
- 6. DataAggregation Customer 15s Position Front Page 15s 15s URL Aggregate 15s Aggregate Arrangement 5m 5m Spout Tweet @Handle Bolt 15s 15s
- 7. Learning / Ideas 1. Kafka + zookeeper is extremely scalable and easy to setup. Check out the Brod library if you are doing Python 2. Use the Storm UI (Ganglia based) to monitor your cluster 3. Shell Bolts were inefficient and hard to debug (at least for us) 4. Upgrade to at least Storm version 0.8.2 which gives you capacity metrics on top of other goodies 5. Storm’s anchoring/replay capability is awesome but comes with a visible overhead 6. Use a good framework to manage your cluster, we use Salt Stack 7. Our unit tests are built in Junit. Most built in unit tests for Storm are only available in Clojure for now
- 8. Thank You Alex Poon @alexpoon06 @Outbrain Yes, it is true. We are Hiring!! www.visualrevenue.com/jobs