Scaling Apache Storm - Strata + Hadoop World 2014
- 1. Scaling Apache Storm P. Taylor Goetz, Hortonworks @ptgoetz
- 2. About Me Member of Technical Staff / Storm Tech Lead @ Hortonworks Apache Storm PMC Chair @ Apache
- 3. About Me Member of Technical Staff / Storm Tech Lead @ Hortonworks Apache Storm PMC Chair @ Apache Volunteer Firefighter since 2004
- 4. 1M+ messages / sec. on a 10-15 node cluster How do you get there?
- 5. How do you fight fire?
- 8. Put the wet stuff on the red stuff. Water, and lots of it.
- 10. When you're dealing with big fire, you need big water.
- 11. Static Water Sources Lakes Streams Reservoirs, Pools, Ponds
- 13. Data Hydrant Active source Under pressure
- 17. How does this relate to Storm?
- 18. Little’s Law L=λW The long-term average number of customers in a stable system L is equal to the long-term average effective arrival rate, λ, multiplied by the average time a customer spends in the system, W; or expressed algebraically: L = λW. http://en.wikipedia.org/wiki/Little's_law
- 20. Batch Processing Operates on data at rest Velocity is a function of performance Poor performance costs you time
- 21. Stream Processing Data in motion At the mercy of your data source Velocity fluctuates over time Poor performance….
- 22. Poor performance bursts the pipes. Buffers fill up and eat memory Timeouts / Replays “Sink” systems overwhelmed
- 23. What can developers do?
- 24. Keep tuple processing code tight public class MyBolt extends BaseRichBolt { ! public void prepare(Map stormConf, TopologyContext context, OutputCollector collector) { // initialize task } ! public void execute(Tuple input) { // process input — QUICKLY! } ! public void declareOutputFields(OutputFieldsDeclarer declarer) { // declare output } ! } Worry about this!
- 25. Keep tuple processing code tight public class MyBolt extends BaseRichBolt { ! public void prepare(Map stormConf, TopologyContext context, OutputCollector collector) { // initialize task } ! public void execute(Tuple input) { // process input — QUICKLY! } ! public void declareOutputFields(OutputFieldsDeclarer declarer) { // declare output } ! } Not this.
- 26. Know your latencies L1 cache reference 0.5 ns Branch mispredict 5 ns L2 cache reference 7 ns 14x L1 cache Mutex lock/unlock 25 ns Main memory reference 100 ns 20x L2 cache, 200x L1 cache Compress 1K bytes with Zippy 3,000 ns Send 1K bytes over 1 Gbps network 10,000 ns 0.01 ms Read 4K randomly from SSD* 150,000 ns 0.15 ms Read 1 MB sequentially from memory 250,000 ns 0.25 ms Round trip within same datacenter 500,000 ns 0.5 ms Read 1 MB sequentially from SSD* 1,000,000 ns 1 ms 4X memory Disk seek 10,000,000 ns 10 ms 20x datacenter roundtrip Read 1 MB sequentially from disk 20,000,000 ns 20 ms 80x memory, 20X SSD Send packet CA-‐>Netherlands-‐>CA 150,000,000 ns 150 ms https://gist.github.com/jboner/2841832
- 27. Use a Cache Guava is your friend.
- 28. Expose your knobs and gauges. DevOps will appreciate it.
- 29. Externalize Configuration Hard-coded values require recompilation/repackaging. conf.setNumWorkers(3); builder.setSpout("spout", new RandomSentenceSpout(), 5); builder.setBolt("split", new SplitSentence(), 8).shuffleGrouping("spout"); builder.setBolt("count", new WordCount(), 12).fieldsGrouping("split", new Fields("word")); Values from external config. No repackaging! conf.setNumWorkers(props.get(“num.workers")); builder.setSpout("spout", new RandomSentenceSpout(), props.get(“spout.paralellism”)); builder.setBolt("split", new SplitSentence(), props.get(“split.paralellism”)).shuffleGrouping("spout"); builder.setBolt("count", new WordCount(), props.get(“count.paralellism”)).fieldsGrouping("split", new Fields("word"));
- 30. What can DevOps do?
- 31. How big is your hose?
- 32. Text Find out!
- 33. Performance testing is essential! Text
- 34. How to deal with small pipes? (i.e. When your output is more like a garden hose.)
- 36. Parallelism == Manifold Take input from one big pipe and distribute it to many smaller pipes The bigger the size difference, the more parallelism you will need
- 38. Every fire is different.
- 39. Text
- 40. Every streaming use case is different.
- 41. Sizeup — Fire What are my water sources? What GPM can they support? How many lines (hoses) do I need? How much water will I need to flow to put this fire out?
- 42. Sizeup — Storm What are my input sources? At what rate do they deliver messages? What size are the messages? What's my slowest data sink?
- 43. There is no magic bullet.
- 44. But there are good starting points.
- 45. Numbers Where to start.
- 46. 1 Worker / Machine / Topology Keep unnecessary network transfer to a minimum
- 47. 1 Acker / Worker Default in Storm 0.9.x
- 48. 1 Executor / CPU Core Optimize Thread/CPU usage
- 49. 1 Executor / CPU Core (for CPU-bound use cases)
- 50. 1 Executor / CPU Core Multiply by 10x-100x for I/O bound use cases
- 51. Example 10 Worker Nodes 16 Cores / Machine 10 * 16 = 160 “Parallelism Units” available
- 52. Example 10 Worker Nodes 16 Cores / Machine 10 * 16 = 160 “Parallelism Units” available ! Subtract # Ackers: 160 - 10 = 150 Units.
- 53. Example 10 Worker Nodes 16 Cores / Machine (10 * 16) - 10 = 150 “Parallelism Units” available
- 54. Example 10 Worker Nodes 16 Cores / Machine (10 * 16) - 10 = 150 “Parallelism Units” available (* 10-100 if I/O bound) Distrubte this among tasks in topology. Higher for slow tasks, lower for fast tasks.
- 55. Example 150 “Parallelism Units” available Emit Calculate Persist 10 40 100
- 56. Watch Storm’s “capacity” metric This tells you how hard components are working. Adjust parallelism unit distribution accordingly.
- 57. This is just a starting point. Test, test, test. Measure, measure, measure.
- 58. Internal Messaging Handling backpressure.
- 60. Key Settings topology.max.spout.pending Spout/Bolt API: Controls how many tuples are in-flight (not ack’ed) Trident API: Controls how many batches are in flight (not committed)
- 61. Key Settings topology.max.spout.pending When reached, Storm will temporarily stop emitting data from Spout(s) WARNING: Default is “unset” (i.e. no limit)
- 62. Key Settings topology.max.spout.pending Spout/Bolt API: Start High (~1,000) Trident API: Start Low (~1-5)
- 63. Key Settings topology.message.timeout.secs Controls how long a tuple tree (Spout/Bolt API) or batch (Trident API) has to complete processing before Storm considers it timed out and fails it. Default value is 30 seconds.
- 64. Key Settings topology.message.timeout.secs Q: “Why am I getting tuple/batch failures for no apparent reason?” A: Timeouts due to a bottleneck. Solution: Look at the “Complete Latency” metric. Increase timeout and/or increase component parallelism to address the bottleneck.
- 65. Turn knobs slowly, one at a time.
- 66. Don't mess with settings you don't understand.
- 67. Storm ships with sane defaults Override only as necessary
- 69. Nimbus Generally light load Can collocate Storm UI service m1.xlarge (or equivalent) should suffice Save the big metal for Supervisor/Worker machines…
- 70. Supervisor/Worker Nodes Where hardware choices have the most impact.
- 71. CPU Cores More is usually better The more you have the more threads you can support (i.e. parallelism) Storm potentially uses a lot of threads
- 72. Memory Highly use-case specific How many workers (JVMs) per node? Are you caching and/or holding in-memory state? Tests/metrics are your friends
- 73. Network Use bonded NICs if necessary Keep nodes “close”
- 75. Don’t “Pancake!” Separate concerns.
- 76. Don’t “Pancake!” Separate concerns. CPU Contention I/O Contention Disk Seeks (ZooKeeper)
- 77. Keep this guy happy. He has big boots and a shovel.
- 78. ZooKeeper Considerations Use dedicated machines, preferably bare-metal if an option Start with 3 node ensemble (can tolerate 1 node loss) I/O is ZooKeeper’s main bottleneck Dedicated disk for ZK storage SSDs greatly improve performance
- 79. Recap Know/track your latencies and code appropriately Externalize configuration Scaling is a factor of balancing the I/O and CPU requirements of your use case Dev + DevOps + Ops coordination and collaboration is essential
- 80. Thanks! P. Taylor Goetz, Hortonworks @ptgoetz