Optimizing Your Cluster with Coordinator Nodes (Eric Lubow, SimpleReach) | Cassandra Summit 2016
5 likes3,717 views
Eric Lubow discusses the optimization of Cassandra clusters using 'fat clients' to improve performance while reducing the number of nodes. The presentation highlights the differences between thick and thin clients, detailing how fat clients enhance coordination without storing data, leading to increased efficiency and lower resource requirements. Key results from implementing this approach include a reduction in nodes from 72 to 30 and a significant boost in capacity and performance.
Optimizing Your Cluster with Coordinator Nodes (Eric Lubow, SimpleReach) | Cassandra Summit 2016
- 1. SOMETHING SOMETHING COORDINATOR NODES Cheating Our Way to Better Performance
- 2. JON HADDAD LEARN DATA MODELING BY EXAMPLE THIS IS AWESOME!! GO TO ROOM 210A NOW!
- 3. Eric Lubow @elubow #CassandraSummit PERSONAL VANITY àč CTO of SimpleReach àč Co-Author of Practical Cassandra àč Skydiver, Mixed Martial Artist, Motorcyclist, Dog Dad (IG: @charliedognyc), NY Giants fan
- 4. Eric Lubow @elubow #CassandraSummit SIMPLEREACH àč Help marketers organize àč Identify the best content àč Use engagement metrics àč WorkïŹow solution àč Stream processing ingest àč Many metrics, time sliced àč Multiple data stores
- 5. Eric Lubow @elubow #CassandraSummit CONCEPTS YOU SHOULD UNDERSTAND 1. Thick clients and thin clients 2. CPU utilization and load average 3. Database tuning may not have anything to do with the database
- 6. Eric Lubow @elubow #CassandraSummit A fat client (also called heavy, rich, or thick client) is a computer (client) in clientâserver architecture or networks that typically provides rich functionality independent of the central server. â Wikipedia WHAT IS A FAT CLIENT?
- 7. Eric Lubow @elubow #CassandraSummit âą Thin clients typically canÊŒt operate without the âserverâ âą Thick clients try to do more locally compared to thin clients which try to do more remotely âą Thick clients require more resources, but fewer servers. âą Thin clients require fewer resources and more servers. THIN CLIENTS AND THICK CLIENTS
- 8. Eric Lubow @elubow #CassandraSummit âą Fat clients have the Cassandra binary, but no data âą Data nodes are denser and more focused on storage âą For context, we can call them proxy nodes âą Proxy nodes are more compute heavy âą Fat clients only handles coordination ONCE MORE, BUT WITH CASSANDRA
- 9. Eric Lubow @elubow #CassandraSummit àč Fat clients are effectively just changing settings àč < Cassandra 2.2 -Djoin_ring=false (hack) àč No data on the nodes, just coordination responsibility àč Intentionally sidestepping Cassandra homogenous nature in favor of performance àč Can be lots of room for adding proxy nodes without incurring additional performance loss from increasing the ring size àč Reduces per node work on the data nodes WHATâS REALLY GOING ON HERE?
- 10. Eric Lubow @elubow #CassandraSummit NORMAL CASSANDRA SETUP CASSANDRA CLUSTERAPPLICATION TIER C* App 1 App 2 C* C* C* C* C* C* C*
- 11. Eric Lubow @elubow #CassandraSummit
- 12. Eric Lubow @elubow #CassandraSummit CASSANDRA PROXY TIER SETUP CASSANDRA CLUSTER App 1 App 2 PROXY TIER Proxy Proxy Data Data Data Data Data Data Data Data APPLICATION TIER Data Nodes: c3.4xlargeProxy Nodes: c3.4xlarge
- 13. Eric Lubow @elubow #CassandraSummit àč More compute power for token calculations àč More compute power for writing data àč More focused compute on coordination tasks àč Smarter allocation of instance types àč Cheaper hardware for proxy instances TRADEOFFS àč More instance types to manage àč More infrastructure overhead àč Requires different monitoring àč High potential for nasty accident (forget to make proxy node)
- 14. Eric Lubow @elubow #CassandraSummit Before Before After After AVERAGE CLUSTER CPU UTILIZATION
- 15. Eric Lubow @elubow #CassandraSummit HOW DID WE DO?
- 16. Eric Lubow @elubow #CassandraSummit àč Why are we talking about CPU utilization and load average àč Terminology is important àč Understanding gains/losses is important àč LetÊŒs talk about CPU utilization and load average HOW DO WE KNOW?
- 17. Eric Lubow @elubow #CassandraSummit àč LetÊŒs use a trafïŹc analogy for load average àč Imagine you are the bridge operator of a single lane bridge (single CPU): àč 0.00 means there's no trafïŹc on the bridge at all. In fact, between 0.00 and 1.00 means there's no backup, and an arriving car will just go right on. àč 1.00 means the bridge is exactly at capacity. All is still good, but if trafïŹc gets a little heavier, things are going to slow down. àč over 1.00 means there's backup. How much? Well, 2.00 means that there are two lanes worth of cars total -- one lane's worth on the bridge, and one lane's worth waiting. 3.00 means there are three lane's worth total -- one lane's worth on the bridge, and two lanes' worth waiting. Etc. àč Best load average for a single CPU system is between 0.7 and 0.8 (headroom) àč Different for multi-core systems WHAT IS LOAD AVERAGE?
- 18. Eric Lubow @elubow #CassandraSummit àč Each core on a CPU has itÊŒs own utilization graph àč CPU utilization isnÊŒt straight forward àč Assume you have a single core processor ïŹxed at a frequency of 2.0 GHz. CPU utilization in this scenario is the percentage of time the processor spends doing work (as opposed to being idle). If this 2.0 GHz processor does 1 billion cycles worth of work in a second, it is 50% utilized for that second. àč Current multiple cores processors exist with dynamically changing frequencies, hardware multithreading, and shared caches all of which effect reporting. àč Resource sharing makes monitoring CPU utilization difïŹcult NOTES ABOUT CPU UTILIZATION
- 19. Eric Lubow @elubow #CassandraSummit
- 20. Eric Lubow @elubow #CassandraSummit Before Before After After AVERAGE CLUSTER CPU UTILIZATION
- 21. Eric Lubow @elubow #CassandraSummit àč Batches work better when there is a coordinator dispatching batches to the correct data node without additional processing on the part of the data node àč One of the many downsides of vnodes is massive coordination requirements àč Removing coordination responsibilities from data nodes makes them more performant àč less context switching àč less network trafïŹc/gossip/GC àč less CPU utilization WHAT ACTUALLY HAPPENED 1/2
- 22. Eric Lubow @elubow #CassandraSummit àč 30 nodes * 256 tokens/node = 7,680 token ranges àč Queries go through a nearly 8,000 item list, slow, context switch, lots of GCable objects àč Considering just reads, at 30k requests per second, this is a signiïŹcant reduction in work on a per query basis àč We are able to tune the JVMs differently WHAT ACTUALLY HAPPENED 2/2
- 23. Eric Lubow @elubow #CassandraSummit RESULTS
- 24. Eric Lubow @elubow #CassandraSummit àč Went from 72 nodes down to 30 nodes àč All data is now stored on AWS ST1 EBS volumes àč Works best for write heavy workloads àč Roughly 300% increase in available and burstable capacity àč Less footprint to watch over; fewer machines, more roles RESULTS
- 25. Eric Lubow @elubow #CassandraSummit àč Command line option or cassandra.yaml option for coordinator only mode àč Code path short cuts for performance àč SpeciïŹc JMX beans around query coordination àč Allow query mutation by coordinator nodes (Lua?) FEATURE NOT HACK
- 26. Eric Lubow @elubow #CassandraSummit WHAT DID I SAY? àč Fat clients can save you money àč DonÊŒt start out with complexity àč Know the basics àč Know what your baseline measurements are àč Monitor everything àč Sometimes database tuning doesnÊŒt require making changes to the database
- 27. Eric Lubow @elubow #CassandraSummit QUESTIONS IN LIFE ARE GUARANTEED, ANSWERS ARENâT. Eric Lubow @elubow