Diagnosing Problems in Production: Cassandra Summit 2014
3 likes1,428 views
The document discusses various methods for diagnosing problems in production environments, specifically within Datastax's system. It outlines preventative measures, techniques for identifying issues with consistency, compaction, and garbage collection, as well as tools for monitoring system performance. Key recommendations involve the use of OpsCenter for metrics integration and system utilities for real-time monitoring and profiling.
Diagnosing Problems in Production: Cassandra Summit 2014
- 1. Diagnosing Problems in Production Jon Haddad, Technical Evangelist, Datastax Blake Eggleston, Software Developer, Datastax ©2013 DataStax Confidential. Do not distribute without consent. 1
- 2. Preventative Measures • Opscenter • Metrics Integration • Munin • Monit • Nagios / Icinga • Graphite / Statsd (application level) • Variety of 3rd party monitoring services
- 3. Narrow Down the Problem • Weird consistency issues - NTP? • Last write wins - if servers have different time, which is the last write? • Problems with Streaming / Repair - version conflicts • Cleanup after you add nodes (reclaim disk space) • Slow queries • Compaction • Histograms • Tracing • Nodes flapping / failing • Check ops center • Dig into system metrics • JVM GC issues
- 4. Compaction • Compaction merges SSTables • Too much compaction? • Opscenter provides insight into compaction cluster wide • nodetool • compactionhistory • getcompactionthroughput • Leveled vs Size Tiered • Leveled on SSD + Read Heavy • Size tiered on Spinning rust • Size tiered is great for write heavy time series workloads
- 5. System Utilities • iostat • disk level statistics • htop • process overview • iftop & netstat • network utilities • dstat • all the above in 1 tool • strace • …for the hardcore
- 6. Histograms • proxyhistograms • High level read and write times • Includes network latency • cfhistograms <keyspace> <table> • reports stats for single table on a single node • Used to identify tables with performance problems
- 8. JVM GC Overview • What is garbage collection? • Manual vs automatic memory management • Generational garbage collection (ParNew & CMS) • New Generation • Old Generation
- 9. New Generation • New objects are created in the new gen • Minor GC • Occurs when new gen fills up • Stop the world • Dead objects are removed • Live objects are promoted into old gen • Removing objects is fast, promoting objects is slow
- 10. Old Generation • Objects are promoted to new gen from old gen • Major GC • Old generations fills up some percentage. • Mostly concurrent • 2 short stop the world pauses • Full GC • Occurs when old gen fills up or objects can’t be promoted • Stop the world • Collects all generations • These are bad!
- 11. GC Profiling • Opscenter gc stats • Look for correlations between gc spikes and read/write latency • Cassandra GC Logging • Can be activated in cassandra-env.sh • jstat • prints gc activity
- 12. GC Profiling • What to look out for: • Long, multi-second pauses • Caused by Full GCs. Old gen is filling up faster than the concurrent GC can keep up with it. Typically means garbage is being promoted out of the new gen too soon • Long minor GC • Many of the objects in the new gen are being promoted to the old gen. • Most commonly caused by new gen being too big • Sometimes caused by objects being promoted prematurely
- 13. Jon: @rustyrazorblade Blake: @blakeeggleston ©2013 DataStax Confidential. Do not distribute without consent. 13