Cassandra Tuning - Above and Beyond (Matija Gobec, SmartCat) | Cassandra Summit 2016
Download as PPTX, PDF12 likes4,242 views
The document outlines tuning methods for Cassandra aimed at achieving high availability and performance. It emphasizes the importance of proper investigation, hardware configuration, JVM and Cassandra settings, as well as effective metrics and testing methodologies. Additionally, it discusses AWS deployment strategies and considerations for achieving optimal performance.
Cassandra Tuning - Above and Beyond (Matija Gobec, SmartCat) | Cassandra Summit 2016
- 1. Cassandra tuning - above and beyond Matija Gobec Co-founder & Senior Consultant @ SmartCat.io
- 2. © DataStax, All Rights Reserved. Why this talk We were challenged with an interesting requirement… “99.999%” 2
- 3. © DataStax, All Rights Reserved. 1 Initial investigation and setup 2 Metrics and reporting 3 Test setup 4 AWS deployment 5 Did we make it? 3
- 4. © DataStax, All Rights Reserved. What makes a distributed system? A bunch of stuff that magically works together 4
- 5. © DataStax, All Rights Reserved. How to start? Investigate the current setup (if any) Understand your use case Understand your data Set a base configuration Define target performance (goal) 5
- 6. © DataStax, All Rights Reserved. Initial investigation • What type of deployment are you working with? • What is the available hardware? • CPU cores and threads • Memory amount and type • Storage size and type • Network interfaces amount and type • Limitations 6
- 8. © DataStax, All Rights Reserved. Hardware configuration 8-16 cores 32GB ram Commit log SSD Data drive SSD 10GbE Placement groups Availability zones Enhanced networking 8
- 9. © DataStax, All Rights Reserved. OS - Swap, storage, cpu 1. Swap is bad • remove swap from stab • disable swap: swapoff -a 2. Optimize block layer • echo 1 > /sys/block/XXX/queue/nomerges • echo 8 > /sys/block/XXX/queue/read_ahead_kb • echo deadline > /sys/block/XXX/queue/scheduler 3. Disable cpu scaling 9
- 10. © DataStax, All Rights Reserved. sysctl.d - network net.ipv4.tcp_rmem = 4096 87380 16777216 net.ipv4.tcp_wmem = 4096 65536 16777216 net.ipv4.tcp_ecn = 0 net.ipv4.tcp_window_scaling = 1 net.ipv4.ip_local_port_range = 10000 65535 net.ipv4.tcp_tw_recycle = 1 net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.core.somaxconn = 4096 net.core.netdev_max_backlog = 16384 10 # read buffer space allocatable in units of pages # write buffer space allocatable in units of pages # disable explicit congestion notification # enable window scaling (higher throughput) # allowed local port range # enable fast time-wait recycle # max socket receive buffer in bytes # max socket send buffer in bytes # number of incoming connections # incoming connections backlog
- 11. © DataStax, All Rights Reserved. sysctl.d - vm and fs 11 vm.swappiness = 1 vm.max_map_count = 1073741824 vm.dirty_background_bytes = 10485760 vm.dirty_bytes = 1073741824 fs.file-max = 1073741824 vm.min_free_kbytes = 1048576 # memory swapping threshold # max memory map areas a process can have # dirty memory amount threshold (kernel) # dirty memory amount threshold (process) # max number of open files # min number of VM free kilobytes
- 12. © DataStax, All Rights Reserved. JVM - CMS MAX_HEAP_SIZE=“8G" # Good starting point HEAP_NEWSIZE=“2G" # Good starting point JVM_OPTS="$JVM_OPTS -XX:+PerfDisableSharedMem" JVM_OPTS="$JVM_OPTS -XX:-UseBiasedLocking” # Tunable settings JVM_OPTS="$JVM_OPTS -XX:SurvivorRatio=2" JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=16" JVM_OPTS="$JVM_OPTS -XX:+UnlockDiagnosticVMOptions" JVM_OPTS="$JVM_OPTS -XX:ParGCCardsPerStrideChunk=4096” # Instagram settings JVM_OPTS="$JVM_OPTS -XX:+CMSScavengeBeforeRemark" JVM_OPTS="$JVM_OPTS -XX:CMSMaxAbortablePrecleanTime=60000" JVM_OPTS="$JVM_OPTS -XX:CMSWaitDuration=30000" 12
- 13. © DataStax, All Rights Reserved. JVM - G1GC JVM_OPTS="$JVM_OPTS -XX:+UseG1GC" JVM_OPTS="$JVM_OPTS -XX:MaxGCPauseMillis=500" JVM_OPTS="$JVM_OPTS -XX:G1RSetUpdatingPauseTimePercent=5" JVM_OPTS="$JVM_OPTS -XX:InitiatingHeapOccupancyPercent=25” JVM_OPTS="$JVM_OPTS -XX:ParallelGCThreads=16” # Set to number of full cores JVM_OPTS="$JVM_OPTS -XX:ConcGCThreads=16” # Set to number of full cores 13
- 14. © DataStax, All Rights Reserved. Cassandra concurrent_reads: 128 concurrent_writes: 128 concurrent_counter_writes: 128 memtable_allocation_type: heap_buffers memtable_flush_writers: 8 memtable_cleanup_threshold: 0.15 memtable_heap_space_in_mb: 2048 memtable_offheap_space_in_mb: 2048 trickle_fsync: true trickle_fsync_interval_in_kb: 1024 internode_compression: dc 14
- 15. Data model and compaction strategy
- 16. © DataStax, All Rights Reserved. Data model Data model impacts performance a lot Optimize so that you read from one partition Make sure your data can be distributed SSTable compression depending on the use case 16
- 17. © DataStax, All Rights Reserved. Compaction strategy 1. Size tiered compaction strategy • Good as a default • Performance and size constraints 2. Leveled compaction strategy • Great for low latency read requirements • Constant compactions 3. Date tiered / Time window compaction strategy • Good fit for time series use cases 17
- 18. © DataStax, All Rights Reserved. Ok, what now? After we set the base configuration it’s time for testing and observing 18
- 19. Metrics and reporting stack
- 20. © DataStax, All Rights Reserved. Metrics and reporting stack OS metrics (SmartCat) Metrics reporter config (AddThis) Cassandra diagnostics (SmartCat) Filebeat Riemann InfluxDB Grafana Elasticsearch Logstash Kibana 20
- 21. © DataStax, All Rights Reserved. Grafana 21
- 22. © DataStax, All Rights Reserved. Kibana 22
- 23. © DataStax, All Rights Reserved. Slow queries Track query execution times above some threshold Gain insights into the long processing queries Relate that to what’s going on on the node Compare app and cluster slow queries https://github.com/smartcat-labs/cassandra-diagnostics 23
- 24. © DataStax, All Rights Reserved. Slow queries - cluster 24
- 25. © DataStax, All Rights Reserved. Slow queries - cluster vs app 25
- 26. © DataStax, All Rights Reserved. Ops center Pros: Great when starting out Everything you need in a nice GUI Cluster metrics Cons: Metrics stored in the same cluster Issues with some of the services (repair, slow query,...) Additional agents on the nodes 26
- 27. Test setup
- 28. © DataStax, All Rights Reserved. Test setup Make sure you have repeatable tests Fixed rate tests Variable rate tests Production like tests Cassandra Stress Various loadgen tools (gatling, wrk, loader,...) 28
- 29. © DataStax, All Rights Reserved. Coordinated omission 29
- 30. © DataStax, All Rights Reserved. Tuning methodology 30
- 31. AWS
- 32. © DataStax, All Rights Reserved. AWS deployment Choose your instance based on calculations Use placement groups and availability zones Don’t overdo it just because you can ($$$) Are you sure you need ephemeral storage? Go for EBS volumes (gp2) 32
- 33. © DataStax, All Rights Reserved. EBS volumes Pros: 3.4TB+ volume has 10.000 IOPs Average latency is ~0.38ms Durable across reboots AWS snapshots Can be attached/detached Easy to recreate 33 Cons: Rare latency spikes Average latency is ~0.38ms Degrading factor
- 34. © DataStax, All Rights Reserved. EBS volumes - problems 34
- 35. © DataStax, All Rights Reserved. End result Did we meet our goal? Can we go any further? 35
- 36. © DataStax, All Rights Reserved. Whats next? Torture testing Failure scenarios Latency and delay inducers Automate everything 36
- 37. Q&A