Cassandra Performance Tuning Like You've Been Doing It for Ten Years
0 likes172 views
The document discusses Cassandra performance tuning and the importance of methodologies such as utilization, saturation, and error metrics. It highlights observability anti-patterns and emphasizes the importance of using appropriate system tools for accurate performance monitoring. Techniques for effective performance practices are also mentioned, alongside insights on configurations and benchmarks.
Cassandra Performance Tuning Like You've Been Doing It for Ten Years
- 1. Jon Haddad, Rustyrazorblade Consulting rustyrazorblade.com Cassandra Performance Tuning Like You've Been Doing It for Ten Years
- 3. • DataStax • The Last Pickle • Committer & PMC • Apple • Net fl ix • Out on my own! Last 10 Years
- 4. Why do we care about performance?
- 6. Methodology
- 8. OODA Loop
- 9. USE Method
- 10. For every resource, check •Utilization •Saturation •Errors
- 11. utilization: the average time that the resource was busy servicing work
- 12. utilization: the average time that the resource was busy servicing work example: cpu at 90% utilization
- 13. saturation: the degree to which the resource has extra work which it can't service, often queued
- 14. saturation: the degree to which the resource has extra work which it can't service, often queued example: i/o device queue of 100
- 15. errors: the count of error events
- 16. errors: the count of error events example: tcp retransmit
- 17. Observability
- 19. Anti-Patterns • Using averages (latency) or anything under p99 • Hiding outliers • Averaging Summary Data • Not understanding your tool's output
- 20. Anti-Patterns • Using averages (latency) or anything under p99 • Hiding outliers • Averaging Summary Data • Not understanding your tool's output
- 21. System Tools
- 23. # mpstat -P ALL 1 10 Linux 5.15.0-89-generic (ubuntu-vm) 12/08/2023 _aarch64_ (2 CPU) 12:21:55 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 12:21:56 AM all 2.16 0.00 23.24 74.59 0.00 0.00 0.00 0.00 0.00 0.00 12:21:56 AM 0 2.17 0.00 22.83 75.00 0.00 0.00 0.00 0.00 0.00 0.00 12:21:56 AM 1 2.15 0.00 23.66 74.19 0.00 0.00 0.00 0.00 0.00 0.00 12:21:56 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 12:21:57 AM all 3.16 0.00 25.79 71.05 0.00 0.00 0.00 0.00 0.00 0.00 12:21:57 AM 0 3.19 0.00 25.53 71.28 0.00 0.00 0.00 0.00 0.00 0.00 12:21:57 AM 1 3.12 0.00 26.04 70.83 0.00 0.00 0.00 0.00 0.00 0.00
- 24. ebpf / bcc-tools
- 25. Q: What happens if we give Cassandra a bigger heap?
- 26. # cachestat-bpfcc 1 10 HITS MISSES DIRTIES HITRATIO BUFFERS_MB CACHED_MB 760 20159 0 3.63% 23 454 828 21858 0 3.65% 23 381 202 16282 0 1.23% 23 414 995 15487 5 6.04% 23 475 1433 12313 0 10.42% 23 523 2966 16771 0 15.03% 23 589 5002 19346 0 20.54% 23 664 6451 18185 0 26.19% 23 735 8391 18053 0 31.73% 23 806 11036 17924 0 38.11% 23 876
- 27. # ext4dist-bpfcc -m 1 20 Tracing ext4 operation latency... Hit Ctrl-C to end. 23:28:18: operation = read msecs : count distribution 0 -> 1 : 18781 |****************************************| 2 -> 3 : 15 | | 4 -> 7 : 5 | | operation = write msecs : count distribution 0 -> 1 : 38182 |****************************************| 2 -> 3 : 2 | | 4 -> 7 : 2 | |
- 28. # biolatency-bpfcc -QDmT 1 10 Tracing block device I/O... Hit Ctrl-C to end. 23:13:48 disk = b'vda' msecs : count distribution 0 -> 1 : 10960 |****************************************| 2 -> 3 : 53 | | 4 -> 7 : 8 | | 8 -> 15 : 1 | |
- 29. Flame Graphs
- 35. $ bin/nodetool tablehistograms test foo test/foo histograms Percentile Read Latency Write Latency SSTables Partition Size Cell Count (micros) (micros) (bytes) 50% 785.94 379.02 1.00 35 1 75% 4055.27 454.83 1.00 35 1 95% 4055.27 454.83 1.00 35 1 98% 4055.27 454.83 1.00 35 1 99% 4055.27 454.83 1.00 35 1 Min 654.95 315.85 1.00 30 0 Max 4055.27 454.83 1.00 35 1
- 36. $ bin/nodetool tablehistograms test foo test/foo histograms Percentile Read Latency Write Latency SSTables Partition Size Cell Count (micros) (micros) (bytes) 50% 785.94 379.02 1.00 35 1 75% 4055.27 454.83 1.00 35 1 95% 4055.27 454.83 1.00 35 1 98% 4055.27 454.83 1.00 35 1 99% 4055.27 454.83 1.00 35 1 Min 654.95 315.85 1.00 30 0 Max 4055.27 454.83 1.00 35 1
- 37. ALTER TABLE test.foo WITH compression = {'sstable_compression': 'LZ4Compressor', 'chunk_length_kb': 4};
- 38. Read Before Write • Lightweight Transactions • Counters
- 40. Read Ahead More read ampli fi cation Turn it down!
- 41. # blockdev --report RO RA SSZ BSZ StartSec Size Device ro 256 512 1024 0 62124032 /dev/loop0 ro 256 512 1024 0 62140416 /dev/loop1 ro 256 512 1024 0 48668672 /dev/loop2 ro 256 512 1024 0 114929664 /dev/loop3 ro 256 512 1024 0 37240832 /dev/loop4 rw 256 512 4096 0 68719476736 /dev/vda blockdev --setra 8 /dev/vda
- 42. Practice
- 43. Set Up A Lab Environment • tlp-cluster • AxonOps
- 44. Benchmark • tlp-stress • nosqlbench • ndbench
- 45. • Methodology • Observability • Practice
- 47. Thank You!!