Cassandra Summit 2010 - Operations & Troubleshooting Intro
21 likes2,697 views
The document discusses troubleshooting strategies for Cassandra, emphasizing the importance of understanding system tools and node management for balancing load. It highlights common issues like uneven data distribution, slow reads, and disk bandwidth problems, offering solutions such as adjusting memtable settings or adding nodes. The author encourages community involvement in sharing experiences and solutions related to these challenges.
Cassandra Summit 2010 - Operations & Troubleshooting Intro
- 1. Cassandra troubleshooting: out of the shadows Benjamin Black, b@b3k.us
- 3. The Allegory of the Cave
- 4. Most people start troubleshooting problems interpreting shadows on the wall.
- 6. Paths out of the cave.
- 7. Combination of basic system tools & nodetool/JMX
- 8. I’m using RP. My ring is very unbalanced.
- 9. I’m using RP. My ring is very unbalanced. WTF?
- 10. nodetool ring
- 11. Address Status Load Range Ring 148873535527910577765226390751398592511 10.248.54.192 Up 5.59 GB 0 |<--| 10.248.254.15 Up 10.58 GB 42535295865117307932921825928971026431 | ^ 10.248.135.239Up 11.01 GB 85070591730234615865843651857942052863 v | 10.248.223.191Up 5.42 GB 106338239662793269832304564822427566079 | ^ 10.248.122.240Up 5.51 GB 127605887595351923798765477786913079295 v | 10.248.34.80 Up 5.45 GB 148873535527910577765226390751398592511 |-->|
- 12. Address Status Load Range Ring 148873535527910577765226390751398592511 10.248.54.192 Up 5.59 GB 0 |<--| 10.248.254.15 Up 10.58 GB 42535295865117307932921825928971026431 | ^ 10.248.135.239Up 11.01 GB 85070591730234615865843651857942052863 v | 10.248.223.191Up 5.42 GB 106338239662793269832304564822427566079 | ^ 10.248.122.240Up 5.51 GB 127605887595351923798765477786913079295 v | 10.248.34.80 Up 5.45 GB 148873535527910577765226390751398592511 |-->|
- 14. Automatic token algorithm: Assign a token that will give me half the range of the most loaded node.
- 15. 32 16 16 8 8 16 8888 44888 444488
- 16. Address Status Load Range Ring 148873535527910577765226390751398592511 10.248.54.192 Up 5.59 GB 0 |<--| 10.248.254.15 Up 10.58 GB 42535295865117307932921825928971026431 | ^ 10.248.135.239Up 11.01 GB 85070591730234615865843651857942052863 v | 10.248.223.191Up 5.42 GB 106338239662793269832304564822427566079 | ^ 10.248.122.240Up 5.51 GB 127605887595351923798765477786913079295 v | 10.248.34.80 Up 5.45 GB 148873535527910577765226390751398592511 |-->|
- 17. nodetool move + Manual token assignment
- 18. 0-(2**127 - 1) def tokens(nodes) 0.upto(nodes - 1) do |n| p (n * (2**127 - 1) / nodes) end end
- 20. YES: This means you need to change tokens on most of the nodes in your cluster whenever you add a node.
- 21. Writes are fast. Reads keep getting slower.
- 22. Writes are fast. Reads keep getting slower. WTF?
- 23. iostat -x look at %util
- 24. nodetool tpstats
- 25. Pool Name Active Pending Completed STREAM-STAGE 0 0 0 RESPONSE-STAGE 0 0 516280 ROW-READ-STAGE 8 4096 1164326 LB-OPERATIONS 0 0 0 MESSAGE-DESERIALIZER-POOL 1 682008 1818682 GMFD 0 0 6467 LB-TARGET 0 0 0 CONSISTENCY-MANAGER 0 0 661477 ROW-MUTATION-STAGE 0 0 998780 MESSAGE-STREAMING-POOL 0 0 0 LOAD-BALANCER-STAGE 0 0 0 FLUSH-SORTER-POOL 0 0 0 MEMTABLE-POST-FLUSHER 0 0 4 FLUSH-WRITER-POOL 0 0 4 AE-SERVICE-STAGE 0 0 0 HINTED-HANDOFF-POOL 0 0 3
- 26. Pool Name Active Pending Completed STREAM-STAGE 0 0 0 RESPONSE-STAGE 0 0 516280 ROW-READ-STAGE 8 4096 1164326 LB-OPERATIONS 0 0 0 MESSAGE-DESERIALIZER-POOL 1 682008 1818682 GMFD 0 0 6467 LB-TARGET 0 0 0 CONSISTENCY-MANAGER 0 0 661477 ROW-MUTATION-STAGE 0 0 998780 MESSAGE-STREAMING-POOL 0 0 0 LOAD-BALANCER-STAGE 0 0 0 FLUSH-SORTER-POOL 0 0 0 MEMTABLE-POST-FLUSHER 0 0 4 FLUSH-WRITER-POOL 0 0 4 AE-SERVICE-STAGE 0 0 0 HINTED-HANDOFF-POOL 0 0 3
- 27. Pool Name Active Pending Completed STREAM-STAGE 0 0 0 RESPONSE-STAGE 0 0 516280 ROW-READ-STAGE 8 4096 1164326 LB-OPERATIONS 0 0 0 MESSAGE-DESERIALIZER-POOL 1 682008 1818682 GMFD 0 0 6467 LB-TARGET 0 0 0 CONSISTENCY-MANAGER 0 0 661477 ROW-MUTATION-STAGE 0 0 998780 MESSAGE-STREAMING-POOL 0 0 0 LOAD-BALANCER-STAGE 0 0 0 FLUSH-SORTER-POOL 0 0 0 MEMTABLE-POST-FLUSHER 0 0 4 FLUSH-WRITER-POOL 0 0 4 AE-SERVICE-STAGE 0 0 0 HINTED-HANDOFF-POOL 0 0 3
- 28. Pool Name Active Pending Completed STREAM-STAGE 0 0 0 RESPONSE-STAGE 0 0 516280 ROW-READ-STAGE 8 4096 1164326 LB-OPERATIONS 0 0 0 MESSAGE-DESERIALIZER-POOL 1 682008 1818682 GMFD 0 0 6467 LB-TARGET 0 0 0 CONSISTENCY-MANAGER 0 0 661477 ROW-MUTATION-STAGE 0 0 998780 MESSAGE-STREAMING-POOL 0 0 0 LOAD-BALANCER-STAGE 0 0 0 FLUSH-SORTER-POOL 0 0 0 MEMTABLE-POST-FLUSHER 0 0 4 FLUSH-WRITER-POOL 0 0 4 AE-SERVICE-STAGE 0 0 0 HINTED-HANDOFF-POOL 0 0 3
- 29. YOU ARE OUT OF DISK BANDWIDTH
- 30. You can:
- 31. Throttle reads at clients
- 32. Adjust memtable settings (size/ops/time)
- 33. Less frequent memtable flush Less frequent compaction Less disk bandwidth demand
- 34. Add more nodes
- 35. Add more spindles per node
- 36. Switch to SSDs
- 37. I inserted a bunch of data. Now my nodes are flapping.
- 38. I inserted a bunch of data. Now my nodes are flapping. WTF?
- 39. iostat -x look at %util
- 41. INFO 13:27:35,309 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
- 42. mmap() in Cassandra consumes up to 2GB.
- 43. mmap() in Cassandra consumes up to 2GB. Per segment.
- 44. NOT tracked as JVM heap. *See: https://issues.apache.org/jira/browse/CASSANDRA-1214
- 45. NOT tracked as JVM heap. JVM heap not locked in memory. *See: https://issues.apache.org/jira/browse/CASSANDRA-1214
- 46. When your data set exceeds memory, this is likely.
- 47. Swapping can delay gossip long enough to cause a node to be marked down.
- 51. Most people start troubleshooting problems interpreting shadows on the wall.
- 52. You can now see the path and the sunlight outside.
- 53. YOU CAN HELP!
- 54. What things have confused you?
- 55. What problems have you solved?
- 56. What tools have you used to solve them?
- 57. GET INVOLVED!