Apache cassandra - survivre en production
0 likes270 views
This document summarizes best practices for operating Apache Cassandra clusters in production environments. It covers topics like hardware requirements, monitoring, garbage collection settings, compaction strategies, repair procedures, tombstone handling, and bootstrap recommendations. The goal is to help Cassandra users optimize performance and ensure high availability of their production databases.
![nodetool cfstats/tablestats
Table: cluster
SSTable count: 1
SSTables in each level: [1, 0, 0, 0, 0, 0, 0, 0, 0]
Space used (live): 5136
Space used (total): 5136
…
Bloom filter false positives: 0
Bloom filter false ratio: 0,00000
Bloom filter space used: 16
Bloom filter off heap memory used: 8
Index summary off heap memory used: 17
Compression metadata off heap memory used: 16
Compacted partition minimum bytes: 104
Compacted partition maximum bytes: 124
Compacted partition mean bytes: 124
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
www.thelastpickle.com](https://image.slidesharecdn.com/apachecassandra-survivreenproduction-180202073402/85/Apache-cassandra-survivre-en-production-15-320.jpg)
![SSTable 1 [——————]
SSTable 2 [———————]
Tombstones [-]
www.thelastpickle.com](https://image.slidesharecdn.com/apachecassandra-survivreenproduction-180202073402/85/Apache-cassandra-survivre-en-production-32-320.jpg)
Apache cassandra - survivre en production
- 1. Apache Cassandra Survivre en production
- 2. PARIS CASSANDRA MEETUP - JANVIER 2018 Alexander Dejanovski @alexanderdeja Consultant www.thelastpickle.com Datastax MVP for Apache Cassandra Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License
- 3. www.bigdatahebdo.com Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License
- 4. AboutThe Last Pickle Apache Cassandra consulting 4 committers / 3 PMC / 1VP Apache Cassandra 6 pays : Nouvelle Zélande, Australie, France, Espagne, USA et Estonie
- 6. CPU 8 à 16 threads www.thelastpickle.com
- 7. RAM 32 à 64 Go www.thelastpickle.com
- 9. Monitoring Grafana / Prometheus / DataDog www.thelastpickle.com
- 10. GC settings G1 vs CMS ? Taille de heap ? www.thelastpickle.com
- 11. Versions de Cassandra 2.x.x vs 3.0.x vs 3.11.x www.thelastpickle.com
- 14. Partitionement !! Les « grosses » partitions tuent silencieusement vos clusters www.thelastpickle.com
- 15. nodetool cfstats/tablestats Table: cluster SSTable count: 1 SSTables in each level: [1, 0, 0, 0, 0, 0, 0, 0, 0] Space used (live): 5136 Space used (total): 5136 … Bloom filter false positives: 0 Bloom filter false ratio: 0,00000 Bloom filter space used: 16 Bloom filter off heap memory used: 8 Index summary off heap memory used: 17 Compression metadata off heap memory used: 16 Compacted partition minimum bytes: 104 Compacted partition maximum bytes: 124 Compacted partition mean bytes: 124 Average live cells per slice (last five minutes): NaN Maximum live cells per slice (last five minutes): 0 Average tombstones per slice (last five minutes): NaN Maximum tombstones per slice (last five minutes): 0 Dropped Mutations: 0 www.thelastpickle.com
- 16. nodetool cfhistograms/tablehistograms reaper_db/cluster histograms Percentile SSTables Write Latency Read Latency Partition Size Cell Count (micros) (micros) (bytes) 50% 0,00 0,00 0,00 124 4 75% 0,00 0,00 0,00 124 4 95% 0,00 0,00 0,00 124 4 98% 0,00 0,00 0,00 124 4 99% 0,00 0,00 0,00 124 4 Min 0,00 0,00 0,00 104 4 Max 0,00 0,00 0,00 124 4 www.thelastpickle.com
- 17. Réduire la durée de vie des partitions www.thelastpickle.com
- 18. Batches vs Async queries www.thelastpickle.com
- 19. Pas de « read-before-write » pour matérialiser un état RACE CONDITIONS ! www.thelastpickle.com
- 20. Pas de changements incrémentaux sur une collection avec TTL www.thelastpickle.com
- 22. Index secondaires et SASI www.thelastpickle.com
- 24. Débit et parallélisme 8 à 16MB/s min par compacteur www.thelastpickle.com
- 25. Choix de stratégie STCS / LCS / TWCS www.thelastpickle.com
- 26. Beaucoup d’overwrites et SSD ? LCS www.thelastpickle.com
- 27. Données avec TTL ? TWCS www.thelastpickle.com
- 29. Pas de compactions manuelles (sauf sur petits volumes) www.thelastpickle.com
- 30. Limites de la STCS www.thelastpickle.com
- 31. Tombstone compactions unchecked_tombstone_compaction: true tombstone_threshold: 0.2 tombstone_compaction_interval: 86400 www.thelastpickle.com
- 32. SSTable 1 [——————] SSTable 2 [———————] Tombstones [-] www.thelastpickle.com
- 33. Limites de la LCS www.thelastpickle.com
- 34. Limites de la TWCS www.thelastpickle.com
- 35. Trop de SSTables ? www.thelastpickle.com
- 37. Réparation avant C* 2.2 www.thelastpickle.com
- 38. How does anti-entropy repair works ? www.thelastpickle.com
- 39. Merkle tree is requested to all replicas www.thelastpickle.com
- 43. The problem with dense nodes Overstreaming Chaque feuille de l’arbre de Merkle peut contenir plusieurs partitions. Résolution max de 32 000 feuilles www.thelastpickle.com
- 44. Réparation après C* 2.2 www.thelastpickle.com
- 46. How does incremental repair work ? www.thelastpickle.com
- 48. Anticompaction (repair on all ranges on local node) www.thelastpickle.com
- 49. Incremental repair Sépare les SSTables en 2 pools non compactables www.thelastpickle.com
- 50. Anticompaction N’apprécie pas la concurrence www.thelastpickle.com
- 52. Repair best practices Dans Cassandra 2.2/3.x : full repair = anticompaction. subrange = pas d’anticompaction www.thelastpickle.com
- 53. Repair best practices Attendez la 4.0.x pour faire de la réparation incrémentale…? CASSANDRA-9143 Fix committed for 4.0 www.thelastpickle.com
- 54. Repair best practices Ajustez l’intensité de la réparation www.thelastpickle.com
- 55. C’est quoi le conseil sur la réparation alors ? www.thelastpickle.com
- 56. http://cassandra-reaper.io Reaper c’est bon, mangez-en ! www.thelastpickle.com
- 57. Une dernière chose Vos réparations sont soumises au débit de compaction max (il est partagé par tous vos threads de compaction) www.thelastpickle.com
- 58. Tombstones cell / row / range / partition www.thelastpickle.com
- 60. gc_grace_seconds Jamais < hint window (sauf si TTL < hint window) www.thelastpickle.com
- 61. gc_grace_seconds Réduire à 3h si TTL et pas de DELETE www.thelastpickle.com
- 62. gc_grace_seconds Caler sur votre cycle de repair si DELETE www.thelastpickle.com
- 63. Bootstrap avec vnodes, 1 à la fois (par rack) www.thelastpickle.com
- 65. Bootstrap le cleanup c’est LONG ! www.thelastpickle.com