Compaction, Compaction Everywhere
11 likes6,471 views
The document provides an overview of compaction strategies in Apache Cassandra, highlighting the importance of compaction for maintaining performance and reducing latency. It outlines various compaction strategies such as Size-Tiered Compaction Strategy (STCS), Leveled Compaction Strategy (LCS), and Date-Tiered Compaction Strategy (DTCS), along with their specific configurations and use cases. The author, Aaron Morton, emphasizes the need for compaction to manage sstables efficiently and prevent issues such as increased read latency and wasted disk space.
![LCS in nodetool cfstats
Column Family: HappyPandaCF
SSTable count: 21
SSTables in each level: [1, 7, 13, 0, 0, 0, 0, 0, 0]
Column Family: SadPandaCF
SSTable count: 710
SSTables in each level: [1, 10, 117/100, 582, 0, 0,
0, 0, 0]](https://image.slidesharecdn.com/southbaycassandrausers-compactioncompactioneverwhere-141117125029-conversion-gate02/85/Compaction-Compaction-Everywhere-35-320.jpg)
Compaction, Compaction Everywhere
- 1. SOUTH BAY CASSANDRA USERS NOVEMBER 2014 COMPACTION, COMPACTION, EVERYWHERE. Aaron Morton @aaronmorton Co-Founder & Principal Consultant Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License
- 2. About The Last Pickle. Work with clients to deliver and improve Apache Cassandra based solutions. Apache Cassandra Committer, DataStax MVP, Apache Usergrid Committer. Based in New Zealand & USA.
- 3. Compaction? STCS LCS DTCS
- 5. No compaction? Row fragmentation would result in dramatically increased read latency.
- 6. No compaction? Increased file count would increase memory usage.
- 7. No compaction? Overwrites and deletions would result in wasted disk space.
- 9. Compaction? Log Structured Merge Tree.
- 10. Compaction? Creating new files when flushing to disk improves performance and reduces complexity.
- 11. Compaction? SSTable 1 foo: dishwasher (ts 10): tomato purple (ts 10): cromulent SSTable 2 foo: frink (ts 20): flayven monkey (ts 10): embiggins SSTable 3 SSTable 4 foo: dishwasher (ts 15): tomacco SSTable 5
- 12. Demo.
- 13. nodetool cfhistograms foo bar SSTables per Read 1 sstables: 149 2 sstables: 62 3 sstables: 65 4 sstables: 50 5 sstables: 45 6 sstables: 44 7 sstables: 76 8 sstables: 72 10 sstables: 305 12 sstables: 390
- 14. Compaction? STCS LCS DTCS
- 15. SizeTieredCompactionStrategy The first compaction strategy. Group files of a similar size for compaction.
- 16. SizeTieredCompactionStrategy Works well when data is written to initially and then only read from.
- 17. STCS - After flush. Tier 0 ( < 50 MB) Tier 1 ~125MB
- 18. STCS - Compaction Starts Tier 0 ( < 50 MB) Tier 1 ~125MB
- 19. STCS - New SSTable Tier 0 ( < 50 MB) Tier 1 ~125MB
- 20. STCS - Purge old SSTables Tier 0 ( < 50 MB) Tier 1 ~125MB
- 21. STCS - Compaction Starts (again) Tier 0 ( < 50 MB) Tier 1 ~125MB
- 22. STCS - Final State Tier 0 ( < 50 MB) Tier 1 ~200MB Tier 2 ~800MB
- 23. STCS - min_sstable_size Maximum size of SSTables in the “small” bucket. Default 50
- 24. STCS - bucket_low Lower bound of the bucket size compared to the average size in the bucket. Default 0.5
- 25. STCS - bucket_high Upper bound of the bucket size compared to the average size in the bucket. Default 1.5
- 26. STCS - cold_reads_to_omit Maximum percentage of reads SSTables ignored by STCS may be responsible for. Default 0
- 27. min_compaction_threshold Compact buckets with at least this many SSTables. Default 4
- 28. max_compaction_threshold Compact no more than this many SSTables in a bucket. Default 32
- 29. Compaction? STCS LCS DTCS
- 30. LeveledCompactionStrategy Based on LevelDB from the Chromium team. http://leveldb.org/
- 31. LeveledCompactionStrategy Works well with overwrites and tombstones. Provides low read latency.
- 32. LeveledCompactionStrategy “Uses twice the disk IO”
- 33. DataStax Blogs “Leveled Compaction in Apache Cassandra” “When to Use Leveled Compaction”
- 34. LCS - “It’s going to be all levels Jerry” Level Number of Files 0 Unlimited* 1 100 2 1000 3 10000
- 35. LCS in nodetool cfstats Column Family: HappyPandaCF SSTable count: 21 SSTables in each level: [1, 7, 13, 0, 0, 0, 0, 0, 0] Column Family: SadPandaCF SSTable count: 710 SSTables in each level: [1, 10, 117/100, 582, 0, 0, 0, 0, 0]
- 36. LCS - Starting out level 0
- 37. LCS - New File in Level 1 level 0 level 1
- 38. LCS - Later, Compact L0 With Overlapping L1 level 0 level 1
- 39. LCS - Another File in L1 level 0 level 1
- 40. LCS - Level 1 Full, compact overlapping level 0 level 1
- 41. LCS - New Files in Level 2 level 0 level 1 level 2
- 42. LCS - sstable_size_in_mb Maximum* size of each SSTable at all levels. Default 160
- 43. Compaction? STCS LCS DTCS
- 44. DateTieredCompactionStrategy CASSANDRA-6602 In 2.0.11 and 2.1.1 “Experimental”
- 45. DTCS - Compact Newest Time Bucket 4 hours 20 hours 80 hours > 365 days
- 46. DTCS - New File in First Bucket 4 hours 20 hours 80 hours > 365 days
- 47. DTCS - Promoted to Later Bucket 4 hours 20 hours 80 hours > 365 days
- 48. DTCS - base_time_seconds Target size. Multiplied by min_sstable_size.
- 49. DTCS - timestamp_resolution What TimeUnit you are using for your WriteTime.
- 50. DTCS - max_sstable_age_days Do not compact SSTables where the youngest WRITETIME is older than this.
- 51. Thanks.
- 52. Aaron Morton @aaronmorton Co-Founder & Principal Consultant www.thelastpickle.com Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License