Object Storage in ScyllaDB by Ran Regev, ScyllaDB
0 likes145 views
In this talk we take a look at how Object Storage is used by Scylla. We focus on current usage, namely - for backup, and we look at the shift in implementation from an external tool to native Scylla. We take a close look at the complexity of backup and restore mostly in the face of topology changes and token assignments. We also take a glimpse to the future and see how Scylla is going to use Object Storage as its native storage. We explore a few possible applications of it and understand the tradeoffs.
Object Storage in ScyllaDB by Ran Regev, ScyllaDB
- 1. A ScyllaDB Community Object Storage in ScyllaDB Ran Regev Software Team Leader
- 2. Ran Regev ■ Software developer ■ 25 years in different domains ■ Communication (embedded devices) ■ Traffic (waze-like) ■ Cyber-security (threat simulation and mitigation) ■ Medical (equipment, not embedded) ■ ISO C++ committee member
- 3. Agenda ■ What is Object Storage ■ In a nutshell only ■ ScyllaDB’s usage of Object Storage ■ Backup ■ Tiering
- 4. Object Storage 4 object API ScyllaDB Node NVMe object object object object object object object ⏫ Unlimited Storage ⏫ Durable ⏫ Cheap ⏬ High Latency
- 5. Backup and Restore ■ Current Approach, old and new ■ Old: rclone ■ New: Native ScyllaDB
- 6. ScyllaDB Node NVMe Scylla process rclone process user queries SSTable 1. Uncontrolled contention on the Disk rclone SSTable SSTable SSTable SSTable SSTable SSTable 2. Data is downloaded from object storage, saved and then read by ScyllaDB
- 7. ScyllaDB Node NVMe ScyllaDB process user queries Native ScyllaDB SSTable SSTable SSTable SSTable SSTable SSTable SSTable
- 8. Restore ■ Challenges ■Fast Restore ■The Holy Grail: Restore To Any Topology
- 9. ■ Given an existing backup restore to any topology ■ Make sure the data is spread efficiently ■ Restore as fast as possible Challenges SSTable 1A node1 node2 SSTable 1B SSTable 1C SSTable 2A SSTable 2C SSTable 2B SSTable 3C node3 SSTable 1A SSTable 1B SSTable 1C SSTable 2A SSTable 2C SSTable 2B SSTable 3C Original Cluster Target Cluster node1 nodeN
- 10. Backup and Restore ■ Understanding the Complexity ■Moving Parts (variables) ■ Token Ranges of ● SSTables ● Nodes ■Data Replication ■Vnodes vs. Tablets
- 11. SSTable 90,000 94,291 Complexity: Node example SSTable with the range of tokens it holds 25,002 98,968 104,254 187,965 547,831 602,337 25,104 26,200 SSTable 70,687 71,404 SSTable 97,875 98,627 SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable Start token range owned by the node End token range owned by the node
- 12. Complexity: Data Replication - fact RF = 3 The same token (data) exists on different SSTables on multiple nodes
- 13. Complexity: Data Replication - Multiple Backups SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable The same token is kept in more than one SSTable
- 14. SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable One SStable may contain data belonging to more than one node. Complexity: Data Replication - Multiple Restore
- 15. SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable vNodes Tablets Complexity: vNodes Vs. Tablets
- 16. ■ Naive, inefficient Restore ■ One node downloads all SSTables and rebuilds the database: mutation by mutation with load-and-stream. ■ Non Naive, Faster and more efficient Restore ■ Storing Token-Ranges of SStables is required. ■ Reading target topology ■ Preprocessing the combined information: stored token ranges and target topology ■ All nodes participating in the restoration - each nodes does the bare minimum required. Complexity Conclusion
- 17. Fast Restore Approaches ■ “Download & Stream” locally only ■ 1:1 Restore ■ Instant Readiness
- 18. SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable Fast Restore: Preamble - load and stream All SStables are downloaded by one node and mutations are streamed to their destinations
- 19. SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable Fast Restore: “Download & Stream” locally only
- 20. SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable Fast Restore: 1:1 Restore SSTable Each SSTables goes exactly to where it belongs. No load and stream at all.
- 21. SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable Fast Restore: Instant Readiness Instead of downloading and streaming - serve from remote
- 22. ScyllaDB future with Object Storage ■ Tiering ■Time Windowed S3 ■General Purpose Tiering
- 23. ■ Tiering is NOT backup. ■ Tiering means serving data from object storage, as if it is local. ■ Example for Differences ■ Backup are not compacted (may be compacted into themselves) ■ backups are not read unless for restoring data. ■ Tiering enables more storage with less costs. ■ At the expanse of latency. ■ Tiering enables “fast backup” for tiered data. ■ We only need to mark a remote SSTable as being used for backup - no copy. Tiering
- 24. SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable Tiering: General Solution SSTables should be moved from and to object storage based on their usage, aka “temperature”
- 25. SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable SSTable Tiering: Complexity ● Algorithm(s) for Hot and Cold Data ● Tiered data is (probably) not an entire SSTable, but rather smaller fragments of data ● Storage calculations - we need to keep metadata that points to object storage, and even for metadata we have limited local space ● We can count on Object Storage durability when promising Replication Factor Challenges
- 26. Scylla Node 8 days TWCS-S3 … N days 7 days 1 day Today 2 days 3 days 4 days 5 day 6 days ■ Serving from Remote (new feature) ■ Move Time To Tier SStables to Object Storage, as part of their life cycle management ■ No cold and hot data - just cold. ■ We may need to keep some parts of the SSTable (e.g. index) on local disk
- 27. Native Support for any Vendor ■Amazon (exists) ■Google ■Microsoft ■Oracle ■More?
- 28. ■ Amazon is not the only object storage provider ■ Google, Microsoft, Oracle also suggest Object Storage Solutions ■ At ScyllaDB, we strive to support each vendor as natively as possible ■ And thus gain efficiency ■ And thus propose best solutions for our customers Object Storage Vendors
- 29. Stay in Touch Ran Regev ran.regev@scylladb.com regevran