GlusterFS w/ Tiered XFS
2 likes969 views
Richard Wareing presented on using XFS realtime subvolumes to improve GlusterFS metadata performance. Traditional solutions like page caching are limited, while dedicated metadata stores add complexity. XFS realtime subvolumes combine benefits by storing metadata on SSDs for improved performance without changing GlusterFS core. Facebook is working on kernel patches to optimize realtime allocation and integration. The presentation addressed strengths and weaknesses of GlusterFS and opportunities to improve scaling and code quality.
![Quick Tangent
Some Perspective: Things we need to work on…
▪ Scaling: There continues to be a huge disparity between GlusterFS and
it’s contemporaries. Ceph & HDFS both scale dramatically more.
▪ Code Quality: More commenting! This helps bring in more developers,
ease code review.
▪ Lack of JBOD support: GlusterFS requires some form of RAID[5 -6] which
adds complexity and expense
▪ Drains/Rebuilds: Without use of some XFS tricks, this is still quite slow,
taking weeks vs. days.](https://image.slidesharecdn.com/glustersummit2017-richardwareing-realtimexfs-180523220659/85/GlusterFS-w-Tiered-XFS-5-320.jpg)
GlusterFS w/ Tiered XFS
- 1. (c) 2009 Facebook, Inc. or its licensors. "Facebook" is a registered trademark of Facebook, Inc.. All rights reserved. 1.0
- 2. Richard Wareing Production Engineer October 27 th, 2017 Realtime XFS Solving The Metadata Problem
- 3. 1 The Problem 2 XFS w/ Realtime Subvolumes 3 GlusterFS Application 4 Facebook Enhanacements 5 Questions? Agenda
- 4. Quick Tangent Some Perspective: Strengths ▪ GlusterFS xlator architecture is elegant, don’t give this up…ever ▪ Tends to enforce good design on new developers ▪ Good isolation between components (with some cheating in some places) ▪ IO Latencies: Yet to see anything in the open source realm come close to the latencies we see on GlusterFS for metadata ▪ Simplicity: Very easy to setup, configure & manage. ▪ Open Source: Community is strong, growing with diverse use-cases
- 5. Quick Tangent Some Perspective: Things we need to work on… ▪ Scaling: There continues to be a huge disparity between GlusterFS and it’s contemporaries. Ceph & HDFS both scale dramatically more. ▪ Code Quality: More commenting! This helps bring in more developers, ease code review. ▪ Lack of JBOD support: GlusterFS requires some form of RAID[5 -6] which adds complexity and expense ▪ Drains/Rebuilds: Without use of some XFS tricks, this is still quite slow, taking weeks vs. days.
- 6. The Problem GlusterFS & Metadata ▪ Preface this… ▪ Leveraging underlying FS for metadata storage was a good decision ▪ -Simplicity of design, reliability ▪ But this design choice isn’t without problems… ▪ Heavy reliance on metadata ▪ xattrs à AFR “journal”, xlator attributes for DHT, quota etc
- 7. The Problem GlusterFS & Metadata ▪ DHT directory operations “magnify” à Single ”ls” à 1000’s of VFS system calls Brick 8 ls –l /foo 10 ms 8 ms 5 ms 15 ms 2 ms 11 ms 50 ms 15 ms 20 ms 50 ms Subvol 0 readdir Optimize Subvol 1 Subvol 2 Brick 1 Brick 4 Brick 7 Brick 2 Brick 5 Brick 3 Brick 6 Brick 9 AFR Optimization s (Slow Sub-Vol)
- 8. The Problem Quantifying the Problem ▪ blktrace to examine physical block device Ios ▪ Examine RWBS field ▪ Example 1: CREATE heavy ▪ Blob store ’ish workload ▪ 51% metadata CREATE heavy workload
- 9. The Problem Quantifying the Problem ▪ Example 2: MKDIR heavy workload ▪ Deep directory structures ▪ ”FS as database” use-case ▪ 83% metadata MKDIR heavy workload
- 10. The Problem Traditional Solutions ▪ Page Cache ▪ Pro: Simplicity à just “works”; works with any storage system ▪ Cons: DRAM is expensive, limited space to cache objects, doesn’t help write heavy use-cases. ▪ Dedicated Metadata Stores ▪ Pros: Good designs can scale well; works with write heavy workloads ▪ Cons: Added complexity à Reliability, management overhead; maintaining consistency can be challenging; very specific to storage system ▪ Can we combine the best of both?
- 11. XFS w/ Realtime Subvol Overview Realtime Block Device Standard Block Device Metadata Intent Log (Journal) Data Blocks XFS Metadata Intent Log (Journal) Data Blocks Realtime Data Blocks XFS Filesystem Optional: Can be moved. XFS Filesystem
- 12. XFS w/ Realtime Subvol Realtime Allocator RT Extent 1 RT Extent 2 RT Extent 3 RT Extent 4 RT Extent 5 RT Extent 6 RT Extent 7 RT Extent 8 RT Extent 9 RT Extent 10 … … … … … … … … … … … … … … RT Extent n Tunable to any Fixed size, guaranteed contiguous! Realtime Bitmap Block Allocator
- 13. GlusterFS Application Brick Filesystem Layout /mnt/brick1 /mnt/brick2 sdb1 Metadata Intent Log File Cache (Optional) sdb2 Metadata Intent Log File Cache (Optional) sdc1 Data Blocks sdd1 Data Blocks SSD (sdb) HDDs (sdc- sdd)
- 14. GlusterFS Application Benefits: Combines best of both worlds (mostly) ▪ Pros ▪ Simplicity à just “works”, no changes to GlusterFS core ▪ Works with any storage system ▪ Works with write or ready heavy metadata workloads ▪ SSD based File caching à Trivial to implement à “.cache” directory ▪ Cons ▪ Does not scale independently of bricks ▪ Changes require kernel patches potentially ▪ Realtime allocator is single-thread optimized, this may present problems for some workloads à JBOD support?
- 15. Facebook Enhancements Kernel Patches (Pending XFS maintainer review) ▪ statfs ▪ Return realtime device usage if rtinherit flag is set à More intuitive ▪ rtallocsize ▪ Small (initial) allocations stored on realtime subvolume automatically ▪ rtfallbackpct ▪ Allocate data to realtime device if data block device (e.g. SSD) usage is above rtfallbackpct
- 17. (c) 2009 Facebook, Inc. or its licensors. "Facebook" is a registered trademark of Facebook, Inc.. All rights reserved. 1.0