Sdc challenges-2012
0 likes468 views
GlusterFS is a distributed file system that shards and replicates files across multiple servers without a central metadata server. It uses modular "translators" to handle functions like replication and distribution. Some challenges GlusterFS faces include multi-tenancy, distributed quota management, efficient data rebalancing, reducing replication latency, optimizing directory traversal, and handling many small files. The speaker argues these challenges are not unique to GlusterFS and that incremental, modular improvements are preferable to monolithic solutions.
Sdc challenges-2012
- 1. GlusterFS Challenges and Futures Jeff Darcy Storage Developer Conference September 17, 2012
- 2. What Is GlusterFS? ● Just your basic distributed file system ● sharding, replication, striping ● Decentralized ● no central metadata server ● core functionality on clients ● Modular ● “translators”
- 3. Translator Stacking FUSE DHT AFR-1 Client-3Client-1 AFR-0 Client-2Client-0 ● One to one, one to many, one to zero (?) ● Rearrange, move from client to server, ...
- 4. Distribution (now) Server A Server BServer C File X File Y
- 5. Replication (now) ● Based on changelog (“dirty flags”) ● set flags, do operation, clear flags ● use flags to determine repair (“self-heal”) after failure ● Latency sensitive ● 3+ network round trips per user request ● implementation heavily optimized
- 6. Challenge: Multitenancy ● Focus of CloudFSHekaFS ● to be merged with GlusterFS soon-ish ● Isolate name and ID spaces ● Encrypt in flight and at rest ● Auth* ● Quota (next slide) ● Performance isolation ● cgroups
- 7. Challenge: Distributed Quota ● Can't trust clients to enforce quota ● Can't just divide equally among servers ● unequal usage (e.g. due to explicit placement) ● EDQUOT on one while still space on another ● “Quota rebalancing daemon” ● monitor/adjust continuously ● interesting problem at high scale
- 8. Challenge: Better Rebalancing ● Optimal placement vs. minimal data movement ● Different kinds of weighting Node A Node B Node C Node D Node E
- 9. Challenge: Replication Latency ● Reaching limits of current approach ● Have to go async? ● but still ordered ● exploit compute/data locality (e.g. Hadoop) ● journaling, conflict resolution
- 10. Challenge: Directory Traversal ● Piggyback attrs (and xattrs) on readdir ● Even better: cursor approach ● read everything in opendir ● zero network activity for readdir ● less current, but more consistent
- 11. Challenge: Many Small Files ● Prefetch whole directories ● if marked, below size threshold, ... ● Exploit async journal ● only works if compute/data colocated ● Weaken consistency? ● allow create/write/close to be buffered ● directory-level fsync (magic xattr)
- 12. Conclusions ● Most of these challenges are not unique to GlusterFS ● Modularity and incremental progress are preferable to monolithic “solve all problems at once” ● GlusterFS provides a good environment in which to experiment with solutions