Progress on adapting BlobSeer to WAN scale
1 like677 views
This document discusses integrating the BlobSeer data management system with the HGMDS distributed metadata management system to build a global file system deployed across multiple sites. Two approaches are considered: having multiple BlobSeer instances, one per site, or having a single BlobSeer-WAN instance spanning sites. The authors propose the latter approach with distributed version managers and provider managers to leverage data locality. Preliminary evaluations show BlobSeer-WAN performance on two sites is similar to vanilla BlobSeer on a single site. Next steps include further evaluation, integrating BlobSeer-WAN with HGMDS, and submitting a co-authored paper.
Progress on adapting BlobSeer to WAN scale
- 1. Towards a Scalable File System Progress on adapting BlobSeer to WAN scale for the HGMDS distributed metadata system Viet-Trung Tran, Gabriel Antoniu, Alexandru Costan (INRIA - Rennes) In collaboration with Kohei Hiraga, Osamu Tatebe (U Tsukuba) FP3C meeting Bordeaux, 2 – 3 September 2011
- 2. Plan 1. Background and context 2. Goal 3. Approach and solution 4. Preliminary evaluation 5. Conclusion FP3C meeting – Bordeaux, 2-3 September 2011 -2
- 3. 1 Background BlobSeer & HGMDS FP3C meeting – Bordeaux, 2-3 September 2011 -3
- 4. BlobSeer: A large-scale data management service Generic data-management platform for huge, unstructured data • Huge data (TB) : BLOBs • Highly concurrent, fine-grain access (MB): R/W/A • Prototype available Key design features • Decentralized metadata management • Beyond MVCC: multiversioning exposed to the user • Lock-free write access through versioning A back-end for higher-level, sophisticated data management systems FP3C meeting – Bordeaux, 2-3 September 2011 -4
- 5. BlobSeer: Architecture Clients Providers • Perform fine grain blob accesses Providers • Store the pages of the blob Provider manager • Monitors the providers • Favours data load balancing Provider Clients manager Metadata providers • Store information about page location Version Version manager manager • Ensures concurrency control Metadata providers FP3C meeting – Bordeaux, 2-3 September 2011 -5
- 6. HGMDS: A distributed metadata management system for global file systems • Multi-master file system The Internet metadata server (MDS). Site A Site B • Managing inode structure. File system Clients • High latency networks don't affect metadata operation HGMD HGMD performance. S S - Both reading and writing. • One MDS per site. • Metadata versioning using mkdir/rmdir/ Propagate create/stat/ updates in vector clocks for collision unlink background detection. Site C • Automatic collision resolution by system side. FP3C meeting – Bordeaux, 2-3 September 2011 -6
- 7. 2 Goal A joint architecture integrating BlobSeer and HGMDS FP3C meeting – Bordeaux, 2-3 September 2011 -7
- 8. Goal BlobSeer HGMDS Data management Metadata management Typically on a single site Global scale, multiple sites Idea: build a global file system deployed on multiple site by integrating BlobSeer to HGMDS Potential benefits: • HGMDS: efficient multi-site file metadata management • BlobSeer: concurrency-optimized access to globally shared data FP3C meeting – Bordeaux, 2-3 September 2011 -8
- 9. 3 Our approach and solution FP3C meeting – Bordeaux, 2-3 September 2011 -9
- 10. Two approaches Multiple BlobSeer instances • One BlobSeer / site One single BlobSeer-WAN over distributed geographic sites FP3C meeting – Bordeaux, 2-3 September 2011 - 10
- 11. 1st approach: 1 BlobSeer instance / site Client FP3C meeting – Bordeaux, 2-3 September 2011 - 11
- 12. 1st approach: Zoom High latency when accessing remote BLOBs: • Too many remote requests for small metadata EMETTEUR - NOM DE LA PRESENTATION - 12
- 13. 2nd approach: 1 BlobSeer-WAN instance over distributed geographic sites Multiple version managers • 1 version manager/site Multiple provider managers • 1 provider manager/site On each site • Multiple data providers and metadata servers • Data providers are under control of local provider manager EMETTEUR - NOM DE LA PRESENTATION - 13
- 14. Idea: leverage locality for remote metadata accesses 2 Metadata I/O is resolved locally EMETTEUR - NOM DE LA PRESENTATION - 14
- 15. 2nd approach: I/O scheme in BlobSeer-WAN Writing • Publish version on local version manager • Locally write metadata on local metadata servers • Locally write data on local data providers Reading (Read your write in many cases) • Ask a version to local version manager • Local metadata accesses • Access remote/local providers if necessary FP3C meeting – Bordeaux, 2-3 September 2011 - 15
- 16. Vector clocks and optimistic metadata replication FP3C meeting – Bordeaux, 2-3 September 2011 - 16
- 17. Expected benefits • On WAN: BlobSeer coordinates with HGMDS to provide a global versioning file system - Low latency metadata I/O - Eventually consistency model - Load balancing/fault tolerance • On LAN: - Distributed version management - Load balancing/fault tolerance FP3C meeting – Bordeaux, 2-3 September 2011 - 17
- 18. 4 Preliminary evaluation BlobSeer-WAN on G5K FP3C meeting – Bordeaux, 2-3 September 2011 - 18
- 19. Testbed Using 2 sites of G5K • Rennes: 40 nodes • 30 nodes reserved for BlobSeer services • 10 nodes for clients • Grenoble: 40 nodes • 30 nodes reserved for BlobSeer services • 10 nodes for clients • Interconnect network between sites 10 Gbps FP3C meeting – Bordeaux, 2-3 September 2011 - 19
- 20. Concurrent appending: 512 MB/client FP3C meeting – Bordeaux, 2-3 September 2011 - 20
- 21. 5 Conclusion On going work FP3C meeting – Bordeaux, 2-3 September 2011 - 21
- 22. Summary Discussed the integration of BlobSeer and HGMDS: • BlobSeer-WAN extension is required BlobSeer-WAN • Preliminary results look encouraging • Performance of BlobSeer-WAN on two sites similar to that of vanilla BlobSeer on a single site • Prototype available at BlobSeer’s repository/branches/ BlobSeer-WAN-dev/ HGMDS • Implementation almost done • Works on multi-sites • Collisions automatically solved by a rule FP3C meeting – Bordeaux, 2-3 September 2011 - 22
- 23. Next steps • A more extensive evaluation for BlobSeer-WAN • Integrate BlobSeer-WAN to HGMDS • Preliminary evaluation of HGMDS BlobSeer-WAN on Grid5000 and on the Japanese Clusters • Submit co-authored paper by Spring 2012 • Next internships: Kohei @Inria Rennes FP3C meeting – Bordeaux, 2-3 September 2011 - 23
- 24. Thank you! FP3C meeting 2 – 3 September 2011