Osdc2012 xtfs.talk
0 likes580 views
XtreemFS is a distributed file system designed for cloud storage. It uses object storage devices to store file data and metadata servers to manage file metadata and replication. Key features include multi-platform support, abstraction of storage via UUIDs, separation of communication layers, and flexibility in choosing object storage backends. The presentation discusses the architecture, high availability approaches including master-slave replication for metadata servers and primary-secondary replication for object storage devices, security including SSL encryption of communication channels, and areas still needing improvement like encryption without a PKI.
Osdc2012 xtfs.talk
- 1. XtreemFS Extreme cloud file system?! Udo Seidel OSDC 2012
- 2. Agenda ● Background/motivation ● High level overview ● High Availability ● Security ● Summary OSDC 2012
- 3. Distributed file systems ● Part of shared file systems family ● Around for a while ● “back” in scope ● Storage challenges – More – Faster – Cheaper ● XaaS OSDC 2012
- 4. Shared file systems family ● Multiple server access the same data ● Different approaches ● Network based, e.g. NFS, CIFS ● Clustered – Shared disk, e.g. CXFS, CFS, GFS(2), OCFS2 – Distributed, e.g. Lustre, CephFS, GlusterFS .... and XtreemFS OSDC 2012
- 5. Distributed file systems – why? ● More efficient utilization of distributed hardware ● Storage ● CPU/Network ● Scalability ... capacity demands ● Amount ● I/O requirements OSDC 2012
- 6. Distributed file systems – which? ● HDFS (Hadoop) ● CephFS .. ● GlusterFS .. RedHat ● ... ● XtreemFS OSDC 2012
- 7. History ● European Research project (2006-2010) ● Part of XtreemOS ● Linux based grid O/S ● Member of OpenGridForum ● Need of distributed file system OSDC 2012
- 8. Implementation I ● Java ● Supported O/S – Linux – MacOS X with manual work – Free/Net/OpenBSD? – No Windows anymore ● Server and Client (fuse) ... both in user space ● Non-privileged user OSDC 2012
- 9. Implementation II ● IP based ● Different ports for different XtreemFS services ● Clear text vs. encrypted ● Object based storage ● Software implementation ● OSD features in XtreemFS code – Copy on write – Snapshotting OSDC 2012
- 10. XtreemFS – the architecture I ● 4 components ● Object based Storage Devices ● Meta Data and Replica Catalogue Servers ● Directory Service ● Clients ;-) OSDC 2012
- 11. XtreemFS – the architecture II OSDC 2012
- 12. XtreemFS services ● Several ● OSD ● MRC ● Volumes ● UUID's ● Abstraction from network ● Change requires outage ● Plans for topology OSDC 2012
- 13. XtreemFS – DIR/MRC data ● Data stored locally ● BabuDB ● Independent of OSD ● Write buffers Modus Description ASYNC Asynchronous log entry write FSYNC Fsync() called after log entry write and before ack'ing of operation SYNC_WRITE Synchronous log entry write, ack'ing of operation before meta data update SYNC_WRITE_METADATA Synchronous log entry write and meta data update before ack'ing of operation OSDC 2012
- 14. XtreemFS – OSD data ● File cut in 128 Kbyte pieces ● Default: entire file on one OSD ● Distribution across multiple OSD's possible ● RAID 0 implemented ● RAID 5 planned ● Parallel reads/writes OSDC 2012
- 15. XtreemFS interfaces ● HTTP ● Read-only ● Read-write planned ● Command line ● All purposes OSDC 2012
- 16. XtreemFS interfaces OSDC 2012
- 17. XtreemFS – high level summary ● Multi-platform ● Abstraction via UUID ● Communication separation ● Freedom of choice of OSD backend file system ● HPC out of scope OSDC 2012
- 18. XtreemFS – HA in general ● One part: OSD ● Replication via policies ● Other part: MRC and DIR ● Local data stored in BabuDB's ● Synchronization via BabuDB methods OSDC 2012
- 19. XtreemFS – HA for MRC/DIR ● Master/slave ● Master changes -> log file without buffering ● Log file entries propagation to slaves ● Quorum needed => at least 3 instances ● No automation for DIR ● Synchronization ● in clear text ● Encryption via SSL possible OSDC 2012
- 20. XtreemFS OSD replication ● File replication ● Read-only – Since 1.0 – Easy to handle ● Read-write – Only since 1.3 – Later more ● Copies ● Full ● Partial aka on-demand OSDC 2012
- 21. XtreemFS r/o replication ● Arbitrary amount of replicas ● Equally treated replicas ● Only OSD local access ● No sync needed ● Use case ● Static files :-) ● Low bandwidth (partial replica) ● Big static files (partial replica) OSDC 2012
- 22. XtreemFS r/w replication ● Primary/secondary ● Election on demand with leases ● Read/write access ● First primary ● Propagated to secondaries OSDC 2012
- 23. XtreemFS r/w replication - failure ● Secondary ● Behaviour configurable ● Write failure vs. Write on remaining – Quorum needed ● Primary ● Behaviour configurable ● Write failure vs. Write on remaining – Quorum needed OSDC 2012
- 24. XtreemFS OSD/replica policies ● OSD selection for new files ● Replica selection for new/additional copies ● Categories: filter, group, sort ● Combination of rules Policy Category Standard OSD filter FQDN based filter, group, sort UUID based filter Data center topology group, sort random sort OSDC 2012
- 25. XtreemFS HA summary ● Homework needed for DIR and MRC ● OSD ● Lateness of OSD read-write replication ● OSD Read-only replication – Mature and WAN ready ● Access time improvement via striping ● Flexibility of policies OSDC 2012
- 26. XtreemFS encryption ● Not on file system level ● For communication ● Interaction of DIR, MRC and OSD ● Data replication for HA for DIR and/or MRC OSDC 2012
- 27. XtreemFS channel encryption ● Via SSL ● PCKS#12 or Java Key Store (JKS) ● Locally stored – service/client certificates – root CA certificates ● Two modes ● All-Or-Nothing approach ● Grid-SSL – just authentication OSDC 2012
- 28. XtreemFS secure channel encryption ● Password protection of certificates ● MRC/DIR/OSD: stored service configuration ● Client: via CLI!! OSDC 2012
- 29. XtreemFS encryption summary ● Data encryption on POSIX layer? ● SSL obvious choice for TCP/IP channels ● Missing PKI contradicts scalability ● Password protection needs re-design OSDC 2012
- 30. Summary ● High self-defined goals ● Some dropped? ● Some partially implemented ● Ok for R&D Labs ● HA and housekeeping improvement needed ● Encryption w/o PKI OSDC 2012
- 31. References ● http://www.xtreemfs.org ● http://babudb.googlecode.com OSDC 2012
- 32. Thank you! OSDC 2012