GlusterFs: a scalable file system for today's and tomorrow's big data
5 likes2,256 views
Roberto Franchini gave a presentation on GlusterFS, an open source, scalable distributed file system. GlusterFS allows storage to be scaled out by adding more servers with disks. It uses a flexible peer-to-peer architecture with data distributed across server bricks. Franchini discussed how his company implemented GlusterFS to store large indexes for information extraction tasks, scaling from 4TB to 28TB as storage needs grew. He provided recommendations on best practices like dedicated servers, same-sized bricks, and planning for future expansion.
GlusterFs: a scalable file system for today's and tomorrow's big data
- 1. Roberto Franchini franchini@celi.it Codemotion Milano 29/11/2014
- 2. GlusterFS A scalable distributed file system
- 3. whoami(1) 15 years of experience, proud to be a programmer Writes software for information extraction, nlp, opinion mining (@scale ), and a lot of other buzzwords Implements scalable architectures Plays with servers Member of the JUG-Torino coordination team franchini@celi.it http://www.celi.it http://www.blogmeter.it github.com/robfrank github.com/uim-celi twitter.com/robfrankie linkedin.com/in/robfrank
- 4. The problem Identify a distributed and scalable file system for today's and tomorrow's Big Data
- 5. Once upon a time 2008: One nfs share 1,5TB ought to be enough for anybody 2010: Herd of shares (1,5TB x N) ought to be enough for anybody Nobody couldn’t stop the data flood It was the time for something new
- 6. Requirements Can be enlarged on demand No dedicated HW OS is preferred and trusted No specialized API No specialized Kernel POSIX compliance Zilions of big and small files No NAS or SAN (€€€€€)
- 7. Clustered Scale-out General Purpose Storage Platform − POSIX-y Distributed File System − ...and so much more Built on commodity systems − x86_64 Linux ++ − POSIX filesystems underneath (XFS, EXT4) No central metadata Server (NO SPOF) Modular architecture for scale and functionality
- 8. Common use cases Large Scale File Server Media / Content Distribution Network (CDN) Backup / Archive / Disaster Recovery (DR) High Performance Computing (HPC) Infrastructure as a Service (IaaS) storage layer Database offload (blobs) Unified Object Store + File Access
- 9. Features ACL and Quota support Fault-tolerance Peer to peer Self-healing Fast setup up Enlarge on demand Shrink on demand Snapshot On premise phisical or virtual On cloud
- 10. Architecture
- 11. Architecture Peer / Node − cluster servers (glusterfs server) − Runs the gluster daemons and participates in volumes Brick − A filesystem mountpoint on servers − A unit of storage used as a capacity building block
- 12. Bricks on a node
- 13. Architecture Translator − Logic between bricks or subvolume that generate a subvolume with certain characteristic − distribute, replica, stripe are special translators to generate simil-RAID configuration − perfomance translators Volume − Bricks combined and passed through translators − Ultimately, what's presented to the end user
- 14. Volume
- 15. Volume types
- 16. Distributed The default configuration Files “evenly” spread across bricks Similar to file-level RAID 0 Server/Disk failure could be catastrophic
- 17. Distributed
- 18. Replicated Files written synchronously to replica peers Files read synchronously, but ultimately serviced by the first responder Similar to file-level RAID 1
- 19. Replicated
- 20. Distributed + replicated Distribued + replicated Similar to file-level RAID 10 Most used layout
- 22. Striped Individual files split among bricks (sparse files) Similar to block-level RAID 0 Limited Use Cases HPC Pre/Post Processing File size exceeds brick size
- 23. Striped
- 24. Moving parts
- 25. Components glusterd Management daemon One instance on each GlusterFS server Interfaced through gluster CLI glusterfsd GlusterFS brick daemon One process for each brick on each server Managed by glusterd
- 26. Components glusterfs Volume service daemon One process for each volume service NFS server, FUSE client, Self-Heal, Quota, ... mount.glusterfs FUSE native client mount extension gluster Gluster Console Manager (CLI)
- 27. Clients
- 28. Clients: native FUSE kernel module allows the filesystem to be built and operated entirely in userspace Specify mount to any GlusterFS server Native Client fetches volfile from mount server, then communicates directly with all nodes to access data Recommended for high concurrency and high write performance Load is inherently balanced across distributed volumes
- 29. Clients:NFS Standard NFS v3 clients Standard automounter is supported Mount to any server, or use a load balancer GlusterFS NFS server includes Network Lock Manager (NLM) to synchronize locks across clients Better performance for reading many small files from a single client Load balancing must be managed externally
- 30. Clients: libgfapi Introduced with GlusterFS 3.4 User-space library for accessing data in GlusterFS Filesystem-like API Runs in application process no FUSE, no copies, no context switches ...but same volfiles, translators, etc.
- 31. Clients: SMB/CIFS In GlusterFS 3.4 – Samba + libgfapi No need for local native client mount & re-export Significant performance improvements with FUSE removed from the equation Must be setup on each server you wish to connect to via CIFS CTDB is required for Samba clustering
- 32. Clients: HDFS Access data within and outside of Hadoop No HDFS name node single point of failure / bottleneck Seamless replacement for HDFS Scales with the massive growth of big data
- 33. Scalability
- 34. Under the hood Elastic Hash Algorithm No central metadata No Performance Bottleneck Eliminates risk scenarios Location hashed intelligently on filename Unique identifiers (GFID), similar to md5sum
- 35. Scalability 3TB 3TB 3TB 3TB 3TB 3TB 3TB 3TB 3TB 3TB Gluster Server 3TB 3TB 3TB 3TB 3TB 3TB 3TB 3TB 3TB 3TB Gluster Server 3TB 3TB 3TB 3TB 3TB 3TB 3TB 3TB 3TB 3TB Gluster Server Scale out performance and availability Scale out capacitry
- 36. Scalability Add disks to servers to increase storage size Add servers to increase bandwidth and storage size Add servers to increase availability (replica factor)
- 37. What we do with glusterFS
- 38. What we do with GFS Daily production of more than 10GB of Lucene inverted indexes stored on glusterFS more than 200GB/month Search stored indexes to extract different sets of documents for every customers YES: we open indexes directly on storage (it's POSIX!!!)
- 39. 2010: first installation Version 3.0.x 8 (not dedicated) servers Distributed replicated No bound on brick size (!!!!) Ca 4TB avaliable NOTE: stuck to 3.0.x until 2012 due to problems on 3.1 and 3.2 series, then RH acquired gluster (RH Storage)
- 40. 2012: (little) cluster New installation, version 3.3.2 4TB available on 8 servers (DELL c5000) still not dedicated 1 brick per server limited to 1TB 2TB-raid 1 on each server Still in production
- 41. 2012: enlarge New installation, upgrade to 3.3.x 6TB available on 12 servers (still not dedicated) Enlarged to 9TB on 18 servers Bricks size bounded AND unbounded
- 42. 2013: fail 18 not dedicated servers: too much 18 bricks of different sizes 2 big down due to bricks out of space Didn’t restart after a move but… All data were recovered (files are scattered on bricks, read from them!)
- 43. 2014: consolidate 2 dedicated servers 12 x 3TB SAS raid6 4 bricks per server 28 TB available distributed replicated 4x1Gb bonded NIC ca 40 clients (FUSE) (other servers)
- 44. Consolidate brick 1 brick 2 brick 3 brick 4 Gluster Server 1 brick 1 brick 2 brick 3 brick 4 Gluster Server 2
- 45. Scale up brick 11 brick 12 brick 13 brick 31 Gluster Server 1 brick 21 brick 22 brick 32 brick 24 Gluster Server 2 brick 31 brick 32 brick 23 brick 14 Gluster Server 3
- 46. Do Dedicated server (phisical or virtual) RAID 6 or RAID 10 (with small files) Multiple bricks of same size Plan to scale
- 47. Do not Multi purpose server Bricks of different size Very small files Write to bricks
- 48. Some raw tests read Total transferred file size: 23.10G bytes 43.46M bytes/sec write Total transferred file size: 23.10G bytes 38.53M bytes/sec
- 49. Raw tests NOTE: ran in production under heavy load, no clean test room
- 50. Resources http://www.gluster.org/ https://access.redhat.com/documentation/en- US/Red_Hat_Storage/ https://github.com/gluster http://www.redhat.com/products/storage-server/ http://joejulian.name/blog/category/glusterfs/ http://jread.us/2013/06/one-petabyte-red-hat-storage-and-glusterfs- project-overview/
- 51. Thank you!