Glusterfs for sysadmins-justin_clift

Editor's Notes

• #3: -Dustin Black -Red Hat Certified Architect -Senior Technical Account Manager with Red Hat Global Support Services -More than 10 years of systems and infrastructure experience, with focus on Linux, UNIX, and networking -I am not a coder. I'll hack a script together, or read code from others reasonably well, but I would never presume to be a developer. -I believe strongly in the power of openness in most all aspects of life and business. Openness only improves the interests we all share.
• #4: I hope to make the GlusterFS concepts more tangible. I want you to walk away with the confidence to start working with GlusterFS today.
• #6: -Commodity hardware: aggregated as building blocks for a clustered storage resource. -Standards-based: No need to re-architect systems or applications, and no long-term lock-in to proprietary systems or protocols. -Simple and inexpensive scalability. -Scaling is non-interruptive to client access. -Aggregated resources into unified storage volume abstracted from the hardware.
• #9: -Bricks are “stacked” to increase capacity -Translators are “stacked” to increase functionality
• #10: -So if you're going to deploy GlusterFS, where do you start? -Remember, for a datacenter deployment, we're talking about commodity server hardware as the foundation. -In the public cloud space, your option right now is Amazon Web Services, with EC2 and EBS. -RHS is the only supported high-availability option for EBS.
• #11: -XFS is the only filesystem supported with RHS. -Extended attribute support is necessary because the file hash is stored there
• #13: -gluster console commands can be run directly, or in interactive mode. Similar to virsh, ntpq
• #14: -The native client uses fuse to build complete filesystem functionality without gluster itself having to operate in kernel space. This offers benefits in system stability and time-to-end-user for code updates.
• #20: No metadata == No Performance Bottleneck or single point of failure (compared to single metadata node) or corruption issues (compared to distributed metadata). Hash calculation is faster than metadata retrieval Elastic hash is the core of how gluster scales linerally
• #21: Modular building blocks for functionality, like bricks are for storage
• #26: When you configure geo-replication, you do so on a specific master (or source) host, replicating to a specific slave (or destination) host. Geo-replication does not scale linearly since all data is passed through the master and slave nodes specifically. Because geo-replication is configured per-volume, you can gain scalability by choosing different master and slave geo-rep nodes for each volume.
• #29: Limited Read/Write performance increase, but in some cases the overhead can actually cause a performance degredation
• #31: -Graphic is wrong -- -Replica 0 is exp1 and exp3 -Replica 1 is exp2 and exp4
• #32: This graphic from the RHS 2.0 beta documentation actually represents a non-optimal configuration. We'll discuss this in more detail later.
• #34: -Native client will be the best choice when you have many nodes concurrently accessing the same data -Client access to data is naturally load-balanced because the client is aware of the volume structure and the hash algorithm.
• #35: ...mount with nfsvers=3 in modern distros that default to nfs 4 The need for this seems to be a bug, and I understand it is in the process of being fixed. NFS will be the best choice when most of the data access by one client and for small files. This is mostly due to the benefits of native NFS caching. Load balancing will need to be accomplished by external mechanisms
• #36: -Use the GlusterFS native client first to mount the volume on the Samba server, and then share that mount point with Samba via normal methods. -GlusterFS nodes can act as Samba servers (packages are included), or it can be an external service. -Load balancing will need to be accomplished by external mechanisms
• #38: An inode size smaller than 512 leaves no room for extended attributes (xattr). This means that every active inode will require a separate block for these. This has both a performance hit as well as a disk space usage penalty.
• #39: -peer status command shows all other peer nodes – excludes the local node -I understand this to be a bug that's in the process of being fixed
• #40: -Support for striped replicated volumes is added in RHS 2.0 -I'm using this example because it's straight out of the documentation, but I want to point out that this is not an optimal configuration. -With this configuration, the replication happens between bricks on the same node. -We should alter the order of the bricks here so that the replication is between nodes.
• #41: -Brick order is corrected to ensure replication is between bricks on different nodes. -Replica is always processed first in building the volume, regardless of whether it's before or after stripe on the command line. -So, a 'replica 2' will create the replica between matching pairs of bricks in the order that the bricks are passed to the command. 'replica 3' will be matching trios of bricks, and so on.
• #42: Must add and remove in multiples of the replica and stripe counts
• #46: ??where is this stored, and how does it impact performance when on??
• #47: Specifying the double-colin between the remote host name and destination tells geo-replication that the destination is a glusterfs volume name. With a single colon, it will treat the destination as an absolute filesystem path. Preceding the remote host with user@ causes geo-replication to interpret the communication protocol as ssh, which is generally preferred. If you use the simple remote syntax of: hostname:volume It will cause the local system to mount the remote volume locally with the native client, which will result in a necessary performance degradation because of the added fuse overhead. Over a WAN, this can be significant.
• #53: Cloud-based online video platform
• #54: Cloud-based online video platform
• #57: Patch written by Jeff Darcy
• #61: Question notes: -Vs. CEPH -CEPH is object-based at its core, with distributed filesystem as a layered function. GlusterFS is file-based at its core, with object methods (UFO) as a layered function. -CEPH stores underlying data in files, but outside the CEPH constructs they are meaningless. Except for striping, GlusterFS files maintain complete integrity at the brick level. -With CEPH, you define storage resources and data architecture (replication) separate, and CEPH actively and dynamically manages the mapping of the architecture to the storage. With GlusterFS, you manually manage both the storage resources and the data architecture.