Gluster d2
0 likes107 views
GlusterD 2.0 is a redesign of the Gluster distributed file system management daemon (GlusterD) to improve scalability and stability. It uses Go for the programming language instead of C and has a new architecture with components like REST interfaces, a central etcd store for management data, RPC for communication, a transaction framework, and a plugin framework. The goals are to make GlusterD able to manage thousands of nodes, provide stronger configuration consistency, and support new Gluster features in a modular way. There will be no rolling upgrade available initially and service disruption is expected during the upgrade from GlusterD 1.0 to 2.0.
Gluster d2
- 1. Vault - April 21, 2016 1 GlusterD 2.0
- 2. Vault - April 21, 2016 2 Atin Mukherjee “An engineer by profession, a musician by passion” Gluster Co Maintainer Senior S/W Engineer @ Red Hat Reach me @ IRC: atinm on #freenode, Twitter: @mukherjee_atin, mailto : amukherj@redhat.com, github: github.com/atinmu
- 3. Vault - April 21, 2016 3 Agenda ● GlusterFS – a brief overview ● GlusterFS concepts ● Introduction to legacy GlusterD (GlusterD 1.0) ● Why GlusterD 2.0 ● High level architecture ● Components ● Upgrades consideration ● Q&A
- 4. Vault - April 21, 2016 4 GlusterFS â—Ź Open-source general purpose scale-out distributed file system â—Ź Aggregates storage exports over network interconnect to provide a single unified namespace â—Ź Layered on disk file systems that support extended attributes No meta-data server Modular Architecture for Scale and Functionality Heterogeneous commodity hardware Scalable to petabytes & beyond
- 5. Vault - April 21, 2016 5 GlusterFS Requirements ● Server ● Intel/AMD x86 64-bit processor ● Disk: 8GB minimum using direct-attached- storage, RAID, Amazon EBS, and FC/Infiniband/iSCSI SAN disk backends using SATA/SAS/FC disks ● Memory: 1GB minimum A node is server capable of hosting GlusterFS bricks ● Networking ● 10 Gigabit ethernet ● Infiniband (OFED 1.5.2 or later) ● Logical Volume Manager ● LVM2 with thin provisioning ● File System ● POSIX w/ Extended Attributes (EXT4, XFS, BTRFS, …)
- 6. Vault - April 21, 2016 6 GlusterFS Concepts – Trusted Storage Pool ● Also known as cluster ● Trusted Storage Pool is formed by invitation – “probe” ● Members can be dynamically added and removed from the pool ● Only nodes in a Trusted Storage Pool can participate in volume creation A collection of storage servers (nodes) Node 1 Node 2 Node 3
- 7. Vault - April 21, 2016 7 GlusterFS Concepts – Bricks ● A brick is the directory on the local storage node ● Layered on posix compliant file-system (e.g. XFS, ext4) ● Each brick inherits limits of the underlying filesystem ● It is recommended to use an independent thinly provisioned LVM as brick – Thin provisioning is needed by Snapshot feature A unit of storage used as a capacity building block
- 8. Vault - April 21, 2016 8 GlusterFS Concepts – Volume ● Node hosting these bricks should be part of a single Trusted Storage Pool ● One or more volumes can be hosted on the same node A volume is a logical collection of one or more bricks
- 9. Vault - April 21, 2016 9 What is GlusterD â—Ź Manages the cluster configuration for Gluster âś” Peer membership management âś” Elastic volume management âś” Configuration consistency âś” Distributed command execution (orchestration) âś” Service management (manages Gluster daemons)
- 10. Vault - April 21, 2016 10 Issues with legacy GlusterD design ● N * N exchange of Network messages for peer handshaking ● Not scalable when N is probably in hundreds or thousands ● Replicate configuration (management) data in all nodes ● Initialization time can be very high ● Can end up in a situation like “whom to believe, whom not to” - popularly known as split brain ● Lack of transaction rollback mechanism
- 11. Vault - April 21, 2016 11 Why GlusterD 2.0 “Configure and deploy a 'thousand- node' Gluster cloud”
- 12. Vault - April 21, 2016 12 Why GlusterD 2.0 (ii) ● More efficient/stable membership – Especially at high scale ● Stronger configuration consistency ● Non trivial effort in adding management support for Gluster features (modularity & plugins)
- 13. Vault - April 21, 2016 13 Language choice ● Legacy GlusterD is in C “It wasn't enough just to add features into existing language because sometimes you can get more in the long run by taking things away...They wanted to start from scratch and rethink everything.” - Robert.C.Pike ● GD2 is in Go and written from scratch! – Suits best in writing a management plane of a file system (distributed) – Garbage collection – Standard libraries support – One binary approach – Built in rich features like go routines, channels for concurrency
- 14. Vault - April 21, 2016 14 High Level Architecture
- 15. Vault - April 21, 2016 15 Component breakdown â—Ź ReST interfaces â—Ź Central store - etcd management & bootstrapping â—Ź RPC Mechanism â—Ź Transaction framework â—Ź Feature plug-in framework â—Ź Flexi volgen
- 16. Vault - April 21, 2016 16 ReST interface â—Ź HTTP ReST Interface â—Ź ReST API support â—Ź CLI to work as ReST Client
- 17. Vault - April 21, 2016 17 Central store - etcd ● Configuration management (replication, consistency) handling by etcd ● etcd bootstrapping – etcd store initiatilization at GD2 boot up – Modes – client(proxy)/server – Interface to toggle between client to server and vice versa ● External etcd integration too
- 18. Vault - April 21, 2016 18 RPC Framework ● Brick to client (and vice versa) communication over existing sun rpc model ● Protobuf RPC for GD2 to GD2 / daemons communication ● Considerations – Language support for both C & Go – Auto code generation – Support for SSL transports – Non-blocking I/O support – Programmer friendly implementation pattern (unlike thrift using Glib style)
- 19. Vault - April 21, 2016 19 Transaction framework â—Ź Drives the life cycle of a command including daemons â—Ź Executes actions in a given order â—Ź Modular - plan to make it consumable by other projects â—Ź To be built around central store â—Ź execution on required nodes only â—Ź Originator only commits into the central store
- 20. Vault - April 21, 2016 20 Feature plug-in framework ● Ease of integration of Gluster features ● Reduce burden on maintenance & ownership ● Feature interface aims to provide – insert xlators into a volume graph – set options on xlators – define and create custom volume graphs – define and manage daemons – create CLI commands – hook into existing CLI commands – query cluster and volume information – associate information with objects (peers, volumes)
- 21. Vault - April 21, 2016 21 Flexi volgen ● Volfile – source of information by which xlator stack is built up ● Currently pretty much static in nature ● Goal - make it easy for devs/users to add/remove xlators ● SystemD-units style approach for the new volgen (Still under discussion)
- 22. Vault - April 21, 2016 22 Other improvements â—Ź Transaction based logging, probably centralized logging too! â—Ź Unit tests â—Ź Better op version management â—Ź Focus on improved documentation
- 23. Vault - April 21, 2016 23 Upgrades consideration â—Ź No rolling upgrade, service disruption is expected â—Ź Smooth upgrade from 3.x to 4.x (migration script) â—Ź Rollback - If upgrade fails, revert back to 3.x, old configuration data shouldn't be wiped off
- 24. Vault - April 21, 2016 24 References â—Ź Design documents - https://github.com/gluster/glusterfs- specs/tree/master/design/GlusterD2 â—Ź Source code - https://github.com/gluster/glusterd2 â—Ź Reach us @: gluster-devel@gluster.org, gluster- users@gluster.org #gluster-dev, #gluster on IRC
- 25. Vault - April 21, 2016 25 Q & A
- 26. Vault - April 21, 2016 26 THANK YOU