Gluster fs hadoop_fifth-elephant
Download as ODP, PDF1 like474 views
This document discusses using GlusterFS for Hadoop. GlusterFS is an open source distributed file system that aggregates storage and provides a unified global namespace. It can be used with Hadoop as the underlying storage system instead of HDFS. Using GlusterFS offers advantages like no need for a metadata server and ability to use the same storage for both MapReduce jobs and application data. It also supports features like geo-replication and erasure coding that are useful for big data workloads.
Gluster fs hadoop_fifth-elephant
- 1. GlusterFS For Hadoop– Overview Vijay Bellur GlusterFS co-maintainer Lalatendu Mohanty GlusterFS Community
- 2. 05/17/16 Agenda ● What is GlusterFS? ● Overview ● Use Cases ● Hadoop on GlusterFS ● Q&A
- 3. 05/17/16 What is GlusterFS? ● A general purpose scale-out distributed file system. ● Aggregates storage exports over network interconnect to provide a single unified namespace. ● Filesystem is stackable and completely in userspace. ● Layered on disk file systems that support extended attributes.
- 4. 05/17/16 Typical GlusterFS Deployment Global namespace Scale-out storage building blocks Supports thousands of clients Access using GlusterFS native, NFS, SMB and HTTP protocols Linear performance scaling
- 5. 05/17/16 GlusterFS Architecture – Foundations ● Software only, runs on commodity hardware ● No external metadata servers ● Scale-out with Elasticity ● Extensible and modular ● Deployment agnostic ● Unified access ● Largely POSIX compliant
- 7. 05/17/16 GlusterFS concepts – Trusted Storage Pool ● Trusted Storage Pool (cluster) is a collection of storage servers. ● Trusted Storage Pool is formed by invitation – “probe” a new member from the cluster and not vice versa. ● Membership information used for determining quorum. ● Members can be dynamically added and removed from the pool.
- 8. 05/17/16 A brick is the combination of a node and an export directory – for e.g. hostname:/dir Each brick inherits limits of the underlying filesystem No limit on the number bricks per node Ideally, each brick in a cluster should be of the same size /export3 /export3 /export3 Storage Node /export1 Storage Node /export2 /export1 /export2 /export4 /export5 Storage Node /export1 /export2 3 bricks 5 bricks 3 bricks GlusterFS concepts - Bricks
- 9. 05/17/16 GlusterFS concepts - Volumes ● A volume is a logical collection of bricks. ● Volume is identified by an administrator provided name. ● Volume is a mountable entity and the volume name is provided at the time of mounting. – mount -t glusterfs server1:/<volname> /my/mnt/point ● Bricks from the same node can be part of different volumes
- 10. 05/17/16 GlusterFS concepts - Volumes Node2Node1 Node3 /export/brick1 /export/brick2 /export/brick1 /export/brick2 /export/brick1 /export/brick2 music Videos
- 11. 05/17/16 Volume Types ➢ Type of a volume is specified at the time of volume creation ➢ Volume type determines how and where data is placed ➢ Following volume types are supported in glusterfs: a) Distribute b) Stripe c) Replication d) Distributed Replicate e) Striped Replicate f) Distributed Striped Replicate
- 12. 05/17/16 Distributed Volume ➢ Distributes files across various bricks of the volume. ➢ Directories are present on all bricks of the volume. ➢ Single brick failure will result in loss of data availability. ➢ Removes the need for an external meta data server.
- 13. 05/17/16 How does a distributed volume work? ➢ Uses Davies-Meyer hash algorithm. ➢ A 32-bit hash space is divided into N ranges for N bricks ➢ At the time of directory creation, a range is assigned to each directory. ➢ During a file creation or retrieval, hash is computed on the file name. This hash value is used to locate or place the file. ➢ Different directories in the same brick end up with different hash ranges.
- 14. 05/17/16 Replicated Volume ● Synchronous replication of all directory and file updates. ● Provides high availability of data when node failures occur. ● Transaction driven for ensuring consistency. ● Changelogs maintained for re-conciliation. ● Any number of replicas can be configured.
- 15. 05/17/16 How does a replicated volume work?
- 16. 05/17/16 Distributed Replicated Volume ● Distribute files across replicated bricks ● Number of bricks must be a multiple of the replica count ● Ordering of bricks in volume definition matters ● Scaling and high availability ● Reads get load balanced. ● Most preferred model of deployment currently.
- 18. 05/17/16 Striped Volume ● Files are striped into chunks and placed in various bricks. ● Recommended only when very large files greater than the size of the disks are present. ● A brick failure can result in data loss. Redundancy with replication is highly recommended (striped replicated volumes).
- 19. 05/17/16 Elastic Volume Management Application transparent operations that can be performed in the storage layer. ● Addition of Bricks to a volume ● Remove brick from a volume ● Rebalance data spread within a volume ● Replace a brick in a volume ● Performance / Functionality tuning
- 20. 05/17/16 Access Mechanisms Gluster volumes can be accessed via the following mechanisms: – FUSE based Native protocol – NFSv3 and v4 – SMB – libgfapi – ReST/HTTP – HDFS
- 22. 05/17/16 Translators in GlusterFS ● Building blocks for a GlusterFS process. ● Based on Translators in GNU HURD. ● Each translator is a functional unit. ● Translators can be stacked together for achieving desired functionality. ● Translators are deployment agnostic – can be loaded in either the client or server stacks.
- 24. 05/17/16 Ecosystem Integration ● Currently integrated with various ecosystems: ● OpenStack ● Samba ● Ganesha ● oVirt ● qemu ● Hadoop ● pcp ● Proxmox ● uWSGI
- 25. 05/17/16
- 26. 05/17/16 Use Cases - current ● Unstructured data storage ● Archival ● Disaster Recovery ● Virtual Machine Image Store ● Cloud Storage for Service Providers ● Content Cloud ● Big Data ● Semi-structured & Structured data
- 27. 05/17/16 Hadoop And GlusterFS ● GlusterFS can be used for Hadoop ● GlusterFS Hadoop plugin replaces HDFS with GlusterFS ● MapReduce jobs can be run on GlusterFS volumes. ● https://github.com/gluster/glusterfs-hadoop
- 28. 05/17/16 Advantage Of Using GlusterFS ● Advantage of a POSIX compliant filesystem. ● Same volume/storage can be used for MapReduce and storing application data. ● E.g. : log files, unstructured data. ● No need to copy data from storage to HDFS for running MapReduce. ● No need for “NameNode” i.e. metadata server. ● Advantage of GlusterFS features (e.g. Geo-replication, Erasure Coding) ● Geo-replication is a distributed, continuous, asynchronous, and incremental replication service for disastrous recovery ● It can replicate data from one site to another over Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.
- 29. 05/17/16 Advantage Of Using GlusterFS ● Erasure Coding provides the fundamental technology for storage systems to add redundancy and tolerate failures. ● On GlusterFS, MapReduce jobs use “data locality optimization”. ● That means Hadoop tries its best to run map tasks on nodes where the data is present locally to optimize on the network and inter-node communication latency. ● GlusterFS works with Apache Spark Project and The Apache Ambari project.
- 30. 05/17/16 Apache Spark Project ● Apache Spark is an open-source data analytics cluster computing framework ● Spark fits into the Hadoop open-source community, building on top of the Hadoop Distributed File System (HDFS) ● Spark is not tied to the two-stage MapReduce paradigm and promises performance up to 100 times faster than Hadoop MapReduce, for certain applications. ● Spark provides primitives for in-memory cluster computing. ● https://spark.apache.org/docs/0.8.1/cluster-overview.html
- 31. 05/17/16 Apache Ambari Project ● The Apache Ambari project is for provisioning, managing, and monitoring Apache Hadoop clusters. ● It provides an intuitive, easy-to-use Hadoop management web UI backed by its RESTful APIs. ● Apache Ambari project supports the automated deployment and configuration of Hadoop on top of GlusterFS. ● http://www.gluster.org/2013/10/automated-hadoop-deployment-o ● http://ambari.apache.org/
- 33. 05/17/16 Resources Mailing lists: gluster-users@gluster.org gluster-devel@nongnu.org IRC: #gluster and #gluster-dev on freenode Links: http://www.gluster.org http://hekafs.org http://forge.gluster.org http://www.gluster.org/community/documentation/index.php/Arch http://hadoopecosystemtable.github.io/
- 34. Thank you! Lalatendu Mohanty lmohanty@redhat.com Twitter: @lalatenduM