Hadoop distributed file system

Hadoop distributed file System
www.kellytechno.com Page 1
I. INTRODUCTION AND RELATED WORK
Hadoop [1][16][19] provides a distributed file system and a framework for the analysis and
transformation of very large data sets using the MapReduce [3] paradigm. An important characteristic of
Hadoop is the partitioning of data and computation across many (thousands) of hosts, and executing
application computations in parallel close to their data.
A Hadoop cluster scales computation capacity, storage capacity and IO bandwidth by simply adding
commodity servers. Hadoop clusters at Yahoo! span 25 000 servers, and store 25 petabytes of
application data, with the largest cluster being 3500 servers. One hundred other organizations
worldwide report using Hadoop. HDFS Distributed file system Subject of this paper! MapReduce
Distributed computation framework HBase Column-oriented table service Pig Dataflow language and
parallel execution framework Hive Data warehouse infrastructure ZooKeeper Distributed coordination
service Chukwa System for collecting management data Avro Data serialization system Table
1. Hadoop project components Hadoop is an Apache project; all components are available via the
Apache open source license. Yahoo! has developed and contributed to 80% of the core of Hadoop (HDFS
and MapReduce). HBase was originally developed at Powerset, now a department at Microsoft. Hive
[15] was originated and develdeveloped at Facebook. Pig [4], ZooKeeper [6], and Chukwa were
originated and developed at Yahoo! Avro was originated at Yahoo! and is being co-developed with
Cloudera. HDFS is the file system component of Hadoop. While the interface to HDFS is patterned after
the UNIX file system, faithfulness to standards was sacrificed in favor of improved performance for the
applications at hand. HDFS stores file system metadata and application data separately.
As in other distributed file systems, like PVFS [2][14], Lustre [7] and GFS [5][8], HDFS stores metadata on
a dedicated server, called the NameNode. Application data are stored on other servers called
DataNodes. All servers are fully connected and communicate with each other using TCP-based protocols.
Unlike Lustre and PVFS, the DataNodes in HDFS do not use data protection mechanisms such as RAID to
make the data durable. Instead, like GFS, the file content is replicated on multiple DataNodes for
reliability. While ensuring data durability, this strategy has the added advantage that data transfer
bandwidth is multiplied, and there are more opportunities for locating computation near the needed
data. Several distributed file systems have or are exploring truly distributed implementations of the
namespace. Ceph [17] has a cluster of namespace servers (MDS) and uses a dynamic subtree
partitioning algorithm in order to map the namespace tree to MDSs evenly. GFS is also evolving into a
distributed namespace implementation [8]. The new GFS will have hundreds of namespace servers
(masters) with 100 million files per master. Lustre [7] has an implementation of clustered namespace on
its roadmap for Lustre 2.2 release. The intent is to stripe a directory over multiple metadata servers
(MDS), each of which contains a disjoint portion of the namespace. A file is assigned to a particular MDS
using a hash function on the file name.
II. ARCHITECTURE A. NameNode The HDFS namespace is a hierarchy of files and directories. Files and
directories are represented on the NameNode by inodes, which record attributes like permissions,
modification and access times, namespace and disk space quotas. The file content is split into large

blocks (typically 128 megabytes, but user selectable file-by-file) and each block of the file is
independently replicated at multiple DataNodes (typically three, but user selectable file-by-file). The
NameNode maintains the namespace tree and the mapping of file blocks to DataNodes
(the physical location of file data). An HDFS client wanting to read a file first contacts the NameNode for
the locations of data blocks comprising the file and then reads block contents from the DataNode closest
to the client. When writing data, the client requests the NameNode to nominate a suite of three
DataNodes to host the block replicas. The client then writes data to the DataNodes in a pipeline fashion.
The current design has a single NameNode for each cluster. The cluster can have thousands of
DataNodes and tens of thousands of HDFS clients per cluster, as each DataNode may execute multiple
application tasks concurrently. HDFS keeps the entire namespace in RAM. The inode data and the list of
blocks belonging to each file comprise the metadata of the name system called the image. The
persistent record of the image stored in the local host’s native files system is called a checkpoint. The
NameNode also stores the modification log of the image called the journal in the local host’s native file
system. For improved durability, redundant copies of the checkpoint and journal can be made at other
servers. During restarts the NameNode restores the namespace by reading the namespace and
replaying the journal. The locations of block replicas may change over time and are not part of the
persistent checkpoint.
B. DataNodes Each block replica on a DataNode is represented by two files in the local host’s native file
system. The first file contains the data itself and the second file is block’s metadata including checksums
for the block data and the block’s generation stamp. The size of the data file equals the actual length of
the block and does not require extra space to round it up to the nominal block size as in traditional file
systems. Thus, if a block is half full it needs only half of the space of the full block on the local drive.
During startup each DataNode connects to the NameNode and performs a handshake.
The purpose of the handshake is to verify the namespace ID and the software version of the DataNode.
If either does not match that of the NameNode the DataNode automatically shuts down. The
namespace ID is assigned to the file system instance when it is formatted. The namespace ID is
persistently stored on all nodes of the cluster. Nodes with a different namespace ID will not be able to
join the cluster, thus preserving the integrity of the file system. The consistency of software versions is
important because incompatible version may cause data corruption or loss, and on large clusters of
thousands of machines it is easy to overlook nodes that did not shut down properly prior to the
software upgrade or were not available during the upgrade.
A DataNode that is newly initialized and without any namespace ID is permitted to join the cluster and
receive the cluster’s namespace ID. After the handshake the DataNode registers with the NameNode.
DataNodes persistently store their unique storage IDs. The storage ID is an internal identifier of the
DataNode, which makes it recognizable even if it is restarted with a different IP address or port. The
storage ID is assigned to the DataNode when it registers with the NameNode for the first time and never
changes after that. A DataNode identifies block replicas in its possession to the NameNode by sending a
block report. A block report contains the block id, the generation stamp and the length for each block
replica the server hosts. The first block report is sent immediately after the DataNode registration.

Subsequent block reports are sent every hour and provide the NameNode with an up-todate view of
where block replicas are located on the cluster. During normal operation DataNodes send heartbeats to
the NameNode to confirm that the DataNode is operating and the block replicas it hosts are available.
The default heartbeat interval is three seconds. If the NameNode does not receive a heartbeat from a
DataNode in ten minutes the NameNode considers the DataNode to be out of service and the block
replicas hosted by that DataNode to be unavailable. The NameNode then schedules creation of new
replicas of those blocks on other DataNodes. Heartbeats from a DataNode also carry information about
total storage capacity, fraction of storage in use, and the number of data transfers currently in progress.
These statistics are used for the NameNode’s space allocation and load balancing decisions. The
NameNode does not directly call DataNodes. It uses replies to heartbeats to send instructions to the
DataNodes.
The instructions include commands to:
• replicate blocks to other nodes;
• remove local block replicas;
• re-register or to shut down the node;
• send an immediate block report.
These commands are important for maintaining the overall system integrity and therefore it is critical to
keep heartbeats frequent even on big clusters. The NameNode can process thousands of heartbeats per
second without affecting other NameNode operations.
C. HDFS Client User applications access the file system using the HDFS client, a code library that exports
the HDFS file system interface. Similar to most conventional file systems, HDFS supports operations to
read, write and delete files, and operations to create and delete directories. The user references files
and directories by paths in the namespace.
The user application generally does not need to know that file system metadata and storage are on
different servers, or that blocks have multiple replicas. When an application reads a file, the HDFS client
first asks the NameNode for the list of DataNodes that host replicas of the blocks of the file. It then
contacts a DataNode directly and requests the transfer of the desired block.
When a client writes, it first asks the NameNode to choose DataNodes to host replicas of the first block
of the file. The client organizes a pipeline from node-to-node and sends the data. When the first block is
filled, the client requests new DataNodes to be chosen to host replicas of the next block. A new pipeline
is organized, and the

Hadoop distributed file system

More Related Content

What's hot (20)

Viewers also liked (10)

Similar to Hadoop distributed file system (20)

Recently uploaded (20)

Hadoop distributed file system