Hadoop Distributed File System in Big data

Vu Pham
Preface
Content of this Lecture:
In this lecture, we will discuss design goals of HDFS, the
read/write process to HDFS, the main configuration
tuning parameters to control HDFS performance and
robustness.
Big Data Computing Hadoop Distributed File System (HDFS)

Vu Pham
Introduction
Hadoop provides a distributed file system and a framework for
the analysis and transformation of very large data sets using
the MapReduce paradigm.
An important characteristic of Hadoop is the partitioning of
data and computation across many (thousands) of hosts, and
executing application computations in parallel close to their
data.
A Hadoop cluster scales computation capacity, storage capacity
and IO bandwidth by simply adding commodity servers.
Hadoop clusters at Yahoo! span 25,000 servers, and store 25
petabytes of application data, with the largest cluster being
3500 servers. One hundred other organizations worldwide
report using Hadoop.

Vu Pham
Introduction
Hadoop is an Apache project; all components are available via
the Apache open source license.
Yahoo! has developed and contributed to 80% of the core of
Hadoop (HDFS and MapReduce).
HBase was originally developed at Powerset, now a department
at Microsoft.
Hive was originated and developed at Facebook.
Pig, ZooKeeper, and Chukwa were originated and developed at
Yahoo!
Avro was originated at Yahoo! and is being co-developed with
Cloudera.

Vu Pham
Hadoop Project Components
Big Data Computing
HDFS Distributed file system
MapReduce Distributed computation framework
HBase Column-oriented table service
Pig
Dataflow language and parallel execution
framework
Hive Data warehouse infrastructure
ZooKeeper Distributed coordination service
Chukwa System for collecting management data
Avro Data serialization system
Hadoop Distributed File System (HDFS)

Vu Pham
HDFS Design Concepts
Scalable distributed filesystem: So essentially, as you add disks
you get scalable performance. And as you add more, you're
adding a lot of disks, and that scales out the performance.
Distributed data on local disks on several nodes.
Low cost commodity hardware: A lot of performance out of it
because you're aggregating performance.
Big Data Computing
Node 1
B1
Node 2
B2
Node n
Bn
…

Vu Pham
HDFS Design Goals
Hundreds/Thousands of nodes and disks:
It means there's a higher probability of hardware failure. So the design
needs to handle node/disk failures.
Portability across heterogeneous hardware/software:
Implementation across lots of different kinds of hardware and software.
Handle large data sets:
Need to handle terabytes to petabytes.
Enable processing with high throughput

Vu Pham
Techniques to meet HDFS design goals
Simplified coherency model:
The idea is to write once and then read many times. And that simplifies
the number of operations required to commit the write.
Data replication:
Helps to handle hardware failures.
Try to spread the data, same piece of data on different nodes.
Move computation close to the data:
So you're not moving data around. That improves your performance and
throughput.
Relax POSIX requirements to increase the throughput.

Vu Pham
Basic architecture of HDFS

Vu Pham
HDFS Architecture: Key Components
Single NameNode: A master server that manages the file system
namespace and basically regulates access to these files from
clients, and it also keeps track of where the data is on the
DataNodes and where the blocks are distributed essentially.
Multiple DataNodes: Typically one per node in a cluster. So
you're basically using storage which is local.
Basic Functions:
Manage the storage on the DataNode.
Read and write requests on the clients
Block creation, deletion, and replication is all based on instructions from
the NameNode.

Vu Pham
Original HDFS Design
Single NameNode
Multiple DataNodes
Manage storage- blocks of data
Serving read/write requests from clients
Block creation, deletion, replication
Big Data Computing Big Data Enabling Technologies

Vu Pham
HDFS in Hadoop 2
HDFS Federation: Basically what we are doing is trying to have
multiple data nodes, and multiple name nodes. So that we can
increase the name space data. So, if you recall from the first design
you have essentially a single node handling all the namespace
responsibilities. And you can imagine as you start having thousands of
nodes that they'll not scale, and if you have billions of files, you will
have scalability issues. So to address that, the federation aspect was
brought in. That also brings performance improvements.
Benefits:
Increase namespace scalability
Performance
Isolation

Vu Pham
HDFS in Hadoop 2
How its done
Multiple Namenode servers
Multiple namespaces
Data is now stored in Block pools
So there is a pool associated with each namenode or
namespace.
And these pools are essentially spread out over all the data
nodes.

Vu Pham
HDFS in Hadoop 2
High Availability-
Redundant NameNodes
Heterogeneous Storage
and Archival Storage
ARCHIVE, DISK, SSD, RAM_DISK

Vu Pham
Federation: Block Pools
So, if you remember the original design you have one name space and a bunch of
data nodes. So, the structure looks similar.
You have a bunch of NameNodes, instead of one NameNode. And each of those
NameNodes is essentially right into these pools, but the pools are spread out over the
data nodes just like before. This is where the data is spread out. You can gloss over
the different data nodes. So, the block pool is essentially the main thing that's
different.

Vu Pham
HDFS Performance Measures
Determine the number of blocks for a given file size,
Key HDFS and system components that are affected
by the block size.
An impact of using a lot of small files on HDFS and
system

Vu Pham
Recall: HDFS Architecture
Distributed data on local disks on several nodes
Big Data Computing
Node 1
B1
Node 2
B2
Node n
Bn
…

Vu Pham
HDFS Block Size
Default block size is 64 megabytes.
Good for large files!
So a 10GB file will be broken into: 10 x 1024/64=160 blocks
Big Data Computing
Node 1
B1
Node 2
B2
Node n
Bn
…

Vu Pham
Importance of No. of Blocks in a file
NameNode memory usage: Every block that you create basically
every file could be a lot of blocks as we saw in the previous case,
160 blocks. And if you have millions of files that's millions of
objects essentially. And for each object, it uses a bit of memory on
the NameNode, so that is a direct effect of the number of blocks.
But if you have replication, then you have 3 times the number of
blocks.
Number of map tasks: Number of maps typically depends on the
number of blocks being processed.

Vu Pham
Large No. of small files: Impact on Name node
Memory usage: Typically, the usage is around 150 bytes per
object. Now, if you have a billion objects, that's going to be like
300GB of memory.
Network load: Number of checks with datanodes proportional
to number of blocks

Vu Pham
Large No. of small files: Performance Impact
Number of map tasks: Suppose we have 10GB of data to
process and you have them all in lots of 32k file sizes? Then we
will end up with 327680 map tasks.
Huge list of tasks that are queued.
The other impact of this is the map tasks, each time they spin up
and spin down, there's a latency involved with that because you
are starting up Java processes and stopping them.
Inefficient disk I/O with small sizes

Vu Pham
HDFS optimized for large files
Lots of small files is bad!
Solution:
Merge/Concatenate files
Sequence files
HBase, HIVE configuration
CombineFileInputFormat

Vu Pham
Big Data Computing
Read/Write Processes in HDFS

Vu Pham
Read Process in HDFS

Vu Pham
Write Process in HDFS

Vu Pham
Big Data Computing
HDFS Tuning Parameters

Vu Pham
Overview
Tuning parameters
Specifically DFS Block size
NameNode, DataNode system/dfs parameters.

Vu Pham
HDFS XML configuration files
Tuning environment typically in HDFS XML configuration files,
for example, in the hdfs-site.xml.
This is more for system administrators of Hadoop clusters, but
it's good to know what changes affect impact the performance,
and especially if your trying things out on your own there some
important parameters to keep in mind.
Commercial vendors have GUI based management console

Vu Pham
HDFS Block Size
Recall: impacts how much NameNode memory is used, number
of map tasks that are showing up, and also have impacts on
performance.
Default 64 megabytes: Typically bumped up to 128 megabytes
and can be changed based on workloads.
The parameter that this changes dfs.blocksize or dfs.block.size.

Vu Pham
HDFS Replication
Default replication is 3.
Parameter: dfs.replication
Tradeoffs:
Lower it to reduce replication cost
Less robust
Higher replication can make data local to more workers
Lower replication ➔ More space

Vu Pham
Lot of other parameters
Various tunables for datanode, namenode.
Examples:
Dfs.datanode.handler.count (10): Sets the number of server
threads on each datanode
Dfs.namenode.fs-limits.max-blocks-per-file: Maximum number
of blocks per file.
Full List:
http://hadoop.apache.org/docs/current/hadoop-project-
dist/hadoop-hdfs/hdfs-default.xml

Vu Pham
Big Data Computing
HDFS Performance and
Robustness

Vu Pham
Common Failures
DataNode Failures: Server can fail, disk can crash, data
corruption.
Network Failures: Sometimes there's data corruption because
of network issues or disk issue. So, all of that could lead to a
failure in the DataNode aspect of HDFS. You could have network
failures. So, you could have a network go down between a
particular and the name node that can affect a lot of data nodes
at the same time.
NameNode Failures: Could have name node failures, disk failure
on the name node itself or the name node itself could corrupt
this process.

Vu Pham
HDFS Robustness
NameNode receives heartbeat and block reports from
DataNodes

Vu Pham
Mitigation of common failures
Periodic heartbeat: from DataNode to NameNode.
DataNodes without recent heartbeat:
Mark the data. And any new I/O that comes up is not going to be sent to
that data node. Also remember that NameNode has information on all
the replication information for the files on the file system. So, if it knows
that a datanode fails which blocks will follow that replication factor.
Now this replication factor is set for the entire system and also you could
set it for particular file when you're writing the file. Either way, the
NameNode knows which blocks fall below replication factor. And it will
restart the process to re-replicate.

Vu Pham
Checksum computed on file creation.
Checksums stored in HDFS namespace.
Used to check retrieved data.
Re-read from alternate replica

Vu Pham
Multiple copies of central meta data structures.
Failover to standby NameNode- manual by default.

Vu Pham
Performance
Changing blocksize and replication factor can improve
performance.
Example: Distributed copy
Hadoop distcp allows parallel transfer of files.

Vu Pham
Replication trade off with respect to robustness
One performance tradeoff is, actually when you go out
to do some of the map reduce jobs, having replicas
gives additional locality possibilities, but the big trade
off is the robustness. In this case, we said no replicas.
Might lose a node or a local disk: can't recover because
there is no replication.
Similarly, with data corruption, if you get a checksum
that's bad, now you can't recover because you don't
have a replica.
Other parameters changes can have similar effects.

Vu Pham
Conclusion
In this lecture, we have discussed design goals of HDFS,
the read/write process to HDFS, the main configuration
tuning parameters to control HDFS performance and
robustness.

Hadoop Distributed File System in Big data

More Related Content

Similar to Hadoop Distributed File System in Big data (20)

Recently uploaded (20)

Hadoop Distributed File System in Big data