Hadoop Architecture in Depth
2 likes1,068 views
The document discusses big data and Hadoop. It defines big data as the large volumes of data created daily by companies like Twitter, Facebook, and Google. It then introduces Hadoop as a framework for distributed processing of large datasets across clusters of computers. The document provides an overview of the key Hadoop components like HDFS for storage and MapReduce for processing. It also describes the Hadoop architecture including the roles of the NameNode, DataNodes and how data is read and written in HDFS.
Hadoop Architecture in Depth
- 1. BigData Syed Solutions Engineer - Big Data mail.syed786@gmail.com info.syedacademy@gmail.com +91-9030477368
- 2. Need For A New Processing Platform (Big Data) What is Big Data ? Twitter (over 7~ TB/day) Facebook (over 10~ TB/day) Google (over 20~ PB/day) Where does it come from ? Existing systems (vertical scalibility) Why Hadoop (horizontal scalibility)?
- 3. Yahoo Google Facebook LinkedIn IBM Amazon HortonWorks Cloudera NY Times … the list goes on. Companies Using Hadoop
- 4. What is Hadoop? Flexible infrastructure for large scale computation & data processing on a network of commodity hardware. Completely written in java. Open source & distributed under Apache license Hadoop Core Components: HDFS & MapReduce. The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models.
- 5. What Hadoop is Not? A database An online transaction processing (OLTP) system Replacement of all programming language
- 6. Hadoop Introduction and Architecture
- 8. HDFS - Hadoop Distributed File System Design of HDFS Where HDFS is not a good fit Why Is a Block in HDFS So Large?
- 10. Let us Zoom into HDFS
- 11. NameNode Deeper Things about Name Node
- 13. What is DataNode? DataNode
- 15. Feature Matrix
- 16. Yahoo Study
- 17. Still need to be fixed
- 18. How Do We Fix a Single NameNode Feature
- 20. NameNode HA(V2)
- 21. NameNode HA – Shared Storage
- 22. NameNode HA
- 23. HDFS Federation
- 24. Hadoop JournalNode JournalNode machines - the machines on which you run the JournalNodes. The JournalNode daemon is relatively lightweight, so these daemons may reasonably be collocated on machines with other Hadoop daemons, for example NameNodes, the JobTracker, or the YARN ResourceManager. Note: There must be at least 3 JournalNode daemons, since edit log modifications must be written to a majority of JNs. This will allow the system to tolerate the failure of a single machine. You may also run more than 3 JournalNodes, but in order to actually increase the number of failures the system can tolerate, you should run an odd number of JNs, (i.e. 3, 5, 7, etc.). Note that when running with N JournalNodes, the system can tolerate at most (N - 1) / 2 failures and continue to function normally.
- 25. Hadoop 1 Limited up to 4,000 nodes per cluster O(# of tasks in a cluster) JobTracker bottleneck - resource management, job scheduling and monitoring Only has one namespace for managing HDFS Map and Reduce slots are static Only job to run is MapReduce
- 26. Hadoop 1 - Reading Files Rack1 Rack2 Rack3 RackN read file (fsimage/edit) Hadoop Client NameNode SNameNode return DNs, block ids, etc. DN | TT DN | TT DN | TT DN | TT DN | TT DN | TT DN | TT DN | TT DN | TT DN | TT DN | TT DN | TT checkpoint heartbeat/ block reportread blocks
- 27. Hadoop 1 - Writing Files Rack1 Rack2 Rack3 RackN request write (fsimage/edit) Hadoop Client NameNode SNameNode return DNs, etc. DN | TT DN | TT DN | TT DN | TT DN | TT DN | TT DN | TT DN | TT DN | TT DN | TT DN | TT DN | TT checkpoint block report write blocks replication pipelining
- 28. Hadoop 2 Potentially up to 10,000 nodes per cluster O(cluster size) Supports multiple namespace for managing HDFS Efficient cluster utilization (YARN) MRv1 backward and forward compatible Any apps can integrate with Hadoop
- 29. Hadoop 2 - Basics
- 30. Hadoop 2 - Reading Files (w/ NN Federation) Rack1 Rack2 Rack3 RackN read file fsimage/edit copy Hadoop Client NN1/ns1 SNameNode per NN return DNs, block ids, etc. DN | NM DN | NM DN | NM DN | NM DN | NM DN | NM DN | NM DN | NM DN | NM DN | NM DN | NM DN | NM checkpoint register/ heartbeat/ block report read blocks fs sync Backup NN per NN checkpoint NN2/ns2 NN3/ns3 NN4/ns4 or ns1 ns2 ns3 ns4 dn1, dn2 dn1, dn3 dn4, dn5 dn4, dn5 Block Pools
- 31. Hadoop 2 - Writing Files Rack1 Rack2 Rack3 RackN request write Hadoop Client return DNs, etc. DN | NM DN | NM DN | NM DN | NM DN | NM DN | NM DN | NM DN | NM DN | NM DN | NM DN | NM DN | NM write blocks replication pipelining fsimage/edit copy NN1/ns1 SNameNode per NN checkpoint block report fs sync Backup NN per NN checkpoint NN2/ns2 NN3/ns3 NN4/ns4 or