Hadoop Developer
9 likes4,155 views
In YARN, the functionality of JobTracker has been replaced by ResourceManager and ApplicationMaster. The ResourceManager replaces the JobTracker and manages the resources across the cluster. It schedules the applications on the nodes based on their resource requirements and availability. The ApplicationMaster coordinates and manages the execution of individual applications submitted to YARN, such as MapReduce jobs. It negotiates resources from the ResourceManager and works with the NodeManagers to execute and monitor the tasks. So in summary, the JobTracker's functionality is replaced by: - ResourceManager (for resource management and scheduling) - ApplicationMaster (for coordinating individual application execution)
Hadoop Developer
- 2. Course Topics Slide 2 www.edureka.in/hadoop Module 1 Understanding Big Data Hadoop Architecture Module 2 Hadoop Cluster Configuration Data loading Techniques Hadoop Project Environment Module 3 Hadoop MapReduce framework Programming in Map Reduce Module 4 Advance MapReduce MRUnit testing framework Module 5 Analytics using Pig Understanding Pig Latin Module 6 Analytics using Hive Understanding HIVE QL Module 7 Advance Hive NoSQL Databases and HBASE Module 8 Advance HBASE Zookeeper Service Module 9 Hadoop 2.0 – New Features Programming in MRv2 Module 10 Apache Oozie Real world Datasets and Analysis Project Discussion
- 3. Topics for Today Slide 3 www.edureka.in/hadoop Revision Hadoop 2.0 New Features HDFS High Availability HDFS Federation YARN or MRv2 YARN and Hadoop ecosystem YARN Components YARN Application Execution Running an Application with YARN Writing a YARN Application
- 4. Lets’s Revise – HBase and ZooKeeper Master HFile RegionServers ReRgeigoinoSneSrevrevresrs memstore WAL /hbase/region 1 /hbase/region 2 ….. ….. /hbase/region Zookeeper HDFS Slide 4 www.edureka.in/hadoop
- 5. Client HDFS Map Reduce Secondary Name Node Data Blocks …. Data Node Name Node Job Tracker Task Tracker Map Reduce Data Node Task Tracker Map Reduce Hadoop 1.0 – In Summary Data Node Data NodeTask Tracker Map Reduce Task Tracker Map Reduce Slide 5 www.edureka.in/hadoop
- 6. Hadoop 1.0 - Challenges Slide 6 www.edureka.in/hadoop Problem Description NameNode – No Horizontal Scalability Single NameNode and Single Namespaces, limited by NameNode RAM NameNode – No High Availability (HA) NameNode is Single Point of Failure, Need manual recovery using Secondary NameNode in case of failure Job Tracker – Overburdened Spends significant portion of time and effort managing the life cycle of applications MRv1 – Only Map and Reduce tasks Humongous Data stored in HDFS remains unutilized and cannot be used for other workloads such as Graph processing etc.
- 7. NameNode – Scale and HA NameNode - No High Availability NameNode - No Horizontal Scale Data Node Data Node Data Node …. Client get Block Locations Block Management Read Data NameNode NS Slide 7 www.edureka.in/hadoop
- 8. Secondary NameNode: “Not a hot standby” for the NameNode Connects to NameNode every hour* Housekeeping, backup of NemeNode metadata Saved metadata can build a failed NameNode You give me metadata every hour, I will make it secure Secondary NameNode Single Point of FailureNameNode metadata Slide 8 www.edureka.in/hadoop metadata Name Node –Single Point of Failure
- 9. Job Tracker – Overburdened CPU Spends a very significant portion of time and effort managing the life cycle of applications Network Single Listener Thread to communicate thousands of Map and Reduce Jobs with Task Tracker Task Tracker Task Tracker…. Job Tracker Slide 9 www.edureka.in/hadoop
- 10. MRv1 – Unpredictability in Large Clusters As the cluster size grow and reaches to 4000 Nodes Cascading Failures The DataNode failures results in a serious deterioration of the overall cluster performance because of attempts to replicate data and overload live nodes, through network flooding. Multi-tenancy As clusters increase in size, you may want to employ these clusters for a variety of models. MRv1 dedicates its nodes to Hadoop and cannot be re-purposed for other applications and workloads in an Organization. With the growing popularity and adoption of cloud computing among enterprises, this becomes more important. Slide 10 www.edureka.in/hadoop
- 11. Terabytes and Petabytes of data in HDFS can be used only for MapReduce processing Unutilized Data in HDFS www.edureka.in/hadoop
- 12. Hadoop 2.0 New Features Slide 12 www.edureka.in/hadoop Other important Hadoop 2.0 features HDFS Snapshots NFSv3 access to data in HDFS Support for running Hadoop on MS Windows Binary Compatibility for MapReduce applications built on Hadoop 1.0 Substantial amount of Integration testing with rest of the projects (such as Hadoop ecosystem PIG, HIVE) in Property Hadoop 1.0 Hadoop 2.0 Federation One Namenode and Namespaces Multiple Namenode and Namespaces High Availability Not present Highly Available YARN - Processing Control and Multi- tenancy JobTracker, Task Tracker Resource Manager, Node Manager, App Master, Capacity Scheduler
- 13. Hadoop 2.0 Cluster Architecture - Federation Namenode Block Management NS Storage Datanode Datanode… NamespaceBlockStorage Namespace NS1 NSk NSn NN-1 NN-k NN-n Common Storage Datanode 1 … Datanode 2 … Datanode m … BlockStorage Pool 1 Pool k Pool n Block Pools … … Hadoop 1.0 Hadoop 2.0 Slide 13 www.edureka.in/hadoop http://hadoop.apache.org/docs/stable2/hadoop-project-dist/hadoop-hdfs/Federation.html
- 14. - provides cross-data center (non- Hlo eca lll o) s Tup hp eo rr et !fo !r HDFS, allowing Slide 14 www.edureka.in/hadoop My name is Annie. I love quizzes and puzzles and I am here to make you guys think and answer my questions. How does HDFS Federation help HDFS Scale horizontally? - reduces the load on any single NameNode by using the multiple, independent NameNode to manage individual parts of the filesystem namespace a cluster administrator to split the Block Storage outside the local cluster. Annie’s Question
- 15. A. In order to scale the name service horizontally, HDFS federation uses multiple independent Namenodes. The Namenodes are federated, that is, the Namenodes are independent and don’t require coordination with each other. Slide 15 www.edureka.in/hadoop Annie’s Answer
- 16. /finance respectively. What will happ Hen elif loyo Tu htr ey reto !!put a file in Slide 16 www.edureka.in/hadoop My name is Annie. I love quizzes and puzzles and I am here to make you guys think and answer my questions. You have configured two name nodes to manage /marketing and /accounting directory? Annie’s Question
- 17. The put will fail. None of the namespace will manage the file and you will get an IOException with a No such file or directory error. Slide 17 www.edureka.in/hadoop Annie’s Answer
- 18. Node Manager HDFS YARN Resource Manager Shared edit logs All name space edits logged to shared NFS storage; single writer (fencing) Read edit logs and applies to its own namespace Secondary Name Node Data Node Standby NameNode Active NameNode Container App Master Node Manager Data Node Container App Master Data Node Container Data Node NDoadtae NMoadneager Client App Master Node Manager Container App Master Hadoop 2.0 Cluster Architecture - HA NameNode High Availability Slide 18 www.edureka.in/hadoop Next Generation MapReduce HDFS HIGH AVAILABILITY http://hadoop.apache.org/docs/stable2/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithNFS.html *Not necessary to configure Secondary NameNode
- 19. NameNode Recovery Vs. Failover High Availability in Hadoop 2.0 NameNode recovery in Hadoop 1.0 Secondary Name Node Standby NameNode Active NameNode Secondary Name Node NameNode Edit logs Meta-Data Automatic failover to Standby NameNode Manually Recover using Secondary NameNode FSImage www.edureka.in/hadoop
- 20. HDFS HA was developed to overcome the following disadvantage in Hadoop 1.0? - Single Point Of Failure Of Name•Node - Only one version can be run in cHlaseslilcoMTaph•eRreed!u!ce - Too much burden on Job TracMkeyr name is Annie. I love quizzes and puzzles and I am here to make you guys think and answer my questions. Slide 20 www.edureka.in/hadoop Annie’s Question
- 21. Single Point of Failure of NameNode. Slide 21 www.edureka.in/hadoop Annie’s Answer
- 22. Hadoop 2.0 – In Summary Client HDFS Distributed Data Storage Active NameNode YARN Resource Manager Applications Secondary Name Node Standby NameNode Distributed Data Processing Data Node Node Manager Container App Master ……. Masters Slaves Node Manager Data Node Container App Master Data Node Node Manager Container App Master Shared edit logs Scheduler Manager (AsM) NameNode High Availability www.edureka.in/hadoop Next Generation MapReduce
- 23. YARN and Hadoop Ecosystem Apache Oozie (Workflow) HDFS (Hadoop Distributed File System) Pig Latin Data Analysis Hive DW System MapReduce Framework HBase Apache Oozie (Workflow) HDFS (Hadoop Distributed File System) Pig Latin Data Analysis Hive DW System MapReduce Framework HBase Other YARN Frameworks (MPI, GIRAPH) framework Slide 23 www.edureka.in/hadoop YARN Cluster Resource Management YARN adds a more general interface to run non-MapReduce jobs (such as Graph Processing) within the Hadoop
- 24. BATCH (MapReduce) INTERACTIVE (Text) ONLINE (HBase) STREAMING (Storm, S4, …) GRAPH (Giraph) IN-MEMORY (Spark) HPC MPI (OpenMPI) OTHER (Search) (Weave..) YARN – Moving beyond MapReduce Slide 24 www.edureka.in/hadoop http://hadoop.apache.org/docs/stable2/hadoop-yarn/hadoop-yarn-site/YARN.html
- 25. Multi-tenancy - Capacity Scheduler Slide 25 www.edureka.in/hadoop Organizes jobs into queues Queue shares as %’s of cluster FIFO scheduling within each queue Data locality-aware Scheduling Hierarchical Queues To manage the resource within an organization. Capacity Guarantees A fraction to the total available capacity allocated to each Queue. Security To safeguard applications from other users. Elasticity Resources are available in a predictable and elastic manner to queues. Multi-tenancy Set of limit to prevent over-utilization of resources by a single application. Operability Runtime configuration of Queues. Resource-based scheduling If needed, Applications can request more resources than the default.
- 26. Executing MapReduce Application on YARN www.edureka.in/hadoop MapReduce Application Execution
- 27. YARN MR Application Execution Flow Slide 27 www.edureka.in/hadoop MapReduce Job Execution Job Submission Job Initialization Tasks Assignment Tasks’ Memory Status Updates Failure Recovery Client Submit a MapReduce Job Resource Manager Manage the resource utilization across Hadoop Cluster Node Manager Runs on each Data Node Creates execution container Monitors Container’s usage MapReduce Application Master Coordinates and Manages MapReduce Jobs Negotiates with Resource Manager to schedule asks The tasks are started by NodeManager(s) HDFS shares resources and Job’s artefacts among YARN components
- 28. YARN = Yet Another Resource Negotiator Node Manager Container Container Node Manager App Master Container Node Manager Container App Master Resource Manager Client Client MapReduce Status Job Submission Node Status Resource Request Resource Manager Cluster Level resource manager Long Life, High Quality Hardware Node Manager One per Data Node Monitors resources on Data Node Application Master One per application Short life Manages task /scheduling Hadoop 2.0 - YARN JobHistory Server Slide 28 www.edureka.in/hadoop
- 29. YARN MR Application Execution Flow Client Resource ManagerApplication Node Manager YARNChild Job Object 1. Run Job 2. Get New Application Client JVM 3. Copy Job Resources 4. Submit Application Map or Reduce Task HDFS MR AppMaster DataNode Slide 29 www.edureka.in/hadoop Management Node Task JVM
- 30. YARN MR Application Execution Flow Client Resource ManagerApplication Node Manager YarnChild Job Object 1. Run Job 2. Get New Application Client JVM 3. Copy Job Resources 4. Submit Application 5. Start MR AppMaster container Map or Reduce Task HDFS MR AppMaster DataNode Slide 30 www.edureka.in/hadoop 7. Get Input Splits 8. Request Resources 9. Start container 6. Create container Management Node Task JVM
- 31. YARN MR Application Execution Flow Client Resource Manager HDFS Application Node Manager MR AppMaster YarnChild Map or Reduce Task Job Object 1. Run Job 2. Get New Application Client JVM 3. Copy Job Resources 4. Submit Application 5. Start MR AppMaster container 7. Get Input Splits 8. Request Resources 9. Start container 6. Create container DataNode 12. Execute 10. Create Container 11. Acquire Job Resources Slide 31 www.edureka.in/hadoop Management Node Task JVM
- 32. YARN MR Application Execution Flow Client Resource Manager HDFS Application Node Manager MR AppMaster YarnChild Map or Reduce Task Job Object Client JVM DataNode Management Node Task JVM Poll for Status Slide 32 www.edureka.in/hadoop Update Status
- 33. Hadoop 2.0 : YARN Workflow Resource Manager Node Manager Node Manager App Master 2 Node Manager Node Manager Node Manager Container 2.2 Node Manager Container 2.3 Node Manager Node Manager Container 1.1 Container 2.1 Node Manager Container 1.2 Scheduler Applications Manager (AsM) Node Manager Node Manager Node Manager App Master 1 www.edureka.in/hadoop
- 34. YARN was developed to overcome the following disadvantage in Hadoop 1.0 MapReduce framework? - Single Point Of Failure Of Name•Node - Only one version can be run in cHlaseslilcoMTaph•eRreed!u!ce - Too much burden on Job TracMkeyr name is Annie. I love quizzes and puzzles and I am here to make you guys think and answer my questions. Slide 34 www.edureka.in/hadoop Annie’s Question
- 35. Too much burden on Job Tracker. Slide 35 www.edureka.in/hadoop Annie’s Answer
- 36. Hello There!! My name is Annie. I love quizzes and puzzles and I am here to make you guys think and answer my questions. In YARN, the functionality of JobTracker has been replaced by which of the following YARN features: Slide 36 www.edureka.in/hadoop - Job Scheduling - Task Monitoring - Resource Management - Node management Annie’s Question
- 37. Task Monitoring and Resource Management. The fundamental idea of YARN is to split up the two major functionalities of the JobTracker, i.e. resource management and job scheduling/monitoring, into separate daemons. A global ResourceManager (RM) for resources and per-application ApplicationMaster (AM) for task monitoring. Slide 37 www.edureka.in/hadoop Annie’s Answer
- 38. Hello There!! My name is Annie. I love quizzes and puzzles and I am here to make you guys think and answer my questions. In YARN, which of the following daemons takes care of the container and the resource utilization by the applications?: Slide 38 www.edureka.in/hadoop - Node Manager - Job Tracker - Task tracker - Application Master - Resource manager Annie’s Question
- 40. Hello There!! My name is Annie. I love quizzes and puzzles and I am here to make you guys think and answer my questions. Can we run MRv1 Jobs in a YARN enabled Hadoop Cluster? Slide 40 www.edureka.in/hadoop - Yes - No Annie’s Question
- 41. Yes. You need to recompile the Jobs in MRv2 after enabling YARN to run the Job successfully in a YARN enabled Hadoop Cluster. Slide 41 www.edureka.in/hadoop Annie’s Answer
- 42. Hello There!! My name is Annie. I love quizzes and puzzles and I am here to make you guys think and answer my questions. Which of the following YARN/MRv2 daemon is responsible for launching the tasks? Slide 42 www.edureka.in/hadoop - Task Tracker - Resource Manager - ApplicationMaster - ApplicationMasterServer Annie’s Question
- 44. Write the MapReduce programs using MRv2 for your Module-3 assignments. Assignment Programs – WordCount , Patents, Temperature, and Alphabets Create a Single Node Apache Hadoop Cluster using the documents present in the LMS. Execute the ‘teragen’ example to test the Cluster Slide 44 www.edureka.in/hadoop Module-10 Pre-work
- 45. Thank You See You in Class Next Week
- 46. www.edureka.in/hadoop NN High Availability – Quorum Journal Manager Standby Name Node Active Name Node Data Node Data Node Data Node Data Node …. Data Blocks Journal Node Write namespace modification to edit logs; only Active Name Node is the writer Journal Node Journal Node Read edit logs and applies to its own namespace Data Nodes are configured with the location of both Name Nodes, and send block location information and heartbeats to both.