Hadoop ecosystem
4 likes3,465 views
Hadoop is a distributed processing framework for large datasets. It utilizes HDFS for storage and MapReduce as its programming model. The Hadoop ecosystem has expanded to include many other tools. YARN was developed to address limitations in the original Hadoop architecture. It provides a common platform for various data processing engines like MapReduce, Spark, and Storm. YARN improves scalability, utilization, and supports multiple workloads by decoupling cluster resource management from application logic. It allows different applications to leverage shared Hadoop cluster resources.
Hadoop ecosystem
- 1. HADOOP ECOSYSTEM STANLEY WANG SOLUTION ARCHITECT, TECH LEAD @SWANG68 http://www.linkedin.com/in/stanley-wang-a2b143b
- 2. What is Hadoop? • Distributed, scalable system on commodity hardware • HDFS – Distributed file system • MapReduce – Programming Paradigm • Tools: Hive, Pig, SQOOP, HCatalog, HBase, Flume, Mahout, YARN, Tez, Spark, Stinger, Oozie, ZooKeeper, Flume, Storm, etc • Ideal for processing huge volumes of data, is inadequate for analyzing that data in real time streaming; • Low cost and robust environment to support fault- tolerant for extremely large datasets • Capable of capturing of unstructured, semi-structured, and structured in batch or real-time • Not necessary to create data models, schema-on-read or schema-on-demand data lake • Provides scalable analytics via distributed storage and distributed processing
- 4. Roles in Hadoop Ecosystem
- 6. Not just HADOOP, but with Much More … Manage & store huge volume of any data Hadoop File System MapReduce Manage streaming data Stream Computing Analyze unstructured data Text Analytics Engine Data WarehousingStructure and control data Integrate and govern all data sources Integration, Data Quality, Security, Lifecycle Management, MDM Understand and navigate federated big data sources Federated Discovery and Navigation
- 7. 1st Generation Hadoop: Batch Focus HADOOP 1.0 Built for Web-Scale Batch Apps Single App BATCH HDFS Single App INTERACTIVE Single App BATCH HDFS All other usage patterns MUST leverage same infrastructure Forces Creation of Silos to Manage Mixed Workloads Single App BATCH HDFS Single App ONLINE
- 8. Hadoop 1 Architecture JobTracker Manage Cluster Resources & Job Scheduling TaskTracker Per-node agent Manage Tasks
- 9. Hadoop 1 Limitations Scalability Max Cluster size ~5,000 nodes Max concurrent tasks ~40,000 Coarse Synchronization in JobTracker Availability Failure Kills Queued & Running Jobs Hard partition of resources into map and reduce slots Non-optimal Resource Utilization Lacks Support for Alternate Paradigms and Services Iterative applications in MapReduce are 10x slower
- 12. Hadoop 2 - YARN Architecture ResourceManager (RM) Manages and allocates cluster resources Central agent NodeManager (NM) Manage Tasks, Enforce Allocations Per-Node Agent Resource Manager MapReduce Status Job Submission Client Node Manager Node Manager Container Node Manager App Mstr Node Status Resource Request
- 13. Data Processing Engines Run Natively IN Hadoop BATCH MapReduce INTERACTIVE Tez STREAMING Storm, S4, … GRAPH Giraph MICROSOFT REEF SAS LASR, HPA ONLINE HBase OTHERS Apache YARN HDFS2: Redundant, Reliable Storage YARN: Cluster Resource Management Flexible Enables other purpose-built data processing models beyond MapReduce (batch), such as interactive and streaming Efficient Double processing IN Hadoop on the same hardware while providing predictable performance & quality of service Shared Provides a stable, reliable, secure foundation and shared operational services across multiple workloads The Data Operating System for Hadoop 2.0
- 14. 5 Key Benefits of YARN 1. New Applications & Services 2. Improved cluster utilization 3. Scale 4. Experimental Agility 5. Shared Services
- 15. Key Improvements in YARN Framework supporting multiple applications – Separate generic resource brokering from application logic – Define protocols/libraries and provide a framework for custom application development – Share same Hadoop Cluster across applications Application Agility and Innovation – Use Protocol Buffers for RPC gives wire compatibility – Map Reduce becomes an application in user space unlocking safe innovation – Multiple versions of an app can co-exist leading to experimentation – Easier upgrade of framework and applications
- 16. Key Improvements in YARN Scalability – Removed complex app logic from RM, scale further – State machine, message passing based loosely coupled design Cluster Utilization – Generic resource container model replaces fixed Map/Reduce slots. Container allocations based on locality, memory (CPU coming soon) – Sharing cluster among multiple applications Reliability and Availability – Simpler RM state makes it easier to save and restart (work in progress) – Application checkpoint can allow an app to be restarted. MapReduce application master saves state in HDFS.
- 17. YARN as Cluster Operating System NodeManager NodeManager NodeManager NodeManager map 1.1 vertex1.2.2 NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager map1.2 reduce1.1 Batch vertex1.1.1 vertex1.1.2 vertex1.2.1 Interactive SQL ResourceManager Scheduler Real-Time nimbus0 nimbus1 nimbus2
- 18. YARN APIs & Client Libraries Application Client Protocol: Client to RM interaction –Library: YarnClient –Application Lifecycle control –Access Cluster Information Application Master Protocol: AM – RM interaction –Library: AMRMClient / AMRMClientAsync –Resource negotiation –Heartbeat to the RM Container Management Protocol: AM to NM interaction –Library: NMClient/NMClientAsync –Launching allocated containers –Stop Running containers Use external frameworks like Weave/REEF/Spring
- 19. © Hortonworks Inc. 2013 - Confidential YARN Application Flow Application Client Resource Manager Application Master NodeManager YarnClient App Specific API Application Client Protocol AMRMClient NMClient Application Master Protocol Container Management Protocol App Container
- 20. © Hortonworks Inc. 2013 - Confidential YARN Best Practices Use provided Client libraries Resource Negotiation –You may ask but you may not get what you want - immediately. –Locality requests may not always be met. –Resources like memory/CPU are guaranteed. Failure handling –Remember, anything can fail ( or YARN can pre-empt your containers) –AM failures handled by YARN but container failures handled by the application. Checkpointing –Check-point AM state for AM recovery. –If tasks are long running, check-point task state.
- 21. © Hortonworks Inc. 2013 - Confidential YARN Best Practices Cluster Dependencies –Try to make zero assumptions on the cluster. –Your application bundle should deploy everything required using YARN’s local resources. Client-only installs if possible –Simplifies cluster deployment, and multi-version support Securing your Application –YARN does not secure communications between the AM and its containers.
- 22. © Hortonworks Inc. 2013 - Confidential YARN Future Work ResourceManager High Availability and Work-preserving restart –Work-in-Progress Scheduler Enhancements –SLA Driven Scheduling, Low latency allocations –Multiple resource types – disk/network/GPUs/affinity Rolling upgrades Long running services –Better support to running services like HBase –Discovery of services, upgrades without downtime More utilities/libraries for Application Developers –Failover/Checkpointing