Discover HDP 2.1: Apache Hadoop 2.4.0, YARN & HDFS
18 likes4,761 views
The document discusses Hortonworks Data Platform (HDP) 2.1, focusing on its integration of Apache Hadoop 2.4.0, YARN, and HDFS. It highlights new features such as enhanced security measures, resource management, and data processing capabilities. The agenda includes an overview of YARN in HDFS, the new features in HDP 2.1, and a Q&A session.
Discover HDP 2.1: Apache Hadoop 2.4.0, YARN & HDFS
- 1. Page 1 © Hortonworks Inc. 2014 Discover HDP 2.1 Apache Hadoop 2.4.0, YARN & HDFS Hortonworks. We do Hadoop.
- 2. Page 2 © Hortonworks Inc. 2014 Speakers Justin Sears Hortonworks Product Marketing Manager Rohit Bakhshi Hortonworks Senior Product Manager & PM for Apache Hadoop & Apache Solr in Hortonworks Data Platform Vinod Vavilapalli Foundational Hadoop Architect, Hortonworks Engineer, PMC for Apache Hadoop & Leads YARN Development at Hortonworks
- 3. Page 3 © Hortonworks Inc. 2014 Agenda • Overview of YARN in HDFS • New YARN & HDFS Features in HDP 2.1 • Q & A
- 4. Page 4 © Hortonworks Inc. 2014 OPERATIONS TOOLS Provision, Manage & Monitor DEV & DATA TOOLS Build & Test A Modern Data ArchitectureAPPLICATIONS DATA SYSTEM REPOSITORIES RDBMS EDW MPP Business Analy<cs Custom Applica<ons Packaged Applica<ons Governance &Integration ENTERPRISE HADOOP Security Operations Data Access Data Management SOURCES OLTP, ERP, CRM Systems Documents, Emails Web Logs, Click Streams Social Networks Machine Generated Sensor Data GeolocaCon Data
- 5. Page 5 © Hortonworks Inc. 2014 HDP 2.1: Enterprise Hadoop HDP 2.1 Hortonworks Data Platform Provision, Manage & Monitor Ambari Zookeeper Scheduling Oozie Data Workflow, Lifecycle & Governance Falcon Sqoop Flume NFS WebHDFS YARN : Data Opera<ng System DATA MANAGEMENT DATA ACCESS GOVERNANCE & INTEGRATION OPERATIONS Script Pig Search Solr SQL Hive/Tez, HCatalog NoSQL HBase Accumulo Stream Storm Others In-‐Memory AnalyCcs, ISV engines 1 ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° N HDFS (Hadoop Distributed File System) Batch Map Reduce SECURITY Authen<ca<on Authoriza<on Accoun<ng Data Protec<on Storage: HDFS Resources: YARN Access: Hive, … Pipeline: Falcon Cluster: Knox
- 6. Page 6 © Hortonworks Inc. 2014 HDP 2.1: Data Management HDP 2.1 Hortonworks Data Platform Provision, Manage & Monitor Ambari Zookeeper Scheduling Oozie Data Workflow, Lifecycle & Governance Falcon Sqoop Flume NFS WebHDFS DATA ACCESS GOVERNANCE & INTEGRATION OPERATIONS Script Pig Search Solr SQL Hive/Tez, HCatalog NoSQL HBase Accumulo Stream Storm Others In-‐Memory AnalyCcs, ISV engines Batch Map Reduce SECURITY Authen<ca<on Authoriza<on Accoun<ng Data Protec<on Storage: HDFS Resources: YARN Access: Hive, … Pipeline: Falcon Cluster: Knox YARN : Data Opera<ng System DATA MANAGEMENT 1 ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° N HDFS (Hadoop Distributed File System)
- 7. Page 7 © Hortonworks Inc. 2014 Agenda Overview Features Q & A
- 8. Page 8 © Hortonworks Inc. 2014 Apache Hadoop YARN and HDFS Flexible Enables other purpose-built data processing models beyond MapReduce (batch), such as interactive and streaming Efficient Double processing IN Hadoop on the same hardware while providing predictable performance & quality of service Shared Provides a stable, reliable, secure foundation and shared operational services across multiple workloads The Data Operating System for Hadoop 2.0 Data Processing Engines Run Na<vely IN Hadoop BATCH MapReduce INTERACTIVE Tez STREAMING Storm IN-‐MEMORY Spark GRAPH Giraph SAS LASR, HPA ONLINE HBase, Accumulo OTHERS HDFS: Redundant, Reliable Storage YARN: Cluster Resource Management
- 9. Page 9 © Hortonworks Inc. 2014 Agenda Overview Features Q & A
- 10. Page 10 © Hortonworks Inc. 2014 HDP 2.1 HDFS: What’s New HDFS Extended ACLs • Provides granular access control to datasets in HDFS Security THEME HTTPs Wire Encryp<on • swebhdfs: HTTPs support for WebHDFS • HTTPs support for Hadoop WebUI Security THEME HDFS DataNode Caching • Enhanced read performance via in memory caching of files Performance THEME
- 11. Page 11 © Hortonworks Inc. 2014 HDFS Coordinated DataNode Caching • In memory cache for HDFS file - enhanced read performance • Identify files to be cached through centralized management controls • Manage caching through pools and directives
- 12. Page 12 © Hortonworks Inc. 2014 HDP 2.1 YARN: What’s New Resource Manager High Availability • No service disrupCon in YARN Reliability THEME Applica<on Timeline Server • Operational monitoring across all YARN applications Monitoring THEME Capacity Scheduler Pre-‐emp<on • Enforce SLAs across applicaCons and organizaCons Scheduling THEME
- 13. Page 13 © Hortonworks Inc. 2014 YARN Resource Manager (RM) HA Automated failover HDP detects and reacts to Resource Manager host & process failures Active/Standby Standby ResourceManager with access to shared state store Fencing Protection against Split Brain Full stack resiliency - Entire HDP Stack certified with ResourceManager HA - RM Restart enables application recovery Integrated into HDP stack - No external HA Frameworks - No external storage needed
- 14. Page 14 © Hortonworks Inc. 2014 Client Standby RM Active RM ZooKeeper Service Cluster Monitor and try to take active lock Monitor and maintain active lock Store State YARN RM HA: Architecture NodeManager NodeManager NodeManager
- 15. Page 15 © Hortonworks Inc. 2014 Application Timeline Server Entity and Event collection Applications of all types can create entities and send events Pluggable store Depending on site requirements REST APIs Applications and user-interfaces can access information via REST Visualizations Users can build tools and visualizations using the APIs Users and Admins Applications as well as the system entities/ events
- 16. Page 16 © Hortonworks Inc. 2014 Application Timeline Server App Timeline Server AMBARI Custom App Monitoring Client
- 17. Page 17 © Hortonworks Inc. 2014 Capacity Scheduler Preemption • Enforce SLAs • Preempt across queues 1. Current Capacity 2. Guaranteed Capacity 3. Pending Requests Gather Queue State STEP 1 1. Figure out what is needed to achieve capacity balance 2. Select applications to preempt: Over cap. Qs and FIFO order 3. Respect bounds on amount of preemption allowed for each round Iden<fy set of preemp<ons STEP 2 1. Remove reservations from the most recently assigned app 2. Issue preemptions for containers of same app (reverse chronological order, last assigned container first) 3. App Master pre-emption is last resort. Preempt applica<on(s) STEP 3 1. Track containers that have been issued by not yet executed preemption 2. After a set of execution periods, forcibly kill these containers Kill containers STEP 4
- 18. Page 18 © Hortonworks Inc. 2014 Agenda Overview Features Q & A
- 19. Page 19 © Hortonworks Inc. 2014 Learn More About the Hadoop Operating System Hortonworks.com/labs/yarn/ Register for the remaining 3 Discover HDP 2.1 Webinars Hortonworks.com/ webinars Next Webinar: Apache Solr for Hadoop Search Thursday, June 12, 10am Pacific