The Future of Hadoop by Arun Murthy, PMC Apache Hadoop & Cofounder Hortonworks
4 likes2,508 views
The document discusses the evolution and future directions of Hadoop, emphasizing innovations like YARN and Apache Hive for improved data management and integration. It highlights the shift from batch processing to interactive and real-time applications, focusing on the importance of data governance and security. Additionally, it outlines trends in data applications and the need for real-time intelligence in cybersecurity.
The Future of Hadoop by Arun Murthy, PMC Apache Hadoop & Cofounder Hortonworks
- 1. Hadoop – Looking to the Future Arun C. Murthy Hortonworks Co-Founder @acmurthy
- 2. Page 2 © Hortonworks Inc. 2011 – 2015. All Rights Reserved ACTIONABLE INTELLIGENCE INTERNET OF ANYTHING Hortonworks Connected Data Platforms that power modern data apps with actionable intelligence from ALL data. PERISHABLE INSIGHTS HISTORICAL INSIGHTS HDP for DATA AT REST HDF for DATA IN MOTION MODERN DATA APPS
- 5. 1 ° ° ° ° ° ° ° ° ° ° N HDFS (Hadoop Distributed File System) MapReduce Largely Batch Processing 2006 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Hadoop w/ MapReduce Traditional Hadoop allowed early adopters to deal with data at scale however… • Single purpose clusters, specific data sets • Primarily a batch system using MapReduce • Difficult to natively integrate existing applications • Limited enterprise capabilities: Operations, Security & Governance In the beginning…
- 6. 20092006 1 ° ° ° ° ° ° ° ° ° ° N HDFS (Hadoop Distributed File System) MapReduce Largely Batch Processing Hadoop w/ MapReduce MAPREDUCE-‐279 Common data, multiple applications • Support multi-tenant cluster • Batch, interactive & real-time use cases can leverage the most appropriate engine Architectural Center • Consistent security, governance & operations • Ecosystem applications run natively in Hadoop Apache Hadoop 2.0 & YARN October 23, 2013 YARN : Data Operating System 1 ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° N HDFS (Hadoop Distributed File System) Batch Interactive Real-Time
- 7. YARN : Data Operating System 1 ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° N HDFS (Hadoop Distributed File System) Legacy MapReduce Interactive SQL Apache Tez Other Engines & Workloads Apache Hive SQL Business Analytics Custom Apps Apache Hive and the Power of YARN Stinger Initiative Next generation SQL based interactive query in Hadoop Speed Performance increased 100x for interactive & batch use cases Scale Queries from GBs, to TBs to PBs SQL Broadest range of SQL semantics Apache Hive Community 1,672 Jira Tickets Closed 145 Developers 44 Companies ~390,000 Lines Of Code Added… (2x) 13 Months Hive 13 Hive 12 Hive 10 Dramatically faster queries speeds time to insight seconds thousands of seconds
- 8. YARN : Data Operating System 1 ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° N HDFS (Hadoop Distributed File System) Legacy MapReduce Interactive SQL Apache Tez Other Engines & Workloads Apache Hive SQL Business Analytics Custom Apps Apache Hive – Interactive SQL in Hadoop Stinger Next generation SQL based interactive query in Hadoop ORC IO Improvements Efficient processing via complex pushdown Tez Powerful primitives for the SQL Planner VQP Efficient CPU utilization in Inner Loop
- 9. Sub-Second SQL with Hive LLAP Stinger.Next Sub-second SQL in Hadoop via Hive/ LLAP CBO The “right” plan executed violently… LLAP Metastore Extensive stats & scalability YARN : Data Operating System 1 ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° N HDFS (Hadoop Distributed File System) LLAP Apache Tez Other Engines & Workloads Apache Hive Sub-second SQL Business Analytics Custom Apps Long-lived daemon for low- latency startup, caching & CPU efficiency via JIT
- 10. © Hortonworks Inc. 2015. All Rights Reserved Apache Atlas - Data Governance Initiative Requirements 1. Hadoop must snap in to the existing frameworks and openly exchange metadata 2. Hadoop must address governance within its own stack of technologies Engineers from a group of companies dedicated to meeting these requirements in the open New Apache project proposal Knowledge Store Audit Store (Ranger) ModelsType-System Policy RulesTaxonomies Tag Based Policies Data Lifecycle Management (Falcon) Real-time Tag-based Access Control (Ranger) REST API Services Search Lineage Exchange Healthcare HIPAA HL7 Financial SOX Dodd-Frank Energy PPDM Retail PCI PII Other CWM
- 11. © Hortonworks Inc. 2015. All Rights Reserved HDFS – Tiered Storage
- 12. © Hortonworks Inc. 2015. All Rights Reserved Looking Ahead…
- 13. © Hortonworks Inc. 2015. All Rights Reserved Data Trends Internet of Anything Aggregate any and all IoAT data from sensors, machines, geolocation, clicks, files, social Mediate secure point-to- point and bi-directional data flows
- 14. © Hortonworks Inc. 2015. All Rights Reserved 14 Data Trends It is cheap to create, collect and curate all data
- 15. © Hortonworks Inc. 2015. All Rights Reserved 15 Data Trends ContainerizaCon Virtualization
- 16. © Hortonworks Inc. 2015. All Rights Reserved 16 Data Trends Modern Applications Are Data Applications Easy to Consume & Operate Secure Repeatable
- 17. © Hortonworks Inc. 2015. All Rights Reserved Assemble Modern Data ApplicaHons YARN.NEXT Assemble Select Engines & Services Wire Secure & Operate Service Container Container Container Data Service Data Service Service Engine Service Data Service Service Container Engine Data Security Admin Container
- 18. © Hortonworks Inc. 2015. All Rights Reserved Real-Time Cyber Security It’s not just how quickly or how much data you ingest – it’s about ingesting and enriching data in real-time in order to provide actionable intelligence that stop cyber threats…
- 19. © Hortonworks Inc. 2015. All Rights Reserved Ex. Real-Time Cyber Security with Hortonworks Raw Network Stream Network Metadata Stream Data Stores Syslog Raw Application Logs Other Streaming Telemetry ParseandFormat Enrich Persist Applications and Analyst Tools (ex. Zeppelin on Spark) Log Mining and Analysis Network Packet Mining and PCAP Reconstruction Big Data Exploration, Predictive Modeling SOLR HBase Hive Threat Intelligence Feeds Enrichment Data Real-Time Index Raw Packet Store Long-Term Store Deliver actionable insights from real-time and historical network threat alerts Key components: NiFi -> Kafka -> Storm