Hive2 Introduction -- Interactive SQL for Big Data
7 likes1,371 views
The document presents an overview of Apache Hive 2, highlighting its advantages as a data warehouse on Hadoop, including dynamic security, scalability, and compatibility with major BI tools. Key features of HDP 2.6 for Hive include significant improvements like Hive LLAP for performance, ACID merge for data management, and integration with Druid for real-time analytics. The document also discusses various use cases and optimization strategies for enterprise data warehousing.
Hive2 Introduction -- Interactive SQL for Big Data
- 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved About Me à 蒋 燚峰 (Yifeng Jiang) à Solutions Engineering Lead, Hortonworks – Hadooper since 2009 – HBase book author – Software engineer, cloud, PaaS, DevOps à Jogger, hiker à Twitter: @uprush
- 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Hive’s Unique Advantages Why Hive: • The data warehouse on Hadoop • Per-User dynamic row and column security. • Replication and DR for critical workloads. • Compatible with every major BI Tool. • Proven at 300+ PB Scale. • Significant innovation in Hive 2
- 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved HDP 2.6 Continues Strong Momentum for Hive à At a High Level: – 1200+ features, improvements and bug fixes in Hive since HDP 2.5. – 400+ of these from outside of Hortonworks. à Major Improvements: – Hive LLAP Now GA – ACID MERGE – SQL: All 99 TPC-DS out-of-the-box with only trivial rewrites – Tech Preview: Hive OLAP Indexes powered by Druid 820 413 From Hortonworks From Community Hive 2 in HDP 2.6 Improvements Hive LLAP GA+ SQL MERGE+ All TPC-DS Queries+
- 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Hive LLAP – MPP Performance at Hadoop Scale Deep Storage YARN Cluster LLAP Daemon Query Executors LLAP Daemon Query Executors LLAP Daemon Query Executors LLAP Daemon Query Executors Query Coordinators Coord- inator Coord- inator Coord- inator HiveServer2 (Query Endpoint) ODBC / JDBC SQL Queries In-Memory Cache (Shared Across All Users) HDFS and Compatible S3 WASB Isilon
- 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Hive LLAP in HDP 2.6: Stable Performance with High Concurrency 4x Queries, 2.8x Runtime Difference 5x Queries, 4.6x Runtime Difference Mark Concurrent Queries Average Runtime 5 7.76s 25 36.24s 100 102.89s
- 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Hive LLAP RAM & SSD Cache à Use the combination of DRAM and SSD to dynamically cache data. à Cache 4x more data than using DRAM alone. à Deliver fast analytics on larger datasets with higher concurrency. à Especially good for cloud environment. Highlights DRAM Cache SSD Cache Deep Storage Deep Storage Deep Storage
- 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved ACID MERGE Ã ACID MERGE in Hive, based on ANSI standard SQL. Ã Efficiently perform record-level inserts, updates and deletes within Hive tables. Ã Delivers real Data Management in Hadoop, massively simplifying updates, deletes and change data capture. Highlights
- 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Hive View 2.0 à Create and manage database and tables. à View & compute table and column stats with one click à View query explain plans with costs. Highlights
- 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Tez UI Ã Powerful new search capabilities to help you find queries that need to be optimized. Ã New Total Timeline View shows exactly where query time is spent to quickly pinpoint Hive query bottlenecks. Highlights
- 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved What Is Druid? Druid is a distributed, real-time, column-oriented datastore designed to quickly ingest and index large amounts of data and make it available for real-time query. Features: • Streaming Data Ingestion • Real-Time Query • Merge Historical and Real-Time Data • Approximate Computation
- 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved (Tech Preview) Hive + Druid = Insight When You Need It OLAP Cubes SQL Tables Streaming Data Historical Data Unified SQL Layer Pre-Aggregate ACID MERGE Easily ingest event data into OLAP cubes Keep data up-to-date with Hive MERGE Build OLAP Cubes from Hive Archive data to Hive for history Run OLAP queries in real-time or Deep Analytics over all history Deep AnalyticsReal-Time Query
- 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Preview: OLAP Analytics in Milliseconds with Hive over Druid .0 .5 1.0 1.5 2.0 2.5 3.0 Q1.1 Q1.2 Q1.3 Q2.1 Q2.2 Q2.3 Q3.1 Q3.2 Q3.3 Q3.4 Q4.1 Q4.2 Q4.3 Runtime (s) Star Schema Benchmark 1TB Scale with Hive over 10 Druid Nodes
- 18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Create Druid Cube from Hive
- 19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Create Druid Cube from Hive
- 20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Query Druid Cube from Hive
- 25. 25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Hive2 EDW Optimization Use Cases ETL/ELT DATA MART DATA LANDING & DEEP ARCHIVE CUBE MART END USER APPLICATIONS APPLICATIONS APPLICATIONS END USERS AND APPS EDW OPTIMIZATIONUse Case HDP Advantage Fast BI on Hadoop Hive LLAP in-memory architecture makes Fast BI a reality using Hadoop- native technologies. ETL Offload Save 50-90% of EDW CPU cycles by offloading ETL to the scale-out HDP platform. Active Archive With cost per terabyte on-par with tape, HDP lets you store and analyze years of data rather than months.