Cloudera Showcase: SQL-on-Hadoop
12 likes3,113 views
This document provides an overview of Cloudera's SQL-on-Hadoop technologies and how to choose the right tool for different jobs. It discusses Hive for batch processing, Impala for interactive SQL and analytics, and SparkSQL for machine learning applications. The document also summarizes performance benchmarks that show Impala outperforming other SQL-on-Hadoop engines in terms of throughput, latency, and scalability. Finally, it briefly outlines new features in Cloudera 5.5 including improvements to Impala, Hive, SparkSQL, and the introduction of Kudu and RecordService.
Cloudera Showcase: SQL-on-Hadoop
- 2. 2© Cloudera, Inc. All rights reserved. § The information in this document is proprietary to Cloudera. No part of this document may be reproduced, copied or transmitted in any form for any purpose without the express prior written permission of Cloudera. § This document is a preliminary version and not subject to your license agreement or any other agreement with Cloudera. This document contains only intended strategies, developments and functionalities of Cloudera products and is not intended to be binding upon Cloudera to any particular course of business, product strategy and/or development. Please note that this document is subject to change and may be changed by Cloudera at any time without notice. § Cloudera assumes no responsibility for errors or omissions in this document. Cloudera does not warrant the accuracy or completeness of the information, text, graphics, links or other items contained within this material. This document is provided without a warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose or non-infringement. § Cloudera shall have no liability for damages of any kind including without limitation direct, special, indirect or consequentialdamages that may result from the use of these materials. The limitation shall not apply in cases of gross negligence.
- 3. 3© Cloudera, Inc. All rights reserved. Cloudera is Built for Production Success Hadoop delivers: • One place for unlimited data • Unified, multi-framework data access Cloudera delivers: • Leading Performance • Enterprise Security • Data Management • Simple Administration Security and Administration Unlimited Storage Process Discover Model Serve Deployment Flexibility On-Premises Appliances Engineered Systems Public Cloud Private Cloud Hybrid Cloud A modern data platform plus what the enterprise requires.
- 4. 4© Cloudera, Inc. All rights reserved. One Platform, Many Workloads Batch, Interactive, and Real-Time. Leading performance and usability in one platform. • End-to-end analytic workflows • Access more data • Work with data in new ways • Enable new users Security and Administration Process Ingest Sqoop, Flume, Kafka Transform MapReduce, Hive, Pig, Spark Discover Analytic Database Impala Search Solr Model Machine Learning SAS, R, Spark, Mahout Serve NoSQL Database HBase Streaming Spark Streaming Unlimited Storage HDFS, HBase YARN, Cloudera Manager, Cloudera Navigator
- 6. 6© Cloudera, Inc. All rights reserved. Hive Batch Processing • User: • SQL-based ETL developers • Designed for: • Handful of concurrent, very long- running batch jobs • Strengths: • Custom file formats • Very long-running ETL, data preparation, or batch processing • Massive ETL sorts with joins • Existing Hive jobs
- 7. 7© Cloudera, Inc. All rights reserved. Impala BI and Analytics User: • Data Analysts • BI Users Designed for: • Interactive SQL for large number of BI users and analysts Strengths: • Multi-user scale • Interactive latency • Compatibility (BI tools, ANSI SQL, and vendor-specific SQL) • Usability
- 8. 8© Cloudera, Inc. All rights reserved. SparkSQL Machine Learning Applications User: • Data Engineers • Data Scientists Designed for: • Ease of development for Spark developers • Handful of concurrent Spark jobs Strengths: • Ease of embedding SQL into Java or Scala applications • SQL for common functionality in developer flow (eg. aggregations, filters, samples)
- 9. 9© Cloudera, Inc. All rights reserved. SQL-on-Hadoop Benchmark Impala, Presto, Stinger, SparkSQL Versions: • Impala 1.4.0 • Presto 0.74 • Stinger Phase 3 (Final) => Hive 0.13.0 • SparkSQL 1.1 • Benchmark Details • Based on industry standards (TPC) • Repeatable (https://github.com/cloudera/impala-tpcds-kit) • Methodical testing with multiple runs on same hardware • Help competing software do well • SQL-92 join style for engines without CBO • JVM tuning for Presto • Run on optimal file formats for each Full Details: http://blog.cloudera.com/blog/2014/09/new-benchmarks-for-sql-on-hadoop-impala-1-4-widens-the- performance-gap/
- 10. 10© Cloudera, Inc. All rights reserved. Impala Multi-User Performance Over 10x Faster with Just 10 Users 0 50 100 150 200 250 300 350 Impala Spark SQL Presto Hive-on-Tez Time (in seconds) Single User vs 10 User Response Time/Impala Times Faster (Lower bars = better) Single User, 5 10 Users, 11 Single User, 25 10 Users, 120 10 Users, 302 10 Users, 202 Single User, 37 Single User, 77 5.0x 10.6x 7.4x 27.4x 15.4x 18.3x
- 11. 11© Cloudera, Inc. All rights reserved. Impala Enables Over 8.7x Throughput More Work Done in Less Time 2333 266 106 175 0 500 1000 1500 2000 2500 Impala Spark SQL Presto Hive-on-Tez Queries per Hour Query Throughput/Impala Throughput Times More Than (Higher bars = better) 8.7x 22.0x 13.3x
- 12. 12© Cloudera, Inc. All rights reserved. Performance Benchmark Takeaways • Impala unlocks BI usage directly on Hadoop • Meets BI low-latency and multi-user requirements • Advantage expands from 5x for single-user to >10x with just 10 users • Hive is designed (and still great) for batch processing • Most Impala customers use Hive for data preparation • Hive is the most commonly used ETL framework • Spark SQL enables easier Spark application development • Enables mixed procedural Spark (Java/Scala) and SQL job development • Mid-term trends will further favor Impala’s design approach for latency and concurrency • More data sets move to memory (HDFS caching, in-memory joins, Intel joint roadmap) • CPU efficiency will increase in importance • Native code enables easy optimizations for CPU instruction sets • Intel joint roadmap support these opportunities
- 13. 13© Cloudera, Inc. All rights reserved. IBM Research Validation • VLDB academic paper compares Impala and Hive (both MR and Tez) for SQL-on-Hadoop • http://www.vldb.org/pvldb/vol7/p1295-floratou.pdf • Impala’s significantly more efficient than Hive/Tez or Hive/MR • Impala’s lead due to CPU efficiency, I/O manager, and overall architecture that resembles a shared-nothing parallel database • Parquet more efficient than ORC • Additional Notes: • Impala 1.4 and higher is significantly faster on selective joins than Impala 1.2.2 used in the paper • Impala 2.0 has disk-based joins and aggregations • Paper compares single-user only. Multi-user would perform even better “Impala’s database-like architecture provides significant performance gains, compared to Hive’s MapReduce or Tez- based runtime” “The Parquet format skips data more efficiently than ORC, which tends to pre-fetch unnecessary data, especially when a table contains a large number of columns”
- 14. 14© Cloudera, Inc. All rights reserved. Major new SQL features in Cloudera 5.5 • Impala • Reliability (particularly with concurrency and scale) • Nested types • Column-level security • Additional functions • Hive • Quality • S3 support • CM monitoring • Navigator lineage • SparkSQL (with DataFrames) • Now supported in CDH 5.5 (recommend HiveContext) • Thriftserver and JDBC not ready for support • Navigator Optimizer (beta) • Helps assess and offload workloads onto Hadoop
- 15. 15© Cloudera, Inc. All rights reserved. 15 Kudu Fills a Critical Gap: Fast Analytics on Fast Changing Data Fast Scans, Analytics and Processing of Stored Data Fast On-Line Updates & Data Serving Unchanging Fast Changing Frequent Updates HDFS HBase Arbitrary Storage (Active Archive) Append-Only Fast Analytics (on fast-changing or frequently-updated data) Real-Time Kudu Kudu fills the Gap Modern analytic applications often require complex data flow & difficult integration work to move data between HBase & HDFS Analytic Gap Pace of Analysis Pace of Data
- 16. 16© Cloudera, Inc. All rights reserved. Current Security Architecture: Inconsistency = Limited Access Policy B Impala (column-level) Policy A Impala ...than others. Some engines support more granular restrictions... Unified, Granular Policy Enforcement Challenge: Hadoop access engines respect policies differently, forcing reliance on lowest common denominator file- or table-based policies, or restricted access. Policy management only solves part of the problem. Solution: RecordService is a new high-performance security layer that centrally enforces access control policy. Complementing Apache Sentry, which provides unified policy definition, it delivers unified row- and column-based security, and dynamic data masking, to every Hadoop access path. Benefits: ● Security: Fine-grained permissions and enforcement across Hadoop, building on Sentry. ● Interoperability: Developers don’t need to be aware of on-disk formats; transparently swap components. RecordService: Unified Access Control Enforcement Spark (table-level) RecordService (policy enforcement) Spark Sentry (policy definition) Sentry (policy definition) ...