Pivotal HAWQ and Hortonworks Data Platform: Modern Data Architecture for IT Transformation
Download as PPTX, PDF14 likes3,447 views
The document outlines an overview of the Hortonworks Data Platform (HDP) and Pivotal's Big Data Suite, including Pivotal HAWQ’s features and benefits. It discusses the advantages of using HAWQ with HDP, emphasizing performance, ANSI SQL compliance, integrated machine learning, flexible deployment options, and query optimization. Additionally, it highlights the collaboration between Pivotal and Hortonworks in engineering and support for open-source software solutions.
Pivotal HAWQ and Hortonworks Data Platform: Modern Data Architecture for IT Transformation
- 1. 1© 2015 Pivotal Software, Inc. All rights reserved.
- 2. 2© 2015 Pivotal Software, Inc. All rights reserved. Agenda • Hortonworks Data Platform Overview • Pivotal Big Data Suite Overview • Pivotal HAWQ • Demo • Pivotal HAWQ & HDP Business Value and Use Cases • Q&A HAWQ
- 3. 3© 2015 Pivotal Software, Inc. All rights reserved. Your Hosts Parham Parvizi PRODUCT MANAGER, PIVOTAL HAWQ, PIVOTAL Parham Parvizi is a Product Manager at Pivotal, where he is responsible for driving the technical product roadmap of the company's flagship SQL on Hadoop product – Pivotal HAWQ. Shivaji Dutta DEVELOPER EVANGELIST / SR PARTNER SOLUTIONS ENGINEERING, HORTONWORKS Shivaji is Sr. Partner Engineer with Hortonworks. He has over 18 years of Software Development and Consulting Experience.
- 4. Page4 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Hadoop for the Enterprise: Implement a Modern Data Architecture with HDP Customer Momentum • 437 customers (as of March 31, 2015) • 105 customers added in Q1 2015 Hortonworks Data Platform • Completely open multi-tenant platform for any app and any data. • A centralized architecture of consistent enterprise services for resource management, security, operations, and governance. Partner for Customer Success • Open source community leadership focus on enterprise needs • Unrivaled world class support • Founded in 2011 • Original 24 architects, developers, operators of Hadoop from Yahoo! • 600+ Employees • 1100+ Ecosystem Partners
- 5. Page 5 © Hortonworks Inc. 2011 – 2015. All Rights Reserved HDP Makes Hadoop Enterprise-Ready Hortonworks Data Platform Multi-tenant data platform built on a centralized architecture of shared enterprise services YARN: data operating system Governance Security Operations Resource management Existing applications New analytics Partner applications Data access: batch, interactive, real-time Storage Key Benefits • Consolidates all data sets • Delivers real-time insights • Integrates with data center • Scalable and affordable
- 6. Page 6 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Hortonworks Data Platform
- 7. Page 7 © Hortonworks Inc. 2011 – 2015. All Rights Reserved SQL Engines on HDP • Apache Hive + Tez + ORC • Apache Phoenix • Spark SQL (Tech Preview) • HAWQ
- 8. Page 8 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Pivotal and Hortonworks • Joint engineering Pivotal HD and HDP based on a common core Pivotal HAWQ certified on HDP • Co-founders of Open Data Platform PIVOTAL AND HORTONWORKS ARE STRONG DRIVERS OF OPEN SOURCE SOFTWARE PIVOTAL AND HORTONWORKS ARE STRONG DRIVERS OF OPEN SOURCE SOFTWARE ODP# (Enterprise Hardening) Hortonworks Data Platform (HDP) Pivotal Hadoop Distribution Other apps/tools* *ex. Analytics apps and visualization tools #OpenDataPlatform.org
- 9. Page 9 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Pivotal HAWQ on HDP Certified
- 10. 10© 2015 Pivotal Software, Inc. All rights reserved. BUSINESS VALUE FROM DATA Transforming companies into data-driven enterprises with open, agile, cloud-ready end- to-end solutions PLATFORM AT YOUR SERVICE Pioneering an open vision for cloud-based, agile application development A BETTER WAY TO BUILD PRODUCTS World-class application development services, ‘Pivots’, & transformative methodologies Cloud Foundry Big Data Suite Pivotal Labs Pivotal – Business Groups
- 11. 11© 2015 Pivotal Software, Inc. All rights reserved.
- 12. 12© 2015 Pivotal Software, Inc. All rights reserved. HAWQ + HDP HDP Open Enterprise Hadoop Pivotal HAWQ 100% ANSI SQL Performance Complex Query Accessible Pivotal HAWQ
- 13. 13© 2015 Pivotal Software, Inc. All rights reserved. HAWQ + HDP Pivotal HAWQ • Discover New Relationships • Enable Data Science • Analyze External Sources • Query All Data Types! Multi-level Fault Tolerance Granular Authorization Resource Pools High multi-tenancy 100% ANSI SQL Standard OLAP Extensions JDBC ODBC Connectivity MPP Architecture Online Expansion HDFS Petabyte Scale Cost Based Optimizer Dynamic Pipelining ACID + Transaction al Multi-Language UDF Support Built-in Data Science Library Extensible (PXF) Query External Sources Hardened, 10+ Years Tested, Production Proven Accessibility + Usability HDFS Native File Formats • Manage Multiple Workloads • Petabyte Scale Analytics • Sub-second Performance • Leverage Existing Skills & Tools • Easily Integrate with Other Tools Compression + Partitioning core compliance • Well Integrated with Hortonworks Data Platform
- 14. 14© 2015 Pivotal Software, Inc. All rights reserved. Reasons Why Customers Will Prefer HDP + HAWQ 5 Pivotal HAWQ
- 15. 15© 2015 Pivotal Software, Inc. All rights reserved. Reasons Why Customers Will Prefer HDP + HAWQ • Up to 30x SQL on Hadoop performance advantage • Faster time to insight • Massive MPP scalability to petabytes Benefits: Near real-time latency, complex queries and advanced analytics at scale 1. Advanced Analytics Performance Pivotal HAWQ 5
- 16. 16© 2015 Pivotal Software, Inc. All rights reserved. Reasons Why Customers Will Prefer HDP + HAWQ • ANSI SQL-92, -99, -2003 • All 99 TPC-DS queries tested, no modifications • Plus, OLAP extensions • Complete ACID integrity and reliability Benefits: 100% SQL compliant No risk to SQL applications All native on HDP via HAWQ 2. 100% ANSI SQL Compliant Pivotal HAWQ 5
- 17. 17© 2015 Pivotal Software, Inc. All rights reserved. Reasons Why Customers Will Prefer HDP + HAWQ • Advanced machine learning for big data • Local, in database operation • Exceptional MPP/parallel performance • Open source, Postgres-based Benefits: Advanced, highly scalable, machine learning, directly on HDP data 3. Integrated Machine Learning Pivotal HAWQ 5
- 18. 18© 2015 Pivotal Software, Inc. All rights reserved. Reasons Why Customers Will Prefer HDP + HAWQ • HDP and Pivotal HD, easily managed via Ambari • On premises, in cloud, or PaaS • Hbase, Avro, Parquet, ORC and more • Plus, connectors to make HAWQ data available to other SQL query tools Benefits: Flexibility Accessibility Portability 4. Flexible Deployment Pivotal HAWQ 5
- 19. 19© 2015 Pivotal Software, Inc. All rights reserved. Reasons Why Customers Will Prefer HDP + HAWQ • Cost-based query optimization • Robust query plan optimization • Complex big data management Benefits: Optimize performance and costs Maximize HDP cluster resources Offload EDW without compromise 5. Query Optimization Options Pivotal HAWQ 5
- 20. 20© 2015 Pivotal Software, Inc. All rights reserved. HAWQ over Competition - Impala • 100% TPC-DS Compatible • HAWQ completed 58/99 TPC-DS queries 12 hours faster! • Multi-dimensional queries with subqueries, dynamic partition elimination, large table joins, and roll-ups • Higher concurrency • Only partial ANSI-SQL compatibility 58/99 TPC-DS queries • Exposure to application errors due to SQL incompatibilities • Single-dimension queries. • No nesting, small table joins, no roll-ups • Limited performance range • No machine learning! 100% ANSI SQL Querycomplexity&speedrequirements + - 0% Pivotal HAWQ
- 21. 21© 2015 Pivotal Software, Inc. All rights reserved. TPC-DS Results vs. Impala Pivotal HAWQ HAWQ Faster 88% of queries 12 hours Impala Faster 12% of queries Subset of TPC-DS Queries Comparison of HAWQ vs. Impala TPC-DS Queries QueryRuntimeDifference(s)(+HAWQFaster/-ImpalaFaster)
- 22. 22© 2015 Pivotal Software, Inc. All rights reserved. HAWQ Integration with Hive & Value-Add Query all Hive tables via PXF Easily move between HAWQ and Hive Value-Add: – Application sub-second performance and faster time to insight – Integration with traditional BI Reporting tools and complex machine generated SQL – Data Science driven application requiring built-in machine learning – Large queries across multiple dataset to find new relationship and patterns – Silo Analytical application with large ad-hoc users and high multi-tenancy – Complex SQL statements with multi-level selects, partitions and rollups + Pivotal HAWQ
- 23. 23© 2015 Pivotal Software, Inc. All rights reserved. Collaboration on support Full SQL on Hadoop Performance Leadership Common ODP core BDS + HDP Focus on solution life-cycle Exceptional performance, applications run without SQL errors, leverage existing SQL skills No vendor lock-in protects investment and grows ecosystem Apache open source availability of software Benefits Summary
- 24. 24© 2015 Pivotal Software, Inc. All rights reserved. + Open Enterprise Hadoop Powering Digital Transformation Working together to digitally transform companies into innovative enterprises
- 25. 25© 2015 Pivotal Software, Inc. All rights reserved.