2 - Trafodion and Hadoop HBase
Download as PPTX, PDF3 likes848 views
Trafodion is a transactional SQL engine that runs on Hadoop and HBase, providing ANSI SQL access via ODBC/JDBC drivers. It maintains compatibility with Hadoop APIs while adding relational schema support, distributed transactions, secondary indexes and automatic parallelism. Trafodion uses HBase for storage but adds features like ACID compliance across rows and tables and more optimized performance for transactional workloads. By running SQL on Hadoop, Trafodion allows users to leverage existing SQL skills while gaining scalability and flexibility of big data platforms.
2 - Trafodion and Hadoop HBase
- 1. Trafodion Transactional SQL-on-HBase Trafodion and Hadoop / HBase www.trafodion.org HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 2. Trafodion innovation built upon Hadoop stack Leverages Hadoop and HBase for core modules • Maintains API compatibility • Inherited scalability and availability Differentiation • ANSI SQL via ODBC/JDBC • Relational schema abstraction • Distributed transaction protection • Mature SQL technology • Automatic parallelism Hadoop Trafodion Client Application using ODBC/JDBC on Windows/Linux Client Services for ODBC and JDBC HBase HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject 2 to change without notice. Hive HDFS Zookeeper SQL Compiler / Optimizer / Executor Distributed Transaction Manager +
- 3. HBase vs. Trafodion comparison HBase Trafodion + HBase Data abstraction Key and value pair Relational schema Physical Layout Column family store where row data is stored together by cells Same except there is a single column family with space-saving column encoding Column values Uninterpreted array of bytes Explicitly defined and enforced data types ACID Guarantee Single row atomicity Multi- SQL statements, tables, and rows defined as part of transaction Language API Get/put/delete SQL (Trafodion invokes native HBase API) Row Key Index Single (string) row key Composite (multi-column) row key Secondary Indexes Not supported Arbitrary secondary key columns HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject 3 to change without notice.
- 4. Salting of row keys How it works • HBase table gets created, pre-split with one region per salt value • A hash value column, “_SALT_”, is added as a prefix to the row key • Salting is transparent to SQL statements – Automatically computed during insert/update statements – Predicates automatically generated where feasible – Minimal overhead for direct lookup by key value Benefits • Even data distributions across HBase regions • Avoids region hotspots caused by insertion of data in row key order INSERT(s) SELECT(s) HBase Region PART 1 PART 2 PART 3 PART 4 HDFS CREATE TABLE t(a integer not null primary key, b integer) SALT USING 4 PARTITIONS; HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject 4 to change without notice. HBase Region HDFS HBase Region HDFS HBase Region HDFS
- 5. Trafodion and Hadoop – Better Together! Leverages and extends Hadoop for transactional SQL workloads Complete: Full-function ANSI SQL Reuse existing SQL skills and improve developer productivity Protected: Distributed ACID transactions Guarantees data consistency across multiple rows, tables, SQL statements Efficient: Optimized for low-latency read and write transactions Supports real-time transaction processing applications Flexible: Schema flexibility and multi-structured data Seamlessly integrates structured, unstructured, and semi-structured data Interoperable: Standard ODBC/JDBC access Works with existing tools and applications Open: Hadoop and Linux distribution neutral Easy to add to your existing infrastructure and no vendor lock-in HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject 5 to change without notice. Reuse SQL skills Scale without complexity Complements Hadoop Reduce Costs Real-time Performanc e +
- 6. See for yourself… Come discover and develop on Trafodion www.trafodion.org HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject 6 to change without notice.
Editor's Notes
- #2: Welcome to the Trafodion briefing series. Trafodion is a HP sponsored open-source project to deliver an enterprise-class Transactional SQL-on-HBase DBMS solution. The purpose of this segment is to discuss how Trafodion both leverages and extends elements of the Hadoop/HBase infrastructure to support operational workload requirements.
- #3: Trafodion is designed to build upon and leverage Apache Hadoop and HBase core modules and thereby transactional/operational applications using Trafodion transparently gain Hadoop’ s advantages of affordable performance, scalability, elasticity, availability, etc. This graphic depicts a subset of the Hadoop software stack colored in dark grey that are specifically leveraged by Trafodion, namely HBase, HDFS, and Zookeeper. To this stack, Trafodion adds (items colored in blue) the Trafodion ODBC/JDBC drivers, the Trafodion database software, and the DTM for distributed transaction protection. Trafodion interfaces to Hadoop services using their standard APIs. By maintaining API compatibility, Trafodion becomes Hadoop distribution neutral thereby eliminating vendor lock-in by offering customers a choice of distributions to choose from. Trafodion is targeted to deliver differentiation on top of Hadoop in these key areas: A full-featured ANSI SQL implementation whose database services are accessible via a standard ODBC/JDBC connection Provides a SQL relational schema abstraction which makes Trafodion look and feel like any other relational database Distributed ACID transaction protection Mature SQL technology Parallel optimizations for transactional workloads
- #4: Although Trafodion stores its database objects in HBase/HDFS storage structures, it differs and brings value-add over vanilla HBase in a multitude of ways as depicted on this slide. Trafodion provides a relational schema abstraction on top of HBase which allows customers to leverage known and well tested relational design methodologies and SQL programming skills. From a physical layout perspective, Trafodion uses standard HBase storage mechanisms (column family store using key-value pairs) to store and access objects. Trafodion currently stores all columns in a single column family to improve access efficiency and speed for operational data. Additionally Trafodion incorporates a column name encoding mechanism to save space on disk and to reduce messaging overhead for the purposes of improving SQL performance. Unlike vanilla HBase that treats stored data as an uninterpreted array of bytes, Trafodion defined columns are assigned specific data types that are enforced by Trafodion when inserting or updating its data contents. This not only greatly improves data quality/integrity, it also eliminates the need to develop application logic to parse and interpret the data contents. Vanilla HBase provides ACID transaction protection only at the row level. Trafodion extends ACID protection to application defined transactions that can span multiple SQL statements, multiple tables, and rows. This greatly improves database integrity by protecting the database against partially completed transactions i.e. ensuring that either the whole transaction is completely materialized in the database or none of it. HBase’s native API is at a very low level and is not a commonly used programming API. In contrast, Trafodion’ s API is ANSI SQL which is a familiar and well known programming interface and allows companies to leverage existing SQL knowledge and skills. Unlike HBase’s key structure that is comprised of a single uninterpreted array of bytes, Trafodion supports the common relational practice of allowing the primary key to be a composite key comprised of multiple columns. Finally unlike vanilla HBase, Trafodion supports the creation of secondary indexes that can be used to speed transaction performance when accessing row data by a column value that is not the row key. In summary, Trafodion incorporates a number of enhancements over vanilla HBase for the purposes of improving transaction performance, data integrity, and DBA/developer productivity (i.e. by reducing application complexity through the use of standard and well known relational practices and APIs).
- #5: One known problematic area for HBase is supporting transactional workloads where data is inserted into a table in row key order. When this happens, all of the I/O gets concentrated to a single HBase region which in turn creates a server and disk hotspot and performance bottleneck. To alleviate this problem, Trafodion provides a innovative feature called “salting the row key”. To enable this feature the DBA specifies the number of partitions (i.e. regions) the table is to be split over when creating the table i.e. “SALT USING 4 PARTITIONS”. Trafodion creates the table pre-split with one region per salt value. An internal hash value column, “_SALT_”, is added as a prefix to the row key. Salting is handled automatically by Trafodion and is transparent to application written SQL statements. As data is inserted into the table, Trafodion computes the salt value and directs the insert to the appropriate region. Likewise, Trafodion calculates the salt value when data is retrieved from the table and automatically generates predicates where feasible. This is a very lightweight operation with little overhead or impact to direct key access operations. The benefits of salting are that you get more even data distributions across regions and improved performance via hotspot elimination. Next let’s look at other ways Trafodion brings innovation and value add to vanilla HBase.
- #6: Trafodion delivers on the promise of a full featured and optimized transactional SQL-on-HBase DBMS solution with full transactional data protection. This combination of HBase and an enterprise-class transactional SQL engine overcomes Hadoop’s weaknesses in terms of supporting operational workloads. Customers gain the following recognized benefits: Complete: Is a comprehensive and full-functioned ANSI SQL DBMS which allows companies to leverage their in-house SQL learnings and expertise versus having to learn complex map/reduce programming. Protected: Extends Hadoop HBase by adding support for guaranteed transactional consistency across multiple SQL statements, tables, and rows. Efficient: Includes many optimizations for low-latency read and write transactions in support of the fast response time requirements of the operational SQL workloads Trafodion is targeting. Flexible: Trafodion hosted applications gain schema flexibility and seamless integration of data from Trafodion, native HBase, and Hive tables without expensive replication or data movement overhead. Interoperable: Provides investment protection via interoperability with your existing tools and applications using standard ODBC and JDBC access. Open: Is designed to seamlessly fit within customer’s existing IT infrastructure with no vendor lock-in by remaining neutral to the underlying Linux and Hadoop distribution. It complements existing Hadoop investments and benefits. HP open-source sponsorship and investment