Offload, Transform, and Present - The New World of Data Integration
0 likes605 views
Gluent is focused on liberating enterprise data through innovative data integration solutions, specifically emphasizing offloading data from Oracle to Hadoop and enabling seamless data access. The document discusses the challenges of traditional ETL processes, introducing a new paradigm termed 'offload, transform, present' (OTP) that streamlines data movement and analytics. Gluent also offers webinars and tools to facilitate data integration and management across various platforms.
Offload, Transform, and Present - The New World of Data Integration
- 2. • Michael Rainey - Technical Advisor • Spreading the good word about gluent products with the world • Data Integration expertise • Oracle ACE Director • @mRainey 2 Introduction we liberate enterprise data
- 3. 3 About Gluent Globally distributed team with a deep background in building high performance enterprise applications and systems … Now on a mission to liberate enterprise data. Gluent Data Platform Advisor Open Data Formats Security Orchestration Access Offload Present
- 5. 4 Gluent is running one-hour webinars each Wednesday in July beginning tomorrow, July 12. See details below and register for your free spot today! Apache Impala Internals Speaker: Tanel Poder, Gluent Wednesday, July 19 @ 12 PM CDT gluent.com/event/gluent-webinar-apache-impala-internals-with-tanel-poder Building an Analytics Platform with Oracle & Hadoop Speakers: Gerry Moore & Suresh Irukulapati, Vistra Energy Wednesday, July 26 @ 9 AM CDT gluent.com/event/gluent-webinar-building-an-integrated-analytics-platform-with-oracle-and-hadoop Gluent Webinars - JULY 2017
- 20. 9 Typical ETL development - too much time spent on “E” and “L” reader = new TransformingReader(reader) .add(new SetCalculatedField("AvailableCredit", "parseDouble(CreditLimit) - parseDouble(Balance)")) .add(new ExcludeFields("CreditLimit", "Balance")); DataWriter writer = new JdbcWriter(getJdbcConnection(), "dp_credit_balance") .setAutoCloseConnection(true); JobTemplate.DEFAULT.transfer(reader, writer);
- 21. 9 Typical ETL development - too much time spent on “E” and “L” ETL Developer / Data Engineer should spend their time on Transformations! reader = new TransformingReader(reader) .add(new SetCalculatedField("AvailableCredit", "parseDouble(CreditLimit) - parseDouble(Balance)")) .add(new ExcludeFields("CreditLimit", "Balance")); DataWriter writer = new JdbcWriter(getJdbcConnection(), "dp_credit_balance") .setAutoCloseConnection(true); JobTemplate.DEFAULT.transfer(reader, writer);
- 24. • Sharing data without moving data • Metadata is stored about each “source” database • Queries are passed through a metadata layer, which provides info about where the data lives and how to translate the query 12 What about data federation? Source 1 Source 2 Source X Source 1 Source 2 Source X App 1 App 2 App 1 App 2
- 25. • Sharing data without moving data • Metadata is stored about each “source” database • Queries are passed through a metadata layer, which provides info about where the data lives and how to translate the query • But… 12 What about data federation? Source 1 Source 2 Source X Source 1 Source 2 Source X App 1 App 2 App 1 App 2
- 26. • Sharing data without moving data • Metadata is stored about each “source” database • Queries are passed through a metadata layer, which provides info about where the data lives and how to translate the query • But… • We must recode applications! • Data remains in RDBMS silos and are limited to CPU/storage constraints 12 What about data federation? Source 1 Source 2 Source X Source 1 Source 2 Source X App 1 App 2 App 1 App 2
- 27. 13 Three things we need the ability to do: 1. Offload data from Oracle to Hadoop 2. Load data from Hadoop to Oracle 3. Query Hadoop data in Oracle
- 28. • “Offload” - copy or move data from the relational database to Hadoop • Why offload? • Store data in a centralized location for access and sharing • Use the distributed, parallel processing power of Hadoop for transformations • Enable the use of “new world” technologies (Spark, Impala, etc) • Why Hadoop? • We can now afford to keep a copy of all enterprise data for data sharing reasons! 14 Offload, not Extract
- 30. • Command line client used to bulk copy data from a relational database to HDFS over JDBC connection • Also works in reverse, HDFS to RDBMS • Sqoop generates MapReduce jobs for the work 16 Bulk data load with Sqoop sqoop import --connect jdbc:oracle:thin:@myserver:1521/MYDB1 --username myuser --null-string '' --null-non-string '' --target-dir=/user/hive/warehouse/ssh/sales --append -m1 --fetch-size=5000 --fields-terminated-by '','' --lines-terminated-by ''n'' --optionally-enclosed-by ''"'' --escaped-by ''"'' --split-by TIME_ID --query ""SELECT * FROM SSH.SALES WHERE TIME_ID < DATE'1998-01-01'""
- 31. • Gluent Advisor helps determine which tables and/or partitions are candidates for offload • Offload options: • Move or copy 100% of data • Move “cold” partitions only • Data can be kept in-sync with an incremental offload • Example: when a partition goes inactive (cold), it can be offloaded to Hadoop • Data can be updated from RDBMS to Hadoop 17 Offload data with Gluent Data Platform
- 32. • Gluent Advisor helps determine which tables and/or partitions are candidates for offload • Offload options: • Move or copy 100% of data • Move “cold” partitions only • Data can be kept in-sync with an incremental offload • Example: when a partition goes inactive (cold), it can be offloaded to Hadoop • Data can be updated from RDBMS to Hadoop 17 Offload data with Gluent Data Platform gluent. 80% - 20% offload -x -t EDW.BALANCE_DETAIL_MONTHLY --less-than-value=2017-05-01
- 33. • Gluent Advisor helps determine which tables and/or partitions are candidates for offload • Offload options: • Move or copy 100% of data • Move “cold” partitions only • Data can be kept in-sync with an incremental offload • Example: when a partition goes inactive (cold), it can be offloaded to Hadoop • Data can be updated from RDBMS to Hadoop 17 Offload data with Gluent Data Platform gluent. 80% - 20% offload -x -t EDW.BALANCE_DETAIL_MONTHLY --less-than-value=2017-05-01
- 34. 18 Three things we need the ability to do: 1. Offload data from Oracle to Hadoop 2. Load data from Hadoop to Oracle 3. Query Hadoop data in Oracle
- 35. 18 Three things we need the ability to do: 1. Offload data from Oracle to Hadoop 2. Load data from Hadoop to Oracle 3. Query Hadoop data in Oracle Present
- 39. 21 Load the data from Hadoop to RDBMS? Tool Offload Data Load Data Sqoop Yes Yes Oracle Loader for Hadoop Yes Oracle SQL Connector for HDFS Yes ODBC Gateway Yes Big Data SQL Yes Gluent Data Platform Yes Present Present
- 40. 21 Load the data from Hadoop to RDBMS? Tool Offload Data Load Data Sqoop Yes Yes Oracle Loader for Hadoop Yes Oracle SQL Connector for HDFS Yes ODBC Gateway Yes Big Data SQL Yes Gluent Data Platform Yes Present Present
- 41. • Oracle SQL Connector for HDFS • Uses Oracle External Table to access delimited file or Hive table in Hadoop • When queried, all data is read from Hadoop to Oracle, then processed • Oracle to Hadoop over database links • Create a database link from Oracle to Hadoop using the Impala (or Hive) ODBC gateway • Pushes filters to Hadoop but not grouping aggregates • Big Data SQL • Access Hive tables via an Oracle external table • Predicate pushdown (not aggregations & joins) 22 Present data to Oracle
- 43. 23 Gluent Present gluent. present -t "SH.SALES" -xv --target-name="SH.SALES_DIVISION_X"
- 44. 23 Gluent Present gluent. present -t "SH.SALES" -xv --target-name="SH.SALES_DIVISION_X" Gluent Present can pushdown predicates, aggregations, and joins
- 45. 24 Three things we need the ability to do: 1. Offload data from Oracle to Hadoop 2. Load data from Hadoop to Oracle 3. Query Hadoop data in Oracle Present
- 46. 25 How to query the data? There are many options Tool Offload Data Load Data Allow Query Data Offload Query Parallel Execution Sqoop Yes Yes Yes Oracle Loader for Hadoop Yes Yes Oracle SQL Connector for HDFS Yes Yes Yes ODBC Gateway Yes Yes Yes Big Data SQL Yes Yes Yes Yes Gluent Data Platform Yes Yes Yes YesPresent
- 47. 25 How to query the data? There are many options Tool Offload Data Load Data Allow Query Data Offload Query Parallel Execution Sqoop Yes Yes Yes Oracle Loader for Hadoop Yes Yes Oracle SQL Connector for HDFS Yes Yes Yes ODBC Gateway Yes Yes Yes Big Data SQL Yes Yes Yes Yes Gluent Data Platform Yes Yes Yes YesPresent
- 63. • No upfront massive project cost to move data around. • Can happen in minutes / hours, not weeks / months • Large corporations can move as fast as startups • Data Engineers / ETL Developers can focus on transformations! • A new approach to data sharing across enterprise • Offload all data for simple access • Takes a different mindset 32 OTP simplifies the process of data movement!
- 66. 33 Data Consolidation SALES (customer A) Customers Products Preferences Promotions Prices SALES (customer B) Customers Products Preferences Promotions Prices Gluent Gluent Offload Transform (optional) Customers A Customers B -> Raw data SQL
- 67. 33 Data Consolidation SALES (customer A) Customers Products Preferences Promotions Prices SALES (customer B) Customers Products Preferences Promotions Prices Gluent Gluent Offload Transform (optional) Customers A Customers B Customers ALL -> Raw data Consolidated data Union all, SQL, Spark SQL
- 68. 33 Data Consolidation SALES (customer A) Customers Products Preferences Promotions Prices ALL ALL SALES (customer B) Customers Products Preferences Promotions Prices Gluent Gluent Offload Transform (optional) Present Customers A Customers B Products ALL Preferences ALL Prices Gluent Customers ALL Promotions SALES (ALL) -> -> Raw data Consolidated data Customers ALL Union all, SQL, Spark SQLSQL Virtualized tables
- 69. 34 we liberate enterprise data thank you!
- 70. 35 Gluent is running one-hour webinars each Wednesday in July beginning tomorrow, July 12. See details below and register for your free spot today! Apache Impala Internals Speaker: Tanel Poder, Gluent Wednesday, July 19 @ 12 PM CDT gluent.com/event/gluent-webinar-apache-impala-internals-with-tanel-poder Building an Analytics Platform with Oracle & Hadoop Speakers: Gerry Moore & Suresh Irukulapati, Vistra Energy Wednesday, July 26 @ 9 AM CDT gluent.com/event/gluent-webinar-building-an-integrated-analytics-platform-with-oracle-and-hadoop Gluent Webinars - JULY 2017