Data Virtualization And Information As A Service (IaaS)
Download as PPTX, PDF0 likes22 views
The document discusses data virtualization and information as a service, led by Anil Allewar from Synerzip, explaining its benefits over traditional ETL processes, such as real-time data access and reduced errors. It details the architecture, use cases, and components involved in data virtualization, specifically through the open-source platform JBoss Teiid. The conclusion emphasizes the emerging utility of data virtualization in solving traditional business intelligence challenges and improving enterprise data accessibility.
Data Virtualization And Information As A Service (IaaS)
- 1. www.synerzip.comwww.synerzip.com Data Virtualization & Information As A Service (IaaS) By Anil Allewar Senior Solutions Architect - Synerzip 1
- 2. www.synerzip.comwww.synerzip.com About Me!! Confidential 2 Anil Allewar Senior Solutions Architect @ Synerzip Technology Evangelist & speaker Core interests: JEE, EAI, EII
- 3. www.synerzip.comwww.synerzip.com • Use cases Agenda • What does it mean? • Implementation Frameworks • Demo • Questions? 3 • Architecture explained
- 4. www.synerzip.comwww.synerzip.com Why it makes sense? Confidential 4
- 5. www.synerzip.comwww.synerzip.com Use Cases Data Warehouse ETL Financial Data OLTP Data ETL 3rd Party Data Data Mart ETL Web Service 1 Web Service 2 Legacy Data Custom Program Excel files Confidential 5
- 6. www.synerzip.comwww.synerzip.com Traditional Data Integration Enterprise Information System ETL Source System Source System ETL Confidential 6 Business Applications
- 7. www.synerzip.comwww.synerzip.com Problems with ETL Confidential 7 More than 1 copy of data for staging Intermediate data => Errors Lead time to add new source Domain knowledge for mapping Batch Process => No real time data
- 8. www.synerzip.comwww.synerzip.com Problems with DBMS consolidation Confidential 8 Alternate approach => Single EIS (say RDBMS) Extensive changes to existing apps Might not satisfy everyone’s requiremets
- 9. www.synerzip.comwww.synerzip.com • Use cases Agenda • What does it mean? • Implementation Frameworks • Demo • Questions? 9 • Architecture explained
- 10. www.synerzip.comwww.synerzip.com Data Virtualization & Federation Confidential 10 Single API to access data Only metadata stored at virtualization layer Real time access without copying/moving data Federate data across hetero/homogenous sources
- 12. www.synerzip.comwww.synerzip.com • Use cases Agenda • What does it mean? • Implementation Frameworks • Demo • Questions? 12 • Architecture explained
- 13. www.synerzip.comwww.synerzip.com Architecture Confidential 13 User Application CommonAccess API Connector 1 Connector 2 RUNTIME & QUERY ENGINE Virtual Database Translator 1 Translator 2
- 14. www.synerzip.comwww.synerzip.com • Use cases Agenda • What does it mean? • Implementation Frameworks • Demo • Questions? 14 • Architecture explained
- 15. www.synerzip.comwww.synerzip.com Vendors • Commercial Products – Composite Software • http://www.compositesw.com/data-virtualization/ – Denodo • http://www.denodo.com/en/product/overview.php?n=h – IBM • http://www-03.ibm.com/software/products/en/ibminfofedeserv – Informatica • http://www.informatica.com/us/data-virtualization/ – Red Hat • http://www.redhat.com/products/jbossenterprisemiddleware/data-virtualization/ • Open Source – Jboss Teiid • http://teiid.jboss.org/ Confidential 15
- 16. www.synerzip.comwww.synerzip.com Selected Platform – JBoss Teiid Confidential 16 Open Source Number of relational/NoSQL/E RP/CRM data stores JEE standards Add custom EIS support using JEE components Active & responsive community Synerzip contribution: Defect discovery, root cause analysis, feature verification
- 17. www.synerzip.comwww.synerzip.com Teiid Components • Virtual Database – container for components used to integrate data from multiple data sources • Source Models – structure and characteristics of physical data sources • View Models – structure and characteristics of abstract structures you want to expose to your applications • Teiid Designer – Eclipse based UI to dynamically discover data source objects and apply data federation – Generate virtual database from 1 or more sources Confidential 17
- 18. www.synerzip.comwww.synerzip.com Teiid Components • Translator – Provides abstraction later between Teiid Query Engine and source system – Convert Teiid SQL commands to source specific execution commands – Convert result data from source system to Teiid specific format • Resource Adapter – Provides connectivity to the physical data source – Integration provided through Java Connector Architecture (JCA) API Confidential 18
- 19. www.synerzip.comwww.synerzip.com Teiid – Supported EIS • Amazon SimpleDB • Apache Accumulo • Apache SOLR • Cassandra • File • Google Spreadsheet • JPA • LDAP • Excel – as file • SalesForce • JDBC – MS access, DB2, derby, excel- odbc, greenplum, h2 , hive(for accessing Hadoop), oracle, teradata and most RDBMS • MongoDB • Object • OData • OLAP • Web Services • SAP Netweaver Gateway Confidential 19
- 20. www.synerzip.comwww.synerzip.com Performance Characteristics • Access same data using Oracle and Teiid drivers – Retrieval times comparable when accessing tables having no Blobs Confidential 20 0 5,000 10,000 15,000 20,000 25,000 No. of rows Vs Time: No Blobs Oracle-JDBC Teiid-JDBC No. of rows ms
- 21. www.synerzip.comwww.synerzip.com Performance Characteristics – Teiid slower when accessing Blob data • Can be tuned Confidential 21 0 5,000 10,000 15,000 20,000 25,000 30,000 0 0 2 42 21,804 32,531 185,454 No. of rows Vs Time: Blobs Oracle-JDBC Teiid-JDBC ms No. of rows
- 22. www.synerzip.comwww.synerzip.com • Use cases Agenda • What does it mean? • Implementation Frameworks • Demo • Questions? 22 • Architecture explained
- 23. www.synerzip.comwww.synerzip.com Demo Confidential 23 JDBC Client JDBC API RDBMS Resource Adapter MongoDB Resource Adapter TEIID RUNTIME & QUERY ENGINE Federated VDB mySQL Translator MongoDB Translator mySQL
- 24. www.synerzip.comwww.synerzip.com Demo-Steps • Pre-requisites – mySQL server 5.5+ installed – MongoDB 2.4.x+ installed • Steps – Load the mySql and MongoDB database with sample data – Setup environment – JBoss, Eclipse – Create Teiid project in Eclipse using Teiid designer • Import source model using JDBC • Create the virtual model and federate data from the source model • Create a virtual database (VDB) and deploy to JBoss – Access data using JDBC client or through browser using OData Confidential 24
- 25. www.synerzip.comwww.synerzip.com Demo – Scenario Confidential 25 Federated Data
- 26. www.synerzip.comwww.synerzip.com Demo – Connection Profile Confidential 26
- 27. www.synerzip.comwww.synerzip.com Demo – Source Model Confidential 27
- 28. www.synerzip.comwww.synerzip.com Demo - Source Model Generation Confidential 28
- 29. www.synerzip.comwww.synerzip.com Demo – Map Source To View Confidential 29
- 31. www.synerzip.comwww.synerzip.com Demo – Data Federation Confidential 31
- 32. www.synerzip.comwww.synerzip.com Demo – Source Code • Source code –https://github.com/Synerzip/JBoss- Teiid –Contains • Configuration files • Instructions • “How-to” videos • VDBs, source models and view models Confidential 32
- 33. www.synerzip.comwww.synerzip.com Conclusion • Data Virtualization and Federation is a rapidly emerging technology that solves traditional BI/ETL problems. • It provides lower time to market, distributes data across the enterprise as a service and provides real time access to enterprise data. Confidential 33
- 34. www.synerzip.comwww.synerzip.com • Use cases Agenda • What does it mean? • Implementation Frameworks • Demo • Questions? 34 • Architecture explained
- 37. www.synerzip.comwww.synerzip.com Synerzip in a Nutshell 1. Software product development partner for small/mid-sized technology companies • Exclusive focus on small/mid-sized technology companies, typically venture-backed companies in growth phase • By definition, all Synerzip work is the IP of its respective clients • Deep experience in full SDLC – design, dev, QA/testing, deployment 2. Dedicated team of high caliber software professionals for each client • Seamlessly extends client’s local team, offering full transparency • Stable teams with very low turn-over • NOT just “staff augmentation”, but provide full mgmt support 3. Actually reduces risk of development/delivery • Experienced team - uses appropriate level of engineering discipline • Practices Agile development – responsive, yet disciplined 4. Reduces cost – dual-shore team, 50% cost advantage 5. Offers long term flexibility – allows (facilitates) taking offshore team captive – aka “BOT” option
- 39. www.synerzip.comwww.synerzip.com Call Us for a Free Consultation! Hemant Elhence hemant@synerzip.com 469.322.0349 Thanks!
Editor's Notes
- #8: Require more than 1 copy of data for staging Creating, storing and manipulating this intermediate data can lead to errors in data quality Lead time required to add data from new sources Depends on domain knowledge of mapping entities between different data sources Batch processing – information lagging behind real time data
- #9: Alternate approach is to move all enterprise data to a common Enterprise Information System (typically RDBMS) Extensive changes to existing applications resulting in end user impact Might not satisfy every group’s requirements – say group 1 has partitioned data but the target RDBMS doesn’t support partitioning
- #11: Single API to access data from heterogeneous sources Only metadata stored at virtualization layer Real time access of data without copying/moving data from the source Enterprise Information System (EIS) Federate data across multiple heterogeneous/homogenous sources An enterprise information system (EIS) is any kind of information system which improves the functions of an enterprise business processes by integration. An EIS could use a database/web service/flat files or any other custom system for storing this information.
- #17: Jboss Teiid Open Source Supports number of relational and non relational data sources Integrated with the JBoss Application Server and JEE architecture Ability to add custom data sources using standard JEE components Very active and responsive community
- #20: Amazon SimpleDB - web service for running queries on structured data in real time Apache Accumulo - sorted, distributed key value store Apache SOLR - search system for indexing data/services Cassandra - NoSQL database File - exposes stored procedures to leverage file system resources JPA - reverse a JPA object model into a relational model LDAP - exposes an LDAP directory tree relationally MongoDB - NoSQL database Object - reading java objects from external sources (i.e., Infinispan Cache or Map cache) OData - Consume OData web services and also act as web server to expose VDB as an OData service OLAP - online analytical processing exposing data as 3-D arrays called cubes SalesForce - CRM product SAP Netweaver Gateway - Web service calls to SAP Web Services - exposes stored procedures for calling web services
- #38: 37
- #39: 38
- #40: 39