Seamless, Real-Time Data Integration with Connect
Download as PPTX, PDF0 likes202 views
The document outlines a comprehensive data integration solution designed to seamlessly connect legacy systems, particularly mainframe and IBM I data, with modern cloud and big data platforms. It emphasizes the software's capabilities in real-time data replication, transformation, and the ability to operate without extensive mainframe expertise, enabling businesses to confidently make data-driven decisions. Key features include high performance, flexibility in data formats, and a design that allows for easy deployment across various environments without the need for extensive coding or reworking.
Seamless, Real-Time Data Integration with Connect
- 1. Seamless, real-time data integration Precisely Connect Overview Liz Thompson | Sales Engineer
- 2. The global leader in data integrity Trust your data. Build your possibilities. Our data integrity software and data enrichment products deliver accuracy and consistency to power confident business decisions. Brands you trust, trust us Data leaders partner with us of the Fortune 100 90 Customers in more than 100 2,000 employees customers 12,000 countries
- 3. Modern Data Integration + Legacy Data = Easier Said Than Done!
- 4. Integrate data seamlessly from legacy systems into next-gen cloud and data platforms with one solution. Batch data transformation Data replication via CDC High-performance, lightweight batch data processing for distributed platforms Real-time data replication to streaming platforms, cloud, databases and data warehouses Connect
- 5. Connect is the best solution for accessing and integrating mainframe and IBM i data with cloud data platforms Quickly and efficiently integrate ALL enterprise data – including mainframe and IBM i Design-once, deploy anywhere approach to building data pipelines Minimal development costs and time Unrivaled scalability and performance
- 6. Connect speaks mainframe Data Translation • Connect can translate mainframe on the fly • No mainframe expertise required • No need to stage extra copies Complex Copybooks • Connect leverages copybooks as-is - no matter how complex • Multiple redefines • Directives, SKIP, EJECT, etc. • Compiler tags • ODO’s, even nested • No padding/exploding the data • Smaller files = faster execution Variable Length Records • Record descriptor words make it hard to split variable length data • Connect handles distributing variable length data with ease. • Retains the integrity of data • If you choose, data can remain unchanged on object storage, HDFS, etc. while still matching the original copybook.
- 7. Or, get your mainframe data into big data, in a big data format Want your data in a cloud-native format? Connect can translate on the fly! • Big Data lacks robust connectivity to mainframe. • With Connect, you can: • Securely access mainframe data with FTPS, Connect:Direct. • Transform data on the fly – no staging. • Import hundreds or Db2 tables to your cluster with a few mouse clicks. Access • Mainframe data needs to be joined with other data sources for full benefit. • With Connect, you can: • Integrate mainframe, relational data, files, streaming data, … • Handle all kinds of data formats, old and new: RDBMS, mainframe variable, HDFS, DBFS, Hive, Delta Lake, text, Avro, Kafka, Parquet, S3, … Integrate • SLA’s are shorter and data is growing. • With Connect, you can: • Access data in parallel. • Integrate even streaming data with Spark support. • Keep Big Data in sync with mainframe and relational database changes in real-time with Connect’s CDC capabilities. Fast
- 8. Connect ETL
- 9. Mainframe IBM i Relational databases, EDW, DBMS Flat files, XML, JSON Sources Strategic Projects: Real-time analytics, AI and machine learning Targets Deploy Connect ETL on a cluster, Windows, Unix, Linux, or cloud Integrate, Prepare, Load, Cleanse, Transform, Deliver Amazon S3 On-prem legacy stores Cloud platforms Big data Connect ETL Precisely Connect
- 10. Offload data and processing • Offload data and processing from mainframe to Cloudera, Databricks, Snowflake, and more • Direct, high-performance connectivity to the mainframe • Ingest from a variety of file formats and data types • Examples: VSAM, fixed length, variable length, OCCURS, OCCURS DEPENDING ON, REDEFINES, packed decimals, COMP data types • Translations from EBCDIC to a variety of encodings • Examples: ASCII, UTF-8, ICU codepage encodings • Convert compressed number formats to displayable number formats • Archive in raw format to target and process without the need for conversion
- 11. Connect to any source • Native drivers to extract data for guaranteed optimal speed and efficiency • Direct mainframe connectivity • Support for distributed sources, external sources, flat files, weblogs, etc. • Join, transform, de-normalize, before landing to the target • Unique approach to convert the data to native Big Data formats, such as Avro, Parquet, and ORC • Quickly create reports showing off the insights from their data • Connections for Tableau and Qlik • Read and write to/from Apache Kafka
- 12. Design once, deploy anywhere No re-design, re-compile or re-work – ever. • Design future-proof workflows for batch and streaming data • Cleanse, blend and transform data for context and meaning • Move from dev to test to production • Move from on-premises to Cloud • Move from one Cloud to another Drive results through high performance • Get excellent performance every time without tuning, load balancing, etc. Use existing skills to drive new projects • No parallel programming – Java, MapReduce, Spark … No worries about: • Mappers, Reducers • Big side or small side of joins … Design Once in visual UX Deploy Anywhere! On- premises , Cloud Spark, MapReduce, Future Platforms Windows, Unix, Linux Batch, Streaming Single Node, Cluster
- 13. Connect comes equipped with Intelligent Execution • No administration or management, Connect’s ETL engine dynamically selects the most efficient algorithms based on the data structures and systems attributes at run-time • Highly performant with small-footprint and comprehensive support for growing modern data warehouses • Visual design that enables jobs to be created once, and deployed them anywhere — Spark, Hadoop, Linux, Unix, Windows — Public, private, hybrid or multi-cloud • Simplicity and ease of use - move applications from standalone server environments and from MapReduce to Spark – as easy as clicking on a dropdown menu • Future-proof your organization from underlying complexities of your technology stack
- 14. Connect ETL Advantages • Connect ETL requires no installation on the mainframe • No mainframe MIPS used • Extract from the mainframe, apply translation and transformation in memory and load directly to RDBMS in one step (no source staging needed), Cloud, Big Data platforms • No source staging keeps data more secure • Future proofing • Design once, deploy anywhere • Simple, lightweight architecture streamlines development and deployment • No data repository required • High performance light weight, self-tuning engine • No coding, no tuning
- 15. Connect ETL Advantages • We are the experts in mainframe data conversion with excellent support • Seamlessly read or write mainframe data based on COBOL Copybook • Apply multiple copybooks to a single mainframe file in one step • No data loss or data type mismatches - • Support for complex hierarchical data from Mainframe and IBM i • Automatically convert from EBCDIC to ASCII including binary values! • Support for Occurs Depending On (ODO) • Support high/low values, • Support for packed decimal • Support for COMP data
- 16. Connect Data Replication (CDC)
- 17. Transform your business with Connect CDC capabilities • Build streaming data pipelines to power business decision-making with real-time data • Get a consistent view of the data across the enterprise and keep your business in sync • Keep data lakes fresh with changes made on transactional systems – including mainframe • Enable timely reporting and meet tightening SLAs • Migrate data with zero downtime for database/application upgrades and system re-platforming • Resilient data delivery and support data governance and security requirements
- 18. Connect – Data Replication Overview • IBM Db2 for i • IBM Db2 for z • IBM Db2 LUW • IBM Informix • IMS Real-time Data Sync • Without overloading networks. • Without affecting source database performance. • Without coding or tuning. Real-Time Data Replication + Transformation • Apache Kafka • Oracle • Sybase • VSAM • IMS • Snowflake Resilient Delivery – Reliable transfer of data even if connectivity fails on either side. No data loss. Auto restart. High Performance – Captures changes in source as they happen. Updates table statistics for faster queries. Flexible – RDBMS, Cloud, Cluster • Replicate data to RDBMSs, enterprise data warehouses, cloud systems, and Kafka • Oracle • Oracle RAC • Sybase • VSAM • MS SQL Server • IBM Db2 (i, z, LUW) • MS Azure SQL Database • MS SQL Server • Teradata • IBM Informix Conflict Resolution, Collision Monitoring, Tracking and Auditing Example Sources Example Targets
- 19. Connect’s CDC High-level Architecture Source Database PROJECT METADATA CHANGE DATA CAPTURE Target Database PROJECT METADATA Connect SaaS
- 20. Flexible Replication Options One Way Two Way Cascade Bi-Directional Distribute Consolidate Choose a topology or combine them to meet your data sharing needs
- 21. Resilient replication: enable information accuracy Ensures ongoing integrity • Changes collected in queue on source • Moved to target only after committed on source • Ensures write-order-consistency retained • Queues retained until successfully applied • No database table locking Ensures failure integrity • Automatically detects communications errors • Automatically recovers the connection and processes • Alerts administrator • No data is lost Email Alerting
- 22. Accurate tracking and data auditing Detects and resolves conflicts • Maintains data integrity Model verification • Validates data movement model Audit Journal Mapping tracks all updates and changes • Records • Before and after values for every column • Type of transaction • Type of sending DBMS • Table name • User name • Transaction information • Records to flat file or to database table • Can assist with SOX, HIPAA audit requirements
- 23. Replicate exactly what you need Filters determine what data gets moved • Select specific column and table • Select specific rows and table
- 24. Transforms data exactly how you need to • Transforms data into useful information • 80+ built-in transformation methods • Field transformations, such as: • DECIMAL(5,2) • nulltostring(ZIP_CODE,'00000') • Table transformation, such as: • Column merging • Column splitting • Creating derived columns • Custom lookup tables • Create custom data transformations using powerful Java scripting interface
- 25. Connect CDC Advantages • Extract from the mainframe, apply translation and transformation in memory and load directly to RDBMS in one step (no source staging needed), Cloud, Big Data platforms • No staging of source data keeps data more secure • Connect CDC uses minimal mainframe MIPS • No data loss or data type mismatches - • Support for complex hierarchical data from Mainframe and IBM i • Automatically convert from EBCDIC to ASCII including binary values! • Support for Occurs Depending On (ODO) • Support high/low values, • Support for packed decimal • Support for COMP data • Fast real-time capture for IMS, VSAM, and DB2. Our capture uses the active logs, where other products can only look at the IMS archive logs • Automatic recovery from network failures for all sources including MF • Simplicity – no coding, no tuning • We are the experts in mainframe data conversion with excellent support
- 26. Precisely data integration top differentiators 1 Resilient & Fault- Tolerant Replication 2 Scalability & Performance 3 Design Once, Deploy Anywhere 4 Native Tech Partner Integration