Snowflake + Syncsort: Get Value from Your Mainframe Data
2 likes314 views
The document presents a webcast discussing the integration of mainframe data with Snowflake, highlighting the advantages of using cloud data warehouses (CDWs) for analytics and insights while overcoming challenges with legacy data formats. It emphasizes best practices for accessing and transforming mainframe data into usable formats for Snowflake, using Syncsort Connect as a solution. The session also intends to facilitate questions and provide additional resources post-webcast.
Snowflake + Syncsort: Get Value from Your Mainframe Data
- 1. Snowflake + Syncsort: Get Value from Your Mainframe Data Presented by: Ashwin Ramachandran Senior Product Manager
- 2. Housekeeping Webcast Audio • Today’s webcast audio is streamed through your computer speakers. • If you need technical assistance with the web interface or audio, please reach out to us using the chat window. Questions Welcome • Submit your questions at any time during the presentation using the chat window. • We will answer them during our Q&A session following the presentation. Recording and slides • This webcast is being recorded. You will receive an email following the webcast with a link to download both the recording and the slides.
- 3. • Cloud Data Warehouses • Benefits of Snowflake • Best practices for extracting mainframe and IBM i data • Syncsort + Snowflake What we will talk about today Snowflake + Syncsort: Get Value from Your Mainframe Data - © Syncsort 2019
- 4. “ ” Companies are using CDWs to accelerate analytics and insights, lower data warehouse costs, and allow users to focus on business issues rather than deal with technical complexity. While a majority of the data warehouses built in the cloud today support new business requirements, more organizations are starting to move their existing on-premises data warehouse to the cloud. Forrester, The Forrester Wave™: Cloud Data Warehouse, October 2018 Snowflake + Syncsort: Get Value from Your Mainframe Data - © Syncsort 2019
- 5. Snowflake + Syncsort: Get Value from Your Mainframe Data - © Syncsort 2019 Why choose a cloud data warehouse approach? Build scalable data warehouses very quickly Support migrations from on-prem data warehouses to the cloud Enable analytics and insights at a faster rate Lower hardware and software costs
- 6. • Analytics and insights: enable right-time business insights that require quicker, adaptive and self-service analytics • Break down data silos: 68% of organization struggle with disparate, siloed datasets that negatively impact operations. CDWs can serve as a centralized repository, ensuring these datasets always remain connected. • Evolve IoT applications: efficiently store, process, and access large volumes of IoT data from sensors and devices • Improve customer experiences: create a customized experience for customers based on common behaviors and correlations that data can reveal. • Re-hosting of on-prem data warehouses: deploy data warehousing workloads in the cloud in a cost- effective, flexible fashion. Common applications of cloud data warehouses Snowflake + Syncsort: Get Value from Your Mainframe Data - © Syncsort 2019
- 8. • Fully managed data warehouse as a service on AWS and Microsoft Azure • Google Cloud Platform announced in 2019 • Launched in 2012, based in San Mateo, California, U.S. • Approach to cloud data warehousing: • Built for the cloud • Storage and compute were decoupled and each individually scalable • Access through SQL • Syncsort and Snowflake partnership announced May 2019 Snowflake + Syncsort: Get Value from Your Mainframe Data - © Syncsort 2019 What is Snowflake?
- 9. The importance of legacy data Your traditional systems – including mainframes, IBM i servers & data warehouses – adapt and deliver increasing value with each new technology wave 91% of executives predict long-term viability of the mainframe as the platform continues evolving to meet digital business demands >100k companies today use IBM i technology to run significant workloads & power critical business applications $1.65trillion invested by enterprise IT to support data warehouse & analytics workloads over the past decade Snowflake + Syncsort: Get Value from Your Mainframe Data - © Syncsort 2019 BMC 12th Annual Mainframe Research Results – Nov. 2017 Wikibon “10-Year Worldwide Enterprise IT Spending 2008-2017” Don’t forget about the mainframe!
- 10. Snowflake + Syncsort: Get Value from Your Mainframe Data - © Syncsort 2019 Mainframes Host the Most Critical Applications 71%Fortune 500 2.5 BillionTransactions / day / per MF Top World Banks92 of World’s Top Insurers10 of Top 25 US Retailers23
- 11. Snowflake + Syncsort: Get Value from Your Mainframe Data - © Syncsort 2019 Cloud Data Warehouses– Mainframe Data = Missed Opportunity • The value of the organizations big data investments are diminished. • Large, rich enterprise datasets never even get analyzed. • Analytics aren’t accurate or complete without the full picture. When mainframe data is left out of the Modern Data Architecture: Mainframe Files Databases Cloud
- 12. Challenges of integrating legacy data with cloud data warehouses Snowflake + Syncsort: Get Value from Your Mainframe Data - © Syncsort 2019 • Legacy data formats such as mainframe and IBM i are not readily readable and consumable within cloud data warehouses • Binary data formats • REDEFINEs in data fields • Complex record structures • Making data movement between on-premises and the cloud data warehouse speedy, secure and cost effective • Plans may require complex integration architectures, making it harder to connect to the cloud data warehouse
- 13. Communication and documentation. Communication and documentation between teams is critical for ensuring that mainframe data is use is alignment with organizational policies Determine needs first. Mainframe data, like any other data set, once pulled from the source can be interpreted in many ways. It’s important that across teams, everyone knows the end goal of what the data will feed as well as the tools the data will be fed into. Have data mapped out. Having a good understanding with your data architect and business analysts on how data needs to be mapped will help you get a firm hold how mainframe data should be restructured for Snowflake. Leverage your existing metadata. Your mainframe data already has rich metadata associated with it and being used by applications on the mainframe. Don’t reinvent the wheel, and use a tool that can properly harness that metadata. Snowflake + Syncsort: Get Value from Your Mainframe Data - © Syncsort 2019 Best practices for accessing legacy data
- 14. Syncsort Connect offers: • Hierarchical mainframe data set integration with relational Snowflake tables • High performance loads to Snowflake on AWS and Azure • Syncsort Connect requires no installation of software on the mainframe to restructure mainframe data into readable formats for use with Snowflake • Syncsort Connect enables you to leverage existing copybooks in order to integrate with Snowflake metadata — Convert packed decimal, zoned decimals, and other binary formats to readable data types — Handle COBOL high/low values upon ingestion — Quickly and easily convert EBCDIC text to Unicode — Handle REDEFINEs in COBOL copybooks, ensuring the correct data is made available in Snowflake — Ingest and performantly transform OCCURS DEPENDING ON variable length arrays for downstream consumption Snowflake + Syncsort: Get Value from Your Mainframe Data - © Syncsort 2019 Snowflake + Syncsort
- 15. Snowflake + Syncsort: Get Value from Your Mainframe Data - © Syncsort 2019 Snowflake & Syncsort IBM i Traditional ETL sources, files, RDBMS, etc. Convert mainframe, IBM i and data from other sources to be shared anywhere on Snowflake BI and Analytics Tools Deploy Syncsort on-prem, in cloud or hybrid environments Mainframe (DB2, VSAM)
- 16. Questions?