Keys to a Streaming-First Architecture
1 like163 views
The document discusses the growing importance of a streaming-first architecture for organizations, highlighting the benefits of real-time data insights, breaking down data silos, and improving customer experiences. It outlines challenges that companies face with streaming data adoption and details how Syncsort's Connect CDC can facilitate building streaming data pipelines across various platforms. The document emphasizes the significance of guaranteed data delivery, scalability, and maintaining metadata integrity in the streaming process.
Keys to a Streaming-First Architecture
- 1. Keys to a Streaming-First Architecture Ashwin Ramachandran
- 2. Housekeeping Webcast Audio • Today’s webcast audio is streamed through your computer speakers. • If you need technical assistance with the web interface or audio, please reach out to us using the chat window. Questions Welcome • Submit your questions at any time during the presentation using the chat window. • We will answer them during our Q&A session following the presentation. Recording and slides • This webcast is being recorded. You will receive an email following the webcast with a link to download both the recording and the slides. Keys to a Streaming-First Architecture - © Syncsort 2019
- 3. • Common applications of streaming data • Why streaming data? • Keys to success • Connect CDC Agenda Keys to a Streaming-First Architecture - © Syncsort 2019
- 4. “ ” Importance of streaming has grown enormously in recent years because it provides a competitive advantage that reduces the time gap between data arrival and its analysis. Some insights are more valuable immediately after an event occurs, but the value diminishes very rapidly with time. Gartner, Stream Processing: The New Data Processing Paradigm
- 5. Keys to a Streaming-First Architecture - © Syncsort 2019 Despite the rise in importance, challenges exist 47% 12% 9% Need to build streaming pipelines to support digital business Currently incorporate streaming into their analytics and data requirements Very effective in getting value from the data generated or collected across the enterprise Sources: Stream Processing: The New Data Processing Paradigm, Gartner 2019 Data Trends for 2019: Extracting Value from Data, Syncsort 2019
- 6. • Real-time insights: streaming data reveals up-to-the-moment markers of the state of the business, allowing organizations to be more agile and maintain competitive advantage. • Break down data silos: 68% of organization struggle with disparate, siloed datasets that negatively impact operations. Streaming data can ensure that these datasets always remain connected. • Evolve business applications: leverage streaming data to feed applications that require real-time responses, such as machine learning-driven fraud detection or AML applications. • Improve customer experiences: create a customized experience for customers based on common behaviors and correlations that streaming data can reveal. Keys to a Streaming-First Architecture - © Syncsort 2019 Common applications of streaming data
- 7. Identify the right data Keys to a Streaming-First Architecture - © Syncsort 2019 • Port workloads to less-expensive, strategic platforms • Connect applications together, leveraging the existing transactional capabilities of the current application platform, and the wealth of new capabilities of the cloud • Feed analytics with up-to-date information so your business runs on current insight Legacy data can provide a treasure-trove of information that can transform your business when leveraged via a streaming paradigm
- 8. Keys to a Streaming-First Architecture - © Syncsort 2019 Leverage the right technology • Guaranteed data delivery and fault tolerance • Scalability and performance • As your business grows, so will the demands on your streaming pipeline. • Lightweight footprint and impact • When replicating from the systems that run your business, ensure the solution won’t negatively impact your running applications • Maintain your metadata • Integrate with technologies like Kafka and the schema registry to maintain metadata lineage Keep in mind the following must-haves:
- 10. Keys to a Streaming-First Architecture - © Syncsort 2019 Connect CDC enables organizations to build streaming data pipelines and easily share application data across the enterprise – from mainframes to the cloud. Connect CDC powers insights with the most up-to-date information and helps to keep the business in sync.
- 11. Top use cases for Connect CDC Stream real-time application data from traditional systems to mission critical business applications and analytics platforms that demand the most up-to-date information for accurate insights • Traditional systems examples: mainframes, EDWs, IBM i • Business application examples: fraud detection, hotel reservations, mobile banking, etc. Support Cloud and hybrid architectures by sharing application data to, from and among cloud platforms • A variety of application topologies that can be combined, including: one-way, two-way, bidirectional, consolidated, distributed, and cascade Keys to a Streaming-First Architecture - © Syncsort 2019
- 12. Self-service data integration through browser based interface • Design, deploy and monitor real-time change replication from a variety of traditional systems (mainframe, RDBMS, EDW) to next- generation distributed streaming platforms like Apache Kafka Guaranteed data delivery • Fault tolerant • Protects against loss of data if a connection is temporarily lost • Keeps track of exactly where data transfer left off and automatically restarts at that exact point – no manual intervention • No missing or duplicate data! Maintain metadata integrity • Integrates with Kafka schema registry Connect CDC Keys to a Streaming-First Architecture - © Syncsort 2019
- 13. Connect Portal The Connect Portal serves as the next-generation UI for Connect CDC, Connect ETL and Connect for Big Data • First Release includes: • High-performance database loading (former DataFunnel functionality) • Connect CDC – ability to design, deploy and monitor the real-time replication of changes from Oracle to Kafka • Future releases will include the following Connect CDC endpoints: • Sources — SQL Server — IBM i — Db2 LUW • Targets — Amazon Kinesis The new Syncsort Connect UI Keys to a Streaming-First Architecture - © Syncsort 2019
- 14. Keys to a Streaming-First Architecture - © Syncsort 2019 Streaming data pipelines with Syncsort Connect Custom Real-Time Applications Sources Destinations Syncsort Connect CLIENT OBJECTIVES • Connecting business-critical legacy applications with downstream applications and analytics platforms PRIMARY • Keeping the data lake fresh, building a real-time data pipeline OTHERS Syncsort Connect Mainframe IBM i RDBMS
- 15. Questions?