DM Radio Webinar: Adopting a Streaming-Enabled Architecture
2 likes293 views
The document discusses the rise of streaming-enabled architecture, emphasizing its importance in achieving real-time data processing and business agility. It highlights the significant advancements in data engineering driven by major tech companies through open-source innovations, leading to the adoption of stream-first architectures. Additionally, it covers technical challenges, solutions such as Apache Kafka, and showcases successful use cases from various industries, underlining the need for automated, efficient data delivery systems.
DM Radio Webinar: Adopting a Streaming-Enabled Architecture
- 1. DEEP DIVE Building a Streaming-Enabled Architecture www.dmradio.biz
- 3. Hardware (network, storage, servers) Data Sources Data Staging Data Volumes Data Flow Data Governance Data Usage Data Structures Schema Definition Ingest Speeds Data Workloads Everything Is In Flux
- 4. The Impact of Parallelism We used to see 10x performance improvement every 6 years, now we see 1000x (and that’s just an approximation) regularly
- 5. You Can’t Build Skyscrapers with Bricks & Mortar
- 6. A Renaissance in Data Engineering Is Underway - Web giants innovated to solve their own challenges - Facebook, Google, LinkedIn, Yahoo! and others… - By open-sourcing software, these companies changed how the industry operates, how tech is built - The result is a new world of scale-out software - Innovations span the spectrum of functionality: database, analytics, networking, data flow, security - Paramount among these in terms of significance is the world of streaming data and supporting tools
- 8. Competition Demands Digital Transformation Adopted by the EU, but affects the USA Behemoths are straws in the wind - They identified huge opportunities - Upended entire industries - Built bulletproof infrastructure - Deconstructed business processes - Re-architected processes at scale - Instilled a data-driven approach
- 10. Stream First Architecture Technical Concepts
- 11. Streaming as Primary Method • Must do it: at or near real-time, partial updates with low cycle time • Events: financial fraud, stock trading, high frequency sensor to controller (autonomous vehicles) • Many different items co-mingled (Internet of Things) • Older examples: Internet packets; digital sensors; computer high density disks • Boosts business: faster awareness leads to faster response leads to improved business (consumer activity monitoring) • Add flexibility and lower Total Cost Ownership (if x and if y and if z) • Avoid committee-itis • Faster process-analyze-change cycles • Allow personnel to address more topics in same time period
- 12. Processing Logic Processing Logic Processing Logic Processing Logic Processing Logic Processing Logic End User End User End UserAccumulator Streams parallel processing multiple inputs. Streaming means processing occurs as data flows through. Straight through or accumulate • Timing • Iterative • Correlation • Referencing • Coding • Real-time • Near real- time • Non-time
- 13. Technical Issues: High Level • Latency: built in delay time caused by many factors. Are you able and willing to invest in finding and removing these items? Can be significant cost to do so. Inherent latency in your use case. • Order misalignment: data arrives or produced out of order • Errors: detect and correct (don’t underestimate this). Embedded data quality problems. Processing logic flaws. How to surgically update. • Power usage: Higher power use per compute period (mobile) • Storage space: overhead for parallel and streaming look ahead, look back requires multiple copies. Content mgmt. procedures become very important.
- 15. Technical Issues: Low Level • Requires deep expertise (hard to find usually) • Memory: must contain all data needed per calculation including lookup codes, cross-references, accumulation arrays • Memory mgmt. is usually weak in high level languages like Java, C# • Cross-stream data exchange: how to truly separate data for full independence (age old parallel computing challenge) or how to store and forward across streams (what is needed, how much, how long) • Dependencies: state mgmt., server health • Latency: network slow down, security hand shakes (e.g. TLS), database freezes, file contention, cluster IO
- 16. Buyer Advice: Look for the Special Things • In memory: of course but especially for engineering of known state- of-art problems like heap space, garbage collection, swapping • Integration: both done for you (framework) and well documented interface APIs using standard computer languages • Error reporting: this is critical. Your personnel cost will go up significantly without this because you will (really!!) experience many crashes. Need meaningful error messages pointing to actual problem. • Demonstrates knowledge and work in next generation: expanded memory spaces; integrated memory-compute. • Shows end-to-end more complicated use cases than yours. (Since yours will become more complicated than you know quickly).
- 18. September 2018 Kevin Petrie Sr Director, Product Marketing ADOPTING A STREAMING- ENABLED ARCHITECTURE ATTUNITY
- 19. 2© 2018 Attunity 2© 2017 Attunity CLOUDSTREAMINGDATA LAKES ON PREMDATA WAREHOUSE BATCH MODERN DATA ARCHITECTURES
- 20. 3© 2018 Attunity 3© 2017 Attunity #1 provider of change data capture (CDC) Support most sources with best performance and least impact #1 cloud database migration technology Already moved over 80,000 databases to Cloud platforms like AWS #1 in ease of use Designed to accelerate deployments by data architects and DBAs instead of developers The leading platform for delivering data efficiently and in real-time to data lake, streaming and cloud architectures ATTUNITY
- 21. 4© 2018 Attunity 4© 2017 Attunity Automated, universal and real-time data delivery Accelerate creation of analytics- ready data structures ATTUNITY REPLICATE ATTUNITY COMPOSE ATTUNITY ENTERPRISE MANAGER Intelligent management, metadata & control THE ATTUNITY PLATFORM
- 22. 5© 2018 Attunity 5© 2017 Attunity Pre-packaged automation of complex tasks Modern user experience Zero source footprint Change data capture (CDC) Stream live updates Optimized for high- performance movement All major platforms DB | DW | Hadoop | Legacy On Premises | Cloud SAP | Mainframe Simplified Real-Time Universal DATA DELIVERY WITH ATTUNITY REPLICATE
- 23. 6© 2018 Attunity Configure, execute and monitor data flows Multiple data centers On premises and cloud Scale to 1,000s of tasks OPERATIONS ANALYTICS CONTROL Historical and real-time reporting Capacity planning Performance monitoring Visually optimize operations Create and discover operational metadata Microservices integration via .NET and REST APIs ATTUNITY ENTERPRISE MANAGER
- 24. 7© 2018 Attunity 7© 2017 Attunity WHAT IT IS ESSENTIAL CHARACTERISTICS OF MODERN DATA STREAMING Producers and consumers are independent Communication is asynchronous Records are persisted for future use High throughput – e.g., records/second Records are sent in re-playable, ordered sequence Geo-distributed replication Fault tolerance
- 25. 8© 2018 Attunity 8© 2017 Attunity WHAT IT IS APACHE KAFKA Open source distributed streaming platform for moving, storing and processing high volumes of data in real time Developed by Jay Kreps and colleagues to process many continuous data flows at LinkedIn More scalable, more fault-tolerant and higher-performance than traditional message-oriented middleware Used for building real-time streaming data pipelines and streaming applications
- 26. 9© 2018 Attunity 9© 2017 Attunity WHAT IT IS KAFKA VS. ENTERPRISE SERVICE BUS PREDECESSORS ActiveMQ, RabbitMQ and IBM MQSeries Centralized, highly scalable cluster to serve all applications across large enterprise environments Persistent storage system for configurable time periods, including forever Abstracted stream processing to easily create derived streams and datasets with minimal coding
- 27. 10© 2018 Attunity 10© 2017 Attunity WHY DATA STREAMING MATTERS “LIFE DOESN’T HAPPEN IN BATCHES” Data streams enable businesses to react to events as they happen Streaming data improves efficiency and scalability Can be used for multiple purposes, by multiple users
- 28. 11© 2018 Attunity 11© 2017 Attunity HOW IT WORKS HIGH LEVEL ARCHITECTURE Producers send records to brokers to be read by consumers Broker persists records to file system on disk for subsequent usage Records are grouped into topics for selected consumer use Topics can be partitioned to improve throughput and redundancy CONSUMER KAFKA BROKER CONSUMER CONSUMER PRODUCER PRODUCER PRODUCER DISK
- 29. 12© 2018 Attunity 12© 2017 Attunity HOW IT WORKS KEY COMPONENTS (a.k.a. message): unit of data, similar to DB row or record (a.k.a. publisher/writer): process that creates and publishes records (a.k.a. subscriber/reader): process that reads records Kafka instance that receives records from producers, persists, provides them to consumers Group of two or more brokers that provide redundancy and scalability (a.k.a. stream): category of record to which a consumer subscribes Subset of topic created to enable redundancy, and parallel reading/writing for higher performance RECORD PRODUCER CONSUMER BROKER CLUSTER TOPIC PARTITION
- 30. 13© 2018 Attunity 13© 2017 Attunity HOW IT WORKS DETAILED ARCHITECTURE Key TOPIC(S) Data 0111 1110 PRODUCER(S) Creates records with serialized key and value Identifies location of write partition for topic RECORD BROKER 0111 1110 0111 1110 Partition Leader RECORD 0111 1110 CONSUMER(S) Deserializes record to create original key and value Key Data 0111 1110 BROKER 0111 1110 Partition Replica TOPIC(S) *Adapted from Enabling Streaming Architectures for Continuous Data and Events with Kafka; Gartner; TJ Craig, Gary Oliffe, Soyeb Barot; 23 May 2018
- 31. 14© 2018 Attunity 14© 2017 Attunity HOW IT WORKS DETAILED ARCHITECTURE Key TOPIC(S) Data 0111 1110 PRODUCER(S) Creates records with serialized key and value Identifies location of write partition for topic RECORD BROKER 0111 1110 0111 1110 Partition Leader RECORD 0111 1110 CONSUMER(S) Deserializes record to create original key and value Key Data 0111 1110 TOPIC(S) CHANGE DATA CAPTURE Eliminates manual scripting to configure record creation from source database transactions *Adapted from Enabling Streaming Architectures for Continuous Data and Events with Kafka; Gartner; TJ Craig, Gary Oliffe, Soyeb Barot; 23 May 2018 BROKER 0111 1110 Partition Replica
- 32. 15© 2018 Attunity 15© 2017 Attunity Generate real-time events Multi topic, multi partition One-to-many event publication Schema evolution; easy schema registry integration DATABASE AS A STREAM UNIVERSAL STREAMING METADATA Amazon KinesisAzure Event Hub ATTUNITY AND DATA STREAMING
- 33. 16© 2018 Attunity 16© 2017 Attunity USE CASES STREAMING INGESTION MESSAGE/EVENT BROKER PREPROCESSING FOR MACHINE LEARNING EVENT STREAM PROCESSING DATA PERSISTENCE REAL-TIME ANALYTIC PROCESSING MICROSERVICES
- 34. 17© 2018 Attunity 17© 2017 Attunity KAFKA STREAMING INGESTION AND MESSAGE BROKER FORTUNE 100 FOOD PROCESSOR CDC to HDP data lake Attunity Replicate feeds HDFS, HBase for timely reporting and product delivery Needed real-time view of production capacity and customer orders Nightly batch loads couldn’t keep up => Fulfilment delays, inaccurate reports PROBLEM SOLUTION ATTUNITY REPLICATE Log based CDC SAP ECC 10 tables (purchase orders, production plans) HDP DATA LAKE
- 35. 18© 2018 Attunity 18© 2017 Attunity Copies live transactions without touching production Securely transfers them for client usage on global AWS microservices platform Need to efficiently roll out extensive cloud-based microservices platform Must minimize latency and security risk while synchronizing massive transactional updates globally PROBLEM SOLUTION $1 Trillion ATTUNITY REPLICATEDB2 z/OS (on prem) KINESIS DYNAMO DB MICROSERVICE HUB ON RDS DYNAMO DB STREAMS RDS EMEA CUSTOMER APJ CUSTOMER AWS CLOUD CDC MICROSERVICES LEADING ASSET MANAGEMENT FIRM Assetsunder Management
- 36. 19© 2018 Attunity 19© 2017 Attunity EVENT STREAM PROCESSING FORTUNE 100 PHARMACEUTICAL FIRM CDC to Kafka to Lambda Architecture Multi-pronged analysis of clinical data at scale Minimal administrative burden; no PROD impact Needed efficient, scalable delivery of clinical data for analytics Lacked tools for low-impact data capture PROBLEM SOLUTION ATTUNITY REPLICATE Log based CDC Lambda Architecture Clinical Systems Structured Analysis Clinical Systems Batch Historical Data Real Time Updates Stream Processing Graph Analysis Natural Language Processing Machine Learning KAFKA
- 37. 20© 2018 Attunity 20© 2017 Attunity Improved ease of use Consistent, 100% automated processes across end points Reduced impact on MF production Improved performance to Kafka Better TCO Needed 360 degree customer view for CSAT initiatives Inefficient OGG solution Lack of data consistency and standardization PROBLEM ATTUNITY SOLUTION Fortune 100 company $100 BILLION STREAMING INGESTION FORTUNE 100 HEALTH BENEFITS FIRM DB2 Z/OS SQL SERVER ORACLE KAFKA ATTUNITY REPLICATE Log based CDC HDP DATA LAKE
- 38. 21© 2018 Attunity MACHINE LEARNING PRE-PROCESSING LEADING PAYMENT PROCESSOR BANK/ MERCHANT GATEWAY APPLICATION AND DATABASE OPENSCORING.IO DECISION SERVICE ENGINE Credit Check, Authentication Decision < 100 MS CHANGE DATA CAPTURE A B Transactions Decisions logged real time MACHINE LEARNING Data analyzed over days Decision models published with new insights Data delivered in seconds Data delivered in minutes Decision performance monitored Real time decisions Decision measurement and tuning C D E F A
- 39. 22© 2018 Attunity Overall Rating Product Capabilities Ease of Deployment 4.5 out of 5 4.4 4.2 4.1 out of 5 4.2 3.7 4.2 out of 5 4.3 4.0 4.1 out of 5 4.3 4.0 VENDOR OF CHOICE Replicate has been working great for several years, implementation was a breeze - DBA, Retail Industry Great vendor to work with and an incredibly easy tool to use - Senior Member of Technical Staff, Communications Industry
- 40. 23© 2018 Attunity 23© 2017 Attunity Trusted by Microsoft with 3 OEMs, bundled inside SQL Server Trusted by Amazon (AWS) with strategic partnership for cloud database migration Trusted by IBM and Oracle with respective OEMs of Attunity technology Trusted by Teradata and HP as resellers for data warehouse and analytics Trusted by global system integrators Trusted by over 2000 customers for commitment, flexibility and speed 2000+ Trusted by SAP as certified solution in use with over 200 SAP customers Trusted by big data leaders for data lake solutions Trusted by IBM and Oracle with respective OEMs of Attunity technology Trusted by Teradata and HP as resellers for data warehouse and analytics PARTNER OF CHOICE
- 41. 24© 2018 Attunity FINANCIAL SERVICES MANUFACTURING/ INDUSTRIAL HEALTH CARE GOVERNMENT TECHNOLOGY / TELECOM RETAIL OTHER INDUSTRIES 2000 CUSTOMERS AND HALF THE FORTUNE 100