Building Modern Streaming Analytics with Confluent on AWS
0 likes229 views
This document discusses building modern streaming analytics architectures with Confluent on AWS. It outlines challenges with traditional analytics approaches, and how a modern data strategy using real-time streaming analytics addresses those challenges. It describes how to build seamless streaming architectures using services like Amazon Kinesis and Confluent Kafka, integrated through AWS and Confluent. Examples of real-time analytics use cases are also provided.
Building Modern Streaming Analytics with Confluent on AWS
- 1. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates. Nae Ittaboon (he/him) Solutions Architect Amazon Web Services (AWS) Building Modern Streaming Analytics with Confluent on AWS #DataInMotionTour
- 2. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates. Agenda 2 ❑ Develop a modern data strategy ❑ Act on events in real time with data streaming ❑ Build seamless streaming with AWS and Confluent ❑ Innovate together to power customer success ❑ Key takeaways
- 3. © 2023, Amazon Web Services, Inc. or its affiliates. Challenges of data analytics at scale 3 Data volume and velocity Multiple analytics needs Variety of sources and data types Difficult to manage systems Complex to scale Inflexible tools Security, compliance Slow performance Increasing and unpredictable cost
- 4. © 2023, Amazon Web Services, Inc. or its affiliates. Catalog Governance Data sources People, apps, and devices Analytics Machine learning Databases Data lakes Modern data strategy
- 5. © 2023, Amazon Web Services, Inc. or its affiliates. Amazon OpenSearch Service Amazon Aurora Amazon EMR Amazon SageMaker Amazon DynamoDB Amazon Redshift Amazon S3 Modern data architecture on AWS Key Pillars Data at any scale Seamless data access and movement Purpose-built for best price performance Built-in ML to solve business challenges Unified governance
- 6. © 2023, Amazon Web Services, Inc. or its affiliates. The benefits of data lakes Data lake Catalog Store all your data in open formats Decouple storage from compute Cost-effectively scale storage to exabytes Process data in place Choice of analytical and ML engines
- 7. © 2023, Amazon Web Services, Inc. or its affiliates. Sharing across data movement with Data Mesh Data producers Data mesh Data consumers Unique modern data architecture Suited to business function Teams that want to share data Unique modern data architecture Suited to business function Team that runs the marketplace Teams that want to use data
- 8. © 2023, Amazon Web Services, Inc. or its affiliates. Source: Perishable insights, Mike Gualtieri, Forrester Real time Seconds Minutes Hours Days Months Value of data to decision-making Preventive/predictive Actionable Reactive Historical Time-critical decisions Traditional “batch” business intelligence Information half-life in decision-making Why real-time data streaming analytics ? Data loses value quickly over time
- 9. © 2023, Amazon Web Services, Inc. or its affiliates. Common real-time analytics use cases Anomaly and fraud detection Empowering IoT analytics Nourishing marketing campaigns Real-time personalization Tailoring customer experience in real time Supporting healthcare and emergency services
- 10. © 2023, Amazon Web Services, Inc. or its affiliates. Devices and/or applications that produce real-time data at high velocity Data from tens of thousands of data sources can be written to a single stream Data are stored in the order they were received for a set duration of time and can be replayed indefinitely during that time Records are read in the order they are produced, enabling real- time analytics or streaming ETL Database (NoSQL most common) Data lake Data warehouse Event driven Applications ` Analytics dashboard Real-time streaming analytics pipeline Ingest, Process & Analyze High Volumes of High-Velocity Data from Various Sources in Real Time
- 11. © 2023, Amazon Web Services, Inc. or its affiliates. Mobile device Metering Click streams IoT sensors AWS SDKs Kinesis Agent/KPL ` Apache Kafka Amazon Kinesis Data Streams Amazon Kinesis Data Firehose Apache Kafka Amazon Kinesis Data Streams Amazon Kinesis Data Analytics AWS Glue Streaming Amazon EMR Amazon DynamoDB Amazon EMR Amazon S3 Amazon Redshift Amazon OpenSearch Amazon QuickSight Apache Kafka Streaming analytics on AWS
- 12. © 2023, Amazon Web Services, Inc. or its affiliates. Rich front-end customer experiences Real-time Event Streams and Analysis A Sale A shipment A Trade A Customer Experience Real-time backend operations Event streaming with Kafka to set Data in Motion: Continuously processing evolving streams of data in real time with Confluent
- 13. © 2023, Amazon Web Services, Inc. or its affiliates. Out-of-box integration with popular services Certified and validated by AWS AWS Native Services Top-5 Global ISV for S3 Data Volume 3rd-Party ISV Services • Amazon RDS Ready • AWS Lambda Ready • Amazon Redshift Ready • AWS PrivateLink Ready • AWS Outposts Ready Validated Service Designations Confluent integrations with AWS
- 14. © 2023, Amazon Web Services, Inc. or its affiliates. S Confluent Cloud on AWS – reference architecture S3
- 15. © 2023, Amazon Web Services, Inc. or its affiliates. Confluent + AWS: Accelerate Customer Business Outcomes Topline Impacting New Experiences ● Event-driven & real-time ● Unify data across org. w/ Kafka data fabric (Schema Reg,..) ● AWS Analytics, Redshift, ML connectors Mitigate Risk ● Higher Service Quality & Resilience with 99.99% SLA ● Deep Kafka expertise & innovation ● Elastic billing/pricing Developer Agility ● Focus on innovation (not data infrastructure) ● Leverage full Kafka OSS ecosystem + AWS services Faster Time to Market ● ~50-75% faster time to market* ● Streamline hybrid cloud migration with no complex lift-n- shift ● Maintain business continuity Lower Kafka TCO ● ~25-50% lower TCO * ● GBps-scale & fast deployments for global expansion ● Deploy Kafka at scale in 1 week Maximize ROI ● ~200% ROI per Forrester study ● Save 10s of $Ms with legacy offload to AWS with Confluent Replicator * For customers that don’t already have Kafka based system in-market * TCO assessment to be analyzed for specific customer scenarios
- 16. © 2023, Amazon Web Services, Inc. or its affiliates. Accelerate modernization from on-prem to AWS Redshift Sink Lambda Sink AWS Direct Connect Replicator Legacy Applications Mainframe Legacy Data Systems JDBC / CDC connectors Connect Leverage 120+ Confluent pre-built connectors Modernize Value added apps, increase agility, reduce TCO On-prem AWS Cloud Bridge Hybrid cloud streaming Amazon Athena AWS Glue Amazon SageMaker AWS Lake Formation Amazon DynamoDB Amazon Amazon Aurora S3 Sink Data Streams Apps ksqlDB
- 17. © 2023, Amazon Web Services, Inc. or its affiliates. Increase developer agility & speed of innovation Serverless integration Connect for effortless integrations with Lambda & data stores AWS serverless platform Free up backend operation & Infras. management Apps Microservices ksqlDB Schema Registry COMPUTE AWS Lambda Data stores REST Proxy & Clients Source Connectors Lambda Sink DATA STORES Amazon DynamoDB Amazon Aurora STORAGE Amazon S3 S3 Sink ANALYTICS Amazon Athena Amazon Redshift
- 18. © 2023, Amazon Web Services, Inc. or its affiliates. Amazon Redshift Warehousing with Confluent Cloud Serverless with AWS and Confluent Cloud Real-time Sentiment Analysis with Confluent Amazon ElastiCache and Confluent Cloud confluent.awsworkshop.io Try it out yourself
- 19. © 2023, Amazon Web Services, Inc. or its affiliates. Learn more Working with streaming data on AWS https://aws.amazon.com/streaming-data/ Modern Data Architecture on AWS https://go.aws/3OJDhFk Build Modern Data Streaming Analytics Architectures on AWS https://go.aws/3bt0HAm Derive Insights from Modern Data https://go.aws/3xVU3dn
- 20. © 2023, Amazon Web Services, Inc. or its affiliates. Free trial available through the AWS Marketplace $400 in free credits to spend during your first 30 days
- 21. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates. Thank you! Nae Ittaboon ittaboon@amazon.co.th