Building Serverless EDA w_ AWS Lambda (1).pptx
- 1. Serverless Stream Processing for Financial Services Ahmed Zamzam AWS Partner Solutions Architect - Confluent Veda Raman Specialist Solutions Architect - AWS Jason Demby Senior Business Development Leader - AWS
- 2. Agenda Introducing Confluent Rearchitected Kafka, together with the features you need to rapidly deploy production use cases 2 Serverless stream processing Building event streaming applications using ksqlDB and AWS Lambda Best Practices Best practices when using AWS Lambda as a stateless stream processor The shift towards data streaming and Apache Kafka and the value it provides The rise of data streaming
- 3. Confluent for Financial Services: Use Cases Deliver differentiated customer experiences Increase digital engagement & improve omni-channel experience Secure your enterprise Detect and respond to fraud, threats and attacks in real-time Modernize your infrastructure Drive massive operational efficiency, developer velocity and reduce costs Automate business resiliency Mitigate risks in service offerings and market risk exposure Drive regulatory compliance Stay compliant across global banking regulations such as open banking, FINRA, trade reporting, payments and more Enable a sharing economy Decentralize asset ownership and increase market opportunity to deliver profitable services in the future
- 4. What is streaming data? Typical characteristics Low-latency Continuous Ordered, incremental High volume
- 5. Why streaming data? Source: Perishable insights, Mike Gualtieri, Forrester Data loses value quickly over time Real-time Seconds Minutes Hours Days Months Value of data to decision-making Preventive/Predictive Actionable Reactive Historical Time critical decisions Traditional “batch” business intelligence Information half-life in decision-making
- 6. Event Streaming is the Central Nervous System for today’s enterprises. Apache Kafka® is the technology.
- 7. ...many more Other Systems Other Systems Kafka Connect Kafka Cluster Kafka Connect Apache Kafka is an Event Streaming Platform
- 8. © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Core Kafka Features 01 Publish & Subscribe to Streams of Events 02 Store your Event Streams 03 Process & Analyze your Events Streams
- 9. Real-time Data Trades Market data Positions Collateral A new paradigm for financial services Continuously Process Data in Real-time “We need to shift our thinking from everything at rest, to everything in motion.” — Real-Time Stream Processing Orders, Execution, Algorithms, Pricing Models Firm-wide Risk Management
- 10. “We look at events as running our business. Business people within our organization want to be able to react to events—and oftentimes it's a combination of events.” VP of Streaming Data Engineering
- 11. Hall of Innovation CTO Innovation Award Winner 2019 Enterprise Technology Innovation AWARDS ...Confluent is the Only Company Focused on Data in Motion Vision ● Original creators of Kafka ● Data in Motion pioneers Category Leadership ● 80% of Kafka commits ● 1M+ hours of Kafka technical experience ● Operate 5K+ clusters Value ● Remove risk ● Deploy at scale ● Accelerate time-to- market Product ● Extends Kafka to be secure and enterprise-ready ● Software or cloud- native service
- 12. ... Device Logs ... ... ... Data Stores Logs 3rd Party Apps Custom Apps / Microservices Real-time Customer 360 Financial Fraud Detection Real-time Risk Analytics Real-time Payments Machine Learning Models ... Real-time Applications Universal Event Pipeline Amazon S3 SaaS apps Confluent: Central Nervous System For Enterprise
- 13. Confluent Enables Endless Financial Services Use Cases Hybrid & Multi-Cloud Messaging & Mainframe Modernization Streaming Analytics Event Driven Microservices CDC Patterns from Systems Of Records Corporate & Investment Banking, Capital Markets Trade Processing (Equities, FICC, Derivatives...) Real Time Payments and Payments Tracking Risk Analytics Market, Reference, & Security Master Data Distribution Trading System Integrations & Automation CTO - Technology Modernization Finance, Risk, Compliance, IT, Cyber Credit & Market Risk (CCAR, BCBS 239, FRTB ) OATS / CAT reporting Operational Log Hub IT Observability Cyber Security | SIEM Modernization Retail Banking, Wealth & Asset Management Fraud Detection Open Banking Customer 360 (omni channel banking, alerts & notifications) Client Advisor Workstations Data and Analytics for Asset Managers
- 14. Everywhere Be everywhere our customers want to be Cloud-Native Re-imagined Kafka experience for the Cloud Complete Enable developers to reliably & securely build next- gen apps faster The Confluent Product Advantage
- 15. Confluent runs everywhere 18 SELF-MANAGED SOFTWARE Confluent Platform The Enterprise Distribution of Apache Kafka In the datacenter VM FULLY-MANAGED SOFTWARE Confluent Cloud Apache Kafka Re-Engineered for the Cloud In the cloud
- 16. Federated streaming, hybrid and multi-cloud. Data syndication and replication across and between clouds and on-premises, with self-service APIs, data governance, and visual tooling. Reliable & real-time data streams between all customer sites, so you can run always-on streaming analytics on the data of the entire enterprise, despite regional or cloud provider outages. Everywhere: Cluster Linking Global Central Nervous System
- 17. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Augment Messaging and Mainframe Systems and Migrate Over Time with our Support 20 1. Current middleware communication 2. Decouple from consumer app 3. Make your events available for downstream systems Customer Payment Jay $10 Sue $15 Grace $5 ... ... Application producing data Traditional Messaging and Mainframe Systems Consumer application 1 2 3 Downstream data store
- 18. Accelerate modernization from on-prem to AWS Redshift Sink Lambda Sink AWS Direct Connect Replicator LEGACY EDW MAINFRAME LEGACY DB JDBC / CDC connectors Connect Leverage +100 Confluent pre-built connectors to continuously bring valuable data from existing services on-prem including enterprise data warehouse, databases and mainframes Modernize Increase agility in getting applications to market and reduce TCO when freeing up resources to focus on value generating activities and not in managing servers On-prem AWS Cloud Bridge Hybrid cloud streaming with consistent, event- driven architecture for modern apps On-prem to AWS modernization Amazon Athena AWS Glue SageMaker Lake Formation Amazon DynamoDB Amazon Aurora S3 Sink Data Streams Apps ksqlDB
- 19. © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Powering real-time analytics for credit scoring, fraud detection, and merchant assessment services Challenge Drive a digital transformation at the largest bank in Indonesia to improve the bank’s market position and increase financial inclusion across the country. Solution Use Confluent to deploy an event-driven microservices architecture that powers big data analytics for real-time credit scoring, fraud detection, and merchant assessment services. Results ● Fraud detection performed in real-time ● Loan disbursement times cut from two weeks to two minutes ● ISO-certified open API created ● Loan defaults predicted proactively; NPL at 0% “Confluent Platform and Apache Kafka, by enabling us to build and deploy real-time event-driven systems for credit scoring, have helped BRI become the most profitable bank in Indonesia.” — Kaspar Situmorang, Executive Vice President
- 20. © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Build Real-time Dashboard And Reports For Compliance, Audit And Monitoring Messaging Systems Mainframe Databases Big Data / Analytics Systems SQL & NoSQL Databases Visualization Tools Regulatory reporting Real-time dashboards In-flight stream processing
- 22. Stateful Stateless Two types of ESP
- 23. Kafka clients Kafka Streams ksqlDB ConsumerRecords<String, String> records = consumer.poll(100); Map<String, Integer> counts = new DefaultMap<String, Integer>(); for (ConsumerRecord<String, Integer> record : records) { String key = record.key(); int c = counts.get(key) c += record.value() counts.put(key, c) } for (Map.Entry<String, Integer> entry : counts.entrySet()) { int stateCount; int attempts; while (attempts++ < MAX_RETRIES) { try { stateCount = stateStore.getValue(entry.getKey()) stateStore.setValue(entry.getKey(), entry.getValue() + stateCount) break; } catch (StateStoreException e) { RetryUtils.backoff(attempts); } } } builder .stream("input-stream", Consumed.with(Serdes.String(), Serdes.String())) .groupBy((key, value) -> value) .count() .toStream() .to("counts", Produced.with(Serdes.String(), Serdes.Long())); SELECT x, count(*) FROM stream GROUP BY x EMIT CHANGES; Flexibility Simplicity 3 modalities of stream processing with Confluent
- 24. ksqlDB at a Glance What is it? ksqlDB is an event streaming database for working with streams and tables of data. All the key features of a modern streaming solution. Aggregations Joins Windowing Event-Time Dual Query Support Exactly-Once Semantics Out-of-Order Handling User-Defined Functions Compute Storage CREATE TABLE activePromotions AS SELECT rideId, qualifyPromotion(distanceToDst) AS promotion FROM locations GROUP BY rideId EMIT CHANGES How does it work? It separates compute from storage, and scales elastically in a fault-tolerant manner. It remains highly available during disruption, even in the face of failure to a quorum of its servers. ksqlDB Kafka
- 25. © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Reduce Data Indexing, Analysis and Storage Costs Application Logs Network Logs Database logs OS Logs Collect all data sources into Confluent Filter events streams and only send priority events to SIEM Shorten SIEM retention window; store in cheaper storage for long-term retention Offload fast query and search Send high priority data to SIEM Send all data to S3/HDFS for cold storage Open up data access to new use cases Preserve data in low-cost storage Tier and route data to cloud storage infrastructure for long term retention with pre-built connectors Save on ingest and indexing costs Moderate ingest and index volume by aggregating, and filtering events to reduce license spend Lower overall operating expenses Reduce the need for proprietary forwarders like Splunk Heavy Forwarder with Splunk S2S connector
- 26. Serverless integration Connect existing and apps & data stores in a repeatable way without having to manage- Apache Kafka, Schema Registry to maintain app compatibility, ksqlDB to develop real-time apps with SQL syntax and Connect for effortless integrations with Lambda & data stores AWS serverless platform Stop provisioning, maintaining or administering servers for backend components such as compute, databases and storage so that you can focus on increasing agility and innovation for your developer teams Increase developer agility & speed of innovation Apps Microservices ksqlDB Schema Registry COMPUTE AWS Lambda Data stores REST Proxy & Clients Source Connectors Lambda Sink DATA STORES Amazon DynamoDB Amazon Aurora STORAGE Amazon S3 S3 Sink ANALYTICS Amazon Athena Amazon Redshift Serverless app integration
- 28. No provisioning, no management Pay for value Automatic scaling Highly available and secure What is serverless?
- 29. No servers to manage Only pay for stream consumption when processing messages Automatically scales consumers Benefits of Serverless stream processing Write less code
- 30. Serverless processing Server-based processing ✔ Stream polling logic is separate from application logic ✔ Stream polling logic is baked into your application code ✔ Event driven processing ✔ Consumer must be running to poll the kafka clusters ✔ Scaling is handled automatically ✔ Scaling is done using consumer groups. ✔ Poller: Open source APIs/libraries( KafkaStreams javalibrary, kafka-python ) • Poller: Lambda ESM Confluent Lambda Sink Connector
- 31. Apache Kafka anatomy 101
- 32. Apache Kafka – Writes to partitions
- 33. Apache Kafka – Reads from partitions
- 34. Lambda consumer options Event Source Mapping Lambda Service Confluent Kafka Sink Connector
- 35. Confluent Lambda Sink connector • Sink connector polls Kafka partitions and calls your function • Lambda can be called synchronously or asynchronously. • At least once semantics • Provides a dead letter queue (DLQ) for any failed invocations
- 36. Confluent Lambda Sink connector – Scaling and Error Handling • Sink connector scales upto a soft maximum of 10 connectors. • Error handling semantics similar to sync and async lambda invocations. • Async: Lambda service retries twice (three total attempts) • Sync: By default, fails and stop processing for that partition. Option to log to another kafka topic and continue processing • Option to batch records. Configured through aws.lambda.batch.size
- 37. Lambda ESM consumer for Kafka Lambda Function instance Poller • Starts with one concurrent poller and customer function • Lambda service polls the Kafka partitions and invokes your lambda function synchronously
- 38. Lambda ESM consumer for Kafka – Scaling and Batching Lambd a Function instance Polle r Function instance Function instance • Scaling: • Lambda service checks every 3 mins if scaling is needed. • Starts with 1 poller and scales upto <= #partitions • Batching: Batch records based on a BatchSize or Batchwindow.
- 39. Best Practices
- 40. Capture and log exceptions data producer Lambda service function A (instance 1) batch size = 200 300 records ✔ function A (instance 1) ✔ Catch exceptions and log to CloudWatch Logs CloudWatch Logs Return successfully from Lambda function • Ensure processing moves forward by catching exceptions and returning successfully !
- 41. Optimize batch-size/batch- window to lower cost Lambda Function instance Poller • Lambda’s maximum execution time is 15 minutes • Adjust the batch size (max 10,000) to ensure execution time is optimal • For sparse topics, consider batch window to aggregate over a time period
- 42. Kafka Producer in Lambda (create once, use many) • Create Producer in the constructor • Producer will be re-used across executions for the life of the Lambda instance • Reduce strain on brokers by minimizing connections and producer clients Producer Producers!!!! This Not this
- 43. Consider using ksqlDB for state • A powerful combination of ksqlDB and Lambda provides a stateful -> stateless -> stateful pattern
- 44. Enrich Transaction events for Fraud scoring Customer Transactio n Jay $10 ksqlDB
- 45. Enrich Transaction events for Fraud scoring Customer Transaction Avg 7 days Num trans 10m Jay $10 $8.5 1
- 46. Enrich Transaction events for Fraud scoring Customer Transaction Avg 7 days Num trans 10m Jay $10 $8.5 1 Amazon SageMaker AWS Lambda ksqlDB
- 47. Next Steps How did we do? Enter your feedback <CSAT link & QR Code> Schedule an executive briefing Schedule a briefing for your business and technology leadership team Join or schedule a workshop ● Join our FSI workshop on July 27th <link/QR code to registration> ● Schedule a workshop for your team Learn more about serverless Visit Serverlessland.com for self guided workshops, videos, and resources 1 2 3
- 48. Thank You! Don’t forget to fill out the survey!