Confluent & GSI Webinars series - Session 3
0 likes130 views
The document discusses the importance of data in motion for financial services, emphasizing the need for real-time processing and total connectivity to enhance customer experiences and operational efficiency. It highlights the capabilities of the Confluent platform, built on Apache Kafka, which supports various use cases such as fraud detection, cloud migration, and trading platform modernization. Additionally, it outlines training opportunities and educational resources available for organizations looking to implement event streaming solutions.
Confluent & GSI Webinars series - Session 3
- 1. Data in Motion for Financial Services Nov 2023 Oli Watson - Staff Solutions Engineer, Confluent owatson@confluent.io
- 2. Agenda 2 • Intros • Introduction to Data in Motion • Use Cases in Financial Services
- 3. We will be running Instructor Led Virtual Training Sessions in November - please contact Paul Archer parche@confluent.io or Ozan Güzeldereli oguzeldereli@confluent.io if you are interested in attending.
- 5. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Old World ...for productivity tools at the edges of a company Software is... New World ...a platform for directly transacting business
- 6. 6
- 7. New use cases need new capabilities. This requires total connectivity and instant reaction, all the time, in real-time.
- 8. At the heart of every software application is data. Databases 8
- 9. Databases are fundamentally incomplete. Databases are designed for disconnected and UI-centric applications. Databases Slow, daily batch processing Simple, static real-time queries 9
- 10. It’s all about data!
- 11. Initial Database Driven Architecture DATABASE WEB APP WEB APP
- 13. STATE EVENT > I changed my job from Couchbase to Confluent. I work at Confluent.
- 14. JOB CHANGE RECOMMENDATION ENGINE SEARCH INDEX EMAIL SERVICE
- 15. REALISATION 2 LEVERAGE ALL DIGITISED EVENTS
- 17. DATABASE IS A MISMATCH FOR REALISATION 1 & 2
- 18. SQL SQL SQL RECOMMENDATION ENGINE SEARCH INDEX EMAIL SERVICE Mismatch 1: No First Class Treatment for Events DATABASE
- 19. Mismatch 2: Not Suitable for Volume of All Digitized Events DATABASE 1000x more volume NON-TRANSACTIONAL EVENTS TRANSACTIONAL EVENTS
- 20. The foundational assumption of every database: data at rest. Databases Slow, daily batch processing Simple, static real-time queries Data is at rest 20
- 21. Databases bring point-in-time queries to stored data. This leads to a Giant Mess in Data Architecture. LINE OF BUSINESS 01 LINE OF BUSINESS 02 PUBLIC CLOUD 21
- 22. Data in motion: Ubiquitous real-time data and continuous real-time processing.
- 23. A New Paradigm is Required for Data in Motion: Continuously processing evolving streams of data in real-time 23 Rich front-end customer experiences Real-time Events Real-time Event Streams A Sale A shipment A Trade A Customer Experience Real-time backend operations
- 24. Confluent - Who we are and what we do Hall of Innovation CTO Innovation Award Winner 2019 Enterprise Technology Innovation AWARDS Confluent founders are original creators of Kafka Confluent team wrote 80% of Kafka commits and has over 1M hours technical experience with Kafka Confluent helps enterprises successfully deploy event streaming at scale and accelerate time to market Confluent Platform extends Apache Kafka to be a secure, enterprise-ready platform
- 25. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. How Confluent technology is used Confluent Platform The Enterprise Distribution of Apache Kafka Confluent Cloud Apache Kafka Re-engineered for the Cloud Self-Managed Software Fully-Managed Service VM Deploy on any platform, on-prem or cloud Available on the leading public clouds
- 26. Confluent is so much more than Apache Kafka Complete: Go beyond Kafka with all the essential tools for a complete data streaming platform Enterprise-grade Security RBAC | Audit Logs | Encryption | BYOK | Private Networking Stream Governance Schema Registry | Schema Validation | Stream Lineage | Stream Catalog Complete Engagement Model Data in Motion Blueprint Business Case Justification TCO | ROI | Risk Management & Monitoring Cloud UI | Metrics API | Control Center | Health+ Flexible DevOps Automation Admin REST APIs | Terraform APIs | Confluent for K8s | Ansible Playbooks Efficient Operations at Scale Production-stage Prerequisites Partnership for Business Success Multi-language Development Non-Java Clients | REST Proxy | MQTT Proxy Stream Processing & Integration Connectors | ksqlDB | Stream Designer Unrestricted Developer Productivity High Availability 99.99% SLA | Multi-AZ Clusters | Multi-Region Clusters Infinite Storage Infinite Storage | Tiered Storage Elastic Scalability Expand | Shrink | Self-Balancing Clusters Cloud Native: Apache Kafka© , fully managed and re-architected to harness the power of the Cloud Everywhere: Connect your data in real time with a platform that spans from on-prem to cloud and across clouds Hybrid and Multicloud Cluster Linking | Replicator Self-managed software Kubernetes | VMs | Bare Metal Fully managed cloud service AWS | Azure | GCP Committer-driven Expertise Training Partners Professional Services Enterprise Support OPERATOR DEVELOPER ARCHITECT EXECUTIVE
- 27. Financial Services Use Cases
- 28. 28 Increase Revenue → Customer Experience, Loyalty Decrease Costs → Increase Operational Efficiency Mitigate Risks → Regulatory Compliance IB and Wealth MIFID, OATS, CAT reporting Risk: Credit, Equities, FX Intraday Liquidity Legacy IT Replacement (e.g. Middleware replacement) Cyber Security (incl. SIEM) Fraud Prevention (incl. Anti-Money-Laundering - AML / Real-time ATM dispute resolution) Legacy IT Modernization (e.g. Mainframe off-load / augmentation) Example Financial Services Solutions Retail Open Banking APIs Real-time Notifications Call Center - Know Your Customer - KYC Software defined business Account Opening / Loans / Mortgages / Next Best Action / Targeted Offers) Pre/Post Trade Market + Trade Data Distribution Trade Validation Position Management Migration to the Cloud (Hybrid on-prem / Cloud. Also Hybrid Public Cloud vendors) (Choreographed) Microservices Architecture Data Infrastructure layer Business Application layer - the use cases Data Pipelines Messaging Microservice/ Event Sourcing Stream Processing Data Integration Streaming ETL Log Aggregation
- 29. Some Highlights Payments • Fraud • Payment Processing • PSD2 • Settlements Customer Experience: • Identity • Customer 360 • KYC • Loan Approvals • Open Banking 29 Multi Cloud / Hybrid Cloud • Event Streaming standardisation across environments Messaging • MQ and TIBCO offload / deco Data Warehouse Modernisation • Netezza, Teradata etc to Cloud Data Services Mainframe • Data offload Security and Cyber • SIEM Regulatory • MIFID, OATS, CAT etc • Risk: Credit, Equities, FX • Intraday Liquidity Trading: • Market Data • Trade Validation • Trade Data Distribution • Position Management Wealth / Private Banking • Customer Experience • Free up data: • e.g offload (e.g. T24) Retail and Corp Banking IB and Wealth Technology Transformation
- 30. Payments
- 31. Challenges ● Growth in multi channel fraud ○ Attacks based on multiple LOBs ○ LOBs very siloed - separate fraud systems ○ Get a business wide view, and bring much more data into the fraud management process ● Operational Cost of Fraud ○ Fraud case management ○ % of cases which require manual intervention ○ Regulatory requirements ● Customer Experience ○ Mobile app integrations ○ Getting the right signal to noise ratio ○ False Positive <-> False Negatives ● Expensive solutions based off legacy technology stacks ○ Not suited to operating in a Cloud environment ○ Costly to license and run Use Cases: -Fraud Detection, with Machine Learning against large historical data sets Major Global Bank and Payments Processor ● 8-10M payments/day avg ● 3.5billion + payments year ● Processor, Issuer, Merchant Acquirer Realtime Fraud Scoring
- 32. payments credit card transactions debit card transactions credit applications mobile app data realtime fraud scoring ML model enhancement fraud case management back testing on 1 year of trx Realtime Fraud Scoring Major Global Bank and Payments Processor ● 8-10M payments/day avg ● 3.5billion + payments year ● Processor, Acquirer, Merchant Acquirer
- 33. Realtime Fraud Scoring Major Global Bank and Payments Processor ● 8-10M payments/day avg ● 3.5billion + payments year ● Processor, Acquirer, Merchant Acquirer On Prem Tokenization Micro Service AZ-1a AZ-1b AZ-1c Fraud Processing Results AWS Realtime Scoring Case Management Model Enhancement Model Testing
- 34. ● Payments Processing and Gateways ○ Modernise legacy messaging solutions and move to cloud ○ Integrate the bank with SaaS payment solutions (Volante, Form3 etc) ○ Payments Hubs, SEPA ○ Realtime Gross Settlement ○ Provide value add services with realtime analytics ● Regulatory ○ Intraday Liquidity - batch to realtime ○ Sanctions Enforcement ○ PSD2 ● Customer Experience ○ Corporate User Experience ○ Payments Transparency More in payments...
- 36. UK retail finance organisation who are a major provider of mortgages, and other financial services Use Cases: -Mainframe Offloading -Legacy to modern application Integrations -Data analytics and reporting Challenges ● Competition from digital first banks driving disruption and modernization ● Digital disruption efforts including open banking, regulatory requirements and expose data through APIs ● High and unpredictable data volumes, 24x7 SLA and availability requirements, ● Protect core Systems of Record (SORs) from the external loads / applications. ● Drive Cloud adoption and IT modernization strategy Solution ● Developed an event based real-time data platform on Confluent called “Speed Layer” ● Speed Layer - preferred source of data for high-volume read-only data requests and event sourcing. ● Delivered secure, near real time customer, account and transaction information from back end systems to front end systems with speed and resilience. ● Microservices architectures to onboard new use cases quickly and easily ● Maintain service availability despite unprecedented demand, agility and autonomy in digital development teams Mainframe Offloading - Major UK FS Org and Mortgage Provider
- 38. Confluent Cloud
- 39. Major international IB and Wealth Manager. Use Cases: -Kafka as a service -Infrastructure log management -IB Workloads - Globally distributed Wealth Management workloads Scenario ● Bank is a big user of Kafka and already has a platform based Confluent Deployment with multiple tenant applications and various standalone deployments ● Cloud first strategy of the Bank leading to difficulty expanding on premise infrastructure ● High Cost of ownership for internally managed Kafka instance Solution ● Confluent Cloud selected as Kafka provider of choice and first 3rd party cloud service authorised within the bank ● Dedicated clusters provisioned with private link networking ● Service curated by infrastructure team and offered out to internal tenants ● First production use case realised within 3 months of contract ● Projects underway to migrate existing workloads ● New use cases being built directly on cloud infrastructure Kafka as a Service
- 41. “We look at events as running our business. Business people within our organization want to be able to react to events—and oftentimes it's a combination of events.” VP of Streaming Data Engineering 41
- 42. Challenge: Rapidly transform digital services to meet customers’ rising expectations for highly secure, personalized and valuable experiences. Solution: With the ability to combine real-time results with historical events, Confluent is leading the charge in helping enterprises discover new services and modernize existing ones with the power of event streaming. Results: Credit Suisse Names Confluent a Disruptive Technology Award Winner “Enterprise IT is under immense pressure to rapidly transform digital services to meet customers’ rising expectations for highly secure, personalized and valuable experiences. With the ability to combine real-time results with historical events, Confluent is leading the charge in helping enterprises discover new services and modernize existing ones with the power of event streaming.” — David Patten, CIO of Investment Banking Capital Markets
- 43. Challenge: Develop a new trading platform for markets across multiple European countries that supports high-volume, high-speed trading and provides clients with access to real-time data. Solution: Use Confluent Platform to implement a reliable, scalable persistence layer for market orders that supports millisecond latencies and billions of messages per day. Results: ● Reliable 24/5 operations achieved and maintained ● Stringent performance requirements exceeded ● Dedicated, expert support received “We have been very satisfied with Confluent Platform as the backbone of our persistence engine. The platform has been super reliable. We have stringent requirements for real-time performance and reliability, and we have confirmed – from proof-of-concept to deployment of a cutting-edge production trading platform – that we made the right decision.” — Alain Courbebaisse, Chief Information Officer, Euronext
- 44. Patterns and Themes Kafka Use Cases for Financial Services
- 45. Common Patterns and Themes Regardless of whether it’s Retail Banking, Global Markets, Commercial Banking…. There’s common demands and architectural patterns emerging across the board ● Be more agile, and respond to changing customer and regulatory requirements ● Differentiate through technology ● Reduce reliance on, and where possible, decommission legacy ● Build new applications that can be deployed on Cloud ● Respond in real time, not batch
- 46. Appendix
- 47. Education
- 48. Education Package About us - https://www.confluent.io/about/ Confluent is creating the foundational platform for data-in-motion. With Confluent, organizations can harness the full power of continuously flowing data to innovate and win in the modern digital world. What Does Kafka Do? – https://developer.confluent.io/ Learn about the fundamentals of Kafka, event streaming, and the surrounding ecosystem. Hybrid or multicloud architecture – https://www.confluent.io/use-case/hybrid-and-multicloud/ Confluent’s hybrid and multi cloud data streaming solutions power real-time interoperability between any systems, applications, and datastores on any number of on-premises and cloud environments. Innovate faster, reduce risk, and maximize ROI with cloud-native ease and simplicity. Event-Driven Microservices – https://www.confluent.io/use-case/event-driven-microservices-communication/ Completely decouple your architecture and eliminate inter-service dependencies so developers can build business logic faster. Build a new class of highly scalable event-drivenmicroservices that are resilient in design and contextually aware. Explore top use cases – https://developer.confluent.io/tutorials/ Learn stream processing the simple way. Use this cookbook of recipes to easily get started at any level. Resources – https://www.confluent.io/resources/ All available Confluent resources
- 49. Education Package Confluent Developer website Courses – https://developer.confluent.io/learn-kafka/ Blog – https://www.confluent.io/blog/ Podcast - https://developer.confluent.io/podcast/ Articles – https://developer.confluent.io/learn/ FAQs – https://developer.confluent.io/learn/apache-kafka-faqs/ 100-Day Code Challenge – https://developer.confluent.io/100-days-of-code/ Tutorials – https://developer.confluent.io/tutorials/ Demos – https://developer.confluent.io/demos-examples/
- 50. Education Package Free Self-Paced Training For individuals preferring to learn at their own pace, we offer the ability to focus on a particular topic, aligned to a role or technology, or access our entire library of self-paced content. ● Apache Kafka Fundamentals and Accreditation ● Developer Learning Path ● Administrator Learning Path ● Security Learning Path ● Confluent Cloud Learning Path https://training.confluent.io/content
- 51. Education Package E-Books Designing Event-Driven Systems Mastering Kafka Streams and ksqlDB Kafka The Definitive Guide I Heart Logs Practical Data Mesh
- 52. Kafka 101
- 54. Continuous Commit Log Time C C C
- 55. Apache Kafka is a Distributed Event Streaming Platform Process streams of events In real time, as they occur 110101 010111 001101 100010 Publish and subscribe to streams of events Similar to a message queue or enterprise messaging system 110101 010111 001101 100010 Store streams of events In a fault tolerant way 110101 010111 001101 100010 Connect Rich ecosystem of connectors, source and sink to/from hundreds of other systems 110101 010111 001101 100010 55
- 56. Anatomy of a Kafka Topic 1 2 3 4 5 6 8 9 7 Partition 1 Old New 1 2 3 4 5 6 8 7 Partition 0 10 9 11 12 Partition 2 1 2 3 4 5 6 8 7 10 9 11 12 Writes 1 2 3 4 5 6 8 7 10 9 11 12 Producers Writes Consumer A (offset=4) Consumer B (offset=7) Reads
- 57. Kafka Connect and Kafka Streams Sink Source KAFKA STREAMS KAFKA CONNECT KAFKA CONNECT Your App 57
- 58. Instantly Connect Popular Data Sources & Sinks Data Diode 200+ pre-built connectors 80+ Confluent Supported 50+ Partner Supported, Confluent Verified