OSACon 2023_ Unlocking Financial Data with Real-Time Pipelines
0 likes217 views
The document discusses the importance of real-time data pipelines in the financial sector, focusing on how technologies like Apache Flink, Nifi, Kafka, and Iceberg can transform data management processes. It highlights challenges with traditional batch processing and demonstrates how these tools can improve decision-making, risk assessment, and regulatory compliance. The session aims to provide insights into building efficient real-time pipelines to empower financial institutions in a fast-paced data-driven environment.
OSACon 2023_ Unlocking Financial Data with Real-Time Pipelines
- 1. © 2023 Cloudera, Inc. All rights reserved. Unlocking Financial Data with Real-Time Pipelines Tim Spann Principal Developer Advocate 14-December-2023
- 2. © 2023 Cloudera, Inc. All rights reserved.
- 3. © 2023 Cloudera, Inc. All rights reserved. 3 Introduction Overview Apache Kafka Apache Flink Apache NiFi Use Cases Demos Agenda (30 minutes)
- 4. © 2023 Cloudera, Inc. All rights reserved. 4 This week in Apache NiFi, Apache Flink, Apache Kafka, ML, AI, Apache Spark, Apache Iceberg, Python, Java and Open Source friends. https://bit.ly/32dAJft https://www.meetup.com/futureofdata- princeton/ FLaNK Stack Weekly by Tim Spann
- 5. © 2023 Cloudera, Inc. All rights reserved. 5 Confidential—Restricted @PaasDev https://www.meetup.com/futureofdata-princeton/ From Big Data to AI to Streaming to Containers to Cloud to Analytics to Cloud Storage to Fast Data to Machine Learning to Microservices to ... Future of Data - NYC + NJ + Philly + Virtual
- 6. © 2023 Cloudera, Inc. All rights reserved. 6 Tim Spann Twitter: @PaasDev // Blog: datainmotion.dev Principal Developer Advocate. Princeton Future of Data Meetup. ex-Pivotal, ex-Hortonworks, ex-StreamNative, ex-PwC https://medium.com/@tspann https://github.com/tspannhw
- 7. © 2023 Cloudera, Inc. All rights reserved. 7 Financial institutions thrive on accurate and timely data to drive critical decision-making processes, risk assessments, and regulatory compliance. However, managing and processing vast amounts of financial data in real-time can be a daunting task. To overcome this challenge, modern data engineering solutions have emerged, combining powerful technologies like Apache Flink, Apache NiFi, Apache Kafka, and Iceberg to create efficient and reliable real-time data pipelines. In th talk, we will explore how this technology stack can unlock the full potential of financial data, enabling organizations to make data-driven decisions swiftly and with confidence. Introduction: Financial institutions operate in a fast-paced environment where real-time access to accurate and reliable data is crucial. Traditional batch processin falls short when it comes to handling rapidly changing financial markets and responding to customer demands promptly. In this talk, we will delve into the power of real-time data pipelines, utilizing the strengths of Apache Flink, Apache NiFi, Apache Kafka, and Iceberg, to unlock the potential of financial data. Key Points to be Covered: Introduction to Real-Time Data Pipelines: a. The limitations of traditional batch processing in the financial domain. b. Understanding the need for real-time data processing. Apache Flink: Powering Real-Time Stream Processing: a. Overview of Apache Flink and its role in real-time stream processing. b. Use cases for Apache Flink in the financial industry. c. How Flink enables fast, scalable, and fault-tolerant processing of streaming financial data. Apache Kafka: Building Resilient Event Streaming Platforms: a. Introduction to Apache Kafka and its role as a distributed streaming platform. b. Kafka's capabiliti in handling high-throughput, fault-tolerant, and real-time data streaming. c. Integration of Kafka with financial data sources and consumers. Apache NiFi: Data Ingestion and Flow Management: a. Overview of Apache NiFi and its role in data ingestion and flow management. b. Data integration and transformation capabilities of NiFi for financial data. c. Utilizing NiFi to collect and process financial data from diverse sources. Iceberg: Efficient Data Lake Management: a. Understanding Iceberg and its role in managing large-scale data lakes. b. Iceberg's schema evolution and table-leve metadata capabilities. c. How Iceberg simplifies data lake management in financial institutions. Real-World Use Cases: a. Real-time fraud detection using Flink, Kafka, and NiFi. b. Portfolio risk analysis with Iceberg and Flink. c. Streamlined regulatory reporting leveraging all four technologies. Best Practices and Considerations: a. Architectural considerations when building real-time financial data pipelines. b. Ensuring data integrity, security, and compliance in real-time pipelines. c. Scalability and performance optimization techniques. Conclusion: In this talk, we will demonstrate the power of combining Apache Flink, Apache NiFi, Apache Kafka, and Iceberg to unlock financial data's true potentia Attendees will gain insights into how these technologies can empower financial institutions to make informed decisions, respond to market changes swiftly, and comply with regulations effectively. Join us to explore the world of real-time data pipelines and revolutionize financial data management.
- 8. © 2023 Cloudera, Inc. All rights reserved. OVERVIEW
- 9. © 2023 Cloudera, Inc. All rights reserved. 9 DATA VELOCITY in FINANCIAL SERVICES Streaming capabilities vary, all enhance insight Transaction Data Core Banking Risk Data Behavioral Data Cyber Market Data News Feeds Customer Data Connected Devices/ Wearables Chat Bots Normal Streaming Regulatory, Compliance Near-Real Time Streaming Real-Time Streaming Social Media
- 10. © 2023 Cloudera, Inc. All rights reserved. 10 NEXT GEN PLATFORM FOR TACKLING FINANCIAL CRIME Leveraging data and analytics across the enterprise from the Edge to AI. Ingest Streaming Data BANKING DATA Data Flow (CDF) Data Science Workbench FINANCIAL CRIME APPLICATIONS Data Scientists Data Processing Data Engineering Data Warehouse Operational DB Catalog | Schema | Security | Governance Business Analysts EDGE DATA ALTERNATIVE DATA Enterprise Data Store ML Cyber Security AML Fraud Surveillance Analytical Tools BI and Visualization Ingest Data at Rest Deploy Models Ingest Stream or Batch Data Teams speaking the same language ENTERPRISE DATA TRADING DATA ` Ingest 1 2 3 4
- 11. 11 © 2023 Cloudera, Inc. All rights reserved. Kafka & Flink (Flink SQL with Stream SQL Builder) for real time analytics Kafka Kafka topics Database Machine learning Flink SQL w/ SSB Data Warehouse Data Viz Monitoring Alerting F in a n c e D a t a Architecture in the context of Financial Use Cases DataFlow / NiFi
- 12. © 2023 Cloudera, Inc. All rights reserved. 12 NIFI MEETS AI Vector DB AI Model Unstructured file types Data in Motion With Cloudera Capture, process & distribute any data, anywhere Other enterprise data Open Data Lakehouse Materialized Views Structured Sources Applications/API’s Streams
- 13. © 2023 Cloudera, Inc. All rights reserved. 13 Transactions Data Account Data Device Logs Business Event Logic Data Lakehouse Flagged Records Continuous Results Fraud Analyst defined suspicious transaction Real-Time Fraud Scoring Freeze transaction Fraud Monitoring Dashboard Stop Fraud When It Happens—Real Life Example Simplified example of deployed use case DATA RELEVANCE
- 15. © 2023 Cloudera, Inc. All rights reserved. APACHE KAFKA
- 16. © 2023 Cloudera, Inc. All rights reserved. © 2019 Cloudera, Inc. All rights reserved. 16 STREAMS MESSAGING WITH KAFKA • Highly reliable distributed messaging system. • Decouple applications, enables many-to-many patterns. • Publish-Subscribe semantics. • Horizontal scalability. • Efficient implementation to operate at speed with big data volumes. • Organized by topic to support several use cases.
- 17. © 2023 Cloudera, Inc. All rights reserved. APACHE FLINK
- 18. © 2023 Cloudera, Inc. All rights reserved. 18 CONTINUOUS SQL ● SSB is a Continuous SQL engine ● It’s SQL, but a slightly different mental model, but with big implications Traditional Parse/Execute/Fetch model Continuous SQL Model Hint: The query is boundless and never finishes, and time matters AKA: SELECT * FROM foo WHERE 1=0 -- will run forever
- 19. 19 © 2022 Cloudera, Inc. All rights reserved. SQL STREAM BUILDER (SSB) SQL STREAM BUILDER allows developers, analysts, and data scientists to write streaming applications with industry standard SQL. No Java or Scala code development required. Simplifies access to data in Kafka & Flink. Connectors to batch data in HDFS, Kudu, Hive, S3, JDBC, CDC and more Enrich streaming data with batch data in a single tool Democratize access to real-time data with just SQL
- 20. © 2023 Cloudera, Inc. All rights reserved. 20 SSB MATERIALIZED VIEWS Key Takeaway; MV’s allow data scientist, analyst and developers consume data from the firehose
- 21. © 2023 Cloudera, Inc. All rights reserved. APACHE NIFI
- 22. © 2023 Cloudera, Inc. All rights reserved. 22 Confidential—Restricted NiFi 2.0.0-M1 is here… https://medium.com/cloudera-inc/getting-ready-for-apache-nifi-2-0-5a5e6a67f450 - First-class citizen Python API - Rules Engine - NiFi Stateless at Process Group level - Java 21 (virtual threads, perf improvements, etc) https://medium.com/@george.vetticaden/accelerating-ai-data-pipelines-building-an-evernote-chatbot-with-apache-nifi-2-0-and-generative-ai-9d977466ff4c Closing the gap between data engineers and data scientists… - Export documentation (Sharepoint, OCR) to build the knowledge base powering your chatbot - Scrape the internet (Sitemap) to build the knowledge base powering your chatbot - Real-time streaming ingest of Slack to build the knowledge base powering your chatbot
- 23. © 2023 Cloudera, Inc. All rights reserved. 23 PROVENANCE
- 24. © 2023 Cloudera, Inc. All rights reserved. DEMO I Can Haz Data?
- 25. © 2023 Cloudera, Inc. All rights reserved. 25 Cloudera Stream Processing Community Edition ● Zero to Flink in less than an hour ○ Experiment with features ○ Develop apps locally ● One docker compose file of CSP which includes: ○ All dependencies required to run ○ Kafka, Kafka Connect and Flink ○ Streams Messaging Manager ○ Schema Registry ○ SQL Stream Builder ● Licensed under the Cloudera Community License ● Community Group Hub (Discussion Forum) for CSP ● Find it on docs.cloudera.com under Applications
- 26. © 2023 Cloudera, Inc. All rights reserved. Open Source Edition • Apache NiFi in Docker • Runs in Docker • Try new features quickly • Develop applications locally ● Docker NiFi ○ docker run --name nifi -p 8443:8443 -d -e SINGLE_USER_CREDENTIALS_USERNAME=admin -e SINGLE_USER_CREDENTIALS_PASSWORD=ctsBtRBKHRAx69EqUgh vvgEvjnaLjFEB apache/nifi:latest ● Licensed under the ASF License ● Unsupported https://hub.docker.com/r/apache/nifi
- 27. © 2023 Cloudera, Inc. All rights reserved. https://medium.com/@tspann/cdc-not-cat-data-capture-e43713879c03
- 28. © 2023 Cloudera, Inc. All rights reserved. 28 https://events.dzone.com/dzone/Data-Pipelines-Investigating-the-Modern-Day-Stack
- 29. 29 TH N Y U