Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg with Stock Data and LLM
0 likes267 views
The document describes the integration of Apache NiFi, Kafka, Risingwave, and Iceberg for managing stock data using machine learning and AI within Cloudera's open data lakehouse framework. It emphasizes the importance of data streaming in financial services, detailing how these technologies enhance data processing and insights. Additionally, it highlights Apache NiFi's evolution and community engagement as a key player in open-source data solutions.
Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg with Stock Data and LLM
- 1. © 2024 Cloudera, Inc. All rights reserved. Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg with Stock Data and LLM Karin Wolok Developer Relations, Dev Marketing, and Community Programming @ Project Elevate Tim Spann Principal Developer Advocate, Cloudera 29-February-2024
- 2. © 2024 Cloudera, Inc. All rights reserved. 2 Tim Spann Twitter: @PaasDev // Blog: datainmotion.dev Principal Developer Advocate. Princeton Future of Data Meetup. ex-Pivotal, ex-Hortonworks, ex-StreamNative, ex-PwC, ex-HPE https://medium.com/@tspann https://github.com/tspannhw
- 3. © 2024 Cloudera, Inc. All rights reserved. 3 Confidential—Restricted @PaasDev https://www.meetup.com/futureofdata-princeton/ https://www.meetup.com/futureofdata-newyork/ From Big Data to AI to Streaming to Containers to Cloud to Analytics to Cloud Storage to Fast Data to Machine Learning to Microservices to ... Future of Data - NYC + NJ + Philly + Virtual
- 4. © 2024 Cloudera, Inc. All rights reserved. 4 This week in Apache NiFi, Apache Flink, Apache Kafka, ML, AI, Apache Spark, Apache Iceberg, Python, Java, LLM, GenAI, Vector DB and Open Source friends. https://bit.ly/32dAJft https://www.meetup.com/futureofdata- princeton/ FLaNK Stack Weekly by Tim Spann
- 5. © 2024 Cloudera, Inc. All rights reserved. OVERVIEW
- 6. © 2024 Cloudera, Inc. All rights reserved. 6 DATA VELOCITY in FINANCIAL SERVICES Streaming capabilities vary, all enhance insight Transaction Data Core Banking Risk Data Behavioral Data Cyber Market Data News Feeds Customer Data Connected Devices/ Wearables Chat Bots Normal Streaming Regulatory, Compliance Near-Real Time Streaming Real-Time Streaming Social Media
- 7. © 2023 Cloudera, Inc. All rights reserved. 7 NIFI MEETS AI Vector DB AI Model Unstructured file types Data in Motion With Cloudera Capture, process & distribute any data, anywhere Other enterprise data Open Data Lakehouse Materialized Views Structured Sources Applications/API’s Streams
- 9. © 2024 Cloudera, Inc. All rights reserved. FLaNK for Halifax Canada Transit — NiFi, Kafka, Flink, SQL, GTFS-RT | by Tim Spann | Cloudera | Dec, 2023 | Medium Never Get Lost in the Stream. NiFi-Kafka-Flink for getting to work… | by Tim Spann | Cloudera | Dec, 2023 | Medium Iteration 1: Building a System to Consume All the Real-Time Transit Data in the World At Once | by Tim Spann | Cloudera | Medium Watching Airport Traffic in Real-Time | by Tim Spann | Cloudera | Medium
- 10. © 2024 Cloudera, Inc. All rights reserved. APACHE ICEBERG
- 11. © 2024 Cloudera, Inc. All rights reserved.
- 12. © 2024 Cloudera, Inc. All rights reserved.
- 13. © 2024 Cloudera, Inc. All rights reserved. 13 Cloudera’s Open Data Lakehouse ❏ Multi-function analytics for Streaming, Data Engineering, Data Warehouse and AI/ML with integrated data services ❏ Common security and governance policies and data lineage with SDX integration ❏ Common dataset with all CDP analytics engines without data duplication and movement ❏ Deployment freedom with Multi-Hybrid Cloud Iceberg Tables DATA WAREHOUSE MACHINE LEARNING DATA ENGINEERING DATA FLOW STREAM PROCESSING Multi-Hybrid Cloud Metadata | Security | Encryption | Control | Governance
- 14. © 2024 Cloudera, Inc. All rights reserved. 14 Compute Engine Interoperability & SDX Integration ● Snapshot isolation ensures consistent data access and processing with various compute engines including Hive, Spark, Impala and Nifi ● Security & Governance support (e.g. FGAC) through Ranger integration ● Data lineage support through Atlas integration Apache Impala Iceberg Tables Ranger Atlas
- 15. © 2024 Cloudera, Inc. All rights reserved. DATAFLOW APACHE NIFI
- 16. © 2024 Cloudera, Inc. All rights reserved. 16 Apache NiFi - developed 17 years ago by the NSA 2006 NiagaraFiles (NiFi) was first incepted at the National Security Agency (NSA) November 2014 NiFi is donated to the Apache Software Foundation (ASF) through NSA’s Technology Transfer Program and enters ASF’s incubator. July 2015 NiFi reaches ASF top-level project status
- 17. © 2024 Cloudera, Inc. All rights reserved. 17 Apache NiFi in a few numbers A very active project with a dynamic community & comparison with ACEU 2019 2800+ members on the Slack channel (535+ - 4 years ago) 475+ contributors on Github across the repositories (260+ - 4 years ago) 65 committers in the Apache NiFi community (45 - 4 years ago) Apache NiFi 1.25.0 is the latest release, NiFi 2.0.0-M2 is in alpha. 14M+ docker pulls of the Apache NiFi image (1M+ - 4 years ago)
- 18. © 2024 Cloudera, Inc. All rights reserved. © 2023 Cloudera, Inc. All rights reserved. 18 PROVENANCE
- 19. © 2024 Cloudera, Inc. All rights reserved. © 2019 Cloudera, Inc. All rights reserved. 19 RECORD-ORIENTED DATA WITH NIFI • Record Readers - Avro, CSV, Grok, IPFIX, JSAN1, JSON, Parquet, Scripted, Syslog5424, Syslog, WindowsEvent, XML • Record Writers - Avro, CSV, FreeFromText, Json, Parquet, Scripted, XML • Record Reader and Writer support referencing a schema registry for retrieving schemas when necessary. • Enable processors that accept any data format without having to worry about the parsing and serialization logic. • Allows us to keep FlowFiles larger, each consisting of multiple records, which results in far better performance.
- 20. © 2024 Cloudera, Inc. All rights reserved. APACHE KAFKA
- 21. © 2024 Cloudera, Inc. All rights reserved. © 2019 Cloudera, Inc. All rights reserved. 21 STREAMS MESSAGING WITH KAFKA • Highly reliable distributed messaging system. • Decouple applications, enables many-to-many patterns. • Publish-Subscribe semantics. • Horizontal scalability. • Efficient implementation to operate at speed with big data volumes. • Organized by topic to support several use cases.
- 22. © 2024 Cloudera, Inc. All rights reserved. DEMO I Can Haz Data?
- 23. © 2024 Cloudera, Inc. All rights reserved. https://medium.com/@tspann/cdc-not-cat-data-capture-e43713879c03
- 24. © 2024 Cloudera, Inc. All rights reserved.
- 25. © 2024 Cloudera, Inc. All rights reserved.
- 26. © 2024 Cloudera, Inc. All rights reserved. https://github.com/tspannhw/PaK-Stocks https://github.com/tspannhw/FLaNK-Py-Stocks https://medium.com/cloudera-inc/let-nifi-worry-about-those-stoc ks-for-you-57d5f16b5e6b
- 27. 27 TH N Y U