AIDEVDAY_ Data-in-Motion to Supercharge AI
0 likes1,086 views
The document outlines Cloudera's initiatives in leveraging data flow and streaming analytics for AI applications, highlighting tools like Apache NiFi, Kafka, and Flink. It discusses the integration of various data management processes, including data storage, preparation, and model deployment across diverse environments. Additionally, it provides resources for developers to explore new features and community editions of Cloudera’s tools.
AIDEVDAY_ Data-in-Motion to Supercharge AI
- 1. © 2023 Cloudera, Inc. All rights reserved. Data-in-Motion to Supercharge AI Tim Spann Principal Developer Advocate 23-August-2023
- 2. © 2023 Cloudera, Inc. All rights reserved. 2 Tim Spann @PaasDev www.datainmotion.dev github.com/tspannhw medium.com/@tspann Principal Developer Advocate Princeton Future of Data Meetup ex-Pivotal, ex-Hortonworks, ex-StreamNative, ex-PwC, ex-EY, ex-HPE. Apache NiFi x Apache Kafka x Apache Flink x AI
- 3. © 2023 Cloudera, Inc. All rights reserved. 3 REAL-TIME REQUIRES A PLATFORM SQL Stream Builder
- 4. © 2023 Cloudera, Inc. All rights reserved. © 2019 Cloudera, Inc. All rights reserved. 4 Cloudera + LLMs Knowledge Repository Data Storage / Management Data Preparation Data Engineering LLM Fine Tuning Process Training Framework LLM Serving Serving Framework Key: CPU Task GPU Task CML CDE CDP Vector DB CDF Streaming Classification Real-Time Model Deployment
- 5. © 2023 Cloudera, Inc. All rights reserved. Run collection and streaming on any cloud, server, container, bare metal, device or VM Data Sources Cloudera Data Flow Cloudera Streaming Analytics Cloudera Streams Processing Kafka Lake House INGEST
- 6. © 2023 Cloudera, Inc. All rights reserved. ENRICH
- 7. © 2023 Cloudera, Inc. All rights reserved. FUNNEL https://github.com/tspannhw/FLaNK-HuggingFace-DistilBert-SentimentAnalysis https://github.com/tspannhw/FLaNK-LLM
- 8. © 2023 Cloudera, Inc. All rights reserved. DISTRIBUTE DEPLOY https://github.com/tspannhw/FLaNK-Edge-Models
- 9. © 2023 Cloudera, Inc. All rights reserved. STORE
- 10. © 2019 Cloudera, Inc. All rights reserved. 10 APACHE NIFI WITH PYTHON CUSTOM PROCESSORS Python as a First Class Citizen https://github.com/apache/nifi/blob/614947e4ac6798ad80817e82514c39349d5faacb/nifi-docs/src/main/ asciidoc/python-developer-guide.adoc
- 11. 11 © 2023 Cloudera, Inc. All rights reserved. Future of Data - Princeton + Virtual @PaasDev https://www.meetup.com/futureofdata-princeton/ From Big Data to AI to Streaming to Containers to Cloud to Analytics to Cloud Storage to Fast Data to Machine Learning to Microservices to ...
- 12. FLaNK Stack Weekly This week in Apache NiFi, Apache Flink, Apache Kafka, Apache Spark, Apache Iceberg, Python, Java, AI, ML, LLM and Open Source friends. https://bit.ly/32dAJft
- 13. © 2023 Cloudera, Inc. All rights reserved. 13 CSP Community Edition • Kafka, KConnect, SMM, SR, Flink, and SSB in Docker • Runs in Docker • Try new features quickly • Develop applications locally ● Docker compose file of CSP to run from command line w/o any dependencies, including Flink, SQL Stream Builder, Kafka, Kafka Connect, Streams Messaging Manager and Schema Registry ○ $> docker compose up ● Licensed under the Cloudera Community License ● Unsupported ● Community Group Hub for CSP ● Find it on docs.cloudera.com under Applications
- 14. Open Source Edition • Apache NiFi in Docker • Runs in Docker • Try new features quickly • Develop applications locally ● Docker NiFi ○ docker run --name nifi -p 8443:8443 -d -e SINGLE_USER_CREDENTIALS_USERNAME=admin -e SINGLE_USER_CREDENTIALS_PASSWORD=ctsBtRBKHRAx69EqUghv vgEvjnaLjFEB apache/nifi:latest ● Licensed under the ASF License ● Unsupported https://hub.docker.com/r/apache/nifi
- 15. © 2023 Cloudera, Inc. All rights reserved. 15 Resources threads.net/@tspannhw