TSPANN-2024-Nov-CloudX-Adding Generative AI to Real-Time Streaming Pipelines
0 likes79 views
The document discusses the integration of generative AI into real-time streaming pipelines, focusing on handling unstructured data using vector embeddings and neural networks. It reviews features of Apache NiFi, including new functionalities for processing unstructured data and examples of applications like image captioning and classification. Additionally, it provides resources for further exploration and development in AI and streaming technologies.
TSPANN-2024-Nov-CloudX-Adding Generative AI to Real-Time Streaming Pipelines
- 1. Adding Generative AI to Real-Time Streaming Pipelines Tim Spann Principal Developer Advocate Nov-2024
- 3. 3 Tim Spann Twitter: @PaasDev // Blog: datainmotion.dev Principal Developer Advocate / Field Engineer NY AI Meetups ex-Pivotal, ex-Cloudera, ex-StreamNative, ex-PwC, ex-HPE, ex-E&Y. https://medium.com/@tspann https://github.com/tspannhw
- 4. 4 This week in Apache NiFi, Apache Flink, Apache Kafka, ML, AI, Apache Spark, Apache Iceberg, Python, Java, LLM, GenAI, Vector DB and Open Source friends. https://bit.ly/32dAJft https://www.meetup.com/futureofdata- princeton/ AI + Streaming Weekly by Tim Spann
- 5. The challenge of Unstructured Data ● Problem: Unstructured data comes in lots of forms, no easy way to interact with it all ● Solution: Vector embeddings ● How: Neural networks e.g. embedding models Vector Databases
- 6. Unstructured Data is Everywhere Unstructured data is any data that does not conform to a predefined data model. Currently, 90% of unstructured data is never analyzed. Images Videos and more! Text
- 7. Image from Nvidia Vector Search Overview
- 8. How Similarity Search Works Vn, 1 … … … 1 2 3 4 5 Transform into Vectors Unstructured Data Images User Generated Content Video Documents Audio Vector Embeddings Perform Approximate Nearest Neighbor Similarity Search Perform Query Get Results Store in Vector Database
- 9. Real-Time Pipelines Can Help External Context Ingest Ingesting, routing, clean, enrich, transforming, parsing, chunking and vectorizing structured, unstructured, semistructured, binary data and documents Prompt engineering Crafting and structuring queries to optimize LLM responses Context Retrieval Enhancing LLM with external context such as Retrieval Augmented Generation (RAG) Roundtrip Interface Act as a Discord, REST, Kafka, SQL, Slack bot to roundtrip discussions
- 10. https://medium.com/cloudera-inc/getting-ready-for-apache-nifi-2-0-5a5e6a67f450 NiFi 2.0.0 Features ● Python Integration ● Parameters ● JDK 21+ ● JSON Flow Serialization ● Rules Engine for Development Assistance ● Run Process Group as Stateless ● flow.json.gz https://cwiki.apache.org/confluence/display/NIFI/NiFi+2.0+Release+Goals
- 11. Extract Company Names ● Python 3.10+ ● Hugging Face, NLP, SpaCY, PyTorch https://github.com/tspannhw/FLaNK-python-ExtractCompanyName-processor
- 12. CaptionImage ● Python 3.10+ ● Hugging Face ● Salesforce/blip-image-captioning-large ● Generate Captions for Images ● Adds captions to FlowFile Attributes ● Does not require download or copies of your images https://github.com/tspannhw/FLaNK-python-processors
- 13. RESNetImageClassification ● Python 3.10+ ● Hugging Face ● Transformers ● Pytorch ● Datasets ● microsoft/resnet-50 ● Adds classification label to FlowFile Attributes ● Does not require download or copies of your images https://github.com/tspannhw/FLaNK-python-processors
- 14. NSFWImageDetection ● Python 3.10+ ● Hugging Face ● Transformers ● Falconsai/nsfw_image_detection ● Adds normal and nsfw to FlowFile Attributes ● Gives score on safety of image ● Does not require download or copies of your images https://github.com/tspannhw/FLaNK-python-processors
- 15. FacialEmotionsImageDetection ● Python 3.10+ ● Hugging Face ● Transformers ● facial_emotions_image_detection ● Image Classification ● Adds labels/scores to FlowFile Attributes ● Does not require download or copies of your images https://github.com/tspannhw/FLaNK-python-processors
- 16. Let’s do a metamorphosis on your data. Don’t fear changing data. You don’t need to be a brilliant writer to stream data. Franz Kafka was a German-speaking Bohemian novelist and short-story writer, widely regarded as one of the major figures of 20th-century literature. His work fuses elements of realism and the fantastic. Wikipedia YES, FRANZ, IT’S KAFKA
- 17. Open Source Edition •Apache NiFi in Docker •Try new features quickly •Develop applications locally ● Docker NiFi ○ docker run --name nifi -p 8443:8443 -d -e SINGLE_USER_CREDENTIALS_USERNAME=admin -e SINGLE_USER_CREDENTIALS_PASSWORD=ctsBtRBKHRAx69EqUghv vgEvjnaLjFEB apache/nifi:latest ● Licensed under the ASF License ● Unsupported ● NiFi 1.25 and NiFi 2.0.0-M2 https://hub.docker.com/r/apache/nifi
- 18. https://medium.com/@tspann/unstructured-data-processing-with-a-raspberry-pi-ai-kit-c959dd7fff47 Raspberry Pi AI Kit Hailo Edge AI
- 19. https://medium.com/@tspann/from-the-edge-to-the-cloud-and-back-again-01095e95a783 Raspberry Pi AI Kit Hailo Edge AI Pose Estimation