[AI Dev World 2022] Build ML Enhanced Event Streaming
3 likes209 views
The document discusses building machine learning enhanced event streaming applications using Java microservices, particularly focusing on Apache Pulsar, Flink, and NiFi. It highlights the capabilities of Apache Pulsar, including its community support, message storage architecture, and lightweight stream processing functions. Additionally, it outlines the usage of Apache NiFi for data flow management and the training resources available for developers and engineers working with Pulsar.
![[AI Dev World 2022] Build ML Enhanced Event Streaming](https://image.slidesharecdn.com/aidevworld2022buildmlenhancedeventstreaming-221102182253-edfc66ae/85/AI-Dev-World-2022-Build-ML-Enhanced-Event-Streaming-8-320.jpg)
![[AI Dev World 2022] Build ML Enhanced Event Streaming](https://image.slidesharecdn.com/aidevworld2022buildmlenhancedeventstreaming-221102182253-edfc66ae/85/AI-Dev-World-2022-Build-ML-Enhanced-Event-Streaming-9-320.jpg)
![[AI Dev World 2022] Build ML Enhanced Event Streaming](https://image.slidesharecdn.com/aidevworld2022buildmlenhancedeventstreaming-221102182253-edfc66ae/85/AI-Dev-World-2022-Build-ML-Enhanced-Event-Streaming-17-320.jpg)
[AI Dev World 2022] Build ML Enhanced Event Streaming
- 1. BUILD ML ENHANCED EVENT STREAMING APPLICATIONS WITH JAVA MICROSERVICES Tim Spann | Developer Advocate
- 2. ● Introduction ● What is Apache Pulsar? ● Function ● Apache NiFi ● Continuous ETL Flink ● Demo ● Q&A
- 3. Tim Spann Developer Advocate Tim Spann Developer Advocate at StreamNative ● FLiP(N) Stack = Flink, Pulsar and NiFi Stack ● Streaming Systems & Data Architecture Expert ● Experience: ○ 15+ years of experience with streaming technologies including Pulsar, Flink, Spark, NiFi, Big Data, Cloud, MXNet, IoT, Python and more. ○ Today, he helps to grow the Pulsar community sharing rich technical knowledge and experience at both global conferences and through individual conversations.
- 4. FLiP Stack Weekly This week in Apache Flink, Apache Pulsar, Apache NiFi, Apache Spark and open source friends. https://bit.ly/32dAJft
- 5. Apache Pulsar has a vibrant community 560+ Contributors 10,000+ Commits 7,000+ Slack Members 1,000+ Organizations Using Pulsar
- 6. ● “Bookies” ● Stores messages and cursors ● Messages are grouped in segments/ledgers ● A group of bookies form an “ensemble” to store a ledger ● “Brokers” ● Handles message routing and connections ● Stateless, but with caches ● Automatic load-balancing ● Topics are composed of multiple segments ● ● Stores metadata for both Pulsar and BookKeeper ● Service discovery Store Messages Metadata & Service Discovery Metadata & Service Discovery Pulsar Cluster Metadata Storage Pulsar Cluster
- 7. Streaming Consumer Consumer Consumer Subscription Shared Failover Consumer Consumer Subscription In case of failure in Consumer B-0 Consumer Consumer Subscription Exclusive X Consumer Consumer Key-Shared Subscription Pulsar Topic/Partition Messaging
- 10. • Functions - Lightweight Stream Processing (Java, Python, Go) • Connectors - Sources & Sinks (Cassandra, Kafka, …) • Protocol Handlers - AoP (AMQP), KoP (Kafka), MoP (MQTT) • Processing Engines - Flink, Spark, Presto/Trino via Pulsar SQL • Data Offloaders - Tiered Storage - (S3) Sources, Sinks and Processing
- 11. ● Lightweight computation similar to AWS Lambda. ● Specifically designed to use Apache Pulsar as a message bus. ● Function runtime can be located within Pulsar Broker. ● Java Functions A serverless event streaming framework Pulsar Functions
- 12. ● Consume messages from one or more Pulsar topics. ● Apply user-supplied processing logic to each message. ● Publish the results of the computation to another topic. ● Support multiple programming languages (Java, Python, Go) ● Can leverage 3rd-party libraries to support the execution of ML models on the edge. Pulsar Functions
- 13. https://www.influxdata.com/integration/mqtt-monitoring/ https://www.influxdata.com/integration/mqtt-monitoring/ • Guaranteed delivery • Data buffering - Backpressure - Pressure release • Prioritized queuing • Flow specific QoS - Latency vs. throughput - Loss tolerance • Data provenance • Supports push and pull models • Hundreds of processors • Visual command and control • Over a 300 components • Flow templates • Pluggable/multi-role security • Designed for extension • Clustering • Version Control Apache NiFi Basics
- 14. Apache NiFi - Apache Pulsar Connector
- 15. ● Unified computing engine ● Batch processing is a special case of stream processing ● Stateful processing ● Massive Scalability ● Flink SQL for queries, inserts against Pulsar Topics ● Streaming Analytics ● Continuous SQL ● Continuous ETL ● Complex Event Processing ● Standard SQL Powered by Apache Calcite Apache Flink
- 16. Apache Flink Job Dashboard
- 19. Apache Pulsar Training ● Instructor-led courses ○ Pulsar Fundamentals ○ Pulsar Developers ○ Pulsar Operations ● On-demand learning with labs ● 300+ engineers, admins and architects trained! Now Available On-Demand Pulsar Training Academy.StreamNative.io StreamNative Academy
- 20. ● https://github.com/tspannhw/pulsar-pychat-function ● https://streamnative.io/apache-nifi-connector/ ● https://nightlies.apache.org/flink/flink-docs-master/docs/conne ctors/datastream/pulsar/ ● https://streamnative.io/en/blog/release/2021-04-20-flink-sql-o n-streamnative-cloud ● https://github.com/streamnative/flink-example ● https://pulsar.apache.org/docs/en/adaptors-spark/ Apache Pulsar Links
- 21. Deploying AI With an Event-Driven Platform https://dzone.com/trendreports/enterprise-ai-1