Using FLiP with influxdb for EdgeAI IoT at Scale
1 like2,279 views
The document discusses the integration of Apache Flink with InfluxDB and Apache Pulsar for scalable IoT applications. It highlights StreamNative's offerings, including their cloud-native event streaming platform that enables real-time data streaming and processing. Various Apache projects and connectors are mentioned to support efficient data handling and analytics in edge AI environments.
Using FLiP with influxdb for EdgeAI IoT at Scale
- 2. Using FLaNK with InfluxDB for EdgeAI IoT at Scale Tim Spann | Developer Advocate
- 3. streamnative.io Tim Spann, Developer Advocate DZone Zone Leader and Big Data MVB Data DJay
- 4. streamnative.io Founded by the original developers of Apache Pulsar and Apache BookKeeper, StreamNative builds a cloud-native event streaming platform that enables enterprises to easily access data as real-time event streams.
- 5. streamnative.io Apache is an open source, cloud-native distributed messaging and streaming platform.
- 6. streamnative.io Powered by Apache Pulsar, StreamNative provides a cloud-native, real-time messaging and streaming platform to support multi-cloud and hybrid cloud strategies. Built for Containers Cloud Native Flink SQL StreamNative Cloud
- 7. © 2021 InfluxData Inc. All Rights Reserved. ● Apache Flink ● Apache Pulsar ● StreamNative's Flink Connector for Pulsar ● Apache NiFi ● Apache +++ FLiP(N) Stack
- 8. © 2021 InfluxData Inc. All Rights Reserved. Apache Pulsar InfluxDB Sink https:/ /pulsar.apache.org/docs/en/io-influxdb-sink/
- 9. © 2021 InfluxData Inc. All Rights Reserved. Apache Pulsar InfluxDB Sink bin/pulsar-admin sink stop --name influxdb-sink-jetson --namespace default --tenant public bin/pulsar-admin sinks delete --tenant public --namespace default --name influxdb-sink-jetson bin/pulsar-admin sinks create --archive ./connectors/pulsar-io-influxdb-2.8.0.nar --tenant public --namespace default --name influxdb-sink-jetson --sink-config-file conf/influxcloud.yml --inputs jetsoninflux --parallelism 1 bin/pulsar-admin sinks get --tenant public --namespace default --name influxdb-sink-jetson bin/pulsar-admin sinks status --tenant public --namespace default --name influxdb-sink-jetson https:/ /pulsar.apache.org/docs/en/io-influxdb-sink/
- 10. © 2021 InfluxData Inc. All Rights Reserved. InfluxDB Telegraf MQTT https:/ /www.influxdata.com/integration/mqtt-monitoring/ https:/ /github.com/streamnative/mop MQTT on Pulsar (MoP)
- 11. © 2021 InfluxData Inc. All Rights Reserved. InfluxDB Telegraf RabbitMQ https:/ /www.influxdata.com/integration/mqtt-monitoring/ https:/ /github.com/streamnative/aop AMQP on Pulsar (AoP) https:/ /hub.streamnative.io/connectors/amqp-1-0-sink/ https:/ /hub.streamnative.io/connectors/amqp-1-0-source
- 12. streamnative.io • Guaranteed delivery • Data buffering - Backpressure - Pressure release • Prioritized queuing • Flow specific QoS - Latency vs. throughput - Loss tolerance • Data provenance • Supports push and pull models • Hundreds of processors • Visual command and control • Over a 300 sources • Flow templates • Pluggable/multi-role security • Designed for extension • Clustering • Version Control Why Apache NiFi?
- 13. © 2021 InfluxData Inc. All Rights Reserved. InfluxDB via Apache NiFi https:/ /www.influxdata.com/integration/mqtt-monitoring/ https:/ /github.com/streamnative/mop https://github.com/influxdata/nifi-influxdb-bundle https://www.influxdata.com/blog/building-a-data-stream-for-iot-with-nifi-and-influxdb/
- 14. streamnative.io ● Unified computing engine ● Batch processing is a special case of stream processing ● Stateful processing ● Massive Scalability ● Flink SQL for queries, inserts against Pulsar Topics ● Streaming Analytics ● Continuous SQL ● Continuous ETL ● Complex Event Processing ● Standard SQL Powered by Apache Calcite Why Apache Flink?
- 16. StreamNative Hub StreamNative Cloud Unified Batch and Stream COMPUTING Batch (Batch + Stream) Unified Batch and Stream STORAGE Offload (Queuing + Streaming) Apache Flink - Apache Pulsar - Apache NiFi <-> Devices <-> influxdb Tiered Storage Pulsar --- KoP --- MoP --- Websocket --- HTTP Pulsar Sink Pulsar Sink Streaming Edge Gateway Protocols End-to-End Streaming Edge App
- 18. NVIDIA Device
- 19. Apache NiFi
- 20. Apache NiFi Data to Enhance IoT Stream
- 21. Apache NiFi
- 22. Apache NiFi Influxdb properties Controller properties
- 23. Apache NiFi
- 24. Apache NiFi
- 25. Apache NiFi
- 26. Apache Pulsar
- 43. ● https://github.com/tspannhw/FLiP-InfluxDB ● https://pulsar.apache.org/docs/en/io-influxdb-sink/ ● https://pulsar.apache.org/docs/en/io-overview/ ● https://github.com/tspannhw/Flip-solr ● https://github.com/tspannhw/Flip-transit ● https://github.com/tspannhw/Flip-iot ● https://www.datainmotion.dev/2020/10/flank-strea ming-edgeai-on-new-nvidia.html ● https://github.com/tspannhw/minifi-jetson-nano ● https://github.com/tspannhw/SpeakerProfile/ ● https://hub.streamnative.io/connectors/influxdb-sin k/2.5.1/ ● https://github.com/tspannhw/minifi-xaviernx/
- 44. Let’s Keep in Touch! Tim Spann Developer Advocate @PaasDev https://www.linkedin.com/in/timothyspann https://github.com/tspannhw
- 45. Questions?
- 46. Thank You