Codeless pipelines with pulsar and flink
1 like692 views
This document summarizes Tim Spann's presentation on codeless pipelines with Apache Pulsar and Apache Flink. The presentation discusses how StreamNative's platform uses Pulsar and Flink to enable end-to-end streaming data pipelines without code. It provides an overview of Pulsar's capabilities for messaging, stream processing, and integration with other Apache projects like Kafka, NiFi and Flink. Examples are given of ingesting IoT data into Pulsar and running real-time analytics on the data using Flink SQL.
![streamnative.io
Python IoT Program Sending Data to MQTT and Pulsar
row['gasKO'] = str(readings)
json_string = json.dumps(row)
json_string = json_string.strip()
client.connect("192.168.1.181", 1883, 180)
client.publish("persistent://public/default/mqtt-2", payload=json_string, qos=0, retain=True)
producer = KafkaProducer(bootstrap_servers='192.168.1.181:9092',retries=3)
producer.send('rp4-kafka-1', json.dumps(row).encode('utf-8'))
We need to install MQTT, Kafka and Pulsar libraries. You can choose your protocol to communicate with the messaging cluster.
pip3 install paho-mqtt
pip3 install kafka-python
pip3 install pulsar-client](https://image.slidesharecdn.com/codelesspipelineswithpulsarandflink-210920163203/85/Codeless-pipelines-with-pulsar-and-flink-8-320.jpg)
Codeless pipelines with pulsar and flink
- 1. streamnative.io Codeless Pipelines with Pulsar and Flink Timothy Spann Developer Advocate
- 2. StreamNative Solution Application Messaging Data Pipelines Real-time Contextual Analytics Tiered Storage APP Layer Computing Layer Storage Layer StreamNative Platform IaaS Layer Micro Service NotiïŹcation Dashboard Risk Control Auditing Payment ETL
- 3. streamnative.io Speaker Bio DZone Zone Leader and Big Data MVB @PaasDev https://github.com/tspannhw https://www.datainmotion.dev/ https://github.com/tspannhw/SpeakerProfile https://dev.to/tspannhw https://sessionize.com/tspann/ https://www.slideshare.net/bunkertor Developer Advocate
- 4. streamnative.io FLaNK and FLiP Stacks â Apache Flink â Apache NiFi â Apache Kafka â Apache Flink â Apache Pulsar â StreamNative's Flink Connector for Pulsar â Apache +++ Apache projects are the way for all streaming use cases.
- 5. streamnative.io Todayâs Data. IoT JSON {"uuid": "rpi4_uuid_vml_20210902151842", "amplitude100": 1.1, "amplitude500": 0.5, "amplitude1000": 0.4, "lownoise": 0.5, "midnoise": 0.2, "highnoise": 0.2, "amps": 0.3, "ipaddress": "192.168.1.244", "host": "rp4", "host_name": "rp4", "macaddress": "3e:f6:33:38:9e:d3", "systemtime": "2021-09-02T11:18:43.765279", "endtime": "1630595923.76", "runtime": "49.71", "starttime": "09/02/2021 11:17:52", "cpu": 0.0, "cpu_temp": "34.0", "diskusage": "37206.6 MB", "memory": 5.5, "id": "20210902151842_b1126bd3-f6e1-4eb1-84d0-bd8557a560cf", "temperature": "22.1", "adjtemp": "16.8", "adjtempf": "42.2", "temperaturef": "51.8", "pressure": 1008.0, "humidity": 37.8, "lux": 156.5, "proximity": 0, "oxidising": 12.4, "reducing": 147.3, "nh3": 25.3, "gasKO": "Oxidising: 12444.44 OhmsnReducing: 147300.33 OhmsnNH3: 25266.49 Ohms"}
- 6. streamnative.io End to End Streaming Codeless Pipeline Enterprise sources Sensors Errors Aggregates Alerts IoT ETL Analytics Streaming SQL Clickstream Market data Machine logs Social
- 7. streamnative.io All Data - Anytime - Anywhere - Multi-Cloud - Multi-Protocol Multi- inges t Multi- inges t Multi-ingest Merge Priority
- 8. streamnative.io Python IoT Program Sending Data to MQTT and Pulsar row['gasKO'] = str(readings) json_string = json.dumps(row) json_string = json_string.strip() client.connect("192.168.1.181", 1883, 180) client.publish("persistent://public/default/mqtt-2", payload=json_string, qos=0, retain=True) producer = KafkaProducer(bootstrap_servers='192.168.1.181:9092',retries=3) producer.send('rp4-kafka-1', json.dumps(row).encode('utf-8')) We need to install MQTT, Kafka and Pulsar libraries. You can choose your protocol to communicate with the messaging cluster. pip3 install paho-mqtt pip3 install kafka-python pip3 install pulsar-client
- 9. streamnative.io Running This All Yourself See: https://github.com/tspannhw/FLiP-SQL https://github.com/tspannhw/FLiP-IoT Run Apache Pulsar Standalone - locally, docker or native cloud (https://console.streamnative.cloud/) Run Apache Flink Standalone - locally, docker or native cloud (https://console.streamnative.cloud/) Run Apache NiFi Single - locally, docker or cloud I run MQTT on Pulsar (MoP) to allow Pulsar to use MQTT protocol (https://github.com/streamnative/mop/releases/tag/v2.8.0.10) I run Kafka on Pulsar (KoP) to allow Pulsar to use Kafka protocol (https://github.com/streamnative/kop)
- 10. streamnative.io Apache NiFi Consuming From Pulsar Cluster via Multiple Protocols
- 11. streamnative.io Apache Pulsar is Cloud-Native Messaging and Event-Streaming Platform
- 12. streamnative.io Apache Pulsar Overview Enable Geo-Replicated Messaging â Pub-Sub â Geo-Replication â Pulsar Functions â Horizontal Scalability â Multi-tenancy â Tiered Persistent Storage â Pulsar Connectors â REST API â CLI â Many clients available â Four Different Subscription Types â Multi-Protocol Support â MQTT â AMQP â JMS â Kafka â ...
- 13. streamnative.io What are the BeneïŹts of Pulsar? Data Durability Scalability Geo-Replication Multi-Tenancy Unified Messaging Model
- 14. streamnative.io Upcoming - Flink + Pulsar (FLiP) https://flink.apache.org/2019/05/03/pulsar-flink.html https://github.com/streamnative/pulsar-flink https://streamnative.io/en/blog/release/2021-04-20-flink-sql- on-streamnative-cloud
- 15. streamnative.io Apache Flink Apache Flink is a distributed stream processing system. It is capable of providing high throughput, near real-time processing of streams from Pulsar. It is ideal for ambitious Stream Processing compared to Pulsarâs model of lightweight Stream Processing. End-to-end exactly-once stream processing https://streamnative.io/en/blog/release/2021-06-14-exactly-once-semantics-with-transactions-in-pulsar
- 16. streamnative.io Connect with the Community & Stay Up-To-Date â Join the Pulsar Slack channel - Apache-Pulsar.slack.com â Follow @streamnativeio and @apache_pulsar on Twitter â Subscribe to Monthly Pulsar Newsletter for major news, events, project updates, and resources in the Pulsar community
- 17. streamnative.io â https://github.com/tspannhw/FLiP-SQL â https://github.com/tspannhw/StreamingSQLExamples â https://github.com/streamnative/pulsar-flink â https://www.linkedin.com/pulse/2021-schedule-tim-spann/ â https://github.com/tspannhw/SpeakerProfile/blob/main/2021/talks/20210729_HailHydrate!FromStream toLake_TimSpann.pdf â https://streamnative.io/en/blog/release/2021-04-20-flink-sql-on-streamnative-cloud â https://docs.streamnative.io/cloud/stable/compute/flink-sql Deeper Content @PaasDev https://www.pulsardeveloper.com/ timothyspann
- 18. streamnative.io Pulsar Summit Europe October 6, 2021 Pulsar Summit Asia November 20-21, 2021 Contact us at partners@pulsar-summit.org to become a sponsor or partner
- 19. streamnative.io