Integrating Sparkplug IoT Edge of Network Nodes with Kafka with Yves Kurz
0 likes713 views
The document discusses the integration of Sparkplug IoT EON nodes with Kafka, highlighting an example from Paul Tech AG, which implements adaptive hydraulic balancing in drinking water systems. It outlines the architecture, optimization benefits achieved through AI, the technical details of MQTT and Sparkplug protocols, and the process of connecting IoT devices with Kafka using specific connectors. Additionally, it provides best practices for architecture selection, message format, and testing strategies.
![Sparkplug
MQTT Topic Namespace
16.05.23 15
Message Format
l spAv1.0: JSON
l spBv1.0: Protobuf
namespace/group_id/message_type/edge_node_id/[device_id]
Type of Message
l NBIRTH: Sent by edge node when it gets online.
l NDEATH: Sent by broker (LWT) when an edge node goes
offline.
l DBIRTH, DDEATH: Similar to NBIRTH, NDEATH but sent by
edge node when a connected device gets online/offline
l NDATA: Sent by edge node when the state of the edge node
changes.
l DDATA: Sent by edge node when the state of a device
changes
l NCMD: Contains a command to an edge node
l DCMD: Contains a command to a device
Group ID allows dividing the
namespace into arbitrary
groups. E.g by region.
Unique identifier of an edge
node
Unique (within edge node)
identifier of a device](https://image.slidesharecdn.com/bs13-20230517-ksl23-kurzyves-230602114942-91644155/85/Integrating-Sparkplug-IoT-Edge-of-Network-Nodes-with-Kafka-with-Yves-Kurz-14-320.jpg)
![Connecting IoT with Kafka
MQTT Source Connector – With Custom Partitioner
16.05.23 19
l We have a requirement that message ordering is guaranteed per Edge Node
l Default Partitioner is hashing message key to determine the partition
l Custom partitioner guarantee that all messages from one Edge Node are store in the same partition
namespace/group_id/message_type/edge_node_id/[device_id]
Partition number: #(full_key) MOD num_partition
namespace/group_id/message_type/edge_node_id/[device_id]
Partition number: #(edge_node_id) MOD num_partition](https://image.slidesharecdn.com/bs13-20230517-ksl23-kurzyves-230602114942-91644155/85/Integrating-Sparkplug-IoT-Edge-of-Network-Nodes-with-Kafka-with-Yves-Kurz-18-320.jpg)
Integrating Sparkplug IoT Edge of Network Nodes with Kafka with Yves Kurz
- 1. Integrating Sparkplug IoT EoN Nodes with Kafka - A Real-world Example Yves Kurz – PAUL Tech AG
- 2. Outline 16.05.23 3 What does PAUL do? 3 IoT Architecture @ PAUL 9 MQTT Primer 12 Introduction to Sparkplug 13 Connecting IoT and Kafka 15 Do‘s and Don‘ts 24
- 3. 4 In 2023 PAUL will save ~ 21‘500 Tons of CO₂! To compensate that amount of CO₂ it would need an area of 10’000 sqm of forest or… … ~80% the size of Airport Heathrow! … ~3x the size of Central Park! … ~4x the size of Disneyland California!
- 4. What does PAUL do? PAUL implements the adaptive hydraulic balancing in drinking water systems and heating. 16.05.23 5
- 5. PAUL Smart Valve PAUL Gateway PAUL Temperature Sensor PAUL Energy Valve What does PAUL do? PAUL implements the adaptive hydraulic balancing in drinking water systems and heating. 16.05.23 6
- 6. The optimization through PAUL's AI and its advantages AI technology analyzes the current state of the entire system and optimizes it Consumption Behaviour Vacancy Weather Sun side orientation Day time / Season Combines billions of data points from other houses. PAUL AI ADVANTAGES Up to 40% energy savings! Optimization of ESG ratings! More attractive interest rates for refinancing! Sizing of new heating systems based on consumption data instead of assumptions! Perfect preparation for the use of alternative heat generators! 16.05.23 7
- 7. PAUL Tech AG in Numbers 16.05.23 8 4 Offices: Mannheim, Nürnberg, Jacksonville, Bern >130 Employees 2017 Year of Establishment >150‘000 Residential Units >500‘000‘000 Data Points https://www.paul.tech/en/career
- 8. PAUL’s Legacy Architecture 2017-2022 16.05.23 9 l Traditional Industrial Automation Architecture l Expensive l Does not scale l Hard to maintain Input Output PLC SCADA/Supervisory Reporting AWS Field Level
- 9. New Architecture 2022- 16.05.23 10 AWS Field Level Control Center & Dashboarding Services Kafka IoT Services Kafka Connect MQTT Broker l Concept and architecture validation from Oct ‘21 – March’22 l Implementation May ’22- l In productive use since Dec ‘22 l No PLC and SCADA system l Router is Sparkplug EoN node l From Polling to Report by Exception
- 10. Covered in this Session 16.05.23 11 AWS Field Level Control Center & Dashboarding Services Kafka IoT Services Kafka Connect MQTT Broker l MQTT - Transport Protocol between Edge Nodes l Sparkplug – MQTT Topic and Payload Definition l Kafka Connect – Bridging MQTT and Kafka
- 11. MQTT Primer 16.05.23 12 l Message Queuing Telemetry Transport l Lightweight open protocol l Used in (I)IoT in a constrained environment l Low Bandwidth, High Latency l Low Power Equipment l Publish-Subscriber Model with Central Broker Topics & Subscriptions l Topics names form a hierarchical tree l Messages published on a topic l Subscriptions on a specific topic or with wildcards to a topic sub-tree l MQTT5: Shared Subscription Brocker Client 1 Client 2 Client 3 Messages l Quality of Service – Message Delivery guarantees l QoS 0: At most once l QoS 1: At least once l QoS 2: Exactly once l Retained Messages l LWT – Last Will Telegram ✉ → m y/t1 my/t1 → ✉ my/t2 ← ✉ ✉ ← my/t1 ✉ ← my/t2 https://mqtt.org
- 12. Sparkplug MQTT + Sparkplug = 'Plug & Play' IIoT 16.05.23 13 l Specification originated by Cirrus Link Solutions now maintained by the Eclipse Foundation l Designed to allow interoperability between IIoT devices l Allows realtime communication between devices in the system l Ensures devices and applications are aware of state and configuration Extends MQTT l Defines a Topic Namespace l Specifies Message payload content and serialization formats l Sparkplug A: JSON l Sparkplug B: Protobuf l Defines device state management l Broker failover https://sparkplug.eclipse.org
- 14. Sparkplug MQTT Topic Namespace 16.05.23 15 Message Format l spAv1.0: JSON l spBv1.0: Protobuf namespace/group_id/message_type/edge_node_id/[device_id] Type of Message l NBIRTH: Sent by edge node when it gets online. l NDEATH: Sent by broker (LWT) when an edge node goes offline. l DBIRTH, DDEATH: Similar to NBIRTH, NDEATH but sent by edge node when a connected device gets online/offline l NDATA: Sent by edge node when the state of the edge node changes. l DDATA: Sent by edge node when the state of a device changes l NCMD: Contains a command to an edge node l DCMD: Contains a command to a device Group ID allows dividing the namespace into arbitrary groups. E.g by region. Unique identifier of an edge node Unique (within edge node) identifier of a device
- 15. Sparkplug Payload content 16.05.23 16 Payload Fields l Timestamp l Array of Metrics l Name l Data type l Value l Timestamp l Optional: l Alias l Historical, Transient, Null l Properties (property set) l Quality Code,… l Metadata (seq, filename, filetype md5sum, description …) l Optional l UUID l Body (byte data)
- 16. Connecting IoT with Kafka 16.05.23 17 MQTT Broker Topics to which Edge Nodes publish Topics to which Edge Nodes subscribe Kafka iot-mqtt-source iot-mqtt-sink MQTT Source Connector Kafka Connect MQTT Sink Connector IoT Services
- 17. Connecting IoT with Kafka MQTT Source Connector 16.05.23 18 l We use Confluent’s MQTT Source Connector l Copy messages from topics written by Edge Nodes to Kafka iot-mqtt-source topic l /spBv1.0/+/NBIRTH/# l /spBv1.0/+/DBIRTH/# l /spBv1.0/+/NDEATH/# l /spBv1.0/+/DDEATH/# l /spBv1.0/+/NDATA/# l /spBv1.0/+/DDATA/# l Don’t use Clean Session, to not loose messages when connector temporarely goes offline l Kafka Message l Key: MQTT Topic name to which payload was published l Data: MQTT Raw message (Protobuf, Sparkplug b) https://docs.confluent.io/kafka-connectors/mqtt/current/mqtt-source-connector l What about the ordering of messages in Kafkas partitions?
- 18. Connecting IoT with Kafka MQTT Source Connector – With Custom Partitioner 16.05.23 19 l We have a requirement that message ordering is guaranteed per Edge Node l Default Partitioner is hashing message key to determine the partition l Custom partitioner guarantee that all messages from one Edge Node are store in the same partition namespace/group_id/message_type/edge_node_id/[device_id] Partition number: #(full_key) MOD num_partition namespace/group_id/message_type/edge_node_id/[device_id] Partition number: #(edge_node_id) MOD num_partition
- 19. Custom SMT 16.05.23 20 No easy way to access the # partitions of destination Kafka topic within an SMT
- 20. Better Solution - Custom Partitioner 16.05.23 21 In a partitioner we have access to the a Cluster object, this gives us access to the # of partitions
- 21. Connecting IoT with Kafka MQTT Sink Connector 16.05.23 22 l We use Confluent’s MQTT Sink Connector l Source Topic: iot-mqtt-sink l Kafka Message l Key: MQTT Topic name to which message shall be published l Data: MQTT Raw message (Protobuf, Sparkplug b) l By default the connector publishes into the MQTT topic that has the same name as the Kafka topic. l Change the behaviour by using ExtractTopic SMT provided by Confluent, to publish to the correct MQTT topic https://docs.confluent.io/kafka-connectors/mqtt/current/mqtt-sink-connector/
- 22. Do’s & Don’t’s 16.05.23 23 l Select the right architecture depending on your needs. Fat vs Thin Edge Devices. Pushing Kafka to the Edge may not make sense. l When working with Sparkplug, start with JSON, when the system is more mature, switch to Protobuf l Use good tools. For MQTT use MQTT Explorer, and build from source to get Protobuf deserializer support. l Write defensive code in SMTs/Partitioners l Test your system, esp. for performance, and outages of brokers and connectors. l Don’t invent your own IoT protocol l Don’t use SMTs to do the work of a partitioner l Try avoiding using custom SMTs and/or partitioners if you plan to use cloud hosted Kafka Connect
- 24. Thank You ©2023. ALL RIGHTS RESERVED. Principal Engineer l Email: yves.kurz@paul.tech l Web: https://paul.tech/ l Twitter: @yveskurz l Sources: https://github.com/yveskurz/kafka-summit-2023- sparkplug-kafka Yves Kurz