Apache StreamPipes – Flexible Industrial IoT Management
0 likes1,537 views
The document discusses Apache StreamPipes, an open-source industrial IoT toolbox designed for non-technical users to easily connect, analyze, and manage IoT data streams. It outlines the components of StreamPipes, including its architecture, use cases like continuous asset monitoring, and its functionality such as data processing and visualization. Additionally, it provides insights into the roadmap for contributions and advanced features like Python integration and edge deployment options.
![streampipes.apache.org/ | @streampipes | github.com/apache/incubator-streampipes
Anatomy of a data processor
Numerical Filter
API Data Processor
Description(s)
Runtime
Processing Function(s)
Transport Layer Transport Layer
User Configuration Output Strategy Stream requirements
Event Schema
(data type, ...)
Text Input,
Selections, Slider, ...
Keep, Append,
Custom, ...
Event Grounding
Supported Format,
Supported Protocol
Numerical Filter
[Json,Thrift],
[Kafka, MQTT]numerickeep
threshold: 100,
operator: “>“,
property: “temp“
Function Thread](https://image.slidesharecdn.com/20200929apcnapwiv7-final-201009174358/85/Apache-StreamPipes-Flexible-Industrial-IoT-Management-22-320.jpg)
![Pipeline Elements Microservices
Runtime Options
* [STREAMPIPES-174] New Python Wrapper
streampipes.apache.org/ | @streampipes | github.com/apache/incubator-streampipes
Standalone
(Java, Siddhi, Python*)
Distributed
(Flink, Spark, Kafka Streams)
Runtime Wrapper](https://image.slidesharecdn.com/20200929apcnapwiv7-final-201009174358/85/Apache-StreamPipes-Flexible-Industrial-IoT-Management-27-320.jpg)
Apache StreamPipes – Flexible Industrial IoT Management
- 1. Apache StreamPipes – Flexible Industrial IoT Management Patrick Wiener | PPMC Apache StreamPipes IoT Track | Sep 29, 2020
- 2. Who am I and how did I get here? § Researcher by day @FZI (Germany), Open Source enthusiast by night § ACNA19 talk @dominikjriemer, entered Apache incubator w/ StreamPipes in November 2019 § PPMC member of Apache StreamPipes § GIF-lover § Twitter: @patrickwiener, @streampipes Stream Processing IoTDistributed Systems this is me flamingo Philipp Dominik streampipes.apache.org/ | @streampipes | github.com/apache/incubator-streampipes
- 3. Agenda Industrial IoT in the Factory 4.01 StreamPipes 1012 Roadmap & How to Contribute5 Technical Deepdive3 Demo Extending StreamPipes4 SDK
- 4. Industrial IoT in the Factory 4.0 1
- 5. 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 0 1 0 0 1 0 0 1 1 0 1 0 0 1 0 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 0 1 0 0 1 0 0 1 1 0 1 0 0 1 0 1 0 1 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 1 0 0 1 1 0 1 0 0 1 0 1 0 1 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 1 0 0 1 1 0 1 0 0 1 0 1 0 1 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 0 1 1 0 1 0 0 1 0 1 0 1 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 0 1 1 0 1 0 0 1 0 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 0 1 0 0 1 0 0 1 1 0 1 0 0 1 0 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 0 1 0 0 1 0 0 1 1 0 1 0 0 1 0 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 0 1 0 0 1 0 0 1 1 0 1 0 0 1 0 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 0 1 0 0 1 0 0 1 1 0 1 0 0 1 0 IIoT in the Factory 4.0 Data streams everywhere machine data gripping force camera images sensor data environmental data gripping force ROS ROS MQTT Modbus PLC REST
- 6. 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 0 1 0 0 1 0 0 1 1 0 1 0 0 1 0 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 0 1 0 0 1 0 0 1 1 0 1 0 0 1 0 1 0 1 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 1 0 0 1 1 0 1 0 0 1 0 1 0 1 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 1 0 0 1 1 0 1 0 0 1 0 1 0 1 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 0 1 1 0 1 0 0 1 0 1 0 1 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 0 1 1 0 1 0 0 1 0 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 0 1 0 0 1 0 0 1 1 0 1 0 0 1 0 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 0 1 0 0 1 0 0 1 1 0 1 0 0 1 0 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 0 1 0 0 1 0 0 1 1 0 1 0 0 1 0 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 0 1 0 0 1 0 0 1 1 0 1 0 0 1 0 IIoT in the Factory 4.0 Collecting them is NOT easy but there are great universal protocols machine data gripping force camera images sensor data environmental data gripping force PLC ROS ROS MQTT Modbus REST
- 7. 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 0 1 0 0 1 0 0 1 1 0 1 0 0 1 0 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 0 1 0 0 1 0 0 1 1 0 1 0 0 1 0 1 0 1 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 1 0 0 1 1 0 1 0 0 1 0 1 0 1 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 1 0 0 1 1 0 1 0 0 1 0 1 0 1 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 0 1 1 0 1 0 0 1 0 1 0 1 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 0 1 1 0 1 0 0 1 0 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 0 1 0 0 1 0 0 1 1 0 1 0 0 1 0 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 0 1 0 0 1 0 0 1 1 0 1 0 0 1 0 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 0 1 0 0 1 0 0 1 1 0 1 0 0 1 0 1 1 0 1 1 0 1 0 1 1 1 0 1 1 0 0 1 0 0 1 0 0 1 1 0 1 0 0 1 0 IIoT in the Factory 4.0 StreamPipes sits on top as an industrial IoT toolbox for domain experts machine data gripping force camera images sensor data environmental data gripping force PLC ROS ROS MQTT Modbus REST
- 9. What is StreamPipes? streampipes.apache.org/ | @streampipes | github.com/apache/incubator-streampipes "An open source industrial IoT toolbox to enable non-technical users to connect, analyze and exploit (IoT) data streams." Connect Pipeline Editor Dashboard Data Explorer Notifications beta
- 10. How does it work? streampipes.apache.org/ | @streampipes | github.com/apache/incubator-streampipes standalone distributed Leverage reuseable data processors & sinks2 x86 arm Deploy & execute the way you like3 Realize use cases 4 Continuous Asset Monitoring Calculate live-KPI Wrap your ML models and use them on sensor/image data ... Connect IoT sources1 OPC UA csv > 30 Adapters > 100 Pipeline Elements
- 11. Training Data Collection Pre-processing of input streams to build up training data repository IIoT in the Factory 4.0 Typical StreamPipes use cases streampipes.apache.org/ | @streampipes | github.com/apache/incubator-streampipes Continuous Asset Monitoring Dashboard to provide live insights of current state KPI Analytics Live overview of relevant production or asset KPI Process / Product Quality Inspection Detect time-critical situations (CEP vs. ML)
- 12. Pipeline Editor: Types of Pipeline Elements streampipes.apache.org/ | @streampipes | github.com/apache/incubator-streampipes Data stream. Ordered sequence of events provided by a StreamPipes Connect adapter instance connecting to an industrial event source (e.g. UR Robot via ROS). Data Streams Data Streams Data Processors Data Sinks + + Pipeline DAG =
- 13. streampipes.apache.org/ | @streampipes | github.com/apache/incubator-streampipes Data ProcessorsData Streams Data processors. Applies a configurable function on one or more input event streams (transform, enrich, filter, etc) and produces an output event stream. Data Streams Data Processors Data Sinks + + Pipeline DAG = Pipeline Editor: Types of Pipeline Elements
- 14. streampipes.apache.org/ | @streampipes | github.com/apache/incubator-streampipes Data SinksData Streams Data Processors Data Streams Data Processors Data Sinks + + Pipeline DAG = Data sinks. Marks the end of a pipeline and either forwards an input event stream to § internal sinks (Dashboard, Datalake) or § third-party sink (Kafka, IoTDB, ROS, etc) Pipeline Editor: Types of Pipeline Elements
- 15. StreamPipes IIoT toolbox complementing the Pipeline Editor streampipes.apache.org/ | @streampipes | github.com/apache/incubator-streampipes Connect Notifications Data Explorer Dashboard beta
- 16. Demo Source: https://bit.ly/3fGaqke Look ma, (almost) no hands!
- 18. StreamPipes Microservice Architecture Transport Layer Data sources Connect Adapters web UI streampipes.apache.org/ | @streampipes | github.com/apache/incubator-streampipes Core More than a web app Pipeline Element Microservices
- 19. Filter temp > 100 Temp Sensor Dashboard MQTT streampipes.apache.org/ | @streampipes | github.com/apache/incubator-streampipes industrial event source 100 011 010 001 010 010 100 101 000 111 { "tstamp": 1453478160, "machineId": "ID5", "temperature": 73.5 } Datalake Let‘s look a basic Pipeline It all starts with an event (stream)
- 20. API Connect Worker Description Runtime Adapter Library Transport LayerMQTT Adapter MQTT streampipes.apache.org/ | @streampipes | github.com/apache/incubator-streampipes industrial event source 100 011 010 001 010 010 100 101 000 111 { "tstamp": 1453478160, "machineId": "ID5", "temperature": 73.5 } Anatomy of a connect worker MQTT Adapter Event Quality User ConfigurationEvent Schema Data type, runtime name, semantic type Frequency, latency, measurement unit Host, Port, Topic double C° 192.168.0.10, 1883, my-topic Adapter Thread
- 21. API Connect Worker Description Runtime Adapter Library streampipes.apache.org/ | @streampipes | github.com/apache/incubator-streampipes Anatomy of a connect worker MQTT Adapter Transport LayerMQTT Adapter Connect Masterinvoke runtime UI MQTT industrial event source 100 011 010 001 010 010 100 101 000 111 { "tstamp": 1453478160, "machineId": "ID5", "temperature": 73.5 } start adapter http://adapter-url:8099/mqtt host: 192.168.0.10 port: 1883, topic: my-topicpayload user config protocol: kafka format: json host: kafka:9092 outputtopic: <topic> Event grounding Adapter Thread
- 22. streampipes.apache.org/ | @streampipes | github.com/apache/incubator-streampipes Anatomy of a data processor Numerical Filter API Data Processor Description(s) Runtime Processing Function(s) Transport Layer Transport Layer User Configuration Output Strategy Stream requirements Event Schema (data type, ...) Text Input, Selections, Slider, ... Keep, Append, Custom, ... Event Grounding Supported Format, Supported Protocol Numerical Filter [Json,Thrift], [Kafka, MQTT]numerickeep threshold: 100, operator: “>“, property: “temp“ Function Thread
- 23. streampipes.apache.org/ | @streampipes | github.com/apache/incubator-streampipes Anatomy of a data processor Numerical Filter Pipeline Management API Data Processor Description(s) Runtime Processing Function(s) Transport Layer Transport Layer Numerical Filter UI http://svc-url:8090/numericalfilter value: 100 operator: “>“ property: “temp“payload protocol: kafka format: json host: kafka:9092 inputtopic: <topic> outputtopic: <topic> provided user config Event grounding start pipeline ✓ ✓ invoke runtime Function Thread
- 24. streampipes.apache.org/ | @streampipes | github.com/apache/incubator-streampipes Pipeline Management API Data Sink Description(s) Runtime Sink Function(s) UI Datalake Websocket InfluxDB Anatomy of a data sink Dashboard & Datalake sink Dashboard Transport Layer http://svc-url:8090/dashboard http://svc-url:8090/datalake payload invoke runtime Sink Thread 1 Sink Thread 2
- 25. streampipes.apache.org/ | @streampipes | github.com/apache/incubator-streampipes Why do we need such a description layer? User guidance, pipeline element matching & verification Supporting and guiding users through pipeline modeling process Preventing non-compatible elements to be connected early on Pre-selecting suitable event fields in user configuration dialog Supporting and guiding users Pre-selecting suitable event fields Preventing non-compatible elements data stream contains only numeric event fields requires at least one boolean event field
- 27. Pipeline Elements Microservices Runtime Options * [STREAMPIPES-174] New Python Wrapper streampipes.apache.org/ | @streampipes | github.com/apache/incubator-streampipes Standalone (Java, Siddhi, Python*) Distributed (Flink, Spark, Kafka Streams) Runtime Wrapper
- 28. streampipes.apache.org/ | @streampipes | github.com/apache/incubator-streampipes MyGreeter data processor Input requirements, user-defined parameters & output strategy Input Params Output Transport Layer Params
- 29. Write your application logic in a few lines streampipes.apache.org/ | @streampipes | github.com/apache/incubator-streampipes Transport LayerTransport Layer MyGreeter data processor
- 30. streampipes.apache.org/ | @streampipes | github.com/apache/incubator-streampipes https://github.com/wipatrick/apachecon-demo-processor
- 31. SDK Source: https://bit.ly/3fGaqke Write your own data processor
- 32. Roadmap & How to Contribute 5
- 33. streampipes.apache.org/ | @streampipes | github.com/apache/incubator-streampipes Python Runtime Wrapper Improve SDK to develop pipeline elements using Python (data science audience) Edge Deployment Advanced deployment options to assign pipeline elements to individual nodes (e.g., edge/cloud nodes) StreamPipes Client Define/create pipelines from code, which are automatically deployed in StreamPipes StreamPipes Platform Services API to interact with live/historical data, pipelines and pipeline elements Resilience/Fault Tolerance Better failure handling/recovery of state Pipeline Monitoring/Statistics Inspect current pipeline execution state, receive statistics (e.g., processed messages) and see errors. Feature roadmap
- 34. Feature roadmap streampipes.apache.org/ | @streampipes | github.com/apache/incubator-streampipes Python Runtime Wrapper Edge Deployment StreamPipes Client StreamPipes Platform Services Resilience/Fault Tolerance Pipeline Monitoring/Statistics Want to get involved? § Check our issues in Jira to get started § See StreamPipes Improvement Proposals in Confluence § Subscribe to mailing lists and slack channel and get involved in the discussions § Help us grow the community § Looking to contribute? § Core/UI features § Extensions (adapters, processors, sinks) § Docs/Blog posts
- 35. Don’t miss the next talk Analyzing IIoT data with PLC4X and StreamPipes Philipp Zehnder | PPMC Apache StreamPipesChristofer Dutz | PMC Apache PLC4X streampipes.apache.org/ | @streampipes | github.com/apache/incubator-streampipes right after this talk + Toddy
- 36. Let‘s connect! … and leave a star on Github if you like it J streampipes.apache.org streampipes.apache.org/docs apache/incubator-streampipes @streampipes