IoT Architecture - Are Traditional Architectures Good Enough or do we Need New Approaches?
5 likes1,994 views
The document discusses the Internet of Things (IoT) and its architectural considerations, including the integration of devices, data analysis, and enterprise infrastructure adaptation. It outlines key challenges and solutions associated with building IoT applications, such as connectivity, data processing, and analytics, while emphasizing the need for appropriate architecture based on specific use cases. Additionally, it highlights the significance of advanced analytics and real-time processing in handling the influx of data generated by IoT devices.
IoT Architecture - Are Traditional Architectures Good Enough or do we Need New Approaches?
- 1. BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH IoT Architecture Guido Schmutz @gschmutz guidoschmutz.wordpress.com Are Traditional Architectures Good Enough or do we Need New Approaches?
- 2. Guido Schmutz Working for Trivadis for more than 19 years Oracle ACE Director for Fusion Middleware and SOA Consultant, Trainer, Software Architect for Java, SOA & Big Data / Fast Data Member of Trivadis Architecture Board Technology Manager @ Trivadis More than 25 years of software development experience Contact: guido.schmutz@trivadis.com Blog: http://guidoschmutz.wordpress.com Slideshare: http://www.slideshare.net/gschmutz Twitter: gschmutz
- 3. Agenda 1. Introduction 2. Towards an IoT Architecture 3. Summary
- 4. Introduction
- 5. Internet of Things (IoT) Wave Internet of Things (IoT): Enabling communication between devices, people & processes to exchange useful information & knowledge that create value for humans Term was first proposed by Kevin Ashton in 1999 Source: The Economist Source: Ericsson, June 2016
- 6. Reasons why IoT opportunity is occurring now ? Affordable hardware • Costs of actuators & sensors have been cut in half over last 10 years Smaller, more powerful hardware • Form factors of hardware have shrunk to millimeter or even nanometer levels Ubiquitous & cheap mobility • Cost for mobile devices, bandwidth and data processing has declined over last 10 years Availability of supporting tools • Big data tools & cloud based infrastructure have become widely available Mass market awareness • IoT has surpassed a critical tipping point • Vision of a connected world has reached such a followership that companies have initiated IoT developments • Commitment is irreversible
- 8. Towards an IoT Architecture
- 9. Key Challenges for building an IoT application 1. Connect: How to collect data from intelligent devices? • Abstract complexity associated with device connectivity • Standardize integration of devices with enterprise 2. Analyze: How to analyze IoT data? • Reduce noise and detect business event at real-time • Enable historical big-data analysis 3. Integrate: How to integrate IoT data & events with enterprise infrastructure? • Make enterprise processes IoT friendly • Allow enterprise & mobile applications to control devices
- 10. Today) Existing Service-/API Architecture as a base 10 Mobile Apps DB Rich (Web) Client Apps API Gateway Enterprise Service Bus (ESB) / Data Integration Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB REST / SOAP REST / SOAP REST / SOAP SOAP Various SQL SOAP REST Service BusOracle Data Integrator API Gateway SOA Suite BPM Suite Business Activity Monitoring = one way = request/response DB
- 11. REST / SOAP REST / SOAP IoT 1a) Reuse exiting Service-/API-based Architecture IoT Smart Devices 11 Mobile Apps D B Rich (Web) Client Apps D B Enterprise Service Bus (ESB) / Data Integration Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB REST / SOAP REST REST JMS / REST SOAP Various SQL SOAP REST WebSocket JMS Service BusOracle Data Integrator API Gateway API Gateway JMS JMS WeblogicJMS SOA Suite BPM Suite Business Activity Monitoring = one way = request/response
- 12. IoT 1a) Challenges • Are IoT devices powerful enough to communicate over the internet (HTTP protocol) ? • Device only collects data (sense) or is there also the way back necessary (actuator) ? • Can JMS be used from external devices (firewalls allow traffic over JMS) ? • How many IoT devices are planned short and long term ? • How frequently do they produce data ? • Is JMS server and ESB capable for dealing with resulting message volume ? • What operations are needed on IoT data (messages / events) ‘ • only simple transformations, filter and routing operations (VETRO pattern) ?
- 13. REST / SOAP REST / SOAP IoT 1b) Reuse existing Service-/API-based Architecture IoT Smart Devices 13 Mobile Apps D B Rich (Web) Client Apps D B API Gateway Enterprise Service Bus (ESB) / Data Integration Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB REST HTTP REST REST HTTP JMS API Gateway Service BusOracle Data Integrator SOAP Various SQL SOAP REST WebSocket JMS JMS JMS REST / SOAP WeblogicJMS SOA Suite BPM Suite Business Activity Monitoring = one way = request/response
- 14. REST / SOAP REST / SOAP IoT 2) Adding Event Hub and optional IoT Gateway 14 Mobile Apps D B Rich (Web) Client Apps D B ESB / Data Integration IoT Devices IoT Gateways IoT Smart Devices Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB REST Kafka / MQTT / REST REST Kafka / MQTT / REST MQTT Kura Event Hub Kafka Service Bus Oracle Data Integrator REST REST SOAP Various SQL SOAP REST WebSocket Kafka JMS JMS API Gateway API Gateway REST Kafka REST / SOAP REST REST SOA Suite BPM Suite Business Activity Monitoring MQTT MQTT = one way = request/response
- 15. How to implement an Event Hub? Apache Kafka to the rescue • Distributed publish-subscribe messaging system • Designed for processing of high-volume, real time activity stream data • Stateless (passive) architecture, offset-based consumption • Provides Topics, but does not implement JMS standard • Initially developed at LinkedIn, now part of Apache • Peak Load on single cluster: 2 million messages/sec, 4.7 Gigabits/sec inbound, 15 Gigabits/sec outbound
- 16. Oracle’s Service Bus as a consumer of Kafka Service Bus 12c Cloud Apps Business Service Cloud Proxy Service Kafka Cloud API Mobile Apps Pipeline Routing Kafka Sensor / IoT Web Apps Business Service REST Business Service WSDL Backend Apps REST Backend Apps WSDL Proxy Service Kafka Pipeline Routing Database DB CDC Stream Processing
- 17. IoT 2) Solutions & Challenges Solutions • Event Hub solves the potential scalability issue of JMS • Event Hub can interoperate / integrate with existing SOA platform (ESB) • IoT Gateway makes sure that lightweight sensors can connect to the internet / send their data Challenges • Where to do complex analytics on the events? Is it scalable? • Can we really send all data down to backend? Network bandwidth?
- 18. REST / SOAP REST / SOAP IoT 3) Adding Stream Processing / Analytics in Backend 18 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices IoT Gateways IoT Smart Devices Event Hub Stream Processing ESP / CEP DB DB Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB Stream Analytics Service Bus Oracle Data IntegratorKafka MQTT Kafka Kafka SOAP Various SQL SOAP REST WebSocket Kafka JMS JMSAPI GatewayAPI Gateway REST MQTT REST REST Kafka Kafka / MQTT / REST Kafka / MQTT / REST REST Kafka REST / SOAP REST REST Kura SOA Suite BPM Suite Business Activity Monitoring = one way = request/response KafkaMQTT
- 19. IoT 3) Solutions & Challenges Solutions • Stream Processing handles complex analytics on events in a scalable manner before sending events to ESB / backend systems Challenges • Can we really send all data down to backend? Network bandwidth?
- 20. Oracle’s Stream Analytics as consumer of Kafka/MQTT Oracle Stream Analytics Stream Analytics Kafka Kafka Mobile Apps Kafka Sensor / IoT Web Apps Machine Data DB CDC Kafka MQTT
- 21. REST / SOAP REST / SOAP IoT 4) Adding Industry 4.0 Data Sources (machine data) 21 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices IoT Gateways IoT Smart Devices Event Hub Stream Processing ESP / CEP DB DB Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB Stream Analytics Service Bus Oracle Data IntegratorKafka MQTT Kafka Kafka SOAP Various SQL SOAP REST WebSocket Kafka JMS JMSAPI GatewayAPI Gateway REST MQTT REST REST Kafka Kafka / MQTT / REST Kafka / MQTT / REST REST Kafka REST / SOAP REST REST Kura SOA Suite BPM Suite Business Activity Monitoring = one way = request/response I 4.0 Machine DB CDC GoldenGate Kafka / MQTT / REST KafkaMQTT
- 22. GoldenGate Gateway Oracle’s GoldenGate for Change Data Capture of existing database Machine Data DB Kafka Oracle GoldenGate Delivery Capture Pump JMS Machine Data DB Oracle GoldenGate Delivery Capture Pump HBase HDFS
- 23. REST / SOAP REST / SOAP IoT 5) Adding Stream Processing / Analytics at Edge 23 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices IoT Gateways IoT Smart Devices Event Hub Stream Processing ESP / CEP DB DB Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB Stream Analytics Service Bus Oracle Data IntegratorKafka Kafka Kafka SOAP Various SQL SOAP REST WebSocket Kafka JMS JMSAPI GatewayAPI Gateway Kafka / MQTT / REST REST REST Kafka Kafka REST RESTREST REST / SOAP REST MQTT SOA Suite BPM Suite Business Activity Monitoring ESP/CEP Edge Analytics MQTT Kafka / MQTT / REST MQTT = one way = request/response I 4.0 Machine DB CDC GoldenGate Kafka / MQTT / REST Kafka
- 24. IoT 5) Solutions & Challenges Solutions • Stream Processing at the edge / gateway allows to reduce the amount of messages send to the backend (cloud / on premises) if necessary Challenges • How to handle integration between the components on the IoT side (fan in)
- 25. REST / SOAP REST / SOAP IoT 6) Adding Data Flow Engines 25 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices IoT Gateways IoT Smart Devices Event Hub Stream Processing ESP / CEP DB DB Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB Stream Analytics Service Bus Oracle Data IntegratorKafka Kafka Kafka SOAP Various SQL SOAP REST WebSocket Kafka JMS JMSAPI GatewayAPI Gateway Kafka / MQTT / REST REST REST Kafka Kafka REST RESTREST REST / SOAP REST MQTT SOA Suite BPM Suite Business Activity Monitoring ESP/CEP Edge Analytics MQTT Kafka / MQTT / REST MQTT = one way = request/response I 4.0 Machine DB CDC GoldenGate Kafka / MQTT / REST Kafka NiFiNiFi
- 26. Continuous Ingestion / Fan-In from the Edge DB Source Big Data Log Stream Processing IoT Sensor Event Hub Topic Topic REST Topic IoT GW CDC GW Connect CDC DB Source Log CDC Native IoT Sensor IoT Sensor 26 Dataflow GW Topic Topic Queue Message GW Topic Dataflow GW Dataflow TopicREST 26 File Source Log Log Log Social Native Topic Topic
- 27. Challenges for Ingesting Sensor Data • Multitude of sensors • Multiple Firmware versions • Bad Data from damaged sensors • Data Quality
- 28. IoT 6) Solutions & Challenges Solutions • Data Flow engines allow for passing information onwards, potentially over multiple hops (fan-in) Challenges • What if we want to keep the raw data? Where to persist it?
- 29. REST / SOAP REST / SOAP IoT 7) Adding Raw Data Storage and Batch Analytics 29 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices IoT Gateways IoT Smart Devices Event Hub Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB Service Bus Oracle Data IntegratorKafka Kafka Kafka SOAP Various SQL SOAP REST WebSocket JMS JMSAPI GatewayAPI Gateway REST REST Kafka Kafka SQL REST REST REST / SOAP Stream Processing ESP/CEP DB DB Big Data Processing HDFS Batch Processing DB Kafka Kafka HDFS ESP/CEP Edge Analytics MQTT MQTT Stream Analytics Hadoop / Spark Oracle Big Data Appliance SOA Suite BPM Suite Business Activity Monitoring Kafka / MQTT / REST Kafka / MQTT / REST = one way = request/response I 4.0 Machine DB CDC GoldenGate MQTT Kafka / MQTT / REST Kafka NiFiNiFi
- 30. IoT 7) Solutions & Challenges Solutions • Adding Big Data platform allows to store all raw data in a distributed file system in a scalable and reliable manner Challenges • How can we leverage the Big Data platform for more than just storing raw data? • How does it integrate with the stream processing?
- 31. REST / SOAP REST / SOAP IoT 7a) Adding Data Mining / Machine Learning and Model execution 31 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices IoT Gateways IoT Smart Devices Event Hub Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB Service Bus Oracle Data IntegratorKafka Kafka Kafka SOAP Various SQL SOAP REST WebSocket JMS JMSAPI GatewayAPI Gateway REST REST Kafka Kafka SQL REST REST REST / SOAP Stream Processing ESP/CEP DB DB Big Data Processing HDFS Batch Processing DB Kafka Kafka HDFS ESP/CEP Edge Analytics MQTT MQTT Stream Analytics Hadoop / Spark Oracle Big Data Appliance SOA Suite BPM Suite Business Activity Monitoring Kafka / MQTT / REST Kafka / MQTT / REST = one way = request/response I 4.0 Machine DB CDC GoldenGate MQTT Kafka / MQTT / REST Kafka NiFiNiFi
- 32. REST / SOAP REST / SOAP Oracle Cloud Services and Internet of Things / Big Data 32 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices Gateway IoT Smart Devices Event Hub Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB = one way = request/response Kafka Kafka SOAP Various SQL SOAP REST WebSocket JMS JMSAPI Gateway Kafka / REST REST REST Kafka Kafka SQL REST REST REST / SOAP Stream Processing ESP/CEP DB DB Big Data Processing HDFS Batch Processin g DB Kafka Kafka HDFS MQTT Hadoop / Spark Oracle IoT Cloud Service Oracle Big Data Cloud Service Oracle BI Cloud Service Oracle Integration Cloud Service Oracle Messaging Cloud Service Oracle Process Cloud Service Oracle SOA Cloud Service Oracle Mobile Cloud Service ESP/CEP Edge Analytics Oracle IoT CS Gateway Oracle IoT CS Client Library = one way = request/response
- 33. Summary
- 34. Summary Treat events as events! Infrastructures for handling lots of events are available! IoT tends to make Big Data / Fast Data infrastructures necessary Know your use case/requirements to choose the right architecture! • Can my existing backend landscape handle the new IoT load? • Do I have to handle huge amount of events in “real-time”? • Do I need to filter/aggregate data before invoking existing backend systems? • Do I want to do Advanced Analytics (predictive analytics) where historical information is necessary? • What is the network bandwidth between device/gateway and cloud/backend? • Centralized or Decentralized IoT solution?
- 35. J
- 36. Q & A Guido Schmutz Technology Manager guido.schmutz@trivadis.com
- 37. IoT Services IoT Logical Reference Architecture IoT Device Sensor Actuator IoT Gateway Storage UIApp Streaming Analytics Enterprise Applications BPM and SOA PlatformStreaming Analytics Storage Endpoint Management Event Hub Service Bus Event Hub Event Hub Service Bus Big Data / BI Storage Services Processes UIApp Storage Bulk Analytics UI Bulk Analytics UI Storage Streaming Analytics Service Bus API REST SOAP HTTP KAFKA MQTT CoAP XMPP DDS AMQP KAFKA WIFI BLE ZigBee WIFI Wired Internet of Things (IoT) and Big Data37
- 38. IoT Services IoT Logical Reference Architecture – Oracle on premises IoT Device Sensor Actuator IoT Gateway Storage UIApp Streaming Analytics Enterprise Applications BPM and SOA PlatformStreaming Analytics Storage Endpoint Management Event Hub Service Bus Event Hub Event Hub Service Bus Big Data / BI Storage Services Processes UIApp Storage Bulk Analytics UI Bulk Analytics UI Storage Streaming Analytics Service Bus API REST SOAP HTTP KAFKA MQTT CoAP XMPP DDS AMQP KAFKA WIFI BLE ZigBee WIFI Wired Edge Analytics Business Activity Monitoring SOA Suite BPM Suite Service Bus Oracle Data Integrator Stream Analytics Big Data Appliance Stream Analytics Service Bus API Gateway Internet of Things (IoT) and Big Data38 Oracle IoT CS Gateway Oracle IoT CS Client Library
- 39. IoT Services IoT Logical Reference Architecture – Oracle Cloud Services IoT Device Sensor Actuator IoT Gateway Storage UIApp Streaming Analytics Enterprise Applications BPM and SOA PlatformStreaming Analytics Storage Endpoint Management Event Hub Service Bus Event Hub Event Hub Service Bus Big Data / BI Storage Services Processes UIApp Storage Bulk Analytics UI Bulk Analytics UI Storage Streaming Analytics Service Bus API REST SOAP HTTP KAFKA MQTT CoAP XMPP DDS AMQP KAFKA WIFI BLE ZigBee WIFI Wired Edge Analytics Oracle BI CS Oracle Big Data CS Oracle SOA CS Oracle Integration CS Oracle IoT CS Oracle Streaming Analytics CS Oracle Messaging CS Oracle Big Data Discovery CS Oracle Mobile CS Internet of Things (IoT) and Big Data39 Oracle IoT CS Gateway Oracle IoT CS Client Library Oracle Process CS Oracle DataFlow ML CS Big Data Preparation CS Application Container CS Container CS
- 40. IoT Services IoT Logical Reference Architecture – Microsoft Azure IoT Device Sensor Actuator IoT Gateway Storage UIApp Streaming Analytics Enterprise Applications BPM and SOA PlatformStreaming Analytics Storage Endpoint Management Event Hub Service Bus Event Hub Event Hub Service Bus Big Data / BI Storage Services Processes UIApp Storage Bulk Analytics UI Bulk Analytics UI Storage Streaming Analytics Service Bus API REST SOAP HTTP KAFKA MQTT CoAP XMPP DDS AMQP KAFKA WIFI BLE ZigBee WIFI Wired Azure IoTHub Azure Streaming Analytics Azure Service Bus Azure Power BI HD Insight Azure Storage Azure IoT Gateway SDK Azure Event Hub Azure Storage Azure Machine Learning Internet of Things (IoT) and Big Data40
- 41. IoT Services IoT Device Sensor Actuator IoT Gateway Storage UIApp Streaming Analytics Enterprise Applications BPM and SOA PlatformStreaming Analytics Storage Endpoint Management Event Hub Service Bus Event Hub Event Hub Service Bus Big Data / BI Storage Services Processes UIApp Storage Bulk Analytics UI Bulk Analytics UI Storage Streaming Analytics Service Bus API REST SOAP HTTP KAFKA MQTT CoAP XMPP DDS AMQP KAFKA WIFI BLE ZigBee WIFI Wired IoT Logical Architecture – Open Source Internet of Things (IoT) and Big Data41