5º MeetUP ARQconf 2016 - IoT: What is it really and how does it work?
2 likes1,300 views
The document outlines the Internet of Things (IoT) architecture, including its components such as sensors, actuators, and cloud infrastructure, which enable real-time data processing and contextually aware decision-making. It discusses various technologies and challenges related to scalability, interoperability, security, and analytics in IoT systems. Additionally, it presents proposed solutions and platform architectures for building IoT applications, emphasizing the need for adaptability in a rapidly evolving ecosystem.
5º MeetUP ARQconf 2016 - IoT: What is it really and how does it work?
- 1. ©2014 GlobalLogic Inc. All Rights Reserved. Juan Caracoche juan.caracoche@globallogic.com @jcaracoche
- 2. 2 ©2015 GlobalLogic Inc. All Rights Reserved. IoT: What is it really and how does it work?
- 3. 3 ©2015 GlobalLogic Inc. All Rights Reserved. Similar quote attributed in various forms to Nobel prize-winning physicists Albert Einstein and Richard Feynman “If you can’t explain something to an intelligent twelve year old, you probably don’t understand it yourself.”
- 4. 4 ©2015 GlobalLogic Inc. All Rights Reserved. Lego Mindstorm Robot courtesy of TM & © 2015 Marvel & Subs.
- 5. 5 ©2015 GlobalLogic Inc. All Rights Reserved. + “Brain” + Sensors + Actuators
- 6. 6 ©2015 GlobalLogic Inc. All Rights Reserved. “Brain” Cloud
- 7. 7 ©2015 GlobalLogic Inc. All Rights Reserved. + Sensors “Brain” Cloud
- 8. 8 ©2015 GlobalLogic Inc. All Rights Reserved. + Sensors “Brain” Cloud + Actuators
- 9. 9 IoT Architecture Paradigm ©2015 GlobalLogic Inc. All Rights Reserved. Immediate Actions (Alerts, Actuators, Notifications) End User and Management Clients Complex Actions (Workflow) Resources Platform Applications External Federated Data Context Historical Data Deep Insights Fast Decision Making Observations and Requests Connected Devices and Systems Action Triggers ActuatorsSensors Sensor s Brain Actuators
- 10. 10 IoT Architecture Paradigm ©2015 GlobalLogic Inc. All Rights Reserved. Immediate Actions (Alerts, Actuators, Notifications) End User and Management Clients Observations and Requests Sensors External Federated Data ConnectedDevices and Systems Action Triggers Complex Actions (Workflow) Actuators Resources Platform Applications Context Historical Data Deep Insights Fast Decision Making
- 11. 11 The Internet of Things ©2015 GlobalLogic Inc. All Rights Reserved. A set of technologies that connects data from a variety of sensors with the ability to make contextually aware decisions and act on them in near real-time
- 12. 12 IoT Architecture Paradigm: Sensor Technologies ©2015 GlobalLogic Inc. All Rights Reserved. Resources Sensors Sensor s • Alljoyn • MQTT • HomeNET (Apple) • Thread (Google) • EMIT (Panasonic) • Electric Imp • CCNx (PARC) • OIC / IIC (Intel) Sensors
- 13. 13 Speed-layer Technologies: • Spark Streaming • Storm • Samza • IBM Streams • Esper • Sidhi CEP • AWS Kinesis (Cloud based) IoT Architecture Paradigm: Analytics Technologies ©2015 GlobalLogic Inc. All Rights Reserved. External Federated Data Context Historical Data Deep Insights Fast Decision Making Brain Batch-layer Technologies: • Hadoop Map-Reduce (Cloudera, Hortonworks, MapR ) • HPCC • Spark • Hive • Spark SQL • Pig • Analytic Databases (MarkLogic, Vertica, Infobright) Fast Decision Making Deep Insights
- 14. 14 • Impala • VoltDB • ElephantDB • SploutSQL • Hive on Spark • Hbase+Phoenix • Couchbase • MongoDB IoT Architecture Paradigm: Analytics Technologies ©2015 GlobalLogic Inc. All Rights Reserved. External Federated Data Context Historical Data Deep Insights Fast Decision Making Brain • Oracle Data Federation • Redhat Data Virtualization • Mule ESB + Custom code • EspressoLogic View Layer Data Federation
- 15. 15 IoT Architecture Paradigm: Applications Technologies ©2015 GlobalLogic Inc. All Rights Reserved. Immediate Actions (Alerts, Actuators, Notifications) End User and Management Clients Complex Actions (Workflow) Applications Connected Devices and Systems Actuators Actuators Technologies: • MBaaS (Mobile Backend as a service • mPaaS / aPaaS systems • Workflow and Rules systems • Pub / Sub • Mobile app frameworks • JS / Browser apps • Sensors / “M2M” Technologies listed Actuators and Applications
- 16. 16 Internet of Things Architectural Challenges And Design Considerations
- 17. 17 CONFIDENTIAL Architectural Challenges Internet of Things • Scalability - Billions of devices and ZBs of data • Interoperability - Heterogeneous systems and distributed resources • Extensibility - New things and new opportunities • Resiliency - Network partition and system errors • Performance - Near real-time processing and high processing • Security - Privacy and Deceit • Maintenance - Resource and Platform
- 18. 18 CONFIDENTIAL Design Considerations Internet of Things • Identity Management - Identity of things and identity of people • Context awareness - Location and identity • Data generation mode - Event driven and time driven • Resource capability - Processing power and memory size • Forward and backward capability
- 19. 19 CONFIDENTIAL Reference Architecture and Platform Internet of Things External Federated Data z z z M2M, Devices, and Intelligent Gateways Fast Decision Making Historical Information Notifications Deep Insights Complex Actions Platform Functions External Federated Data End User and Management Clients
- 20. ©2015 GlobalLogic Inc. CONFIDENTIAL Embedded systems in IoT
- 21. CONFIDENTIAL21 Internet of Things • Secure communication capabilities • Interoperability means talking to each other • Sensing/interacting embedded technology IoT = network of physical objects
- 22. 22 Internet of Things Real world interaction is provided by sensors and actuators • Light, pressure, gyro, movement, temperature, etc. sensors • MEMS, motors, valves, relays, etc. • Machine vision, human interaction and other systems Communication capabilities are provided by protocols CoAP Well-defined and secure interoperability means: • AAA, provisioning and network management • Inter-system capabilities negotiations and configuration • Cloud/edge/fog optimal work distribution • Security, security and also security
- 23. 23 Internet of Things • SSL/TLS based security (certificates) • Flexible transport plugins – AllJoyn or MQTT or REST or … • Transport-independent initial provisioning mechanism • Capabilities negotiation between all endpoints based on GUIDs • “Edge” and “Fog” computing concepts support Security Transport AllJoyn MQTT REST … Provisioning Capabilities negotiation IFTTT rules & computing Data flow Vision of embedded systems in IoT
- 24. 24 CONFIDENTIAL Definitions Internet of Things • Sensor/Actuator - Pressure sensors, camera, stepper motor, etc. • Controller -Collect, process and possibly store data, implement some custom IFTTT or other logic -Can have a set of Sensors or Actuators connected to it -Can talk to other Controllers or M2M Gateway • M2M Gateway - Provides Internet access to one or some set of Controllers - Can be smart enough to implement some custom IFTTT or other logic
- 25. 25 Device Gateway PlatformController M2M Gateway Request certificate signing M2M transport connection Sign GW certificate with Private Key Create private key by serial Create platform certificate Return signed certificateCreate private key by serial Request certificate signing Sign CTL certificate with Private Key Return signed certificate Controller-Gateway certificate exchange (Transport authentication mechanism) Device registration process Gateway and Controller are authorized by the Platform, securedconnectionis established Provisioning
- 26. 26 Device Gateway Platform Controller M2M Gateway Authentication Check certificate authorize data processing actions Connection → announce DeviceInfo + certificate Device Authorized,setResourceID getData events setData events Message loop Announce capabilities Update Device Info AAA
- 27. ©2015 GlobalLogic Inc. CONFIDENTIAL Platform Core
- 28. 28 CONFIDENTIAL Reference Architecture and Platform Internet of Things External Federated Data z z z M2M, Devices, and Intelligent Gateways Fast Decision Making Historical Information Notifications Deep Insights Complex Actions Platform Functions External Federated Data End User and Management Clients
- 29. 29 CONFIDENTIAL Platform Core | Cloud Internet of Things Heterogeneous Data Large Volume Near Real-Time Processing Security Notifications
- 30. 30 Heterogeneous Data Solutions Internet of Things • Transform any format of data to ONE format • Apply basic rules • Cache & Aggregate data • Platform Gateways
- 31. 31 Large Volume Solutions Internet of Things • Fast Hundreds of MBs of RW per second, serialize/deserialize • Scalable Elastically and transparently expanded without downtime • Durable persisted and replicated to prevent data loss • Distributed fault-tolerance guarantees
- 32. 32 Near Real-Time Processing Internet of Things • Low Latency persisted and replicated to prevent data loss • Scale High throughput and scale-out by adding instances • Dynamic Change the rules dynamically using API interface • Complex detecting relationships between events
- 33. 33 Security Solutions Internet of Things • Privacy Multi tenancy and data sharing • Security Policies apply security policies on the data before storing • In Memory Process non-persistable data in memory • Transport Secure data transfer to the DMZ
- 34. ©2015 GlobalLogic Inc. CONFIDENTIAL IoT Analytics
- 35. 35 CONFIDENTIAL Reference Architecture and Platform Internet of Things External Federated Data z z z M2M, Devices, and Intelligent Gateways Fast Decision Making Historical Information Notifications Deep Insights Complex Actions Platform Functions External Federated Data End User and Management Clients
- 36. 36 Internet of Things The Internet of Things Analytics(IoTA) , the measurement and transformation into business intelligence of the Internet of Things Data Paradigms Volume application and transactional data Velocity Machine Variety Social Data
- 37. 37 CONFIDENTIAL IoT Analytics Challenges Internet of Things • Data Integration – Aggregation and integration of the collected data streams in a manner that makes them suitable for analysis. – Key challenge in dealing with more diverse and disparate data sources before meaningful analysis is viable. • Data Storage – Implementation and management of the data store for the analytics process. – Key issue to address in this context is how to store time-series sensor data, which can increase dramatically in volume compared to e.g., transactional data readings • Different components of IOT analytics
- 38. 38 CONFIDENTIAL IoT Analytics Challenges (Cont’d) Internet of Things • Core Analytics – Processing of the data by an analytics engine and the subsequent delivery of insights. • Data Presentation – Further presentation of the delivered analytical insights to the end-user. – In IoT analytics, the geography of data is a particularly important presentation element, given that the location of physical things matters more to analysis than, say, the location of an ecommerce transaction. • Different components of IOT analytics
- 39. 39 CONFIDENTIAL Enabling IOT Analytics using Lambda Architecture Internet of Things Components of Lambda Architecture new data batch layer serving layer speed layer Query Query master data set batch view batch view real time view real time view
- 40. 40 Technology Stack Internet of Things Component Technology Choices Rationale Batch View Generator (Batch Layer) Hive on Spark , Impala •Queries, especially those involving multiple reducer stages, runs faster. • Can anytime move to using only Hive by just a JDBC connection parameters change. • Easy of use than Map Reduce and other processing framework. • Support SQL by default. Real Time View Generator (Speed Layer) Volt DB , Spark Streaming ,Storm •Processes data fast by keepingit in memory and runs on commodity hardware. •Data can be ingested directly from Kafka queues so would be on low latency as thereis no intermediate layer betweendataingestion. Consolidate View Generator (Serving Layer) Hbase + Phoenix , Couchbase •Is easy to use and provides real time read and write. •Good for time series based evaluation. • Phoenix is used to provide SQL support over hbase Analytics Engine(Statistical programming language) R , Mahout •Gives wide variety of statistical and graphical algorithm. •It has linear and non linear modeling, time series analysis, classification,clustering etc. Real time Machine Learning (Machine Learning Library) SparkML •Distributed machine learning library. • Has support for common algorithms. Data Ingestion Layer (Messaging System) Kafka , RabbitMQ •High Throughput , data canbe pulledat consumer's own pace. •Can replay any messages or set of messages given the necessary selection criteria.
- 41. 41 IOT Analytics Industry Trends Internet of Things Informatica Vibe Data Stream for Machine Data ParStream IOT Analytic Framework Source- http://www.informatica.com/in/products/big-data/vibe-data-stream/, https://www.parstream.com/
- 42. 42 IOT Analytics Industry Trends Internet of Things Source- http://voltdb.com/
- 43. ©2015 GlobalLogic Inc. CONFIDENTIAL Applications in IoT
- 44. 44 CONFIDENTIAL GlobalLogic Reference Architecture and Platform Internet of Things External Federated Data z z z M2M, Devices, and Intelligent Gateways Fast Decision Making Historical Information Notifications Deep Insights Complex Actions Platform Functions External Federated Data End User and Management Clients
- 45. 45 CONFIDENTIAL “There’s an app for that” – Apple’s advertising tagline, circa 2011 With the brain layer disappearing and becoming part of the background, how do we build apps for IoT?
- 46. 46 CONFIDENTIAL What are the Challenges? Internet of Things The Past The Present The Near Future The Future Concept from Jeffery S. Engelhardt Courtesy Sony Courtesy Nokia / Issam Trabelsi Courtesy bushlemon.deviantart.com A rapidly changing ecosystem
- 47. 47 CONFIDENTIAL What are the Challenges? Internet of Things Very high fluctuation in usage
- 48. 48 CONFIDENTIAL What are the Challenges? Internet of Things Ability to handle all types of use cases Control home appliances (Connected Home) Gather meteorological data from sensors for weather forecasting Communicate to an aero engine manufacturer that this engine needs servicing
- 49. 49 CONFIDENTIAL What are the Challenges? Internet of Things A completely new UX paradigm How do we do UX for agriculture? Will apps on TVs require a new UX? How will appliances work in a vehicle?
- 50. 50 What are the Challenges? The Future is an Unknown Internet of Things 2020 2010 2000 1990 1980 1970 1960 1950 1944 1 10 1000s 100s of 1000s 1 Million 100s of Millions 1 Billion 8 Billion 50 Billion One Ten Thousands Hundreds of Thousands One Million Hundreds of Millions Billion 8 Billion 50 Billion
- 51. 51 CONFIDENTIAL What are the Challenges? Internet of Things Being comfortable with legacy
- 52. 52 CONFIDENTIAL What are the Challenges? Internet of Things Lack of a default standard What will become the default standard is just a guess for now
- 53. 53 CONFIDENTIAL Summary Internet of Things With challenges of this kind, it is best to build IoT applications on a platform So: how do we do it, and what would an application platform look like?
- 54. 54 CONFIDENTIAL Plenty of Platforms Already Exist Internet of Things Xively There are plenty more being developed . . . And all of them do ingestion and analytics ThingWorx Mnubo Bug Labs
- 55. 55 App Platform Feature Set Internet of Things Challenges Solutions • A rapidly changing ecosystem • Extreme difference in consumption based on success • Ability to handle all types of use cases • A completely new UX paradigm • The future is an unknown • Being comfortable with legacy • A lack of standardization • Use of Auto-update like features (similar to Google’s Chrome • Have a plug n play approach to building applications • Use of cloud and cloud patterns to handle varying levels of consumption and to make this scalable and fault tolerant and highly available • Have the ability to Integrate with all existing systems via an ESB • Ability to incorporate new devices displays in a seamless manner by separating out the rendering out of the view from the data and the view • Build once, Reuse everywhere, Continuously update • Follow the Rapid Application Delivery (RAD model) With the right Platform as your partner and with the right vendor, go to market would be reduced by months
- 56. 56 CONFIDENTIAL Proposed Architecture of the Application Platform Internet of Things Views AppEngineSDK AppRenderer WebApps (MV*) Native App Custom AppDesktops / Laptops / Tablets Smart Phones Devices & Gateways Business Logic AppPlatformAPI Worker VM (Task A) Worker VM (Task B) Worker VM (Task N) Orchestration Manager Integration ESB Integration layer (3rd Party Adapters) Auto Scale Manager PlatformAccessLayer App DB Platform Tier Data Services Streaming Service Platform Config Services Analytics Services Platform Management …
- 57. 57 CONFIDENTIAL Proposed Architecture of the Application Platform Internet of Things Views AppEngineSDK AppRenderer WebApps (MV*) Native App Custom AppDesktops / Laptops / Tablets Smart Phones Devices & Gateways Business Logic AppPlatformAPI Worker VM (Task A) Worker VM (Task B) Worker VM (Task N) Orchestration Manager Integration ESB Integration layer (3rd Party Adapters) Auto Scale Manager PlatformAccessLayer App DB Platform Tier Data Services Streaming Service Platform Config Services Analytics Services Platform Management … Custom built once for each stack Custom built once for each stack Vert.x / Node.js / Web API Polyglot DB Polyglot Programming Custom / Drools / WWF Mule / Neuron / Fuse / etc. Vert.x / Node.js / Web API
- 58. 58 CONFIDENTIAL Reference Architecture and Platform Internet of Things External Federated Data z z z M2M, Devices, and Intelligent Gateways Fast Decision Making Historical Information Notifications Deep Insights Complex Actions Platform Functions External Federated Data End User and Management Clients
- 59. ©2014 GlobalLogic Inc. All Rights Reserved.