![Traditional Table Approach
Informix TimeSeries Approach
Meter_ID Time KWH Voltage ColN
1 1-1-11 12:00 Value 1 Value 2 ……… Value N
2 1-1-11 12:00 Value 1 Value 2 ……… Value N
3 1-1-11 12:00 Value 1 Value 2 ……… Value N
… … … … ……… …
1 1-1-11 12:15 Value 1 Value 2 ……… Value N
2 1-1-11 12:15 Value 1 Value 2 ……… Value N
3 1-1-11 12:15 Value 1 Value 2 ……… Value N
… … … … ……… …
Meter_ID Series
1 [(1-1-11 12:00, value 1, value 2,…, value N), (1-1-11 12:15, value 1, value 2, …, value N), …]
2 [(1-1-11 12:00, value 1, value 2,…, value N), (1-1-11 12:15, value 1, value 2, …, value N), …]
3 [(1-1-11 12:00, value 1, value 2,…, value N), (1-1-11 12:15, value 1, value 2, …, value N), …]
4 [(1-1-11 12:00, value 1, value 2,…, value N), (1-1-11 12:15, value 1, value 2, …, value N), …]
…
Traditional Sensor data storage vs Informix TimeSeries
Storage
10](https://image.slidesharecdn.com/ibmiotarchitechtureandcapabilities-160106074247/85/IBM-Internet-of-Things-architecture-and-capabilities-10-320.jpg)
IBM Internet-of-Things architecture and capabilities
- 1. © 2015 IBM Corporation IBM Analy*cs IBM Internet of Things -‐ Architecture and Capabili*es Kevin Brown – Informix CTO, IoT Solu7ons and Services August 17, 2015
- 2. Agenda • IoT Architectural Components Overview • Informix for the Intelligent Gateway • Informix for the Cloud • IoT Partners and Platforms 2
- 4. 4 Internet of Things Topography Streams Deep Analytics Zone Device/Sensors Smart Gateways Sensor Analytics Zone 4
- 5. Simplified IoT Data Flow Sensor Data History Sensors In-memory Analytics Predictive Analytics Publish / Subscribe Cloud Infrastructure Real-time Analytics Real-time Analytics Operational Analytics Big Data Analytics (no gateway) (Gateways) HDFS / Hadoop Big Data Analytics MessageSight / MQTT Gateways for local analytics InfoSphere Streams Informix / Cloudant / IOT Foundation Service / TimeSeries Service Informix Warehouse Accelerator / DashDB PMQ / SPSS / Cognos Softlayer / BluMix Watson / DashDB / BigSQL Informix / Node-Red
- 6. • Gateways can reduce the cost of the backend cloud • Reduces cloud storage by filtering/aggregating/analyzing data locally • Reduces cloud CPU requirements by precomputing values • Reduces latency since actions can be taken immediately • Intelligent gateways can detect and respond to local events as they happen rather than waiting for transfer to the cloud • Some users are not comfortable putting all their data in the cloud • Gateways allow customers to capture and get value from their sensors without sending data to the cloud • Protocol Consolidation • Cloud does not need to implement the 100’s of IoT protocols Over time more and more of the processing will move from the cloud to gateway devices How Do Gateways Help IoT Solutions? 6
- 7. What are the Requirements for a Gateway Database? • The database management system must: ▪ Have a small install footprint, less than 100 MB ▪ Run with low memory requirements – less than 256 MB ▪ Use lossless compression or other techniques to minimize storage space ▪ Have built-in support for common types of IoT data like time series and spatial/GIS data ▪ Simple application development supporting both NoSQL and SQL ▪ Driverless, easy access to the data ▪ Require absolutely no administration ▪ Ideally should be able to network multiple gateways together to create a single distributed database 7 The database must be powerful enough to ingest, process and analyze data in real-time
- 8. IBM Informix: The Ideal Database for Gateways Simple to use ▪ Hands-Free operation – No administration ▪ Supports popular interfaces such as REST and Mongo as well as ODBC/JDBC ▪ Handles SQL and JSON data in the same database Performance ▪ One of a kind support for Time Series and Spatial data ▪ Stream data continuously into the database ▪ Run analytics as data arrives ▪ Dynamically add and update analytics when needed ▪ Storage is typically 1/3 the size compared to other vendors Invisible Agile 8 Informix is the only database management system perfectly suited to run in Gateways
- 9. Sensor Data is Time Series Data • What is a Time Series? ▪ A logically connected set of records ordered by time • What are the Key Strengths of Informix TimeSeries? ▪ Much less space required • Typically about 1/3 the space required by other vendors ▪ Queries run orders of magnitude faster • Unique optimized storage means codes paths are shorter and more data fits in memory ▪ Purpose built streaming data loader for sensor data • Automatically run analytic and/or aggregate functions on new data ▪ Can store structured (SQL) or unstructured (JSON) data for quick application development • REST/ODBC/JDBC/JSON interfaces available ▪ 100’s of functions predefined • Programming APIs available to create your own analytics 9
- 10. Traditional Table Approach Informix TimeSeries Approach Meter_ID Time KWH Voltage ColN 1 1-1-11 12:00 Value 1 Value 2 ……… Value N 2 1-1-11 12:00 Value 1 Value 2 ……… Value N 3 1-1-11 12:00 Value 1 Value 2 ……… Value N … … … … ……… … 1 1-1-11 12:15 Value 1 Value 2 ……… Value N 2 1-1-11 12:15 Value 1 Value 2 ……… Value N 3 1-1-11 12:15 Value 1 Value 2 ……… Value N … … … … ……… … Meter_ID Series 1 [(1-1-11 12:00, value 1, value 2,…, value N), (1-1-11 12:15, value 1, value 2, …, value N), …] 2 [(1-1-11 12:00, value 1, value 2,…, value N), (1-1-11 12:15, value 1, value 2, …, value N), …] 3 [(1-1-11 12:00, value 1, value 2,…, value N), (1-1-11 12:15, value 1, value 2, …, value N), …] 4 [(1-1-11 12:00, value 1, value 2,…, value N), (1-1-11 12:15, value 1, value 2, …, value N), …] … Traditional Sensor data storage vs Informix TimeSeries Storage 10
- 11. IoT Requirements for SpatioTemporal Data • Many IoT applications have a spatial component to them ▪ Vehicles, cell phones, even pets… tracking is common • In these cases both location and time is important ▪ Show me the vehicles that have passed by location X in the last hour ▪ Where has my car been over the last few hours? • Informix allows you to combine Time series and Spatial data in the same query 11
- 12. 12 Both Structured and Unstructured Data is Common in IoT Environments JSON Collection SQL Driver NoSQL Driver SQL Data Join Data • Informix can store SQL and JSON data in the same database • Write programs using SQL drivers or Mongo/NoSQL drivers • SQL data automatically transformed into JSON documents when needed • NoSQL data automatically transformed into SQL when needed Embedded Device or Database server Horizontal Scale-out with Shards
- 13. Informix Data Access Options 13 MongoDB Client REST Client SQLI Client DRDA Client Informix DBMS Informix NoSQL Listener Informix • NoSQL ↔ SQL Translation • REST, MongoDB Protocol Support • SQLI, DRDA Protocol Support • Relational, Collection, Time Series, and Spatial Data Support Spatial Time Series JSON Collection Relational Table A REST client is any program capable of making a HTTP request
- 14. Informix Data Access Options 14 MongoDB Client REST Client SQLI Client DRDA Client Informix DBMS Informix NoSQL Listener Informix • NoSQL ↔ SQL Translation • REST, MongoDB Protocol Support • SQLI, DRDA Protocol Support • Relational, Collection, Time Series, and Spatial Data Support Spatial Time Series JSON Collection Relational Table You can use all the client drivers that are available for MongoDB with the NoSQL Listener
- 15. • Rapid Development • Simple use with JSON • Simple REST • Simple MQTT and other adapters • Contributor Nodes – simple to use other services 15 Use Node-Red for Quick Gateway App. Development
- 16. Node-RED 16
- 17. IBM IoT Smart Gateway Kit 1. Login as installing user. ▪ Ex: pi 2. mkdir iot-gateway-kit-depend 3. Run git clone https://github.com/ibm-iot/iot-gateway-kit.git 4. cd iot-gateway-kit 5. Run ./iot_install 17
- 18. IBM IoT Smart Gateway Kit • The iot-gateway-kit will install the following: ▪ Node.js ▪ Node-red ▪ Timeseries nodes ▪ Bluetooth node.js application sample 18
- 19. Smart Gateway Kit – TI Sensor Tag 19 1. IoT gateway Kit Designed using node- red/node.js to work with the TI Sensor Tag" 2. Stores data in the Timeseries database" 3. Aggregate data and push to the cloud" 4. IoT Foundation or other."
- 20. IoT Developers - Get Started! • Gateway starter - https://ibm.biz/BdXr2W • Code samples - https://ibm.biz/BdX4QV • Github - https://github.com/IBM-IoT/ 20
- 21. Informix on Docker Hub https://registry.hub.docker.com/u/ibmcom/informix-innovator-c/ • IBM Informix Innovator-C • 12.10.FC4 21 Docker Hub $docker pull ibmcom/informix-innovator-c!
- 22. 22 Informix for the Cloud and Operational Zone
- 23. 23 What are the IoT Requirements for the Cloud? • Requirements - similar to gateways (but for different reasons): • Potentially 1000’s of servers means zero administration is a must • Data volume adds up very quickly so low storage overhead is required • Data flows into the cloud continuously and must be processed in real-time • Must be able to handle time series, spatial, and NoSQL data natively • Additional requirements • Must be able to scale-out • Must be available as a service The database must be able to ingest, process and analyze data in real-time
- 24. 24 Why use Informix in the “Operational Zone”? Simple to use • Hands-Free operation • Supports REST and Mongo APIs as well as ODBC/JDBC • Stores SQL and JSON database in the same database Highly Available • Close to zero down time • Partition or Hash your data across servers in the cloud • Dynamically add/remove additional servers Performance • Continuous High Performance Analytics • Specialized support for Time Series and Spatial data Invisible Agile Resilient
- 25. 25 Shards: Scale-out your Database across Servers or Gateways • Distribute data among servers by range or hash partitioning • Each shard can have an associated secondary server for high availability • Run queries across all shards or a subset of the shards • Only shards that could qualify are searched • Shards are searched in parallel • Ignores shards that are offline Shards in a Cloud
- 26. 26 IoT Analytics - Operational and Big Data Analytics • Operational Analytics • Needed when results are required in (near) real-time • Real-time monitoring, situational detection, correlation of events, e.g. • Dynamic advertising based on customer movement • Real-time equipment failure prediction • Operational analytics are required in gateways as well as the cloud • Gateways need to aggregate, filter, monitor for conditions • Analytics must run efficiently while new data is being loaded • Must be able to dynamically add and update analytics in the cloud and gateways • Big Data analytics are required when you have the time to do “Deeper/Wider” analysis • Latency between data arrival and results not an issue • Efficient ETL process from the operational repository is a must
- 27. 27 TCP/IP Bulk Loader SQL Queries (from apps) Informix Warehouse Accelerator Compressed DB partition Query Processor Data Warehouse Informix SQL Query Router Results Informix Warehouse Accelerator: • Connects to Informix via TCP/IP & DRDA • Analyzes, compresses, and loads to memory • Copy of (portion of) warehouse • Processes routed SQL query and • returns answer to Informix Use Informix Warehouse Accelerator for Mixed Operational/Analytic Workloads Informix: • Routes SQL queries to accelerator • User need not change SQL or apps. • Can always run query in Informix • Too short an est. execution time
- 28. 28 TimeSeries Service on BlueMix Every IoT deployment will need to store time series data and possibly spatial data • Provides REST, JSON, ODBC/JDBC based services to: • Store, retrieve, and join time series, spatial, relational, and NoSQL data • Built-in time series analytics • Multi-tenancy support • Scales out automatically as needed • No Administration needed IoT Foundation Service on BlueMix BlueMix Cloud Services • Quickly and simply add new sensors • Interface for continuously loading sensor data • Adapters for MQTT and MessageSight
- 30. IBM Informix and Intel Deliver Leading Edge Solutions for IoT ▪ Informix support for Intel’s new Quark processor, now suppor*ng Intel family, from Quark to Xeon. ▪ Quark port in the free Informix Developer Edi*on ▪ Informix Developer Edi*on available as part of standard Intel Gateway developer SDK Combining IBM and Intel’s strength at the Edge, Gateway and in the Cloud provides an intelligent, e2e IoT solu7on for rapid 7me to market. Intel® Quark™ SoC / Intel® Atom™ / Intel® Xeon™ Windriver McAfee Security Supports OSGI/TR69 Informix Database 30
- 31. Benchmark: Informix vs SQLite 31 Tests on Intel Quark Informix SQLite Data loading – high-speed performance (records per second) 950 / 1050 secs (DK100 / DK50)1 700 secs (Average)2 Storage space that is required for 1 day of data 275 MB 1200 MB Aggregation query (seconds) 2 secs 4-25 secs Moving average (seconds) 25 secs 2592003 secs Missing interval search (seconds) 2 secs 14-30 secs 1. The two figures for data loading with Informix reflect a slight difference in performance between the DK100 and DK50. DK100 had more running components causing a drop in performance vs DK50 2. Data loading with SQLite had significant variations in load performance as the database size increased. 3. The moving average result for SQLite is a projected figure that is based on a partial result after 10 minutes.
- 32. 32 Metric Competitor Informix Daily processing time Maximum number of cores used 11 hours 62 5 hour 50 min 32 Maximum amount of memory used 192GB 192GB Size of database per month of data 15TB 5TB # Records processed each day 2.88 Billion 2.88 Billion Billing determinants creation (1/21 of the total meter population) 51,322 ~2 million reads per second TimeSeries Meter Data Management Benchmark - 30 million smart meters sending data every 15 minutes - 2.88 billion records inserted each day - Workload: data Ingestion, data cleanup, and a daily billing cycle
- 33. Summary • IBM Informix - best fit for IoT architecture • IoT gateway • IoT cloud analytics • Supported on a wide array of platforms • Best in class embeddability • Native support for sensor data - TimeSeries & Spatial data • Native support for unstructured (JSON) data • Ease of application development - REST access • High availability and dynamic scaling 33
- 34. 34 Kevin Brown – kbrown3@us.ibm.com