SPARQL-to-SQL on Internet of Things Databases and Streams
1 like139 views
The document discusses the challenges and solutions related to interoperability and data management in the Internet of Things (IoT), emphasizing the limitations of current product silos and cloud dependency. It introduces linked data concepts to address data integration and efficiency, particularly for time-series data, through the implementation of a SPARQL-to-SQL engine. Additionally, it presents performance benchmarks illustrating improved query efficiency for environmental data in low-resource computing environments like Raspberry Pi.
SPARQL-to-SQL on Internet of Things Databases and Streams
- 1. INTERNET OF THINGS DEVICES ANDSTREAMS SPARQL-TO-SQLON EUGENE SIOW THANASSIS TIROPANIS WENDY HALL
- 2. Trekking up 913m Mt. rokko
- 3. Sensor DATAFROMMULTIPLE SOURCES “The Internet of Things is currently beset by product silos.” W3C Web of Things Interest Group CURRENT STATE OFTHE INTERNET OF THINGS PRODUCT &DATASILOS DEPENDENCYON THE CLOUD LIMITED INTEROPERABILITY OF APPS
- 4. LINKED DATA ON LIGHTWEIGHT COMPUTERS APPLE GOOGLE HONEYWELL CISCO HUAWEI GENERALELECTRIC IBM AMAZON INTEL MICROSOFT
- 5. INTRODUCING LINKED DATA FOR INTEROPERABILITY URI andontologies Establish commondata structures& References ENABLES RICH METADATA what,where, WHEN,HOW of DATA Integrates with Linked open data cyber, social, physical LOD on web Barnaghi, P., Wang, W.: Semantics for the Internet of Things: early progress and back to the future. International Journal on Semantic Web and Information Systems (2012) http://thing.io/1 is a http://ont/weather_sensor CLASS produces http://thing.io/obs/1 http://ont/temp_observation is a 13.0 has value CLASS ℃ unit located at http://thing.io/loc/1 latitude longitude -1.4150.9
- 6. LINKEDDATA ON DISTRIBUTEDLIGHTWEIGHT COMPUTERS Tiered levels of applications between the “Ground” and “Cloud” for Meteorological Data Irrigation Application Query on Rainfall Meteorological Station on Lightweight Computer Data Stream Environmental Sensors GROUND Weather Data State Inclement Weather Planning Application SRBench:National Weather and Disaster Monitoring CLOUD Broadcast Queries
- 7. PERFORMANCE STORES DON’T SCALE & PERFORM WELLON WEB YET Buil-Aranda, C., Hogan, A.: SPARQL Web-Querying Infrastructure: Ready for Action? ISWC 2013 CHALLENGES RESOURCE CONSTRAINED DEVICES ~500mhz CPU,512mb ram,SD CARD historical and streaming data Need storage andqueryefficiency ON BOTH
- 8. THE SHAPE OFIOT TIME-SERIES DATA { timestamp : 1467673132, temperature : { max: 22.0, min: 15.0, current: 17.0, error: { percentage: 5.0 } } } FLAT { timestamp : 1467673132, temperature : 32.0, wind_speed : 10.5, pressure : 1016 } COMPLEX 20kUNIQUE DEVICES dweet.io 99.5%FLAT SCHEMATA 0.5%COMPLEX SCHEMATA 1 2,3 4 5 6+ Width { timestamp : 1467673132, temperature : 32.0, humidity : 10.5, pressure : 1016, light: 120.0, } 1 2 3 4
- 9. EFFICIENTQUERIESWITH TIME-SERIES DATA THING TEMPERATURE OBS HUMIDITY OBS WIND SPEED OBS 13.0 2016-01-0106:00:00 CELCIUS 93.0 2016-01-0106:00:00 PERCENT 10.5 2016-01-0106:00:00 MPH LOCATION produces produces located produces has value unit time RDF GRAPH
- 10. OBSERVATION DATA OBSERVATION METADATA SENSOR METADATA THING TEMPERATURE OBS HUMIDITY OBS WIND SPEED OBS 13.0 LOCATION produces produces located produces has value THING THING THING TEMPERATURE OBS timeTEMPERATURE OBS 2016-01-0106:00:00 unitTEMPERATURE OBS celcius 93.0has valueHUMIDITY OBS timeHUMIDITY OBS 2016-01-0106:00:00 unitHUMIDITY OBS PERCENT 10.5has valueWIND SPEED OBS timeWIND SPEED OBS 2016-01-0106:00:00 unitWIND SPEED OBS MPH EFFICIENTQUERIESWITH TIME-SERIES DATA RDF TRIPLES
- 11. OUR APPROACH EFFICIENTQUERIESWITH TIME-SERIES DATA THING TEMPERATURE OBS WIND SPEED OBS CELCIUS PERCENT MPH LOCATION produces located HUMIDITY OBS unit TEMPERATURE HUMIDITY WIND SPEED 13.0 93.0 10.5 TIME 2016-01-01 06:00:00
- 12. SENSOR METADATA OBSERVATION DATA OUR APPROACH EFFICIENTQUERIESWITH TIME-SERIES DATA THING TEMPERATURE OBS WIND SPEED OBS CELCIUS PERCENT MPH LOCATION produces located HUMIDITY OBS unit TEMPERATURE HUMIDITY WIND SPEED 13.0 93.0 10.5 TIME 2016-01-01 06:00:00 OBSERVATION METADATA
- 13. DESIGNING OURENGINE THING TEMPERATURE OBS WIND SPEED OBS CELCIUS PERCENT MPH LOCATION produces located HUMIDITY OBS unit TEMPERATURE HUMIDITY WINDSPEED 13.0 93.0 10.5 TIME 2016-01-01 06:00:00 Table1 TABLE1.TEMPERATURE has value has value TABLE1.HUMIDITY has value TABLE1.WINDSPEED
- 14. DESIGNING OURENGINE THING TEMPERATURE OBS WIND SPEED OBS CELCIUS PERCENT MPH LOCATION produces located HUMIDITY OBS unit TEMPERATURE HUMIDITY WINDSPEED 13.0 93.0 10.5 TIME 2016-01-01 06:00:00 Table1 TABLE1.TEMPERATURE has value has value TABLE1.HUMIDITY has value TABLE1.WINDSPEED
- 15. DESIGNING OURENGINE THING TEMPERATURE OBS CELCIUS PERCENT produces loc HUMIDITY OBS unit TEMPERATURE HUMID 13.0 93.0 TIME 2016-01-01 06:00:00 TABLE1.TEMPERATURE has value has va TABLE1.H MAX( )?TEMPERATURESELECT ?OBS TEMPERATURE OBSa has value?OBS ?TEMPERATURE has unit?OBS ?uom { } 𝞹 𝞬 (max ( ))?TEMPERATURE ?OBS TEMPERATURE OBSa has value?OBS ?TEMPERATURE has unit?OBS ?uom BGP
- 16. DESIGNING OURENGINE TEMPERATURE OBS CELCIUS TEMPERATURE 13.0 TABLE1.TEMPERATURE has value MAX( )?TEMPERATURESELECT ?OBS TEMPERATURE OBSa has value?OBS ?TEMPERATURE has unit?OBS ?uom { } 𝞹 𝞬 (max ( ))?TEMPERATURE ?OBS TEMPERATURE OBSa has value?OBS ?TEMPERATURE has unit?OBS ?uom BGP
- 17. DESIGNING OURENGINE MAX( )?TEMPERATURESELECT ?OBS TEMPERATURE OBSa has value?OBS ?TEMPERATURE has unit?OBS ?uom { } 𝞹 MAX( )?TEMPERATURE ?OBS ?TEMPERATURE ?uom TABLE1.TEMPERATURE CELCIUSNODE_TEMP 𝞹 𝞬 (max ( ))?TEMPERATURE ?OBS TEMPERATURE OBSa has value?OBS ?TEMPERATURE has unit?OBS ?uom BGP
- 18. DESIGNING OURENGINE MAX( )?TEMPERATURESELECT ?OBS TEMPERATURE OBSa has value?OBS ?TEMPERATURE has unit?OBS ?uom { } MAX( )TEMPERATURE 𝞹 𝞬 (max ( ))?TEMPERATURE ?OBS TEMPERATURE OBSa has value?OBS ?TEMPERATURE has unit?OBS ?uom BGP FROM TABLE1 𝞹
- 19. SPARQL DESIGNING OURENGINE MAX( )?TEMPERATURESELECT ?OBS TEMPERATURE OBSa has value?OBS ?TEMPERATURE has unit?OBS ?uom { } SQL SELECT MAX( )TEMPERATURE FROM TABLE1
- 20. LOWVISIBILITYSTATIONSEXAMPLE ?OBS VISIBILITY OBSa has value?OBS ?VISIBILITY has unit?OBS ?uom } { FILTER( ,<10 )?VISIBILITY UNION ?OBS RAINFALLOBSa has value?OBS ?RAINFALL has unit?OBS ?uom } { FILTER( ,>30 )?RAINFALL ?SENSORSELECT { } UNION ?OBS SNOWFALLOBSa has value?OBS ?SNOWFALL has unit?OBS ?uom } { 𝞹 ?OBS SNOWFALLOBSa has value?OBS ?SNOWFALL has unit?OBS ?uom BGP UNION UNION FILTER( ,<10 )?VISIBILITY FILTER( ,>30 )?RAINFALL ?OBS VISIBILITY OBSa has value?OBS ?VISIBILITY has unit?OBS ?uom BGP ?OBS RAINFALLOBSa has value?OBS ?RAINFALL has unit?OBS ?uom BGP FILTER( ,TRUE)?SNOWFALL
- 21. LOWVISIBILITYSTATIONSEXAMPLE ?OBS VISIBILITY OBSa has value?OBS ?VISIBILITY has unit?OBS ?uom } { FILTER( ,<10 )?VISIBILITY UNION ?OBS RAINFALLOBSa has value?OBS ?RAINFALL has unit?OBS ?uom } { FILTER( ,>30 )?RAINFALL ?SENSORSELECT } UNION ?OBS SNOWFALLOBSa has value?OBS ?SNOWFALL has unit?OBS ?uom } { 𝞹 UNION UNION FILTER( ,<10 )?VISIBILITY FILTER( ,>30 )?RAINFALL ?OBS ?snowfall ?uom TABLE1.snowfall BOOLNODE_SNOW ?OBS ?VISIBILITY ?uom TABLE1.VISIBILITY cmNODE_vis ?OBS ?rainfall ?uom TABLE1.rainfall cmNODE_RAIN { FILTER( ,TRUE)?SNOWFALL
- 22. LOWVISIBILITYSTATIONSEXAMPLE ?OBS VISIBILITY OBSa has value?OBS ?VISIBILITY has unit?OBS ?uom } { FILTER( ,<10 )?VISIBILITY UNION ?OBS RAINFALLOBSa has value?OBS ?RAINFALL has unit?OBS ?uom } { FILTER( ,>30 )?RAINFALL ?SENSORSELECT { } UNION ?OBS SNOWFALLOBSa has value?OBS ?SNOWFALL has unit?OBS ?uom } { 𝞹 UNION UNION FILTER( ,<10 )VISIBILITY FILTER( ,>30 )RAINFALL FROM TABLE1FROM TABLE1 FROM TABLE1 FILTER( ,TRUE)SNOWFALL
- 23. LOWVISIBILITYSTATIONSEXAMPLE ?OBS VISIBILITY OBSa has value?OBS ?VISIBILITY has unit?OBS ?uom } { FILTER( ,<10 )?VISIBILITY UNION ?OBS RAINFALLOBSa has value?OBS ?RAINFALL has unit?OBS ?uom } { FILTER( ,>30 )?RAINFALL ?SENSORSELECT { } UNION ?OBS SNOWFALLOBSa has value?OBS ?SNOWFALL has unit?OBS ?uom } { 𝞹 UNION FROM TABLE1 WHERE RAINFALL>30 SELECT SENSOR FROM TABLE1 WHERE VISIBILITY<10 UNION SELECT SENSOR FROM TABLE1 FILTER( ,TRUE)SNOWFALL
- 24. SQL SPARQL LOWVISIBILITYSTATIONSEXAMPLE ?OBS VISIBILITY OBSa has value?OBS ?VISIBILITY has unit?OBS ?uom } { FILTER( ,<10 )?VISIBILITY UNION ?OBS RAINFALLOBSa has value?OBS ?RAINFALL has unit?OBS ?uom } { FILTER( ,>30 )?RAINFALL ?SENSORSELECT { } UNION ?OBS SNOWFALLOBSa has value?OBS ?SNOWFALL has unit?OBS ?uom } { SELECT UNION WHERE RAINFALL>30 WHERE VISIBILITY<10 UNION SENSOR FROM SELECT SENSOR FROM TABLE1 WHERE SNOWFALL==TRUE ( ) SELECT SENSOR FROM TABLE1 SELECT SENSOR FROM TABLE1
- 25. HOWMAPPINGSWORK SPARQL2SQL Mapping S2SML MAPPING TABLE1.TEMPERATURE has value THING TEMPERATURE OBS WIND SPEED OBS CELCIUS PERCENT MPH LOCATION produces located HUMIDITY OBS unit TABLE1.humidity has value TABLE1.WINDSPEED has value ?OBS TEMPERATURE OBSa has value?OBS ?TEMPERATURE has unit?OBS ?uom BGP SPARQL ?OBS TEMPERATURE OBSa has value?OBS ?TEMPERATURE has unit?OBS ?uom SELECT * WHERE { } JENA SESAME ?? INTERFACE
- 26. S2SML MAPPING WRITTEN IN RDF REFLECTSTHE RDF ITREPRESENTS DIRECTLY SUPPORTS COLLAPSING NODES EFFICIENT QUERIES WITHIOT TIME-SERIES DATA R2rML COMPATIBLE CAN BE TRANSLATEDTOAND FROM R2RML SUBJECT object predicate IRImapIRI BNODE IRI FNODE IRImapIRI BNODE FNODE LITERAL LITERALMAP
- 27. S2SML MAPPING FauxNODE HUMIDITYOBS TABLE1.COL1 hasValue LITERALMAP <http://knoesis.wright.edu/ssw/obs/{readings.uuid}> if (HUMIDITY OBS is projected): if(readings.uuid !exists): create_col(readings.uuid) return IRImap(readings.uuid)
- 28. RDF STREAMPROCESSING sparql2stream Same engine and mappings but translates to EPL instead of SQL TRANSLATE QUERY 2 Stream Window SPARQL query specifying stream window size REGISTER QUERY 1 Stream Sockets Supports multiple platforms and streams with ZeroMQ STREAMDATA 3 Real-time analytics PIOTRe & Smart Home Freeboard demo github.com/eugenesiow/iotwo RECEIVE PUSH RESULTS 4
- 29. SPARQL2stream RSP QUERY MAX( )?TEMPERATURESELECT ?OBS TEMPERATURE OBSa has value?OBS ?TEMPERATURE has unit?OBS ?uom { } FROM NamedStream Obsstream Range 1h EPL SELECT MAX( )TEMPERATURE FROM Weather.win:time(1hour) Event Processing Language
- 30. EVALUATIONWITH BENCHMARKS SRBENCH ~20,000 Stations 100 – 300k triples Wind, Rainfall, etc. 10 SRBench Queries Zhang, Y, et al. (2012) "SRBench: a streaming RDF/SPARQL benchmark.”The 11th International Semantic Web Conference. SMART HOME BENCH Siow, E., Tiropanis, T., Hall, W. (2016). "Interoperable and Efficient: Linked Data for the Internet of Things." The 3rd International Conference on Internet Science. 3 months, 1 home ~30k triples Motion, energy, environment 4 Analytics Queries GraphDB (OWLIM) Ontop Our Approach (S2S) TDB G Morph O S M T Raspberry Pi 2 Model B+ 1GB RAM, 900MHz Quad Core ARM Cortex A7, Class 10 SD Cards
- 32. Get the rainfall observed in a particular hour from all stations01 02 SRBENCH QUERYRESULTS Q01 with an optional clause on unit of measure x5 S2S S TDB GraphDB Ontop Morph x3 x13 x4k x2 x4 x4 x5k
- 33. 03 04 05 Detect if a hurricane has been observed Get the average wind speed at the stations where the air temperature is >32 Join between wind observation and temperature observation subtrees time-consuming in low resource environment (Raspberry Pi) Detect if a station is observing a blizzard x3 x6 x6 x88 x3 x3
- 34. 06 07 08 Get the stations with extremely low visibility Detect stations that are recently broken Get the daily minimal and maximal air temperature observed by the sensor at a given location x2 x14 x4 x6 x6 x5 x2
- 35. 09 10 Get the daily average wind force and direction observed by the sensor at a given location Get the locations where a heavy snowfall has been observed Our Approach (s2s) is shown to be faster on all queries in the Distributed Meteorological System with SRBench Join between wind force and wind direction observation subtrees is time-consuming in low resource environment (Raspberry Pi) x3 x3k x2 x7
- 36. Temperature aggregated by hour on a specified day01 02 SMARTHOME RESULTS Minimum and maximum temperature each day for a particular month S2S TDB GraphDB x7 x29 x3 x9
- 37. 03 04 Energy Usage Per Room By Day Diagnose unattended appliances consuming energy with no motion in room Our Approach (s2s) is shown, once again, to be faster on all queries for Smart Home Analytics Involves motion and meter data (much larger set), with space-time aggregations and joins between motion and meter tables/subgraphs. Involves meter data (larger set), with space-time aggregations. x69 x13 x4
- 38. STREAMPROCESSING EFFICIENCY 1 2 3 4 5 7 8 9 10 SRBENCH 294 261 306 277k 3243k 5245 426 280k 98 Le-Phuoc, D., et al. (2011) "A native and adaptive approach for unified processing of linked streams and linked data.” The 10th International Semantic Web Conference.CQELSPerformance Improvement For IoT Data Over SMART HOME 196 21 167 xImprovement Query
- 39. STREAMPROCESSING SCALABILITY VELOCITY >99% <1ms latency increasing from 1 to 1000 rows/ms VOLUME 33.5million rows, projected ~2.5 billion triples! <1ms 10-100ms 1 2 5 10 100 1000 99% 100% Rateinrows/ms Percentage Latency in ms Bands
- 40. PERSONAL IOT REPOSITORY Siow, E., Tiropanis, T. and Hall, W. (2016) PIOTRe: Personal Internet of Things Repository: The 15th International Semantic Web Conference P&D github.com/eugenesiow/piotresparql2streamsparql2sql github.com/eugenesiow/sparql2sql PIOTRE Apps sparql2stream sparql2sql Metadata
- 41. “It's a long road, it's a long and narrow way. If I can't work up to you, you'll surely have to work down to me someday.” Narrow Way by Bob Dylan eugene_siow Travel Supported By: LINKED DATA FORINTEROPERABILITY A rich model todescribe thingsand integrateconnected thing’sdata OPTIMISED PERFORMANCEFORSTREAMS AND HISTORICAL DATA FROM 2to3orders of magnitudeperformance improvement