What we do to improve scalability in our RDF processing system
0 likes553 views
This document outlines work done to improve the scalability of an RDF stream processing system. It describes transforming sensor data streams into tuples that are published to multiple channels to parallelize preprocessing. It also gives an example of applying these techniques to data from CSIRO's Sensor Cloud, annotating phenomena with SWEET and converting tuples to RDF on demand in response to queries. Future work discussed includes evaluation, interface improvements, and monitoring scalability metrics for adaptive query processing.
![Towards efficient processing of RDF data streams
Goal: to develop a stream processing engine
capable of adapting to variable conditions, such as
changing rates of input data, failure of processing
nodes, or distribution of workload, while serving
complex continuous queries.
Example of query execution parallelization (OrdRing 2014)
Project
Project
Project
Project
Simple Join
Simple Join
Simple Join
Simple Join
Windowing
<t1, t3...>
Windowing
<t0, t2...>
Windowing
<t1, t3...>
Windowing
<t0, t2...>
Storm topology example (4 nodes)
SELECT ?obs.value ?sensors.location
FROM NAMED STREAM <obs> [60 SEC TO NOW]
FROM NAMED STREAM <sensors> [60 SEC TO NOW]
WHERE obs.sensorId = sensors.id ;
SPOUT
<obs>
Triple2Graph
Output
SPOUT
<sensors> Triple2Graph
t0
t1
t2
t3](https://image.slidesharecdn.com/rspworkshop-scalability-allaves-oeg-p7-150601073737-lva1-app6891/85/What-we-do-to-improve-scalability-in-our-RDF-processing-system-4-320.jpg)
![Example of use: CSIRO's Sensor Cloud (2/3)
1.Sensor Cloud messages to field-named tuples
2.SWEET annotations for phenomena
<sample time=”20150528T16:30” value=”48” sensor=”bom_gov_au.94961.air.rel_hum”/>
[“20150528T16:32”, “20150528T16:30”, “48”, “bom_gov_au”, “94961”, “air”, “rel_hum”,
“43.3167”, “147.0075”]
network
phenomenon
platform sensorsampling time
system time
latitude longitude](https://image.slidesharecdn.com/rspworkshop-scalability-allaves-oeg-p7-150601073737-lva1-app6891/85/What-we-do-to-improve-scalability-in-our-RDF-processing-system-8-320.jpg)
What we do to improve scalability in our RDF processing system
- 1. Alejandro Llaves and Oscar Corcho Ontology Engineering Group Universidad Politécnica de Madrid Madrid, Spain allaves@fi.upm.es May 31 2015 Scalability in RDF Stream Processing Systems
- 2. Alejandro Llaves and Oscar Corcho Ontology Engineering Group Universidad Politécnica de Madrid Madrid, Spain allaves@fi.upm.es May 31 2015 What we do to improve Scalability in our RDF Stream Processing System
- 3. Outline Towards efficient processing of RDF data streams Architecture overview Parallelizing the pre-processing of sensor data streams Example of use: CSIRO's Sensor Cloud Discussion and future work
- 4. Towards efficient processing of RDF data streams Goal: to develop a stream processing engine capable of adapting to variable conditions, such as changing rates of input data, failure of processing nodes, or distribution of workload, while serving complex continuous queries. Example of query execution parallelization (OrdRing 2014) Project Project Project Project Simple Join Simple Join Simple Join Simple Join Windowing <t1, t3...> Windowing <t0, t2...> Windowing <t1, t3...> Windowing <t0, t2...> Storm topology example (4 nodes) SELECT ?obs.value ?sensors.location FROM NAMED STREAM <obs> [60 SEC TO NOW] FROM NAMED STREAM <sensors> [60 SEC TO NOW] WHERE obs.sensorId = sensors.id ; SPOUT <obs> Triple2Graph Output SPOUT <sensors> Triple2Graph t0 t1 t2 t3
- 6. Parallelizing the pre-processing of sensor data streams Methodology 1. Transform data input into field-named tuples 2. Add semantic annotations (if needed) 3. Publish tuples to multiple channels 4. Convert tuples to RDF on (query) demand Focus Storm topologies Environmental sensor observations Using Semantic Sensor Network (SSN) ontology
- 7. Example of use: CSIRO's Sensor Cloud (1/3) Sensor Cloud Viticulture, water management, weather monitoring, oyster farming... RESTful API – JSON Network → Platform → Sensor → Phenomenon → Observation Lack of semantic descriptions, e.g. rain_trace vs Rain. Multiple HTTP requests to query various streams. Source: CSIRO
- 8. Example of use: CSIRO's Sensor Cloud (2/3) 1.Sensor Cloud messages to field-named tuples 2.SWEET annotations for phenomena <sample time=”20150528T16:30” value=”48” sensor=”bom_gov_au.94961.air.rel_hum”/> [“20150528T16:32”, “20150528T16:30”, “48”, “bom_gov_au”, “94961”, “air”, “rel_hum”, “43.3167”, “147.0075”] network phenomenon platform sensorsampling time system time latitude longitude
- 9. Example of use: CSIRO's Sensor Cloud (3/3) 3.Publish to multiple channels 4.Convert tuples to SSN model on (query) demand Publish2Kafka Publish2Kafka SWEET Annotation SWEET Annotation SC2Tuple SC2Tuple Sensor Cloud spout sweet:Rainfall All tuples sweet:RelativeHumidity sweet:Rainfall All tuples sweet:RelativeHumidity Kafka spout Tuple2SSN Tuple2SSN to be continued... Query: heavy rainfall events
- 10. Discussion and future work Conclusion Division of work into simple tasks. Parallelize any parallelizable task. Delay RDF generation and convert on demand. Future work Evaluation and benchmarking. SSN mapping interface. Topology package: executing distributed queries (Storm). theObserver (theO) package: monitoring scalability metrics for adaptive query processing.
- 11. The presented research has has been funded by Ministerio de Economía y Competitividad (Spain) under the project ”4V: Volumen, Velocidad, Variedad y Validez en la Gestión Innovadora de Datos” (TIN2013-46238-C4-2-R), by the EU Marie Curie IRSES project SemData (612551), and supported by an AWS in Education Research Grant award. Alejandro Llaves allaves@fi.upm.es Thanks!