Towards efficient processing of RDF data streams

Towards efficient
processing of
RDF data streams
Alejandro Llaves
Javier D. Fernández
Oscar Corcho
Ontology Engineering Group
Universidad Politécnica de Madrid
Madrid, Spain
allaves@fi.upm.es
OrdRing workshop - ISWC 2014
Riva del Garda
October 20th, 2014

Efficient?? Scalable?
http://blog.mikiobraun.de/

Outline
 Introduction
 Background: Storm and Lambda Architecture
 Efficient processing of queries over RDF streams
 Architecture overview
 Storm-based operators for querying RDF streams
 Adaptive query processing for data streams
 RDF stream compression
 Conclusions & future work
 Open questions

Introduction
 Origins of Linked Stream Data
 Extracting information from data streams is complex:
heterogeneity, rate of generation, volume, provenance,…
 Challenges
 C1. Efficient processing of user queries over RDF streams
 C2. Continuous transmission of data increases latency
 C3. Integration of historical and real-time data with background
knowledge
Source: http://webnotations.com

Background
Storm - http://storm.incubator.apache.org/
 Distributed system for real-time processing of streams
 Why Storm?
 Simple processing model (parallelization)
 Open source community backing the project
 Used by relevant companies, e.g. Twitter.
Lambda Architecture (Marz 2013)
 Batch layer: stores ALL the incoming data in an immutable
master dataset and pre-computes batch views on historic data.
 Serving layer: indexes views on the master dataset.
 Real-time processing layer: requests data views depending on
incoming queries.

Efficient processing of RDF streams
Goal: to develop a stream processing engine capable of
adapting to variable conditions, such as changing rates of
input data, failure of processing nodes, or distribution of
workload, while serving complex continuous queries.
Methodology
 State of the art of (RDF) stream processing
 Evaluate how to parallelize SPARQLStream queries
 Implementation of RDF query operators
 Optimize parallelization for common queries
 Design self-adaptive strategies that allow the engine to
react in front of changes

Storm-based operators for querying RDF streams
 Triple2Graph operator
 Time Window operator
 Simple Join operator
 Projection operator
Simple Join Stream<Set<Tuple>>
(join attribute)
Stream<Set<Tuple>, Set<Tuple>>
Windowing Stream<Set<Graph>>
(window size,
emission time)
Stream<Graph, t>
Project Stream<Tuple<o1, o2,..., on>>
(input, output)
Stream<Tuple<i1, i2,..., in>>
Triple2Graph Stream<Graph, t>
(graph starter)
Stream<s, p, o>

Storm-based operators for querying RDF streams
Project
Project
Project
Project
Storm topology example (4 nodes)
SELECT ?obs.value ?sensors.location
FROM NAMED STREAM <obs> [60 SEC TO NOW]
FROM NAMED STREAM <sensors> [60 SEC TO NOW]
WHERE obs.sensorId = sensors.id ;
Simple Join
Simple Join
Simple Join
Simple Join
Windowing
<t0, t2...>
Windowing
<t1, t3...>
Windowing
<t0, t2...>
Windowing
<t1, t3...>
SPOUT
<obs>
Triple2Graph
Output
SPOUT
<sensors> Triple2Graph
t0
t1
t2
t3

RDF stream compression (1/2)
Efficient RDF Interchange (ERI) format
 Based on Efficient XML Interchange (EXI)
 Main assumption: RDF streams have regular structure and
are redundant
 Information encoded at 2 levels
 Structural dictionary
 Presets (values)
 Example: SSN observations

Where can we apply RDF stream compression?

Conclusions and future work
Conclusions
 We have addressed challenges C1 (scalability) and C2 (transmission)
 Catalogue of Storm-based operators to parallelize query processing over RDF
streams.
 New format for RDF stream compression called ERI.
 Challenge C3 (integration) involves storage of historical data and the
deployment of batch and serving layers OR the migration to a more
general system, e.g. Apache Spark.
Future work
 Finish the implementation of RDF query operators
 Test the parallelization of a set of common queries → SRBench
 Adaptive strategies based on Adaptive Query Processing
 Evaluation → Benchmarking: comparison to CQELS Cloud
 Integration of ERI into our engine

Open questions
 How does the order of tuple arrival affect the
parallelization of join processing tasks?
 Are the spatial (or spatio-temporal) properties of a
tuple a dimension to have into account for ordering?
In such case, what influence does it have on
reasoning tasks? And on parallelization tasks?
 How does the out-of-order tuples affect the
processing of streams? In case of discarding out-of-order
tuples, how to communicate this in the
results?

Thanks!
The research leading to this results has received funding from the
EU's Seventh Framework Programme (FP7/2007-2013) under
grant agreement no. 257641, PlanetData network of excellence,
from Ministerio de Economía y Competitividad (Spain) under the
project “4V: Volumen, Velocidad, Variedad y Validez en la Gestión
Innovadora de Datos” (TIN2013-46238-C4-2-R), and has been
supported by an AWS in Education Research Grant award.
Alejandro Llaves
allaves@fi.upm.es

Adaptive Query Processing (AQP) for data streams
 Traditional databases include a query optimizer that
designs the execution plan based on the data
statistics.
 AQP (Deshpande 2007) techniques allow adjusting
the query execution plan to varying conditions of
the data input, the incoming queries, and the
system.

RDF stream compression (2/2)
Evaluation
 Datasets: streaming, statistical, and general static.
 Compression ratio, compression time, and parsing
throughput (transmission + decompression)
 Comparison to other formats, such as N-Triples, Turtle,
RDSZ, HDT, with different configurations of ERI w.r.t.
transmitted data block (1K – 4K) and the presence of
dictionary.
 Conclusion: ERI produces state-of-the-art compression for
RDF streams and excels for regularly-structured static RDF
datasets. ERI compression ratios remain competitive in
general datasets and the time overheads for ERI
processing are relatively low.
http://dataweb.infor.uva.es/wp-content/uploads/2014/07/iswc14.pdf

Towards efficient processing of RDF data streams

More Related Content

What's hot (20)

Similar to Towards efficient processing of RDF data streams (20)

Recently uploaded (20)

Towards efficient processing of RDF data streams