Towards efficient processing of RDF data streams
1 like975 views
The document discusses the efficient processing of RDF data streams using Storm and Lambda architecture, focusing on developing a scalable and adaptive query processing engine. It outlines challenges facing RDF stream queries, provides an overview of Storm-based operators, and introduces a new format for RDF stream compression called ERI. Future work includes testing and optimizing query operators while addressing the integration of historical data with real-time processing.
![Storm-based operators for querying RDF streams
Project
Project
Project
Project
Storm topology example (4 nodes)
SELECT ?obs.value ?sensors.location
FROM NAMED STREAM <obs> [60 SEC TO NOW]
FROM NAMED STREAM <sensors> [60 SEC TO NOW]
WHERE obs.sensorId = sensors.id ;
Simple Join
Simple Join
Simple Join
Simple Join
Windowing
<t0, t2...>
Windowing
<t1, t3...>
Windowing
<t0, t2...>
Windowing
<t1, t3...>
SPOUT
<obs>
Triple2Graph
Output
SPOUT
<sensors> Triple2Graph
t0
t1
t2
t3](https://image.slidesharecdn.com/ordring2014allaves-141020081440-conversion-gate01/85/Towards-efficient-processing-of-RDF-data-streams-9-320.jpg)
Towards efficient processing of RDF data streams
- 1. Towards efficient processing of RDF data streams Alejandro Llaves Javier D. Fernández Oscar Corcho Ontology Engineering Group Universidad Politécnica de Madrid Madrid, Spain allaves@fi.upm.es OrdRing workshop - ISWC 2014 Riva del Garda October 20th, 2014
- 3. Outline Introduction Background: Storm and Lambda Architecture Efficient processing of queries over RDF streams Architecture overview Storm-based operators for querying RDF streams Adaptive query processing for data streams RDF stream compression Conclusions & future work Open questions
- 4. Introduction Origins of Linked Stream Data Extracting information from data streams is complex: heterogeneity, rate of generation, volume, provenance,… Challenges C1. Efficient processing of user queries over RDF streams C2. Continuous transmission of data increases latency C3. Integration of historical and real-time data with background knowledge Source: http://webnotations.com
- 5. Background Storm - http://storm.incubator.apache.org/ Distributed system for real-time processing of streams Why Storm? Simple processing model (parallelization) Open source community backing the project Used by relevant companies, e.g. Twitter. Lambda Architecture (Marz 2013) Batch layer: stores ALL the incoming data in an immutable master dataset and pre-computes batch views on historic data. Serving layer: indexes views on the master dataset. Real-time processing layer: requests data views depending on incoming queries.
- 6. Efficient processing of RDF streams Goal: to develop a stream processing engine capable of adapting to variable conditions, such as changing rates of input data, failure of processing nodes, or distribution of workload, while serving complex continuous queries. Methodology State of the art of (RDF) stream processing Evaluate how to parallelize SPARQLStream queries Implementation of RDF query operators Optimize parallelization for common queries Design self-adaptive strategies that allow the engine to react in front of changes
- 8. Storm-based operators for querying RDF streams Triple2Graph operator Time Window operator Simple Join operator Projection operator Simple Join Stream<Set<Tuple>> (join attribute) Stream<Set<Tuple>, Set<Tuple>> Windowing Stream<Set<Graph>> (window size, emission time) Stream<Graph, t> Project Stream<Tuple<o1, o2,..., on>> (input, output) Stream<Tuple<i1, i2,..., in>> Triple2Graph Stream<Graph, t> (graph starter) Stream<s, p, o>
- 9. Storm-based operators for querying RDF streams Project Project Project Project Storm topology example (4 nodes) SELECT ?obs.value ?sensors.location FROM NAMED STREAM <obs> [60 SEC TO NOW] FROM NAMED STREAM <sensors> [60 SEC TO NOW] WHERE obs.sensorId = sensors.id ; Simple Join Simple Join Simple Join Simple Join Windowing <t0, t2...> Windowing <t1, t3...> Windowing <t0, t2...> Windowing <t1, t3...> SPOUT <obs> Triple2Graph Output SPOUT <sensors> Triple2Graph t0 t1 t2 t3
- 10. RDF stream compression (1/2) Efficient RDF Interchange (ERI) format Based on Efficient XML Interchange (EXI) Main assumption: RDF streams have regular structure and are redundant Information encoded at 2 levels Structural dictionary Presets (values) Example: SSN observations
- 11. Where can we apply RDF stream compression?
- 12. Conclusions and future work Conclusions We have addressed challenges C1 (scalability) and C2 (transmission) Catalogue of Storm-based operators to parallelize query processing over RDF streams. New format for RDF stream compression called ERI. Challenge C3 (integration) involves storage of historical data and the deployment of batch and serving layers OR the migration to a more general system, e.g. Apache Spark. Future work Finish the implementation of RDF query operators Test the parallelization of a set of common queries → SRBench Adaptive strategies based on Adaptive Query Processing Evaluation → Benchmarking: comparison to CQELS Cloud Integration of ERI into our engine
- 13. Open questions How does the order of tuple arrival affect the parallelization of join processing tasks? Are the spatial (or spatio-temporal) properties of a tuple a dimension to have into account for ordering? In such case, what influence does it have on reasoning tasks? And on parallelization tasks? How does the out-of-order tuples affect the processing of streams? In case of discarding out-of-order tuples, how to communicate this in the results?
- 14. Thanks! The research leading to this results has received funding from the EU's Seventh Framework Programme (FP7/2007-2013) under grant agreement no. 257641, PlanetData network of excellence, from Ministerio de Economía y Competitividad (Spain) under the project “4V: Volumen, Velocidad, Variedad y Validez en la Gestión Innovadora de Datos” (TIN2013-46238-C4-2-R), and has been supported by an AWS in Education Research Grant award. Alejandro Llaves allaves@fi.upm.es
- 15. Adaptive Query Processing (AQP) for data streams Traditional databases include a query optimizer that designs the execution plan based on the data statistics. AQP (Deshpande 2007) techniques allow adjusting the query execution plan to varying conditions of the data input, the incoming queries, and the system.
- 16. RDF stream compression (2/2) Evaluation Datasets: streaming, statistical, and general static. Compression ratio, compression time, and parsing throughput (transmission + decompression) Comparison to other formats, such as N-Triples, Turtle, RDSZ, HDT, with different configurations of ERI w.r.t. transmitted data block (1K – 4K) and the presence of dictionary. Conclusion: ERI produces state-of-the-art compression for RDF streams and excels for regularly-structured static RDF datasets. ERI compression ratios remain competitive in general datasets and the time overheads for ERI processing are relatively low. http://dataweb.infor.uva.es/wp-content/uploads/2014/07/iswc14.pdf