SANSA ISWC 2017 Talk
3 likes5,833 views
The document discusses combining big data processing frameworks like Spark and Flink with semantic technologies. It presents SANSA, a system that loads RDF data into Spark and enables distributed semantic processing including querying, reasoning, and analytics. SANSA allows expressing ontologies and rules as RDDs to leverage Spark's scalability while supporting W3C standards like RDF and OWL. The document provides examples of using SANSA to load and partition RDF graphs, perform reasoning using rules, and write inferred data back to disk.
![•
•
•
•
•
val rdd = ManchesterSyntaxOWLAxiomsRDDBuilder.build(spark, "file.owl")
// get all subclass-of axioms
val sco = rdd.filter(_.isInstanceOf[OWLSubClassOfAxiom])](https://image.slidesharecdn.com/sansaiswc2017talk-171023111647/85/SANSA-ISWC-2017-Talk-11-320.jpg)
SANSA ISWC 2017 Talk
- 3. • • • Disk In-memory Iteration 1 Iteration 2 Iteration n Intermediate Dataset (in cluster memory) Intermediate Dataset (in cluster memory) Output
- 6. “Big Data” Processing (Spark/Flink) Semantic Technology Stack Data Integration Manual pre-processing Partially automated, standardised Modelling Simple (often flat feature vectors) Expressive Support for data exchange Limited (heterogeneous formats with limited schema information) Yes (RDF & OWL W3C Standards) Business value Direct Indirect Horizontally scalable Yes No Idea: combine advantages of both worlds
- 9. • • • val graph: TripleRDD = NTripleReader.load(spark, uri) graph.find(ANY, URI("http://dbpedia.org/ontology/influenced"), ANY) val rdf_stats_prop_dist = PropertyUsage(graph, spark).PostProc()
- 10. •
- 11. • • • • • val rdd = ManchesterSyntaxOWLAxiomsRDDBuilder.build(spark, "file.owl") // get all subclass-of axioms val sco = rdd.filter(_.isInstanceOf[OWLSubClassOfAxiom])
- 13. val graphRdd = NTripleReader.load(spark,input) val partitions = RdfPartitionUtilsSpark.partitionGraph(graphRdd) val rewriter = SparqlifyUtils.createSparqlSqlRewriter(spark, partitions) val qef = new QueryExecutionFactorySparqlifySpark(spark, rewriter) SANSA Engine RDF Layer Data Ingestion Partitioning Query Layer Sparqlifying Distributed Data Structures ResultsViews Views
- 15. • • • → →
- 16. val graph = RDFGraphLoader.loadFromDisk(spark, uri) val reasoner = new ForwardRuleReasonerOWLHorst(spark.sparkContext) val inferredGraph = reasoner.apply(graph) RDFGraphWriter.writeToDisk(inferredGraph, output) RDFS rule dependency graph (simplified)
- 19. • •
- 20. Visit our demo at 6pm!
- 21. • • • • • •
- 22. • • • • • • •
- 23. Web: http://sansa-stack.net Twitter: @SANSA_Stack Github: https://github.com/SANSA-Stack Mail: sansa-stack@googlemail.com
- 24. • • •
- 27. • • • • • • •
- 28. • • • • • •
- 29. • • • • •