Flink Forward Berlin 2017: Max Kiessling, Martin Junghanns - Cypher-based Graph Pattern Matching in Apache Flink

Cypher-based
Graph Pattern
Matching
in Apache Flink

About us
Martin Junghanns
Software Engineer @ Neo4j
PhD Student @ University of Leipzig
Max Kiessling
Software Engineer @ Neo4j

Graph = Set of Vertices + Set of Edges

A Real World Example - OrcBook

Sauron’s Data Analyst: “Who are the closest enemies of each Orc?”
What Can you do with it?

Flink Forward Berlin 2017: Max Kiessling, Martin Junghanns - Cypher-based Graph Pattern Matching in Apache Flink

Cypher
MATCH (c1:Clan)<-[:LEADER_OF]-(o1:Orc),
(o1)-[:HATES]->(o2:Orc),
(o2)-[:LEADER_OF]->(c2:Clan),
(o2)-[:KNOWS*1..10]->(h:Hobbit)
WHERE NOT (c1 = c2 AND o1 = o2)
AND h.name = "Frodo"
RETURN o1.name, o2.name

Flink Gelly
or any other non-declarative
graph processing system

Graph Fundamentals - Property Graphs
:Orc
name: Helmut
weapon: Axe
weight: 199
:Hobbit
name: Bilbo
job: Burglar
:Clan
name: Saruman
members: 1337
:HATES
:LEADER_OF
since: 3204
:Hobbit
name: Sam
age: 42
:KNOWS

What is Cypher?
Neo4j’s declarative graph query language
● Used to insert, update and retrieve data from Neo4j
● Designed to be easily understood by people with SQL background
● Support for Pattern Matching, Filtering, Aggregation, Projection
● Results are (multidimensional) Tables
Specified, maintained and extended in the openCypher project
by several academic and industry contributors.
http://www.opencypher.org/

Cypher Query Syntax
Find all vertices with
label Clan and assign
them to c1
Traverse incoming edges of
type LEADER_OF
Describes the pattern that should be
matched
Filters the match results
Specifies which fields to return

Cypher Query Syntax

Apache Flink (DataSet API)
Apache Flink Operator Implementation
Extended Property Graph Model (EPGM)
Graph Dataflow Operators
I/O
Cypher
„An open-source graph dataflow framework for
declarative analytics of heterogeneous graph data.“

2
1|Area|title:Mordor
|Area|title:Shire
Extended Property Graph Model (EPGM)
1 2
3
4
5
1 2 3
4
5
:Hobbit
name : Samwise
:Orc
name : Azog
:Clan
name : Tribes of Moria
founded : 1981
:Orc
name : Bolg
:Hobbit
name : Frodo
yob : 2968
:LEADER_OF
since : 2790
:MEMBER_OF
since : 2013
:HATES
since : 2301
:HATES
:KNOWS
since : 2990
= Property Graph Model
+ Logical Graphs
+ Graph Transformations
○ Subgraph
○ Aggregation
○ Transformation
○ Grouping
○ Cypher
○ ...

Pattern
Graph Collection
LogicalGraph graph1 = new CSVDataSource(“hdfs:///path/to/graph”, conf).getLogicalGraph();
a b1
1 2 3
4
5
2|map:{a:1, b:2} 3|map:{a:3, b:4}
1 2 3
4
Cypher Operator
GraphCollection collection = graph1.cypher(pattern);
String pattern = “MATCH (a:Green)-[:orange]->(b:Orange)”;

Internal Representation
2|map:{a:1,e:10,b:2} 3|map:{a:3,e:30,b:4}
1 2 3
4
id label
1 Green
2 Orange
3 Green
4 Orange
5 Orange
id src trgt label
10 1 2 ORANGE
20 3 2 BLUE
30 3 4 ORANGE
40 3 5 BLUE
1
1 2 3
4
5
2010
30
40
DataSet<Vertex> DataSet<Edge>
a be
10
30
id label graphs
1 Green {2}
2 Orange {2}
3 Green {3}
4 Orange {4}
id src trgt label graphs
10 1 2 ORANGE {2}
30 3 4 ORANGE {3}
DataSet<Vertex> DataSet<Edge>
id label properties
2 Green map:{a:1, e:10, b:2}
3 Orange map:{a:3, e:30, b:4}
DataSet<GraphHead>

Parsing Execution
c1
o2 h
c2
o1
((c1 != c2) AND (o1 != o2)
AND (h.name = Frodo Baggins)
=> 23
=> 42
=> 84
=> 123
=> 456
=> 789
0
3 4
1
2
0
3 5
1
2 4
0
3 6
1
2 4
0
3 6
1
2
4
7
Planning
Query Overview

Parsing and Query Rewriting
o1 h
__edge01
Query Graph
(o1.weapon = "Axe" AND o1.weight > h.weight)
OR o1.weapon = "Sword"
(o1.weapon = "Axe" OR o1.weapon = "Sword")
AND
(o1.weight > h.weight OR o1.weapon = "Sword")
Conjunctive-Normal-Form
Transformation
ANTLR
MATCH (o1:Orc)-[:KNOWS]->(h:Hobbit)
WHERE (o1.weapon = "Axe" AND o1.weight > h.weight)
OR o1.weapon = "Sword"

Intermediate Result Representation - Embedding
1 2
10
Embedding = Mapping between Query graph and Input (Sub-)Graph
a be
Embedding f
f(a) = 1
f(e) = 10
f(b) = 2

Query operators - Filter and Project
Filter
WHERE h.name = ‘Frodo’
Project
DataSet<Vertex> DataSet<Embedding>
vertices.flatMap(FilterAndProject)
id label properties
1 Orc {...}
2 Clan {...}
3 Hobbit {...}
... ... ...
h.id h.name h.height ...
31 Frodo 1.22 ...
h.id
31

Query operators - Join Embeddings
DataSet<Embedding> lhs DataSet<Embedding>
lhs.flatJoin(rhs).with(Combine)DataSet<Embedding> rhs
Combine
JoinEmbeddings
Left: (c1:Clan)<-[:HAS_LEADER]-(o1:Orc)
Right: (o1:Orc)-[:HATES]->(o2.Orc)
c.id _e1.id o1.id
51 11 2
52 12 3
... ... ...
c.id _e1.id o1.id _e2.id o2.id
51 11 2 13 5
52 12 3 14 3
... ... ... ... ...
o1.id _e2.id o2.id
2 13 5
3 14 3
... ... ...

Query operators - Expand Embeddings
ExpandEmbeddings
Left: (o2:Orc)
Edges: (o2)-[:KNOWS*1..10]->(h:Hobbit)
DataSet<Embedding> lhs
DataSet<Embedding>
DataSet<Embedding> rhs
BulkIteration ws = lhs.join(rhs))
filteredPaths = ws.filter(filterByLength)
newPaths = filteredPaths.flatJoin(rhs, combine)
nextWs = ws.union(newPaths)
Combine
Check for vertex/edge isomorphism
o2.id
5
_e3.sid _e3.id _e3.tid
5 26 31
31 27 32
32 28 33
o2.id _e3.id h.id
3 [26] 31
3 [26,31,27] 32
3 [26,31,27,32,28] 33

Query operators - Filter embeddings
FilterEmbeddings
o1.sid _e2.id o2.tid
2 13 5
3 14 3
... ... ...
o1.sid _e2.id o2.tid
2 13 5
... ... ...
DataSet<Embedding> DataSet<Embedding>embeddings.filter(ByPredicate)

Cost-based greedy query planning
● Problem: Query can be computed in a factorial number of ways
○ Goal: Find a way (plan) with minimal / low computation costs
● Use statistics about the input graph
○ Vertex-/Edge counts by label, i.e., label distributions
○ Distinct value counts (source, target) by edge label
○ Property value distributions
● Cost calculation for computing intermediate results
○ Primarily based on join result estimation
○ Filters and projections are evaluated as early as possible
● Planner iteratively builds a physical query plan
○ Greedy: picks plan with minimum cost with each iteration

PlanTableEntry | type: GRAPH | all-vars: [...] | proc-vars: [...] | attr-vars: [] | est-card: 23 | prediates: () | Plan :
|-FilterEmbeddingsNode{filterPredicate=((c1 != c2) AND (o1 != o2))}
|.|-JoinEmbeddingsNode{joinVariables=[o2], vertexMorphism=H, edgeMorphism=I}
|.|.|-JoinEmbeddingsNode{joinVariables=[o1], vertexMorphism=H, edgeMorphism=I}
|.|.|.|-JoinEmbeddingsNode{joinVariables=[c1], vertexMorphism=H, edgeMorphism=I}
|.|.|.|.|-FilterAndProjectVerticesNode{vertexVar=c1, filterPredicate=((c1.label = Clan)), projectionKeys=[]}
|.|.|.|.|-FilterAndProjectEdgesNode{sourceVar='o1', edgeVar='_e0', targetVar='c1', filterPredicate=((_e0.label = leaderOf)), projectionKeys=[]}
|.|.|.|-JoinEmbeddingsNode{joinVariables=[o1], vertexMorphism=H, edgeMorphism=I}
|.|.|.|.|-FilterAndProjectVerticesNode{vertexVar=o1, filterPredicate=((o1.label = Orc)), projectionKeys=[]}
|.|.|.|.|-FilterAndProjectEdgesNode{sourceVar='o1', edgeVar='_e1', targetVar='o2', filterPredicate=((_e1.label = hates)), projectionKeys=[]}
|.|.|-JoinEmbeddingsNode{joinVariables=[o2], vertexMorphism=H, edgeMorphism=I}
|.|.|.|-JoinEmbeddingsNode{joinVariables=[h], vertexMorphism=H, edgeMorphism=I}
|.|.|.|.|-FilterAndProjectVerticesNode{vertexVar=h, filterPredicate=((h.label = Hobbit) AND (h.name = Frodo Baggins)), projectionKeys=[]}
|.|.|.|.|-ExpandEmbeddingsNode={startVar='o2', pathVar='_e3', endVar='h', lb=1, ub=10, direction=OUT, vertexMorphism=H, edgeMorphism=I}
|.|.|.|.|.|-FilterAndProjectVerticesNode{vertexVar=o2, filterPredicate=((o2.label = Orc)), projectionKeys=[]}
|.|.|.|.|.|-FilterAndProjectEdgesNode{sourceVar='o2', edgeVar='_e3', targetVar='h', filterPredicate=((_e3.label = knows)), projectionKeys=[]}
|.|.|.|-JoinEmbeddingsNode{joinVariables=[c2], vertexMorphism=H, edgeMorphism=I}
|.|.|.|.|-FilterAndProjectVerticesNode{vertexVar=c2, filterPredicate=((c2.label = Clan)), projectionKeys=[]}
|.|.|.|.|-FilterAndProjectEdgesNode{sourceVar='o2', edgeVar='_e2', targetVar='c2', filterPredicate=((_e2.label = leaderOf)), projectionKeys=[]}

Future work
● Optimizations
○ DP-Planner
○ Improve cost model (more statistics, Flink optimizer hints)
○ Reuse of intermediate results
● Support more Cypher features
○ e.g. Aggregation and Functions
● Introduce new Cypher features
○ e.g. regular path queries

Further reading / Contributing
Gradoop: http://www.gradoop.com
Demo: https://github.com/dbs-leipzig/gradoop_demo
Paper: https://event.cwi.nl/grades/2017/03-Junghanns.pdf
Neo4j: https://neo4j.com/
openCypher: http://www.openCypher.org

Flink Forward Berlin 2017: Max Kiessling, Martin Junghanns - Cypher-based Graph Pattern Matching in Apache Flink

More Related Content

What's hot (20)

Similar to Flink Forward Berlin 2017: Max Kiessling, Martin Junghanns - Cypher-based Graph Pattern Matching in Apache Flink (20)

More from Flink Forward (20)

Recently uploaded (20)

Flink Forward Berlin 2017: Max Kiessling, Martin Junghanns - Cypher-based Graph Pattern Matching in Apache Flink