Graph Databases: insight, scandal and the speed you always wanted!
0 likes1,000 views
The document discusses the use of graph databases, specifically Neo4j, for investigative journalism, highlighting their applications in analyzing data from the Panama and Paradise papers. It outlines various graph algorithms, community detection methods, and the benefits of graph models in revealing connections within complex datasets. The document also emphasizes collaboration among journalists globally to uncover hidden financial dealings of public officials through effective data modeling.
![Cypher: Powerful and Expressive Query
Language
CREATE (:Person { name:“Dan”} ) -[:LOVES]-> (:Person { name:“Ann”} )
LOVES
Dan Ann
LABEL PROPERTY
NODE NODE
LABEL PROPERTY](https://image.slidesharecdn.com/smmeetup2019-190320034239/85/Graph-Databases-insight-scandal-and-the-speed-you-always-wanted-31-320.jpg)
![(you)-[:HAVE]->(questions)<-[:ANSWERS]-(neo)](https://image.slidesharecdn.com/smmeetup2019-190320034239/85/Graph-Databases-insight-scandal-and-the-speed-you-always-wanted-53-320.jpg)
Graph Databases: insight, scandal and the speed you always wanted!
- 1. Offshore Leaks: Graph Databases For Investigative Journalism Justin Fine Santa Monica, CA Feb 2019 Panama Papers Paradise Papers
- 2. 2 GRAPHTOUR – San Francisco https://neo4j.com/graphtour/ https://www.eventbrite.com/e/graphtour-san-francisco-ca-tickets-58600670182
- 4. Machine Learning and Graph Algorithms in Neo4j • Parallel Breadth First Search & DFS • Shortest Path • Single-Source Shortest Path • All Pairs Shortest Path • Minimum Spanning Tree • A* Shortest Path • Yen’s K Shortest Path • K-Spanning Tree (MST) • Degree Centrality • Closeness Centrality • CC Variations: Harmonic, Dangalchev, & Wasserman & Faust • Betweenness Centrality • Approximate Betweenness Centrality • PageRank • Personalized PageRank • ArticleRank • Triangle Count • Clustering Coefficients • Connected Components (Union Find) • Strongly Connected Components • Label Propagation • Louvain Modularity – 1 Step & Multi-Step • Balanced Triad (identification) • Euclidean Distance • Cosine Similarity • Jaccard Similarity • Overlap Similarity • Random Walk • One Hot Encoding Reference Implementations for Graph Embeddings (Node to Vector) • DeepGL • DeepWalk Pathfinding & Search Centrality / Importance Community Detection Similarity & ML Workflow https://neo4j.com/docs/graph-algorithms/3.5/
- 5. Justin Fine Field Engineer , Neo4j Justin.Fine@neo4j.com https://www.linkedin.com/in/justin-fine- a7604523 https://github.com/JustinFineNeo/ IntroToNeo4J
- 19. What’s a Graph Database???
- 20. Chart Graph
- 22. Common Use Case ● Real time recommendations ● Fraud Detection ● Network & IT Management ● Social Networks ● Bill of Materials ● Knowledge Graphs ● Master Data Management ● Access Management ● Microservices Analysis ● IoT ● ...
- 23. Fixed Sized Records “Joins” on Creation Spin Spin Spin through this data structure Pointers instead of Lookups1 2 3 4 Neo4j Secret Sauce
- 24. Neo4j Mission Statement To help the world make sense of data
- 28. SQLER Diagrams
- 29. Relational Versus Graph Models Relational Model Graph Model KNOWS KNOWS KNOWS ANDREAS TOBIAS MICA DELIA Person FriendPerson- Friend ANDREAS DELIA TOBIAS MICA
- 30. Property Graph Model Components Nodes • The objects in the graph • Can have name-value properties • Can be labeled Relationships • Relate nodes by type and direction • Can have name-value properties CAR DRIVES name: “Dan” born: May 29, 1970 twitter: “@dan” name: “Ann” born: Dec 5, 1975 since: Jan 10, 2011 brand: “Volvo” model: “V70” LOVES LOVES LIVES WITH OW NS PERSON PERSON
- 31. Cypher: Powerful and Expressive Query Language CREATE (:Person { name:“Dan”} ) -[:LOVES]-> (:Person { name:“Ann”} ) LOVES Dan Ann LABEL PROPERTY NODE NODE LABEL PROPERTY
- 32. Cypher Query Language SQL for graphs
- 34. Almost 200 journalists Based in 65 countries “Our aim is to bring journalists from different countries together in teams - eliminating rivalry and promoting collaboration. Together, we aim to be the world’s best cross-border investigative team.” icij.org/about
- 35. Exposed the offshore holdings of 12 current and former world leaders. And dealings of 128 more politicians and public officials around the world. Exposure of hidden Secrets
- 37. INSIDE THE 2.6 TB
- 38. Context is King name: “John” last: „Miller“ role: „Negotiator“ name: "Maria" last: "Osara" name: “Some Media Ltd” value: “$70M” PERSON PERSON $ @ name: ”Jose" last: “Pereia“ position: “Governor“ name: “Alice” last: „Smith“ role: „Advisor“ PERSON PERSON
- 39. Context is King name: “John” last: „Miller“ role: „Negotiator“ name: "Maria" last: "Osara" name: “Some Media Ltd” value: “$70M” PERSON PERSON $ @ name: ”Jose" last: “Pereia“ position: “Governor“ name: “Alice” last: „Smith“ role: „Advisor“ PERSON MENTIONS SUPPORTS PERSONSENT WROTE CONTAINS OW NS since: Jan 10, 2011 CREATED
- 40. We need a data model Meta Data Entities • Document, Email, Contract, DB- Record • Meta: Author, Date, Source, Keywords • Conversation: Sender, Receiver, Topic • Money Flows Actual Entities • Person • Representative (Officer) • Address • Client • Company • Account Either based on our use cases & questions On the entities present in our meta-data and data.
- 41. Data model – relationships Meta-Data • sent, received, cc‘ed • mentioned, topic-of • created, signed • attached • roles • family relationships Activities • open account • manage • has shares • registered address • money flow
- 42. The Basic ICIJ Data Model
- 43. The Real ICIJ Data Model
- 44. The Predictions
- 45. Actually found because of Neo4j
- 47. Bit more connected though
- 48. Make Sense Of Data https://neo4j.com/ sandbox-v2/
- 49. Neo4j Desktop For ICIJ https://offshoreleaks.icij.org/pages/database