How to Import JSON Using Cypher and APOC

© 2022 Neo4j, Inc. All rights reserved.
How to Import JSON Using
Cypher and APOC
Eric Monk,
Principal Solutions Engineer

2
JSON Data Overview

Data we will use
• Start with :play movies
• Run Cypher that creates the sample data
◦ Movie, Person
◦ ACTED_IN, DIRECTED, …
• Using Cypher + APOC the following files have been exported
◦ play-movies-db.json
◦ play-movies-roles.json
• Files exist in the <db_root>/import directory
3

DB Export Cypher
WITH "
MATCH (p:Person)-[r:ACTED_IN]->(m:Movie)
WHERE r.roles IS NOT NULL
WITH apoc.coll.sort(apoc.coll.toSet(apoc.coll.flatten(collect(r.roles)))) as allRoles
UNWIND range(1, size(allRoles), 1) as index
WITH apoc.map.fromPairs(collect([allRoles[index], index])) as roleIndexMap
MATCH (m:Movie)
WITH m,
[(m)<-[r:ACTED_IN]-(person:Person) | {
roleIndexes: [x IN r.roles | roleIndexMap[x]],
person: person { .name, .born }
}] as actors,
[(m)<-[:DIRECTED]-(person:Person) | person { .name, .born }] as directors,
[(m)<-[:PRODUCED]-(person:Person) | person { .name, .born }] as producers,
[(m)<-[:WROTE]-(person:Person) | person { .name, .born }] as writers,
[(m)<-[r:REVIEWED]-(person:Person) | {
review: r { .rating, .summary },
person: person { .name, .born }
}] as reviewers
4
Converting
roles to
indexes
makes import
more difficult
Relationships
with
properties
have deeper
nesting

DB Export Cypher (continued)
WITH apoc.map.merge({
title: m.title,
tagline: m.tagline,
released: m.released,
actors: actors,
producers: producers,
directors: directors,
writers: writers
}, CASE WHEN size(reviewers) = 0 THEN {} ELSE { reviewers: reviewers } END) as movieJson
RETURN movieJson
" as query
CALL apoc.export.json.query(query, "play-movies-db.json")
YIELD file, source, format, nodes, relationships, properties, time, rows, batchSize, batches, done, data
RETURN file, source, format, nodes, relationships, properties, time, rows, batchSize, batches, done,
data
Note: apoc.export.file.enabled=true needs to in neo4j.conf
5
reviewers is
optionally
present

6
DB JSON
{"movieJson":{"actors":[{"person":{"born":1978,"name":"Emil
Eifrem"},"roleIndexes":[55]},...
{"movieJson":{"actors":[{"person":{"born":1960,"name":"Hugo
Weaving"},"roleIndexes":[4]},...
{"movieJson":{"actors":[{"person":{"born":1960,"name":"Hugo
Weaving"},"roleIndexes":[4]},...
{"movieJson":{"actors":[{"person":{"born":1940,"name":"Al
Pacino"},"roleIndexes":[99]},...
{"movieJson":{"actors":[{"person":{"born":1967,"name":"James
Marshall"},"roleIndexes":[141]},...
{"movieJson":{"actors":[{"person":{"born":1959,"name":"Val
Kilmer"},"roleIndexes":[80]},...
{"movieJson":{"actors":[{"person":{"born":1974,"name":"Jerry
O'Connell"},"roleIndexes":[62]},...
We have a
file with a
JSON string
on each line

7
DB JSON Example 1
{
"movieJson": {
"actors": [
{
"person": { "born": 1960, "name": "Hugo Weaving" },
"roleIndexes": [4]
},
...
],
"directors": [
{ "born": 1965, "name": "Lana Wachowski" },
{ "born": 1967, "name": "Lilly Wachowski" }
],
"tagline": "Welcome to the Real World",
"writers": [],
"title": "The Matrix",
"released": 1999,
"producers": [{ "born": 1952, "name": "Joel Silver" }]
}
}
We'll need to
look this up
before writing
reviewers is
missing

8
DB JSON Example 2
{
"movieJson": {
"actors": [
{
"person": { "born": 1974, "name": "Jerry O'Connell" },
"roleIndexes": [62]
},
...
],
"directors": [{ "born": 1957, "name": "Cameron Crowe" }],
"tagline": "The rest of his life begins now.",
"writers": [{ "born": 1957, "name": "Cameron Crowe" }],
"title": "Jerry Maguire",
"reviewers": [
{
"person": { "born": null, "name": "Jessica Thompson" },
"review": { "summary": "You had me at Jerry", "rating": 92 }
}
],
"released": 2000,
"producers": [{ "born": 1957, "name": "Cameron Crowe" }]
}
}
reviewers is
present

Roles Export Cypher
WITH "
MATCH (p:Person)-[r:ACTED_IN]->(m:Movie)
WHERE r.roles IS NOT NULL
WITH apoc.coll.sort(apoc.coll.toSet(apoc.coll.flatten(collect(r.roles)))) as
allRoles
UNWIND range(1, size(allRoles), 1) as index
WITH collect({roleId: index, role: allRoles[index]}) as allRoles
RETURN { roles: allRoles } as rolesJson
" as query
CALL apoc.export.json.query(query, "play-movies-roles.json")
YIELD file, source, format, nodes, relationships, properties, time, rows,
batchSize, batches, done, data
RETURN file, source, format, nodes, relationships, properties, time, rows,
batchSize, batches, done, data
9
Use a roleId
to identify a
role

10
Play Movies Roles JSON
{
"rolesJson": {
"roles": [
{ "role": ""Wild Bill" Wharton", "roleId": 1 },
{ "role": "Ace Merrill", "roleId": 2 },
{ "role": "Admiral", "roleId": 3 },
{ "role": "Agent Smith", "roleId": 4 },
{ "role": "Albert Goldman", "roleId": 5 },
...more roles…
{ "role": "Walter", "roleId": 181 },
{ "role": "Warden Hal Moores", "roleId": 182 },
{ "role": "Zachry", "roleId": 183 },
{ "role": null, "roleId": 184 }
]
}
}
We have a
null we'll
have to deal
with

11
JSON Exploration

Exploring the JSON
• Use apoc.load.jsonArray(arg1, arg2)
• Arg1 - path to JSON file
• Arg2 - (optional) JSON path to fetch nested data
Example
CALL apoc.load.jsonArray('file:///play-movies-db.json', '$.movieJson')
Note
apoc.import.file.enabled=true needs to in neo4j.conf
12
file:/// points to /import directory

Run Examples in Neo4j Browser
• E1: Examine Movies JSON
• E2: Examine Roles JSON
• E3: Select Movies JSON w/Path
• E4: Select Roles JSON w/Path
• E5: Look at Movie Properties
13

14
JSON Import

Loading from JSON
We want to…
• CREATE/MERGE nodes
• CREATE/MERGE relationships
• Set node and relationship properties
• Handle optional data
• Lookup data from another file
15

L1: Load Movies
CALL apoc.load.jsonArray('file:///play-movies-db.json',
'$.movieJson') YIELD value
WITH value as movieData
MERGE (m:Movie {title: movieData.title})
SET m += {
tagline: movieData.tagline,
released: movieData.released // <-- note you could do
something like toInteger() here but in this case it's already a
number
}
16
MERGE
Movie on
title
Set properties
Convert /
data quality
check if
needed

L2: Load Directors
// L2: Load Directors
// ...previous Cypher...
FOREACH (directorInfo IN movieData.directors |
MERGE (p:Person {name: directorInfo.name})
SET p.born = directorInfo.born
MERGE (p)-[:DIRECTED]->(m)
)
17
Use FOREACH
loops through directors array
MERGE Person on name
Set born property
Add DIRECTED relationship

L3: Add Writers and Producers
// L3: Add writers and producers
// writers
FOREACH (personInfo IN movieData.writers |
MERGE (p:Person {name: personInfo.name})
SET p.born = personInfo.born
MERGE (p)-[:WROTE]->(m)
)
// producers
FOREACH (personInfo IN movieData.producers |
MERGE (p:Person {name: personInfo.name})
SET p.born = personInfo.born
MERGE (p)-[:PRODUCED]->(m)
)
18
Using same pattern as before
loops through arrays
MERGE nodes
set properties
create relationships

L4.1: Add Reviewers - Option 1
// L4.1: Add Reviewers - Option 1
// reviewers - option 1
FOREACH (reviewer IN movieData.reviewers |
MERGE (p:Person {name: reviewer.person.name})
SET p.born = reviewer.person.born
MERGE (p)-[r:REVIEWED]->(m)
SET r += reviewer.review
)
Works because no additional processing required,
but that's not always the case…
19
Using same pattern as before
Note extra level of nesting
reviewer.person.name
reviewer.person.born

L4.2: Add Reviewers - Option 2
// L4: Add Reviewers - Option 2
WITH m, movieData
CALL apoc.do.when(movieData.reviewers IS NOT NULL,
"
UNWIND reviewers as reviewer
MERGE (p:Person {name: reviewer.person.name})
SET p.born = reviewer.person.born
MERGE (p)-[r:REVIEWED]->(m)
SET r += reviewer.review
WITH collect(reviewer) as _
RETURN 1 as result
",
"
RETURN 0 as result
", { m: m, reviewers: movieData.reviewers }) YIELD value
20
Use apoc.do.when for
conditional processing
If NOT NULL, perform logic to
MERGE, SET
If NULL, do nothing

L5.1: Return Roles as a Map
// L5.1 - Return Roles as a Map
WITH {
jsonFile: 'file:///play-movies-roles.json',
jsonPath: '$.rolesJson.roles'
} as params
CALL apoc.load.jsonArray(params.jsonFile,
params.jsonPath) YIELD value
RETURN apoc.map.fromPairs(collect([value.roleId,
value.role])) as roleMap
Result
{
"1": ""Wild Bill" Wharton",
"2": "Ace Merrill",
"3": "Admiral",
...
}
21
Use apoc.map.fromPairs
to create a map

L5.2: Load Roles First
// L5.2 - Load Actors and Roles
WITH {
jsonFile: 'file:///play-movies-db.json',
jsonPath: '$.movieJson',
roleJsonFile: 'file:///play-movies-roles.json',
roleJsonPath: '$.rolesJson.roles'
} as params
CALL apoc.load.jsonArray(params.roleJsonFile,
params.roleJsonPath) YIELD value
WITH params, apoc.map.fromPairs(collect([value.roleId,
value.role])) as roleMap
CALL apoc.load.jsonArray(params.jsonFile,
params.jsonPath) YIELD value
WITH params, roleMap, value as movieData
22
Load roles JSON
Create roleMap
Load movies JSON
Pass roleMap in WITH

L5.2: Load Actors
// ...previous Cypher…
// actors
FOREACH (actor IN movieData.actors |
MERGE (p:Person {name: actor.person.name})
SET p.born = actor.person.born
MERGE (p)-[r:ACTED_IN]->(m)
SET r += {
roles: [x IN
[roleIndex IN actor.roleIndexes |
roleMap[toString(roleIndex)]]
WHERE x IS NOT NULL | x
]
}
)
23
For each roleIndex use
roleMap to get the role
string
Filter out NULL by
wrapping it another list
comprehension

24
Summary

Summary
• Use apoc.load.jsonArray() to load JSON
• MERGE create nodes and relationships, SET to set properties
• FOREACH to process JSON lists
• Use apoc.do.when for conditional processing
• Convert "lookup" JSON file to map, and make it available via WITH
25

26
Thank you!
Contact us at
sales@neo4j.com

How to Import JSON Using Cypher and APOC

More Related Content

What's hot (20)

Similar to How to Import JSON Using Cypher and APOC (20)

More from Neo4j (20)

Recently uploaded (20)

How to Import JSON Using Cypher and APOC