ETL into Neo4j
Download as PPTX, PDF16 likes16,875 views
This document summarizes Max De Marzi's presentation on ETL (extract, transform, load) processes for loading data into Neo4j. It discusses using the Neo4j REST API, Gremlin and Groovy, and the Neo4j Batch Importer for ETL. It also provides an example of ETL from a SQL database by identifying relationships between rows and importing the data without node IDs.
![START user = node(1)
MATCH user -[user_skill]-> skill
RETURN skill, user_skill](https://image.slidesharecdn.com/etlintoneo4j-120531001746-phpapp02/85/ETL-into-Neo4j-8-320.jpg)
![name: “Canada”
languages_spoken: “[ „English‟, „French‟ ]”
language:“English” spoken_in
name: “USA”
name: “Canada”
language:“French” spoken_in
name: “France”](https://image.slidesharecdn.com/etlintoneo4j-120531001746-phpapp02/85/ETL-into-Neo4j-11-320.jpg)
ETL into Neo4j
- 1. ETL into Neo4j Max De Marzi
- 2. About Me Built the Neography Gem (Ruby Wrapper to the Neo4j REST API) Playing with Neo4j since 10/2009 • My Blog: http://maxdemarzi.com • Find me on Twitter: @maxdemarzi • Email me: maxdemarzi@gmail.com • GitHub: http://github.com/maxdemarzi
- 3. Agenda • ETL your mind • ETL with Batch and the REST API • ETL with Gremlin and Groovy • ETL with the Batch Importer • ETL from SQL
- 4. ETL your Mind You have to start there
- 5. More Relational than Relational Stop thinking about how Start thinking about relationships Tables are related
- 6. Objects like to mingle Optimized for “trees” of data Optimized for seeing the forest and the trees, and the branches, and the trunks
- 7. SELECT skills.*, user_skill.* FROM users JOIN user_skill ON users.id = user_skill.user_id JOIN skills ON user_skill.skill_id = skill.id WHERE users.id = 1
- 8. START user = node(1) MATCH user -[user_skill]-> skill RETURN skill, user_skill
- 10. Language LanguageCountry Country language_code language_code country_code language_name country_code country_name word_count primary flag_uri Language Country name name IS_SPOKEN_IN code code word_count as_primary flag_uri
- 11. name: “Canada” languages_spoken: “[ „English‟, „French‟ ]” language:“English” spoken_in name: “USA” name: “Canada” language:“French” spoken_in name: “France”
- 12. Country name flag_uri language_name number_of_words yes_in_langauge no_in_language currency_code currency_name Country Language name name flag_uri SPEAKS number_of_words yes no Currency code name
- 13. ETL with Batch and the REST API
- 14. Batch command from REST API Great for importing Facebook/Twitter friends Keep each request under 10k commands Preferably send a request every 2k to 5k commands
- 15. Using Batch from Neography
- 16. Why Batch Transactional: any failures not committed. Ordered: responses guaranteed to be in the same order as sent. Continuous loading/updating nodes and relationships in spurts or streaming.
- 17. ETL with Gremlin and Groovy
- 18. Commit every 1000 changes or so, make sure to stop the transaction to commit the last few changes at the very end. Look into auto-indexing to make life easier. Disabled by default. See Docs for trick to make it full text instead of exact index. http://docs.neo4j.org/chunked/milestone/auto-indexing.html
- 19. Crazy Format is ok Id :: Title :: Genre|Genre|Genre But it’s preferable to stay clear of escape characters like “|” String location of data file, converted to URL, then processed one line at a time. Movie vertex created, genre vertex created unless it exists (index lookup), edge from movie to genre is created. Full walk-through on http://maxdemarzi.com/2012/01/13/neo4j-on-heroku-part- one/
- 20. ETL with the Batch Importer
- 22. Testing it 7.5M nodes, 42M relationships in just over 3 minutes on a laptop.
- 23. Loading it into Neo4j Full walk-through on http://maxdemarzi.com/2012/02/28/batch-importer-part-1/
- 24. When to use the Batch Importer? • 1st time loading or periodic reloading • When you need Speed • When you don’t mind a little Java
- 25. ETL from SQL
- 26. Identities who vouched for each other row_number() and INTO are our friends
- 27. The “term” vouched for will serve as our relationship type, status is a relationship property.
- 28. Notice there are no node ids. These are automatic, clkao is node 1
- 29. No time to get coffee >8-[
- 30. What about multiple types of nodes? No problem, just add the MAX(node_id) from the first table. Full walk-through at: http://maxdemarzi.com/2012/02/28/batch-importer-part-2/ Need help? E-mail me, catch me on Google chat or Skype. Please don’t be shy…. and read my blog: http://maxdemarzi.com