How medium uses Neo4j
Download as PPTX, PDF3 likes808 views
This document discusses how Medium uses Neo4j as its graph database. It begins with an introduction to Medium and its goals. It then describes Medium's evaluation of different data store options before choosing Neo4j for its ability to model Medium's social network graph and enable complex queries. The rest of the document outlines Medium's architecture using Neo4j, process of migrating to Neo4j, techniques for optimizing performance, and includes a live demo of Neo4j queries on Medium's data.
How medium uses Neo4j
- 1. How Medium uses Neo4j Nathaniel Felsen May 20th, 2015
- 2. About Me Nathaniel Felsen Data & DevOps engineer nathanielfelsen n@medium.com @faitlezen
- 3. Agenda • What is Medium and what problem we are solving through Neo4j ? • Why did we pick Neo4j ? • Our architecture • Steps taken and obstacles encountered while going live • Improve Neo4j’s performances • Live Demo
- 4. • Easiest, fastest way to create a beautiful story • Seamless integration of photos, audio & video • Optimized for web, tablet & mobile Medium is a beautiful publishing experience.
- 5. Medium is a home for influential contributors.
- 6. Medium is a place for important ideas.
- 7. • Follow, share, recommend • Personalized story feed • Customized daily emails & notifications Medium is a network that builds audience.
- 9. DynamoDB • Need to nail the schema ≠ Experimentation • Limited ways of querying data • Things like short path between users won’t perform well Pros • Expertise • Already used to store user info • No maintenance • No hardware Cons
- 10. • Using Relational database to create graphs • Sharding Pros • Used by lots of people and heavily vetted. • Less rampup for learning querying language. • Strong community. Cons
- 11. Pros Flock DB • Not maintained anymore • 2 tiers model • Deal with sharding in the near future • Expertise • SQL Lite syntax • Open Source / Free Cons
- 12. • Not free • No expertise in house • Requires hardware Pros • Easy to start • Easy to experiment • Good community • Enteprise edition: HA, Backup, Support Cons
- 13. Architecture
- 14. Our Social Service Architecture
- 15. Nodes • User • Post • Collection Relationships • Edited • Wrote • Published • Recommended • Followed • …
- 16. Use queues for the writes Write are done to the master only If you lose your master, you need to wait for a new election
- 20. Metrics / Monitoring • Architecture • Systems • Neo4j • Dataset • Java • Services that interact with Neo4j
- 21. Logs aggregation / Indexing • ElasticSearch • Logstash • Logstash forwarder • Kibana
- 22. Backups • Incremental Backup • Full Backup
- 24. Getting optimal performance with Neo4j
- 25. Talk to the support
- 26. What Neo4j is good and not as good at Long Traversal Where NOT Dense / Super Node
- 28. Tune the configuration over time • Java Garbage collection (stop the world) • Neo4j settings
- 30. Neo4J2.2
- 31. Server Plugins & Unmanaged Extensions • Easy to Deploy • The server’s functionality can be extended by adding plugins. • RESTful Web Services (JAX-RS) • Put more logic in the code like caching • Sharp tool
- 32. Demo
- 33. Followers who recommended a story
- 35. People Recommended to follow