Q con london2011-matthewwall-whyichosemongodbforguardiancouk
2 likes1,079 views
The document discusses the author's experience with database selection and migration for the guardian.co.uk website, describing how they evolved from a relational database to a NoSQL database (MongoDB) to address scalability and flexibility issues. It outlines the benefits of MongoDB for the author's use case, including flexible schema, simple data model mapping, and ease of queries. MongoDB was selected to build a new identity management system after issues were encountered with scaling the existing relational database solution.
Q con london2011-matthewwall-whyichosemongodbforguardiancouk
- 1. Why I chose mongodb for guardian.co.uk Mat Wall Lead Software Architect, guardian.co.uk
- 2. “It is not the strongest of the species that survives, nor the most intelligent. It is the one that is most adaptable to change.”
- 3. Early Period circa ’95 The “Lash It Together” era
- 4. Early Period (95, the “Lash It Together” era) Perl, CGI, apache Experimental Manual processes Bespoke software RDBMS, scripts & static files
- 5. Mid Period circa ’00 The “Vendor CMS” era
- 6. Mid Period: 2000s (The “Vendor CMS era”) Vignette / AOLserver TCL, Apache, Oracle Platform for online publishing Initially scales well with acceleration in delivery of features
- 7. Mid Period: 2000s (The “Vendor CMS era”) Surprise! Vendor’s CMS doesn’t do what we want! Mish-mash in templates: HTML, JavaScript, TCL, SQL, PL-SQL No model in app tier, only in RDBMS schema created in Oracle Designer
- 8. Mid Period: 2000s (The “Vendor CMS era”)
- 9. Mid Period: 2000s (The “Vendor CMS era”)
- 10. Mid Period: 2000s (The “Vendor CMS era”) After a few years, very difficult to extend Database schema becomes fixed due to dependencies in templates
- 11. Mid Period: 2000s (The “Vendor CMS era”) If you can’t change the system:
- 12. Modern Period circa ’05-09 The “J2EE Monolithic” era
- 14. Web server Web server Web server I bring you NEWS!!! App server App server App server Oracle CMS Data feeds
- 15. Web server Web server Web server Modern java app I bring you NEWS!!! App server App server App server Spring / Hibernate DDD / TDD Strong Oracle in java model Database abstracted away with ORM CMS Data feeds
- 16. Problems
- 17. Each release involves schema upgrade Schema upgrade = downtime for journalists
- 18. Complexity still increasing: 300+ tables, 10,000 lines of hibernate XML config 1,000 domain objects mapped to database 70,000 lines of domain object code Very tight binding to database
- 19. ORM not really masking complexity: Database has strong influence on domain model: many domain objects made more complex mapping joins in RDBMS Complex hibernate features used, interceptors, proxies Complex caching strategy Lots of optimisations And: We still hand code complex queries in SQL!
- 20. Load becoming an issue RDBMS difficult to scale
- 21. Partial NoSQL circa ’09-10 The “Sticking Plaster” era
- 22. Introduce yet more caching to patch up load problems Decouple applications from database by building APIs Power APIs using alternative, more scalable technologies APIs used to scale out database reads Writes still go to RDBMs
- 23. Core Api Web servers Solr/API App server Solr/API Memcached (20Gb) Solr/API rdbms Solr Solr/API M/Q Solr/API CMS Cloud, EC2
- 24. Content API Mutualised news! Apache Solr Read API delivered using Hosted in EC2 Document oriented search engine Loose schema: records, fields, facets Scales well for read operations
- 25. Related content from Solr Introduction of memcached
- 26. Mutualised news! We’ve solved our load problem (for now) but Increased our complexity
- 27. Mutualised news! We now have 3 models! RDBMS tables Java Objects JSON API
- 28. Mutualised news!
- 29. Mutualised news!
- 30. Mutualised news!
- 31. MutualisedAPI is very simple JSON news! Multiple domain concepts expressed in single document Can be designed in forwardly extensible way What if the JSON API was our primary model?
- 32. Full NoSQL in development The “It’s the future!” era
- 33. The first project: Identity Current login/registration system still in TCL/PL-SQL 3M+ users in relational database Very complex schema + PL-SQL New system required Can we migrate from Oracle to NoSql?
- 34. Database selection Simple keystore. Too simple? Huge scalability. Do we need it? Schema design difficult. Simple to use, can execute similar queries to RDBMs
- 35. MongoDB Mutualised news! database Document oriented Stores parsed JSON documents Can express complex queries Can be flexible about consistency Malleable schema: can easily change at runtime Can work at both large & small scales
- 36. MongoDB concepts Mutualised news! RDBMS MongoDB Table Collection Row JSON Document Index Index Join Embedding & Linking Partition Shard
- 39. Flexible Schema Mutualised news! Can easily represent different classes of tag as documents Both documents can be inserted into same collection Far simpler than equivalent hibernate mapped subclass configuration
- 40. Flexible Schema Simple to query: Mutualised news!
- 41. Flexible Schema Simple to query: Mutualised news! Query operators: $ne, $nin, $all, $exists, $gt, $lt, $gte ...
- 42. Modifying the schema Mutualised news!
- 43. Modifying the schema Mutualised news!
- 44. Modifying the schema Mutualised news!
- 45. Schema upgrades Mutualised news! Schema can be upgraded simply by upgrading the application version Application must deal with differing document versions Can become complex over time
- 46. Schema upgrades Mutualised news! by: This can be mitigated Adding a “version” key to each document Updating the version each time the application modifies a document Using MapReduce capability to forcibly migrate documents from older versions if required
- 47. Mongodb architecture mongod Single node Durability only possible in upcoming 1.8 release (databse fsync from buffer every min)
- 48. Mongodb architecture master replicas mongod mongod mongod Replica set mongod mongod Can choose to read & Can choose to run reads write from master for full on slaves to scale reads consistency
- 49. Mongodb architecture master replicas Durability achieved (<1.8) via replication mongod mongod Reads can be scaled out onto replicas mongod (eventual consistency) Replica set mongod All writes to master mongod If master fails, new master nominated by election Can choose to read & Can choose to accept dirty write from drivers handle most cluster complexity scale DB master for full reads from slaves to consistency reads
- 50. Mongodb architecture Aggregator mongos consistent shard shard shard shard (master) replica replica replica replica inconsistent (replica) replica replica replica replica replica replica replica replica
- 51. Mongodb architecture Writes scaled by sharding Aggregator mongos Shards populated by ranges consistent shard shard shard shard (master) mongos queries appropriate shard(s) Shards automatically balanced replica replica replica replica inconsistent (replica) replica replica replica replica Developers (essentially) unaware of shards replica replica replica replica
- 52. Mongodb durability Relies (pre 1.8) on replication for durability 1.8 features optional journaling & redo logs Database users need to be cluster aware, each query can specify: No error checking / write confirmation Write confirmed on master Write replicated to N slave servers
- 53. Old Idenity system Hundreds of tables & stored procedures Mutualised news! New Identity model User List Fields Text Dates Date/Time Statuses Boolean
- 55. Very simple domain objects Simple, flexible objects No hibernate session
- 56. Very simple domain objects Flexible schema embraced in domain object design
- 57. Very simple domain objects Using casbah scala drivers = significant reduction in LOC vs SQL implementation
- 58. Build API that can support both backends Registration app guardian.co.uk API MongoDB Oracle
- 59. Build API that can support both backends Registration app guardian.co.uk API This bit is hard! MongoDB Oracle
- 60. Migrate using API & decommision Registration app guardian.co.uk API MongoDB
- 61. Add new stuff! Registration app guardian.co.uk API MongoDB Solr? Redis?
- 62. MongoDB Simple, flexible schema with similar query & indexing to RDBMS Great at small or large scale Easy for developers to get going Commercial support available (10Gen) One day may power all of guardian.co.uk No transactions / joins: developers must cater for this Produces a net reduction in lines of code / complexity
- 63. Shameless plug We’re hiring: http://www.careersatgnl.co.uk