![Recap
Design documents that simply map to
your application
post
=
{author:
roger ,
date:
new
Date(),
text:
Down
Under... ,
tags:
[ rockstar , men
at
work ]}](https://image.slidesharecdn.com/datamodeling-melbournemongousergroup-120308000715-phpapp02/85/Intro-to-MongoDB-and-datamodeling-8-320.jpg)
![Extending the Schema
{ _id : ObjectId(4c4ba5c0672c685e5e8aabf3),
author : ”roger,
date : Sat Jul 24 2010 19:47:11 GMT-0700 (PDT),
text : ”Down
Under...,
tags : [ ”rockstar, ”men at work ],
comments_count: 1,
comments : [
{
author : ”Bruce,
date : Sat Jul 24 2010 20:51:03 GMT-0700 (PDT),
text : ” Love Men at Work!!!!
}
]}](https://image.slidesharecdn.com/datamodeling-melbournemongousergroup-120308000715-phpapp02/85/Intro-to-MongoDB-and-datamodeling-13-320.jpg)
![Many – Many
products:
{ _id: ObjectId(4c4ca23933fb5941681b912e),
name: Sumatra Dark Roast,
category_ids: [ ObjectId(4c4ca25433fb5941681b912f),
ObjectId(4c4ca25433fb5941681b92af”]}](https://image.slidesharecdn.com/datamodeling-melbournemongousergroup-120308000715-phpapp02/85/Intro-to-MongoDB-and-datamodeling-22-320.jpg)
![Many – Many
products:
{ _id: ObjectId(4c4ca23933fb5941681b912e),
name: Sumatra Dark Roast,
category_ids: [ ObjectId(4c4ca25433fb5941681b912f),
ObjectId(4c4ca25433fb5941681b92af”]}
categories:
{ _id: ObjectId(4c4ca25433fb5941681b912f),
name: Indonesia,
product_ids: [ ObjectId(4c4ca23933fb5941681b912e),
ObjectId(4c4ca30433fb5941681b9130),
ObjectId(4c4ca30433fb5941681b913a]}](https://image.slidesharecdn.com/datamodeling-melbournemongousergroup-120308000715-phpapp02/85/Intro-to-MongoDB-and-datamodeling-23-320.jpg)
![Many - Many
products:
{ _id: ObjectId(4c4ca23933fb5941681b912e),
name: Sumatra Dark Roast,
category_ids: [ ObjectId(4c4ca25433fb5941681b912f),
ObjectId(4c4ca25433fb5941681b92af”]}
categories:
{ _id: ObjectId(4c4ca25433fb5941681b912f),
name: Indonesia,
product_ids: [ ObjectId(4c4ca23933fb5941681b912e),
ObjectId(4c4ca30433fb5941681b9130),
ObjectId(4c4ca30433fb5941681b913a]}
//All categories for a given product
db.categories.find({product_ids: ObjectId(4c4ca23933fb5941681b912e)})](https://image.slidesharecdn.com/datamodeling-melbournemongousergroup-120308000715-phpapp02/85/Intro-to-MongoDB-and-datamodeling-24-320.jpg)
![Many - Many
products:
{ _id: ObjectId(4c4ca23933fb5941681b912e),
name: Sumatra Dark Roast,
category_ids: [ ObjectId(4c4ca25433fb5941681b912f),
ObjectId(4c4ca25433fb5941681b92af”]}
categories:
{ _id: ObjectId(4c4ca25433fb5941681b912f),
name: Indonesia,
product_ids: [ ObjectId(4c4ca23933fb5941681b912e),
ObjectId(4c4ca30433fb5941681b9130),
ObjectId(4c4ca30433fb5941681b913a]}
//All categories for a given product
db.categories.find({product_ids: ObjectId(4c4ca23933fb5941681b912e)})
//All products for a given category
db.products.find({category_ids: ObjectId(4c4ca25433fb5941681b912f)})](https://image.slidesharecdn.com/datamodeling-melbournemongousergroup-120308000715-phpapp02/85/Intro-to-MongoDB-and-datamodeling-25-320.jpg)
![Alternative
products:
{ _id: ObjectId(4c4ca23933fb5941681b912e),
name: Sumatra Dark Roast,
category_ids: [ ObjectId(4c4ca25433fb5941681b912f),
ObjectId(4c4ca25433fb5941681b92af”]}
categories:
{ _id: ObjectId(4c4ca25433fb5941681b912f),
name: Indonesia}](https://image.slidesharecdn.com/datamodeling-melbournemongousergroup-120308000715-phpapp02/85/Intro-to-MongoDB-and-datamodeling-26-320.jpg)
![Alternative
products:
{ _id: ObjectId(4c4ca23933fb5941681b912e),
name: Sumatra Dark Roast,
category_ids: [ ObjectId(4c4ca25433fb5941681b912f),
ObjectId(4c4ca25433fb5941681b92af”]}
categories:
{ _id: ObjectId(4c4ca25433fb5941681b912f),
name: Indonesia}
// All products for a given category
db.products.find({category_ids: ObjectId(4c4ca25433fb5941681b912f)})](https://image.slidesharecdn.com/datamodeling-melbournemongousergroup-120308000715-phpapp02/85/Intro-to-MongoDB-and-datamodeling-27-320.jpg)
![Alternative
products:
{ _id: ObjectId(4c4ca23933fb5941681b912e),
name: Sumatra Dark Roast,
category_ids: [ ObjectId(4c4ca25433fb5941681b912f),
ObjectId(4c4ca25433fb5941681b92af”]}
categories:
{ _id: ObjectId(4c4ca25433fb5941681b912f),
name: Indonesia}
// All products for a given category
db.products.find({category_ids: ObjectId(4c4ca25433fb5941681b912f)})
// All categories for a given product
product = db.products.find(_id : some_id)
db.categories.find({_id : {$in : product.category_ids}})](https://image.slidesharecdn.com/datamodeling-melbournemongousergroup-120308000715-phpapp02/85/Intro-to-MongoDB-and-datamodeling-28-320.jpg)
![Trees
Full Tree in Document
{ comments: [
{ author: rpb , text: ... ,
replies: [
{author: Fred , text: ... ,
replies: []}
]}
]}
Pros: Single Document, Performance, Intuitive
Cons: Hard to search, 16MB limit](https://image.slidesharecdn.com/datamodeling-melbournemongousergroup-120308000715-phpapp02/85/Intro-to-MongoDB-and-datamodeling-29-320.jpg)
![Array of Ancestors
- Store Ancestors of a node
{ _id: a }
{ _id: b, ancestors: [ a ], parent: a }
{ _id: c, ancestors: [ a, b ], parent: b }
{ _id: d, ancestors: [ a, b ], parent: b }
{ _id: e, ancestors: [ a ], parent: a }
{ _id: f, ancestors: [ a, e ], parent: e }
{ _id: g, ancestors: [ a, b, d ], parent: d }](https://image.slidesharecdn.com/datamodeling-melbournemongousergroup-120308000715-phpapp02/85/Intro-to-MongoDB-and-datamodeling-31-320.jpg)
![Array of Ancestors
- Store Ancestors of a node
{ _id: a }
{ _id: b, ancestors: [ a ], parent: a }
{ _id: c, ancestors: [ a, b ], parent: b }
{ _id: d, ancestors: [ a, b ], parent: b }
{ _id: e, ancestors: [ a ], parent: a }
{ _id: f, ancestors: [ a, e ], parent: e }
{ _id: g, ancestors: [ a, b, d ], parent: d }
//find all descendants of b:
db.tree2.find({ancestors: b })](https://image.slidesharecdn.com/datamodeling-melbournemongousergroup-120308000715-phpapp02/85/Intro-to-MongoDB-and-datamodeling-32-320.jpg)
![Array of Ancestors
- Store Ancestors of a node
{ _id: a }
{ _id: b, ancestors: [ a ], parent: a }
{ _id: c, ancestors: [ a, b ], parent: b }
{ _id: d, ancestors: [ a, b ], parent: b }
{ _id: e, ancestors: [ a ], parent: a }
{ _id: f, ancestors: [ a, e ], parent: e }
{ _id: g, ancestors: [ a, b, d ], parent: d }
//find all descendants of b:
db.tree2.find({ancestors: b })
//find all ancestors of f:
ancestors = db.tree2.findOne({_id: f }).ancestors
db.tree2.find({_id: { $in : ancestors})](https://image.slidesharecdn.com/datamodeling-melbournemongousergroup-120308000715-phpapp02/85/Intro-to-MongoDB-and-datamodeling-33-320.jpg)
Intro to MongoDB and datamodeling
- 1. Schema Design Roger Bodamer roger@analytica.com @rogerb
- 2. A brief history of Data Modeling • ISAM • COBOL • Network • Hiearchical • Relational • 1970 E.F.Codd introduces 1st Normal Form (1NF) • 1971 E.F.Codd introduces 2nd and 3rd Normal Form (2NF, 3NF • 1974 Codd Boyce define Boyce/Codd Normal Form (BCNF) • 2002 Date, Darween, Lorentzos define 6th Normal Form (6NF) • Object
- 3. So why model data?
- 4. Modeling goals Goals: • Avoid anomalies when inserting, updating or deleting • Minimize redesign when extending the schema • Make the model informative to users • Avoid bias towards a particular style of query * source : wikipedia
- 5. Relational made normalized data look like this
- 6. Document databases make normalized data look like this
- 7. Some terms before we proceed RDBMS Document DBs Table Collection View / Row(s) JSON Document Index Index Join Embedding Linking across documents Partition Shard Partition Key Shard Key
- 8. Recap Design documents that simply map to your application post = {author: roger , date: new Date(), text: Down Under... , tags: [ rockstar , men at work ]}
- 9. Query operators Conditional operators: $ne, $in, $nin, $mod, $all, $size, $exists, $type, .. $lt, $lte, $gt, $gte, $ne, // find posts with any tags db.posts.find({tags: {$exists: true}})
- 10. Query operators Conditional operators: $ne, $in, $nin, $mod, $all, $size, $exists, $type, .. $lt, $lte, $gt, $gte, $ne, // find posts with any tags db.posts.find({tags: {$exists: true}}) Regular expressions: // posts where author starts with k db.posts.find({author: /^r*/i })
- 11. Query operators Conditional operators: $ne, $in, $nin, $mod, $all, $size, $exists, $type, .. $lt, $lte, $gt, $gte, $ne, // find posts with any tags db.posts.find({tags: {$exists: true}}) Regular expressions: // posts where author starts with k db.posts.find({author: /^r*/i }) Counting: // posts written by mike db.posts.find({author: roger }).count()
- 12. Extending the Schema new_comment = {author: Bruce , date: new Date(), text: Love Men at Work!!!! } new_info = { $push : {comments: new_comment}, $inc : {comments_count: 1}} db.posts.update({_id: ... }, new_info)
- 13. Extending the Schema { _id : ObjectId(4c4ba5c0672c685e5e8aabf3), author : ”roger, date : Sat Jul 24 2010 19:47:11 GMT-0700 (PDT), text : ”Down Under..., tags : [ ”rockstar, ”men at work ], comments_count: 1, comments : [ { author : ”Bruce, date : Sat Jul 24 2010 20:51:03 GMT-0700 (PDT), text : ” Love Men at Work!!!! } ]}
- 14. Extending the Schema // create index on nested documents: db.posts.ensureIndex({comments.author: 1}) db.posts.find({comments.author:”Bruce”}) // find last 5 posts: db.posts.find().sort({date:-1}).limit(5) // most commented post: db.posts.find().sort({comments_count:-1}).limit(1) When sorting, check if you need an index
- 16. Modeling Patterns Single table inheritance One to Many Many to Many Trees Queues
- 17. Single Table Inheritance db.shapes.find() { _id: ObjectId(...), type: circle, area: 3.14, radius: 1} { _id: ObjectId(...), type: square, area: 4, d: 2} { _id: ObjectId(...), type: rect, area: 10, length: 5, width: 2} // find shapes where radius 0 db.shapes.find({radius: {$gt: 0}}) // create index db.shapes.ensureIndex({radius: 1})
- 18. One to Many - Embedded Array / Using Array Keys - slice operator to return subset of array - hard to find latest comments across all documents
- 19. One to Many - Embedded Array / Array Keys - slice operator to return subset of array - hard to find latest comments across all documents - Embedded tree - Single document - Natural
- 20. One to Many - Embedded Array / Array Keys - slice operator to return subset of array - hard to find latest comments across all documents - Embedded tree - Single document - Natural - Normalized (2 collections) - most flexible - more queries
- 21. Many - Many Example: - Product can be in many categories - Category can have many products Products Category - product_id - category_id Prod_Categories - id - product_id - category_id
- 22. Many – Many products: { _id: ObjectId(4c4ca23933fb5941681b912e), name: Sumatra Dark Roast, category_ids: [ ObjectId(4c4ca25433fb5941681b912f), ObjectId(4c4ca25433fb5941681b92af”]}
- 23. Many – Many products: { _id: ObjectId(4c4ca23933fb5941681b912e), name: Sumatra Dark Roast, category_ids: [ ObjectId(4c4ca25433fb5941681b912f), ObjectId(4c4ca25433fb5941681b92af”]} categories: { _id: ObjectId(4c4ca25433fb5941681b912f), name: Indonesia, product_ids: [ ObjectId(4c4ca23933fb5941681b912e), ObjectId(4c4ca30433fb5941681b9130), ObjectId(4c4ca30433fb5941681b913a]}
- 24. Many - Many products: { _id: ObjectId(4c4ca23933fb5941681b912e), name: Sumatra Dark Roast, category_ids: [ ObjectId(4c4ca25433fb5941681b912f), ObjectId(4c4ca25433fb5941681b92af”]} categories: { _id: ObjectId(4c4ca25433fb5941681b912f), name: Indonesia, product_ids: [ ObjectId(4c4ca23933fb5941681b912e), ObjectId(4c4ca30433fb5941681b9130), ObjectId(4c4ca30433fb5941681b913a]} //All categories for a given product db.categories.find({product_ids: ObjectId(4c4ca23933fb5941681b912e)})
- 25. Many - Many products: { _id: ObjectId(4c4ca23933fb5941681b912e), name: Sumatra Dark Roast, category_ids: [ ObjectId(4c4ca25433fb5941681b912f), ObjectId(4c4ca25433fb5941681b92af”]} categories: { _id: ObjectId(4c4ca25433fb5941681b912f), name: Indonesia, product_ids: [ ObjectId(4c4ca23933fb5941681b912e), ObjectId(4c4ca30433fb5941681b9130), ObjectId(4c4ca30433fb5941681b913a]} //All categories for a given product db.categories.find({product_ids: ObjectId(4c4ca23933fb5941681b912e)}) //All products for a given category db.products.find({category_ids: ObjectId(4c4ca25433fb5941681b912f)})
- 26. Alternative products: { _id: ObjectId(4c4ca23933fb5941681b912e), name: Sumatra Dark Roast, category_ids: [ ObjectId(4c4ca25433fb5941681b912f), ObjectId(4c4ca25433fb5941681b92af”]} categories: { _id: ObjectId(4c4ca25433fb5941681b912f), name: Indonesia}
- 27. Alternative products: { _id: ObjectId(4c4ca23933fb5941681b912e), name: Sumatra Dark Roast, category_ids: [ ObjectId(4c4ca25433fb5941681b912f), ObjectId(4c4ca25433fb5941681b92af”]} categories: { _id: ObjectId(4c4ca25433fb5941681b912f), name: Indonesia} // All products for a given category db.products.find({category_ids: ObjectId(4c4ca25433fb5941681b912f)})
- 28. Alternative products: { _id: ObjectId(4c4ca23933fb5941681b912e), name: Sumatra Dark Roast, category_ids: [ ObjectId(4c4ca25433fb5941681b912f), ObjectId(4c4ca25433fb5941681b92af”]} categories: { _id: ObjectId(4c4ca25433fb5941681b912f), name: Indonesia} // All products for a given category db.products.find({category_ids: ObjectId(4c4ca25433fb5941681b912f)}) // All categories for a given product product = db.products.find(_id : some_id) db.categories.find({_id : {$in : product.category_ids}})
- 29. Trees Full Tree in Document { comments: [ { author: rpb , text: ... , replies: [ {author: Fred , text: ... , replies: []} ]} ]} Pros: Single Document, Performance, Intuitive Cons: Hard to search, 16MB limit
- 30. Trees - continued Parent Links - Each node is stored as a document - Contains the id of the parent Child Links - Each node contains the id s of the children - Can support graphs (multiple parents / child)
- 31. Array of Ancestors - Store Ancestors of a node { _id: a } { _id: b, ancestors: [ a ], parent: a } { _id: c, ancestors: [ a, b ], parent: b } { _id: d, ancestors: [ a, b ], parent: b } { _id: e, ancestors: [ a ], parent: a } { _id: f, ancestors: [ a, e ], parent: e } { _id: g, ancestors: [ a, b, d ], parent: d }
- 32. Array of Ancestors - Store Ancestors of a node { _id: a } { _id: b, ancestors: [ a ], parent: a } { _id: c, ancestors: [ a, b ], parent: b } { _id: d, ancestors: [ a, b ], parent: b } { _id: e, ancestors: [ a ], parent: a } { _id: f, ancestors: [ a, e ], parent: e } { _id: g, ancestors: [ a, b, d ], parent: d } //find all descendants of b: db.tree2.find({ancestors: b })
- 33. Array of Ancestors - Store Ancestors of a node { _id: a } { _id: b, ancestors: [ a ], parent: a } { _id: c, ancestors: [ a, b ], parent: b } { _id: d, ancestors: [ a, b ], parent: b } { _id: e, ancestors: [ a ], parent: a } { _id: f, ancestors: [ a, e ], parent: e } { _id: g, ancestors: [ a, b, d ], parent: d } //find all descendants of b: db.tree2.find({ancestors: b }) //find all ancestors of f: ancestors = db.tree2.findOne({_id: f }).ancestors db.tree2.find({_id: { $in : ancestors})
- 34. Variable Keys How to index ? { _id : uuid1, field1 : { ctx1 : { ctx3 : 5, … }, ctx8 : { ctx3 : 5, … } }} db.MyCollection.find({ field1.ctx1.ctx3 : { $exists : true} }) Rewrite: { _id : uuid1, field1 : { key: ctx1 , value : { k:ctx3 , v : 5, … }, key: ctx8 , value : { k: ctx3 , v : 5, … } }} db.x.ensureIndex({ field1.key.k , 1})
- 35. findAndModify Queue example //Example: find highest priority job and mark job = db.jobs.findAndModify({ query: {inprogress: false}, sort: {priority: -1), update: {$set: {inprogress: true, started: new Date()}}, new: true})
- 36. Thanks !