Retail Reference Architecture Part 2: Real-Time, Geo Distributed Inventory
Download as PPTX, PDF6 likes3,292 views
The document outlines a retail reference architecture using MongoDB to create a real-time, geo-distributed inventory system that addresses challenges in traditional inventory management. Key features include a single view of inventory accessible across channels, local and real-time data updates, and support for rapid application development. It details technical decisions, schema design, and querying strategies for efficient inventory data handling, including aggregations and geospatial queries.
![> db.stores.findOne()
{
"_id" : ObjectId("53549fd3e4b0aaf5d6d07f35"),
"className" : "catalog.Store",
"storeId" : "store0",
"name" : "Bessemer store",
"address" : {
"addr1" : "1st Main St",
"city" : "Bessemer",
"state" : "AL",
"zip" : "12345",
"country" : "US"
},
"location" : [ -86.95444, 33.40178 ],
...
}
Stores – Sample Document](https://image.slidesharecdn.com/mongodbworldinventorypublic-140805153530-phpapp01/85/Retail-Reference-Architecture-Part-2-Real-Time-Geo-Distributed-Inventory-10-320.jpg)
![Stores – Sample Geo Queries
• Get nearby stores sorted by distance
db.runCommand({
geoNear : "stores",
near : {
type : "Point",
coordinates : [-82.8006, 40.0908] },
maxDistance : 10000.0,
spherical : true
})](https://image.slidesharecdn.com/mongodbworldinventorypublic-140805153530-phpapp01/85/Retail-Reference-Architecture-Part-2-Real-Time-Geo-Distributed-Inventory-13-320.jpg)
![Stores – Sample Geo Queries
• Get the five nearest stores within 10 km
db.stores.find({
location : {
$near : {
$geometry : {
type : "Point",
coordinates : [-82.80, 40.09] },
$maxDistance : 10000.0 } }
}).limit(5)](https://image.slidesharecdn.com/mongodbworldinventorypublic-140805153530-phpapp01/85/Retail-Reference-Architecture-Part-2-Real-Time-Geo-Distributed-Inventory-14-320.jpg)
![> db.inventory.findOne()
{
"_id": "5354869f300487d20b2b011d",
"storeId": "store0",
"location": [-86.95444, 33.40178],
"productId": "p0",
"vars": [
{ "sku": "sku1", "q": 14 },
{ "sku": "sku3", "q": 7 },
{ "sku": "sku7", "q": 32 },
{ "sku": "sku14", "q": 65 },
...
]
}
Inventory – Sample Document](https://image.slidesharecdn.com/mongodbworldinventorypublic-140805153530-phpapp01/85/Retail-Reference-Architecture-Part-2-Real-Time-Geo-Distributed-Inventory-16-320.jpg)
![Inventory – Sample Aggregations
• Aggregate total quantity for a product
db.inventory.aggregate( [
{ $match : { productId : "p200" } },
{ $unwind : "$vars" },
{ $group : {
_id : "result",
count : { $sum : "$vars.q" } } } ] )
{ "_id" : "result", "count" : 101752 }](https://image.slidesharecdn.com/mongodbworldinventorypublic-140805153530-phpapp01/85/Retail-Reference-Architecture-Part-2-Real-Time-Geo-Distributed-Inventory-20-320.jpg)
![Inventory – Sample Aggregations
• Aggregate total quantity for a store
db.inventory.aggregate( [
{ $match : { storeId : "store100" } },
{ $unwind : "$vars" },
{ $match : { "vars.q" : { $gt : 0 } } },
{ $group : {
_id : "result",
count : { $sum : 1 } } } ] )
{ "_id" : "result", "count" : 29347 }](https://image.slidesharecdn.com/mongodbworldinventorypublic-140805153530-phpapp01/85/Retail-Reference-Architecture-Part-2-Real-Time-Geo-Distributed-Inventory-21-320.jpg)
![Inventory – Sample Aggregations
• Aggregate total quantity for a store
db.inventory.aggregate( [
{ $match : { storeId : "store100" } },
{ $unwind : "$vars" },
{ $group : {
_id : "result",
count : { $sum : "$vars.q" } } } ] )
{ "_id" : "result", "count" : 29347 }](https://image.slidesharecdn.com/mongodbworldinventorypublic-140805153530-phpapp01/85/Retail-Reference-Architecture-Part-2-Real-Time-Geo-Distributed-Inventory-22-320.jpg)
![Inventory – Sample Geo-Query
• Get inventory for an item near a point
db.runCommand( {
geoNear : "inventory",
near : {
type : "Point",
coordinates : [-82.8006, 40.0908] },
maxDistance : 10000.0,
spherical : true,
limit : 10,
query : { "productId" : "p200",
"vars.sku" : "sku11736" } } )](https://image.slidesharecdn.com/mongodbworldinventorypublic-140805153530-phpapp01/85/Retail-Reference-Architecture-Part-2-Real-Time-Geo-Distributed-Inventory-24-320.jpg)
![Inventory – Sample Geo-Query
• Get closest store with available sku
db.runCommand( {
geoNear : "inventory",
near : {
type : "Point",
coordinates : [-82.800672, 40.090844] },
maxDistance : 10000.0,
spherical : true,
limit : 1,
query : {
productId : "p200",
vars : {
$elemMatch : { sku : "sku11736",
q : { $gt : 0 } } } } } )](https://image.slidesharecdn.com/mongodbworldinventorypublic-140805153530-phpapp01/85/Retail-Reference-Architecture-Part-2-Real-Time-Geo-Distributed-Inventory-25-320.jpg)
![Inventory – Sample Geo-
Aggregation
• Get count of inventory for an item near a point
db.inventory.aggregate([
{ $geoNear: {
near : { type : "Point",
coordinates : [-82.800672, 40.090844] },
distanceField: "distance",
maxDistance: 10000.0, spherical : true,
query: { productId: "p200",
vars : { $elemMatch : { sku : "sku11736",
q : {$gt : 0} } } },
includeLocs: "dist.location",
num: 5 } },
{ $unwind: "$vars" },
{ $match: { "vars.sku" : "sku11736" } },
{ $group: { _id: "result", count: {$sum: "$vars.q"} } }])](https://image.slidesharecdn.com/mongodbworldinventorypublic-140805153530-phpapp01/85/Retail-Reference-Architecture-Part-2-Real-Time-Geo-Distributed-Inventory-26-320.jpg)
Retail Reference Architecture Part 2: Real-Time, Geo Distributed Inventory
- 1. Senior Consulting Engineer, MongoDB Norman Graham #MongoDBWorld Retail Reference Architecture: Real-Time, Geo-Distributed Inventory
- 2. Inventory Inventory MongoDB External Inventory Internal Inventory Regional Inventory Purchase Orders Fulfillment Promotions
- 3. Inventory – Traditional Architecture Relational DB System of Records Nightly Batches Analytics, Aggregations, Reports Caching Layer Field Inventory Internal & External Apps Point-in-time Loads
- 4. Inventory – Opportunities Missed • Can’t reliably detect availability • Can't redirect purchasers to in-store pickup • Can’t do intra-day replenishment • Degraded customer experience • Higher internal expense
- 5. Inventory – Principles • Single view of the inventory • Used by most services and channels • Read-dominated workload • Local, real-time writes • Bulk writes for refresh • Geographically distributed • Horizontally scalable
- 6. Requirement Challenge MongoDB Single view of inventory Ensure availability of inventory information on all channels and services Developer-friendly, document-oriented storage High volume, low latency reads Anytime, anywhere access to inventory data without overloading the system of record Fast, indexed reads Local reads Horizontal scaling Bulk updates, intra-day deltas Provide window-in-time consistency for highly available services Bulk writes Fast, in-place updates Horizontal scaling Rapid application development cycles Deliver new services rapidly to capture new opportunities Flexible schema Rich query language Agile-friendly iterations Inventory – Requirements
- 7. Inventory – Target Architecture Relational DB System of Records Analytics, Aggregations, Reports Field Inventory Internal & External Apps Inventory Assortments Shipments Audits Products Stores Point-in-time Loads Nightly Refresh Real-time Updates
- 8. Horizontal Scaling Inventory – Technical Decisions Store Inventory Schema Indexing
- 9. Inventory – Collections Stores Inventory Products Audits AssortmentsShipments
- 10. > db.stores.findOne() { "_id" : ObjectId("53549fd3e4b0aaf5d6d07f35"), "className" : "catalog.Store", "storeId" : "store0", "name" : "Bessemer store", "address" : { "addr1" : "1st Main St", "city" : "Bessemer", "state" : "AL", "zip" : "12345", "country" : "US" }, "location" : [ -86.95444, 33.40178 ], ... } Stores – Sample Document
- 11. Stores – Sample Queries • Get a store by storeId db.stores.find({ "storeId" : "store0" }) • Get a store by zip code db.stores.find({ "address.zip" : "12345" })
- 12. What’s near me?
- 13. Stores – Sample Geo Queries • Get nearby stores sorted by distance db.runCommand({ geoNear : "stores", near : { type : "Point", coordinates : [-82.8006, 40.0908] }, maxDistance : 10000.0, spherical : true })
- 14. Stores – Sample Geo Queries • Get the five nearest stores within 10 km db.stores.find({ location : { $near : { $geometry : { type : "Point", coordinates : [-82.80, 40.09] }, $maxDistance : 10000.0 } } }).limit(5)
- 15. Stores – Indices • { "storeId" : 1 }, { "unique" : true } • { "name" : 1 } • { "address.zip" : 1 } • { "location" : "2dsphere" }
- 16. > db.inventory.findOne() { "_id": "5354869f300487d20b2b011d", "storeId": "store0", "location": [-86.95444, 33.40178], "productId": "p0", "vars": [ { "sku": "sku1", "q": 14 }, { "sku": "sku3", "q": 7 }, { "sku": "sku7", "q": 32 }, { "sku": "sku14", "q": 65 }, ... ] } Inventory – Sample Document
- 17. Inventory – Sample Queries • Get all items in a store db.inventory.find({ storeId : "store100" }) • Get quantity for an item at a store db.inventory.find({ "storeId" : "store100", "productId" : "p200" })
- 18. Inventory – Sample Queries • Get quantity for a sku at a store db.inventory.find( { "storeId" : "store100", "productId" : "p200", "vars.sku" : "sku11736" }, { "vars.$" : 1 } )
- 19. Inventory – Sample Update • Increment / decrement inventory for an item at a store db.inventory.update( { "storeId" : "store100", "productId" : "p200", "vars.sku" : "sku11736" }, { "$inc" : { "vars.$.q" : 20 } } )
- 20. Inventory – Sample Aggregations • Aggregate total quantity for a product db.inventory.aggregate( [ { $match : { productId : "p200" } }, { $unwind : "$vars" }, { $group : { _id : "result", count : { $sum : "$vars.q" } } } ] ) { "_id" : "result", "count" : 101752 }
- 21. Inventory – Sample Aggregations • Aggregate total quantity for a store db.inventory.aggregate( [ { $match : { storeId : "store100" } }, { $unwind : "$vars" }, { $match : { "vars.q" : { $gt : 0 } } }, { $group : { _id : "result", count : { $sum : 1 } } } ] ) { "_id" : "result", "count" : 29347 }
- 22. Inventory – Sample Aggregations • Aggregate total quantity for a store db.inventory.aggregate( [ { $match : { storeId : "store100" } }, { $unwind : "$vars" }, { $group : { _id : "result", count : { $sum : "$vars.q" } } } ] ) { "_id" : "result", "count" : 29347 }
- 24. Inventory – Sample Geo-Query • Get inventory for an item near a point db.runCommand( { geoNear : "inventory", near : { type : "Point", coordinates : [-82.8006, 40.0908] }, maxDistance : 10000.0, spherical : true, limit : 10, query : { "productId" : "p200", "vars.sku" : "sku11736" } } )
- 25. Inventory – Sample Geo-Query • Get closest store with available sku db.runCommand( { geoNear : "inventory", near : { type : "Point", coordinates : [-82.800672, 40.090844] }, maxDistance : 10000.0, spherical : true, limit : 1, query : { productId : "p200", vars : { $elemMatch : { sku : "sku11736", q : { $gt : 0 } } } } } )
- 26. Inventory – Sample Geo- Aggregation • Get count of inventory for an item near a point db.inventory.aggregate([ { $geoNear: { near : { type : "Point", coordinates : [-82.800672, 40.090844] }, distanceField: "distance", maxDistance: 10000.0, spherical : true, query: { productId: "p200", vars : { $elemMatch : { sku : "sku11736", q : {$gt : 0} } } }, includeLocs: "dist.location", num: 5 } }, { $unwind: "$vars" }, { $match: { "vars.sku" : "sku11736" } }, { $group: { _id: "result", count: {$sum: "$vars.q"} } }])
- 27. Inventory – Sample Indices • { storeId : 1 } • { productId : 1, storeId : 1 } • { productId : 1, location : "2dsphere" } • Why not "vars.sku"? – { productId : 1, storeId : 1, "vars.sku" : 1 }
- 28. Horizontal Scaling Inventory – Technical Decisions Store Inventory Schema Indexing
- 29. Shard East Shard Central Shard West East DC Inventory – Sharding Topology West DC Central DC Legacy Inventory Primary Primary Primary
- 30. Inventory – Shard Key • Choose shard key – { storeId : 1 } ? – { productId : 1, storeId : 1 }? – { storeId : 1, productId : 1 } ? • Set up sharding – sh.enableSharding("inventoryDB") – sh.shardCollection( "inventoryDB.inventory", { storeId : 1, productId : 1 })
- 31. Inventory – Shard Tags • Set up shard tags – sh.addShardTag("shard0000", "west") – sh.addShardTag("shard0001", "central") – sh.addShardTag("shard0002", "east") • Set up tag ranges – sh.addTagRange("inventoryDB.inventory", { storeId : 0 }, { storeId : 100}, "west" ) – sh.addTagRange("inventoryDB.inventory", { storeId : 100 }, { storeId : 200 }, "central" ) – sh.addTagRange("inventoryDB.inventory", { storeId : 200 }, { storeId : 300 }, "east" )
- 32. Senior Consulting Engineer, MongoDB Norman Graham #MongoDBWorld Thank You