MongoDB and Content Delivery at Aviary by Nir Zicherman and Jack Sisson
1 like1,459 views
The document outlines the presentation from the Aviary MongoDB User Group discussing their content delivery system for photo-editing tools, highlighting the transition from version 1 to version 2 and the implementation of targeted content delivery. Key elements include the use of MongoDB for data management and the ability to dynamically deliver content based on user specifics such as device and location. The presentation emphasizes Aviary's rapid growth and future plans for user targeting and enhancements in their content management system.
![Content Deployment
1. Insert/Update CMS Entry
"identifier": "com.aviary.stickers.234fe"
2. Find Response Formats
"type":"object",
"id": "com.aviary.stickers.234fe",
"properties":{
"metadata": {
"displayName": "Hats"
"name": "Hats",
"id": {
"type": "string",
},
"dataKey": "identifier"
"icon": {
"path": "cds/hats/icon.png"
"iconImagePath": "cds/hats/icon100.png"
"stickers": [
{
},
"identifier": "1"
"name": {
"path-100": "cds/hats/icon100.png"
"type": "string",
},
"imageUrl": "cds/hats/1.png"
"dataKey": "metadata.displayName"
"items": [
},
{
"type": "string",
"imageUrl": "cds/hats/1.png"
}
}
],
"iconImagePath": {
"identifier": "1"
]
3. Generate+Insert Responses
"versionKey": "e4532fd342"
"dataKey": "icon.path-100"
},
"stickers": {
"type": "array",
"dataKey": "items"
J](https://image.slidesharecdn.com/mongopresentation1-131122094404-phpapp01/85/MongoDB-and-Content-Delivery-at-Aviary-by-Nir-Zicherman-and-Jack-Sisson-18-320.jpg)
![Manifests
"stickers": [
{
"id": "com.aviary.stickers.234fe",
"versionKey": "e4532fd342"
},
{
"id": "com.aviary.stickers.fed34",
"versionKey": "c54532343d"
}
],
"frames": [
{
"id": "com.aviary.frames.25435",
"versionKey": "fd4324323"
}
]
J](https://image.slidesharecdn.com/mongopresentation1-131122094404-phpapp01/85/MongoDB-and-Content-Delivery-at-Aviary-by-Nir-Zicherman-and-Jack-Sisson-19-320.jpg)
![Manifest Deployment
Using Aggregate to Find the Newest Versions
Manifest with Correct Version Keys
{
$match:{
"stickers": [
{
formatId:{$in:formatIds},
"id": "com.aviary.stickers.234fe",
identifier:{$in:identifiers}
"versionKey": "e4532fd342"
}
}
},
],
{
$sort: { _id: -1 }
"effects": [
{
},
"id": "com.aviary.effects.25435",
{
"versionKey": "fd4324323"
$group: {
}
_id: "$identifier",
versionKey:{$first:"$versionKey"}
]
}
}
J](https://image.slidesharecdn.com/mongopresentation1-131122094404-phpapp01/85/MongoDB-and-Content-Delivery-at-Aviary-by-Nir-Zicherman-and-Jack-Sisson-20-320.jpg)
![Scopes and Targeting
Deployed Manifests Have Scopes
End Users Have Scope Parameters
Manifest 1
"targetingScope": {
"apiKey": "abc",
"country": ["JP"]
},
/manifest?
"formattingScope": {
"platform": "ios",
"minOsVersion": "7.0.0"
}
apiKey=abc&
country=JP&
language=ja&
platform=ios&
Manifest 2
osVersion=7.2.0
"targetingScope": {
"apiKey": "def",
},
"formattingScope": {
"platform": "android",
"minOsVersion": "6.0.0"
}
N](https://image.slidesharecdn.com/mongopresentation1-131122094404-phpapp01/85/MongoDB-and-Content-Delivery-at-Aviary-by-Nir-Zicherman-and-Jack-Sisson-21-320.jpg)
![API Servers
Scope parameters are converted into queries
Manifest 1
"targetingScope": {
db.manifest.find({
"apiKey": "abc",
"apiKey": {$in: ["abc", null]},
/manifest?
country=JP&
"formattingScope": {
"platform": {$in: ["ios", null]},
7002000
"platform": "ios",
"minOsVersion": {$lte: 7002000}
language=ja&
osVersion=7.2.0
},
"language": {$in: ["ja", null]},
apiKey=abc&
platform=ios&
"country": ["JP"]
"country": {$in: ["JP", null]},
"minOsVersion": "7.0.0"
}).sort({
}
"apiKey": -1,
Manifest 2
"language": -1,
"targetingScope": {
"country": -1,
"apiKey": "abc",
"minOsVersion": -1,
},
"platform": -1,
"formattingScope": {
"_id": -1
"platform": "ios",
}).limit(1)
"minOsVersion": "6.0.0"
}
N](https://image.slidesharecdn.com/mongopresentation1-131122094404-phpapp01/85/MongoDB-and-Content-Delivery-at-Aviary-by-Nir-Zicherman-and-Jack-Sisson-22-320.jpg)
![Versioned Content
Received Manifests Contain VersionKeys
"stickers": [
{
"id": "com.aviary.stickers.234fe",
"versionKey": "e4532fd342"
},
{
"id": "com.aviary.stickers.fed34",
"versionKey": "c54532343d"
db.content.findOne({
/content?
versionKey=e4532fd342
}
"versionKey": “e4532fd342”
});
],
"frames": [
{
"id": "com.aviary.frames.25435",
"versionKey": "fd4324323"
}
]
N](https://image.slidesharecdn.com/mongopresentation1-131122094404-phpapp01/85/MongoDB-and-Content-Delivery-at-Aviary-by-Nir-Zicherman-and-Jack-Sisson-23-320.jpg)
![Response Caching
db.manifests.find({
/manifest?
"apiKey": {$in: ["abc", null]},
apiKey=abc&
"country": {$in: ["JP", null]},
country=JP&
"language": {$in: ["ja", null]},
language=ja&
"platform": {$in: ["ios", null]},
platform=ios&
"minOsVersion": {$gte: 7002000}
osVersion=7.2.0
}).sort({
…,
"_id": -1
}).limit(1)
db.cachedManifests.findOne({
"url": "/manifest?apiKey=abc&country=JP&language=ja&osVersion=7.2.0&platform=ios"
})
N](https://image.slidesharecdn.com/mongopresentation1-131122094404-phpapp01/85/MongoDB-and-Content-Delivery-at-Aviary-by-Nir-Zicherman-and-Jack-Sisson-24-320.jpg)
![Other Mongo Usage
● PAULA permissions in user objects
users collection
{
"name": "nir",
"email": "nir@aviary.com",
"permissions": [
"content",
"dev",
"admin",
"partying"
]
}
● Integration tests interact with schemaless db willy nilly
N](https://image.slidesharecdn.com/mongopresentation1-131122094404-phpapp01/85/MongoDB-and-Content-Delivery-at-Aviary-by-Nir-Zicherman-and-Jack-Sisson-27-320.jpg)
MongoDB and Content Delivery at Aviary by Nir Zicherman and Jack Sisson
- 1. Content Delivery At Aviary MongoDb User Group 11/19/13
- 2. Aviary Photo-Editing SDK & Apps Fully-Baked UI Configurable, High-Quality Tools Over 5,000 Partners Over 50 Million Monthly Users Over 4 Billion Photos Edited iOS, Android, Web, Windows, Server J
- 3. Who Are We? Nir Jack Lead Serverside Director of Engineer Engineering Likes: ● ● ● Automated deployment Big-O notation Brainteasers Hates: ● Cilantro Likes: ● ● ● Parallelizing processes DRY code Seltzer Hates: ● Food after the sell-by date
- 5. The “Fat Tiny” Situation We want to dynamically deliver the Fat Tiny stickers to our users How can we do that? J
- 6. The CDS Aviary’s Content Delivery System
- 7. Version 1 ● Static files on S3 ● Excess data sent ● Data stored in MySQL ● New features meant new code N
- 8. The “Fat Tiny” Situation We want to display the Fat Tiny sticker pack only to Picstitch users in Japan who use iOS 7 How can we do that? N
- 9. CDS V2 Version 2 Overview
- 10. Stack Global Users Akamai AWS Cloud Formation AWS Elastic LoadBalancer (ELB) Node.js AWS Elastic Cloud Computing (EC2) Ubuntu Bash MongoDb JSON JSON Schema Content Delivery Network Load Balancer API Servers Database Cluster Management System N
- 11. Why MongoDB? ● Great with Node (JSON-based) ● Schema-less is easy to change/query ● Read-heavy ● Relatively small data set N
- 12. How It Works A Behind-the-Scenes Look
- 13. Delivered Types ● Manifest JSON ○ Content Set ○ Content Versions ● Content JSON ○ Content Metadata ○ All Assets N
- 14. The “Fat Tiny” Situation We want to manage the Fat Tiny sticker pack as a single entity, but we want to deliver it to each device in its own optimal format. How can we do that? N
- 15. Response Formatting Model Content Entry Response Formats Responses JSON document describing content item JSON documents defining mappings from entry to responses Actual JSON responses delivered to devices J
- 16. Content Formats ● JSON schema { "type":"object", "properties":{ "metadata": { ● Added properties "type": "object", "properties":{ "displayName": { "type": "string", ● Custom types } } }, ● Validates entries "icon": { "type": "object", "customType": "image" "properties":{ "path": { "type": "string", "required": true, J
- 17. Response Formats ● JSON schema { "type":"object", "properties":{ "identifier": { ● dataKey property "type": "string", "dataKey": "identifier" }, ● Defines response "name": { "type": "string", "dataKey": "metadata.displayName" }, structure "iconImagePath": { "type": "string", "dataKey": "icon.path-100" ● Maps content }, "items": { "type": "array", "dataKey": "items" J
- 18. Content Deployment 1. Insert/Update CMS Entry "identifier": "com.aviary.stickers.234fe" 2. Find Response Formats "type":"object", "id": "com.aviary.stickers.234fe", "properties":{ "metadata": { "displayName": "Hats" "name": "Hats", "id": { "type": "string", }, "dataKey": "identifier" "icon": { "path": "cds/hats/icon.png" "iconImagePath": "cds/hats/icon100.png" "stickers": [ { }, "identifier": "1" "name": { "path-100": "cds/hats/icon100.png" "type": "string", }, "imageUrl": "cds/hats/1.png" "dataKey": "metadata.displayName" "items": [ }, { "type": "string", "imageUrl": "cds/hats/1.png" } } ], "iconImagePath": { "identifier": "1" ] 3. Generate+Insert Responses "versionKey": "e4532fd342" "dataKey": "icon.path-100" }, "stickers": { "type": "array", "dataKey": "items" J
- 19. Manifests "stickers": [ { "id": "com.aviary.stickers.234fe", "versionKey": "e4532fd342" }, { "id": "com.aviary.stickers.fed34", "versionKey": "c54532343d" } ], "frames": [ { "id": "com.aviary.frames.25435", "versionKey": "fd4324323" } ] J
- 20. Manifest Deployment Using Aggregate to Find the Newest Versions Manifest with Correct Version Keys { $match:{ "stickers": [ { formatId:{$in:formatIds}, "id": "com.aviary.stickers.234fe", identifier:{$in:identifiers} "versionKey": "e4532fd342" } } }, ], { $sort: { _id: -1 } "effects": [ { }, "id": "com.aviary.effects.25435", { "versionKey": "fd4324323" $group: { } _id: "$identifier", versionKey:{$first:"$versionKey"} ] } } J
- 21. Scopes and Targeting Deployed Manifests Have Scopes End Users Have Scope Parameters Manifest 1 "targetingScope": { "apiKey": "abc", "country": ["JP"] }, /manifest? "formattingScope": { "platform": "ios", "minOsVersion": "7.0.0" } apiKey=abc& country=JP& language=ja& platform=ios& Manifest 2 osVersion=7.2.0 "targetingScope": { "apiKey": "def", }, "formattingScope": { "platform": "android", "minOsVersion": "6.0.0" } N
- 22. API Servers Scope parameters are converted into queries Manifest 1 "targetingScope": { db.manifest.find({ "apiKey": "abc", "apiKey": {$in: ["abc", null]}, /manifest? country=JP& "formattingScope": { "platform": {$in: ["ios", null]}, 7002000 "platform": "ios", "minOsVersion": {$lte: 7002000} language=ja& osVersion=7.2.0 }, "language": {$in: ["ja", null]}, apiKey=abc& platform=ios& "country": ["JP"] "country": {$in: ["JP", null]}, "minOsVersion": "7.0.0" }).sort({ } "apiKey": -1, Manifest 2 "language": -1, "targetingScope": { "country": -1, "apiKey": "abc", "minOsVersion": -1, }, "platform": -1, "formattingScope": { "_id": -1 "platform": "ios", }).limit(1) "minOsVersion": "6.0.0" } N
- 23. Versioned Content Received Manifests Contain VersionKeys "stickers": [ { "id": "com.aviary.stickers.234fe", "versionKey": "e4532fd342" }, { "id": "com.aviary.stickers.fed34", "versionKey": "c54532343d" db.content.findOne({ /content? versionKey=e4532fd342 } "versionKey": “e4532fd342” }); ], "frames": [ { "id": "com.aviary.frames.25435", "versionKey": "fd4324323" } ] N
- 24. Response Caching db.manifests.find({ /manifest? "apiKey": {$in: ["abc", null]}, apiKey=abc& "country": {$in: ["JP", null]}, country=JP& "language": {$in: ["ja", null]}, language=ja& "platform": {$in: ["ios", null]}, platform=ios& "minOsVersion": {$gte: 7002000} osVersion=7.2.0 }).sort({ …, "_id": -1 }).limit(1) db.cachedManifests.findOne({ "url": "/manifest?apiKey=abc&country=JP&language=ja&osVersion=7.2.0&platform=ios" }) N
- 25. PAULA The CDS Management Console
- 27. Other Mongo Usage ● PAULA permissions in user objects users collection { "name": "nir", "email": "nir@aviary.com", "permissions": [ "content", "dev", "admin", "partying" ] } ● Integration tests interact with schemaless db willy nilly N
- 29. The Facts ● Built and deployed in 3 months ● Very few struggles with MongoDB ● Seamless management ● Graceful scaling from 0 to over 20M MAUs ● Happy serverside engineers J
- 30. The Future ● Targeted Translations ● Granular User Targeting ● PAULA for the masses N
- 31. Questions? Comments also welcome nir@aviary.com jack@aviary.com …and by the way, WE’RE HIRING!