MongoDB and the MEAN Stack
18 likes11,087 views
This document summarizes a presentation about building an IoT application using the MEAN stack. It discusses five key things they learned: performance depends on test data; MEAN is fast to develop with but frameworks can obscure what's happening so profiling is important; incremental aggregation works well for IoT; Node.js bottlenecks before MongoDB; and performance tuning patterns like identifying bottlenecks and slam-dunk optimizations. It also describes building user stories for an advertising application, modeling the data, initial measurements that guided prototyping, challenges of scaling, and using "boxes" to identify hot sales areas.
![Document / Model / Controller
Model (advertiser.js) Document
Haystack examples sent us in
wrong direction initially
exports.all = function(req, res) {!
! findQuery = { near: [ Number(req.query.lng), Number(req.query.lat) ],!
! ! maxDistance: Number(req.query.dist) };!
! Advertiser.geoSearch({kind:"pub"}, findQuery, !
! ! function (err, advertisers) {!
// error handling!
! !! res.jsonp(advertisers);!
! ! });!
}
13
{
name: ‘Long Hall’,
pos: [-6.265535, 53.3418364],
kind: “pub”
}
AdvertiserSchema = new Schema({!
name: { type: String,!
default: ‘’},!
pos: [Number],!
kind: { type: String,!
default: ‘place’},!
}); Controller (advertisers.js)](https://image.slidesharecdn.com/mongodbiotrefmeanlondon2014v3-141106090249-conversion-gate02/85/MongoDB-and-the-MEAN-Stack-13-320.jpg)
![To find out what’s happening - debug
We used Express passport-http to add Basic-
Digest auth (client id lookup)
It can be hard to figure out what a framework like
express/mongoose really does
Tip: mongoose.set('debug', true) - detailed logging
Console
Mongoose: clients.findOne({ _id: ObjectId(“…”) })!
Mongoose: advertisers.geoHaystack({…[-6.267765, 53.34087]})!
19](https://image.slidesharecdn.com/mongodbiotrefmeanlondon2014v3-141106090249-conversion-gate02/85/MongoDB-and-the-MEAN-Stack-19-320.jpg)
MongoDB and the MEAN Stack
- 1. Ger Hartnett & Alan Spencer MongoDB Dublin
- 2. Overview • Fictional story of a startup using MongoDB & MEAN stack to build IoT application • We’ll take a devops perspective - show you what to watch out for a framework like MEAN • Tips you can use to help development team focus on the right things when close to production • Questions • How many from operations? • How many from development? 2
- 3. 5 Things we Learned Capacity planning/prototyping is a good idea but performance is sensitive to sample test data The MEAN stack rocks - fast to get started - profiler can help you understand what’s under the hood Realtime/incremental aggregation works well with IoT workloads - the “MMS approach” With NodeJS/Express number of app servers becomes bottleneck before MongoDB Performance tuning patterns apply - "bottleneck whack-a-mole" & “slam-dunk-optimization” 3
- 4. Context: IoT & MEAN
- 5. Internet of Things Big Data => Humongous Data “The rise of device oriented development … new architectural and workflow challenges … distinctly different from … web and mobile development so far.” - Morten Bagai
- 6. Internet of Things • Bosch: “IoT brings root and branch changes to the world of business” • Richard Kreuter's Webinar May 2013 • Earlier bootcamp looked at sharding IoT 6 Photo by jurvetson - Creative Commons Attribution License - http://www.flickr.com/photos/jurvetson/916142
- 7. MEAN stack 7 MongoDB - the database Express - web app framework/router Angular - browser HTML/JS MVC Node - javascript application server Photo by benmizen - Creative Commons ShareAlike License - http://www.flickr.com/photos/benmizen/9456440635
- 8. Learn more about MEAN Valeri Karpov - MongoDB Kernel Tools Team http://thecodebarbarian.wordpress.com/2013/07/22/ introduction-to-the-mean-stack-part-one-setting-up-your-tools/ MEAN.io http://mean.io 8
- 9. About MongoDB Bootcamp We invest in technical new hires Everyone does “bootcamp” NYC for 2 weeks - product internals Then work on a longer project 3-4 weeks In our case: wanted to do a bit of everything, capacity planning, iterate user-stories, MongoDB a component 9
- 10. The Application
- 11. Location based advertising - IoMT 11 ! ! Advertiser ! Advertiser Advertiser ! ! ! ! Customer ! • IoT example 3 from Richard’s Webinar
- 12. User Stories - for the application US1 - customer looks for advertisers near US2 - advertiser wants to see how many customers saw offer US3 - find hot spots where many customers but few advertisers 12 Photo by consumerist - Creative Commons Attribution License - http://www.flickr.com/photos/consumerist/2158190589
- 13. Document / Model / Controller Model (advertiser.js) Document Haystack examples sent us in wrong direction initially exports.all = function(req, res) {! ! findQuery = { near: [ Number(req.query.lng), Number(req.query.lat) ],! ! ! maxDistance: Number(req.query.dist) };! ! Advertiser.geoSearch({kind:"pub"}, findQuery, ! ! ! function (err, advertisers) {! // error handling! ! !! res.jsonp(advertisers);! ! ! });! } 13 { name: ‘Long Hall’, pos: [-6.265535, 53.3418364], kind: “pub” } AdvertiserSchema = new Schema({! name: { type: String,! default: ‘’},! pos: [Number],! kind: { type: String,! default: ‘place’},! }); Controller (advertisers.js)
- 14. CRUD interface & Mongoose CRUD interface ! Raised & fixed bug in Mongoose, pull request merged 14
- 15. 5 Things we Learned Capacity planning/prototyping is a good idea but performance is sensitive to sample test data The MEAN stack rocks - fast to get started - profiler can help you understand what’s under the hood Realtime/incremental aggregation works well with IoT workloads - the “MMS approach” With NodeJS/Express number of app servers becomes bottleneck before MongoDB Performance tuning patterns apply - "bottleneck whack-a-mole" & “slam-dunk-optimization” 15
- 16. US1 Initial Measurements MongoDB shell scripts 9 advertisers, small area, distance 10km MongoDB has 5 kinds of geo query 3 kinds of geo index geoSearch (haystack) looked much better than others (our 1st mistake) TIP: performance is sensitive to test data & query 16
- 17. 5 Things we Learned Capacity planning/prototyping is a good idea but performance is sensitive to sample test data The MEAN stack rocks - fast to get started - profiler can help you understand what’s under the hood Realtime/incremental aggregation works well with IoT workloads - the “MMS approach” With NodeJS/Express number of app servers becomes bottleneck before MongoDB Performance tuning patterns apply - "bottleneck whack-a-mole" & “slam-dunk-optimization” 17
- 18. The good thing about frameworks is… ! they do lot’s of things for developers ! ! ! …and the bad thing about frameworks? ! they do lot’s of things for developers
- 19. To find out what’s happening - debug We used Express passport-http to add Basic- Digest auth (client id lookup) It can be hard to figure out what a framework like express/mongoose really does Tip: mongoose.set('debug', true) - detailed logging Console Mongoose: clients.findOne({ _id: ObjectId(“…”) })! Mongoose: advertisers.geoHaystack({…[-6.267765, 53.34087]})! 19
- 20. Find out what’s happening - profiler Tip: The MongoDB profiler shows operations really happening on DB, check with dev 20 db.system.profile.find {"op":"query", "ns":"tings.clients",...! {“op":"command", "command":{"geoSearch"...! {"op" :"update","ns":"tings.sessions"...! exports.all = function(req, res) {! . . .! ! ! ! req.session = null;! ! !! res.jsonp(advertisers);! } 10% performance improvement Where did that come from? Fixing it is not obvious
- 21. Back to the application
- 22. US2 means we built on US1 US1 - customer looks for advertisers near • Need to store customer location US2 - advertiser wants to see how many customers near 22 Being a startup we decided to take a naive pragmatic approach: • Store all samples • US2 aggregates on-demand Photo by consumerist - Creative Commons Attribution License - http://www.flickr.com/photos/consumerist/2158190589
- 23. 5 Things we Learned Capacity planning/prototyping is a good idea but performance is sensitive to sample test data The MEAN stack rocks - fast to get started - profiler can help you understand what’s under the hood Realtime/incremental aggregation works well with IoT workloads - the “MMS approach” With NodeJS/Express number of app servers becomes bottleneck before MongoDB Performance tuning patterns apply - "bottleneck whack-a-mole" & “slam-dunk-optimization” 23
- 24. US2 - Aggregation of Raw Samples 1 hour of raw samples @ 2k RPS = 7.2M documents ! Aggregation on 7.2M raw samples took 1 second on our instances Significant impact • Run every 2 seconds RPS dropped by factor of 4! (single instance) Samples Query Aggregate 24 Raw Insert Aggregate
- 25. US2 - Pre aggregation Samples Query Aggregate 25 Raw Insert Samples Pre Aggregate ! Update Query Aggregate Aggregate Aggregate An MMS type approach Document for advertiser-customer-month ! Using update multi-true (more on this later) ! Query now only needs to aggregate unique customers
- 26. US1 measurements revisited MongoDB shell scripts More realistic data - old measurements repeated locations 110k advertisers with clusters in DUB and NYC Performance best for near and nearSphere (2x better than Haystack) 26
- 27. Where does the time go? 27 • Express/Mongoose/Node • Customer Lookup • Find ($near) • Save Sample DB • Save Sample File • Preagg=multiple docs (6) • Preagg=multi-update 1 doc
- 28. 5 Things we Learned Capacity planning/prototyping is a good idea but performance is sensitive to sample test data The MEAN stack rocks - fast to get started - profiler can help you understand what’s under the hood Realtime/incremental aggregation works well with IoT workloads - the “MMS approach” With NodeJS/Express number of app servers becomes bottleneck before MongoDB Performance tuning patterns apply - "bottleneck whack-a-mole" & “slam-dunk-optimization” 28
- 29. NodeJS MongoD MongoD Deployment Chrome:Postman 29 NodeJS HAproxy NodeLoad NodeJS NodeJS
- 30. Scaling 30
- 31. 5 Things we Learned Capacity planning/prototyping is a good idea but performance is sensitive to sample test data The MEAN stack rocks - fast to get started - profiler can help you understand what’s under the hood Realtime/incremental aggregation works well with IoT workloads - the “MMS approach” With NodeJS/Express number of app servers becomes bottleneck before MongoDB Performance tuning patterns apply - "bottleneck whack-a-mole" & “slam-dunk-optimization” 31
- 32. 1 - number of Node.JS 2 - HAproxy 3 - load gen threads/BW
- 33. Pattern: “slam dunk optimization" 2 * NodeJS MongoD MongoD Chrome:Postman 33 NodeJS HAproxy NodeLoad NodeJS NodeJS 3 1
- 34. Performance tips 1. Increase number of Node.JS 2. Increase perf of proxy/balancer instance 34 HAproxy more balanced than Amazon ELB 3. Tweak Nodeload (generates/measures REST) Nodeload concurrency 3x Node servers Run Nodeload on same machine as HAproxy Development recommendation: Postman chrome ext - generates REST / Basic Auth
- 35. Back to the application
- 36. US3 Overview What are the top 10 hot sales areas? • What is an “area”…? Requirements • Little impact, easy to calculate • Approx. Regular size • Optimal approx. distance - “bounding areas” • Plays nice with sharding Internals of haystack, 2dsphere? Polygon? MGRS? 36
- 37. US3 - Hot box - Sales, go sell! 37
- 38. MGRS - Military Grid Reference System 38 • 4QFJ123678 precision level 100m Image by Mikael Rittri - Creative Commons ShareAlike License http://en.wikipedia.org/wiki/File:MGRSgridHawaiiSchemeAARealigned.png
- 39. MGRS - But at the poles… 39 Image by Mikael Rittri - Creative Commons ShareAlike License http://en.wikipedia.org/wiki/File:MGRSgridNorthPole.png
- 41. The “box” - the poor-man’s MGRS x • Reinvented the sphere • Long/lat -> box number • Tailored to specific distance • Boxes are at least 1km • Search in current and 8 neighbouring boxes ! • Filter outside circle in JS • Performed relatively well • Can be used to shard 41
- 42. Replication 42
- 43. Impact of Replication 43 Secondary reads ! Worked for this app ! Beware - don’t try this at home!
- 44. Apply the production notes Change from default readahead Disable NUMA & THP ext4 or XFS noatime Load test workload on different configurations Instance Store / EBS (PIOPs) SSDs / spinning rust AWS instance types 44
- 45. Recap
- 46. 5 Things we Learned Capacity planning/prototyping is a good idea but performance is sensitive to sample test data The MEAN stack rocks - fast to get started but profiler can help you understand what’s under the hood Realtime/incremental aggregation works well with IoT workloads - the “MMS approach” Performance tuning patterns apply - "bottleneck whack-a-mole" & “slam-dunk-optimization” With NodeJS/Express number of app servers becomes bottleneck before MongoDB 46
- 47. Next Steps
- 48. Next Steps Plan to publish as blog post series and github project ! Check blog.mongodb.org ! Continue to explore… 48
- 49. Next Steps - continuation Hadoop/YARN for aggregations Use “box” to geo-shard Try 2.6 bulk updates Dynamic angular-google-maps with socket-io Implement in another framework (Go/Clojure) to load MongoDB with less hardware Find balance between batch and pre-aggregation 49 (see next slide)
- 50. Learn More & Thank You Introduction to MEAN - Valeri Karpov http://thecodebarbarian.wordpress.com/2013/07/22/introduction-to-the-mean-stack-part-one-setting-up-your- tools/ MEAN.io http://mean.io Richard Kreuter's webinar - M2M http://www.mongodb.com/presentations/webinar-realizing-promise-machine-machine-m2m-mongodb Building MongoDB Into Your Internet of Things http://blog.mongohq.com/building-mongodb-into-your-internet-of-things-a-tutorial/ Schema design for time series data (MMS) http://blog.mongodb.org/post/65517193370/schema-design-for-time-series-data-in-mongodb 50