Deployment Preparedness
Download as PPTX, PDF2 likes558 views
This document provides guidance on monitoring and optimizing MongoDB deployments. It discusses collecting metrics on operations, memory usage, page faults, locks and B-tree accesses. Key considerations for deployment include profiling usage, monitoring, capacity planning, high availability and disaster recovery. The document also offers tips on warming the cache, using separate disks for journaling and data files, and dynamically changing the log level.
![Log file
Mon Dec 3 15:05:37 [conn81]
getmore scaleout.nodes query: { ts: { $lte: new Date(1354547123142) } }
cursorid:8607875337747748011
ntoreturn:0
keyUpdates:0
numYields: 216
locks(micros) r:615830
nreturned:27055
reslen:4194349
551ms](https://image.slidesharecdn.com/deployment-121212100612-phpapp02/85/Deployment-Preparedness-21-320.jpg)
Deployment Preparedness
- 2. Plan A because there is no Plan B http://bit.ly/QlJULZ
- 3. Part One Before you deploy…
- 4. Prototype Ops Playbook Test Capacity Planning Monitor Reinventing the wheel
- 5. Essentials • Disable NUMA • Pick appropriate file-system (xfs, ext4) • Pick 64-bit O/S – Recent Linux kernel, Win2k8R2 • More RAM – Spend on RAM not Cores • Faster Disks – SSDs vs. SAN – Separate Journal and Data Files
- 6. Key things to consider • Profiling – Baseline/Blue print: Understand what should happen – Ensure good Index usage • Monitoring – SNMP, munin, zabix, cacti, nagios – MongoDB Monitoring Service (MMS) • Sizing – Understand Capability (RAM, IOPs) – Understand Use Cases + Schema
- 7. What is your SLA? • High Availability? – 24x7x365 operation? – Limited maintenance window? • Data Protection? – Failure of a Single Node? – Failure of a Data Center? • Disaster Recovery? – Manual or automatic failover? – Data Center, Region, Continent?
- 8. Build & Test your Playbook • Backups • Restores (backups are not enough) • Upgrades • Replica Set Operations • Sharding Operations
- 9. Part Two Under the cover…
- 10. How to see metrics • mongostat • MongoDB plug ins for – munin, zabix, cacti, ganglia •Hosted Services – MMS - 10gen – Server Density, Cloudkick • Profiling
- 12. Metrics in detail: opcounters • Counts: Insert, Update, Delete, Query, Commands • Operation counters are mostly straightforward: more is better • Some operations in a replica set primary are accounted differently in a secondary • getlastError(), system.status etc are also counted
- 14. Metrics in detail: resident memory • Key metric: to a very high degree, the performance of a mongod is a measure of how much data fits in RAM. • If this quantity is stably lower than available physical memory, the mongod is likely performing well. • Correlated metrics: page faults, B-Tree misses
- 16. Collection 1 Virtual Disk Address Space 1 Physical RAM Index 1 100 ns = 10,000,000 ns =
- 17. Metrics in detail: page faults • This measures reads or writes to pages of data file that aren't resident in memory • If this is persistently non-zero, your data doesn't fit in memory. • Correlated metrics: resident memory, B-Tree misses, iostats
- 18. Working Set > db.blogs.stats() { Size of data "ns" : "test.blogs", "count" : 1338330, "size" : 46915928, Average "avgObjSize" : 35.05557523181876, document size "storageSize" : 86092032, "numExtents" : 12, Size on disk (and "nindexes" : 2, in memory!) "lastExtentSize" : 20872960, "paddingFactor" : 1, "flags" : 0, "totalIndexSize" : 99860480, Size of all "indexSizes" : { indexes "_id_" : 55877632, "name_1" : 43982848 Size of each }, index "ok" : 1 }
- 19. Lock % counter
- 20. Metrics in detail: lock percentage and queues • By itself, lock % can be misleading: a high lock percentage just means that writing is happening. • But when lock % is high and queued readers or writers is non-zero, then the mongod probably at its write capacity. • Correlated metrics: iostats
- 21. Log file Mon Dec 3 15:05:37 [conn81] getmore scaleout.nodes query: { ts: { $lte: new Date(1354547123142) } } cursorid:8607875337747748011 ntoreturn:0 keyUpdates:0 numYields: 216 locks(micros) r:615830 nreturned:27055 reslen:4194349 551ms
- 22. explain, hint // explain() shows the plan used by the operation > db.c.find(<query>).explain() // hint() forces a query to use a specific index // x_1 is the name of the index from db.c.getIndexes() > db.c.find( {x:1} ).hint("x_1")
- 23. B-Tree Counter
- 24. Metrics in detail: B-Tree • Indicates b-tree accesses including page fault service during an index lookup • If misses are persistently non-zero, your indexes don't fit in RAM. (You might need to change or drop indexes, or shard your data.) • Correlated metrics: resident memory, page faults, iostats
- 25. B-Trees' strengths • B-Tree indexes are designed for range queries over a single dimension • Think of a compound index on { A, B } as being an index on the concatenation of the A and B values in documents • MongoDB can use its indexes for sorting as well
- 26. B-Trees' weaknesses • Ranges queries on the first field of a compound index are suboptimal • Range queries over multiple dimensions are suboptimal • In both these cases, a suboptimal index might be better than nothing, but best is to try to see if you can't change the problem
- 27. Indexing dark corners • Some functionality can't currently always use indexes: – $where JavaScript clauses – $mod, $not, $ne – regex • Negation maybe transformed into a range query – Index can be used • Complicated regular expressions scan a whole index
- 28. Other tricks
- 29. Warming the Cache > db.c.find( {unused_key: 1} ).explain() > db.c.find( {unused_key: 1} ) .hint( {random_index:1} ) .explain() # cat /data/db/* > /dev/null // New in 2.2 > db.runCommand( { touch: "blogs", data: true, index: true } )
- 30. Journal on another disk •The journal's write load is very different than the data files – journal = append-only – data files = randomly accessed •Putting the journal on a separate disk or RAID (e.g., with a symlink) will minimize any seek-time related journaling overhead
- 31. --directoryperdb • Allows storage tiering – Different access patterns – Different Disk Types / Speeds • use --directoryperdb • add symlink into database directory
- 32. Dynamically change log level // Change logging level to get more info > db.adminCommand({ setParameter: 1, logLevel: 1 }) > db.adminCommand({ setParameter: 1, logLevel: 0 })
- 33. Because you now have a Plan B http://bit.ly/QlJULZ