NoSQL Infrastructure
0 likes836 views
This document discusses MongoDB infrastructure at Server Density. It notes that Server Density uses 27 MongoDB nodes to store 20TB of data per month from their MySQL database. Some key reasons for choosing MongoDB include replication, official drivers, easy deployment, and fast performance out of the box. The document then discusses various MongoDB performance and infrastructure considerations like network throughput, replication lag, failover processes, disk types, backups, and monitoring.
![Tags
{
_id : "someSet",
members : [
{_id : 0, host : "A", tags : {"dc": "ny"}},
{_id : 1, host : "B", tags : {"dc": "ny"}},
{_id : 2, host : "C", tags : {"dc": "sf"}},
{_id : 3, host : "D", tags : {"dc": "sf"}},
{_id : 4, host : "E", tags : {"dc": "cloud"}}
]
settings : {
getLastErrorModes : {
veryImportant : {"dc" : 3},
sortOfImportant : {"dc" : 2}
}
}
}
> db.foo.insert({x:1})
> db.runCommand({getLastError : 1, w : "veryImportant"})](https://image.slidesharecdn.com/nosql-20infrastructure-202013-130520060326-phpapp02/85/NoSQL-Infrastructure-39-320.jpg)
![{
_id : "someSet",
members : [
{_id : 0, host : "A", tags : {"dc": "ny"}},
{_id : 1, host : "B", tags : {"dc": "ny"}},
{_id : 2, host : "C", tags : {"dc": "sf"}},
{_id : 3, host : "D", tags : {"dc": "sf"}},
{_id : 4, host : "E", tags : {"dc": "cloud"}}
]
settings : {
getLastErrorModes : {
veryImportant : {"dc" : 3},
sortOfImportant : {"dc" : 2}
}
}
}
> db.foo.insert({x:1})
> db.runCommand({getLastError : 1, w : "veryImportant"})
(A or B) + (C or D) + E
Tags](https://image.slidesharecdn.com/nosql-20infrastructure-202013-130520060326-phpapp02/85/NoSQL-Infrastructure-40-320.jpg)
![{
_id : "someSet",
members : [
{_id : 0, host : "A", tags : {"dc": "ny"}},
{_id : 1, host : "B", tags : {"dc": "ny"}},
{_id : 2, host : "C", tags : {"dc": "sf"}},
{_id : 3, host : "D", tags : {"dc": "sf"}},
{_id : 4, host : "E", tags : {"dc": "cloud"}}
]
settings : {
getLastErrorModes : {
veryImportant : {"dc" : 3},
sortOfImportant : {"dc" : 2}
}
}
}
> db.foo.insert({x:1})
> db.runCommand({getLastError : 1, w : "sortOfImportant"})
(A + C) or (D + E) ...
Tags](https://image.slidesharecdn.com/nosql-20infrastructure-202013-130520060326-phpapp02/85/NoSQL-Infrastructure-41-320.jpg)
NoSQL Infrastructure
- 2. ~3333 write ops/s 0.07 - 0.05 ms response
- 5. MongoDB at Server Density
- 6. •27 nodes MongoDB at Server Density
- 7. • June 2009 - 4yrs •27 nodes MongoDB at Server Density
- 8. •MySQL -> MongoDB •27 nodes MongoDB at Server Density • June 2009 - 4yrs
- 9. •MySQL -> MongoDB •20TB data per month •27 nodes MongoDB at Server Density • June 2009 - 4yrs
- 10. Why?
- 12. Why? • Replication • Official drivers
- 13. Why? • Replication • Official drivers • Easy deployment
- 14. Why? • Replication • Official drivers • Easy deployment • Fast out of the box
- 15. It’s a little different.
- 16. Picture is unrelated! Mmm, ice cream. • Fast network Performance
- 17. • Fast network Performance EC2 10 Gigabit Ethernet - Cluster Compute - High Memory Cluster - Cluster GPU - High I/O - High Storage
- 18. • Fast network Performance Workload: Read/Write? Result set size What is being stored?
- 19. • Fast network Performance Use Network Throughput Normal 0-100Mb/s Replication (Initial Sync) Burst +100Mb/s Replication (Oplog) 0-100Mb/s Backup Initial Sync + Oplog
- 21. • Fast network Performance Inter-DC LAN Cross USA Washington, DC - San Jose, CA
- 22. • Fast network Performance Location Ping RTT Latency Within USA 40-80ms Trans-Atlantic 100ms Trans-Pacific 150ms Europe - Japan 300ms
- 25. Failover •Replica sets •Min 3 nodes •Master/slave
- 26. Failover •Replica sets •Min 3 nodes •Master/slave •Automatic failover
- 27. • Replication lag Performance Location Ping RTT Latency Within USA 40-80ms Trans-Atlantic 100ms Trans-Pacific 150ms Europe - Japan 300ms
- 28. Replication Lag 1. Reads: eventual consistency
- 29. Replication Lag 1. Reads: eventual consistency 2. Failover: slave behind
- 32. Eventual Consistency Stale data Inconsistent data Changing data
- 33. Eventual Consistency Use Case Needs consistency? Graphs No User profile Yes Statistics Depends Alert config Yes
- 34. Replication Lag 1. Reads: eventual consistency 2. Failover: slave behind
- 35. Slave behind Failover: out of date master
- 36. MongoDB WriteConcern Changed Nov 27 2012
- 37. • Safe by default >>> from pymongo import MongoClient >>> connection = MongoClient(w=int/str) Value Meaning 0 Unsafe 1 Primary 2 Primary + x1 secondary 3 Primary + x2 secondaries MongoDB WriteConcern
- 39. Tags { _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "veryImportant"})
- 40. { _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "veryImportant"}) (A or B) + (C or D) + E Tags
- 41. { _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "sortOfImportant"}) (A + C) or (D + E) ... Tags
- 42. Picture is unrelated! Mmm, ice cream. • Fast network •More RAM Performance
- 46. More RAM = expensive Performance x2 4GB RAM 12 month Prices
- 54. Tips: rand() •Field names •Covered indexes
- 55. Tips: rand() •Field names •Covered indexes •Collections / databases
- 56. Backups What is the use case?
- 57. Backups What is the use case? Fixing user errors? Point in time restore? Disaster recovery?
- 65. mongostat