Advanced Replication
Download as PPTX, PDF2 likes1,655 views
The document provides a detailed overview of advanced replication in MongoDB, including configuration options for replica sets, operational details such as heartbeat timing, and the use of oplogs. It explains the implementation of multi-data center setups for disaster recovery and analytics, along with commands for managing replica sets and maintaining strong consistency. The document highlights best practices for configurations, monitoring, and write concerns to optimize performance and reliability in replica set environments.
![Configuration Options
> conf = {
_id : "mySet",
members : [
{_id : 0, host : "A"},
{_id : 1, host : "B"},
{_id : 2, host : "C", "arbiter" : true}
]
}
> rs.initiate(conf)](https://image.slidesharecdn.com/8-advancedreplication-marcschrewing-130408150142-phpapp01/85/Advanced-Replication-4-320.jpg)
![Tagging Example
{
_id : "mySet",
members : [
{_id : 0, host : "A", tags : {"dc": "ny"}},
{_id : 1, host : "B", tags : {"dc": "ny"}},
{_id : 2, host : "C", tags : {"dc": "sf"}},
{_id : 3, host : "D", tags : {"dc": "sf"}},
{_id : 4, host : "E", tags : {"dc": "cloud"}}],
settings : {
getLastErrorModes : {
allDCs : {"dc" : 3},
someDCs : {"dc" : 2}} }
}
> db.blogs.insert({...})
> db.runCommand({getLastError : 1, w : "someDCs"})
> db.getLastErrorObj({"someDCs"})](https://image.slidesharecdn.com/8-advancedreplication-marcschrewing-130408150142-phpapp01/85/Advanced-Replication-28-320.jpg)
Advanced Replication
- 1. #MongoDBDays - @m4rcsch Advanced Replication Marc Schwering Solutions Architect, 10gen
- 4. Configuration Options > conf = { _id : "mySet", members : [ {_id : 0, host : "A"}, {_id : 1, host : "B"}, {_id : 2, host : "C", "arbiter" : true} ] } > rs.initiate(conf)
- 7. Implementation details • Heartbeat every 2 seconds – Times out in 10 seconds • Local DB (not replicated) – system.replset – oplog.rs • Capped collection • Idempotent version of operation stored
- 9. Op(erations) Log is idempotent > db.replsettest.insert({_id:1,value:1}) { "ts" : Timestamp(1350539727000, 1), "h" : NumberLong("6375186941486301201"), "op" : "i", "ns" : "test.replsettest", "o" : { "_id" : 1, "value" : 1 } } > db.replsettest.update({_id:1},{$inc:{value:10}}) { "ts" : Timestamp(1350539786000, 1), "h" : NumberLong("5484673652472424968"), "op" : "u", "ns" : "test.replsettest", "o2" : { "_id" : 1 }, "o" : { "$set" : { "value" : 11 } } }
- 11. Single operation can have many entries > db.replsettest.update({},{$set:{name : ”foo”}, false, true}) { "ts" : Timestamp(1350540395000, 1), "h" : NumberLong("- 4727576249368135876"), "op" : "u", "ns" : "test.replsettest", "o2" : { "_id" : 2 }, "o" : { "$set" : { "name" : "foo" } } } { "ts" : Timestamp(1350540395000, 2), "h" : NumberLong("- 7292949613259260138"), "op" : "u", "ns" : "test.replsettest", "o2" : { "_id" : 3 }, "o" : { "$set" : { "name" : "foo" } } } { "ts" : Timestamp(1350540395000, 3), "h" : NumberLong("- 1888768148831990635"), "op" : "u", "ns" : "test.replsettest", "o2" : { "_id" : 1 }, "o" : { "$set" : { "name" : "foo" } } }
- 12. Operations
- 13. Maintenance and Upgrade • No downtime • Rolling upgrade/maintenance – Start with Secondary – Primary last – Commands: • rs.stepDown(<secs>) • db.version() • db.serverBuildInfo()
- 14. Upgrade Demo
- 15. Replica Set – 1 Data Center • Single datacenter • Single switch & power • Points of failure: – Power – Network – Data center – Two node failure • Automatic recovery of single node crash
- 16. Replica Set – 2 Data Centers • Multi data center • DR node for safety • Can’t do multi data center durable write safely since only 1 node in distant DC
- 17. Replica Set – 2 Data Centers • Analytics • Disaster Recovery • Batch Jobs • Options – low or zero priority – hidden – slaveDelay
- 18. Replica Set – 3 Data Centers • Three data centers • Can survive full data center loss • Can do w= { dc : 2 } to guarantee write in 2 data centers (with tags)
- 19. Replica Set – 3+ Data Centers Secondary Primary Secondary Secondary Secondary delayed Secondary Secondary
- 20. Commands • Managing – rs.conf() – rs.initiate(<conf>) & rs.reconfig(<conf>) – rs.add(host:<port>) & rs.addArb(host:<port>) – rs.status() – rs.stepDown(<secs>) • Minority reconfig – rs.reconfig( cfg, { force : true} )
- 21. Options • Priorities • Hidden • Slave Delay • Disable indexes (on secondaries) • Default write concerns
- 25. Write Concern • Network acknowledgement • Wait for error • Wait for journal sync • Wait for replication – number – majority – Tags
- 27. Datacenter awareness (Tagging) • Control where data is written to, and read from • Each member can have one or more tags – tags: {dc: "ny"} – tags: {dc: "ny", subnet: "192.168", rack: "row3rk7"} • Replica set defines rules for write concerns • Rules can change without changing app code
- 28. Tagging Example { _id : "mySet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}}], settings : { getLastErrorModes : { allDCs : {"dc" : 3}, someDCs : {"dc" : 2}} } } > db.blogs.insert({...}) > db.runCommand({getLastError : 1, w : "someDCs"}) > db.getLastErrorObj({"someDCs"})
- 30. Write Concern with timeout settings : { getLastErrorModes : { allDCs : {"dc" : 3}, someDCs : {"dc" : 2}} } } > db.getLastErrorObj({"allDCs"},100); > db.getLastErrorObj({”someDCs"},500); > db.getLastErrorObj(1,500);
- 31. Read Preference Modes • 5 modes (new in 2.2) – primary (only) - Default – primaryPreferred – secondary – secondaryPreferred – Nearest When more than one node is possible, closest node is used for reads (all modes but primary)
- 32. Tagged Read Preference • Custom read preferences • Control where you read from by (node) tags – E.g. { "disk": "ssd", "use": "reporting" } • Use in conjunction with standard read preferences – Except primary
- 33. Tags {"dc.va": "rack1", disk:"ssd", ssd: "installed" } {"dc.va": "rack2", disk:"raid"} {"dc.gto": "rack1", disk:"ssd", ssd: "installed" } {"dc.gto": "rack2", disk:"raid”} > conf.settings = { getLastErrorModes: { MultipleDC : { "dc.va": 1, "dc.gto": 1}} > conf.settings = { "getLastErrorModes" : { "ssd" : { "ssd" : 1 },...
- 34. Tagged Read Preference { disk: "ssd" } JAVA: ReadPreference tagged_pref = ReadPreference.secondaryPreferred( new BasicDBObject("disk", "ssd") ); DBObject result = coll.findOne(query, null, tagged_pref);
- 35. Tagged Read Preference • Grouping / Failover {dc : "LON", loc : "EU"} {dc : "FRA", loc : "EU"} {dc : "NY", loc : "US”} DBObject t1 = new BasicDBObject("dc", "LON"); DBObject t2 = new BasicDBObject("loc", "EU"); ReadPreference pref = ReadPreference.primaryPreferred(t1, t2);
- 36. Conclusion
- 37. Best practices and tips • Odd number of set members • Read from the primary except for – Geographically distribution – Analytics (separate workload) • Use logical names not IP Addresses in configs • Set WriteConcern appropriately for what you are doing • Monitor secondaries for lag (Alerts in MMS)
- 38. #MongoDBDays - @m4rcsch Thank You Marc Schwering Solutions Architect, 10gen
Editor's Notes
- #5: PrimaryData memberSecondaryHot standbyArbitersVoting member
- #6: Remind the audience, that there are more options, multi dc failure etc. but this is for single machine failurePriorityFloating point number between 0..100Highest member that is up to date wins Up to date == within 10 seconds of primaryIf a higher priority member catches up, it will force election and win Slave DelayLags behind master by configurable time delay Automatically hidden from clientsProtects against operator errorsFat fingeringApplication corrupts data
- #8: Schemaoplog, design considerations of MongoDB.. Not only master slave.Tunable: based on your needs.
- #14: Upgrade/maintenanceCommon deployment scenarios
- #20: TODO: How people really using it? …Get Data from secodaries (chaining)A good question to ask the audience : 'Why wouldn't you set w={dc:3}'… Why would you ever do that? What would be the complications?
- #24: ConsistencyWrite preferencesRead preferences
- #30: Using 'someDCs' so that in the event of an outage, at least a majority of the DCs would receive the change. This favors availability over durability.
- #31: Using 'allDCs' because we want to make certain all DCs have this piece of data. If any of the DCs are down, this would timeout. This favors durability over availability.
- #35: Using 'someDCs' so that in the event of an outage, at least a majority of the DCs would receive the change. This favors availability over durability.