Preventing multi master conflicts with tungsten
1 like1,792 views
The document discusses multi-master replication using Continuent Tungsten and how it prevents data conflicts. It describes how Tungsten allows defining rules to specify which database can be updated by each master node. Tungsten then enforces these rules on slave nodes to prevent "unknown" or "unwanted" shards from being updated, which could lead to data duplication, inconsistency, loss or break replication. Examples are provided showing how conflict prevention works in different replication topologies.
![sample incorrect operation (2)
#3 $ trepct services | simple_services
alpha [slave]
seqno: 7 - latency: 0.136 - ONLINE
bravo [slave]
seqno: -1 - latency: -1.000 - OFFLINE:ERROR
charlie [master]
seqno: 66 - latency: 0.440 - ONLINE
©Continuent 2012. 40
Sunday, February 03, 13](https://image.slidesharecdn.com/preventingmultimasterconflictswithtungsten-130203071238-phpapp01/85/Preventing-multi-master-conflicts-with-tungsten-40-320.jpg)
Preventing multi master conflicts with tungsten
- 1. Preventing con!icts in Multi-master replication with Tungsten Giuseppe Maxia, QA Director, Continuent ©Continuent 2012. 1 Sunday, February 03, 13
- 2. Introducing Continuent • The leading provider of clustering and replication for open source DBMS • Our Product: Continuent Tungsten • Clustering - Commercial-grade HA, performance scaling and data management for MySQL • Replication - Open source, Flexible, high- performance data movement ©Continuent 2012 2 Sunday, February 03, 13
- 3. What is Tungsten Replicator? • Open source drop-in replacement for MySQL replication, providing: • Global transaction ID • Multiple masters • Multiple sources • Flexible topologies • Heterogeneous replication • Parallel replication • ... and more ©Continuent 2012. 3 Sunday, February 03, 13
- 4. Tungsten Replicator Overview Master Replicator THL Download transactions via network (Transactions + Metadata) DBMS Logs Slave Replicator THL Apply using JDBC (Transactions + Metadata) ©Continuent 2012 4 Sunday, February 03, 13
- 5. Tungsten Replication Service Pipeline Stage Stage Stage Extract Filter Apply Extract Filter Apply Extract Filter Apply Master Transaction In-Memory Slave DBMS History Log Queue DBMS ©Continuent 2012 5 Sunday, February 03, 13
- 6. Multiple Services Per Replicator NYC Replicator Service nyc Replicator Service nyc Service fra Frankfurt Replicator Service fra London ©Continuent 2012 6 Sunday, February 03, 13
- 7. Multi-Master Replication • Updates on 2+ sites (active-active mode) • Enables geographic distribution of data • No failover necessary if network fails or site becomes unavailable • Not all applications can handle multi-master • Applications must avoid con!icts • Careful testing required • Restoration of broken systems may not be easy ©Continuent 2012 7 Sunday, February 03, 13
- 8. Simple Multi-Master Con!guration NYC Replicator Frankfurt Replicator fra (slave) fra (master) nyc (master) nyc (slave) Database-to-Database ©Continuent 2012 8 Sunday, February 03, 13
- 9. Three-node Multi-Master Con!guration NYC Replicator Replicator Frankfurt fra (slave) fra (master) nyc (master) nyc (slave) lon (slave) lon (slave) Database-to-Database lon (master) fra (slave) nyc (slave) London Replicator ©Continuent 2012 9 Sunday, February 03, 13
- 10. multi-node star Con!guration NYC Replicator Replicator Frankfurt nyc (master) fra (master) lon (slave) lon (slave) Replicator lon (master) Singapore sin (master) fra (slave) lon (slave) nyc (slave) London (hub) sin (slave) Replicator ©Continuent 2012 10 Sunday, February 03, 13
- 11. CONFLICTS ©Continuent 2012 11 Sunday, February 03, 13
- 12. What's a con"ict • Data modi!ed by several sources (masters) • Creates one or more : • data loss (unwanted delete) • data inconsistency (unwanted update) • duplicated data (unwanted insert) • replication break ©Continuent 2012. 12 Sunday, February 03, 13
- 13. Data duplication 4 Matt 140 id name amount bravo alpha 1 Joe 100 2 Frank 110 3 Sue 100 BREAKS REPLICATION 4 Matt 130 charlie ©Continuent 2012. 13 Sunday, February 03, 13
- 14. auto_increment o#sets are not a remedy • A popular recipe • auto_increment_increment + auto_increment_offset • They don't prevent con"icts • They hide duplicates ©Continuent 2012. 14 Sunday, February 03, 13
- 15. Hidden data duplication 11 Matt 140 INSERT bravo alpha id name amount o#set 2 o#set 1 1 Joe 100 2 Frank 110 3 Sue 100 INSERT charlie 13 Matt 130 o#set 3 ©Continuent 2012. 15 Sunday, February 03, 13
- 16. Data inconsistency 3 Sue 108 UPDATE id name amount bravo alpha 1 Joe 100 2 Frank 110 3 Sue 100 UPDATE 3 Sue 105 charlie ©Continuent 2012. 16 Sunday, February 03, 13
- 17. Data loss 3 Sue 108 UPDATE id name amount bravo alpha 1 Joe 100 2 Frank 110 3 Sue 100 DELETE MAY BREAK REPLICATION record #3 charlie ©Continuent 2012. 17 Sunday, February 03, 13
- 18. con"ict handling strategies • resolving • after the fact • Needs information that is missing in async replication • avoiding • requires synchronous replication with 2pc • preventing used by Tungsten • setting and enforcing a split sources policy • Transforming and resolving planned for • all records are converted to INSERTs future use • con!icts are resolved within a given time window ©Continuent 2012 18 Sunday, February 03, 13
- 19. Tungsten Replicator Filters ©Continuent 2012. 19 Sunday, February 03, 13
- 20. Replicator Pipeline Architecture Tungsten Replicator Process Pipeline Stage Stage Stage Assign Extract Shard Apply Extract Apply Extract Apply ID Transaction History Log MySQL filter THL filter filter Binlog Slave DBMS Sunday, February 03, 13
- 21. Restrict replication to some schemas and tables ./tools/tungsten-installer --master-slave -a ... --svc-extractor-filters=replicate "--property=replicator.filter.replicate.do=test,*.foo" ... --start-and-report # test="test.*" -> same drawback as binlog-do-db in MySQL # *.foo = table 'foo' in any database # employees.dept_codes,employees.salaries => safest way ©Continuent 2012. 21 Sunday, February 03, 13
- 22. Multi-master: Con!ict prevention ©Continuent 2012. 22 Sunday, February 03, 13
- 23. Tungsten con"ict prevention in a nutshell 1. de!ne the rules (which master can update which database) 2. tell Tungsten the rules 3. de!ne the policy (error, drop, warn, or accept) 4. Let Tungsten enforce your rules ©Continuent 2012. 23 Sunday, February 03, 13
- 24. Tungsten Con"ict prevention facts • Sharded by database • De!ned dynamically • Applied on the slave services • methods: • error: make replication fail • drop: drop silently • warn: drop with warning ©Continuent 2012. 24 Sunday, February 03, 13
- 25. Tungsten con"ict prevention applicability • unknown shards • The schema being updated is not planned • actions: accept, drop, warn, error • unwanted shards • the schema is updated from the wrong master • actions: accept, drop, warn, error • whitelisted shards • can be updated by any master ©Continuent 2012. 25 Sunday, February 03, 13
- 26. Con"ict prevention directives --svc-extractor-filters=shardfilter replicator.filter.shardfilter.unknownShardPolicy=error replicator.filter.shardfilter.unwantedShardPolicy=error replicator.filter.shardfilter.enforceHomes=false replicator.filter.shardfilter.allowWhitelisted=false ©Continuent 2012. 26 Sunday, February 03, 13
- 27. con"ict prevention in a star topology alpha updates H employees Host1 m A master: alpha d C database: employees H m d ✔ A Host3 master: charlie (hub) B C database: vehicles Host2 ✔ master: bravo database: buildings B C ©Continuent 2012. 27 Sunday, February 03, 13
- 28. con"ict prevention in a star topology alpha updates H vehicles Host1 m A master: alpha d C database: employees H m d ✗ A Host3 master: charlie (hub) B C database: vehicles Host2 master: bravo database: buildings B C ©Continuent 2012. 28 Sunday, February 03, 13
- 29. con"ict prevention in a all-masters topology H alpha updates Host1 m employees A master: alpha d B C database: employees H m d ✔ A Host3 master: charlie (hub) C ✔ B database: vehicles A Host2 master: bravo database: buildings B C ©Continuent 2012. 29 Sunday, February 03, 13
- 30. con"ict prevention in a all-masters topology H charlie updates Host1 m vehicles A master: alpha d B C database: employees H m d ✔ A Host3 master: charlie (hub) C ✔ B database: vehicles A Host2 master: bravo database: buildings B C ©Continuent 2012. 30 Sunday, February 03, 13
- 31. con"ict prevention in a all-masters topology H alpha updates Host1 m buildings A master: alpha d B C database: employees H ✗ m d A Host3 Host2 ✗ A B C master: charlie (hub) database: vehicles master: bravo database: buildings B C ©Continuent 2012. 31 Sunday, February 03, 13
- 32. con"ict prevention in a all-masters topology H charlie updates Host1 m employees A master: alpha d B C database: employees H ✗ m d A Host3 Host2 ✗ A B C master: charlie (hub) database: vehicles master: bravo database: buildings B C ©Continuent 2012. 32 Sunday, February 03, 13
- 33. setting con"ict prevention rules trepctl -host host1 -service charlie shard -insert < shards.map cat shards.map shard_id master critical personnel alpha false buildings bravo false vehicles charlie false test whitelisted false # charlie is slave service in host 1 ©Continuent 2012. 33 Sunday, February 03, 13
- 34. setting con"ict prevention rules trepctl -host host2 -service charlie shard -insert < shards.map cat shards.map shard_id master critical personnel alpha false buildings bravo false vehicles charlie false test whitelisted false # charlie is slave service in host 2 ©Continuent 2012. 34 Sunday, February 03, 13
- 35. setting con"ict prevention rules trepctl -host host3 -service alpha shard -insert < shards.map trepctl -host host3 -service bravo shard -insert < shards.map cat shards.map shard_id master critical personnel alpha false buildings bravo false vehicles charlie false test whitelisted false # alpha and bravo are slave services in host 3 ©Continuent 2012. 35 Sunday, February 03, 13
- 36. Con"ict prevention demo • reminder • Server #1 can update "employees" • Server #2 can update "buildings" • Server #3 can update "vehicles" ©Continuent 2012. 36 Sunday, February 03, 13
- 37. Sample correct operation (1) mysql #1> create table employees.names( ... ) # all servers receive the table # all servers keep working well ©Continuent 2012. 37 Sunday, February 03, 13
- 38. Sample correct operation (2) mysql #2> create table buildings.homes( ... ) # all servers receive the table # all servers keep working well ©Continuent 2012. 38 Sunday, February 03, 13
- 39. Sample incorrect operation (1) mysql #2> create table employees.nicknames( ... ) # Only server #2 receives the table # slave service in hub gets an error # slave service in #1 does not receive anything ©Continuent 2012. 39 Sunday, February 03, 13
- 40. sample incorrect operation (2) #3 $ trepct services | simple_services alpha [slave] seqno: 7 - latency: 0.136 - ONLINE bravo [slave] seqno: -1 - latency: -1.000 - OFFLINE:ERROR charlie [master] seqno: 66 - latency: 0.440 - ONLINE ©Continuent 2012. 40 Sunday, February 03, 13
- 41. sample incorrect operation (3) #3 $ trepct -service bravo status NAME VALUE ---- ----- appliedLastEventId : NONE appliedLastSeqno : -1 appliedLatency : -1.0 (...) offlineRequests : NONE pendingError : Stage task failed: q-to-dbms pendingErrorCode : NONE pendingErrorEventId : mysql-bin.000002:0000000000001241;0 pendingErrorSeqno : 7 pendingExceptionMessage: Rejected event from wrong shard: seqno=7 shard ID=employees shard master=alpha service=bravo (...) ©Continuent 2012. 41 Sunday, February 03, 13
- 42. Fixing the issue mysql #1> drop table if exists employees.nicknames; mysql #1> create table if exists employees.nicknames ( ... ) ; #3 $ trepct -service bravo online -skip-seqno 7 # all servers receive the new table ©Continuent 2012. 42 Sunday, February 03, 13
- 43. Sample whitelisted operation mysql #2> create table test.hope4best( ... ) mysql #1> insert into test.hope4best values ( ... ) # REMEMBER: 'test' was explicitly whitelisted # All servers get the new table and records # But there is no protection against conflicts ©Continuent 2012. 43 Sunday, February 03, 13
- 44. Parting thoughts We are hiring ! http://continuent.com/about/careers ©Continuent 2012. 44 Sunday, February 03, 13
- 45. blog: http://datacharmer.blogspot.com twitter: @datacharmer Continuent Website: http://www.continuent.com Tungsten Replicator 2.0: http://tungsten-replicator.org ©Continuent 2012 45 Sunday, February 03, 13