Training Slides: Advanced 302: Performing Schema Changes in a Multi-Site/Multi-Master Cluster
0 likes64 views
This document outlines a training course covering schema changes in a multi-site multi-master clustering environment, including challenges, rolling maintenance processes, and command line tools for management. Key topics include the architecture of tungsten clusters, zero-downtime maintenance, and best practices for applying schema changes. Additional resources for further learning and support contacts are provided.
![Schema Changes – The Challenges
• Table-Level Locking
– Large tables with millions of rows
– Blocks Application
– Clogs Replication
– Increased Memory usage
• Application Compatibility
– Does you application handle both before/after versions of the table?
• Test and Baseline
• ALTER TABLE <table> ADD <column> <definition> [BEFORE|AFTER] <column>;
– Enable Replicator filters
• https://docs.continuent.com/tungsten-clustering-5.2/deployment-msmm-monitoring-
schemachange.html
7](https://image.slidesharecdn.com/continuent-training-advanced-schema-changes-20171003-200427195249/85/Training-Slides-Advanced-302-Performing-Schema-Changes-in-a-Multi-Site-Multi-Master-Cluster-7-320.jpg)
![Tools : cctrl
• “cctrl” can be run from any node within a cluster to control the local cluster and gather
information
• Type “help” to get a full list of all commands available
• “ls” provides a summary overview of the entire cluster
• “datasource <node> shun”
– Isolates node in cluster
• “replicator <node> offline”
– Stop node receiving replication data
• “datasource <node> recover”
– Re-Introduce node to cluster
• “switch” or “switch to <node>”
13
[LOGICAL] /nyc > ls
COORDINATOR[db1:AUTOMATIC:ONLINE]
ROUTERS:
+----------------------------------------------------------------------------+
|connector@db1.tt-0509[25431](ONLINE, created=0, active=0) |
|connector@db2.tt-0509[24548](ONLINE, created=0, active=0) |
|connector@db3.tt-0509[24803](ONLINE, created=0, active=0) |
+----------------------------------------------------------------------------+
DATASOURCES:
+----------------------------------------------------------------------------+
|db1(master:ONLINE, progress=0, THL latency=0.776) |
|STATUS [OK] [2017/08/17 11:00:53 AM UTC] |
+----------------------------------------------------------------------------+
| MANAGER(state=ONLINE) |
| REPLICATOR(role=master, state=ONLINE) |
| DATASERVER(state=ONLINE) |
| CONNECTIONS(created=0, active=0) |
+----------------------------------------------------------------------------+
. . .
13](https://image.slidesharecdn.com/continuent-training-advanced-schema-changes-20171003-200427195249/85/Training-Slides-Advanced-302-Performing-Schema-Changes-in-a-Multi-Site-Multi-Master-Cluster-13-320.jpg)
![Tools : tpm connector
15
• Simple and quick way to connect to MySQL CLI
• Tungsten Commands to query database and cluster stats
– Connector-based Tungsten commands are NOT available in Bridge Mode
– This is a good way to tell if you are in Bridge mode – if no commands are available, then you are in Bridge mode
– “tungsten help” will show all commands available
mysql> tungsten help;
+---------------------------------------------------------------------------------------------------------------------------------+
| Message |
+---------------------------------------------------------------------------------------------------------------------------------+
| tungsten connection status: display information about the connection used for the last request ran |
| tungsten connection count: gives the count of current connections to each one of the cluster datasources |
| tungsten cluster status: prints detailed information about the cluster view this connector has |
| tungsten show [full] processlist: list all running queries handled by this connector instance |
| tungsten show variables [like '<string>']: list connector configuration options in use. The <string> may contain '%' wildcards |
| tungsten flush privileges: reload user.map and refresh user credentials |
| tungsten mem info: display memory information about current JVM |
| tungsten gc: calls garbage collector |
| tungsten help: display this help message |
+---------------------------------------------------------------------------------------------------------------------------------+
9 rows in set (0.00 sec)
15](https://image.slidesharecdn.com/continuent-training-advanced-schema-changes-20171003-200427195249/85/Training-Slides-Advanced-302-Performing-Schema-Changes-in-a-Multi-Site-Multi-Master-Cluster-15-320.jpg)
Training Slides: Advanced 302: Performing Schema Changes in a Multi-Site/Multi-Master Cluster
- 1. Advanced: Performing Schema Changes in a Multi-Site Multi-Master Environment 1
- 2. Topics In this short course we will: • Review the Cluster Architecture • Explore the challenges with Schema Changes • Describe the Rolling Maintenance Process • Describe the process of applying changes • Demo • Re-Cap Commands and resources used during the demo Course Prerequisite Learning – Basics: Introduction to Clustering – Intermediate: Performing Cluster Maintenance with Zero Downtime – Advanced: Multi-Site/Multi-Master Tungsten Clustering Deployments – Visit the Continuent website or Tungsten University on YouTube to watch these recordings • Continuent website https://www.continuent.com/videos/ • Tungsten University on YouTube https://www.youtube.com/channel/UCZ9iU-7nT1RLNnJvITFCsWA or http://tinyurl.com/TungstenUni 2 2
- 5. Tungsten Multi-Site/Multi-Master Cluster Architecture 5 5
- 7. Schema Changes – The Challenges • Table-Level Locking – Large tables with millions of rows – Blocks Application – Clogs Replication – Increased Memory usage • Application Compatibility – Does you application handle both before/after versions of the table? • Test and Baseline • ALTER TABLE <table> ADD <column> <definition> [BEFORE|AFTER] <column>; – Enable Replicator filters • https://docs.continuent.com/tungsten-clustering-5.2/deployment-msmm-monitoring- schemachange.html 7
- 9. Rolling maintenance proceeds node-by-node starting with the slaves and ending with the master Slave upgrade Slave upgrade Switch Master upgrade • Shun slave • Upgrade MySQL • Return node to cluster • Discard and re-provision upon failure • Repeat for remaining slave(s) • Switch master to promote an upgraded slave • Upgrade old master • Maintenance is now done! 9
- 10. Rolling maintenance • set policy maintenance on both clusters • On Slave Node: – datasource <node> shun – replicator <node> offline • Perform Schema Change ensuring session level bin logging disabled – datasource <node> recover • Repeat on all Slave Nodes in both cluster • Stop cross-cluster replicators • Promote a Slave in each cluster to Master – switch • Repeat steps on old Master after switch complete • Bring cross-cluster replicators online and return cluster Automatic 10
- 11. Rolling Maintenance Demo - Tungsten Clustering and Tungsten Replicator 5.2.1 - Connectors in Port-Based Routing mode - Amazon AWS EC2 - MySQL Community 5.7.19 - Java 1.8 - Ruby 1.9 11
- 13. Tools : cctrl • “cctrl” can be run from any node within a cluster to control the local cluster and gather information • Type “help” to get a full list of all commands available • “ls” provides a summary overview of the entire cluster • “datasource <node> shun” – Isolates node in cluster • “replicator <node> offline” – Stop node receiving replication data • “datasource <node> recover” – Re-Introduce node to cluster • “switch” or “switch to <node>” 13 [LOGICAL] /nyc > ls COORDINATOR[db1:AUTOMATIC:ONLINE] ROUTERS: +----------------------------------------------------------------------------+ |connector@db1.tt-0509[25431](ONLINE, created=0, active=0) | |connector@db2.tt-0509[24548](ONLINE, created=0, active=0) | |connector@db3.tt-0509[24803](ONLINE, created=0, active=0) | +----------------------------------------------------------------------------+ DATASOURCES: +----------------------------------------------------------------------------+ |db1(master:ONLINE, progress=0, THL latency=0.776) | |STATUS [OK] [2017/08/17 11:00:53 AM UTC] | +----------------------------------------------------------------------------+ | MANAGER(state=ONLINE) | | REPLICATOR(role=master, state=ONLINE) | | DATASERVER(state=ONLINE) | | CONNECTIONS(created=0, active=0) | +----------------------------------------------------------------------------+ . . . 13
- 14. Tools : trepctl 14 • “trepctl status” can be run from any node within a cluster to view the status of the local replicator • “trepctl status –r 3” will show status output refreshed every 3 second until CTRL+C • “trepctl qs” provides a quick summary overview of the local replicator • “trepctl perf” provides deeper diagnostics of the different stages in the replicators • “trepctl offline|online” $ trepctl qs State: east Online for 21.069s, running for 45.654s Latency: 0.837s from DB commit time on db1 into THL 21.839s since last database commit Sequence: 1 last applied, 0 transactions behind (0-1 stored) estimate 0.00s before synchronization 14
- 15. Tools : tpm connector 15 • Simple and quick way to connect to MySQL CLI • Tungsten Commands to query database and cluster stats – Connector-based Tungsten commands are NOT available in Bridge Mode – This is a good way to tell if you are in Bridge mode – if no commands are available, then you are in Bridge mode – “tungsten help” will show all commands available mysql> tungsten help; +---------------------------------------------------------------------------------------------------------------------------------+ | Message | +---------------------------------------------------------------------------------------------------------------------------------+ | tungsten connection status: display information about the connection used for the last request ran | | tungsten connection count: gives the count of current connections to each one of the cluster datasources | | tungsten cluster status: prints detailed information about the cluster view this connector has | | tungsten show [full] processlist: list all running queries handled by this connector instance | | tungsten show variables [like '<string>']: list connector configuration options in use. The <string> may contain '%' wildcards | | tungsten flush privileges: reload user.map and refresh user credentials | | tungsten mem info: display memory information about current JVM | | tungsten gc: calls garbage collector | | tungsten help: display this help message | +---------------------------------------------------------------------------------------------------------------------------------+ 9 rows in set (0.00 sec) 15
- 16. Log Files 16 • The /opt/continuent/service_logs/ directory contains both text files and symbolic links for cluster log files. • Links in the service_logs directory go to one of three (3) subdirectories: – /opt/continuent/tungsten/tungsten-connector/log/ – /opt/continuent/tungsten/tungsten-manager/log/ – /opt/continuent/tungsten/tungsten-replicator/log/ tungsten@db1:/opt/continuent/service_logs $ ll total 116 lrwxrwxrwx 1 tungsten tungsten 61 Jun 22 09:52 connector.log -> /opt/continuent/tungsten/tungsten-connector/log/connector.log lrwxrwxrwx 1 tungsten tungsten 62 Jun 22 09:52 mysqldump.log -> /opt/continuent/tungsten/tungsten-replicator/log/mysqldump.log lrwxrwxrwx 1 tungsten tungsten 55 Jun 22 09:52 tmsvc.log -> /opt/continuent/tungsten/tungsten-manager/log/tmsvc.log lrwxrwxrwx 1 tungsten tungsten 60 Jun 22 09:52 trepsvc.log -> /opt/continuent/tungsten/tungsten-replicator/log/trepsvc.log lrwxrwxrwx 1 tungsten tungsten 63 Jun 22 09:52 xtrabackup.log -> /opt/continuent/tungsten/tungsten-replicator/log/xtrabackup.log 16
- 17. Tools : tpm diag 17 • Provides support engineers with an entire overview of the cluster state, by: – Gathering point-in-time status of all components in a cluster – Gathering log files of all components in a cluster, including database logs to provide historical information – Bundles everything into one easy zip file that can be attached to a support case – ALWAYS create a diag package when you contact support for assistance • tungsten_send_diag – Executes “tpm diag” to generate the diagnostic package – Automatically uploads the package to support – https://docs.continuent.com/tungsten-clustering-5.2/cmdline-tools-tungsten_send_diag.html tungsten@db1:/opt/continuent/service_logs $ tungsten_send_diag –d –c 1234 17
- 18. Next Steps • If you are interested in knowing more about the clustering software and would like to try it out for yourself, please contact our sales team who will be able to take you through the details and setup a POC – sales@continuent.com • Read the documentation at http://docs.continuent.com/tungsten-clustering-5.2/index.html • Subscribe to our Tungsten University YouTube channel! http://tinyurl.com/TungstenUni • Visit the events calendar on our website for upcoming Webinars and Training Sessions https://www.continuent.com/events/ 18 18
- 19. For more information, contact us: MC Brown VP Products mc.brown@continuent.com Chris Parker Director, Professional Services EMEA & APAC chris.parker@continuent.com Matthew Lang Director, Professional Services Americas matthew.lang@continuent.com Eero Teerikorpi Founder, CEO eero.teerikorpi@continuent.com +1 (408) 431-3305 Eric Stone COO eric.stone@continuent.com