Training Slides: 252 - Monitoring & Troubleshooting
0 likes65 views
The document is a training overview for monitoring and troubleshooting MySQL replication issues, covering topics such as common incidents, trigger behavior, log file management, and recovery strategies. Key points include setting up monitoring alerts, understanding error logs, and using scripts for effective monitoring. The course concludes with a brief on the next session, which will focus on filtering.
Training Slides: 252 - Monitoring & Troubleshooting
- 1. The MySQL Availability Company Tungsten Replicator Master Class Intermediate: Replicator Monitoring & Troubleshooting Chris Parker, Customer Success Director, EMEA & APAC
- 2. Topics In this short course, we will โข Discuss Monitoring & Troubleshooting โข Review Common Issues โข Triggers โข Finding and understanding Log Files โข Handling (and recovering from) Failures โข Skipping Transactions โข Resetting โข Scripts for Monitoring
- 3. Most Common Issues Common Incidents: โข Network Failures โข Disk Capacity โข OOM โข MySQL Crashes โข MySQL Max Connections โข Human Error โข Deletion of THL โข Incorrect Shutdown Procedures โข Writing direct to a replica Things to check: โข Set up Monitoring / Alerts โข Check status outputs โข Check Log Files โข Check OS โข Check MySQL state
- 4. Triggers โข Triggers are an issue in all MySQL<>MySQL replication topologies โข The combination of the binlog_format (ROW vs MIXED) and the TRIGGER type (DETERMINISTIC vs NONDETERMINISTIC) will result in triggers behaving differently. โข Either: โข Avoid Triggers altogether if possible, or โข Code Triggers to intelligently fire ONLY on Primary hosts โข Read the docs for a full explanation: โข https://docs.continuent.com/tungsten-clustering-6.1/troubleshooting-known-issues- triggers.html
- 5. Log Files โข /opt/continuent/service_logs/ โข trepsvc.log for complete replicator log โข Use logrotate to manage โข `replicator dump` โข Dumps memory stack to log file โข Can be useful in support cases โข Automatically issued by `tpm diag`
- 6. Recovering from Failures โข Enable auto-recovery properties โข Review log files to understand errors โข Simply try restarting replicator โ YES this sometimes works! โข Which node is experiencing an error? โข If statement clash, is skipping safe? โข If database crashed, have you lost binary logs? โข Have you lost THL Files? โข Worse case โ Reset and Re-provision
- 7. Monitoring Scripts โข Nagios-compatible scripts are located in โข These scripts will only check the local host. They do not check other hosts. โข check_tungsten_online โ report ok if all services are online โข check_tungsten_services โr โ report ok if replicator services are healthy โข check_tungsten_latency โ used for Nagios style warning and critical replicator latency thresholds โข Write your own! - New API coming in v7 $CONTINUENT_ROOT/tungsten/cluster-home/bin/check_tungsten_*
- 8. Summary What we have learnt today โข Reviewed Common Issues โข Discussed Triggers โข Looked at Log Files โข Discussed Failures โข Skipping Transactions โข Resetting โข Reviewed Scripts for Monitoring
- 9. Next Steps In the next session we will โข Explore Filtering!
- 10. THANK YOU FOR LISTENING continuent.com The MySQL Availability Company Chris Parker, Customer Success Director, EMEA & APAC