Training Slides: Intermediate 205: Configuring Tungsten Replicator to Extract from Amazon RDS
0 likes63 views
This document outlines the process of configuring Tungsten Replicator to extract data from Amazon RDS, detailing key components such as extraction prerequisites, installation parameters, and direct connection methods. It emphasizes the challenges of accessing binary logs directly from RDS and offers solutions like using direct mode and unprivileged user support. Resources for further information and tools for managing the replication process, including command line options, are also provided.
Training Slides: Intermediate 205: Configuring Tungsten Replicator to Extract from Amazon RDS
- 1. Intermediate : Configuring Tungsten Replicator to Extract from Amazon RDS
- 2. Topics In this short course we will: β’ Review the Replicator β’ Explain how we can extract from Amazon RDS β’ Review prerequisites required to extract from Amazon RDS β’ Discuss installation parameters (tungsten.ini) β’ Walk through an installation β’ Recap key resources and tools Course Prerequisite Learning β Basics: Introduction to Tungsten Replicator β Visit www.continuent.com/videos/ to watch this recording 2
- 3. Extracting from RDS using Tungsten Replicator
- 4. Extraction from RDS β’ When extracting from MySQL, Tungsten Replicator usually reads the MySQL binary logs directly from the filesystem β’ Amazon RDS does NOT provide filesystem access to the binary logs, nor any access to the RDS host itself β’ However, we can configure two aspects of Tungsten Replicator to extract from RDS: β Direct mode β allows us to read from the binary logs from the RDS instance over the network, rather than accessing them on the filesystem β Unprivileged user support β allows the user to access and update information within RDS without requiring SUPER privileges which are unavailable in RDS β’ Tungsten Replicator can then connect to the RDS instance as a regular MySQL client and request the binary logs β’ Note that by using the above features, we can extract from any remote MySQL database. For example, the Tungsten Replicator Extractor can be installed on a secondary server, and extract from a remote MySQL database without having to install Tungsten Replicator directly on the database server 4
- 5. Tungsten Replicator 5 Tungsten Replicator: Extractor THL Slave Replicator: Applier Amazon RDS Offboard Host Slave Replicator: Applier Slave Replicator: Applier Slave Replicator: Applier Slave Replicator: Applier
- 7. Prerequisites (specific to RDS) β’ RDS Instance o Note IP/Endpoint o Also note user name and password for RDS β’ Ensure that security group configuration allows access from the offboard host to the RDS instance β’ Configure binlog retention hours in RDS to allow enough time for Tungsten Replicator to read the binary logs before they are purged. For instance, in RDS: call mysql.rds_set_configuration('binlog retention hours', 24); β’ Configure max_allowed_packet_size in RDS. This should be large enough to read entire transactions from the RDS instance. β’ Configure the tungsten user in RDS: o GRANT ALL ON tungsten_service.* to tungsten@'%' IDENTIFIED BY 'secretβ; o GRANT SELECT ON *.* TO tungsten@'%' IDENTIFIED BY 'secretβ; o GRANT REPLICATION SLAVE ON *.* TO tungsten@'%' IDENTIFIED BY 'secretβ; β’ RDS by default is configured for MIXED-mode replication. Depending on your topology, you may have to change this to ROW-based replication. 7
- 8. Prerequisites (standard Replicator host) β’ Review docs β http://docs.continuent.com/tungsten-replicator-5.3/prerequisite.html β’ Host Prerequisites β OS user β /etc/hosts β sudoers and ssh β Ruby β Java β’ Network Prerequisites β Review port requirements β’ MySQL Prerequisites β my.cnf settings β tungsten user accounts 8
- 9. RDS/Direct Parameters for tungsten.ini β’ toplogy=direct β Replication toplogy, use βdirectβ for reading directly from a MySQL server β’ master=db1 β The master is the host that will be serving the THL to the appliers, typically it is the offboard host where the replicator extractor is installed β’ direct-datasource-host=rds_endpoint.us-east-1.rds.amazonaws.com β The RDS endpoint β’ direct-datasource-port=3306 β The port in RDS where the replicator will connect to read binary logs. Typically 3306 β’ direct-datasource-user=rds_user β RDS user name with privileges to read the binary logs (ie tungsten) β’ direct-datasource-password=rds_password β Password for the rds user above β’ privileged-master=false β Disable privileged updates, which are not available in RDS (i.e. require SUPER privileges) 9
- 10. RDS to Local MySQL Demo Amazon RDS Tungsten Replicator: Extractor THL Slave Replicator: Applier 10
- 12. Tools : trepctl 12 β’ βtrepctl statusβ can be run from any node to view the status of the local replicator β’ βtrepctl status βr 3β will show status output refreshed every 3 second until CTRL+C β’ βtrepctl qsβ provides a quick summary overview of the local replicator β’ βtrepctl perfβ provides deeper diagnostics of the different stages in the replicators $ trepctl qs State: east Online for 21.069s, running for 45.654s Latency: 0.837s from DB commit time on db1 into THL 21.839s since last database commit Sequence: 1 last applied, 0 transactions behind (0-1 stored) estimate 0.00s before synchronization
- 13. Log Files 13 β’ The /opt/continuent/service_logs/ directory contains both text files and symbolic links. β’ Links in the service_logs directory go to following subdirectory: β /opt/continuent/tungsten/tungsten-replicator/log/ tungsten@db1:/opt/continuent/service_logs $ ll total 116 lrwxrwxrwx 1 tungsten tungsten 60 Jan 06 09:52 trepsvc.log -> /opt/continuent/tungsten/tungsten-replicator/log/trepsvc.log
- 14. Next Steps β’ If you are interested in knowing more about the software and would like to try it out for yourself, please contact our sales team who will be able to take you through the details and setup a POC β sales@continuent.com β’ Read the documentation at http://docs.continuent.com/tungsten-replicator-5.3/index.html β’ Follow us on Twitter @Continuent or www.facebook.com/Continuent β’ Subscribe to our Tungsten University YouTube channel! http://tinyurl.com/TungstenUni β’ Visit the events calendar for upcoming Webinars and Training sessions at www.continuent.com/events/ 15
- 15. For more information, contact us: Eero Teerikorpi Founder, CEO eero.teerikorpi@continuent.com Eric Stone COO eric.stone@continuent.com MC Brown VP Products mc.brown@continuent.com Matthew Lang Director, Professional Services Americas matthew.lang@continuent.com