2020 pre fosdem mysql clone
Download as PPTX, PDF2 likes171 views
The document introduces MySQL Clone, a plugin that provides a simple way to create a replica of a MySQL instance. It allows cloning data locally or remotely from a "donor" server to a "recipient" server. The cloning process happens in stages, copying data files, page IDs, and redo logs to keep the recipient synchronized. Monitoring can be done via the performance_schema to check the clone's progress and status. Variables control resources used and networking during the clone operation. It is intended to provision new replication nodes and re-provision returning nodes.
![The big picture
© 2020 Oracle11
Client Network
Network
Clone
Plugin
Innodb
SE-2
[Future]
Clone
Plugin
Innodb
SE-2
[Future]
Recipient
Server
Donor
Server
Clone SQL
mysql> CLONE INSTANCE
FROM
clone_user@donor.host.com:3306
IDENTIFIED BY
"clone_password";
Connect](https://image.slidesharecdn.com/2020prefosdem-mysqlclone-200211155226/85/2020-pre-fosdem-mysql-clone-11-320.jpg)
2020 pre fosdem mysql clone
- 2. Safe harbor statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation. © 2020 Oracle2
- 3. What is MySQL Clone ? A simple and efficient way to create a MySQL instance replica © 2020 Oracle3
- 4. WhyYet Another Method ? • It’s easy to use: a single command • It’s a data snapshot with consistent GTID and binary log position • Data can be cloned locally or from remote • It’s resumable ! © 2020 Oracle4
- 5. It’s FASTER Too ! 5 https://mydbops.wordpress.com/2019/11/14/ mysql-clone-plugin-speed-test/ 2 servers CPU: 2 Cores RAM: 4 GB DISK: 200 GB SSD approx. 122.40 GB data from sysbench © 2020 Oracle
- 6. Clone: Use Cases © 2020 Oracle6
- 7. Provision a new replication node © 2020 Oracle7 Master Slave Slave 2
- 8. Re-provision a returning node © 2020 Oracle8 BinlogMaster SlaveData ??
- 9. Clone: Operation © 2020 Oracle9
- 10. Terminology © 2020 Oracle10 Donor RecipientData
- 11. The big picture © 2020 Oracle11 Client Network Network Clone Plugin Innodb SE-2 [Future] Clone Plugin Innodb SE-2 [Future] Recipient Server Donor Server Clone SQL mysql> CLONE INSTANCE FROM clone_user@donor.host.com:3306 IDENTIFIED BY "clone_password"; Connect
- 12. Prerequisites on the Donor • Install the plugin: INSTALL PLUGIN clone SONAME 'mysql_clone.so'; • Grant BACKUP_ADMIN to a user account CREATE USER foo …; GRANT BACKUP_ADMIN TO foo; © 2020 Oracle12
- 13. Prerequisites on the Recipient • Install the plugin: INSTALL PLUGIN clone SONAME 'mysql_clone.so'; • Grant CLONE_ADMIN to a user account CREATE USER bar …; GRANT CLONE_ADMIN TO bar; • Allow the donor host SET GLOBAL clone_valid_donor_list='donor_computer:3306'; © 2020 Oracle13
- 14. On the recipient, as “bar” CLONE INSTANCE FROM foo@donor_computer:3306 IDENTIFIED BY 'foo_pwd‘ DATA DIRECTORY = 'clone_dir' REQUIRE SSL © 2020 Oracle14
- 15. Clone: What Has Just Happened ? © 2020 Oracle15
- 16. Clone Stages © 2020 Oracle16 Init Data Files File Copy Modified Page IDs Page Copy Redo Logs Redo Copy Done
- 17. Clone: Network Failure Recovery © 2020 Oracle17 Donor RecipientData Less than 5 min
- 18. Clone Monitoring: performance_schema.clone_status mysql> SELECT STATE FROM performance_schema.clone_status; +-----------+ | STATE | +-----------+ | Completed | +-----------+ © 2020 Oracle18
- 19. Clone Monitoring: performance_schema.clone_progress mysql> SELECT STAGE, STATE, END_TIME FROM performance_schema.clone_progress; +-----------+-----------+----------------------------+ | stage | state | end_time | +-----------+-----------+----------------------------+ | DROP DATA | Completed | 2019-10-27 14:45:43.141261 | | FILE COPY | Completed | 2019-10-27 14:45:44.457572 | | PAGE COPY | Completed | 2019-10-27 14:45:44.577330 | | REDO COPY | Completed | 2019-10-27 14:45:44.679570 | | FILE SYNC | Completed | 2019-10-27 14:45:44.918547 | | RESTART | Completed | 2019-10-27 14:45:48.583565 | | RECOVERY | Completed | 2019-10-27 14:45:49.626595 | +-----------+-----------+----------------------------+ © 2020 Oracle19
- 20. Performance And Resource Use Control © 2020 Oracle20 Variable Type Meaning clone_max_concurrency Int (Maximum) number of cloning threads clone_autotune_concurrency Bool Enable dynamic spawning of cloning threads clone_max_network_bandwidth Int, Mbps (recipient only) limit network usage clone_max_data_bandwidth Int, Mbps (recipient only) limit clone transfer rate
- 21. Clone Limitations • Does not clone configuration values • Does not clone binary logs • Supports only InnoDB • Blocks all concurrent DDLs on the Donor • Will FULLY override recipient data • Cloning can be done only to the SAME server version • Doesn’t work over the X protocol © 2020 Oracle21
- 22. Further Reading • Reference Manual • https://dev.mysql.com/doc/refman/8.0/en/clone-plugin.html • Blogs • https://mysqlserverteam.com/clone-create-mysql-instance-replica/ • https://mysqlhighavailability.com/mysql-innodb-cluster-automatic-node-provisioning • https://mysqlhighavailability.com/a-breakthrough-in-usability-automatic-node-provisioning • https://mysqlhighavailability.com/automatic-provisioning-in-group-replication • InnoDB cluster video • https://lefred.be/content/migrate-from-a-single-mysql-instance-to-mysql-innodb-cluster- using-clone-plugin/ © 2020 Oracle22
- 23. Thank you ! Georgi Kodinov, Software Development Director Georgi.Kodinov@oracle.com https://www.slideshare.net/GeorgiKodinov © 2020 Oracle23
Editor's Notes
- #9: * The node is far behind and binary log sync would take long time. * Binary logs are not available to sync
- #11: Recipient: Server instance, where the clone statement is executed. Donor: The source server instance from which the data is going to be cloned. The Donor collects all its data and metadata stored in the InnoDB storage engine at a consistent snapshot to replace the data in Recipient. After successful execution of a CLONE SQL statement the Recipient server is restarted automatically. The restart involves recovery. Once restarted, recipient is a clone of donor server!
- #17: Donor dynamic snapshot is transferred in three stages. File Copy Page Copy Redo Copy
- #18: Recipient: Server instance, where the clone statement is executed. Donor: The source server instance from which the data is going to be cloned. The Donor collects all its data and metadata stored in the InnoDB storage engine at a consistent snapshot to replace the data in Recipient. After successful execution of a CLONE SQL statement the Recipient server is restarted automatically. The restart involves recovery. Once restarted, recipient is a clone of donor server!
- #19: One or zero rows ID A unique cloning operation identifier in the current MySQL server instance. PID Process list ID of the session executing the cloning operation. STATE Current state of the cloning operation. Values include Not Started, In Progress, Completed, and Failed. BEGIN_TIME A timestamp in 'YYYY-MM-DD hh:mm:ss[.fraction]' format that shows when the cloning operation started. END_TIME A timestamp in 'YYYY-MM-DD hh:mm:ss[.fraction]' format that shows when the cloning operation finished. Reports NULL if the operation has not ended. SOURCE The donor MySQL server address in 'HOST:PORT' format. The column displays 'LOCAL INSTANCE' for a local cloning operation. DESTINATION The directory being cloned to. ERROR_NO The error number reported for a failed cloning operation. ERROR_MESSAGE The error message string for a failed cloning operation. BINLOG_FILE The name of the binary log file up to which data is cloned. BINLOG_POSITION The binary log file offset up to which data is cloned. GTID_EXECUTED The GTID value for the last cloned transaction.
- #20: ID A unique cloning operation identifier in the current MySQL server instance. STAGE The name of the current cloning stage. Stages include DROP DATA, FILE COPY, PAGE_COPY, REDO_COPY, FILE_SYNC, RESTART, and RECOVERY. STATE The current state of the cloning stage. States include Not Started, In Progress, and Completed. BEGIN_TIME A timestamp in 'YYYY-MM-DD hh:mm:ss[.fraction]' format that shows when the cloning stage started. Reports NULL if the stage has not started. END_TIME A timestamp in 'YYYY-MM-DD hh:mm:ss[.fraction]' format that shows when the cloning stage finished. Reports NULL if the stage has not ended. THREADS The number of concurrent threads used in the stage. ESTIMATE The estimated amount of data for the current stage, in bytes. DATA The amount of data transferred in current state, in bytes. NETWORK The amount of network data transferred in the current state, in bytes. DATA_SPEED The current actual speed of data transfer, in bytes per second. This value may differ from the requested maximum data transfer rate defined by clone_max_data_bandwidth. NETWORK_SPEED The current speed of network transfer in bytes per second.
- #21: Clone plugin uses multiple connections from recipient to donor server. - clone_max_concurrency limits maximum connections Clone operation starts with 1 connection thread and threads are increased in steps. Auto tuning checks and stop spawning new connections if no significant performance improvement is noticed. - clone_autotune_concurrency If cpu, i/o, Network resource is saturated then increasing connections/threads could degrade performance and could also impact donor performance. Auto tuning helps here.