Session 03 data_migration_at_scale_by_sameer
Download as PPTX, PDF0 likes94 views
The document details a presentation on migrating MySQL databases to AWS, focusing on the benefits of cloud migration for business growth, scalability, and availability. It discusses various approaches and methods for database migration, including logical backups and the use of tools like Percona XtraBackup. The speaker shares statistics on user activity and outlines a typical migration process, including deployment strategies and considerations for successful implementation.
Session 03 data_migration_at_scale_by_sameer
- 1. MySQL Migration To AWS As delivered in tech meetup #01 organized by www.edYoda.com on 22nd April 2018 at zekeLabs Bangalore
- 2. About me: Sameer Kumar DevOps Engineer @ Intuit 10+ years Ex- Siemens, Ex- Paytm, Ex - Snapdeal
- 3. Why Migrate Databases to Cloud ?
- 4. Why Migrate Databases to Cloud ? Business Growth
- 5. Why Migrate Databases to Cloud ? Business Growth Vertical Scaling
- 6. Why Migrate Databases to Cloud ? Business Growth Horizontal ScalingVertical Scaling
- 7. Availability Zone 3 POD-1 POD-2 POD-N Proxy Layer (Routing) Availability Zone 1 Proxy Layer (Routing) POD-1 POD-2 POD-N Availability Zone 2 Proxy Layer (Routing) POD-1 POD-2 POD-N Architecture – Scalability & Availability
- 8. Some statistics ~700K Login per day, ~1MM users login per day during Peak seasons. 103TB of financial data hosted in 118 MySQL physical servers. Size varies from 600 GB to 2 TB Nearly Zero Down Time
- 10. Different Approaches Different Time 2015 Logical Backup Restore mysqldump
- 11. Different Approaches Different Time 2016 DMS 2015 Logical Backup Restore mysqldump
- 12. Different Approaches Different Time 2017 Percona Backup 2016 DMS 2015 Logical Backup Restore mysqldump
- 13. Logical Backup Restore • mysqldump • rsync • tar On-Premise Slave EC2 with EBS RDS • MySQL Client • tar
- 14. Logical Backup Restore • mysqldump • rsync • tar On-Premise Slave EC2 with EBS RDS • MySQL Client • tar • Dump : 18 Hours for 2 TB • Sync to AWS: 180G Per Hour • Restoration: 24~30 Hours • Replication Catchup : 2 Days
- 15. Logical Backup Restore • mysqldump • rsync • tar On-Premise Slave EC2 with EBS RDS • MySQL Client • tar • Dump : 18 Hours for 2 TB • Sync to AWS: 180G Per Hour • Restoration: 24~30 Hours • Replication Catchup : 2 Days • max_allowed_packet • No MyISAM • Ignore MySQL schema • Don’t change user permission during migration
- 16. Percona XtraBackup • Percona Xtrabackup • AWS CLI • tar On-Premise Slave AWS S3 RDS
- 17. Percona XtraBackup • Percona Xtrabackup • AWS CLI • tar On-Premise Slave AWS S3 RDS • Only MySQL 5.6 • 6 TB Limit • Functions, Stored Procedures are not imported automatically • Can’t use Partial Backup
- 18. Percona XtraBackup • Percona Xtrabackup • AWS CLI • tar On-Premise Slave AWS S3 RDS • Only MySQL 5.6 • 6 TB Limit • Functions, Stored Procedures are not imported automatically • Can’t use Partial Backup • S3 Copy: 180G Per Hour • Restoration: 8 Hours for 2 TB • Replication Catchup : 2 Days
- 19. Blue/Green Deployment with DNS switch DB-1 DB-2 DB-N Availability Zone 1 Proxy Layer (Routing & Pod-picking) App-1 App-2 App-N Availability Zone 3 Proxy Layer (Routing & Pod-picking) App-1 App-2 App-N Availability Zone 2 Proxy Layer (Routing & Pod-picking) App-1 App-2 App-N Route 53 Persistence Layer - RDS Jenkins Web and Application Layer Availability Zone 1 Proxy Layer (Routing & Pod-picking) App-1 App-1 App-1 Availability Zone 3 Proxy Layer (Routing & Pod-picking) App-1 App-2 App-N Availability Zone 2 Proxy Layer (Routing & Pod-picking) App-1 App-2 App-N Web and Application Layer 1. Deploy 2. Validate 3. Switch DNS Entry
- 21. Reference links • DMS Limitations • Migration to RDS using Percona • RDS as Slave of External MySQL
- 22. Thank You