Data backup and Disaster recovery with AWS
- 1. Data backup and Disaster recovery with AWS Evgeny Rudinsky, SA by day * Blogger & AWSRus runner by night ** cloudacademy.com * SA - Solution Architect @ CloudBerry Lab ** AWSRus - AWS User Group, Russia (St. Petersburg)
- 2. Introductions Evgeny Rudinsky — Solution Architect @ CloudBerry Lab Twitter: @evgenyrudinsky AWS User Group: https://www.meetup.com/awsrus/ LinkedIn: https://www.linkedin.com/in/evgenyrudinsky/
- 3. AWS Storage services • EFS • Glacier • EBS • S3 • Plus something else.. * Great course to learn about these services — https://cloudacademy.com/learning-paths/aws-fundamentals-1/
- 4. Elastic File System • Highly available and durable, scalable file storage for use with EC2 instances • Great for file sharing across multiple EC2 running in multiple Availability zones • Mounts via NFSv4.1 protocol using standard OS mount commands • Can be mounted to on-premises datacenter servers when connected to VPC through Direct Connect * • Great performance, starts from $0.30/GB-month * AWS Direct Connect — https://aws.amazon.com/directconnect/
- 5. Glacier • Secure, durable and low cost storage • Great for long-term backup (think about tapes on-premises) • Data is stored in archives up to 40 terr. each. Archives are stored in containers called “Vaults” • Price starts from $0.004/GB-month • Data retrieval from few minutes to 3-5 hrs * * Expedited, Standard and Bulk — https://aws.amazon.com/glacier/faqs/#Data-retrievals
- 6. Elastic Block Storage • Highly-available, consistent and low-latency persistent block storage volume for EC2 • Designed for application workloads, automatically replicated within its AZ * • Quick scale-up and scale-down with SSD or HDD backed ** • EBS Provisioned IOPS SSD (io1) & EBS General Purpose SSD (gp2)* • Throughput Optimized HDD (st1) & Cold HDD (sc1) • Encryption at-rest and in-transit between EC2 and EBS • Protected by point-in-time snapshots kept on AWS S3 • Starts from $0.025/GB-month (SC1) & $0.10/GB-month (GP2) * Region & Availability zone — http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html ** EBS details — https://aws.amazon.com/ebs/details/
- 7. Simple Storage Service • One of the most popular service from AWS • Cost-effective object storage with “eleven nines” durability and 99,99% availability • Simple data transfer with Restful API and various SDK for integration simplicity • Each file is an object and is kept in a bucket • Ideal for backup and recovery • Starts from $0.004 per GB (Glacier storage class) * * S3 pricing — https://aws.amazon.com/s3/pricing/
- 8. S3 storage classes * Graph example with Standard Glacier retrieval (3-5 hrs), “Data Transfer OUT From Amazon S3 To” not counted ** More detailed picture — https://aws.amazon.com/s3/storage-classes/
- 9. Versioning Versioning is a means of keeping multiple variants of an object in the same bucket. You can use versioning to preserve, retrieve, and restore every version of every object stored in your Amazon S3 bucket. With versioning, you can easily recover from both unintended user actions and application failures. In one bucket, for example, you can have two objects with the same key, but different version IDs, such as photo.gif (version 111111) and photo.gif (version 121212). * http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html
- 10. Transfer acceleration • Designed for long-distance transfers • Tool to measure the efficency based on current location — http://s3-accelerate-speedtest.s3- accelerate.amazonaws.com/en/acc elerate-speed-comparsion.html • Consider based geo and AWS global infra map * Additional cost $$ 0.04 / 0.08 per GB (depends on region) — https://aws.amazon.com/s3/pricing/ ** Edge location — https://aws.amazon.com/cloudfront/details/ *** Global Infrastructure — https://aws.amazon.com/about-aws/global-infrastructure/
- 11. Multipart upload Essentially, the feature introduced in 2010, helps: • Handle large objects • Object uploaded in chunks in parallel threads • Large objects (up to 5 TB) can be uploaded • 5 Gb max chunk size with 1024 threads ~ 5TB • 5 MB to 5 TB in size for multipart upload * I borrowed this picture from Jeff Barr’s blog post — https://aws.amazon.com/blogs/aws/amazon-s3-multipart-upload/
- 12. Snowball • Helps to import / export to / from AWS • 50TB, 80TB, 100TB (edge) boxes, can be scaled out • Pricing starts from $200 (without days onsite and transfer out if export) Two major interfaces for upload: * • Snowball client • Snowball S3 adapter (multipart upload supported!!!) * References to the user guides — http://docs.aws.amazon.com/snowball/latest/ug/snowball-data-transfer.html
- 13. Storage Gateway Hybrid cloud workloads, — https://aws.amazon.com/storagegateway/
- 14. Backup and Recovery • AWS offers great integration methods via CLI and SDK. • APN contains endless number of third-party tools, especially storage partner solutions. * * https://aws.amazon.com/backup-recovery/partner-solutions/
- 15. AWS CLI • Available for all major platforms (Windows, macOS, Linux) • Manage AWS services from command line • Get started guide * * http://docs.aws.amazon.com/cli/latest/userguide/installing.html
- 16. Ah … how about access control? • Identity and Access Management (IAM) enables you to securely control access to AWS services and resources for your users • Users and Roles have policies (either built-in or custom) • Policy is “json-like” structure defining certain services and its parts to which entity may have / may not have access to • For today’s session we will be using Access and Secret key pair * Best practices around this topic — http://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html
- 17. File level backup • AWS account with IAM user, S3 bucket • Cron • AWS CLI • Bash #!/bin/sh aws s3 cp /path/to/my/files/* s3://my-bucket-goes-here/ You may want to have pre/post scripts for clean up your folders pre/after the process You configure cron accordingly (crontab -e and set your scheduler) * You may want to enable bucket versioning, what makes your backup more powerful! * Cron details — https://en.wikipedia.org/wiki/Cron
- 18. AMI * generator and launch EC2 • AMI description (json) • IAM user to access EC2 and S3 resources • AWS CLI • OVA file (or VMDK, VHD, or RAW) * Amazon Machine Image (for launching EC2) — http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html ** AWS Import VM — https://aws.amazon.com/ec2/vm-import/ #!/bin/sh aws ec2 import-image --description "New AMI" --disk-containers file://new_ami.json aws ec2 describe-import-image-tasks --import-task-ids aws ec2 ec2-import-instance ./WinSvr8-2-32-disk1.vmdk -f VMDK -t m1.small -a i386 -b myawsbucket
- 19. MS SQL restored to RDS? • RDS - Relational DB * • Supports the following Amazon Aurora, PostgreSQL, MySQL, MariaDB, Oracle, and Microsoft SQL Server • Native backup and restore support (think about .bak file) * RDP details — https://aws.amazon.com/rds/ EXEC msdb.dbo.rds_restore_database @restore_db_name='dbName', @s3_arn_to_restore_from='arn:aws:s3:::mybucket/MSSQL_file.bak';
- 20. Wrap up Looks like we have plenty of storage services from AWS, that can be used with really reach CLI and SDK. Leverage them or consider 3d party tool to delivery your backups and handle restore through any of the following solution: https://aws.amazon.com/backup-recovery/partner-solutions/