Training AWS: Module 4 - Storage in AWS
0 likes115 views
The document outlines an AWS training course conducted over 9 days. The course covers various AWS storage services including EBS, EFS, S3, and others. It provides detailed information on EBS volumes such as volume types, snapshots, encryption, and RAID configurations. It also describes EFS as a managed NFS file system that can be mounted on multiple EC2 instances across availability zones.
Training AWS: Module 4 - Storage in AWS
- 1. AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 1 Training Course Amazon Web Service Day Content Lab Day 1 System Operations on AWS Day 2 Computing on AWS X Day 3 Networking on AWS X Day 4 Storage in AWS (EBS, EFS, Instance Store) X Day 5 Elastic Load Balancing & Auto Scaling Group X Day 6 Storage S3 in AWS X Day 7 Route 53 X Day 8 RDS, Aurora, ElastiCache X Day 9 CloudWatch X Author: Bui Quang Lam Phone/Zalo: +84.0365.635.598 Mail: buiquanglam185@gmail.com
- 2. Goal: Understanding the format repository at Amazon. Store how, operate, share data. Distinguishing features and uses of format storage EBS AMI EC2 Instance Store EBS Volume Types Encryption EFS Lab 3: Create and configuring Storage AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 2 Module 4: Storage and Archiving in AWS
- 3. EBS AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 3 Module 4: Storage and Archiving in AWS
- 4. What’s an EBS Volume? • An EBS (Elastic Block Store) Volume is a network drive you can attach to your instances while they run • It allows your instance to persist data, even after their termination • They can only be mouted to one instance at a time • They are bound to a specific availability zone AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 4 Module 4: Storage and Archiving in AWS
- 5. What’s an EBS Volume? • Analogy: Thinks of them as a “network USB stick” • Free tier: 30GB of free EBS storage of type General Purpose (SSD) or Magnetic per month AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 5 Module 4: Storage and Archiving in AWS
- 6. EBS Volume • It’s a network drive (not a physical drive) • It uses the network to communicate the instance, which means there might be a bit of latency • It can be detached from an EC2 instance and attached to another one quickly • It’s locked to an Availability Zone (AZ) • An EBS Volume in us-east-1a cannot be attached to us-east-1b • To move a volume across, you first need to snapshot it AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 6 Module 4: Storage and Archiving in AWS
- 7. EBS Volume • Have a provisioned capacity (size in GBs, and IOPS) • You get billed for all the provisioned capacity • You can increase the capacity of the drive over time AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 7 Module 4: Storage and Archiving in AWS
- 8. EBS Volume AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 8 Module 4: Storage and Archiving in AWS
- 9. EBS – Delete on Termination attribute • Controls the EBS behavior when an EC2 instance terminates • By default, the root EBS volume is deleted (attribute enabled) • By default, any other attached EBS volume is not deleted (attribute disabled) • This can be controlled by the AWS console / AWS CLI • Use case: preserve root volume when instance is terminated AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 9 Module 4: Storage and Archiving in AWS
- 10. EBS – Delete on Termination attribute AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 10 Module 4: Storage and Archiving in AWS
- 11. EBS Snapshots • Make a backup (snapshot) of your EBS volume at a point in time • Not necessary to detach volume to do snapshot (but recommened) • Can copy snapshots across AZ or Region AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 11 Module 4: Storage and Archiving in AWS
- 12. EBS Snapshots AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 12 Module 4: Storage and Archiving in AWS
- 13. AMI Overview AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 13 Module 4: Storage and Archiving in AWS
- 14. AMI Overview • AMI = Amazon Machine Image (!= IAM) • AMI are a customization of an EC2 instance • Can add software, configuration, operation system, monitoring… • Faster boot / configuration time because all software is pre-packaged • AMI are built for a specific region (and can be copied across regions) • You can launch EC2 instances from • A public AMI: AWS provided • Your own AMI: you make and maintain them yourself • An AWS Marketplace AMI: an AMI someone else made (and potentially sells) AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 14 Module 4: Storage and Archiving in AWS
- 15. AMI Process (from an EC2 instance) • Start an EC2 instance and customize it • Stop the instance (for data integrity) • Build an AMI – this will also create EBS snapshots • Launch instances from other AMIs AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 15 Module 4: Storage and Archiving in AWS
- 16. EC2 Instance Store AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 16 Module 4: Storage and Archiving in AWS
- 17. EC2 Instance Store • EBS volumes are networking drives with good but “limited” performance • If need a high-performance hardware disk, use EC2 Instance Store • Better I/O performance • EC2 Instance Store lose data if stopped (ephemeral) • Good for buffer/cache/scratch data/temporary content • Risk of data loss if hardware fails • Backups and Replicaton are your responsibility AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 17 Module 4: Storage and Archiving in AWS
- 18. Local EC2 Instance Store AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 18 Module 4: Storage and Archiving in AWS
- 19. EBS Volume Types AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 19 Module 4: Storage and Archiving in AWS
- 20. EBS Volume Types • EBS Volumes come in 6 types • gp2 / gp3 (SSD): General purpose SSD volume that balances price and performance for a wide variety of workloads • io1 / io2 (SSD): Highest-performance SSD volume for mission- critical low-latency or high-throughput workloads • st 1 (HDD): Low cost HDD volume designed for frequently accessed, throughput-intensive workloads • sc 1 (HDD): Loweset cost HDD volume designed for less frequently accessed workloads AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 20 Module 4: Storage and Archiving in AWS
- 21. EBS Volume Types Use cases • EBS Volumes are characterized in Size | Throughput | IOPS (I/O Ops Per Sec) • When in doubt always consult the AWS documentation – it’s good • Only gp2/gp3 and io 1/io 2 can be used as boot volumes AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 21 Module 4: Storage and Archiving in AWS
- 22. EBS Volume Types Use cases General Purpose SSD • Cost effective storage, low-latency • System boots volumes, Virtual desktops, Development and test enviroments • 1 Gib – 16 TiB AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 22 Module 4: Storage and Archiving in AWS
- 23. EBS Volume Types Use cases General Purpose SSD • Gp3: • Baseline of 3,000 IOPS and throughput of 125 MiB/s • Can increase IOPS up to 16,000 and throughput up to 1000 MiB/s independently • Gp2: • Smaill gp2 volumes can burst IOPS to 3000 • Size of the volume and IOPS are linked, max IOPS is 16000 • 3 IOPS per GB, means at 5,334 GB we are at the max IOPS AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 23 Module 4: Storage and Archiving in AWS
- 24. EBS Volume Types Use cases Provisioned IOPS SSD • Critical business applications with sustained IOPS performance • Or applications that need more than 16,000 IOPS • Great for databases workloads (sensitive to storage perf and consistency) AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 24 Module 4: Storage and Archiving in AWS
- 25. EBS Volume Types Use cases Provisioned IOPS SSD • io 1 /io2 (4 GiB – 16 TiB) • Max PIOPS: 64,000 for Nitro EC2 instances & 32,000 for other • Can increase PIOPS independently from storage size • io2 have more durability and more IOPS per GiB (at the same price as io 1) • io2 Block Express (4 Gib – 64 TiB) • Sub-millisecond latency • Max PIOPS: 256,000 with an IOPS:Gib ratio of 1,000:1 • Supports EBS Multi-attach AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 25 Module 4: Storage and Archiving in AWS
- 26. EBS Volume Types Use cases Provisioned IOPS SSD • io 1 /io2 (4 GiB – 16 TiB) • Max PIOPS: 64,000 for Nitro EC2 instances & 32,000 for other • Can increase PIOPS independently from storage size • io2 have more durability and more IOPS per GiB (at the same price as io 1) • io2 Block Express (4 Gib – 64 TiB) • Sub-millisecond latency • Max PIOPS: 256,000 with an IOPS:Gib ratio of 1,000:1 • Supports EBS Multi-attach AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 26 Module 4: Storage and Archiving in AWS
- 27. EBS Volume Types Use cases Hard Disk Drives (HDD) • Cannot be a boot volume • 125 Mib to 16 Tib • Throughput Optimized HDD (st 1) • Big Data, Data Warehouses, Log Processing • Max throughput 500 MiB/s – max IOPS 500 • Cold HDD (sc 1) • For data that is infrequently accessed • Scenarios where lowest cost is important • Max throughput 250 MiB/s – max IOPS 250 AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 27 Module 4: Storage and Archiving in AWS
- 28. EBS Volume Types Summary AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 28 Module 4: Storage and Archiving in AWS
- 29. EBS Multi-Attach – io1/io2 family • Attach the same EBS volume to multiple EC2 instances in the same AZ • Each instance has full read & write permissions to the volume AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 29 Module 4: Storage and Archiving in AWS
- 30. EBS Multi-Attach – io1/io2 family • Use case: • Archieve higher application availability in clustered Linux applications (ex: Teradata) • Applications must manage concurrent write operations AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 30 Module 4: Storage and Archiving in AWS
- 31. EBS Multi-Attach – io1/io2 family AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 31 Module 4: Storage and Archiving in AWS
- 32. Encryption AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 32 Module 4: Storage and Archiving in AWS
- 33. EBS Encryption • When you create an encrypted EBS volume, you get the following: • All the data in flight moving between the instance and the volume is encrypted • All snapshots are encrypted • All volumes created from the snapshot AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 33 Module 4: Storage and Archiving in AWS
- 34. EBS Encryption • Encryption and decryption are handled transparently (have nothing to do) • Encryption has a minial impact on latency • EBS Encryption leverages keys from KMS (AES-256) • Copying an unencrypted snapshot allow encryption • Snapshots of encrypted volumes are encrypted AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 34 Module 4: Storage and Archiving in AWS
- 35. Encryption: encrypt an unencrypted EBS volume • Create an EBS snapshot of the volume • Encrypt the EBS snapshot (using copy) • Create new EBS volume from the snapshot (the volume will also be encrypted) • Now can attach the encrypted volume to the original instance AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 35 Module 4: Storage and Archiving in AWS
- 36. RAID AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 36 Module 4: Storage and Archiving in AWS
- 37. EBS RAID Options • EBS is already redundant storage (replicated within an AZ) • But if want to increase IOPS to 100 000 IOPS? • What if you want to mirror your EBS volumes? • You would mount volumes in parallel in RAID settings! • RAID is possible as long as your OS supports it AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 37 Module 4: Storage and Archiving in AWS
- 38. EBS RAID Options • Some RAID optiones are: • RAID 0 • RAID 1 • RAID 5 (not recommended for EBS ?) • RAID 6 (not recommended for EBS ?) AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 38 Module 4: Storage and Archiving in AWS
- 39. RAID 0 (increase performance) • Combining 2 or more volumes and getting the total disk space and I/O • But one disk fails , all the data is failed AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 39 Module 4: Storage and Archiving in AWS
- 40. RAID 0 (increase performance) • Use case • An application needs a lot of IOPS and doesn’t need fault- tolerance • A database that has replication already built-in • Using this, we can have a big disk with a lof of IOPS AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 40 Module 4: Storage and Archiving in AWS
- 41. RAID 1 (increase fault tolerance) • RAID 1 = Mirror a volume to another • If one disk fails, our logical volume is still working • We have to send the data to two EBS volume at the same time (2x network) AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 41 Module 4: Storage and Archiving in AWS
- 42. RAID 1 (increase fault tolerance) • Use case: • Application that need increase volume fault tolerance • Application where you need to service disks AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 42 Module 4: Storage and Archiving in AWS
- 43. EFS AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 43 Module 4: Storage and Archiving in AWS
- 44. EFS – Elastic File System • Managed NFS (network file system) that can be mounted on many EC2 • EFS works with EC2 instances in multi-AZ • Highly available, scalable, expensive (3x gp2), par per use AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 44 Module 4: Storage and Archiving in AWS
- 45. EFS – Elastic File System AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 45 Module 4: Storage and Archiving in AWS
- 46. EFS – Performance • EFS Scale • 10 GB+ /S throughput • Grow to Petabyte-scale network file system, automatically • Performance mode (set at EFS creation time) • General purpose (default): latency-sensitive use cases (web server, CMS, etc…) • Max I/O – higher latency, throughput, highly parallel (big data, media processing) AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 46 Module 4: Storage and Archiving in AWS
- 47. EBS vs EFS – Elastic Block Storage • EBS volumes… • Can be attached to only one instance at a time • Are locked at the Availability Zone (AZ) level • Gp2: IO increases if the disk size increases • Io 1: can increase IO independently AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 47 Module 4: Storage and Archiving in AWS
- 48. EBS vs EFS – Elastic Block Storage • To migrate EBS volumes across AZ… • Take a snapshot • Restore the snapshot to another AZ • EBS backups use IO and you shouldn’t run them while your application is handling a lot of traffic • Root EBS Volumes of instances get terminated by default if the EC2 instanc gets terminated (can disable that) AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 48 Module 4: Storage and Archiving in AWS
- 49. EBS vs EFS – Elastic Block Storage • EBS volumes AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 49 Module 4: Storage and Archiving in AWS
- 50. EBS vs EFS – Elastic File System • Mouting 100s of instances across AZ • EFS share websites files (WordPress) • Only for Linux Instance (POSIX) • EFS has a higher price point than EBS • Can leverage EFS-IA for cost savings AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 50 Module 4: Storage and Archiving in AWS
- 51. Remember: EFS vs EBS vs Instance Store!! AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 51 Module 4: Storage and Archiving in AWS
- 52. Thank you!!! AWS COURSE DOCUMENT- BUI QUANG LAM - TRAINING 52 Module 4: Storage and Archiving in AWS