20160503 Amazed by AWS | Tips about Performance on AWS

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
김일호, SolutionsArchitect
03-May-2016
Tips about performance on AWS
Amazed by AWS

InstancesAPIs
Networking
EC2
EC2
Purchase Options
Amazon Elastic Compute Cloud is Big

Host Server
Hypervisor
Guest 1 Guest 2 Guest n
Amazon EC2 Instances

2006 2008 2010 2012 2014 2016
m1.small
m1.large
m1.xlarge
c1.medium
c1.xlarge
m2.xlarge
m2.4xlarge
m2.2xlarge
cc1.4xlarge
t1.micro
cg1.4xlarge
cc2.8xlarge
m1.medium
hi1.4xlarge
m3.xlarge
m3.2xlarge
hs1.8xlarge
cr1.8xlarge
c3.large
c3.xlarge
c3.2xlarge
c3.4xlarge
c3.8xlarge
g2.2xlarge
i2.xlarge
i2.2xlarge
i2.4xlarge
i2.4xlarge
m3.medium
m3.large
r3.large
r3.xlarge
r3.2xlarge
r3.4xlarge
r3.8xlarge
t2.micro
t2.small
t2.med
c4.large
c4.xlarge
c4.2xlarge
c4.4xlarge
c4.8xlarge
d2.xlarge
d2.2xlarge
d2.4xlarge
d2.8xlarge
g2.8xlarge
t2.large
m4.large
m4.xlarge
m4.2xlarge
m4.4xlarge
m4.10xlarge
Amazon EC2 Instances History

What to Expect from the Session
• Defining system performance and how it is
characterized for different workloads
• How Amazon EC2 instances deliver performance
while providing flexibility and agility
• How to make the most of your EC2 instance experience
through the lens of several instance types

• Servers are hired to do jobs
• Performance is measured differently depending on the job
Hiring a Server
?

• What performance means
depend on your perspective:
– Response time
– Throughput
– Consistency
Defining Performance: Perspective Matters
Application
System libraries
System calls
Kernel
Devices
Workload

Simple Performance Model for Single Thread
• Using CPU: executing (in user mode)
• Not using CPU: waiting for turn on CPU, waiting for disk or
network I/O, thread locks, memory paging, or for more work.

Performance Factors
Resource Performance factors Key indicators
CPU Sockets, number of cores, clock
frequency, bursting capability
CPU utilization, run queue length
Memory Memory capacity Free memory, anonymous paging,
thread swapping
Network
interface
Max bandwidth, packet rate Receive throughput, transmit throughput
over max bandwidth
Disks Input / output operations per
second, throughput
Wait queue length, device utilization,
device errors

Resource Utilization
• For given performance, how efficiently are resources being used
• Something at 100% utilization can’t accept any more work
• Low utilization can indicate more resource is being purchased
than needed

Example: Web Application
• MediaWiki installed on Apache with 140 pages of content
• Load increased in intervals over time

• Memory stats

• Disk stats

• Network stats

• CPU stats

• Picking an instance is tantamount to resource performance tuning
• Give back instances as easily as you can acquire new ones
• Find an ideal instance type and workload combination
Instance Selection = Performance Tuning

Delivering Compute Performance with

CPU Instructions and Protection Levels
• CPU has at least two protection levels: ring0 and ring1
• Privileged instructions can’t be executed in user mode to protect
system. Applications leverage system calls to the kernel.
Kernel
Application

Example: Web application system calls

X86 CPU Virtualization: Prior to Intel VT-x
• Binary translation for privileged instructions
• Para-virtualization (PV)
• PV requires going through the VMM, adding latency
• Applications that are system call bound are most affected
VMM
Application
Kernel
PV

X86 CPU Virtualization: After Intel VT-x
• Hardware assisted virtualization (HVM)
• PV-HVM uses PV drivers opportunistically for operations that are
slow emulated:
• e.g. network and block I/O
Kernel
Application
VMM
PV-HVM

Time Keeping Explained
• Time keeping in an instance is deceptively hard
• gettimeofday(), clock_gettime(), QueryPerformanceCounter()
• The TSC
• CPU counter, accessible from userspace
• Requires calibration, vDSO
• Invariant on Sandy Bridge+ processors
• Xen pvclock; does not support vDSO
• On current generation instances, use TSC as clocksource

Tip: Use TSC as clocksource
tsc
source=tsc

CPU Performance and Scheduling
• Hypervisor ensures every guest receives CPU time
• Fixed allocation
• Uncapped vs. capped
• Variable allocation
• Different schedulers can be used depending on the goal
• Fairness
• Response time / deadline
• Shares

Review: C4 Instances
Custom Intel E5-2666 v3 at 2.9 GHz
P-state and C-state controls
Model vCPU Memory (GiB) EBS (Mbps)
c4.large 2 3.75 500
c4.xlarge 4 7.5 750
c4.2xlarge 8 15 1,000
c4.4xlarge 16 30 2,000
c4.8xlarge 36 60 4,000

What’s new in C4: P-state and C-state control
• By entering deeper idle states, non-idle cores can achieve
up to 300MHz higher clock frequencies
• But… deeper idle states require more time to exit, may not
be appropriate for latency sensitive workloads

Tip: P-state control for AVX2
• If an application makes heavy use of AVX2 on all cores, the
processor may attempt to draw more power than it should
• Processor will transparently reduce frequency
• Frequent changes of CPU frequency can slow an application

Review: T2 Instances
• Lowest cost EC2 Instance at $0.013 per hour
• Burstable performance
• Fixed allocation enforced with CPU Credits
Model vCPU CPU Credits
/ Hour
Memory
(GiB)
Storage
t2.micro 1 6 1 EBS Only
t2.small 1 12 2 EBS Only
t2.medium 2 24 4 EBS Only
t2.large 2 36 8 EBS Only

How Credits Work
• A CPU Credit provides the
performance of a full CPU core for
one minute
• An instance earns CPU credits at
a steady rate
• An instance consumes credits
when active
• Credits expire (leak) after 24 hours
Baseline Rate
Credit
Balance
Burst
Rate

Tip: Monitor CPU credit balance

Monitoring CPU Performance in Guest
• Indicators that work is being done
• User time
• System time (kernel mode)
• Wait I/O, threads blocked on disk I/O
• Else, Idle
• What happens if OS is scheduled off the CPU?

Tip: How to interpret Steal Time
• Fixed CPU allocations of CPU can be offered through
CPU caps
• Steal time happens when CPU cap is enforced
• Leverage CloudWatch metrics

Delivering I/O Performance with

I/O and Devices Virtualization
• Scheduling I/O requests between virtual devices and
shared physical hardware
• Split driver model
• Intel VT-d
• Direct pass through and IOMMU for dedicated devices
• Enhanced Networking

Hardware
Split Driver Model
Driver Domain Guest Domain Guest Domain
VMM
Frontend
driver
Frontend
driver
Backend
driver
Device
Driver
Physical
CPU
Physical
Memory
Network
Device
Virtual CPU
Virtual
Memory
CPU
Scheduling

Split Driver Model
• Each virtual device has two main components
• Communication ring buffer
• An event channel signaling activity in the ring buffer
• Data is transferred through shared pages
• Shared pages requires inter domain permissions, or granting

Review: I2 Instances
16 vCPU: 3.2 TB SSD; 32 vCPU: 6.4 TB SSD
365K random read IOPS for 32 vCPU instance
Model vCPU Memory
(GiB)
Storage Read IOPS Write IOPS
i2.xlarge 4 30.5 1 x 800 SSD 35,000 35,000
i2.2xlarge 8 61 2 x 800 SSD 75,000 75,000
i2.4xlarge 16 122 4 x 800 SSD 175,000 155,000
i2.8xlarge 32 244 8 x 800 SSD 365,000 315,000

Granting in pre-3.8.0 Kernels
• Requires “grant mapping” prior to 3.8.0
• Grant mappings are expensive operations due to TLB flushes
read(fd, buffer,…)

Granting in 3.8.0+ Kernels, Persistent and Indirect
• Grant mappings are setup in a pool once
• Data is copied in and out of the grant pool
read(fd, buffer…)
Copy to
and from
grant pool

Tip: Use 3.8+ kernel
• Amazon Linux 13.09 or later
• Ubuntu 14.04 or later
• RHEL7 or later
• Etc.

Event Handling
• Guest vCPUs are interrupted to process events.
• Pre-2.6.36 kernels: notifications went to a single virtual
hardware interrupt
• Post-2.6.36 kernels: allow instance to tell hypervisor to deliver
notification to a specific vCPU for balancing
• Check "dmesg" for the following text: "Xen HVM callback vector for
event delivery is enabled“
• Also, check version of irqbalance is 1.0.7 or higher

Hardware
Split Driver Model: Networking
VMM
Frontend
driver
Frontend
driver
Backend
driver
Device
Driver
Physical
CPU
Physical
Memory
Network
Device
Virtual CPU
Virtual
Memory
CPU
Scheduling
Sockets
Application

Device Pass Through: Enhanced Networking
• SR-IOV eliminates need for driver domain
• Physical network device exposes virtual function to
instance
• Requires a specialized driver, which means:
• Your instance OS needs to know about it
• EC2 needs to be told your instance can use it

Hardware
After Enhanced Networking
VMM
Frontend
driver
NIC
Driver
Backend
driver
Device
Driver
Physical
CPU
Physical
Memory
SR-IOV Network
Device
Virtual CPU
Virtual
Memory
CPU
Scheduling
Sockets
Application

Tip: Use Enhanced Networking
• Highest packets-per-second
• Lowest variance in latency
• Instance OS must support it
• Look for SR-IOV property of instance or image

How to build enhanced network
driver on Linux

Let’s start to create AMI for
Enhanced networking enabled

CentOS 6.5 with c4.8xlarge
Let’s start with the AMI officially provided CentOS, should
be clean.
CentOS is provided in awsmarketplace.

Only missing c4.8xlarge
L
Anyway, let’s go to AMI
search

Find and Select AMI
CentOS 6 x86_64 (2014_09_29) EBS HVM-74e73035-
3435-48d6-88e0-89cc02ad83ee-ami-a8a117c0.2
ami-c2a818aa (IAD)
This is the CentOS AMI at awsmarketplace CentOS. Nice
?!?!

Check the requirements of
Enabling Enhanced Networking on Linux
• C3, C4, D2, I2, M4 and R3
• HVM AMI with Linux kernel above V.2.6.32
• Launch the instance in VPC
• A network driver to support enhanced networking on
Linux.

Check kernel version and network driver
[root@ip-192-168-1-171 ~]# cat /etc/redhat-release
CentOS release 6.5 (Final)
[root@ip-192-168-1-171 ~]# uname -na
Linux ip-192-168-1-171 2.6.32-431.29.2.el6.x86_64 #1 SMP Tue Sep 9 21:36:05 UTC 2014 x86_64 x86_64
x86_64 GNU/Linux
[root@ip-192-168-1-171 ~]# modinfo ixgbevf
modinfo ixgbevf
filename: /lib/modules/2.6.32-431.29.2.el6.x86_64/kernel/drivers/net/ixgbevf/ixgbevf.ko
version: 2.7.12-k
license: GPL
description: Intel(R) 82599 Virtual Function Driver
author: Intel Corporation, <linux.nics@intel.com>
srcversion: E75203124BB105EC871944F
alias: pci:v00008086d00001515sv*sd*bc*sc*i*
alias: pci:v00008086d000010EDsv*sd*bc*sc*i*
depends:
vermagic: 2.6.32-431.29.2.el6.x86_64 SMP mod_unload modversions
parm: debug:Debug level (0=none,...,16=all) (int)
Launch c4.large instance and login

Let’s update all including kernel
root@ip-192-168-1-171 ~]# yum update -y
Loaded plugins: fastestmirror, presto
Loading mirror speeds from cached hostfile
* base: mirrors.mit.edu
* extras: linux.cc.lehigh.edu
* updates: mirrors.lga7.us.voxel.net
Setting up Update Process
Resolving Dependencies
--> Running transaction check
---> Package audit.x86_64 0:2.2-4.el6_5 will be updated
---> Package audit.x86_64 0:2.3.7-5.el6 will be an update
---> Package audit-libs.x86_64 0:2.2-4.el6_5 will be updated
---> Package audit-libs.x86_64 0:2.3.7-5.el6 will be an update
---> Package authconfig.x86_64 0:6.1.12-13.el6 will be updated
---> Package authconfig.x86_64 0:6.1.12-19.el6 will be an update
---> Package bash.x86_64 0:4.1.2-15.el6_5.2 will be updated
---> Package bash.x86_64 0:4.1.2-29.el6 will be an update
---> Package binutils.x86_64 0:2.20.51.0.2-5.36.el6 will be updated
---> Package binutils.x86_64 0:2.20.51.0.2-5.42.el6 will be an update
---> Package ca-certificates.noarch 0:2014.1.98-65.0.el6_5 will be updated
---> Package ca-certificates.noarch 0:2014.1.98-65.1.el6 will be an update
---> Package centos-release.x86_64 0:6-5.el6.centos.11.2 will be updated
---> Package centos-release.x86_64 0:6-6.el6.centos.12.2 will be an update
---> Package coreutils.x86_64 0:8.4-31.el6_5.2 will be updated
---> Package coreutils.x86_64 0:8.4-37.el6 will be an update
---> Package coreutils-libs.x86_64 0:8.4-31.el6_5.2 will be updated
………………………

Reboot and check the updated
[root@ip-192-168-1-171 ~]# cat /etc/redhat-release
CentOS release 6.6 (Final)
[root@ip-192-168-1-171 ~]# uname -na
Linux ip-192-168-1-171 2.6.32-504.12.2.el6.x86_64 #1 SMP Wed Mar 11 22:03:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@ip-192-168-1-171 ~]# modinfo ixgbevf
version: 2.12.1-k
license: GPL
description: Intel(R) 82599 Virtual Function Driver
srcversion: 8797AC845BB302315230490
depends:
parm: debug:Debug level (0=none,...,16=all) (int)
Upgraded network driver installed
version: 2.7.12-k -> 2.12.1-k

Enable SR-IOV
Install AWS CLI or EC2 CLI tools
Not supported at AWS Console yet L
1. Stop the instance
2. Enable SR-IOV of the instance with CLI
3. Check the status
4. Start the instance a82066443ffe:~ ilho$ aws ec2 modify-instance-attribute --instance-id i-681280bf
--sriov-net-support simple
a82066443ffe:~ ilho$ aws ec2 describe-instance-attribute --instance-id i-681280bf
--attribute sriovNetSupport
{
"InstanceId": "i-681280bf",
"SriovNetSupport": {
"Value": "simple"
}
}

A problem with more than 32 vCPUs on Linux
CentOS 6.x does not support more than 32 vCPUs in
kernel.
It can not boot when you launch c4.8xlarge(36 vCPUs) L
• d2.8xlarge and m4.10xlarge? -> L
A solution is to add an option to kernel boot parameter

Add maxcpus option to kernel boot parameter
$ vi /boot/grub/menu.lst
Add maxcpus=32
[root@ip-10-10-10-242 ~]# cat /boot/grub/menu.lst
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You do not have a /boot partition. This means that
# all kernel and initrd paths are relative to /, eg.
# root (hd0,0)
# kernel /boot/vmlinuz-version ro root=/dev/vda1
# initrd /boot/initrd-[generic-]version.img
#boot=/dev/vda
default=0
timeout=1
serial --unit=0 --speed=115200
terminal --timeout=1 serial console
title CentOS (2.6.32-431.29.2.el6.x86_64)
root (hd0,0)
kernel /boot/vmlinuz-2.6.32-431.29.2.el6.x86_64 maxcpus=32 ro root=UUID=dcb1645e-05a6-4311-
8bce-a9c12bec5801 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD console=ttyS0,115200
crashkernel=auto SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM
initrd /boot/initramfs-2.6.32-431.29.2.el6.x86_64.img

Now let’s launch c4.8xlarge instance
I think it’s ready to go.
Reboot
Launch a c4.8xlarge instance
It can not launch.

Why?
AWS Marketplace AMI does not support to launch an
instance type not in the list.

Change the base AMI
Find CentOS 6.5 community version
AMI : CentOS-6.5-base-20150305
(ami-0e80db66)
Repeat three steps
1. $ sudo yum update –y
2. Enable SR-IOV
3. Add maxcpus=32

Network driver version should be checked
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/en
hanced-networking.html
To enable enhanced networking on your instance, you
must ensure that its kernel has the ixgbevf module installed
and that you set the sriovNetSupport attribute for the
instance. For the best performance, we recommend that
the ixgbevf module is version 2.14.2 or higher.

Build and Install the network driver #2
[ec2-user@ip-192-168-1-50 src]$ make;sudo make install
make -C /lib/modules/2.6.32-504.12.2.el6.x86_64/build SUBDIRS=/home/ec2-user/ixgbevf-2.16.1/src
modules
make[1]: Entering directory `/usr/src/kernels/2.6.32-504.12.2.el6.x86_64'
CC [M] /home/ec2-user/ixgbevf-2.16.1/src/ixgbevf_main.o
CC [M] /home/ec2-user/ixgbevf-2.16.1/src/ixgbevf_param.o
CC [M] /home/ec2-user/ixgbevf-2.16.1/src/ixgbevf_ethtool.o
CC [M] /home/ec2-user/ixgbevf-2.16.1/src/kcompat.o
CC [M] /home/ec2-user/ixgbevf-2.16.1/src/ixgbe_vf.o
CC [M] /home/ec2-user/ixgbevf-2.16.1/src/ixgbe_mbx.o
LD [M] /home/ec2-user/ixgbevf-2.16.1/src/ixgbevf.o
Building modules, stage 2.
MODPOST 1 modules
CC /home/ec2-user/ixgbevf-2.16.1/src/ixgbevf.mod.o
LD [M] /home/ec2-user/ixgbevf-2.16.1/src/ixgbevf.ko.unsigned
NO SIGN [M] /home/ec2-user/ixgbevf-2.16.1/src/ixgbevf.ko
make[1]: Leaving directory `/usr/src/kernels/2.6.32-504.12.2.el6.x86_64'
make -C /lib/modules/2.6.32-504.12.2.el6.x86_64/build SUBDIRS=/home/ec2-user/ixgbevf-2.16.1/src
modules
make[1]: Entering directory `/usr/src/kernels/2.6.32-504.12.2.el6.x86_64'
Building modules, stage 2.
MODPOST 1 modules
make[1]: Leaving directory `/usr/src/kernels/2.6.32-504.12.2.el6.x86_64'
gzip -c ../ixgbevf.7 > ixgbevf.7.gz
# remove all old versions of the driver
find /lib/modules/2.6.32-504.12.2.el6.x86_64 -name ixgbevf.ko -exec rm -f {} ; || true
find /lib/modules/2.6.32-504.12.2.el6.x86_64 -name ixgbevf.ko.gz -exec rm -f {} ; || true
install -D -m 644 ixgbevf.ko /lib/modules/2.6.32-
504.12.2.el6.x86_64/kernel/drivers/net/ixgbevf/ixgbevf.ko
/sbin/depmod -a 2.6.32-504.12.2.el6.x86_64 || true
install -D -m 644 ixgbevf.7.gz /usr/share/man/man7/ixgbevf.7.gz
man -c -P'cat > /dev/null' ixgbevf || true
http://sourceforge.net/projects/e1000/files/ixgbevf%20stable/

Check the new driver installed
[ec2-user@ip-192-168-1-50 src]$ modinfo ixgbevf
version: 2.16.1
license: GPL
description: Intel(R) 10 Gigabit Virtual Function Network Driver
srcversion: 3B690FE23A02C25EF74012F
depends:
parm: InterruptThrottleRate:Maximum interrupts per second, per vector, (956-488281,
0=off, 1=dynamic), default 1 (array of int)
reboot
[ec2-user@ip-192-168-1-50 ~]$ ethtool -i eth0
driver: ixgbevf
version: 2.16.1
firmware-version: N/A
bus-info: 0000:00:03.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

Share AMI to the users
Sharing the customized AMI with your team ~
Let’s get some sleep… J

Reply back with 1/2 (OS unreachable)

20160503 Amazed by AWS | Tips about Performance on AWS

Root cause found
the 70-persistent-net.rules file /etc/udev/rules.d/ having an
entry with the MAC address of the original instance the AMI
was taken from. When the image is taken, the MAC from
the original instance persists, which tells the OS after the
new instance is deployed that the now-nonexistent MAC of
the original instance should be eth0. Since the OS cannot
find the device with the original MAC address, eth0 fails to
be identified and isn't brought up.

A solution to avoid this caveat
Before creating AMI, You must remove
/etc/udev/rules.d/70-persistent-net.rules file
Create AMI
Share AMI
Now launch c4 instance successfully with the latest driver.

A small contribution
The note was added in public document “enhanced networking”
section.

Lesson learned
Just use CentOS 7 or Amazon Linux. J

What is EBS?
• Network block storage as a service
• EBS volumes attach to any Amazon EC2 instance in the
same Availability Zone
• Designed for five nines of availability
• 2 million volumes created every day

EBS volume types
Magnetic General purpose (SSD) Provisioned IOPS
(SSD)

EBS volume types
IOPS: Typically 100, best effort
Throughput: 40-90 MB/s
Latency: Read 10-40ms, Write 2-10ms
Best for infrequently accessed data
Magnetic

EBS volume types
IOPS Baseline: 100-10,000 (3 / GiB)
IOPS Burst: 30 minutes @ 3,000
Throughput: Up to 160 MB/s
Latency: Single-digit ms
Performance consistency: 99%
Most workloadsGeneral purpose (SSD)

EBS volume types
IOPS: 100-20,000 (customer provisioned)
Throughput: Up to 320 MB/s
Latency: Single-digit ms
Performance consistency: 99.9%
Mission Critical workloadsProvisioned IOPS (SSD)

Queuing theory
Little’s law is the foundation for performance tuning theory
• Mathematically proven by John Little in 1961
𝑾 =
𝑳
𝑨
W = Wait time = average wait time per request
L = Queue length = average number of requests waiting
A = Arrival rate = the rate of requests arriving
EBS performance is related to this law

Performance optimization is measured by:
IOPS: Read/write I/O rate (IOPS)
Latency: Time between I/O submission
and completion (ms)
Throughput: Read/write transfer rate
(MB/s); throughput = IOPS X I/O size

Key components to performance
EC2 instance
I/O
EBS
Network link

A day in the life: I/O
All I/O must pass through I/O domain
Requires “grant mapping” prior to 3.8.0
Grant mappings are expensive operations due to TLB flushes
EBS
Grant
mappingread(fd, buffer,
BLOCK_SIZE)
I/O domainInstance

A day in the life: I/O (continued)
Responses
Requests
Instance I/O domain
READ
8KB @ 1234
Request queue is a single memory page
Each I/O request has 11 grant references (4KiB/reference)
Maximum data in queue = 1408 KiB

3.8.0+ Kernels – Persistent grants
Grant mappings are setup in a pool once
Data is copied in and out of the grant pool
Copying is significantly faster than remapping
EBS
Grant
poolread(fd, buffer,
BLOCK_SIZE)
I/O domainInstance

3.8.0+ Kernels – Indirect grants
Responses
Requests
Instance
READ
8KB @ 1234
I/O domain
Each I/O request has grant references that contain grant references
Maximum data in queue = 4096 KiB (default)

Instance I/O: Before 3.8.0
0 1 2 3 4 5 30 31
128KiB
44KiB 44KiB 40KiB

Instance I/O: Linux 3.8.0+
0 1 2 3 4 5 30 31
128KiB

Tip: Use 3.8+ kernel
Amazon Linux 2013.09 or later
Ubuntu 14.04 or later
RHEL7 or later
Etc.

Queue depth
An I/O operation
EBS
After it’s gone, it’s gone
EC2
Queue depth is the pending I/O for a volume

Workload/
software
Typical block
size
Random/
Seq?
Max EBS @ 500
MB/s instances
Max EBS @
1 GB/s instances
Max EBS @ 10 GB/s
instances
Oracle DB Configurable:2 KB
–16 KB
Default 8 KB
random ~7,800 IOPS ~15,600 IOPS ~48,000 IOPS
Microsoft SQL
Server
8 KB w/ 64 KB
extents
random ~7,800 IOPS ~15,600 IOPS ~48,000 IOPS
MySQL 16 KB random ~4,000 IOPS ~7,800 IOPS ~48,000 IOPS
PostgreSQL 8 KB random ~7,800 IOPS ~15,600 IOPS ~48,000 IOPS
MongoDB 4 KB sequential ~15,600 IOPS ~31,000 IOPS ~48,000 IOPS
Apache
Cassandra
4 KB random ~15,600 IOPS ~31,000 IOPS ~48,000 IOPS
GlusterFS 128 KB sequential ~500 IOPS ~1,000 IOPS ~6,000 IOPS
Cheat sheet: Sample storage workloads on AWS

Example workload
Transaction (OLTP)
Examples: eCommerce website, metadata storage
Benchmark: MySQL + sysbench

Tip: Workload
Where possible, use real
production workloads for
performance testing

Baseline configuration
Availability Zone: US West (Oregon)
Instance type: m2.4xlarge
vCPU: 8
Memory: 68.4GiB
EBS-optimized
Data volume: 500GiB EBS magnetic
OS: Amazon Linux 2015.03.1

Optimization: Increase parallelism
MySQL threads
Transactions(n)
Baseline
2 n

Tip: Parallelism
Increase parallelism of your
system

m2.4xlarge
CPU: Intel Xeon
vCPU: 8
Memory: 68.4 GiB
Price: $0.98/hour*
Instance selection
r3.2xlarge
CPU: Intel Xeon E5-2670 v2
vCPU: 8
Memory: 61 GiB
Enhanced networking
Price: $0.70/hour*
* All pricing from US West (Oregon)

EBS optimized instances
• Most instancefamilies support the EBS-optimizedflag
• EBS-optimized instances now support up to 4 Gb/s
• Drive 32,000 16K IOPS or 500 MB/s
• Available by defaulton newer instance types
• EC2 *.8xlargeinstances support 10 Gb/s network
• Max IOPS per node supported is ~48,000 IOPS @ 16K I/O

Tip: Use EBS-optimized instances
Use EBS-optimized instances
for consistent EBS performance

Updated configuration: Instance type
Instance type: r3.2xlarge
vCPU: 8
Memory: 61 GiB
EBS-optimized
EBS volume: 500GiB magnetic

25%
Optimization: Current generation instances
MySQL threads
Transactions(n)
Baseline
r3.2xlarge
2 n

Tip: Instance selection
Use the right instance family for
your workload
Use current generation instances

Volume selection
EBS magnetic
Latency:
Read: 10-40ms
Write: 2-10ms
SSD backed
Latency:
Read/Write: Single-digit ms

File systems
Use a modern, journaled filesystem
ext4, xfs, etc.
Ensure partitions are aligned on 4KiB boundaries

Volume initialization
Newly created volumes
• Just attach, mount, and go!
• Pre-warming is no longer recommended
Volumes restored from snapshots
• You can use your volume right away
• Accelerate data loading by reading

Updated configuration: EBS volumes
Instance type: r3.2xlarge
vCPU: 8
Memory: 61 GiB
EBS-optimized
Boot volume: 8 GiB – EBS general purpose
Data volume: 500 GiB – EBS general purpose

Optimization: Volume selection
Transactions(n)
19% 50%
MySQL threads
Baseline
r3.2xlarge
r3.2xlarge gp2
2 n

Tip: Volume selection
Use SSD backed volumes when
performance matters

EBS IOPS vs. Throughput
20,000 IOPS
PIOPS volume
20,000 IOPS
320 MB/s
throughput
You can achieve 20,000 IOPS when
driving smaller I/O operations
You can achieve up to 320 MB/s
when driving larger I/O operations

EBS IOPS vs. Throughput
8,000 IOPS
PIOPS volume
8,000 IOPS
320 MB/s
throughput
8,000 x 8 KB = 64 MB/s
8,000 x 16 KB = 128 MB/s
8,000 x 32 KB = 256 MB/s
16,000 x 8 KB = 128 MB/s
8,000 x 64 KB=512 MB/s
5,000 x 64 KB = 320 MB/s

Striping
Increases performance, or capacity, or both
Don’t mix volume types
Typically RAID 0 or LVM stripe
Avoid RAID for redundancy
EBS
EC2

Striping: Snapshots
Quiesce I/O
1. Database: FLUSH and LOCK tables
2. Filesystem: sync and fsfreeze
3. EBS: snapshot all volumes
When snapshot API returns,
it is safe to resume

EBS-optimized instance
Four key components: Balanced
EC2
A “boatload” of I/O
Right-sized EBS

Amazon CloudWatch
Important Amazon CloudWatch metrics:
• IOPS and bandwidth
• Latency
• Queue depth
All EBS metrics are prefixed with “Volume”

CloudWatch: Instance bandwidth
m4.2xlarge
Instance: 128MB/s
m4.4xlarge
Instance: 256MB/s
m4.10xlarge
Volume: 320MB/s

Distributing Key Names
Don’t do this
<my_bucket>/2013_11_13-164533125.jpg
<my_bucket>/2013_11_13-051033564.jpg
<my_bucket>/2013_11_13-061133789.jpg
<my_bucket>/2013_11_13-051033458.jpg
<my_bucket>/2013_11_12-063433125.jpg
<my_bucket>/2013_11_12-021033564.jpg
<my_bucket>/2013_11_12-065533789.jpg
<my_bucket>/2013_11_12-011033458.jpg
<my_bucket>/2013_11_11-022333125.jpg
<my_bucket>/2013_11_11-153433564.jpg
<my_bucket>/2013_11_11-065233789.jpg
<my_bucket>/2013_11_11-065633458.jpg

Distributing Key Names
Add randomness to the beginning of the key name
<my_bucket>/521335461-2013_11_13.jpg
<my_bucket>/465330151-2013_11_13.jpg
<my_bucket>/987331160-2013_11_13.jpg
<my_bucket>/465765461-2013_11_13.jpg
<my_bucket>/125631151-2013_11_13.jpg
<my_bucket>/934563160-2013_11_13.jpg
<my_bucket>/532132341-2013_11_13.jpg
<my_bucket>/565437681-2013_11_13.jpg
<my_bucket>/234567460-2013_11_13.jpg
<my_bucket>/456767561-2013_11_13.jpg
<my_bucket>/345565651-2013_11_13.jpg
<my_bucket>/431345660-2013_11_13.jpg

Other Techniques for Distributing Key Names
Store objects as a hash of their name
• add the original name as metadata
• “deadmau5_mix.mp3” à 0aa316fb000eae52921aab1b4697424958a53ad9
– watch for duplicate names!
• prepend keyname with short hash
• 0aa3-deadmau5_mix.mp3
Epoch time (reverse)
• 5321354831-deadmau5_mix.mp3

Randomness in a Key Name Can Be an Anti-Pattern
Lifecycle policies
LISTs with prefix filters
Maintaining thumbnails of images
• craig.jpg -> stored as orig-09329jed0fc
• thumb-09329jed0fc
When you need to recover a file with its original name

Solving for the Anti-Pattern
Add additional prefixes to help sorting
Amazon S3 maintains keys lexicographically in its internal
indices
<my_bucket>/images/521335461-2013_11_13.jpg
<my_bucket>/images/465330151-2013_11_13.jpg
<my_bucket>/movies/293924440-2013_11_13.jpg
<my_bucket>/movies/987331160-2013_11_13.jpg
<my_bucket>/thumbs-small/838434842-2013_11_13.jpg

CloudFront dynamic content
accleratoion

Region
Edge Location
12 Regions
32 Availability Zones
54 Edge Locations
Need to update
We’re here J

Configure multiple origins
Elastic Load
Balancing
Dynamic content
Amazon EC2
Static content
Amazon S3
* (default)
/error/*
/assets/*
Amazon CloudFront
example.com

CloudFront Behaviors
CloudFront
Customer Location
www.mysite.com
Path Pattern Matching
/*.jpg; /*.php etc.
GET http://mysite.com/images/1.jpg to ORIGIN A
GET http://mysite.com/index.php to ORIGIN B
GET http://mysite.com/web/home.css to ORIGIN C
GET http://mysite.com/* (DEFAULT) to ORIGIN D
Origin A: S3 bucket
Origin B:
www.mysite.com
Origin C: S3 Bucket
Origin D:
www.mysite.com
Path Pattern Matching
/*.php
/images/*.jpg
/web/*.css
/*.* (DEFAULT)

Region
Edge Location
12 Regions
32 Availability Zones
54 Edge Locations
Need to update AWS optimized network
Internet

Amazon S3 Transfer Acceleration
Embedded WAN acceleration
S3 Bucket
AWS Edge
Location
Uploader
Optimized
Throughput!
Move over long geographic distances
Up to 300% (6x) faster
No firewall mods, no client software
54 global edge locations
Change your endpoint, not your code

Accelerate Speed Comparison
• Test URL
• http://s3-accelerate-speedtest.s3-
accelerate.amazonaws.com/en/accelerate-speed-
comparsion.html
• bit.ly/news3ta
• Test Result (May-02-2016)
• Tested on May-02-2016, LGU+ Wifi at GSTower in Seoul
• http://bit.ly/newss3taresult

Testing S3 Transfer Accelerator by AWSCLI
$ sudo pip install –upgrade
awscli
$ aws configure set
default.s3.use_accelerate_endpo
int true

Testing S3 Transfer Accelerator by AWSCLI
$ aws s3 cp 33MB.pptx s3://ilho-saopaulo-01/
$ aws s3 cp 33MB.pptx s3://ilho-saopaulo-01/ --endpoint-
url http://ilho-saopaulo-01.s3-accelerate.amazonaws.com

S3 Transfer Acceleration Pricing
Starting at $0.04/GB transferred (+ usual
bandwidth charges). Up to $0.08/GB in
some regions
Pay only for what you use
Accelerated performance or no charge
Compare to hardware, per-GB or licenses

20160503 Amazed by AWS | Tips about Performance on AWS

More Related Content

Viewers also liked (20)

Similar to 20160503 Amazed by AWS | Tips about Performance on AWS (12)

More from Amazon Web Services Korea (20)

Recently uploaded (20)

20160503 Amazed by AWS | Tips about Performance on AWS