SlideShare a Scribd company logo
Lessons learned
managing large
AWS Environments
Ronald Bradford
http://ronaldbradford.com
@RonaldBradford
2013.06
SCOPE
Consulting experiences with AWS
Several different clients
Largest - 500+ servers
Some 40-50+ servers
Some 2-5 servers

LAMP/RoR/RDS/Windows
EffectiveMySQL.com - Performance, Scalability & Business Continuity
ABOUT MySELF
Ronald BRADFORD

Enterprise Data Architecture
24 years with RDBMS - 13 years with MySQL
Using AWS 4+ years
Published author - 4 books
Accomplished presenter - 8 years
Work at Independent MySQL Consultant
EffectiveMySQL.com - Performance, Scalability & Business Continuity
Covering
1. Products

5. Instrumentation

2. Cost

6. Failure

3. Web Scale
4. Security

EffectiveMySQL.com - Performance, Scalability & Business Continuity
1

AWS Products &
Ecosystem

EffectiveMySQL.com - Performance, Scalability & Business Continuity
ABOUT AWS
AMAZON WEB SERVICES
30+

Many, many products and
features

EC2, S3, EBS, ELB, RDS,
EMR,VPC, CDN, SWF,
SQS, SES, SNS, IAM, ...
Mechanical Turk
Flexible Payments Service (FPS)
EffectiveMySQL.com - Performance, Scalability & Business Continuity
AWS CONSOLE

May 2013

Aug 2012

EffectiveMySQL.com - Performance, Scalability & Business Continuity
Announcements
Product Announcements
http://aws.amazon.com/about-aws/newsletters/
Pricing Changes
New instance types
New features (e.g. IOPS)
New Products (e.g. Redshift/ OpsWorks)

EffectiveMySQL.com - Performance, Scalability & Business Continuity
Announcements
Product Announcements
Examples in presentation

http://aws.amazon.com/about-aws/newsletters/
Pricing Changes
New instance types
New features (e.g. IOPS)
New Products (e.g. Redshift/ OpsWorks)

EffectiveMySQL.com - Performance, Scalability & Business Continuity
ECOSYSTEM
AWS Marketplace

Over 800

https://aws.amazon.com/marketplace/

EffectiveMySQL.com - Performance, Scalability & Business Continuity
Product growth
When I started
No RDS, In-memory Cache,
DynamoDB, Glacier
No Elastic Beanstalk, OpsWorks
No management console

EffectiveMySQL.com - Performance, Scalability & Business Continuity
2

AWS Costs

EffectiveMySQL.com - Performance, Scalability & Business Continuity
operating cost
Are you monitoring your costs?
Daily
Hourly

EffectiveMySQL.com - Performance, Scalability & Business Continuity
Operating Cost
$ ec2_cost.sh

https://github.com/ronaldbradford/aws
EffectiveMySQL.com - Performance, Scalability & Business Continuity
Operating Cost
$29,000 p.m.

$ ec2_cost.sh

https://github.com/ronaldbradford/aws
EffectiveMySQL.com - Performance, Scalability & Business Continuity
Your Money
What is AWS costing you?
Instance types/sizes
http://aws.amazon.com/ec2/instance-types

Cost options
http://aws.amazon.com/ec2/pricing

EffectiveMySQL.com - Performance, Scalability & Business Continuity
Instance Types
General-purpose
Compute-optimized
Memory-optimized
Storage-optimized
GPU

EffectiveMySQL.com - Performance, Scalability & Business Continuity
$

Instance Prices
Large Instance (m1.large)

EffectiveMySQL.com - Performance, Scalability & Business Continuity
$

Instance Prices
Large Instance (m1.large)
On Demand

$0.24

Reserved

$0.136 *

+ Annual contract ( +$ 0.043)

Spot

$0.03+ *

Can be terminated (budget)

Per hour investment

EffectiveMySQL.com - Performance, Scalability & Business Continuity
$

Instance Prices
Large Instance (m1.large)
On Demand

$0.24

Reserved

$0.136 *

+ Annual contract ( +$ 0.043)

Spot

$0.03+ *

Can be terminated (budget)

Per hour investment

EffectiveMySQL.com - Performance, Scalability & Business Continuity
$

Instance Prices
Large Instance (m1.large)
On Demand

$0.24

Reserved

$0.136 *

+ Annual contract ( +$ 0.043)

Spot

$0.03+ *

Can be terminated (budget)

Per hour investment

EffectiveMySQL.com - Performance, Scalability & Business Continuity
$

Instance Prices
Large Instance (m1.large)

Was $0.32 til 11/19/2012
Was $0.26 til 1/16/2013

On Demand

$0.24

Reserved

$0.136 *

+ Annual contract ( +$ 0.043)

Spot

$0.03+ *

Can be terminated (budget)

Light/Medium/Heavy utilization

Per hour investment

40% saving

up to 80+% saving

EffectiveMySQL.com - Performance, Scalability & Business Continuity
SPOT EXAMPLE
One hour (24 cents)
1 x Large - Reserved
7.5G, 4 CPUs, 850G
8 x Large - Spot

or
1 x Eight Extra Large - Spot (cc2.8xlarge)
60G, 88 CPUs, 3.4T,10Gb NIC
EffectiveMySQL.com - Performance, Scalability & Business Continuity
SPOT EXAMPLE
One hour (24 cents)
1 x Large - Reserved
7.5G, 4 CPUs, 850G
8 x Large - Spot

price has changed 3 times
in 8 months

or
1 x Eight Extra Large - Spot (cc2.8xlarge)
60G, 88 CPUs, 3.4T,10Gb NIC
EffectiveMySQL.com - Performance, Scalability & Business Continuity
SPOT HISTORY
3c to 10c Zone A
3c to 40c Zone B

2013

$ ec2-describe-spot-price-history -t m1.large -d Linux/UNIX
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE
SPOTINSTANCEPRICE

0.030000
0.100000
0.030000
0.100000
0.030000
0.030000
0.100000
0.060000
0.030000
0.240000
0.030000
0.030000
0.045000
0.050000
0.400000
0.260000
0.030000
0.030000
0.026000

2013-05-28T17:20:41-0500
2013-05-28T17:07:02-0500
2013-05-28T16:37:51-0500
2013-05-28T16:31:03-0500
2013-05-28T16:24:48-0500
2013-05-28T16:24:48-0500
2013-05-28T16:15:03-0500
2013-05-28T16:08:34-0500
2013-05-28T16:01:59-0500
2013-05-28T15:55:12-0500
2013-05-28T15:48:32-0500
2013-05-28T15:42:07-0500
2013-05-28T15:35:47-0500
2013-05-28T15:35:47-0500
2013-05-28T15:29:15-0500
2013-05-28T15:22:47-0500
2013-05-28T15:16:01-0500
2013-05-28T15:16:01-0500
2013-05-28T15:09:30-0500

m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large

Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX

us-east-1a
us-east-1a
us-east-1a
us-east-1a
us-east-1d
us-east-1a
us-east-1a
us-east-1d
us-east-1b
us-east-1b
us-east-1b
us-east-1a
us-east-1a
us-east-1b
us-east-1b
us-east-1b
us-east-1d
us-east-1a
us-east-1a

EffectiveMySQL.com - Performance, Scalability & Business Continuity
SPOT HISTORY
2.6c to 17c (1/2 of 34c)
One AZ only

2012

$ ec2-describe-spot-price-history -t m1.large -d Linux/UNIX
0.0260
0.0260
0.0290
0.0370
0.0600
0.1700
0.1600
0.0600
0.0900
0.0260
0.0260

2012-09-27T09:45:46-0800
2012-09-27T09:45:46-0800
2012-09-27T09:38:37-0800
2012-09-27T09:38:37-0800
2012-09-27T09:31:29-0800
2012-09-27T09:31:29-0800
2012-09-27T09:24:20-0800
2012-09-27T09:17:11-0800
2012-09-27T09:17:11-0800
2012-09-27T09:09:55-0800
2012-09-27T09:09:55-0800

m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large
m1.large

Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX
Linux/UNIX

us-east-1b
us-east-1d
us-east-1b
us-east-1d
us-east-1b
us-east-1d
us-east-1d
us-east-1b
us-east-1d
us-east-1c
us-east-1b

EffectiveMySQL.com - Performance, Scalability & Business Continuity
Using SPOTS
Is your volume predicable?
Splitting on-demand/spot instances
Can work be done asynchronously?
i.e. can be queued
Is work restartable?

EffectiveMySQL.com - Performance, Scalability & Business Continuity
Using SPOTS
Is your volume predicable?
Splitting on-demand/spot instances
Can work be done asynchronously?
i.e. can be queued
Is work restartable?

WARNING: Not for
general workloads

EffectiveMySQL.com - Performance, Scalability & Business Continuity
Instance sizes
Evaluating the right instance size
What is your bottleneck?

EffectiveMySQL.com - Performance, Scalability & Business Continuity
Instance sizes
Evaluating the right instance size
What is your bottleneck?

Developing a tool to
recommend savings

EffectiveMySQL.com - Performance, Scalability & Business Continuity
TRUSTED ADVISOR
AWS now offers Trusted Advisor
Recommendations to save money
Improve performance
Close security problems

http://aws.amazon.com/premiumsupport/trustedadvisor/
EffectiveMySQL.com - Performance, Scalability & Business Continuity
COST SAVINGS
Other players
http://www.newvem.com/
http://www.cloudyn.com/

EffectiveMySQL.com - Performance, Scalability & Business Continuity
OTHER COST SAvings
CDN - Cloudfront
Bandwidth

http://aws.amazon.com/cloudfront/

Reduce response size (e.g. 10%)
Storage
old EBS snapshots
Remove unused instances

NEW: Announced 1/9/2103
CloudWatch Alarm Actions

EffectiveMySQL.com - Performance, Scalability & Business Continuity
3

Web Scale
(hint: no humans)

EffectiveMySQL.com - Performance, Scalability & Business Continuity
ABOUT WEB SCALE
GUI = #FAIL
CLI is necessary
Manual CLI use is slow
Automation in crucial
Parallel

EffectiveMySQL.com - Performance, Scalability & Business Continuity
AWS CLI’s
Different for EC2, ELB, RDS etc
Updated frequently (i.e. monthly)
$ git clone https://github.com/ronaldbradford/aws.git
$ cd aws/scripts
$ ./aws_cli_configure.sh

EffectiveMySQL.com - Performance, Scalability & Business Continuity
AWS CLI’s
Different for EC2, ELB, RDS etc
Updated frequently (i.e. monthly)
Simple helper
$ git clone https://github.com/ronaldbradford/aws.git
$ cd aws/scripts
$ ./aws_cli_configure.sh

EffectiveMySQL.com - Performance, Scalability & Business Continuity
RTFM

http://aws.amazon.com/archives/Amazon-EC2
EffectiveMySQL.com - Performance, Scalability & Business Continuity
Identifiers
Access Key ID
Private Access Key
X.509 Certificates (2 of)
Private

(*)

& Public

AWS Account ID
Canonical User ID
https://portal.aws.amazon.com/gp/aws/securityCredentials
EffectiveMySQL.com - Performance, Scalability & Business Continuity
CLI Examples
Launch Script
Demand/Spot or switch between
Verify SSH
Verify MySQL
Verify replication in sync
Add to ELB
EffectiveMySQL.com - Performance, Scalability & Business Continuity
CLI Examples
Audit Script
Consolidates information
Parallel operations
Unused EC2/EBS etc
Feeds reporting
ELB/EC2 usage
EffectiveMySQL.com - Performance, Scalability & Business Continuity
CLI EXAMPLES
Others
Cost Measurement
Cloning (optimizes scale-up)
Move servers between load balancers
Spot History graphing
Spot History email alerts
EffectiveMySQL.com - Performance, Scalability & Business Continuity
4

AWS Security

EffectiveMySQL.com - Performance, Scalability & Business Continuity
SECURITY
Do not give away the front door keys
Do not open all the windows

EffectiveMySQL.com - Performance, Scalability & Business Continuity
SECURITY OPTIONS
Keypairs

Learn the different benefits

Security groups
Virtual Private Cloud (VPC)
Identity and Access Management (IAM)
Multi-factor authentication
http://aws.amazon.com/mfa/
EffectiveMySQL.com - Performance, Scalability & Business Continuity
SECURITY TIPS
Restrict open access to port 80/443
Jump box
Restrict IP Access
Additional authentication
Per user SSH authentication
Do not use keypair
EffectiveMySQL.com - Performance, Scalability & Business Continuity
products
Many Others (AWS Summit 2013)
Cloudaware

Dome9

Enstratius

SafeNet

AlertLogic

EffectiveMySQL.com - Performance, Scalability & Business Continuity
5

Instrumentation

EffectiveMySQL.com - Performance, Scalability & Business Continuity
Instrumentation

EffectiveMySQL.com - Performance, Scalability & Business Continuity
Instrumentation
What is important to you?

EffectiveMySQL.com - Performance, Scalability & Business Continuity
Instrumentation
What is important to you?
All server stats

EffectiveMySQL.com - Performance, Scalability & Business Continuity
Instrumentation
What is important to you?
All server stats
Sampling issues

EffectiveMySQL.com - Performance, Scalability & Business Continuity
Instrumentation
What is important to you?
All server stats
Sampling issues
Deceiving averages (frequency)

EffectiveMySQL.com - Performance, Scalability & Business Continuity
REQUESTS PER SEC

5 second averages, not 1 minute sample
https://github.com/ronaldbradford/reqstat
EffectiveMySQL.com - Performance, Scalability & Business Continuity
REQUESTS PER SEC

5 second averages, not 1 minute sample
https://github.com/ronaldbradford/reqstat
EffectiveMySQL.com - Performance, Scalability & Business Continuity
REQUESTS PER SEC

-1,500 RPS

5 second averages, not 1 minute sample
https://github.com/ronaldbradford/reqstat
EffectiveMySQL.com - Performance, Scalability & Business Continuity
outliers

EffectiveMySQL.com - Performance, Scalability & Business Continuity
outliers
I care about these

EffectiveMySQL.com - Performance, Scalability & Business Continuity
TESTING
End to end testing critical
Network latency
ELB performance

EffectiveMySQL.com - Performance, Scalability & Business Continuity
products
AWS Cloudwatch
Many Others (AWS Summit 2013)
Datadog

AppDynamics

Boundary
CopperEgg

EffectiveMySQL.com - Performance, Scalability & Business Continuity
products
AWS Cloudwatch

What features matter?

Many Others (AWS Summit 2013)
Datadog

AppDynamics

Boundary
CopperEgg

EffectiveMySQL.com - Performance, Scalability & Business Continuity
6

Failure

EffectiveMySQL.com - Performance, Scalability & Business Continuity
FAILURE

EffectiveMySQL.com - Performance, Scalability & Business Continuity
FAILURE
Instances fail

EffectiveMySQL.com - Performance, Scalability & Business Continuity
FAILURE
Instances fail
Outages occur
AWS scheduled reboots

EffectiveMySQL.com - Performance, Scalability & Business Continuity
FAILURE
Instances fail
Outages occur
AWS scheduled reboots
Be prepared
Chaos Monkey
http://www.codinghorror.com/blog/2011/04/working-with-the-chaos-monkey.html

EffectiveMySQL.com - Performance, Scalability & Business Continuity
CONCLUSION

EffectiveMySQL.com - Performance, Scalability & Business Continuity
CONCLUSION
Cost Management (saving money)

EffectiveMySQL.com - Performance, Scalability & Business Continuity
CONCLUSION
Cost Management (saving money)
CLI automation

EffectiveMySQL.com - Performance, Scalability & Business Continuity
CONCLUSION
Cost Management (saving money)
CLI automation
Instrumentation (inc business metrics)

EffectiveMySQL.com - Performance, Scalability & Business Continuity
CONCLUSION
Cost Management (saving money)
CLI automation
Instrumentation (inc business metrics)
Distribute your application & data

EffectiveMySQL.com - Performance, Scalability & Business Continuity
CONCLUSION
Cost Management (saving money)
CLI automation
Instrumentation (inc business metrics)
Distribute your application & data
Disaster is inevitable

EffectiveMySQL.com - Performance, Scalability & Business Continuity
AWS for FREE
http://aws.amazon.com/free/
Free EC2 t1.micro for a year
Free RDS t1.micro for a year
S3, DynamoDB, SimpleDB, +++

EffectiveMySQL.com - Performance, Scalability & Business Continuity
Ronald Bradford
http://effectiveMySQL.com
EffectiveMySQL.com - Performance, Scalability & Business Continuity

More Related Content

PDF
Monitoring your technology stack with New Relic
PDF
Successful Scalability Principles - Part 1
PDF
The History and Future of the MySQL ecosystem
PDF
MySQL Scalability Mistakes - OTN
PDF
MySQL Backup and Recovery Essentials
PPT
My two cents about Mysql backup
PDF
MySQL High Availability and Disaster Recovery with Continuent, a VMware company
PDF
MySQL Enterprise Backup - BnR Scenarios
Monitoring your technology stack with New Relic
Successful Scalability Principles - Part 1
The History and Future of the MySQL ecosystem
MySQL Scalability Mistakes - OTN
MySQL Backup and Recovery Essentials
My two cents about Mysql backup
MySQL High Availability and Disaster Recovery with Continuent, a VMware company
MySQL Enterprise Backup - BnR Scenarios

What's hot (20)

PDF
Moodle Moot Spain: Moodle Available and Scalable with MySQL HA - InnoDB Clust...
PDF
MySQL Best Practices - OTN
PDF
UKOUG 2011: MySQL Architectures for Oracle DBA's
PDF
MySQL 5.7 -- SCaLE Feb 2014
PDF
MySQL Performance Tuning: The Perfect Scalability (OOW2019)
PDF
MySQL :What's New #GIDS16
PDF
Posscon my sql56
PDF
MySQL For Oracle DBA's and Developers
PDF
MySQL Enterprise Backup (MEB)
PDF
MySQL 5.7 New Features to Exploit -- PHPTek/Chicago MySQL User Group May 2014
PDF
MySQL Performance Best Practices
PDF
Scaling MySQl 1 to N Servers -- Los Angelese MySQL User Group Feb 2014
PDF
MySQL Performance Tuning Variables
ODP
MySQL 5.7 - What's new and How to upgrade
PPT
MySQL 5.6 Updates
PDF
MySql's NoSQL -- best of both worlds on the same disks
PDF
What's New in MySQL 8.0 @ HKOSC 2017
PDF
MySQL Cloud Service Deep Dive
KEY
Perf Tuning Short
PDF
MySQL Performance Tuning. Part 1: MySQL Configuration (includes MySQL 5.7)
Moodle Moot Spain: Moodle Available and Scalable with MySQL HA - InnoDB Clust...
MySQL Best Practices - OTN
UKOUG 2011: MySQL Architectures for Oracle DBA's
MySQL 5.7 -- SCaLE Feb 2014
MySQL Performance Tuning: The Perfect Scalability (OOW2019)
MySQL :What's New #GIDS16
Posscon my sql56
MySQL For Oracle DBA's and Developers
MySQL Enterprise Backup (MEB)
MySQL 5.7 New Features to Exploit -- PHPTek/Chicago MySQL User Group May 2014
MySQL Performance Best Practices
Scaling MySQl 1 to N Servers -- Los Angelese MySQL User Group Feb 2014
MySQL Performance Tuning Variables
MySQL 5.7 - What's new and How to upgrade
MySQL 5.6 Updates
MySql's NoSQL -- best of both worlds on the same disks
What's New in MySQL 8.0 @ HKOSC 2017
MySQL Cloud Service Deep Dive
Perf Tuning Short
MySQL Performance Tuning. Part 1: MySQL Configuration (includes MySQL 5.7)
Ad

Similar to Lessons Learned Managing Large AWS Environments (20)

PDF
Scaling MySQL in Amazon Web Services
PDF
Running MySQL in AWS
PDF
Successful MySQL Scalability
PPTX
Optimizing Your Cloud Applications in RightScale
PPTX
Continuous Availability and Scale-out for MySQL with ScaleBase Lite & Enterpr...
PDF
Amazon Web Services - Relational Database Service Meetup
PPTX
MySQL Tech Tour 2015 - Manage & Tune
PDF
High Performance MySQL Optimization Backups Replication and More Second Editi...
PDF
Full Download High Performance MySQL Optimization Backups Replication and Mor...
PPTX
SQL Server in the AWS Cloud
PDF
High Performance MySQL Optimization Backups Replication and More Second Editi...
PDF
[INSIGHT OUT 2011] A17 mastering my sql performance tuning(ronald bradford)
PDF
The Peoper Care and Feeding of a MySQL Server for Busy Linux Admin
PDF
Evolution of DBA in the Cloud Era
PDF
Scaling MySQL -- Swanseacon.co.uk
PDF
Databases in the hosted cloud
PDF
Running MySQL on Linux
PPTX
Managing MySQL Scale Through Consolidation
PDF
MySQL infra readiness-for-peak-sale-events - Kabilesh PR (Co-Founder of Mydbops)
PDF
MySQL Intro JSON NoSQL
Scaling MySQL in Amazon Web Services
Running MySQL in AWS
Successful MySQL Scalability
Optimizing Your Cloud Applications in RightScale
Continuous Availability and Scale-out for MySQL with ScaleBase Lite & Enterpr...
Amazon Web Services - Relational Database Service Meetup
MySQL Tech Tour 2015 - Manage & Tune
High Performance MySQL Optimization Backups Replication and More Second Editi...
Full Download High Performance MySQL Optimization Backups Replication and Mor...
SQL Server in the AWS Cloud
High Performance MySQL Optimization Backups Replication and More Second Editi...
[INSIGHT OUT 2011] A17 mastering my sql performance tuning(ronald bradford)
The Peoper Care and Feeding of a MySQL Server for Busy Linux Admin
Evolution of DBA in the Cloud Era
Scaling MySQL -- Swanseacon.co.uk
Databases in the hosted cloud
Running MySQL on Linux
Managing MySQL Scale Through Consolidation
MySQL infra readiness-for-peak-sale-events - Kabilesh PR (Co-Founder of Mydbops)
MySQL Intro JSON NoSQL
Ad

More from Ronald Bradford (20)

PDF
My SQL Idiosyncrasies That Bite OTN
PDF
MySQL Best Practices - OTN LAD Tour
PDF
MySQL Idiosyncrasies That Bite SF
PDF
MySQL Idiosyncrasies That Bite 2010.07
PDF
Capturing, Analyzing and Optimizing MySQL
PDF
MySQL Idiosyncrasies That Bite
KEY
10x Performance Improvements
PDF
LIFTOFF - MySQLCamp for the Oracle DBA
PDF
IGNITION - MySQLCamp for the Oracle DBA
PDF
10x Performance Improvements - A Case Study
PDF
Dolphins Now And Beyond - FOSDEM 2010
PDF
Drizzle - Status, Principles and Ecosystem
PDF
SQL v No SQL
PDF
MySQL for the Oracle DBA - Object Management
PDF
Know Your Competitor - Oracle 10g Express Edition
PDF
MySQL For Oracle Developers
PDF
The Ideal Performance Architecture
PDF
MySQL Monitoring 101
PDF
Getting started with MySQL on Amazon Web Services
PDF
Best Practices in Migrating to MySQL - Part 1
My SQL Idiosyncrasies That Bite OTN
MySQL Best Practices - OTN LAD Tour
MySQL Idiosyncrasies That Bite SF
MySQL Idiosyncrasies That Bite 2010.07
Capturing, Analyzing and Optimizing MySQL
MySQL Idiosyncrasies That Bite
10x Performance Improvements
LIFTOFF - MySQLCamp for the Oracle DBA
IGNITION - MySQLCamp for the Oracle DBA
10x Performance Improvements - A Case Study
Dolphins Now And Beyond - FOSDEM 2010
Drizzle - Status, Principles and Ecosystem
SQL v No SQL
MySQL for the Oracle DBA - Object Management
Know Your Competitor - Oracle 10g Express Edition
MySQL For Oracle Developers
The Ideal Performance Architecture
MySQL Monitoring 101
Getting started with MySQL on Amazon Web Services
Best Practices in Migrating to MySQL - Part 1

Recently uploaded (20)

PPTX
Programs and apps: productivity, graphics, security and other tools
PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
PDF
Encapsulation theory and applications.pdf
PPTX
Understanding_Digital_Forensics_Presentation.pptx
PPTX
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
PDF
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
PDF
Machine learning based COVID-19 study performance prediction
PDF
NewMind AI Weekly Chronicles - August'25 Week I
PDF
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
PDF
Dropbox Q2 2025 Financial Results & Investor Presentation
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
PPTX
20250228 LYD VKU AI Blended-Learning.pptx
PPT
Teaching material agriculture food technology
PDF
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
PPTX
Big Data Technologies - Introduction.pptx
PPTX
ACSFv1EN-58255 AWS Academy Cloud Security Foundations.pptx
PDF
Unlocking AI with Model Context Protocol (MCP)
PDF
Network Security Unit 5.pdf for BCA BBA.
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
Programs and apps: productivity, graphics, security and other tools
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Encapsulation theory and applications.pdf
Understanding_Digital_Forensics_Presentation.pptx
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
Machine learning based COVID-19 study performance prediction
NewMind AI Weekly Chronicles - August'25 Week I
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Dropbox Q2 2025 Financial Results & Investor Presentation
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
20250228 LYD VKU AI Blended-Learning.pptx
Teaching material agriculture food technology
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
Big Data Technologies - Introduction.pptx
ACSFv1EN-58255 AWS Academy Cloud Security Foundations.pptx
Unlocking AI with Model Context Protocol (MCP)
Network Security Unit 5.pdf for BCA BBA.
Mobile App Security Testing_ A Comprehensive Guide.pdf
Per capita expenditure prediction using model stacking based on satellite ima...

Lessons Learned Managing Large AWS Environments

  • 1. Lessons learned managing large AWS Environments Ronald Bradford http://ronaldbradford.com @RonaldBradford 2013.06
  • 2. SCOPE Consulting experiences with AWS Several different clients Largest - 500+ servers Some 40-50+ servers Some 2-5 servers LAMP/RoR/RDS/Windows EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 3. ABOUT MySELF Ronald BRADFORD Enterprise Data Architecture 24 years with RDBMS - 13 years with MySQL Using AWS 4+ years Published author - 4 books Accomplished presenter - 8 years Work at Independent MySQL Consultant EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 4. Covering 1. Products 5. Instrumentation 2. Cost 6. Failure 3. Web Scale 4. Security EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 5. 1 AWS Products & Ecosystem EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 6. ABOUT AWS AMAZON WEB SERVICES 30+ Many, many products and features EC2, S3, EBS, ELB, RDS, EMR,VPC, CDN, SWF, SQS, SES, SNS, IAM, ... Mechanical Turk Flexible Payments Service (FPS) EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 7. AWS CONSOLE May 2013 Aug 2012 EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 8. Announcements Product Announcements http://aws.amazon.com/about-aws/newsletters/ Pricing Changes New instance types New features (e.g. IOPS) New Products (e.g. Redshift/ OpsWorks) EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 9. Announcements Product Announcements Examples in presentation http://aws.amazon.com/about-aws/newsletters/ Pricing Changes New instance types New features (e.g. IOPS) New Products (e.g. Redshift/ OpsWorks) EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 11. Product growth When I started No RDS, In-memory Cache, DynamoDB, Glacier No Elastic Beanstalk, OpsWorks No management console EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 12. 2 AWS Costs EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 13. operating cost Are you monitoring your costs? Daily Hourly EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 15. Operating Cost $29,000 p.m. $ ec2_cost.sh https://github.com/ronaldbradford/aws EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 16. Your Money What is AWS costing you? Instance types/sizes http://aws.amazon.com/ec2/instance-types Cost options http://aws.amazon.com/ec2/pricing EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 18. $ Instance Prices Large Instance (m1.large) EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 19. $ Instance Prices Large Instance (m1.large) On Demand $0.24 Reserved $0.136 * + Annual contract ( +$ 0.043) Spot $0.03+ * Can be terminated (budget) Per hour investment EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 20. $ Instance Prices Large Instance (m1.large) On Demand $0.24 Reserved $0.136 * + Annual contract ( +$ 0.043) Spot $0.03+ * Can be terminated (budget) Per hour investment EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 21. $ Instance Prices Large Instance (m1.large) On Demand $0.24 Reserved $0.136 * + Annual contract ( +$ 0.043) Spot $0.03+ * Can be terminated (budget) Per hour investment EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 22. $ Instance Prices Large Instance (m1.large) Was $0.32 til 11/19/2012 Was $0.26 til 1/16/2013 On Demand $0.24 Reserved $0.136 * + Annual contract ( +$ 0.043) Spot $0.03+ * Can be terminated (budget) Light/Medium/Heavy utilization Per hour investment 40% saving up to 80+% saving EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 23. SPOT EXAMPLE One hour (24 cents) 1 x Large - Reserved 7.5G, 4 CPUs, 850G 8 x Large - Spot or 1 x Eight Extra Large - Spot (cc2.8xlarge) 60G, 88 CPUs, 3.4T,10Gb NIC EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 24. SPOT EXAMPLE One hour (24 cents) 1 x Large - Reserved 7.5G, 4 CPUs, 850G 8 x Large - Spot price has changed 3 times in 8 months or 1 x Eight Extra Large - Spot (cc2.8xlarge) 60G, 88 CPUs, 3.4T,10Gb NIC EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 25. SPOT HISTORY 3c to 10c Zone A 3c to 40c Zone B 2013 $ ec2-describe-spot-price-history -t m1.large -d Linux/UNIX SPOTINSTANCEPRICE SPOTINSTANCEPRICE SPOTINSTANCEPRICE SPOTINSTANCEPRICE SPOTINSTANCEPRICE SPOTINSTANCEPRICE SPOTINSTANCEPRICE SPOTINSTANCEPRICE SPOTINSTANCEPRICE SPOTINSTANCEPRICE SPOTINSTANCEPRICE SPOTINSTANCEPRICE SPOTINSTANCEPRICE SPOTINSTANCEPRICE SPOTINSTANCEPRICE SPOTINSTANCEPRICE SPOTINSTANCEPRICE SPOTINSTANCEPRICE SPOTINSTANCEPRICE 0.030000 0.100000 0.030000 0.100000 0.030000 0.030000 0.100000 0.060000 0.030000 0.240000 0.030000 0.030000 0.045000 0.050000 0.400000 0.260000 0.030000 0.030000 0.026000 2013-05-28T17:20:41-0500 2013-05-28T17:07:02-0500 2013-05-28T16:37:51-0500 2013-05-28T16:31:03-0500 2013-05-28T16:24:48-0500 2013-05-28T16:24:48-0500 2013-05-28T16:15:03-0500 2013-05-28T16:08:34-0500 2013-05-28T16:01:59-0500 2013-05-28T15:55:12-0500 2013-05-28T15:48:32-0500 2013-05-28T15:42:07-0500 2013-05-28T15:35:47-0500 2013-05-28T15:35:47-0500 2013-05-28T15:29:15-0500 2013-05-28T15:22:47-0500 2013-05-28T15:16:01-0500 2013-05-28T15:16:01-0500 2013-05-28T15:09:30-0500 m1.large m1.large m1.large m1.large m1.large m1.large m1.large m1.large m1.large m1.large m1.large m1.large m1.large m1.large m1.large m1.large m1.large m1.large m1.large Linux/UNIX Linux/UNIX Linux/UNIX Linux/UNIX Linux/UNIX Linux/UNIX Linux/UNIX Linux/UNIX Linux/UNIX Linux/UNIX Linux/UNIX Linux/UNIX Linux/UNIX Linux/UNIX Linux/UNIX Linux/UNIX Linux/UNIX Linux/UNIX Linux/UNIX us-east-1a us-east-1a us-east-1a us-east-1a us-east-1d us-east-1a us-east-1a us-east-1d us-east-1b us-east-1b us-east-1b us-east-1a us-east-1a us-east-1b us-east-1b us-east-1b us-east-1d us-east-1a us-east-1a EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 26. SPOT HISTORY 2.6c to 17c (1/2 of 34c) One AZ only 2012 $ ec2-describe-spot-price-history -t m1.large -d Linux/UNIX 0.0260 0.0260 0.0290 0.0370 0.0600 0.1700 0.1600 0.0600 0.0900 0.0260 0.0260 2012-09-27T09:45:46-0800 2012-09-27T09:45:46-0800 2012-09-27T09:38:37-0800 2012-09-27T09:38:37-0800 2012-09-27T09:31:29-0800 2012-09-27T09:31:29-0800 2012-09-27T09:24:20-0800 2012-09-27T09:17:11-0800 2012-09-27T09:17:11-0800 2012-09-27T09:09:55-0800 2012-09-27T09:09:55-0800 m1.large m1.large m1.large m1.large m1.large m1.large m1.large m1.large m1.large m1.large m1.large Linux/UNIX Linux/UNIX Linux/UNIX Linux/UNIX Linux/UNIX Linux/UNIX Linux/UNIX Linux/UNIX Linux/UNIX Linux/UNIX Linux/UNIX us-east-1b us-east-1d us-east-1b us-east-1d us-east-1b us-east-1d us-east-1d us-east-1b us-east-1d us-east-1c us-east-1b EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 27. Using SPOTS Is your volume predicable? Splitting on-demand/spot instances Can work be done asynchronously? i.e. can be queued Is work restartable? EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 28. Using SPOTS Is your volume predicable? Splitting on-demand/spot instances Can work be done asynchronously? i.e. can be queued Is work restartable? WARNING: Not for general workloads EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 29. Instance sizes Evaluating the right instance size What is your bottleneck? EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 30. Instance sizes Evaluating the right instance size What is your bottleneck? Developing a tool to recommend savings EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 31. TRUSTED ADVISOR AWS now offers Trusted Advisor Recommendations to save money Improve performance Close security problems http://aws.amazon.com/premiumsupport/trustedadvisor/ EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 33. OTHER COST SAvings CDN - Cloudfront Bandwidth http://aws.amazon.com/cloudfront/ Reduce response size (e.g. 10%) Storage old EBS snapshots Remove unused instances NEW: Announced 1/9/2103 CloudWatch Alarm Actions EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 34. 3 Web Scale (hint: no humans) EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 35. ABOUT WEB SCALE GUI = #FAIL CLI is necessary Manual CLI use is slow Automation in crucial Parallel EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 36. AWS CLI’s Different for EC2, ELB, RDS etc Updated frequently (i.e. monthly) $ git clone https://github.com/ronaldbradford/aws.git $ cd aws/scripts $ ./aws_cli_configure.sh EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 37. AWS CLI’s Different for EC2, ELB, RDS etc Updated frequently (i.e. monthly) Simple helper $ git clone https://github.com/ronaldbradford/aws.git $ cd aws/scripts $ ./aws_cli_configure.sh EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 39. Identifiers Access Key ID Private Access Key X.509 Certificates (2 of) Private (*) & Public AWS Account ID Canonical User ID https://portal.aws.amazon.com/gp/aws/securityCredentials EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 40. CLI Examples Launch Script Demand/Spot or switch between Verify SSH Verify MySQL Verify replication in sync Add to ELB EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 41. CLI Examples Audit Script Consolidates information Parallel operations Unused EC2/EBS etc Feeds reporting ELB/EC2 usage EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 42. CLI EXAMPLES Others Cost Measurement Cloning (optimizes scale-up) Move servers between load balancers Spot History graphing Spot History email alerts EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 43. 4 AWS Security EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 44. SECURITY Do not give away the front door keys Do not open all the windows EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 45. SECURITY OPTIONS Keypairs Learn the different benefits Security groups Virtual Private Cloud (VPC) Identity and Access Management (IAM) Multi-factor authentication http://aws.amazon.com/mfa/ EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 46. SECURITY TIPS Restrict open access to port 80/443 Jump box Restrict IP Access Additional authentication Per user SSH authentication Do not use keypair EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 47. products Many Others (AWS Summit 2013) Cloudaware Dome9 Enstratius SafeNet AlertLogic EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 48. 5 Instrumentation EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 49. Instrumentation EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 50. Instrumentation What is important to you? EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 51. Instrumentation What is important to you? All server stats EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 52. Instrumentation What is important to you? All server stats Sampling issues EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 53. Instrumentation What is important to you? All server stats Sampling issues Deceiving averages (frequency) EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 54. REQUESTS PER SEC 5 second averages, not 1 minute sample https://github.com/ronaldbradford/reqstat EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 55. REQUESTS PER SEC 5 second averages, not 1 minute sample https://github.com/ronaldbradford/reqstat EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 56. REQUESTS PER SEC -1,500 RPS 5 second averages, not 1 minute sample https://github.com/ronaldbradford/reqstat EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 57. outliers EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 58. outliers I care about these EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 59. TESTING End to end testing critical Network latency ELB performance EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 60. products AWS Cloudwatch Many Others (AWS Summit 2013) Datadog AppDynamics Boundary CopperEgg EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 61. products AWS Cloudwatch What features matter? Many Others (AWS Summit 2013) Datadog AppDynamics Boundary CopperEgg EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 62. 6 Failure EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 63. FAILURE EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 64. FAILURE Instances fail EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 65. FAILURE Instances fail Outages occur AWS scheduled reboots EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 66. FAILURE Instances fail Outages occur AWS scheduled reboots Be prepared Chaos Monkey http://www.codinghorror.com/blog/2011/04/working-with-the-chaos-monkey.html EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 67. CONCLUSION EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 68. CONCLUSION Cost Management (saving money) EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 69. CONCLUSION Cost Management (saving money) CLI automation EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 70. CONCLUSION Cost Management (saving money) CLI automation Instrumentation (inc business metrics) EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 71. CONCLUSION Cost Management (saving money) CLI automation Instrumentation (inc business metrics) Distribute your application & data EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 72. CONCLUSION Cost Management (saving money) CLI automation Instrumentation (inc business metrics) Distribute your application & data Disaster is inevitable EffectiveMySQL.com - Performance, Scalability & Business Continuity
  • 73. AWS for FREE http://aws.amazon.com/free/ Free EC2 t1.micro for a year Free RDS t1.micro for a year S3, DynamoDB, SimpleDB, +++ EffectiveMySQL.com - Performance, Scalability & Business Continuity