Real life aws_user_data_auto_scale

Real Life AWS: user-data & AutoScale Adam Neumann Solutions Architect at IndexMedia [email_address] &

Part 1: user-data What is user-data? A problem user-data in the solution Further work

What is user-data? user specified instance attribute accessible via HTTP and curl/wget usage wrapped by cloud-init 'runs' the user data once on boot great for configuring instances

The concept any script, or init-cloud format #! /bin/bash etc etc cmd line arg or via UI:

A Problem too hard to spin up new instances complex, undocumented, manual fab, bash, and misc scripts slow: 30 min per instance first: consolidate, then: user-data

IndexMedia example 1 small user-data script, bootstrap load big script from S3 setup apache, hadoop, virtualenv,... 16 mins, vanilla ubuntu to serving automated: yes, optimal: no could be improved...

IndexMedia example 2 split script into 2 scripts :) 1: heavy, general, static = 15 min runs on vanilla ubuntu, creates AMI 2: light, instance specific = 90 secs runs on AMI from 30 min manual to 90 sec auto

Further work investigate cloud-init format include Puppet/Chef (first, learn Puppet/Chef...) integrate into CI process (environment becomes versioned, deployable, etc)

user-data summary great way to configure instances UI and cmd line accessible enables automated+repeatable deployment reduced deployment time, 30 mins to 90 secs

Part 2: AutoScale What is AutoScale Our (common) problem Show me the examples! Learnings

Part 2: AutoScale cmd line only = no UI = hidden = unknown = scary basically: launching & terminating instances within some bounds based upon metrics a second order system :)

AutoScale concepts 1. Launch Configuration individual: instance type, AMI, user-data, etc 2. AutoScale Group collective: ideal/max/min, AZs 3. Policy how: a reaction, increase, decrease 4. Alerts when: an event, high/low load, CW metrics

Our (common) problem maintain consistent experience for users, ie. Pip load time turn on servers when load is high bootstrapped, must turn off servers when load is low! enter AutoScale...

Solution = AutoScale servers start/stop based upon current system state spending on IT matches user demand exactly! 4 commands to set it up...

1/4: Launch Configuration as-create-launch-config gorilla_config_7d2 \ --image-id ami-XXXXXXX \ --instance-type t1.micro \ --group "gorilla-live" --key indexmedia-live \ --user-data-file /path/to/bootstrap.sh

2/4: Group as-create-auto-scaling-group as_group \ --launch-configuration gorilla_config_7d2 \ --availability-zones us-west-1a us-west-1b \ --min-size 10 --max-size 20 \ --load-balancers my_balancer \ --health-check-type EC2 \ --grace-period 120

3/4: Policy as-put-scaling-policy gorilla_scaleup_policy \ --auto-scaling-group as_group \ --type ChangeInCapacity \ --adjustment=1

4/4: Alarm mon-put-metric-alarm "gorilla high load" \ --dimensions "AutoScalingGroupName=as_group" \ --namespace "AWS/EC2" \ --statistic Average --metric-name CPUUtilization \ --comparison-operator GreaterThanThreshold \ --threshold 80 \ --evaluation-periods 5 --period 60 \ --alarm-actions ${POLICY-ARN}

... and done! Learnings 4 steps (well 6...) = elastic system feedback = second order system avoid bad harmonics! shorter instance boot time == quicker system response time validate alarms & behaviour w/ load tests on AWS

AutoScale summary cmd line only; just jump in! 4 distinct concepts, one per cmd quick boot = shorter grace period = quicker scaling up (enter user-data...) key to our multi-AZ, HA elastic cluster, problem solved!

Thanks + Questions? Adam Neumann Solutions Architect at IndexMedia [email_address]

Real life aws_user_data_auto_scale

More Related Content

What's hot (9)

Viewers also liked (7)

Similar to Real life aws_user_data_auto_scale (20)

Recently uploaded (20)

Real life aws_user_data_auto_scale