Effective EC2
1 like512 views
The document discusses Valentino Volonghi's presentation on effective use of Amazon EC2. It provides examples of how AdRoll migrated their infrastructure from self-hosted servers to using Amazon EC2 and other AWS services like S3 and CloudFront. It highlights benefits of EC2 like automatic scaling, easy software upgrades, low latency network access, and real-time monitoring. It also discusses strategies used for deploying application servers on EC2 using custom AMIs and boto monitoring.
![Boto monitoring
def regional_max_allocation(start_adserver=ec2.Image.start_adserver):
zones = [zone.name for zone in ec2.bconn.get_all_zones()]
(availables_with_ip, availables_without_ip,
software_failures, startings) = get_instances_by_status()
def setup_candidate(failure=None, free_ip=None):
if availables_without_ip:
new_instance = get_instance_in_unused_zone(availables_with_ip,
availables_without_ip,
lenient=True)
if failure is not None:
free_ip = failure.disassociate_elastic()
availables_without_ip.remove(new_instance)
availables_with_ip.add(new_instance)
new_instance.associate_elastic(free_ip)
return
if startings:
startings.pop()
return
start_adserver("adserver-" + utils.uuid(),
zone=get_unused_zone(availables_with_ip,
zones))
for software_failure in software_failures:
setup_candidate(failure=software_failure)
free_ips = ec2.get_all_free_elastic_ip(default=[])
for free_ip in free_ips:
Effective EC2 - Valentino Volonghi setup_candidate(free_ip=free_ip)
Wednesday, May 19, 2010](https://image.slidesharecdn.com/effective-ec2-100707042333-phpapp01/85/Effective-EC2-20-320.jpg)
![new_lines = ["127.0.0.1 localhost"]
for reservation in conn.get_all_instances():
for instance in reservation.instances:
if instance.state == "running":
KEYS.discard(instance.key_name)
new_lines.append(
"%s %s.internal %s" % (
socket.gethostbyname(instance.public_dns_name),
instance.key_name,
instance.key_name)
)
for missing_key in KEYS:
new_lines.append(
"127.0.0.1 %s.internal %s" % (missing_key, missing_key)
)
f = open('/etc/hosts.new', 'wb')
f.write("n".join(new_lines))
f.write("n")
f.flush()
f.close()
shutil.move('/etc/hosts.new', '/etc/hosts')
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010](https://image.slidesharecdn.com/effective-ec2-100707042333-phpapp01/85/Effective-EC2-37-320.jpg)
Effective EC2
- 1. Effective EC2 PyCon Italia Qu4ttro - May 7/8/9 2010 Valentino Volonghi - valentino@adroll.com Wednesday, May 19, 2010
- 2. Buzz Why I’ll never own another server... Joe Stump - Digg Lead Architect, CTO SimpleGeo http://stu.mp/2010/04/why-ill-never-own-another-server.html Cloud computing economies of scale James Hamilton - VP & Distinguished Engineer, Amazon Web Services http://live.visitmix.com/MIX10/Sessions/EX01 Moving to the cloud Reddit Team http://blog.reddit.com/2009/11/moving-to-cloud.html Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 3. Buzzwords Cost Reliability Multi-tenancy Agility Scalability Location Independence Security Maintenance Metering Effective EC2 - Valentino Volonghi Source: Wikipedia Wednesday, May 19, 2010
- 4. Competition... Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 5. Google AppEngine • Transparently scalable • No maintenance headaches • Software must be written ad hoc • Little control over infrastructure Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 6. Rackspace Cloud • CloudServers/CloudFiles/CloudSites • Some nice additional features • Fewer datacenters and can’t pick them • 12/11/07, 09/07/08, 03/11/09 Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 7. Others Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 8. Amazon AWS • 4 independent regions - 10 data centers • 3rd party vendor support • Well integrated services accessible via API • “Poor” single-node performance • Flexible upper limits Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 9. Amazon AWS Amazon Web Services can be as big as our retail business Andy Jassy - Amazon Senior VP Cloud Computing Business Revenue: Market Share: $650.000.000 77% Effective EC2 - Valentino Volonghi Source: www.businessweek.com/technology/content/apr2010/tc20100428_085106.htm Wednesday, May 19, 2010
- 10. AdRoll • Scaling High ROI Display Advertising • 60M paid campaigns impressions • 200M advertiser pixels impressions • 99% of requests below 3ms think time • Realtime Dynamic Ads Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 11. AdRoll 2008 • ServePath Housing • NaviSite CDN with mod_python • Self-Hosted DNS • MySQL DB • NFS Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 12. AdRoll 2009 • ServePath Housing • Amazon EC2 auto-managed AdServers • Amazon S3/CloudFront CDN • Dynect DNS Global Server Load Blancing • MySQL DB • NFS Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 13. Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 14. AdRoll 2010 • Fully Amazon EC2 • Akamai with Amazon S3 as origin • Dynect DNS GSLB • PostgreSQL DB • Amazon S3 instead of NFS Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 15. Campaigns/Reports/Plan platforms 3rd party Syncing AdRoll.com Scheduling Plan Ads Akamai DB S3 Plan/ Log Processing Config AdServers Logs/Performance Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 16. AdRoll 2010 Network Uptime: 100% Avg Resp Time: 96ms Effective EC2 - Valentino Volonghi Measured by Pingdom Wednesday, May 19, 2010
- 17. Migration Phase I - May 2009 Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 18. New Ad Servers • Very Quick & faultless boot procedure • Easy software upgrade • Low latency network • Real time monitoring Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 19. New Ad Servers • Custom bundled AMI stored in S3 • Python deploys the AdServer on boot • Hidden real load test • Boto monitoring Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 20. Boto monitoring def regional_max_allocation(start_adserver=ec2.Image.start_adserver): zones = [zone.name for zone in ec2.bconn.get_all_zones()] (availables_with_ip, availables_without_ip, software_failures, startings) = get_instances_by_status() def setup_candidate(failure=None, free_ip=None): if availables_without_ip: new_instance = get_instance_in_unused_zone(availables_with_ip, availables_without_ip, lenient=True) if failure is not None: free_ip = failure.disassociate_elastic() availables_without_ip.remove(new_instance) availables_with_ip.add(new_instance) new_instance.associate_elastic(free_ip) return if startings: startings.pop() return start_adserver("adserver-" + utils.uuid(), zone=get_unused_zone(availables_with_ip, zones)) for software_failure in software_failures: setup_candidate(failure=software_failure) free_ips = ec2.get_all_free_elastic_ip(default=[]) for free_ip in free_ips: Effective EC2 - Valentino Volonghi setup_candidate(free_ip=free_ip) Wednesday, May 19, 2010
- 21. Migration Phase 2 - March 2010 Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 22. Preparation • SQLAlchemy is awesome • PostgreSQL 5-10x faster than MySQL • 1 EBS is slower than 8 in RAID!! :) • m1.small instances are useless Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 23. Automated Deploy • Stock Ubuntu AMI • Setup scripts on S3 • Fabric ➡ 5 minutes to boot a new instance ➡ No maintenance overhead Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 24. Automated Deploy Deploy DB #!/bin/bash -ex wget -O- run.alestic.com/install/runurl | bash runurl "https://$base_s3/setup/add-keys-and-cert?SIGNATURE" runurl $base_s3/machine/db runurl $base_s3/deploy/db Deploy Web #!/bin/bash -ex wget -O- run.alestic.com/install/runurl | bash runurl "https://${base_s3}/setup/add-keys-and-cert?SIGNATURE" runurl $base_s3/machine/web Deploy App runurl $base_s3/deploy/web #!/bin/bash -ex runurl $base_s3/start/service production.ini paster wget -O- run.alestic.com/install/runurl | bash runurl "https://${base_s3}/setup/add-keys-and-cert?SIGNATURE" runurl $base_s3/machine/app runurl $base_s3/deploy/app runurl $base_s3/start/service production.ini watchwoman_inotify watchwoman_pixels watchwoman_amqp spaceman crawlman billman generalman turtl quexecutor sphinx Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 25. DB Instance • m2.2xlarge - 34GB RAM - 4 VCPU • 8 EBS RAID 1+0 • PostgreSQL 8.4 log shipping to S3 • RAID snapshot every 6 hours • Nightly SQL Backup Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 26. Generic Instance • m1.large - 7.5 GB RAM - 2 VCPU • 1 EBS with 500GB space • Store frequently changed data • Exact copies of each other Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 27. Web Instance • c1.medium - 1.7 GB RAM - 2 VCPU • No EBS • Amazon ELB frontend • Easily replaceable • Logs aggregated separately in real time Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 28. AWS Tips/Tricks Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 29. runurl by Eric Hammond. Download and run any script from any URL. Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 30. ec2-consistent-snapshot by Eric Hammond, atomic snapshot of EBS volumes with xfs filesystem. Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 31. Eric Hammond If you want to know more about Amazon AWS, he’s the one you want to talk to. Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 32. Use Public addresses Public addresses are automatically resolved to internal addresses when possible. Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 33. Use ELB and Cloudwatch Very useful for latency and load monitoring… It also makes scaling the web frontend extremely easy! Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 34. EBS RAID EBS is cheap. Instead of one big EBS create one big soft-RAID/LVM volume from many of them, it’s faster and safer. Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 35. SMTPAuth / SendGrid Mail sending from Amazon EC2 was/is crippled. Use a third party service to improve deliverability. Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 36. Use /etc/hosts Setting up a DNS is a lot of work at the beginning. Use SSH key names to setup a custom /etc/hosts file in each instance. Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 37. new_lines = ["127.0.0.1 localhost"] for reservation in conn.get_all_instances(): for instance in reservation.instances: if instance.state == "running": KEYS.discard(instance.key_name) new_lines.append( "%s %s.internal %s" % ( socket.gethostbyname(instance.public_dns_name), instance.key_name, instance.key_name) ) for missing_key in KEYS: new_lines.append( "127.0.0.1 %s.internal %s" % (missing_key, missing_key) ) f = open('/etc/hosts.new', 'wb') f.write("n".join(new_lines)) f.write("n") f.flush() f.close() shutil.move('/etc/hosts.new', '/etc/hosts') Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 38. AWS Wishlist Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 39. Elastic DNS Service Did I say that setting up your own DNS is hard? Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 40. Dedicated Rack Housing Try to get a Netezza rack in AWS today... Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 41. Improve ELB It’s a CNAME… No HTTPS termination Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 42. Thank you! Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
- 43. Answers? Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010