![Extras
BT Infrastructure
• Many small, HTTP webapps working together
• Key off „server groups‟ to define each environment
• “data01we2” - Server group “data” in the “we2” region
"roles": {"we2": {"data": ["dataserve", "auxserve"] } …
• Roles define which puppet modules we install
ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG](https://image.slidesharecdn.com/2013cloudconnect-131025090723-phpapp01/85/2013-cloudconnect-OpenStack-BT-17-320.jpg)
2013-cloudconnect-OpenStack@BT
- 1. OpenStack at BrightTag Joshua Buss – Cloud Connect 2013 ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
- 2. Intro Me ● DevOps since before it was called “Dev Ops” ● Orbitz – NOC/SOC ● Graphite specialist ● High-frequency trading ● Capacity planning, DBA, *nix admin, HW geek ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
- 3. Intro BrightTag ● World‟s leading Tag Management System ● Globally distributed on AWS ● Server-side tag firing through the cloud ● FUSE profile matching and offline processing ● Next-generation data management ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
- 4. Motivation Lay of the land Four environments: • Local (each developer‟s laptop) • Development (in-office only) • Staging (publicly available, in Rackspace) • Production (AWS) ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
- 5. Motivation The problem • Production was on an EOL‟d debian • Staging and dev used same distro • Each region had separate puppetmaster • Local is (mostly) OSX (brew; “btiab”) ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
- 6. Motivation The fix • Moved production to ubuntu 11.10 • Centralized puppet repo • Had to rebuild dev and staging, too ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
- 7. Motivation The opportunity • New dev environment? • Let‟s make it more like production! • Time to pick private cloud software ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
- 8. Process The goals • As similar to Amazon as possible • Open source, active community • libcloud compatible • Highly-available • Web interface ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
- 9. Process The choices • Ganeti • VMWare • CloudStack • Eucalyptus • OpenStack ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
- 10. Process The choices • Ganeti (no web interface) • VMWare ($) • CloudStack (Chaotic at the time) • Eucalyptus (Tried it, didn‟t work out) • OpenStack (Tried it, worked great) ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
- 11. Process The work • Found some cheap servers on eBay • Put Ubuntu 12.04 server on them • Used our new puppet configs to add users, etc. • Followed the docs ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
- 12. Process The work • Found some cheap servers on eBay • Put Ubuntu 12.04 server on them • Used our new puppet configs to add users, etc. • Followed the docs • Yeah, right. ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
- 13. Process The work • Took about a month to learn: • Terminology • How all the pieces fit together • Highly available networking support • Routing and ARP tables • Thank you, IRC ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
- 14. Result The finished product • Hey look, you can use this nice UI • ...or libcloud • ...and our single puppet master • ...and the result is just like our production instances! ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
- 15. Result Open Stack was just the start • Dev environment now has over 50 VMs on 15 servers • Standardize all the environments! • ...even some new CI environments • ...and our local environments (virtualbox) • Jenkins is integral in our continuous deployments ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
- 16. Thank you! ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
- 17. Extras BT Infrastructure • Many small, HTTP webapps working together • Key off „server groups‟ to define each environment • “data01we2” - Server group “data” in the “we2” region "roles": {"we2": {"data": ["dataserve", "auxserve"] } … • Roles define which puppet modules we install ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
- 18. Extras Zerg • libcloud + flask • Provisioning, real-time CMDB, generated configs • HAProxy, Nagios, Cassandra configs • “Manifest” - JSON that include IP Addresses • Use manifest to know where to deploy each app • Use zerg for DNS as well (updating /etc/hosts) ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
- 19. Extras Puppet • Source-controlled through multiple branches • Puppet master has each branch checked out into a different directory • Branches are pulled automatically • Each branch has a matching puppet „environment‟ name • /etc/puppet/puppet.conf “environment” determines the client • All environments, including our guest VMs, default to master ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
- 20. Extras Zero-downtime deploys • /bthc of each webapp determines healthiness of app • Fabric puts each app into maintenance mode before stopping • App stops taking traffic, then gets restarted with new code • New app doesn‟t respond with „healthy‟ until its actually ready for traffic • When there are a lot of an app, we parallelize the deploy ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG