Distributed Release Management
7 likes11,716 views
Etsy deploys code and configuration changes over 40 times per day using a distributed release management system. The process involves automated testing, deployment to pre-production environments, monitoring, and human oversight throughout. Deployments take around 15 minutes on average and involve parallel testing, configuration changes, and code deployment to multiple environments. The system provides improved visibility, communication, and a record for post-mortems. It was developed through iterative improvements over four years.
Distributed Release Management
- 1. Distributed Release Management Deploying etsy.com 40+ times per day Mike Brittain Engineering Director, Etsy @mikebrittain mikebrittain.com/talks
- 2. 1st Day Assignment Put your face on etsy.com/about
- 3. What I’m showing you tonight is the result of four years of iteration.
- 4. Small incremental changes to the application “Dark” features: new classes, methods, controllers Graphics, stylesheets, templates Copy/content changes ! App deploys Turning flags on, off, or % ramp up Config deploys
- 5. Latent bugs and security holes Traffic management, load shedding Adding and removing infrastructure ! Tweaking config flags or releasing patches. “Operating” the site
- 6. IRC, #push
- 7. /topic mbrittain | jgoulah | rsnyder | ekastner
- 8. /topic mbrittain, jgoulah, rsnyder | ekastner
- 9. Keep real people in the loop Queue, with max batch size of seven. Automated deployment run by humans
- 10. 4 people in this deploy. “I’ve pushed my changes to master.” “Everyone has checked in.”
- 11. Build QA and Pre-prod Build progress Status in #push Git SHA1 in for each env. Date, username, deploy log, changeset, link to dashboard from time of deploy
- 13. Reporting what’s going on in Deployinator, and who triggered Status from build cluster
- 14. Pre-prod (“princess”) has been deployed. ! SHA1 of the change Time it took to deploy Link to changeset in GitHub Log of the deploy script
- 15. Btw, there are three bots talking in channel at this point. O_o
- 16. Queuing for next deploy Humans talk to other humans from time to time.
- 17. Talking to pushbot. ! Pushbot knows some Spanish… because, ya know, why not?
- 18. Link to test results for CI environment, along with how long the tests took.Alerting by name.
- 19. 8 minutes have elapsed… We’ve built and tested our release in the CI environment (“QA”). ! QA build failed our 5 min. SLA for tests.
- 20. “Try” is our pre-commit testing cluster.
- 21. Bots help reinforce our values. This is especially helpful for new people on the team.
- 23. Still 8 minutes elapsed… Pre-prod has been deployed and tested. ! This ran in parallel with our QA build and tests.
- 25. Cross-traffic: In a separate channel (#config), our app configs files were deployed to pre-prod.
- 30. Cross-traffic: Ops team deployed a configuration change. And, yes… another non-human.
- 32. Code is live Link to dashboard.
- 34. 13 minutes elapsed… Code is now in production with public traffic.
- 35. Who committed code in the last deploy? And how many lines did each of them change?
- 38. Handoff for the next deploy.
- 39. Entire app deploy took 15 minutes. ! 4 people running the deployment 8 committers Config deploy and Chef change deployed in parallel.
- 40. Optimal queue size Normalized communication Improved visibility Historical record is ideal for post-mortems Organic evolution
- 41. Hold up the queue (.hold) Work the issue with the people available in #push Additional help always available in #sysops Buddy-system for off-hours deploys Ops-on-call, dev-on-call When something goes wrong?
- 42. 25 Million Items listed 60+ Million Monthly unique visitors 200 Countries with annual transactions ! 175+ Committers, everyone deploys Items by anjaysdesigns, betwixxt, OneStarLeatherGoods, mediumcontrol, TheDesignPallet
- 44. Start small. (We did.) Automated tests and production monitoring. Have a story around maintaining quality. “We can always go back to the old way.” Demonstrate value to leadership.
- 45. Go write your own story.
- 46. Thank you. Mike Brittain Engineering Director, Etsy @mikebrittain mikebrittain.com/talks