Scaling Humans - BigPanda's Fabulous ChatOps Adventure - Erik Zaadi, BigPanda - DevOpsDays Tel Aviv 2016
2 likes410 views
The document discusses BigPanda's evolution in deployment practices through the use of chatops, notably with the development of a bot named Robin, which improved deployment management and reduced human errors. Initially facing issues like unreliable deployments and synchronization, the team transitioned from Ansible to a more streamlined process that utilized tmux and later a chatops solution. The implementation resulted in significant growth in deployers, microservices, and servers while also enhancing visibility and collaboration among team members.
Scaling Humans - BigPanda's Fabulous ChatOps Adventure - Erik Zaadi, BigPanda - DevOpsDays Tel Aviv 2016
- 1. SCALING HUMANS: BIGPANDA'S FABULOUS CHATOPS ADVENTURE BY ERIK ZAADI | DEC 13, 2016
- 2. TEAM LEADER AND COMMUNITY GUY ERIKZAADI
- 3. • Continuous Delivery • Deploy by ssh to deploy server (AWS) • Ran Ansible on our playbooks repository A TALE OF DEPLOYMENT EVOLUTION FLASHBACK TO EARLY 2014 8 deployers 5 Microservices 15 Servers
- 4. • Synchronizing - Lack of visibility • Not reliable deploys • Disconnections • Debugging • Ansible was not common knowledge WE SIMPLY WANTED TO DEPLOY PROBLEMS:
- 5. Bash wrapper for Ansible NEXT STEP: LET'S MAKE IT EASY ENTER BASH - DEPLOY.SH DEPLOY.SH IS COMING
- 6. Solved: • No Ansible Knowledge Problems: • Synchronization • Not reliable deploys DEPLOY.SH
- 7. • Wrapped SSH login with TMUX • Used local users with symlinked playbooks repository • Enabled mosh NEXT STEP: LET'S STAY CONNECTED TMUX TO THE RESCUE TMUX IS COMING
- 8. Solved: • Not reliable deploys • Disconnections (session) • Debugging Problems: • Synchronization TMUX AND LOCAL USERS
- 9. • HipChat bot using Hubot • Written in CoffeeScript • Called Beanbot • Limited to staging servers • Used Jenkins to run Ansible • Could query past deploys NEXT STEP: CHATOPS TO THE RESCUE! HELLO THERE HUBOT!
- 10. Problems: • Synchronization • Lack of HipChat usage • Almost no visibility • Didn’t solve the production pain • Only 3 contributors • People really hated Coffeescript BEANBOT
- 11. AND WE CONTINUED GROWING THIS WAS MID 2015 15 deployers 15 Microservices 40 Servers
- 12. Slack channel: #deployment-locks We'd manually notify locking. Escalated quickly to be unusable. Stupid Humans. NEXT STEP: LET'S SOLVE IT IN CHAT SO, CHAT-NOT-OPS?
- 13. • Hackathon rewrite from scratch • Robin (Hubot) • Written in Javascript • Using micro services standards • Tests • Deploys NEXT STEP: CHATOPS TO THE RESCUE AGAIN! FIX THE STUPID HUMANS
- 14. • Managed deployment locks • Deployed to production (!) • Prevented Human Errors • Similar syntax to bash script ZOMG TIS’S DA THINGZ THE HYPE TRAIN TOOK OFF
- 15. • Managed using GitHub issues • 10 contributors (~1/2 RND) • Javascript vs Coffeescript • Slack vs HipChat • Hubot Scripts • Use anything with an API INSTANT POPULARITY GAINED WHY ROBIN WORKED
- 16. • Integrated part of our dev cycle 30 Deployers 30 Microservices 160 Servers ~100 daily deploys and locks TODAY WE CAN’T LIVE WITHOUT ROBIN
- 18. EXAMPLES - KEEP IT SIMPLE AND SAVE HUMAN STEPS
- 19. EXAMPLES - MINIMIZE HUMAN ERRORS AND ADD CONFIRMATION
- 20. EXAMPLES - MINIMIZE HUMAN ERRORS
- 21. EXAMPLES - PROVIDE USEFUL FEEDBACK
- 22. EXAMPLES - BE QUERY-ABLE
- 23. EXAMPLES - ENSURE NAME SPACING CHATOPS COMMANDS
- 24. EXAMPLES - UNIT TEST ALL THE THINGS! CONVERSATION REGEX TBD: SOLARIZED LIGHT?
- 25. EXAMPLES - SOLVE NON VISIBLE OPERATIONS TASKS
- 26. EXAMPLES - DON’T SPAM PUBLIC CHANNELS TOO MUCH
- 27. EXAMPLES - HELP WITH DAY TO DAY TASKS
- 28. EXAMPLES - MAKE THE USAGE OF THE BOT FUN
- 29. • Ensure deploy can work without ChatOps as well • Reuse existing deploy process and syntax • Run the bot both locally and on a staging env • Make contributing to the bot fun and easy • Company bot skeletons (allowing quick POCs) THINGS WE LEARNED THE HARD WAY YOUR BOT WILL BE DOWN
- 30. • SuperPanda - Groceries wish list • Gordon - Sales and Support • Bellboy - Opens our office door • Splinter - Code linting rewarder DIVIDED THEY CONQUER MOAARR BOTS!!!1
- 31. EOF