A culture of Automation - Joe Smith - DevOpsDays Tel Aviv 2017

Editor's Notes

• #4: I do believe that folks in these roles all have different approaches, and that is excellent. There is a focus of continuity with Operations, and deep architectural expertise with Site Reliability Engineering. I’d love to talk to you all afterward to hear your approaches to building out teams!
• #5: We're going to break this talk into 3 pieces, and I would love to talk about any of them for hours But we only have time for one talk this year, so let's see what we'd like to focus on Who here considers them an experienced Operations Engineer or SRE? Who has written at least a few hundred lines of Python? Who has published your own python distribution to Cheeseshop (aka PyPI?)
• #6: This is going to set the stage for priorities and approaches based on years running large web services. The focus is primarily around working with a larger group of people— a team that has gone beyond the “two pizza rule”
• #7: Once you go beyond this number of engineers, your problems begin to center around organization and collaboration instead of technology (usually!)
• #8: There are a few universal best practices which we can use to inform how we structure our work.
• #9: Different approaches to doing this For this talk, we're going to consider three pillars of a great operations team These are aspects of rolling out a new tool, procedure, service, or automation
• #10: ^ Consider strategies for dealing with mistakes vs. just "do a rollback" ^ Is everyone familiar with Rollback strategies? ^ Whenever you make a change in production, you should consider how are you going to rollback if something breaks! ^ DO NOT come up with your rollback strategy under pressure after things break! ^ Use the _feature-flag_ approach: Hide chef/puppet changes behind a per-node attribute ^ Deploy processes should be the same for a roll-forward as a roll-back ^ Specify which version number should be running and structure the config to match
• #11: These written pieces of text will be generally of two types: Design Documentation describing architecture and function. The second, which we will dig into, are called “Runbooks” These are descriptions about how to run a service. They contain important information about how to spin the service up from scratch. How to repair components when they fail. And most importantly, a corresponding runbook for each page/alert that goes to an engineer.
• #12: Documentation is written in the past, and consumed in the present. “Communication” is real-time: this means information is conveyed at the same time it is received.
• #13: Any questions about bringing these together? So these are the ideal, but unfortunately it isn't always the case! Let’s talk about corresponding issues we can run into
• #14: ^ These all get more difficult over time ^ More people, more difficult to communicate
• #16: Here’s where we get to a really fun part of the talk I’ve been on a team where our runbook was one really long wiki page, and now a team with much more structured information. I’m biased, but I think this approach is much better! A little effort has helped make oncall and remediation much easier. Even if you don’t follow anything else I suggest here— make time explicitly for your team to improve documentation!
• #17: First, consider whether what you're writing is really a runbook. If you're documenting an architecture or a workflow that's expected to become muscle memory (like how Logstash works or our Chef workflow) then your file belongs in the ops directory. Runbooks are for reapeatable processes. We are all required to refer to the runbook each and every time we execute a process documented in one so that changes to runbooks will reliably take effect.
• #18: We’re going to talk about three different components to good runbooks.
• #21: Runbooks should be organized into directories in the runbooks directory named descriptively and broadly. For example, message_server as a top level directory, with descriptive files within. Within those files there should be a first-level heading reiterating the name of the file (some_alert_name.md becomes "# Some Alert Name") and many second-level headings, following the template laid out in the example directory.
• #22: There should also be a README.md file with a basic explanation of the service, related services, contacts/channels for that service, and related links/graphs. Now remember, when someone realizes something is wrong, this will likely be the first place they go to get the lay on the land. This should be generally helpful information.
• #23: Each process documented should present, as prose, any context that may be required to decide if use of this runbook is appropriate or otherwise set the stage for execution. Then an ordered list should be presented. Use "1. " as the prefix for every element in the list to keep diffs tidier; when the Markdown is rendered the numbers will be sequential. Finally, any parting considerations should be presented, again as prose.
• #24: It is very important that Markdown be used precisely when writing runbooks. In the heat of the moment, consistent formatting adds clarity and helps us move quickly and confidently.
• #25: Specify where a command is to be run before specifying the command ("From staging:", "From your chef-repo:", etc.) Link to graphs, God pages, and other specific resources required during execution of the runbook Include actions like updating the status site, sending particular messages in Slack, deploying the site, enabling notifications, and so on lest they be forgotten or their omission confuse the operator executing your runbook
• #27: This is an example of something that gives some context— it clearly points the reader to the next services to investigate. Following along with what Corey said this morning... It could be improved in a few ways.. Most notably by helpfully linking to runbooks about how to diagnose if memcache, databases, and other downstreams are healthy or not! This is something that an experienced team member can read and have no issues with. Someone new on the team may need to burn time getting help from a teammate before isolating the root cause.
• #28: Ensure you are creating something that can be acted upon by a teammate at 2am. Otherwise expect to get woken up to chip in!
• #29: So here’s my favorite part! To start we’re going to talk about one of my favorite maxims, followed by an ancient proverb.
• #31: This is the best rule of thumb we have for ensuring we are directing efforts appropriately For something that is so standard that you know you can handle it, and have seen the edge cases, it’s going to be boring Don’t let yourself or your team stay working on the same issue for too long. This is the perfect opportunity to put in some extra effort to make sure it is automated.
• #32: Automatically detect which procedures to follow As we saw, that example runbook was helpful, but sort of passed the buck when it came to further troubleshooting. A potential tool might automatically query health metrics for downstream services to help identify any problems Let’s walk through what that procedure might look like.
• #36: Once you have defined the logic for a process, it can be written out in a language for the future ^ That language could also happen to be understood by a machine! Humans can also read code, but machines can execute explicit instructions.
• #37: ^ We have found that investing the time to write tools as we understand a process has been worth the effort ^ We are able to work on much more interesting problems, and not repeatedly wake up for the same issues again and again
• #38: This can be a large amount of work, and the payoff isn’t always immediate. It can be slow-going to write something which is resilient enough to be run by someone not familiar with both the process and the tool. The payoff will be well-worth it however!
• #39: I’m hopeful I’ve motivated you to not only invest in your runbooks, but also looking into improvements beyond just text. Using machines to do some work for us will allow our brains to focus on bigger and more impactful challenges.
• #40: If you agree, disagree, or have any questions, I’d love to hear it! Thank you!