How to Create a Runbook: A Guide for Sysadmins & MSPs
- 1. 1/12 October 15, 2021 How to Create a Runbook: A Guide for Sysadmins & MSPs process.st/create-a-runbook Benjamin Brandall October 15, 2021 Business Operations, IT How do you name a new server, export config data, or fix that one really annoying bug that keeps popping up every 2nd Thursday? For prepared IT professionals, that information is stored in a runbook. A runbook is a set of standardized documents, references and procedures that explain common recurring IT tasks. Instead of figuring out the same problem time and time again, you can refer to your runbook for an optimal way to get the work done. What’s more, you can also delegate tasks and onboard employees more effectively if you have documentation to train them with. Whenever you do a task, think of this quote:
- 2. 2/12 “Will you remember how to do these things 6 months from now? I find myself having to re- invent a process from scratch if I haven’t done it in a few months (or sometimes just a few days!). Not only do I reinvent the process, I repeat all my old mistakes and learn from them again. What a waste of time.” — Tom Limoncelli, The Operations Report Card In short, the less time wasted figuring out how to do a task, the better it’ll be for your business efficiency, productivity, and sanity. This post will look at: First, let’s look at two example runbooks to give context on what I’m going to talk about. The 2 types of runbook Runbook is a broad term. It can refer to two kinds of documentation: General documentation: Updated by a sysadmin when new procedures arise or evolve. Specialized documentation: Written for one team, one use-case, or one system. With this in mind, it’s likely that any one IT department or sysadmin will have multiple runbooks that assist with their routine tasks and act as reference manuals for special cases. For example, to build a general daily tasks runbook inside Process Street you could use this template as a starting point: Runbook documentation is precise. It’s specific to the systems your company is running, and the custom configurations you’ve made. While the process above can’t cater to your systems and existing procedures, it can act as a demonstration or template you can fill in with your common daily tasks. As for specialized documentation: this varies drastically for each organization. The below example — Installation Runbook for Palo Alto Networks virtual Firewall and Juniper Contrail plugin for Fuel — contains diagrams, tables, and procedures that will guide the user through every step and contingency.
- 3. 3/12 Aside from system-specific documentation, most organizations will prepare use-case specific documentation. The most operationally vital use-case for documentation in IT will always be disaster recovery, which needs to be executed quickly and thoroughly. Xtium has released a 33-page disaster recovery runbook template that runs you through example procedures and recommendations for creating and updating your disaster recovery procedures. A disaster recovery process is something many businesses only make when it’s too late. The top three causes of downtime are power outages, hardware failures, and network failures. How do you respond to these incidents in each case?
- 4. 4/12 For a cloud-based option, you can edit our information security incident response template, as an example of one kind of disaster recovery: How to create a runbook Every runbook is unique and specific. The actual content will be specific to your needs. The methodology, however, is the same as it is for all process creation. The first stage is to plan which procedures need to be documented in your runbook. When you have a list, you can write them up in detail. After field-testing a process, you can make updates and optimizations as necessary. Author and ex-Google sysadmin Tom Limoncelli recommends that each runbook contains 7 sections: 1. Overview: Overview of the service: what is it, why do we have it, who are the primary contacts, how to report bugs, links to design docs, and other relevant information. 2. Build: How to build the software that makes the service. Where to download it from, where the source code repository is, steps for building and making a package, or other distribution mechanisms. If it is software that you modify in any way (open source project you contribute to or a local project) include instructions for how a new developer gets started. Ideally, the result is a package that can be copied to other machines for installation. 3. Deploy: How to deploy the software. How to build a server from scratch: RAM/disk requirements, OS version and configuration, what packages to install, and so on. If this is automated with a configuration management tool like cfengine/puppet/chef (and it should be), then say so. 4. Common Tasks: Step-by-step instructions for common things like provisioning (add/change/delete), common problems and their solutions, and so on. 5. Pager Playbook: A list of every alert your monitoring system may generate for this service and a step-by-step “what do to when…” for each of them. 6. DR: Disaster Recovery Plans and procedure. If a service machine died how would you fail-over to the hot/cold spare? 7. SLA: Service Level Agreement. The (social or real) contract you make with your customers. Typically things like Uptime Goal (how many 9s), RPO (Recovery Point Objective), and RTO (Recovery Time Objective). The way that’s structured is up to you, but best practice is to maintain core policy and procedure documents with a knowledge management tool like Process Street’s Pages, and then use a workflow app to execute the recurring processes associated with your day-to-day. 1. Planning Which procedures do you regularly execute? Some ideas:
- 5. 5/12 Disaster recovery Data backup New user setup Virtual machine troubleshooting Patch management A runbook should contain your most regularly executed tasks, as well as instructions for complex semi-regular issues with easily forgotten details. Our CTO, Cameron, uses Process Street to fire an SSL renewal checklist once every three years (because it’s an archaic and fiddly task), but he also runs daily and weekly checklists for system maintenance. 2. Writing It’s sustainable and scalable to write runbook documentation in language that anyone can understand because later down the line, staff of every level need to be able to understand it for training and onboarding purposes. This means cutting jargon, avoiding abbreviations, and making no assumptions about the skill level of the user. To create a process, it’s best to actually do the task and record your work. You can do this by taking notes of every button you click, and all of the information you enter, but you could also just record your screen and take notes afterwards. That way, you get a realistic step-by-step procedure that can be replicated by non-technical staff, too. To summarize: 1. Choose a process to document (it should be a common cause for error, or a critical regular task) 2. Start recording your screen 3. Work through the task as you normally would 4. Pause the screenshot at each action and note down everything you do in your Pages document (“Open the Start menu. All programs > Accessories > Paint”) 5. For actions that need more explanation, take a screenshot of the screencast and add it to the Process Street Page to use in the runbook 6. Repeat until the end of the screencast Make sure to pull together any resources you used while doing the task: Reference documentation Network diagrams Login credentials Config information
- 6. 6/12 Record your common runbook tasks in an active Process Street workflow. Here you’ll have everything you need to create rich-media processes that include images, videos, sub- checklists, code snippets, and more. You can easily update your processes and have the changes reflected for all users. Documenting runbook procedures in a Process Street workflow allows you to track process instances as they happen, creating active documentation. In addition to this active documentation, you’ll also need to store fixed organizational knowledge, SOPs, and best practices in your runbook for your team. You can do that using Process Street’s Pages. This way you’ll have a procedure workflow plus living documentation to share with your team. You can add a shareable link for your Process Street page to your workflows, and vice-versa, connecting the two. Keep reading to find out how you can use Process Street workflows plus Pages to create your runbook. 3. Improving It’s one thing to document a process for yourself, but having that process executed by someone else in a real-world situation is a different story. The first time your process is battle-tested, you’ll notice any errors you’ve made by seeing the efficiency and accuracy of the work done. A good way to improve processes is to be present when the instructions are being carried out, then hand the process off to be updated and used by more general staff. Process consultant Ian James says that improving a process after it’s documented includes 11 steps: 1. Get everyone on board. When you’re making changes to the generally accepted way of doing work, it’s likely you’re going to need approval from higher-ups, and time from everybody involved. 2. Choose the right process to optimize. Which process is causing pains, needs to be handed off, or is too out of date to be useful? 3. Calculate the time and resources you’ll need. A pitch to upper management will need backing up with data. 4. Act when the time is right. Process improvement is necessary, but often that’s only apparent in a crisis. 5. Set expectations for everyone involved. If you’ll need 5 hours per week of a certain team’s time, and expect the project to take three weeks, make it clear. 6. Offer process training. Thinking critically about business systems is a learned skill. If your team doesn’t have training, prepare by getting them some. 7. Build the process out in a workflow application. Paper or Word documents will eventually fall flat. Paper forms can get lost, and collaboration is impossible in Word.
- 7. 7/12 8. Know why you’re improving the process. Is it because it’s done frequently, has a high margin for variance, or because it needs to be predictable? Categorize the process by these criteria. 9. Select your team size. Choose a small number of trained individuals to help. 10. Pick the right team members. The only people who should be involved with optimizing a process are the people who run it. 11. Get together in a dedicated room. Ian James reports that projects where the team collaborates in a physical space have a 200% higher success rate. We’ve also put together a process specifically for process improvement: Running your IT processes with Process Street Process Street is a workflow documentation platform that lets you document your processes and then track progress each time the process is run. Insert form fields, automation, and supporting files into your process documents, and then assign them to members of your team. Process Street’s folder structure lends itself well to IT runbooks, and we’ve also got plenty of pre-made processes for IT teams, and integration with ITGlue. Structuring your IT processes Process Street has folders, sub-folders, and tags. This gives you a lot of control over the way you structure processes in your organization. For example, an organization for multiple departments might have different folders on their home screen for each department. In the gif below, I go through the layers of an example structure for IT processes, with the home screen divided up by department and then the IT folder divided into folders such as hardware setup, maintenance, security, etc.
- 8. 8/12 You can create new folders with the green ‘New’ button on the left sidebar. Go inside your IT folder to create subfolders: You can use the same ‘New’ button on the sidebar to add a template to the folder. For example, you might want to use one of our VPN processes:
- 9. 9/12 It makes sense to organize files into their proper folders, but you can also use folders to control access management. Read more about folders in our help documentation. With Process Street, you can use and optimize your processes in one central place. This even means you can integrate with tools like ITGlue. Integrating Process Street with ITGlue ITGlue lets you set up notifications when key events happen on your systems that need dealing with: SSL renewal, licenses, maintenance, audits, etc. These are linked to your ITGlue documentation, and live alongside the rest of your key IT knowledge such as network structure and inventory.
- 10. 10/12 Process Street can be integrated into this notification system to act as a runbook. For example, whenever your SSL certificate is running out, ITGlue will notify Process Street which will then run a checklist from our SSL renewal checklist and email the person in charge of getting the job done. Read more about setting up this integration here.
- 11. 11/12 Transfer workflow information into Pages Capture operational knowledge and SOPs for your runbook using Pages. With Process Street’s Pages, you can create living documents to share with your team. Pages helps you capture vital operational knowledge to support your recurring, actionable workflows, and can be used to springboard and power-up your workflow runs by providing a clear operational overview of key processes and helping to you ensure that processes are maintained. To create a new page for your runbook, click on the New button in the top left of the dashboard then click on Page. Start by filling in the title of your new page, which you need to do before the new page can be published.
- 12. 12/12 For more information on how to get started with Pages, watch this video: Get started with your runbook Runbooks make it easier for you to remember the ideal methods you come up with, and to delegate work to a team. You can manage your IT documentation with Process Street, and use documentation templates created specifically for IT.