Incident response orchestration
Download as PPTX, PDF1 like556 views
The document discusses the importance of incident response orchestration in managing failures of always-on services, emphasizing that effective response minimizes business impact. Key components include incident detection, assessment, collaboration among responders, and post-incident analysis to improve future performance. It highlights the necessity of preparation, reliable notifications, and clear communication with stakeholders to ensure efficient incident resolution.
Incident response orchestration
- 1. Incident Response Orchestration October 26, 2017 | Berkay Mollamustafaoglu
- 2. The rise of the always-on services
- 3. Failures may be inevitable... “We understand that we are operating large, complex systems and build in redundancy and resiliency to maximize uptime, we must also understand that failures will still occur.” Incident management for Operations Rob Schnepp, Ron Vidal & Chris Hawley
- 4. It’s how we respond that matters Incident Response is about how we respond to those failures, it's about minimizing the likelihood that failures become business impacting outages.
- 5. Journey from alerting, on-call management & escalations to Incident Response
- 6. What happens during an incident? ● Detection ● Assessment ● Dispatching the responders ● Collaboration during the incident ● Stakeholder communications ● Post incident analysis
- 7. Assessing the priority ● Impact & urgency determines the priority ● There may be multiple dimensions for classification but keep it simple ● There may be multiple dimensions for classification but keep it simple
- 8. Dispatching the responders One or more teams can be notified as the responders (as well as individuals)
- 9. Collaboration during the incident ● Chat ● Conference calls ● Web conference
- 10. Stakeholders communications ● Stakeholders need different information ● Provide a simple way to check status of services and incident updates ● Enable users to control which services they are interested in
- 11. Post incident analysis ● Time to detect ● Time to engage ○ First responder ○ Each responder team ● Time to resolve ● How many people were involved? ● How many man hours were spent? ● How many alerts are received?
- 12. We race against time! Preparation is the key! Effective communications & collaboration is essential.
- 13. Heroism is not sustainable Don’t expect you people to compensate for systemic problems forever Don’t become the crutch for systemic flaws
- 14. What can we do to minimize MTTR? MTTR (Mean Time to Repair) Time to detect Time to respond Time to resolve
- 15. Reliable notifications is essential for rapid response Use multiple notification methods for reliability. Don’t rely on the your own infrastructure for alerting. How to communicate with: ● The first responder ● Best people to resolve the incident ● The stakeholders ● Customers/users
- 16. Effective communications with customers & users is crucial DEVELOPERS OPERATIONS PM MANAGEMENT CUSTOMER SERVICE INTERNAL CUSTOMERS SALES USERS CUSTOMERS
- 17. Easy access to all supporting data accelerates resolution Identify the relevant contextual data in advance: ● Alerts ● Logs ● Configuration changes ● Metrics ● Dependencies Make it easy to access or better yet gather the data in advance.
- 18. Automate or at least document incident resolution ● Investigative steps ● Remedial steps ● Runbooks
- 19. Measure, track and report for continuous improvement Metrics ● Detection time ● First response time ● Diagnosis time ● Resolution time ● Number of alerts ● Number of people involved Update response procedures ● Classification, alerting, escalation policies ● Runbooks ● Communications
Editor's Notes
- #3: There will be increasingly more applications and services that will be required to be “always-on” Since your consumers are always digitally connected your business approach should change. You have to provide them with always-on services, especially in the fastly growing IT market full of innovative solutions. ITIL or ITSM are not enough any more and Composable Incident Service Management results more actual with its broad possibilities, with its customer-centric approach.
- #15: Time is precious, and fast and effective incident resolution is vital when you are running an IT business. Every millisecond counts and end user experience shows that timing is one of the most important aspects.
- #17: Nowadays,the people outside of your company, such as your users or customers, your partners, or individuals who may potentially be impacted by an event such as a critical system outage or any anomalous event happening to any of the components of their IT infrastructure want to stay in the know. To ensure better communication and transparency between you and them you have to notify them via any preferable communication mean.
- #21: No long time ago all the day-to-day IT infrastructure management including the provisioning, capacity, performance and availability of the computing, networking and application environment was managed by Operation Team. IT Operations is responsible for the smooth functioning of the infrastructure and operational environments to internal and external customers, including the network infrastructure; server and device management; computer operations;ITIL management etc.