What To Do When It All Goes So Wrong
Download as PPTX, PDF0 likes758 views
The document outlines a structured approach to managing IT emergencies, particularly during system failures, illustrated by a case involving a sales center issue. Key steps include prompt communication, data collection on impacted systems, the use of a 'war room' for collaboration, and systematic troubleshooting to identify and resolve the problem. The process emphasizes teamwork, documentation, and the importance of minimizing disruption while identifying the root cause of the issues.
What To Do When It All Goes So Wrong
- 1. What To Do When It All Goes So WrongDavid LevyAdventuresInSql.comSQL Saturday #67 Chicago
- 2. More than 11 years in ITSQL Server DBA for over 3 yearsPrevious Life as DeveloperBloggerhttp://adventuresinsql.comSyndicated on SQLServerCentral.comSyndicated on SQLServerPedia.com@dave_levy on TwitterAbout Me
- 3. Peak Time of Peak Sales DayTypical Hourly Sales $100K/HROrder Entry Screen is Locked UpUsers report Slowness InitiallyNow the “Sales Center” Application is Just “Clocking”EMERGENCY!
- 4. Let Everyone Know There is a ProblemPrevent Duplicated EffortsAllows Others to Speak UpRecent ChangesRelated IssuesCommunicatehttp://www.freedigitalphotos.net/images/view_photog.php?photogid=1983
- 5. Send Up a FlareSend to an IT Only Distribution GroupKeep the Subject Line GeneralProvide Broad Overview Including:Systems ImpactedMajor Symptoms Including Error MessagesNumber of People ImpactedAny Location Specific InformationCommunicate
- 6. What Resources Do You need?Subject Matter ExpertsSpecialized EquipmentCommunicate
- 7. Never Assign BlameOnly State FactsCommunicate
- 8. To: IT EmergenciesSubject: Sales Center IssuesSales Center Users are reporting that the Order Entry screen has quit responding. We are currently investigating the issue with the Sales Center Development Team. We will provide updates as we know more.Communicate
- 10. What Are the Symptoms?What Locations are Involved?Collect
- 11. What Systems are Involved?SQL ServerAS400MainframeWeb FarmMajor Network Components like Load BalancersCollect
- 12. What Has Changed?Look at Change Control CalendarTalk to Primary On-Calls for Related SystemsCollect
- 13. Anything in the Logs?Windows LogsApplication Specific LogsCustom Exception Handling SystemsCollect
- 14. What are Performance Indicators Showing?PerfmonSQL Wait StatsThird-party toolsCollect
- 15. Analyze Collected InformationAre There Any Obvious Signs of Trouble?Can the Problem be Linked to a Change?Can Any Patterns be Identified?Process
- 16. Prove It Is Your IssueShows HumilityShows Respect for Everyone Else’s TimeAvoid Appearing ArrogantProcess
- 17. Prove It Is Your IssueConstruct Tests to Prove Theories in Order of Likelihood Until Problem Proven or Theories ExhaustedFaster than arguing about what it is notHow can you know it is not your issue?Process
- 18. List Potential ActionsRank by effort, confidence, level of riskDevelop action plans for best options and re-rankEach potential action should have a rollback planProcess
- 19. Define MeasuresWhat will indicate things have gotten better?Adding this index will reduce Disk IO by 10 million reads per secondThe execution time of query x will drop from 6 minutes to 50 millisecondsProcess
- 20. Define MeasuresWhat will indicate things have gotten worse?Disk IO may go upThe execution time of query x may go upAdding this index may slow inserts from the order upload processProcess
- 21. Communicate Your IntentionsMake the ChangeFollow a written planMake a single changeA single person should make the changeDocument any additional steps takenStart Over by Collecting More DataRespond
- 22. Signs You Need to Convene A War RoomHaving Trouble Finding Anything Wrong30 Minutes Without ProgressAn Issue Appears to Span Multiple SystemsHaving Difficulty Getting People EngagedThe War Room
- 23. Get Everyone in a RoomNo Changes Made Outside the RoomNo HeroesWatch out for people doing a lot of typingAvoid changes that take more than a few minutesHave a Call in Number for Remote CoworkersThe War Room
- 24. Have a Technology KitOld SwitchPatch CordsMice + Mouse PadsPower StripsThe War Room
- 25. Monitor Your Guest List1-2 Representatives From Each TeamTry to Keep Management OutWatch for Disruptive PeopleThe War Room
- 26. To: IT EmergenciesSubject: Sales Center IssuesWe are convening a war room for the Sales Center issue. Everyone working on the issue please meet in the North Conference Room. Remote/WFH coworkers should dial into the conference bridge 888-888-1234, participant code:1234.Communicate
- 28. White Board the IssueEvery System Gets Own ColumnWrite All Facts on White BoardClosed Items Get Crossed Out Not ErasedInclude a Resolution for Each Closed ItemThe War Room
- 29. Share the FloorLikely Issue Owner Has the LeadMake Sure Everyone is HeardContributing Often Involves Staying Out of the WayDon’t Be Afraid to Fade Back and Run The WhiteboardThe War Room
- 30. Never Call “Not-It” and LeaveNot HelpfulYou May be WrongAppears ArrogantThe War Room
- 31. Keep an Eye On TimeProvide Regular Updates to ManagementBring in Food Around Meal TimesRaises SpiritsBrings in More People to HelpThe War Room
- 32. To: IT EmergenciesSubject: Sales Center Issues UpdateThe Sales Center war room is still going. We are currently looking into a driver issue with IBM. All necessary resources have been engaged.Communicate
- 33. Keep People in ReserveEach Team Should Divide up the DayRotate People In and OutSend Someone Home Early to Come in EarlyThe War Room
- 34. Closing OutCommunicate ResolutionCapture Contents of WhiteboardClean Up RoomThe War Room
- 35. To: IT EmergenciesSubject: Sales Center Issues ResolvedThe Sales Center issue has been resolved. The issue was caused by a patch that was applied over the weekend. Now that it has been backed out everything has returned to normal.Communicate
- 36. ?Questions?