A Solution for Identifying the Root Cause of Problems in IT Change Management
1 like732 views
The document proposes a solution for identifying the root cause of problems in IT change management. It involves developing an interactive root cause analysis method that considers information from previous diagnoses and the affected IT infrastructure. The proposed approach includes a diagnosis process, an information model, a conceptual architecture, and a root cause analyzer component. The root cause analyzer identifies potential root causes and questions based on weights and selects the question with the highest weight to iteratively diagnose the problem. The goal is to optimize root cause identification and enable knowledge reuse from past diagnoses.
A Solution for Identifying the Root Cause of Problems in IT Change Management
- 1. A Solution for Identifying the Root Cause of Problems in IT Change Management Ricardo L. dos Santos, Juliano A. Wickboldt, Roben C. Lunardi, Bruno L. Dalmazo, Lisandro Z. Granville and Luciano P. Gaspary Federal University of Rio Grande do Sul, Brazil Claudio Bartolini Hewlett Packard Laboratories, Palo Alto, USA Marianne Hickey Hewlett Packard Laboratories, Bristol, UK © 2009 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice Labs & Institute of Informatics – UFRGS Malia Project – HP
- 2. Outline • Introduction • Related Work • Proposed Solution • Diagnosis Process • Information Model • Conceptual Architecture • Root Cause Analyzer • Conclusion and Future Work Malia Project – HP Labs & Institute of Informatics – UFRGS 2 Wednesday, March 06, 2013
- 3. Introduction • Context • The complexity of IT infrastructure makes the IT processes a critical mission • ITIL has become important for IT processes • IT Change Management • Defines how the IT infrastructure must evolve in a consistent and safe fashion • Defines how changes should be conducted Malia Project – HP Labs & Institute of Informatics – UFRGS 3 Wednesday, March 06, 2013
- 4. Introduction • IT Problem Management • Defines how lifecycle of IT problems is managed • Its main goals are • Preventing the occurrence of IT related problems • Eliminating recurring problems • Minimizing the impact of problems that cannot be prevented • Allows to simplify procedures for detecting the root cause of IT problems and reducing the associated costs Malia Project – HP Labs & Institute of Informatics – UFRGS 4 Wednesday, March 06, 2013
- 5. Introduction • Current Scenario • Organizations use tools for to help in analysis and identification of the root cause of failures • When they do not point directly into the root cause, is able to provide important information to the responsible technical staff • Enables the reuse of knowledge, which results in a reduction of associated costs • However, these tools have serious limitations, such as • The description of cases in diagnostic tools is performed in ad hoc fashion • Poor reusability of cases • Low maintainability Malia Project – HP Labs & Institute of Informatics – UFRGS 5 Wednesday, March 06, 2013
- 6. Introduction • Problems • Often diagnoses are not completed correctly due to outdated cases • The complexity of IT infrastructure where the failure occurs is ignored • Data of previous diagnoses are not considered Malia Project – HP Labs & Institute of Informatics – UFRGS 6 Wednesday, March 06, 2013
- 7. Introduction • Objectives • Propose an interactive method for identifying causes of failure in change processes • Based on information of the diagnosis of previous failures in change activities • Considering the affected IT infrastructure • Optimize the root cause identification process of failures in IT changes • Reuse the knowledge acquired in previous diagnoses Malia Project – HP Labs & Institute of Informatics – UFRGS 7 Wednesday, March 06, 2013
- 8. Proposed Solution Diagnosis Process Help Desk Root Cause Operator Analyzer IR RC Iterative Diagnosis Incident Report Question Answered Root Cause Selection Question Malia Project – HP Labs & Institute of Informatics – UFRGS 9 Wednesday, March 06, 2013
- 9. Proposed Solution Conceptual Architecture Change Change Deployment CI CI Designer Planner System CI Change Management System Root Cause Analyzer Config. Mgmt. RC Database Diagnosis Log Operator Recorder Diagnosis System CI L CI CI CI CI Input Weight Question Question IR RC Processor Calculator Selector Verifier RC RC RC Root Cause Analyzer Malia Project – HP Labs & Institute of Informatics – UFRGS 10 Wednesday, March 06, 2013
- 10. Proposed Solution Information Model ManagedElement ManagedSystem ExchangeElement Element SolutionCategory PRS_ServiceInciden Logical Element SolutionElement Category 1..* * t * * 1 1..* CategoryParentChild 0..1 1 QuestionCategory EnabledLogical PRS_Problem Element 0..1 RecordedProblem 0..1 0..1 RootCause 1 ▲ ServiceProblem MessageLog determinesProblem determinesOthersQuestions 1..* 1..* * 0..* 1..* possiblesAnswers 1 RecordLog Answer Question 1 1 1 1 RecordedAnswers RecordedQuestions Malia Project – HP Labs & Institute of Informatics – UFRGS 11 Wednesday, March 06, 2013
- 11. Proposed Solution Root Cause Analyzer Operator Reports the incident Log Log CI Log CICI CI Priority … Malia Project – HP Labs & Institute of Informatics – UFRGS 14
- 12. Proposed Solution Root Cause Analyzer Category that Identif ication Calculates the has the Obvious? based on IR weight of RCs greatest weight RC Calculates the RC RC Identif ication weight of Question that based on has the 80% with the Questions and categories greatest weight same answer Categories Calculates the Threshold weight of CI L CI Identif ication based on RCs possible Question with higher level Answers Input Processor Weight Calculator Question Selector Question Verifier Root Cause Analyzer Malia Project – HP Labs & Institute of Informatics – UFRGS 15 Wednesday, March 06, 2013
- 13. Proposed Solution Root Cause Analyzer Operator Log Log CI Log CICI IDENTIFIES: •Logs of previous diagnoses •RCs, Questions and Answers (categories) •Affected CIs Malia Project – HP Labs & Institute of Informatics – UFRGS 16
- 14. Proposed Solution Root Cause Analyzer • An example of the affected IT infrastructure Malia Project – HP Labs & Institute of Informatics – UFRGS
- 15. Proposed Solution Root Cause Analyzer Log Log CI Log CICI CI Category CI Category Internet Third Party Services WEB Server (215) System Computer System Server Web Server DNS Server (423) System Computer System Server DNS Server Apache 2.2 Software Web Server PHP 5.0.2 Software Programming Language Hard Disk Device Storage Devices Network Card Device Network Cards Malia Project – HP Labs & Institute of Informatics – UFRGS 18
- 16. Proposed Solution Root Cause Analyzer Category that Identif ication Calculates the has the Obvious? based on IR weight of RCs greatest weight RC Calculates the RC RC Identif ication weight of Question that based on has the 80% with the Questions and categories greatest weight same answer Categories Calculates the Threshold weight of CI L CI Identif ication based on RCs possible Question with higher level Answers Input Processor Weight Calculator Question Selector Question Verifier Root Cause Analyzer Malia Project – HP Labs & Institute of Informatics – UFRGS 19 Wednesday, March 06, 2013
- 17. Proposed Solution Root Cause Analyzer Log Log Log CI CICI Select questions and answers based on the Set of RCs, Answers, and Questions categories of CIs identified Configuration Mgmt. System RC RC RC RC RC Malia Project – HP Labs & Institute of Informatics – UFRGS 20
- 18. Proposed Solution Root Cause Analyzer RC RC RC System Computer System Server Web Server RC RC RC RC RC RC Application Web Server RC RC RC RC RC RC Device Network Cards RC RC RC RC RC RC Third Party Services RC Application System Device 74 95 35 Computer System Network Card Web Server 30 30 70 Server Third Party Services Web Server 5 20 Malia Project – HP Labs & Institute of Informatics – UFRGS 21
- 19. Proposed Solution Root Cause Analyzer Software System Device Computer System Network Card Web Server Server Third Party Services Web Server Malia Project – HP Labs & Institute of Informatics – UFRGS 22
- 20. Proposed Solution Root Cause Analyzer Category that Identif ication Calculates the has the Obvious? based on IR weight of RCs greatest weight RC Calculates the RC RC Identif ication weight of Question that based on has the 80% with the Questions and categories greatest weight same answer Categories Calculates the Threshold weight of CI L CI Identif ication based on RCs possible Question with higher level Answers Input Processor Weight Calculator Question Selector Question Verifier Root Cause Analyzer Malia Project – HP Labs & Institute of Informatics – UFRGS 23 Wednesday, March 06, 2013
- 21. Solução Proposta Root Cause Analyzer Software System Device 74 95 35 Computer System Network Card Web Server 30 70 30 Server 26 Third Party Services Web Server 5 20 Malia Project – HP Labs & Institute of Informatics – UFRGS 24 6 March, 2013
- 22. Proposed Solution Root Cause Analyzer Category that Identif ication Calculates the has the Obvious? based on IR weight of RCs greatest weight RC Calculates the RC RC Identif ication weight of Question that based on has the 80% with the Questions and categories greatest weight same answer Categories Calculates the Threshold weight of CI L CI Identif ication based on RCs possible Question with higher level Answers Input Processor Weight Calculator Question Selector Question Verifier Root Cause Analyzer Malia Project – HP Labs & Institute of Informatics – UFRGS 25 Wednesday, March 06, 2013
- 23. Example Selection of Questions Current Version Application System Device 74 95 35 Computer System Network Card Web Server 30 70 30 Server 26 Third Party Services Web Server 5 20 Malia Project – HP Labs & Institute of Informatics – UFRGS 26 6 March, 2013
- 24. Example Selection of Questions Current Version Software System Device 74 95 35 Computer System Network Card Web Server 30 70 30 Server 26 Third Party Services Web Server 5 20 Malia Project – HP Labs & Institute of Informatics – UFRGS 27 6 March, 2013
- 25. Proposed Solution Root Cause Analyzer Category that Identif ication Calculates the has the Obvious? based on IR weight of RCs greatest weight RC Calculates the RC RC Identif ication weight of Question that based on has the 80% with the Questions and categories greatest weight same answer Categories Calculates the Threshold weight of CI L CI Identif ication based on RCs possible Question with higher level Answers Input Processor Weight Calculator Question Selector Question Verifier Root Cause Analyzer Malia Project – HP Labs & Institute of Informatics – UFRGS 28 Wednesday, March 06, 2013
- 26. Example Selection of Questions Current Version Application Questions RC 1 (10) System Device 74 95 35 Questions RC 2 (20) Computer System Questions RC 3 (40) Network Card Web Server Questions RC 4 (25) 30 70 30 Questions RC 1,3 (50) Server 26 Third Party Services Web Server 5 20 Malia Project – HP Labs & Institute of Informatics – UFRGS 29 6 March, 2013
- 27. Proposed Solution Root Cause Analyzer Category that Identif ication Calculates the has the Obvious? based on IR weight of RCs greatest weight RC Calculates the RC RC Identif ication Question that weight of based on has the 80% with the Questions and categories greatest weight same answer Categories Calculates the Threshold weight of CI L CI Identif ication based on RCs possible Question with higher level Answers Input Processor Weight Calculator Question Selector Question Verifier Root Cause Analyzer Malia Project – HP Labs & Institute of Informatics – UFRGS 30 Wednesday, March 06, 2013
- 28. Example Selection of Questions Current Version Software Questions RC 1 (10) System Device 74 95 35 Questions RC 2 (20) Computer System Questions RC 3 (40) Network Card Web Server Questions RC 4 (25) 30 70 30 Questions RC 1,3 (50) Server 26 Third Party Services Web Server 5 20 Malia Project – HP Labs & Institute of Informatics – UFRGS 31 6 March, 2013
- 29. Proposed Solution Root Cause Analyzer Category that Identif ication Calculates the has the Obvious? based on IR weight of RCs greatest weight RC Calculates the RC RC Identif ication Question that weight of based on has the 80% with the Questions and categories greatest weight same answer Categories Calculates the Threshold weight of CI L CI Identif ication based on RCs possible Question with higher level Answers Input Processor Weight Calculator Question Selector Question Verifier Root Cause Analyzer Malia Project – HP Labs & Institute of Informatics – UFRGS 32 Wednesday, March 06, 2013
- 30. Proposed Solution Root Cause Analyzer Category that Identif ication Calculates the has the Obvious? based on IR weight of RCs greatest weight RC Calculates the RC RC Identif ication weight of Question that based on has the 80% with the Questions and categories greatest weight same answer Categories Calculates the Threshold weight of CI L CI Identif ication based on RCs possible Question with higher level Answers Input Processor Weight Calculator Question Selector Question Verifier Root Cause Analyzer Malia Project – HP Labs & Institute of Informatics – UFRGS 33 Wednesday, March 06, 2013
- 31. Results Question Question Question Question Question Question Exim is installed? POP3 is running? Switch is up? Server is Switch is Apache is reachable by reachable by installed? ping? ping? Executions: 25 Executions: 22 Executions: 48 Executions: 41 Executions: 36 Executions: 16 Answers: 88% YES Answers: 63,6% NO Answers: 75% NO Answers: 73,2% NO Answers: 77,8% NO Answers: 93,7% YES Operator’s Answer: YES Operator’s Answer: YES Operator’s Answer: YES Operator’s Answer: NO Operator’s Answer: YES Operator’s Answer: YES Question Question Question Path of Apache Network adapter Switch ports are cfg is Ok? is configured? configured? Executions: 15 Executions: 18 Executions: 28 Answers: 66,6% NO Answers: 50% YES Answers: 71,4% YES Operator’s Answer: YES Operator’s Answer: YES Operator’s Answer: NO Question Question Question Question Question Question Exim is installed? POP3 is running? Apache is Path of Apache Exim is running? Path of Exim cfg is installed? cfg is Ok? Ok? Executions: 25 Executions: 22 Executions: 16 Executions: 15 Executions: 15 Executions: 8 Answers: 88% YES Answers: 63,6% NO Answers: 93,7% YES Answers: 66,6% NO Answers: 86,6% YES Answers: 100% YES Operator’s Answer: YES Operator’s Answer: YES Operator’s Answer: YES Operator’s Answer: YES Operator’s Answer: YES Operator’s Answer: NO Malia Project – HP Labs & Institute of Informatics – UFRGS 34 Wednesday, March 06, 2013
- 32. Conclusion and Future Work • The proposed solution proved to be feasible with the following features • Interactive • Considering the affected IT infrastructure • Based on information of the previous diagnoses • Reuse the knowledge acquired • Optimize the root cause identification process of failures in IT changes Malia Project – HP Labs & Institute of Informatics – UFRGS 35 Wednesday, March 06, 2013
- 33. Conclusion and Future Work • Melhorar o bootstrpping do sistema • Improve the selection algorithms • Consider other metrics (age) • Consider other selection methods (popularity) Malia Project – HP Labs & Institute of Informatics – UFRGS 36 Wednesday, March 06, 2013
- 34. Thanks for your attention! Questions?
- 35. Proposed Solution Root Cause Analyzer Category that Identif ication Calculates the has the Obvious? based on IR weight of RCs greatest weight RC Calculates the RC RC Identif ication weight of Question that based on has the 80% with the Questions and categories greatest weight same answer Categories Calculates the Threshold weight of CI L CI Identif ication based on RCs possible Question with higher level Answers Input Processor Weight Calculator Question Selector Question Verifier Root Cause Analyzer Malia Project – HP Labs & Institute of Informatics – UFRGS 38 Wednesday, March 06, 2013
- 36. Example Selection of Questions Current Version Software System Device 74 95 35 Computer System Network Card Web Server 30 70 30 Server 26 Third Party Services Web Server 5 20 Malia Project – HP Labs & Institute of Informatics – UFRGS 39 6 March, 2013
- 37. Example Selection of Questions Current Version Application System Device 74 95 35 Computer System Network Card Web Server 30 70 30 Server 26 Third Party Services Web Server 5 20 Malia Project – HP Labs & Institute of Informatics – UFRGS 40 6 March, 2013
- 38. Example Selection of Questions Current Version Software System Device 74 95 35 Computer System Network Card Web Server 30 70 30 Server 26 Third Party Services Web Server 5 20 Malia Project – HP Labs & Institute of Informatics – UFRGS 41 6 March, 2013
- 39. Example Selection of Questions Current Version Software Questions RC 1 (10) System Device 74 95 35 Questions RC 2 (20) Computer System Questions RC 3 (40) Network Card Web Server Questions RC 4 (25) 30 70 30 Qestions RC 1, 3 (50) Server 26 Third Party Services Web Server 5 20 Malia Project – HP Labs & Institute of Informatics – UFRGS 42 6 March, 2013
- 40. Example Selection of Questions Current Version Software Questions RC 1 (10) System Device 74 95 35 Questions RC 2 (20) Computer System Questions RC 3 (40) Network Card Web Server Questions RC 4 (25) 30 70 30 Common Questions Server 26 Third Party Services Web Server 5 20 Malia Project – HP Labs & Institute of Informatics – UFRGS 43 6 March, 2013
- 41. Example Selection of Questions Current Version 16 bits How many 32 bits bits is the application? 64 bits Malia Project – HP Labs & Institute of Informatics – UFRGS 44 6 March, 2013
- 42. Example Selection of Questions Current Version Software Questions RC 1 (10) System Device 74 95 35 Questions RC 2 (20) Computer System Questions RC 3 (40) Network Card Web Server Questions RC 4 (25) 30 70 30 Common Questions Server 26 Third Party Services Web Server 5 20 Malia Project – HP Labs & Institute of Informatics – UFRGS 45 6 March, 2013
- 43. Example Selection of Questions Current Version Software Questions RC 1 (10) System Device 74 95 35 Questions RC 2 (20) Computer System Questions RC 3 (40) Network Card Web Server Questions RC 4 (25) 30 70 30 Common Questions Server 26 Third Party Services Web Server 5 20 Malia Project – HP Labs & Institute of Informatics – UFRGS 46 6 March, 2013
- 44. Example Selection of Questions Current Version Software Questions RC 1 (10) System Device 74 95 35 Questions RC 2 (20) Computer System Questions RC 3 (40) Network Card Web Server Questions RC 4 (25) 30 95 – 70 = 25 70 30 Common Questions Server 26 Third Party Services Web Server 5 20 Malia Project – HP Labs & Institute of Informatics – UFRGS 47 6 March, 2013
- 45. Example Selection of Questions Current Version Software System Device 74 25 35 Computer System Network Card Web Server 30 70 30 Server 26 Third Party Services Web Server 5 20 Malia Project – HP Labs & Institute of Informatics – UFRGS 48 6 March, 2013
- 46. Example Selection of Questions Current Version Software System Device 74 25 35 Computer System Network Card Web Server 30 70 30 Server 26 Third Party Services Web Server 5 20 Malia Project – HP Labs & Institute of Informatics – UFRGS 49 6 March, 2013
- 47. Example Selection of Questions Current Version Software System Device 74 25 35 Computer System Network Card Web Server 30 70 30 Server 26 Third Party Services Web Server 5 20 Malia Project – HP Labs & Institute of Informatics – UFRGS 50 6 March, 2013
- 48. Example Selection of Questions Current Version Software Questions RC 5 (15) System Device 74 Questions RC 6 (5) 25 35 Questions RC 7 (12) Computer System Network Card Web Server Questions RC 8 (12) 30 30 Questions RC 9 (2)70 Questions RC 10 (3) Server Questions RC 11 (25) 26 Common Questions Third Party Services Web Server 5 20 Malia Project – HP Labs & Institute of Informatics – UFRGS 51 6 March, 2013
- 49. Example Selection of Questions Current Version Software Questions RC 5 (15) System Device 74 Questions RC 6 (5) 25 35 Questions RC 7 (12) Computer System Network Card Web Server Questions RC 8 (12) 30 30 Questions RC 9 (2)70 Questions RC 10 (3) Server Questions RC 11 (25) 26 Common Questions Third Party Services Web Server 5 20 Malia Project – HP Labs & Institute of Informatics – UFRGS 52 6 March, 2013
- 50. Example Selection of Questions Current Version 16 bits How many 32 bits bits is the application? 64 bits The devices No Yes are well plugged? Malia Project – HP Labs & Institute of Informatics – UFRGS 53 6 March, 2013
- 51. Example Selection of Questions Current Version Software Questions RC 5 (15) System Device 74 Questions RC 6 (5) 25 35 Questions RC 7 (12) Computer System Network Card Web Server Questions RC 8 (12) 30 30 Questions RC 9 (2)70 Questions RC 10 (3) Server Questions RC 11 (25) 26 Common Questions Third Party Services Web Server 5 20 Malia Project – HP Labs & Institute of Informatics – UFRGS 54 6 March, 2013
- 52. Example Selection of Questions Current Version Software Questions RC 5 (15) System Device 74 Questions RC 6 (5) 25 35 Questions RC 7 (12) Computer System Network Card Web Server Questions RC 8 (12) 30 30 Questions RC 9 (2)70 Questions RC 10 (3) Server Questions RC 11 (25) 26 Common Questions Third Party Services Web Server 5 20 Malia Project – HP Labs & Institute of Informatics – UFRGS 55 6 March, 2013
- 53. Example Selection of Questions Current Version 16 bits How many 32 bits bits is the application? 64 bits The devices No Yes are well plugged? No Network Yes card detected? Malia Project – HP Labs & Institute of Informatics – UFRGS 56 6 March, 2013
- 54. Example Selection of Questions Current Version Software Questions RC 5 (15) System Device 74 Questions RC 6 (5) 25 35 Questions RC 7 (12) Computer System Network Card Web Server Questions RC 8 (12) 30 30 Questions RC 9 (2)70 Questions RC 10 (3) Server Questions RC 11 (25) 26 Common Questions Third Party Services Web Server 5 20 Malia Project – HP Labs & Institute of Informatics – UFRGS 57 6 March, 2013
- 55. Example Selection of Questions Current Version Software Questions RC 5 (15) System Device 74 Questions RC 6 (5) 25 35 Questions RC 7 (12) Computer System Network Card Web Server Questions RC 8 (12) 30 30 Questions RC 9 (2)70 Questions RC 10 (3) Server Questions RC 11 (25) 26 Common Questions Third Party Services Web Server 5 20 Malia Project – HP Labs & Institute of Informatics – UFRGS 58 6 March, 2013
- 56. Example Selection of Questions Current Version 16 bits How many 32 bits bits is the application? 64 bits The devices No Yes are well plugged? No Network Yes card detected? Network No card is configured correctly? Network card Yes not configured Malia Project – HP Labs & Institute of Informatics – UFRGS 59 6 March, 2013