Defect Management Procedure Presentation
- 2. Table of Contents 2 Page # Subject Title Page # Subject Title 3 What is a Defect? 17 Defect Priority 4 Defect Management 18 Testing Evidence 6 Defect Lifecycle 19-22 Assess Defect 6 Defect Management Process Flow 23-24 Analyze Defect 8 Roles & Responsibilities 25-27 Fix Defect 9 Defect User Group 28 Fix Migration 10-11 Log Defect (Assign to Test Case) 29-30 Retest Fix 12 Defects - Key Items to Remember 30 Defect Workflows 13-15 Defect Fields 31 Defect Status Report 16 Defect Severity 32 Questions? 2
- 3. 3 • Defect is defined as a variance from expect testing results & actual results. Some examples of these are: An issue with IT Coding Incorrectly working function Overlooked requirement / Requirement Gap Performance problem Security issue Error in data • Each defect found throughout the testing efforts must be logged, so that it can be tracked and measured. What is a Defect ? 3
- 4. 4 • Defect management is a set of processes to manage, documenting and resolving defects identified during testing and to perform root cause analysis. The test management tool, HP Application Lifecycle Management (ALM), is used by the project team to conduct all defect management activities. • Defects will be analysed and a root cause attributed. Assigning a cause allows analysis to take place for reporting and management purposes such as determining quality levels of delivered software and products. • Defects needs to be resolved. Resolution can result in: No action (e.g. Rejected, Duplicate, Not Reproducible) A configuration or code fix A change to project scope (PCR) Defect Management 4
- 6. Defect Lifecycle 6 Status Description New Defect has been identified and logged. Open Defect has been reviewed and confirmed as a genuine defect . Rejected Not a Defect. Working as design. Duplicate Fix in Progress Analysis of fix is being performed. Fix Pending Migration Fixed Fix is ready for installation. PCR Approved Enhancement or new Requirement has been approved. PCR Deferred Enhancement or new Requirement has been postponed to a future release PCR Pending Approval Enhancement or new Requirement is pending approval PCR Rejected Enhancement or new Requirement has not been accepted or approved Ready for Retest Fix has been installed and is ready for retest. Reopen Defect is still reproducible and needs appropriate actions. Requirement Additional Information Additional Information is needed Required Information Updated Necessary Information is needed Closed Defect has been resolved, delivered and successfully tested. Deferred Defect fix has been postponed to a later date or Phase • Once a defect is identified, it will pass through various statuses that are used to control processing of and actions that can be performed on a defect. Possible statuses defined are listed below. 6
- 7. Defect Management Process Flow 7 • The flow below shows the general process flow for defect management. Each stage of the lifecycle will be described in more detailed later in this document. • E-mail system will be set up within HP ALM so that status changes are automatically mailed to the appropriate person/groups. LEGEND: Process Defect Status 1. Log Defect <Tester> 4. Fix Defect <Fix Agent> 5. Fix Migration <Project Environment Coordinator> 6. Re-test Fix <Tester> 2. Assess / Analyze Defect <Lead Tester> 3. Assign Defect <IT/Technical Lead> NEW OPEN FIXED READY FOR TEST CLOSED OPEN 1. Log Defect <Tester> 4. Analyze Defect <Fix Agent> 5. Fix Defect <Fix Agent> 6. Fix Migration <Project Environment Coordinator> 7. Re-test Fix <Tester> 2. Assess Defect <Test Lead> 3. Assign Defect <IT/Technical Lead> NEW OPEN FIX IN PROGRESS FIXED READY FOR TEST CLOSED OPEN Rock Project Defect Flow Boulder Project Defect Flow 7
- 8. Roles and Responsibilities 8 8 Role Responsibilities Program Manager Support project testing activities and provide program leadership Escalation point for project. Test Lead Coordinate and manage testing issues, including appropriate escalation Work with overall project leadership team to manage required testing activities Obtain final approval and sign-offs for each testing cycle QA Analyst Upload approved test related documentation including requirements and test scripts in ALM 11.5 for each project. Create and distribute project reports Provide ALM support. Functional Testers Responsible for executing test cases as defined in the test plan. Documents and reports the results of test execution. Documents detailed defects using ALM. Lead Tester - Functional Lead Creates valid test plans, test cases for area of responsibility Ensures timely completion of testing activities for their area of responsibility. Coordinates of the resolution of defects for their area of responsibility PMO Analyze possible change requests as a result of defects impacts. Coordinate with Program Manager and Functional leads on possible scope creep . Technical Lead Coordinates review and assignment of technical work for resolution of project defects Fix Agent Completes and tests technical work for resolution of project defects Project Environment Coordinator Monitor Dev to Test Migrations. Move defects to retest once migrations complete.
- 9. Defect User Group 9 • User Groups are defined by the HP ALM admin within each project in HP ALM based upon the needs of the project • Project Manager will coordinate with QA Team to assign access to the correct User Groups • Viewers will only have READ access • Enclosed in the spreadsheet is a list of roles and related defect statuses that each user will update over the lifecycle of a defect during Testing. 9
- 10. Description: The first stage of the Defect Management process is the detection and logging of a defect. This ensures all defects can be worked on and tracked by the project. Once a defect has been identified and a brief initial assessment has been undertaken to confirm it is a defect. A defect must be entered within HP ALM. Mandatory information should be entered regarding the defect to ensure all necessary data is available for efficient processing in later stages. Process: Log Defect (assign to Test Case) 10 1. Log Defect Tester 1.1 Collect Information about Defect Start 1.2 Create New Defect Record 2 10
- 11. Procedures: 1.1 Collect Information about Defect Adequate information about the defect for assessment must be collected. Typical information to be required are: - Outcome (In which point is different from expectation i.e. Expected vs Actual Results) - Data and procedure reproduce - Testing evidence e.g. screenshot 1.2 Create New Defect Record Defect entry must be made in HP ALM. To create a defect, a minimum set of required information needs to be entered. (See Defect Fields page.) Once entered the status of the defect automatically becomes NEW and an email notification will be sent to the LEAD TESTER/TEST LEAD for initial root cause analysis. Log Defect (assign to Test Case) 11 11
- 12. Procedures: Defects: Key Items to Remember 12 Link defects to test cases when they fail; at test step level Description of the defect should be clear and detailed Include the steps required to reproduce the defect and adequate supporting information Add screenshots to defects 12
- 13. Defect Fields • The following minimum set of information is required to create a new defect. Summary – brief description of a defect Project Phase/Workstream – Phase or Workstream in which the defect was found Environment – Development, Test, UAT, & Production Functional Area – this is auto-populated when new defect is submitted. Detected in Cycle – Release Cycle e.g. Shakeout, Component & Functional, System Integration Testing, Performance, & UAT. Severity – impact on the project (See Defect Severity page.) Submit Date –the date on which the defect was identified Priority – this an auto-populated field from the requirement based on business. Status – Defect status (“New” in default, see Defect Lifecycle page.) Business Impact – Free Text area where you can summarize the impact of the defect to the business. Description – This is auto-populated from the linked failed test step 13 13
- 14. Defect Fields • Additional fields as defect status changes. Defect Type – i.e. Code, Configuration, Not a Defect, Program Change Request (PCR) Root Cause – the reason of the defect occurrence e.g. Code Error, Conversion, Environment Issue, Network, Vendor Issue, PCR, or Missed Requirement. Assigned To – user to which the defect is assigned e.g. tmp12345 Estimated Fix Date – estimated date Development provides for target fix Corrective Action – Action taken to resolve the defect i.e. Code Fix, Configuration Fix, Database Fix, Server Restart, & Vendor Fix Vendor Defect ID – This is only populated if the defect was assigned to a vendor for resolution e.g. Oracle SR. Work in Progress by – work that is assigned to either a vendor or KCP&L. Expected Resolution Date – date the business or project team expects resolution 14 14
- 15. Defect Fields • “Defect Type” will be entered by the Lead Tester/Test Lead once the Initial Root Cause Analysis is finished Assigned To – depending on the Defect Type selected, it will be assigned automatically to the appropriate user group Priority – based on Business Requirements • The following fields will be entered by the Lead Tester/Test Lead during analysis and modification processes. Comments – the result of analysis NOTE: You must click on “Add Comment” before filling in the information. This captures your user ID and date/time stamp. • The following fields will be updated by the Tester after re-test successfully passed. Comments – the result of re-test NOTE: You must click on “Add Comment” before filling in the information. This captures your user ID and date/time stamp. 15 15
- 16. Defect Severity Severity Level Definition Example Sev 1 • The system is not functioning and a significant portion of testing is stopped. • The issue is a show-stopper, and must be resolved before progressing to the next testing cycle and system go-live. • No work-around exists and immediate attention is required. Critical Business component unavailable or functionally incorrect. Workaround is NOT available for testing to continue. e.g. Defect has cause testing environment to be inoperable Sev 2 • Major functionality is disabled or not working per requirements. • The issue needs to be resolved before progressing to the next testing cycle and system go-live. • An acceptable work around does not exist. Major component unavailable or functionally incorrect; incorrect calculation results in functionally critical key fields/dates and an acceptable workaround is not available. Sev 3 • A variance from the requirements exists under prescribed conditions. • The defect will not stop progression to the next testing cycle or system go-live • An acceptable work around exists Usability errors; screen or report errors that do not materially affect quality and correctness of function, intended use or results. 16 Defect severity is required for the tester to enter. The Test Lead will determine if that is the correct severity later. 16
- 17. Defect Priority 17 Priority Description High •Must be fixed in any of the upcoming builds but should be included in the release. Medium •May be fixed after the release / in the next release. Low •May or may not be fixed at all • Defect priority is auto-populated by the system. • The defect priority of high, medium or low should be established based on impact to business. When applicable, the priority will default to the priority of the linked business requirement. 17
- 18. Testing Evidence 18 • Testing evidence provided by the Testers should be sufficient enough to support the Lead Tester/Test Lead for root cause analysis. These will include: Outcomes that show an error or a variance from expectations - Screenshots - Output data (attached documents) Testing procedure performed (if needed –these will help the Lead Tester/Test Lead reproduce the defect during their analysis processes.) - Actual test steps taken to reproduce the defect. - Screenshots that show the tester followed the test steps as specified in the test cases. - Input data used in the test set. • Testing evidence need to be attached to the defect in HP ALM. 18
- 19. Description: The purpose of this step in the process is to review the defect and confirm its validity. The LEAD TESTER/TEST LEAD undertakes a peer review to ensure the defect has been correctly completed and confirm the tester’s assessment of severity level is correct. The LEAD TESTER/TEST LEAD identifies correct group/team for resolution and updates the defect has been logged. If the defect is validated, the test team lead takes appropriate action to cancel or close it. Process: 19 2. Assess Defect Lead Tester/Test Lead Start 2.2 Not A Defect 2.1 Initial Root Cause Analysis Missed Requirement or Enhancement? 2.3 Program Change Request (PCR) Yes End 2.5 Assign to Non- IT Config Fix Agent No Duplicate or Not Reproducible? Yes No 3 Is this an IT Fix? No 5 Yes 2.4 Code Defect 4 Assess Defect 19
- 20. Procedures: 2.1 Initial Root Cause Analysis A review is conducted by the Lead Tester/Test Lead to check that all information has been supplied, the defect is not a duplicate, and that severity code is correct. Once reviewed the status of the defect is updated to OPEN. 2.2 Not A Defect After the Initial Root Cause Analysis, If the Lead Tester/Test Lead determines that it is not a defect e.g. Unable to Reproduce, working as designed, or a duplicate defect, the Defect Type should be set to NOT A DEFECT and the Defect Status will automatically set to REJECTED. After the Lead Tester/Test Lead sets the defect status to REJECTED, notification of reject about the defect is automatically e-mailed to the tester that created the defect. The Lead Tester/Test Lead may also manually send an email to the other team members. 2.3 Program Change Request (PCR) After the Initial Root Cause Analysis, If the Lead Tester determined that the Defect is a missed requirement or an enhancement, then the Defect Type should be set to PCR. Once done, defect status will automatically set to PCR Pending Approval & email notification will be sent to PMO Group. The status will remain pending until the PCR has been reviewed and acted upon. Root Cause should be determined as well. 2.4 Code Defect After the Initial Root Cause Analysis, If the Lead Tester/Test Lead determines that the Defect is a code issue or an IT Fix issue, the Defect Type should be set to CODE and the Defect Status will automatically set to OPEN & Root Cause should be determined as well. An email notification will be sent to the IT Lead or Technical Lead for further review and assignment. 2.5 Assign to Non-IT Config Fix Agent After the Initial Root Cause Analysis, If the Lead Tester/Test Lead determines that the Defect is NOT a code issue and is a Non-IT Fix issue, the Defect Type should be set to NON-IT. Email notification will be sent to Lead Tester/Test Lead group for them to assign the defect to a non IT Fix agent. Defect status will be set to OPEN and assigned field will be blank. Root Cause should be determined as well. Lead Tester/Test Lead should reopen the defect and they assign it to the appropriate Non-IT Config fix agent. Assess Defect 20 20
- 21. Description: The purpose of this step in the process is for the PMO to review the PCR defect and submit to the Steering Committee for an approval. It will be decided by the committee if the PCR will approved fixed in the current release or deferred to a future release. Process: 21 3. Assess Defect PMO 2.3 3.2 PCR Rejected 3.1 PCR Defect Review PCR Approved for the Current Release? 3.3 PCR Deferred No End Yes PCR Approved by Steering Committe/ PMO? No Yes 3.4 PCR Approved 4 Assess Defect 21
- 22. Procedures: 3.1 PCR Defect Review After the PCR defect is received from the Lead Tester/Test Lead, PMO reviews the PCR and submit it to the Steering Committee/PMO to make a decision i.e. Rejected, Approved or Deferred. 3.2 PCR Rejected If the PCR is denied the PMO will set the Defect Status to PCR REJECTED and an email notification will be sent to the Tester that created the defect. 3.3 PCR Deferred If the PCR deferred by the Steering Committee to be included in the future release, then the PMO will set the Defect Status to PCR DEFERRED. An email notification will be sent to the Tester that created the defect. 3.4 PCR Approved If the PCR is approved by the Steering Committee, the fix will be included in the current or future release. If the fix will in the current release then the PMO will set the Defect Status to PCR APPROVED. An email notification will be sent to IT/Technical Lead to assign. NOTE: Each time there’s a PCR Defect, PMO should have the PCR document completed and attached in HP ALM prior submitting it to the Steering Committee for approval. Please ensure to follow the PCR flow. Assess Defect, cont’d. 22 22
- 23. Description: The purpose of this step in the process is for the PMO to review the PCR defect and submit to the Steering Committee for an approval. If it has been determined that it is a legit defect, it will be decided by the committee if the PCR will be fixed in the current release or deferred to a future release. (Not PMO – IT LEAD) Process: 23 4. Analyze Defect IT/Technical Lead 2.5 4.1 Non IT Config Fix Defect Not a Defect? 4.2 Not a Defect Yes End No Is this an IT Fix? No Yes 4.3 Assign to Appropriate Fix Agent 5 Analyze Defect 23
- 24. Analyze Defect Procedures: 4.1 Non IT Config Fix Defect After additional Root Cause Analysis, the IT/Technical Lead determines that the Defect is NOT a code issue and is a Non-IT Fix issue, the Defect Type is set to NON-IT. Email notification will be sent to Lead Tester/Test Lead group for them to assign it to the appropriate Non-IT Config fix agent. Defect status will be set to OPEN. Root Cause should be determined as well. 4.2 Not a Defect After additional Root Cause Analysis, the IT/Technical Lead determines that it is not a defect e.g. Unable to Reproduce, working as designed, or a duplicate defect, the Defect Type is set to NOT A DEFECT and the Defect Status will automatically set to REJECTED & Root Cause should be determined as well. Once the IT/TECHNICAL LEAD sets the defect status to REJECTED, information about the defect is automatically e-mailed to the tester that created the defect. The IT/TECHNICAL LEAD may also manually send an email to other team members that a defect has been created. 4.3 Assign to Appropriate Fix Agent Once the PMO/Steering Committee approved the PCR, IT/TECHNICAL LEAD will assign the defect to the responsible/appropriate IT Fix Agent. Select the responsible IT Fix Agent in the ‘Assigned To’ dropdown field. Email notification will be sent to the fix agent. 24 24
- 25. Fix Defect Description: The purpose of this step in the process is to identify and resolve the defect. Regular meetings will be held to review the status of all defects and fixes. Process: 25 5. Fix Defect Tester Fix Agent 4.3 5.1 Defect Fix in Progress Fix Completed? 5.3 Additional Information Required Require more information from Tester? No 5.2 Corrective Action 6 Yes Yes No 5.4 Updated Required Information Yes 25
- 26. Fix Defect Procedures: 5.1 Defect Fix in Progress Once the defect has enough information to start working on a fix, the Fix Agent should change the Defect Status to FIX IN PROGRESS so the team is aware that the fix is in progress. 5.2 Corrective Action Once the defect fix is completed, a Corrective Action information is needed e.g. Code Fix, Configuration Fix, Server Restart, or Vendor Fix. The Root Cause field should only be updated if its different from the one already selected. If the defect fix is made directly in TEST environment then the Defect Status should be set to FIXED. OR if the defect fix is made in Development environment and it needs to be migrated to the Test environment then the Defect Status should be set to FIX PENDING MIGRATION. An email notification will be sent to the Project Environment Coordinator (PEC). 5.3 Additional Information Required If the defect assigned requires additional detail required information in the comments section for the Tester, Defect Status should be set to REQUIRE ADDITIONAL INFORMATION. An email notification will be sent to the Tester. 5.4 Update Required Information The Tester should update the defect with the information requested by the Fix Agent in the comment. Once fulfilled, the Tester should change the Defect Status to REQUIRED INFORMATION UPDATED. An email notification will be sent to the Fix Agent. 26 26
- 27. Fix Migration Description: The purpose of this step in the process is to migrate the fix developed to the appropriate environment. Regular meetings need to be held to review the status of all defects and fixes to discuss test progress. Process: 27 6. Fix Migration Project Environment Coordinator (PEC) 6.2 Update Status of Defect 5.2 6.1 Migration Review 7 Migration Required? Yes 27
- 28. Fix Migration Procedures: 6.1 Migration Review Project Environment Coordinator (PEC) reviews completed fixes for all defects are documented and approved. This could includes changes to architecture of project related environments, configuration/code changes, environment setups & version control. 6.2 Update Status of Defect PEC then updates the status of the defect to READY FOR TEST. An email notification will sent to the Tester for retest. 28 28
- 29. Re-test Fix Description: The purpose of this step in the process is to confirm defect resolution. The original test sets or specific test cases that found the defect are re-executed, and a pass or fail decision is made about the provided fix. The Lead Tester/Test Lead checks the testing performed on the defect and the results obtained. If the re-test is adequate and the outcome successful, then the defect is closed. Process: 7. Re-test Fix Tester 7.2 Close Defect 6 7.1 Perform Re-test End Retest Passed? Yes 7.3 Re-Open Defect No Is Defect Type Code? 2.1 No Yes 4.3 29 29
- 30. Re-test Fix Procedures: 7.1 Perform Re-test Select test case(s) to run to test the defect. The testing should consist of the same test cases that originally identified. 7.2 Close Defect If all selected tests pass then the defect is considered to have passed. The defect needs to be updated with the following information by the tester. - Comments - Closing Date (this is updated automatically by HP ALM when the status of a defect is changed to “closed”) The tester sets the status of the defect to CLOSED after checking that all information has been entered into the defect log and the testing is adequate. 7.3 Re-open Defect If retesting confirms that the problem still exists, then the defect must be re-opened. The tester should append additional information about what has happened during re-testing and set the status of the defect to REOPEN. The defect then routes through the defect lifecycle again. However, if different step fails during retest then submit a new defect. An email notification will be either sent to Lead Tester/Test Lead OR IT/Technical Lead Group depending if the defect occurred during retesting is a Config or Code type defect. 30 30
- 31. Defect Workflow 31 • Boulder ALM Workflow • Rock ALM Workflow 31
- 32. Defect Status Report 32 • Defect status report includes the following information: Defect Id Functional Area Severity Priority Defect Summary Status Submit Date Estimated Fix Date Assigned To • The following is a sample defect status report 32 Defect Id Functional Area Severity Priority Defect Summary Status Submit Date Estimated Fix Date Assigned To 1 Dashboard Sev 1 High Interface Issue Ready For Test 7/20 7/23 tmp123456 2 General Ledger Sev 2 Medium User and Password incorrect Fixed 7/20 7/24 tmp123456 3 Procurement Sev 3 Low Menu Title Typo Open 7/20 8/23 tmp123456
- 33. QUESTIONS? Contact QA Team: ALMtesting@kcpl.com 33 33