[Case study]Utilize STLC data for Process Improvement
1 like2,013 views
The document discusses utilizing STLC (software testing life cycle) data to improve processes. It summarizes a case study where the speaker analyzed over 277,000 records of bugs recorded during testing to identify deployment issues as a major source of wasted time. The speaker gathered data on root causes and potential solutions, finding that deployment problems accounted for 35% of environmental bug resolution time. By addressing deployment issues, the company estimates it could save around 300 hours per year.
![[Case study]
Utilize STLC data for Process Improvement
Jun 30th, 2020
Rajat Dayma
Service Quality Assurance Group
Leisure Product Dept.
Commerce Company
Rakuten, Inc.](https://image.slidesharecdn.com/20200630-200626010539/85/Case-study-Utilize-STLC-data-for-Process-Improvement-1-320.jpg)
![4
[Career Summary]
> I am from India having more than 13+ years of
experience in QA field
> Worked in Banking, Finance, Insurance domains
> Having around 7-8 years' experience working with
Japanese clients
> ISTQB Foundation level and JLPT N2 certified
> Worked on service and product based companies
both
Self-Introduction](https://image.slidesharecdn.com/20200630-200626010539/85/Case-study-Utilize-STLC-data-for-Process-Improvement-4-320.jpg)
![[Case study]Utilize STLC data for Process Improvement](https://image.slidesharecdn.com/20200630-200626010539/85/Case-study-Utilize-STLC-data-for-Process-Improvement-54-320.jpg)
[Case study]Utilize STLC data for Process Improvement
- 1. [Case study] Utilize STLC data for Process Improvement Jun 30th, 2020 Rajat Dayma Service Quality Assurance Group Leisure Product Dept. Commerce Company Rakuten, Inc.
- 2. 2 1. Self Introduction 2. Today’sTheme & Goal 3. What is STLC Data 4. a. Data collection and analysis 5. b. Metrix and analytics flow 6. Actual improvement: deployment issue 7. Future 8. Conclusion Agenda
- 3. 3 1. Self Introduction 2. Today’sTheme & Goal 3. What is STLC Data 4. a. Data collection and analysis 5. b. Metrix and analytics flow 6. Actual improvement: deployment issue 7. Future 8. Conclusion Agenda
- 4. 4 [Career Summary] > I am from India having more than 13+ years of experience in QA field > Worked in Banking, Finance, Insurance domains > Having around 7-8 years' experience working with Japanese clients > ISTQB Foundation level and JLPT N2 certified > Worked on service and product based companies both Self-Introduction
- 5. 5 1. Self Introduction 2. Today’sTheme & Goal 3. What is STLC Data 4. a. Data collection and analysis 5. b. Metrix and analytics flow 6. Actual improvement: deployment issue 7. Future 8. Conclusion Agenda
- 6. 6 Today’s Theme QA Role = Product and Process Quality https://www.kindpng.com/imgv/ihmmowT_this-free-icons-png-design-of-conveyor-with/
- 7. 7 Today’s Theme QA Role = Product and Process Quality Process Product https://www.kindpng.com/imgv/ihmmowT_this-free-icons-png-design-of-conveyor-with/
- 8. 8 Today’s Theme QA Role = Product and Process Quality Process https://www.kindpng.com/imgv/ihmmowT_this-free-icons-png-design-of-conveyor-with/ Today’s Goal
- 9. 9 1. Self Introduction 2. Today’sTheme & Goal 3. What is STLC Data 4. a. Data collection and analysis 5. b. Metrix and analytics flow 6. Actual improvement: deployment issue 7. Future 8. Conclusion Agenda
- 10. 10 STLC Data Plan TestDevelop In the testing phase, we view human error as a bug. As of June 2020, there are 277k of them recorded and many points of improvement to learn from the results It means that you can.
- 11. 11 STLC Data Plan TestDevelop In the testing phase, we view human error as a bug. As of June 2020, there are 277k of them recorded and many points of improvement to learn from the results It means that you can.
- 12. 12 STLC Data Plan TestDevelop In the testing phase, we view human error as a bug. As of June 2020, there are 277k of them recorded and many points of improvement to learn from the results It means that you can.
- 13. 13 STLC Data Plan TestDevelop STLC Database 277k+ results In the testing phase, we view human error as a bug. As of June 2020, there are 277k of them recorded and many points of improvement to learn from the results It means that you can.
- 14. 14 1. Self Introduction 2. Today’sTheme & Goal 3. What is STLC Data 4. a. Data collection and analysis 5. b. Metrix and analytics flow 6. Actual improvement: deployment issue 7. Future 8. Conclusion Agenda
- 15. 15 Metrix and analytics flow Category Meaning Example FUNCTION Do not work as per specification • Spec says ‘A’ link should be clickable but in app its not working DEGRADE Function worker earlier but not working now properly • In previous R1 execution “Button A” was working fine but in R2 its not working as expected REQUIREMENT Not mentioned or wrong explanation in specification • “A” user should able to purchase only one type of ticket but actually it should purchase all types of tickets PRD is wrong. EXISTING Existing product bug • QA found difference in PC an SP behavior but its expected based on usage UI Usability, Security, Performance issues • If any page is taking more time to load then its treated as usability issues ENVIRONMENT Environment issues which affects for our test execution • Because of wrong deployment of STG we raise the ticket INVALID QA misunderstood functionality • Spec completely misunderstood by QA LATER It’s bug but won’t be fixed this release. Future release • Valid bug wont be fixed in this release and considered for future releases NONFUNCTION （userability）（performance（security） • Performance or security related issues OTHER_SERVICE CWD, Coupon, Salesforce, Dependent on other teams to resolve • Bugs which are dependent on other systems to solve
- 16. 16 STLC Database 277k+ results Metrix and analytics flow
- 17. 17 STLC Database 277k+ results Metrix and analytics flow
- 18. 18 STLC Database 277k+ results Metrix and analytics flow
- 19. 19 1. Self Introduction 2. Today’sTheme & Goal 3. What is STLC Data 4. a. Data collection and analysis 5. b. Metrix and analytics flow 6. Actual improvement: deployment issue 7. Future 8. Conclusion Agenda
- 20. 20 STLC Data TestDevelop STLC Database We saw a lot of environmental issues that arose during the development phase and the testing phase. So we decided to focus on that.
- 21. 21 STLC Data TestDevelop STLC Database Developer We saw a lot of environmental issues that arose during the development phase and the testing phase. So we decided to focus on that.
- 22. 22 STLC Data TestDevelop STLC Database Test Environment Developer We saw a lot of environmental issues that arose during the development phase and the testing phase. So we decided to focus on that.
- 23. 23 STLC Data TestDevelop STLC Database Test Environment Developer 3rd party system We saw a lot of environmental issues that arose during the development phase and the testing phase. So we decided to focus on that.
- 24. 24 STLC Data TestDevelop STLC Database Test Environment Developer 3rd party system We saw a lot of environmental issues that arose during the development phase and the testing phase. So we decided to focus on that.
- 25. 25 Proxy Issue 35 % of environment bug treatment time spent on Among them, “Deployment issue” category takes up the most time. Test Environment 3rd party system 3rd PartyTrouble #ofissues cumulativeratio Developer 35% Find the root cause
- 26. 26 Proxy Issue Source: xxx 35 % of environment bug treatment time spent on Among them, “Deployment issue” category takes up the most time. Test Environment 3rd party system Wrong setting 3rd PartyTrouble #ofissues cumulativeratio Developer Deployment Issue is the most 35% Find the root cause
- 27. 27 Once the target of the problem has been determined, we gather qualitative data through interviews as well as quantitative data. Find the root cause
- 28. 28 Ticket Ticket Comment Hearing Note https://ticket.server.com/ticket/browse/BUG-99848 Due to DB data difference. XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Environment settingissue Dev team YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY YYYYY https://ticket.server.com/ticket/browse/BUG-88090 【Expected value 】 is correct. XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XX Environment settingissue Dev team inserted YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY https://ticket.server.com/ticket/browse/BUG-66427 Deployment for group XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Environment settingissue Updating content YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY https://ticket.server.com/ticket/browse/BUG-88726 it is environment problem.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX. Deploytimingissue but unfortunately YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY https://ticket.server.com/ticket/browse/BUG-77062 All the following functions are modified by both our project XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXx Deploytimingissue Issue detail) YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY https://ticket.server.com/ticket/browse/BUG-66345 this is environment issue. Because XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Deploytimingissue Issue detail) Multiple YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY YYYYYYYYYYYYYYYYYYYYYYYYYY Once the target of the problem has been determined, we gather qualitative data through interviews as well as quantitative data. Find the root cause
- 29. 29 Ticket Ticket Comment Hearing Note https://ticket.server.com/ticket/browse/BUG-99848 Due to DB data difference. XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Environment settingissue Dev team YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY YYYYY https://ticket.server.com/ticket/browse/BUG-88090 【Expected value 】 is correct. XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XX Environment settingissue Dev team inserted YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY https://ticket.server.com/ticket/browse/BUG-66427 Deployment for group XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Environment settingissue Updating content YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY https://ticket.server.com/ticket/browse/BUG-88726 it is environment problem.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX. Deploytimingissue but unfortunately YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY https://ticket.server.com/ticket/browse/BUG-77062 All the following functions are modified by both our project XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXx Deploytimingissue Issue detail) YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY https://ticket.server.com/ticket/browse/BUG-66345 this is environment issue. Because XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Deploytimingissue Issue detail) Multiple YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY YYYYYYYYYYYYYYYYYYYYYYYYYY Once the target of the problem has been determined, we gather qualitative data through interviews as well as quantitative data. Find the root cause
- 30. 30 Ticket Ticket Comment Hearing Note https://ticket.server.com/ticket/browse/BUG-99848 Due to DB data difference. XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Environment settingissue Dev team YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY YYYYY https://ticket.server.com/ticket/browse/BUG-88090 【Expected value 】 is correct. XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XX Environment settingissue Dev team inserted YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY https://ticket.server.com/ticket/browse/BUG-66427 Deployment for group XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Environment settingissue Updating content YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY https://ticket.server.com/ticket/browse/BUG-88726 it is environment problem.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX. Deploytimingissue but unfortunately YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY https://ticket.server.com/ticket/browse/BUG-77062 All the following functions are modified by both our project XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXx Deploytimingissue Issue detail) YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY https://ticket.server.com/ticket/browse/BUG-66345 this is environment issue. Because XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Deploytimingissue Issue detail) Multiple YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY YYYYYYYYYYYYYYYYYYYYYYYYYY Once the target of the problem has been determined, we gather qualitative data through interviews as well as quantitative data. Find the root cause
- 31. 31 Calculate ROI : Because the number of man-hours spent on solving the problem and the value of the solution It's to reflect on the economic value or not. Calculate Return on Investment
- 32. 32 Calculate ROI : Because the number of man-hours spent on solving the problem and the value of the solution It's to reflect on the economic value or not. Calculate Return on Investment
- 33. 33 Test Environment Developer Tester Currently we are losing around 300 hours efforts to solve deployment issue Calculate Return on Investment
- 34. 34 Test Environment Developer Tester ①Confirm & Create bug ticket Currently we are losing around 300 hours efforts to solve deployment issue Calculate Return on Investment
- 35. 35 Test Environment Developer Tester ①Confirm & Create bug ticket ②Investigate & Re-Deploy Currently we are losing around 300 hours efforts to solve deployment issue Calculate Return on Investment
- 36. 36 Test Environment Developer Tester ①Confirm & Create bug ticket ②Investigate & Re-Deploy ③Re-Test Currently we are losing around 300 hours efforts to solve deployment issue Calculate Return on Investment
- 37. 37 Test Environment Developer Tester ①Confirm & Create bug ticket ②Investigate & Re-Deploy ③Re-Test ①Confirm Create bug ticket : X hours ②Investigate & Re-Deploy : Y hours ③Preparation & Re-Test : Z hours #of Dep issues / yar Currently we are losing around 300 hours efforts to solve deployment issue Calculate Return on Investment
- 38. 38 Test Environment Developer Tester ①Confirm & Create bug ticket ②Investigate & Re-Deploy ③Re-Test 300+ hours Lost ①Confirm Create bug ticket : X hours ②Investigate & Re-Deploy : Y hours ③Preparation & Re-Test : Z hours #of Dep issues / yar Currently we are losing around 300 hours efforts to solve deployment issue Calculate Return on Investment
- 39. 39 Calculate ROI : Because the number of man-hours spent on solving the problem and the value of the solution It's to reflect on the economic value or not. Calculate Return on Investment
- 40. 40 Pick up any number of projects and test them for additional man-hours and effectiveness Calculate Return on Investment
- 41. 41 Pick up any number of projects and test them for additional man-hours and effectiveness Calculate Return on Investment Project A Project B Project C Project D Project E Project F Project G Project H Project I Project J Project K Project L Project M Project N Project O Project P Project Q Project R
- 42. 42 Pick up any number of projects and test them for additional man-hours and effectiveness Calculate Return on Investment Project A Project B Project C Project D Project E Project F Project G Project H Project I Project J Project K Project L Project M Project N Project O Project P Project Q Project R
- 43. 43 We decided to target this category because this solution has no cost and some of LPD team has already implemented strong solution about that. Objective Key elements Root cause Reduction of environment al issues Deployment issue 3rd PartyTrouble Deploy timing issue Environment setting issueProxy Related Issue Other Solutions Calculate Return on Investment
- 44. 44 We decided to target this category because this solution has no cost and some of LPD team has already implemented strong solution about that. Objective Key elements Root cause Reduction of environment al issues Deployment issue 3rd PartyTrouble Deploy timing issue Environment setting issueProxy Related Issue Other Solutions Solution A $ Solution B $$$ Solution C $$ Calculate Return on Investment
- 45. 45 We decided to target this category because this solution has no cost and some of LPD team has already implemented strong solution about that. Objective Key elements Root cause Reduction of environment al issues Deployment issue 3rd PartyTrouble Deploy timing issue Environment setting issueProxy Related Issue Other Solutions Solution A $ Solution B $$$ Solution C $$ Solution D $ Solution E $$$ Solution F $$ Calculate Return on Investment
- 46. 46 We decided to target this category because this solution has no cost and some of LPD team has already implemented strong solution about that. Objective Key elements Root cause Reduction of environment al issues Deployment issue 3rd PartyTrouble Deploy timing issue Environment setting issueProxy Related Issue Other Solutions Solution A $ Solution B $$$ Solution C $$ Solution D $ Solution E $$$ Solution F $$ Calculate Return on Investment
- 48. 48 Source: xxx Before After 35% 2.38% Result
- 49. 49 Once proven effective, we will apply it to all projects. Adaptation to the all projects Project A Project B Project C Project D Project E Project F Project G Project H Project I Project J Project K Project L Project M Project N Project O Project P Project Q Project R
- 50. 50 1. Self Introduction 2. Today’sTheme & Goal 3. What is STLC Data 4. a. Data collection and analysis 5. b. Metrix and analytics flow 6. Actual improvement: deployment issue 7. Future 8. Conclusion Agenda
- 51. 51 STLC Database 277k+ results Future Of course, there are still many problems that can be solved with STLC database analysis. So in the future we want to keep improving! Branch merge issueEnvironment Isolation issue Requirement issue
- 52. 52 1. Self Introduction 2. Today’sTheme & Goal 3. What is STLC Data 4. a. Data collection and analysis 5. b. Metrix and analytics flow 6. Actual improvement deployment issue 1. Result 01 2. Result 02 3. Summary 7. Future 8. Conclusion Agenda
- 53. 53 Conclusion QA Role = Product and Process Quality Process Product https://www.kindpng.com/imgv/ihmmowT_this-free-icons-png-design-of-conveyor-with/