QUES#19 Automation and Quality 2022.pdf
- 1. 1 Can we automate bug analytics? Sonal Singh Booking QA Group Leisure Product Department Service Quality Assurance Section Rakuten Group, Inc. 2022/11/17
- 2. 2 ➢ Who Am I? ➢ Background of bug analytics ➢ What is bug analytics? ➢ Can we automate it? Agenda
- 3. 3 ➢ My name is Singh Sonal ソナル シング ➢ I’m from India ➢ I have over 10 years of experience into QA and worked with Service as well as Product based companies. ➢ I’m in Japan since Sep, 2017 and working in Rakuten since Feb, 2018 Who am I?
- 4. 4 ➢ I gave presentation on RTM (Requirement traceability matrix) in Nov, 2018 Last QUES Presentation
- 5. 5 ➢ Who Am I? ➢ Background of bug analytics ➢ What is bug analytics? ➢ Can we automate it? Agenda
- 6. 6 Quality Assurance QA is responsible for Process and Product both Process Product https://www.kindpng.com/imgv/ihmmowT_this-free-icons-png-design-of-conveyor-with/
- 7. 7 Today’s Theme Implement process for analytics Process Product https://www.kindpng.com/imgv/ihmmowT_this-free-icons-png-design-of-conveyor-with/
- 8. 8 In STLC Human error = Bug Plan Test Develop In the testing phase, we view human error as a bug. We had 500K such errors and few years back we had to process this data to make improvement.
- 9. 9 As a first step we started to record all this data in our STLC database. Plan Design Implementation Execution Final report STLC data base
- 10. 10 Store STLC Database 500k+ results Step 1 Done What’s Next
- 11. 11 REQUIREMENT NON-FUNCTIONAL ENVIRONMENT INVALID FUNCTION DEGRADE EXISTING LATER Store Categorize STLC Database 500k+ results Next step: Categorize
- 12. 12 Category Meaning Example FUNCTION Do not work as per specification • Spec says ‘A’ link should be clickable but in app its not working DEGRADE Function worker earlier but not working now properly • In previous R1 execution “Button A” was working fine but in R2 its not working as expected REQUIREMENT Not mentioned or wrong explanation in specification • “A” user should able to purchase only one type of ticket but actually it should purchase all types of tickets PRD is wrong. EXISTING Existing product bug • QA found difference in PC an SP behavior but its expected based on usage UI Usability, Security, Performance issues • If any page is taking more time to load then its treated as usability issues ENVIRONMENT Environment issues which affects for our test execution • Because of wrong deployment of STG we raise the ticket INVALID QA misunderstood functionality • Spec completely misunderstood by QA LATER It’s bug but won’t be fixed this release. Future release • Valid bug wont be fixed in this release and considered for future releases NON FUNCTION （userability）（performance（security） • Performance or security related issues OTHER_SERVICE CWD, Coupon, Salesforce, Dependent on other teams to resolve • Bugs which are dependent on other systems to solve Categorize and define for common understanding
- 14. 14 REQUIREMENT NON-FUNCTIONAL ENVIRONMENT INVALID FUNCTION DEGRADE EXISTING LATER Store Categorize Analyze STLC Database 500k+ results Analytics flow in 3 steps
- 15. 15 ➢ Who Am I? ➢ Background of bug analytics ➢ Can we automate it? ➢ What is bug analytics? Agenda
- 16. 16 REQUIREMENT NON-FUNCTIONAL ENVIRONMENT INVALID FUNCTION DEGRADE EXISTING LATER Store Categorize STLC Database 500k+ results What can be automated? Can be automated
- 17. 17 How to Automate? Analytics dashboard Practitest DB Jira DB STLC DB
- 18. 18 BEFORE AFTER Fill bugs in sheet, search on Ticket by manually Use script to export Ticket tickets with bug types, root cause Automation tip – Reduce data collecting time
- 19. 19 ➢ Who Am I? ➢ Background of bug analytics ➢ Can we automate it? ➢ What is bug analytics? Agenda
- 20. 20 Bug Analytics flow Step1 Major Category and QoQ comparison Step 2 RCA (Root cause analysis) Step 3 Jira ticket drill down
- 21. 21 • Divide projects into categories (APP , Web, Front, Back) • Analyzing individually makes result more accurate • Android iOS ver xx • Big Function 01 • Small Function 01 • Backend dashboard • Middleware Update Step1-1: Categorize for devices Step 1: Big category Step 2: RCA Step 3: Drill down
- 22. 22 UI bug ▼27% 41.5% 2.6% 9.3% 3.2% 6.4% 4.5% 1.6% 28.6% 0.3% 0.6% 44.1% 22.0% 13.6% 6.8% 6.8% 1.7% 1.7% 1.7% 0.0% 0.0% 0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0% 35.0% 40.0% 45.0% 50.0% Q2 Q3 Functional bug 3% Step 1: Major category Step 2: RCA Step 3: Drill down QoQ comparison Step1-2: Categorize by Major category and QoQ comparison
- 23. 23 Improvement 2021 2022 1H 2022 2H Data Collection ✅ ✅ ✅ Data Compare ✅ ✅ ✅ Chart Presentation ✅ ✅ ✅ Root Cause Analysis ✅ ✅ Quarter Detail compare ✅ ✅ Improvement Summary ✅ ✅ Analyzing by Categories ✅ QoQ comparison ✅ Ticket ticket detail drilldown ✅ Custom observation and suggestion ✅ Improvement History
- 24. 24 1 1 2 4 11 16 18 34 DEPLOYMENT ISSUE TESTING MISTAKE UNIT TEST MISS INCORRECT REQUIREMENT MISSING IMPLEMENTATION DOW N TIME EXTERNAL INCORRECT CODING Grand Total Step 1: Major category Step 2: RCA Step 3: Drill down Step2: RCA (Root cause analysis for Major category)
- 25. 25 Reference: Final report of each projects Genchi- Genbutsu Step3-1: Ticket drill down Step 1: Big category Step 2: RCA Step 3: Drill down Basic Data: sum, fluctuation of bug type, root cause
- 26. Why why why 26 On the top page reservation calendar icon was not clickable. Deployment issue Why no code review Why couldn’t find in Unit testing Source wasn’t deployed No code review Missing implementation and updating sheet No unit testing Issue Expected Why Step3-2: Ticket drill down on real world Step 1: Big category Step 2: RCA Step 3: Drill down
- 27. 27 Judge projects quality by sum of bug, bug ratio: Refer more condition to judge Condition Good Quality Bad Quality Bug ration below 3% 〇 × Bug ration over 3% × 〇 Can skip R2 〇 × Few bug tickets 〇 × Condition Good Quality Bad Quality Bug ration below 3% 〇 × Bug ration over 3% × 〇 Can skip R2 〇 × Few bug tickets 〇 × Release successfully 〇 × Roll back even no issue reported from QA × 〇 High bug ratio but no hotfix or prod issue △ △ Step3.3: Refer PROD ISSUE, counts of Hotfix / Roll back additionally to judge project quality Earlier Now
- 28. 28 Only share suggestion but don’t know if it’s implemented Add result column to trace action plan progress Step 3.4: TRACK result carefully Previous quarter This quarter
- 29. Can we automate bug analytics: Partially! Quantitative analysis By Automation Qualitative analysis By Manual Solution STLC DB Jira DB Practitest DB
- 30. Thank you