There's no time to test, can you just automate it? by Anna Heiermann
Download as PPTX, PDF0 likes313 views
Anna Heiermann discusses lessons learned from past testing failures and advocates for a risk-based testing approach. A risk-based strategy involves identifying product risks, prioritizing test cases based on risk, and communicating the test plan to stakeholders. This helps ensure the highest priority and most critical areas are tested thoroughly. When risks are missed, it can lead to catastrophic bugs affecting users and lost revenue. With a risk-based approach, testing is targeted efficiently to reduce risks while keeping stakeholders informed.
There's no time to test, can you just automate it? by Anna Heiermann
- 1. There’s No Time to Test Can’t You Just Automate It? A Guide to Managing Test Strategy from a Risk Based Perspective Anna Heiermann – Senior Quality Analyst Consultant - HMB
- 2. QA Career started 10 years ago Found career by accident Senior Consultant with HMB Married, 2 Dogs Interested in Finding Ways to Manage Stress 90% 70% 96% 98% Stress Management Scale Powerlifting Family/Puppies/Friends Music Risk Management Anna Heiermann About Me
- 3. The Worst Bug The Worst Bug I Didn’t Catch • Worst Event in client history • Requested to build a site with a product form in one day • Bug: Form was not saving a very critical field in several cases • Successful submit followed by going back to form using back browser button • Result: Thousands of impacted forms did not contain most critical piece of information Aftermath • Lots of “Lessons Learned” • Unable to shake off the impacts of what happened with what wasn’t caught • Worried about missing something very critical for several projects to come • Easily overwhelmed with what was not in my control
- 4. No Communication of Test Coverage to Client, No Test Strategy Little Release Management No Risk Based Discussions What Happened?
- 5. 5 Software Disasters Unfortunate Defects with Consequences
- 6. 6 Deadly Miss – 6/11/2000 Radiation Therapy Software : Data entry miscalculation when entering beam block data and interpreting beam block data caused overexposure No malfunction, but data caused overexposure • Lack of requirements (procedures) • Lack of testing Result: FDA Statement 2017 • 28 patients overexposed • 20-100 percent over prescribed dose • 9 patients died, 5 confirmed radiation overexposure • Remaining patients expected to develop complications https://www.fda.gov/Radiation- EmittingProducts/RadiationSafety/AlertsandNotices/ucm116533. htm
- 7. 7 The Age of Airline Outages British Airways - 2017 • Systems failure impacted check-in and operational systems • All flights out of Heathrow cancelled for days til resolved Delta – 2016, 2017 • Electrical component failure led to transformer shutdown • Transformer provided power to data center, servers weren’t connected • Lost 100 million in Revenue, suffered same glitch a few months later United Airlines – 2017 • “Computer problem” grounded hundreds of flights causing delays – several disruptions from June - October Southwest - 2017 • 2000 flights canceled after failed network router outage https://www.reuters.com/article/us-ual-flights/united-flights-delayed- after-computer-glitch-grounds-u-s-planes-idUSKBN15705C https://www.npr.org/sections/thetwo-way/2017/05/29/530587512/on- day-3-of-cancellation-chaos-british-airways-edges-closer-to-recovery https://www.cnbc.com/2017/05/27/british-airways-says-computer- outage-causing-global-delays.html
- 8. 8 Starbucks Goes Dark - 2017 Cause: Overnight store register technology update caused store outages in Canada and US • Many stores closed • A few stores only offered drip coffee and took cash • Some handed out free cups of coffee Result: Starbucks Survives, lost revenue • Unexpected outage caused no revenue for several hours • Free coffee – At least customers will return as some stores tried to make it right • Still under investigation – exposure of scammers finding ways to get into Starbuck Customer accounts and drain balances http://fortune.com/2017/05/16/starbucks-computers/
- 9. Risk Based Testing ISQTB Definition An approach to testing to reduce the level of product risks and inform stakeholders of their status, starting in the initial stages of a project. It involves the identification of product risks and the use of risk levels to guide the test process Reference Page 54 https://www.istqb.org/downloads/send/2-foundation-level-documents/3-foundation-level-syllabus-2011.html4 Benefits of Risk Based Testing Collaboration on Strategy and Approach with QA, the Project Team, and client Client gains more education and awareness on high priority and critical items from a technical and business perspective Better trust/confidence with product quality and QA Team Potential to help save/meet budget or time restraints Risk Based Testing “Well, the code broke and we’re not sure of what the fix has impacted, so you’ll have to test everything”
- 10. • Panama Radiation Incident • “Changes” no matter how big or small still can cause catastrophic consequences if testing is not a part of the change • Risk Based Management with a Risk Based Test approach is a MUST even if the change is a last minute “quick” thing • Airlines • Culprits may be presented with the hardware, but affect the software. • Planning testing from a Risk perspective around outages can verify whether/not the software will withstand the glitch • Starbucks • Not life or death, but outages are lost revenue and more expensive to fix and test in production • “Change” to existing products are just as critical to test as a brand new application • “Changes” with a Risk Based Test approach will increase the ability to identify critical defects • RBT encourages the entire team to think critically to identify what should be tested first • My Worst Bug • Should have had a better understanding of the functionality behind the scenes • Better tests specifically around the behavior of the form with navigation would have increased the chance the defect would have been caught • Business would have had more involvement and visibility in our testing process • Risk Based Strategy would increase overall client/dev team communication and increased collaboration on what to test W h a t w e c a n l e a r n f r o m : Lessons Learned
- 11. 11 v Venting Sessions: At the Bar Product Knowledge How well do you know what you’re testing? Do you RAM? Do you use a Risk Assessment Matrix to determine the probability of high risk areas and discuss these with your team as you plan? Communication What’s your test approach? Who is helping you? Is your team and client onboard? Execution What are the state of your Risk BasedTestcases? Are you maintaining traceability? Do you use RBT for every level of testing?
- 12. RBT Tip #1: Know the Code In order to Better Understand Your Product Sit down with the senior developer/lead tech to understand both front end and back end functionality. Benefit – Better test cases and test strategy approach. Assess and determine what aspects behind the scenes share the same code. Understanding the way a product is coded will help uncover potential high risk items that have shared code and allow for better timing around when something should be tested.
- 13. The Challenge Continuous deployments with new templates caused testing to be overwhelmed with a high number of pages to validate content, images, browser testing, etc. Functionality was equal if not lower priority as a result. CMS Session enabled QA to reprioritize what was critical than what they assumed was critical and decreased test time for shared features/pages No Understanding into what CMS was Too many repeat test cases to test all pages using same features and templates CMS and RBT Risk Based Testing against CMS and Site Redesign Ý
- 14. RBT Tip #2: Build a RAM In order to PRIORITIZE RISK RAM = RISK ASSESSMENT MATRIX Meet with Product Owner/client, development team and determine all critical aspects of the application Prioritize the most critical aspects that are to be tested first. What’s most important to your dev team and your client?
- 15. The Challenge Tech Debt card was originally prioritized as a “low impacting” change. However, the way it was implemented had an impact on every single table the team had coded from the beginning of a project. RAM designed with Business and Dev team to assess most critical pieces of data. Matrix helped identify these and then the entire team helped in testing these areas to meet timeline. “T” Shaped Team! Short timeline to deployment, not enough QA Resources Too many test cases in regression bed that wouldn’t cover all the critical concerns from the change Creating a RAM Risk Based Testing using Risk Assessment Matrix Ý D E E P BROAD Able to apply knowledge across situations Functional/SpecificSkillSet
- 16. RBT Tip #3: Communicate the Plan In order to Set Test Scope and Expectation Build a Test Approach defining the test coverage that will be provided based on the outcome of the RAM session. List out testing types that will be done and others that will not (for example, boundary testing is in scope performance is not). Exploratory testing and a UAT even if it’s small, should always be considered. Communicate the plan to the client/business and project team. Everyone needs to be aware and on board with the plan and their role within the plan. Test Plan should include: Execution Process, Defect handling/reporting, daily updates on bugs and progress, Names and Roles helping with testing, instructions for “test helpers”.
- 17. RBT Tip #4: Execute with RBT In order to Meet the Project Need Test highest priority items first followed by critical bug retests, then lower priority items Utilize existing automation where necessary if accessible: Regression tests, browser checks, performance based, etc Hold checkpoints with business throughout allowing them to view/use the product to ensure they are getting what they asked for. There’s a risk they will want more/changes. When this happens, start the Risk BasedApproach discussions again!
- 18. 1 2 3 4 QA became a Top Priority QA became a Trusted Partner QA Test Strategies Improved Decrease in Critical Production Issues QA was planned into Tech Debt cards Trusted business partner Pulled in to provide insight on future projects/enhance ments during planning “Test Everything” was now targeted to critical first followed by impacted applications Increased communication at the start of story card work More plan/test time Developer test coverage increased RBT Wins How a RBT Approach benefits the Team
- 19. THANK YOU
Editor's Notes
- #10: Start with discussing the episode where the fridge broke and his friends came in and he had eaten everything. So similar to the common phrase, well the code broke and we’re not sure what the fix impacted, so you’ll have to test everything
- #12: I find myself asking these common things anytime I’m out to lunch/happy hour with fellow testers. If we were having a drink and you were venting, these are some quick topics I’d bring up
- #14: Client story