Test Strategy-The real silver bullet in testing by Matthew Eakin
Download as PPTX, PDF1 like474 views
This document provides an overview of creating a testing strategy. It begins with explaining why a testing strategy is important, as testing accounts for a large portion of IT budgets. It then discusses the key questions a testing strategy should answer: what to test, where to test, when to test, how to test, and who will test. The document outlines a process for creating a testing strategy, including assessing the current state, defining a future vision, and creating a roadmap to get from the current to future state. It provides examples of what to include under each section of the strategy, such as system architecture under "what to test" and test environments under "where to test". Overall, the document provides guidance on developing a
Test Strategy-The real silver bullet in testing by Matthew Eakin
- 1. NOTICE: PROPRIETARY AND CONFIDENTIAL This material is proprietary to and contains trade secrets and information which is solely the property of Centric Consulting, LLC. It is solely for the Client’s internal use and shall not be used, reproduced, copied, disclosed, transmitted, in whole or in part, without the express consent of Centric Consulting, LLC. © 2016 Centric Consulting, LLC. All rights reserved. Testing Strategy: The Silver Bullet Presented by: Matthew Eakin, Test Strategist Matt.Eakin@centricconsulting.com @MatthewEakin MatthewEakin
- 2. Agenda •Why a Testing Strategy? •What is a Testing Strategy? •What – needs to be tested? •Where – is it going to be tested? •When – is it going to be tested? •Who – is going to test it? •How – are they going to test it? •Steps to creating a solid Test Strategy: 1. Assessment of current state 2. Vision of where you want to be 3. Create Testing Strategy 4. Create Roadmap (based on Strategy) to get from current to future state 2
- 3. Why a Testing Strategy? • 29% of an IT’s budget is typically spent on testing. This is expected to grow to 34% in the next several years • CIO’s are transitioning organizations towards Bi-Modal IT shops, maturing agile into DevOps. Integration of quality activities and the promotion of whole team accountability of quality is paramount to successful transformations. • Meanwhile, 40% of organizations lack sufficient transparency into QA activities and QA evaluations • Agile and DevOps requires a strategic focus on automation to reduce feedback times while improving coverage • Better defect detection (72%) • Reduced cycle times (69%) • Reduced testing costs (67%) • Better test re-use (66%)
- 4. Driving Forces in IT today Historic Testing Response: 4 With Less Do More Historic Testing Response: “Add more Testers” “Buy more tools”
- 5. What is a Test Strategy What needs to be tested
- 6. Strategy vs. Plan Test Strategy Test Plan Written once Re-Written for each Project Written for each application or system Re-Written for each Project Explains WHY? Why use a specific tool? Why do you need x resources? Why you need a dedicated QA env? Details Who tests, What they test, When they test and Where they do the testing 6 “Planning is everything. Plans are nothing.” –Field Marshal Helmuth Graf von Moltke (“Moltke the Younger”) –Often attributed to Dwight D. Eisenhower
- 7. 5 Basic Questions •What – needs to be tested? System Architecture •Where – is it going to be tested? Environment •When – is it going to be tested? Cadence •How – is it going to be tested? Tools •Who – is going to test it? Resources 7
- 8. System Architecture What needs to be tested
- 9. 9
- 10. User Behavior Tests 10 •Acceptance Tests •Functional Tests •End-to-End Tests •UI Smoke Tests •Multi-browsers •Mobile
- 11. Middle-Tier Tests 11 •API Tests •Security Tests •Some Performance Tests •Some Integration Tests
- 12. Server-Side Tests 12 •Database Tests •Some Integration Tests •Unit Tests
- 14. Brian Marick’s Testing Quadrants
- 15. Environment Where is it going to be tested?
- 16. 16
- 17. Environment Purpose What test(s) are best suited for which environment? 17 DEV CI Test/QA UAT Perf Unit X Smoke X X X X Integration X Functional X X API X X E2E X Behavior X X Mobile X X Multi- browser X X Look & Feel X
- 18. Deeper Dive When talking about environments, numerous other discussions will happen: •Databases •Test data needed •Clients needed in each environment •Continuous Integration (CI) •How will code, configuration and databases get migrated from environment to environment? •Smoke tests •Code versioning – which code is in which environment? •Virtual machines (VM’s)? •Integration with external systems 18
- 19. Cadence When is it going to be tested?
- 20. •Run test when Functionality is ready? Daily? End of Sprint? Prior to Release? 20
- 21. Tools How are the tests going to be executed?
- 22. Needs Checklist KEY: you don’t need 1 tool that “does it all.” But you are going to need tool(s) to do what you need done. What have you learned thus far? •Types of tests needed to be executed (API tests, DB validation, UI tests, Integration tests, etc.) •Will you be using CI to migrate code? •When will the tests be executed during a Sprint (cadence)? •Will you need to run a single test? Group of tests? Smoke tests? •Collaboration/Review on scripts? 22
- 23. 23
- 24. Resources Who is going to be executing the tests?
- 25. Skillsets & Ownership •Do you have the skillset internally to execute what needs tested? •Current team, potential contactors, external teams •Dunning-Kruger Effect -> you need to be honest and objective •Who owns testing? •Development? Architects? Business? Testing? KEY: It doesn’t matter who “owns” testing. Everyone has some testing which needs done to ensure the product is great. You need to own the coordination of execution (a.k.a. the Strategy) 25
- 26. Additional Strategies Important areas which need to be addressed
- 27. What else… Out of these initial discussions, quite a few additional topics will come up: •Reporting – when is the best time to report your testing findings? •Metrics – what are the best metrics? •Internal (team), external (product owner) •Defect management – best way to get defects triaged and in the hands of those who can fix them •Training – testers, analysts, developers, product owners •Use of contractors •Coaching, training •Skill & resource gaps fill-in •Use of Off-shore •Is there a place for it? 27
- 28. Where to start
- 29. 3 Things you need to figure out… 1.Where are you? Current state… 2.Where do you want to go? Future state… 3.How are you going to get from Current to Future state? Roadmap… 29
- 30. Current State How to determine where you are…
- 31. Assessments •Many available on internet •Google “software testing assessment”, “Testing Maturity Model”, “test process assessment” •Many consulting companies have canned assessments •All good, but can be costly Key: Do the right kind of assessment to identify your problems •Manual testing assessment, test automation assessment, performance assessment, etc.. Key: Be Objective/honest •Very difficult to do but critical 31
- 32. What is an assessment? 32
- 33. Breakdown of a Category 33
- 34. Future State Where do you want to be…
- 35. Again, Look to the Assessment •You know where you are. How high up do you want to go? •How high up can you realistically go? In 1 year? In 2 years? In 5 years? 35
- 36. Roadmap How to get from Current to Future State
- 37. Again, Look to the Assessment •If you want to attain Optimizing maturity, and are currently at Controlled, the Assessment can provide a roadmap to get from Current to Future state. 37
- 38. Test Strategy Document Putting it all together
- 39. Putting it all together •Putting a Strategy together is like putting together a 1,000 piece puzzle… …Without a picture 39
- 40. What to consider •Pain Points •What is causing the most testing pain? •Budget (testing and department) •What do you have the $$ to do? •Department Priorities •What does everyone else on the team feel is important? •Department Politics & Culture •What can you realistically get done? •What will people realistically do? •What is completely new? 40
- 41. Each Criteria •Documentation of current state •State the facts (good, bad, ugly) •Document what you found •Recommendations •How do you plan to improve? •Tools? •Personnel? •Training? •System resources? •When do you plan to improve? •Immediate (now), Short-Term (over the next few months), Medium- Term (1-2 years), Long-Term (future) 41
- 42. Create a Timeline (Backlog) •Immediate (what can be done now) •List out all the first steps in each criteria •Some criteria may have more than 1 •Prioritize what you want to tackle 1st, 2nd, etc.. •Short Term (what can be done in the near future) •List out all the “2nd” steps in each criteria, and any leftover 1st steps •Prioritize what you want to tackle 1st, 2nd, etc.. •Mid Term (what can be done in 1-2 years) •These take time to plan, get funding, get resources, etc.. •Long Term (completed will into the future) •Don’t waste too much time dreaming these up as they will most likely change by the time you get there 42
- 43. Start with the Biggest Pain Points •What is causing the most pain? Focus on it first •Teams can only handle change in small doses Focus on 2-3 things at a time 43
- 44. Create a KanBan Board 44
- 45. How does this enable you to do more, with less?
- 46. Doing more •Entire stack/team testing •Focus testing efforts on high-risk/high-pain areas •Better view of all tests -> remove duplicates •Make a case for test automation •Make a case for better tools •Make a case for training 46
- 47. With Less •Make a case for open source tools •vs. current high-cost licensed tools •Better view of all tests -> remove duplicates •Make a case for consultants •More resources at reduced cost •Make a case for test automation •Make a case for off-shore resources 47
Editor's Notes
- #2: Alternate for Title Slide: slide 15
- #4: Improved Business Outcomes It’s no longer about rapid delivery, but rapid delivery of value Alignment of testing and business Faster feedback Lower costs
- #6: Alternate for slide 1
- #9: Alternate for slide 1
- #16: Alternate for slide 1
- #20: Alternate for slide 1
- #22: Alternate for slide 1
- #25: Alternate for slide 1
- #27: Alternate for slide 1
- #29: Alternate for slide 1
- #31: Alternate for slide 1
- #35: Alternate for slide 1
- #37: Alternate for slide 1
- #39: Alternate for slide 1
- #46: Alternate for slide 1