Advancing Testing Using Axioms
Download as PPTX, PDF1 like1,786 views
The document discusses how test axioms can be used to advance testing practices. It introduces 16 proposed test axioms grouped into stakeholder, design, and delivery axioms. The axioms represent critical thinking processes for testing any system. The document discusses how the axioms can help testers design test strategies, assess improvement opportunities, and define needed skills. It also proposes a "first equation of testing" that separates axioms, context, values, and thinking to allow for different valid approaches. Additionally, the concept of "quantum testing" is introduced to discuss assigning significance to tests rather than defining their value, which can only be determined by stakeholders.
Advancing Testing Using Axioms
- 1. THEAXIOMSTESTINGOFAdvancing Testing Using AxiomsPaul Gerrard
- 2. Paul GerrardPaul is a consultant, teacher, author, webmaster, programmer, tester, conference speaker, rowing coach and a publisher. He has conducted consulting assignments in all aspects of software testing and quality assurance, specialising in test assurance. He has presented keynote talks and tutorials at testing conferences across Europe, the USA, Australia, South Africa and occasionally won awards for them.Educated at the universities of Oxford and Imperial College London, Paul was the founding chair of the BCS ISEB Testing Certificate Scheme and a member of the Working Party that produced BS 7925 – the Component Test Standard. Currently, he is Principal of Gerrard Consulting Limited and is the host of the UK Test Management Forum.
- 3. AgendaAxioms – a Brief IntroductionAdvancing Testing Using AxiomsFirst Equation of TestingTest Strategy and ApproachTesting ImprovementA Skills Framework for TestersQuantum TestingClose
- 4. Test AxiomsFormulated as a context-neutral set of rules for testing systemsThey represent the critical thinking processes required to test any systemThere are clear opportunities to advance the practice of testing using themTesters Pocketbook: testers-pocketbook.comTest Axioms Website test-axioms.com
- 5. How can we use Test Axioms?Test Axioms are not beginners guides
- 6. They can help you to think critically about testing
- 7. They expose flaws in other people’s thinking and their arguments about testing
- 8. They generate some useful by-products
- 9. They help you to separate context from values
- 11. First Equation of Testing, Testing Uncertainty Principle, Quantum Theory, Relativity, Exclusion Principle...
- 12. You can tell I like physics16 Proposed Axioms(in three groups)Stakeholder, (Test) Design and (Test) DeliveryYou have the little book
- 14. Testing needs stakeholders (p64)Summary:Identify and engage the people or organisations that will use and benefit from the test evidence we are to provideConsequence if ignored or violated:There will be no mandate or any authority for testing. Reports of passes, fails or enquiries have no audience.Questions:Who are they?Whose interests do they represent?What evidence do they want?What do they need it for?When do they want it?In what format?How often?
- 16. Test design is based on models (p68)Summary:Choose test models to derive tests that are meaningful to stakeholders. Recognise the models’ limitations and the assumptions that the models makeConsequence if ignored or violated:Tests design will be meaningless and not credible to stakeholders.QuestionsAre design models available to use as test models? Are they mandatory?What test models could be used to derive tests from the Test Basis?Which test models will be used?Are test models to be documented or are they purely mental models?What are the benefits of using these models?What simplifying assumptions do these models make?How will these models contribute to the delivery of evidence useful to the acceptance decision makers?How will these models combine to provide sufficient evidence without excessive duplication?How will the number of tests derived from models be bounded?
- 18. Advancing Testing Using Axioms
- 19. First Equation of TestingAxioms+ Context+ Values+ Thinking =Approach
- 20. Why is the equation useful?Separation of Axioms, context, values and thinkingTools, methodologies, certification, maturity models promote approaches without reference to your context or valuesNo thinking is required!Without a unifying test theory you have no objective way of assessing these products.
- 21. One context, multiple approachesGiven context, practitioners can promote different approaches based on their valuesValuesare preferences or beliefsPre-planned v exploratoryPredefined v custom processRequirements-driven v goal-basedStandard documentation v face-to-face comms.Some contexts preclude certain practices“No best practices”
- 22. Axioms allow (ensure) different approaches and expose positionsSeparating axioms, context and values clarifies positions, for example:‘Structured’ (certified?) test advocates have little (useful) to say about Agile contextsExploratory test advocates have little (useful) to say about contract/requirements-based acceptanceThe disputes between these positions is more about valuesthan practices in contextIs a consultant recommendation best for the stakeholders or the consultant?
- 23. Test Strategy and ApproachStrategy is a thought process not a document
- 24. Contexts of Test StrategyAxiomsCommunicationEarly TestingRisksDe-DuplicationTestStrategyOpportunitiesGoalsAutomationCultureContractUser involvementConstraintsHuman resourceArtefactsSkillsEnvironmentProcess(lack of?)Timescales
- 25. IEEE 829 Test Plan OutlineTest Plan IdentifierIntroductionTest ItemsFeatures to be TestedFeatures not to be TestedApproachItem Pass/Fail CriteriaSuspension Criteria and Resumption RequirementsTest DeliverablesTesting TasksEnvironmental NeedsResponsibilitiesStaffing and Training NeedsScheduleRisks and ContingenciesApprovalsBased on IEEE Standard 829-1998
- 26. I’m no fan of IEEE 829Used as a strategy checklistScarily vague (don’t go there)Used as a documentation template/standardFlexible, not prescriptive, but encourages copy and edit mentality (documents that no one reads)But many many testers seek guidance onWhat to consider in a test strategyCommunicating their strategy to stakeholders and project participants
- 27. IEEE 829 Plan and AxiomsItems 1, 2 – AdministrationItems 3+4+5 – Scope Management, PrioritisationItem 6 – All the Axioms are relevantItems 7+8 – Good-Enough, ValueItem 9 – Stakeholder, Value, ConfidenceItem 10 – All the Axioms are RelevantItem 11 – EnvironmentItem 12 – StakeholderItem 13 – All the Axioms are RelevantItem 14 – All the Axioms are RelevantItem 15 – Fallibility, EventItem 16 – Stakeholder Axioms
- 28. A Better Test Strategy and PlanStakeholder ObjectivesStakeholder managementGoal and risk managementDecisions to be made and how (acceptance)How testing will provide confidence and be assessedHow scope will be determinedDesign approachSources of knowledge (bases and oracles)Sources of uncertaintyModels to be used for design and coveragePrioritisation approachDelivery approachTest sequencing policyRepeat test policiesEnvironment requirementsInformation delivery approachIncident management approachExecution and end-game approachPlan (high or low-level)ScopeTasksResponsibilitiesScheduleApprovalsRisks and contingencies
- 29. Testing ImprovementTest process improvement is a waste of time
- 32. Actuallyits 11(most were not software related)
- 33. The delusion of process models(e.g. CMM)Google search“CMM” – 22,300,000“CMM Training” – 48,200“CMM improves quality” – 74 (BUT really 11 – most of these have NOTHING to do with software)A Gerrard Consulting client…CMM level 3 and proud of it (chaotic, hero culture)Hired us to assess their overall s/w process and make recommendations (quality, time to deliver is slipping)40+ recommendations, only 7 adopted – they couldn’t changeHow on earth did they get through the CMM 3 audit?
- 34. “Test Process Improvement is a Waste of Time”Using process change to fix cultural or organisational problems is never going to workImproving test in isolation is never going to work eitherNeed to look at changing context rather than values…
- 35. Why you are where you areContext+ Values+ Thinking=Approach<- your context<- your values<- your thinking<- your approach
- 36. Where maturity models come fromContext+ Values+ Thinking=Approach<- someone else's<- someone else's<- someone else's<- someone else's
- 37. Making change happenAxioms+ Context+ Values+ Thinking=Approach<- recognise<- could change?<- hard to change<- just do some<- your approach
- 38. Using the axioms and questionsAxioms represent the critical things to think aboutAssociated questions act as checklists to:Assess your current approachIdentify gaps, inconsistencies in current approachQA your new approach in the futureAxioms represent the WHATYour approach specifies HOW
- 39. Eight stage change process (after Kotter)MissionCoalitionVisionCommunicationActionWinsConsolidationAnchoringChanges identified hereIf you must use one, this is where your ‘process model’ comes into play
- 40. A Skills framework for testersAxioms indicate WHAT to think about......so the Axioms point to SKILLS
- 41. Test design is based on models (p68)Summary:Choose test models to derive tests that are meaningful to stakeholders. Recognise the models’ limitations and the assumptions that the models make.Consequence if ignored or violated:Tests design will be meaningless and not credible to stakeholders.Questions:Are design models available to use as test models? Are they mandatory?What test models could be used to derive tests from the Test Basis?Which test models will be used?Are test models to be documented or are they purely mental models?What are the benefits of using these models?What simplifying assumptions do these models make?How will these models contribute to the delivery of evidence useful to the acceptance decision makers?How will these models combine to provide sufficient evidence without excessive duplication?How will the number of tests derived from models be bounded?
- 42. Test design and modelling skillsA tester needs to understand:Test models and how to use themHow to select test models from fallible sources of knowledgeHow to design test models from fallible sources of knowledgeSignificance, authority and precedence of test modelsHow to use models to communicateThe limitations of test modelsFamiliarity with common modelsIs this all that current certification provides?
- 43. Training and certification must changeIntellectual skills and capabilities are more important than the clerical skillsNeed to re-focus on:Testing thought processes (Axioms)Testing Stakeholder relationship managementTesting as an information provision serviceGoal and risk-based testingReal-world examples, not theoryPractical, hands-on, real-world training, exercises and coaching.
- 44. Quantum TestingIf evidence arrives in discrete quanta......can we assign a value to it?
- 45. How testing builds confidenceAs tests are run, every individual test has some significanceSome tests expose failures but ultimately we want all tests to PASSWhen all tests pass – the stakeholders are happy, aren’t they?Can we measure confidence by counting tests?Not really...
- 46. The value of a test varies by…Coverage model:A test could cover one or hundreds of functional conditions, ten thousand program statements or tenObjective:Criticality of the business goal it examplesCriticality of the risk it informsPrecedent:The first end-to-end test pass is significantThe 100th variation of a similar test is less significantFunctional dependence:A test of shared functionality used thousands of times per hour could be much more important than a peripheral feature used once/dayStakeholder:Are customers tests more or less significant than supplier tests?Context:The same test run at different times in different environments can have different value.
- 47. Quantum TestingOnly a stakeholder can assign a value to a test (but that is very hard thing to do) – (but see relativity)
- 48. Testers can’t quantify value, but could define significance
- 49. A test is significant (to stakeholders) if it:
- 50. Can be related to a meaningful test objective
- 51. Increases coverage with respect to a meaningful test model
- 52. Is considered in an acceptance decision (at any level)
- 53. Significance is a Boolean but could be 0 or 1
- 54. The number of insignificant tests should be zero.Assessing significanceSignificance can only be assessed by testers if:Our test objectives, models, coverage goals are meaningful (to stakeholders) orTesters are authorised to create their own objectives, measures and coverage goals orTesters are their own stakeholderTesters need a close, trusting relationship with their stakeholdersor authorised autonomyE.g. exploratory testing won’t work if stakeholders do not allow autonomyTesters should not ‘go it alone’.
- 55. The significance of significanceTest coverage models and goals that generate uniform distributions of tests are inefficient and uninformativeWe need more and better test modelsModels that are meaningful in contextSignificance varies with context and can be used to explain whye.g. some tests aren’t useful as regression testsHow much testing is enough?Can never be answered by coverage alone.
- 56. Using significance (as booleans)A test objective could be deemed as met when a set of significant tests are passedTest objectives can often be aligned with business goals (or risks) e.g.:Faultless end to end or straight-through processing in an ERP systemCustomer order capture, money-in or complete financial reconciliations without failureCould the business put a value on these goals being achieved?Perhaps the business case provides this level of information?
- 57. Our current understanding of test value measurementIn terms of a mathematical treatment of testing we are some way offCompare with:Understanding of heat and energy in 18thcentury?Economics in the early 20thCentury?A complete theory of test value may never be achievedQuantum testing suggests that:Stakeholders judge value (relatively speaking)Testers judge significance.
- 58. CloseAxioms are context-neutral rules for testingThe Equation of TestingSeparates axioms, context, values and thinkingWe can have sensible conversations about processAxioms and associated questions provide context neutral checklists for test strategy, assessment/improvement and skillsQT separates significance from valueCould it tell us, “how much testing is enough”?Right now, we can ‘feel the heat’, but not measure temperature.