HBT - A Revolutionary Approach to Testing Software
1 like6,185 views
The document presents Hypothesis Based Testing (HBT) as an innovative approach to software testing that seeks to identify and eliminate defects through a scientific methodology inspired by natural processes. It emphasizes the importance of cleanliness criteria and potential defect types to meet user expectations and optimize testing outcomes. A case study illustrates the effectiveness of HBT, showing significant improvements in defect detection and test coverage compared to conventional testing methods.
HBT - A Revolutionary Approach to Testing Software
- 1. Hypothesis Based Testing (HBT) A revolutionary approach to testing software T Ashok ash@stagsoftware.com in.linkedin.com/in/AshokSTAG ash_thiru Webinar: 22 June 2011, 1430-1530 IST
- 2. How can we seek inspirations from nature to deliver “Clean Software”? In the next hour, let us look at the act of testing differently and see how we can think better and test scientifically? © 2011. STAG Software Private Limited. All rights reserved.
- 3. Quality is... “... at least meeting expectations” Meeting expectations implies “satisfying properties” i.e. Expect certain properties to be satisfied. Properties are impeded by “potential defects” © 2011. STAG Software Private Limited. All rights reserved.
- 4. Properties of matter Physical & Chemical properties of matter allow us to: ... classify ... understand behaviors, interactions ... enable checking purity Think! How can we use a similar train of thought to identify “properties of cleanliness” and then “types of defects”? http://www.science.wellsresearch.org/wp-content/uploads/2010/02/CBScienceHSPhysicalScienceSt2.jpg © 2011. STAG Software Private Limited. All rights reserved.
- 5. “Properties of the system” Cleanliness criteria End user expectations “affected by” Issues in specifications, Potential Defect Types (PDT) structure, environment and behavior © 2011. STAG Software Private Limited. All rights reserved.
- 6. Expectations “Properties of the system” Needs Cleanliness criteria Features Environment “impedes” Behavior Structure Potential Defect Types (PDT) Material Expectations delivered by Needs (Requirements) via Features that display Behavior constructed from Materials in accordance to a Structure in a given Environment © 2011. STAG Software Private Limited. All rights reserved.
- 7. Fractional Distillation A technique to separate mixtures that have components of different boiling points Think! There are a variety of defect types that may be present in the system. To optimally uncover the From : http://withfriendship.com defects, can we separate these types of defects on the basis of certain properties and optimally uncover the defects? © 2011. STAG Software Private Limited. All rights reserved.
- 8. Fetal growth - Delivering systems (Babies too!) Visualizing health : - Size, Shape, Parts - Types of issues not present Think! How can we measure the ‘intrinsic quality’ at a stage? Quality Growth At each stage certain types of Source :http://www.environment.ucla.edu/media/images/Fetal_dev5.jpg defects detected & removed. Quality grows. Can we chart this as “cleanliness index”? © 2011. STAG Software Private Limited. All rights reserved.
- 9. Sherlock Holmes Focus on intellectual power to solve mysteries. Strong application of deductive logic. Think! Can we hypothesize as to what can go wrong and prove/disprove? Hypothesize potential defect types and prove presence of instances. If they are not present, can we reason their absence? © 2011. STAG Software Private Limited. All rights reserved.
- 10. Test methodologies focus on activities that are driven by a process which are powered by tools, yet successful outcomes still depend a lot on experience. Typically methodologies are at organizational level. On the other hand HBT ... is a personal scientific methodology enabled by STEMTM , a defect detection technology to deliver “Clean Software” © 2011. STAG Software Private Limited. All rights reserved.
- 11. Scientific approach to uncovering defects Cleanliness criteria What is the end user expectation of “Good Quality”? Potential Defect Types What types of issues can result in poor quality? Evaluation Stage When should I uncover them? Test Types How do I uncover them? Test Techniques What techniques to generate test cases? Scenarios/Cases What are the test cases? Are they enough? Scripts How do I execute them? Metrics & Management How good is it? How am I doing? © 2011. STAG Software Private Limited. All rights reserved.
- 12. HBT: Quality Levels, Aspects & Issues Aspect Issues Usage L9 End user value User flows, experience Environment, Data, Logic L8 Clean Deployment Compatibility, migration Structure, Logic, Usage L7 Attributes met Performance, security, volume, load... Environment L6 Environment cleanliness Resource leaks, Compatibility... Logic, Environment L5 Robustness Error handling Data, Logic L4 Behavior correctness Functionality L3 Structural integrity Structure Internal structural issues Structure, Usage L2 Input interface cleanliness UI issues Data L1 Input cleanliness Input data handling © 2011. STAG Software Private Limited. All rights reserved.
- 13. HBT - A Quick Introduction SIX stages of powered EIGHT disciplines DOING by of THINKING D8 D1 S6 S1 Analysis & Business value Assess & Understand management understanding ANALYSE EXPECTATIONS D7 D2 Execution & Defect D8 D1 reporting STEM Core hypothesis Tooling D7 D2 S5 STEM Understand 32 core SUPPORT S2 D6 D3 CONTEXT concepts D5 D4 Strategy & Visibility planning D3 D6 Devise Formulate HYPOTHESIS Tooling Test design PROOF S4 S3 D5 D4 HBT powered STEM Personal test methodology by Defect detection technology © 2011. STAG Software Private Limited. All rights reserved.
- 14. D1 Business value understanding D2 Defect hypothesis D3 Test strategy & planning Landscaping EFF model Orthogonality principle Viewpoints (Error-Fault-Failure) Tooling needs assessment Reductionist principle Defect centricity principle Defect centered AB Interaction matrix Negative thinking Quality growth principle Operational profiling Orthogonality principle Techniques landscape Attribute analysis Defect typing Process landscape GQM D4 Test design D5 Tooling Reductionist principle Automation complexity Input granularity principle 32 core assessment Box model Minimal babysitting Behavior-Stimuli approach concepts principle Techniques landscape Separation of concerns Complexity assessment Tooling needs analysis Operational profiling D6 Visibility D7 Execution & Reporting D8 Analysis & Management GQM Contextual awareness Gating principle Quality quantification Defect rating principle Cycle scoping model © 2011. STAG Software Private Limited. All rights reserved.
- 15. How is HBT different? from other methodologies... Goal al p ic Powered by experience Ty drives T Activities HB ...................................... detection technology Powered by defect ...................................... ...................................... Activities ...................................... ...................................... (STEM) ...................................... ...................................... ...................................... ...................................... ...................................... ...................................... ...................................... hopefully results in Goal © 2011. STAG Software Private Limited. All rights reserved.
- 16. Hypothesis Based Testing (HBT) S1 Cleanliness criteria Potential defect types S3 S2 Staged & purposeful Expectations detection S4 Complete test cases S6 Goal directed measures Sensible automation S5 © 2011. STAG Software Private Limited. All rights reserved.
- 17. Cleanliness criteria Potential defect types Clear Baseline S1, S2 Staged & purposeful detection Expectations Set a clear goal for quality Complete test cases Example: Clean Water implies Goal directed measures Sensible automation 1.Colorless 2.No suspended particles 3.No bacteria 4.Odorless What information(properties) can be used to identify this? ...Marketplace,Customers, End users ...Requirement(flows), Usage, Deployment ... Features, Attributes ...Stage of development, Interactions ... Environment, Architecture ... Behavior, Structure © 2011. STAG Software Private Limited. All rights reserved.
- 18. Cleanliness criteria Potential defect types A goal focused approach S3 to cleanliness Staged & purposeful detection Identify potential defect types that Expectations can impede cleanliness Complete test cases Example: Data validation Timeouts Goal directed Sensible automation Resource leakage measures Calculation Storage Presentation Transactional ... Scientific approach to hypothesizing defects is about looking at FIVE Aspects - Data, Logic, Structure, Environment & Usage from THREE Views - Error injection, Fault proneness & Failure Use STEM core concepts > Negative thinking (Aspect) > EFF Model (View) “A Holmes-ian way of looking at properties of elements” © 2011. STAG Software Private Limited. All rights reserved.
- 19. Cleanliness criteria Potential defect types Levels, Types & Techniques - STRATEGY Staged & purposeful detection S4 NINE levels to Cleanliness Expectations L9 End user value Complete test cases L8 Clean Deployment Goal directed L7 Attributes met Sensible automation measures L6 Environment cleanliness Quality Levels L3 L5 Robustness PDT7 PDT6 L4 Behaviour correctness L2 PDT5 Test Techniques (T1-T4) PDT4 L1 PDT3 TT5 T4 L3 Structural integrity PDT2 PDT: PDT1 Potential Defect Types TT4 L2 Input interface cleanliness T3 TT3 L1 Input cleanliness Test Types PDT7 TT5 PDT6 TT2 T2 TT4 PDT5 PDT4 TT1 T1 TT3 PDT3 TT2 PDT2 TT: TT1 PDT1 Test Types “Fractional distillation of bug mixture” © 2011. STAG Software Private Limited. All rights reserved.
- 20. Cleanliness criteria Potential defect types Countable test cases & Fault coverage Staged & purposeful detection Expectations Use STEM Core concepts > Box model Complete test cases > Behaviour Stimuli approach S4 > Techniques landscape Goal directed > Coverage evaluation Sensible automation measures to - Model behaviour - Create behaviour scenarios - Create stimuli (test cases) Test Scenarios/Cases R1 PDT1 Irrespective of who designs, #scenarios/ TS1 TC1,2,3 R2 PDT2 cases shall be same - COUNTABLE TT R3 TS2 TC4,5,6,7 PDT3 Requirements & Fault traceability That test cases for a given requirement shall have the ability to detect specific types of defects Guarantee test adequacy. FAULT COVERAGE Guarantee implies that the means to the end is rational & provable © 2011. STAG Software Private Limited. All rights reserved.
- 21. HBT Test Case Architecture Organized by Quality levels sub-ordered by items (features/modules..), Level segregated by type, ranked by importance/priority, Item sub-divided into conformance(+) and robustness(-), classified by early (smoke)/late-stage evaluation, Type tagged by evaluation frequency, linked by optimal execution order, classified by execution mode (manual/automated) Priority Focus Stage A well architected set of test cases is like a effective bait that can ‘attract‘ defects in the Frequency system. Order It is equally important to ensure that they Mode are well organized to enable execution optimisation and have the right set of information to ensure easy automation. © 2011. STAG Software Private Limited. All rights reserved.
- 22. Cleanliness criteria Potential defect types Focused scenarios + Good Automation Architecture Staged & purposeful detection Expectations Level based testcriteria Cleanliness scenarios yield Complete test cases shorter scripts that are more flexible for change and S5 easily maintainable. Goal directed Sensible automation measures L9 End Expectations user value L8 Clean Deployment L7 Attributes met L6 Environment cleanliness L5 Robustness L4 Behavior correctness L3 Structural integrity L2 Input interface cleanliness L1 Input cleanliness © 2011. STAG Software Private Limited. All rights reserved.
- 23. Cleanliness criteria Potential defect types “Cleanliness Index” - Improved visibility Staged & purposeful detection L4 Expectations PDT10 TT8 Complete test cases PDT9 TT7 S6 L3 PDT9 TT6 Cleanliness PDT8 TT5 Goal directed Sensible automation measures PDT7 TT4 L2 PDT6 PDT5 TT3 L1 PDT4 PDT3 TT2 PDT2 PDT1 TT1 Stage Quality report CC1 CC2 CC3 CC4 R1 Met R2 Not met R3 Partially met R4 R5 “Growth of a baby” © 2011. STAG Software Private Limited. All rights reserved.
- 24. HBT - A Case Study Two Teams, one using HBT & The other conventional approach Test case details Test case details (HBT) Module HBT Normal Increase Module Total Positive Negative M1 100 28 257% M1 100 59 41 M2 85 52 63% M2 85 68 17 M3 95 66 44% M3 95 67 28 M4 132 72 83% M4 132 112 20 M5 127 28 354% M5 127 85 42 M6 855 116 637% M6 855 749 106 TOTAL 1394 362 285% TOTAL 1394 1140 254 Nearly 3x increase in #test cases increasing probability of higher defect yield 2x improvement in negative cases increasing probability of better defect yield Defect details Effort details (person-hours) HBT Normal Increase Stage HBT Normal #Defects 32 16 100% Test analysis & 30 20 Design 20 (Major), 12(Minor) Out of these 32 defects, few were Front loading of effort resulted residual defects, one being critical to in lowering support cost corrupt the entire data. © 2011. STAG Software Private Limited. All rights reserved.
- 25. HBT Results 50%-1000% reduction in post-release defects Re-architecting test assets increases test coverage by 250% 30% defect leakage reduction from early stage Smart automation - 3x reduction in time Deskilling - Less experienced staff do better, faster ramp up, lower cost ‘Holes’ found & fixed at requirement stage Test assessment accelerates integration, de-risks deployment © 2011. STAG Software Private Limited. All rights reserved.
- 26. Summarizing... S2 Cleanliness criteria Potential defect types S3 S1 Staged & purposeful detection Expectations S4 Complete test cases Goal directed S6 Sensible automation S6 measures Clear baseline A goal focused approach to cleanliness Levels, Types & Techniques - STRATEGY Countable test cases & Fault coverage Focused scenarios + Good automation architecture “Cleanliness Index” - Improved visibility © 2011. STAG Software Private Limited. All rights reserved.
- 27. Thank you! “...the value addition for us came from a blend of HBT/STEM and the dedication/hard work from senior management to excel in their services....” Praveen Kanipakam, President SHARP Software Development India “...the methodology and team was the potent combination that jumpstarted our QA." Tapesh Kumar Agarwal, CTO & Co-Founder Meddiff Technologies Connect with us... @stagsoft “..the focus on scientific way of doing testing with blog.stagsoftware.com their methodology HBT was reflected in the effectiveness of test artifacts created by their team.” Raj Khare, Chairman & Founder SureWaves “..the quality of test design and completeness of testing as a result of their methodology is evident in every release.” Iype Isaac, CTO Bankbazaar.com STEMTM is the trademark of STAG Software Private Limited © 2011. STAG Software Private Limited. All rights reserved. www.stagsoftware.com