SlideShare a Scribd company logo
Using Test Triggers for Improved Defect Detection Charles P. Schultz, ASQ CSQE Global Software Group United States - Florida IEEE LATW’03
WHY DO YOU TEST? To demonstrate that the device meets its specification?  - quality To find defects (non-conformance to specification) in the device as soon as possible after they are introduced?-  cost To demonstrate that the device is fit to proceed (handoff) to the next stage of its development or evolution?-  readiness
WHAT ARE YOUR RESULTS? The specification does not adequately or fully describe the intended or required device?  Many defects escape notice until very late in the development cycle when they are the most costly? Do some defects escape the entire product development cycle and reach the customer because they are not checked for by testing?
SOME POSSIBLE REASONS... Specification are written mostly from the perspective of “normal” or “intended” use of the device Device capabilities are added incrementally so finding defects related to interactions gets “postponed” until late in the project Testers will sometimes think of “tricky” or “unusual” scenarios, but do not have a method for doing this regularly and consistently
SOME ADDITIONAL CLUES... The attention of requirements, developers, and testers also tends to concentrate on the operation of the device once it has reached a  static operating state Diagramming the possible  operating spaces  of a device may give some clues as to why some defects are missed and what can be done about it...
DEVICE OPERATING SPACE Initialization Activate Post-Init Deactivate Pwrdn/Reset
DEVICE OPERATING SPACE Attention is focused here Initialization Activate Post-Init Deactivate Pwrdn/Reset
HOW CAN WE DO BETTER? The  Orthogonal Defect Classification  system defines a set of categories to classify how defects are triggered, which includes the following: CONFIGURATION STARTUP NORMAL RESTART
WHY IS THIS SIGNIFICANT? These  Defect Triggers  map to the different regions of the Device Operating Space CONFIGURATION  ALL STARTUP  Initialization NORMAL  Post-Initialization RESTART  Powerdown/Reset
HOW DOES THIS APPLY TO ME? Each test can be classified  by which Defect Trigger(s) it uses. Mapping the test “coverage”  of each trigger will reveal opportunities to improve the test set coverage of the Operating Space ability to find more defects Defining these  additional tests  may also reveal missing or ambiguous requirements
TRIGGER COVERAGE MAP EXAMPLE 33%   untested   (12 of 36 possibilities) Boxes with low numbers may be undertested
TEST EXAMPLE - CONFIGURATION SERIAL I/O Device works correctly at default Baud Rate Device cannot properly decode received date when operating at the lowest possible Baud Rate configuration Is device also being tested at high Baud Rate, or Baud Rates that use “unusual” clock multipliers?
IS THAT ALL THERE IS? Further examination of device behavior and Defect Trigger test results reveals that  some faults cannot be revealed by test cases using one trigger These faults may be   masked by “repairs”  that occur in different parts of the operating space Such a fault can only be detected by operating in the space where the defect is revealed, and without the presence of the repair
FAULT MASKING EXAMPLE In order to detect Fault  F , the device must go through a Restart, and then be operated during Initialization, before Repair  R  occurs F R Initialization Activate Post-Init Deactivate Pwrdn/Reset
WHY ELSE USE TRIGGER PAIRS? Some faults are not masked, but their  effects  are not detectable until an operation occurs in a subsequent operating space Such a fault can only be detected by operating in the space that causes the defects effect, and then  observing behavior in the space where the effect can be detected
DELAYED FAULT EFFECT EXAMPLE Fault  F  creates Effect  E  which corrupts an Initialization parameter. The device must be used normally and then operated during Initialization to trigger  E F E Initialization Activate Post-Init Deactivate Pwrdn/Reset
ARE THESE DEFECTS SIGNIFICANT? Non-Normal, single trigger tests revealed 20% of the defects found in one set of features Trigger pairs detected an additional 23% of defects, which went undetected by the single trigger tests Under-tested Feature/Trigger pairs represent an additional opportunity to find  more defects  and/or provide  more confidence  in the product’s quality
ARE THESE DEFECTS SIGNIFICANT? Non-Normal single trigger tests were over twice as efficient as Normal tests at finding defects (0.22 defects/test versus 0.10) Trigger pair test efficiency was similar, finding 0.26 defects/test A high density of defects indicates that more defects may still be present but undetected
RECIPE FOR DEFECT TRIGGER TESTS Map  tests to triggers and add tests to improve the coverage of under-utilized triggers for each functional area of the device Do the same for trigger pairs Create   and use  tests for new devices and features with the goal to provide good trigger coverage Start where you can make the biggest  impact
RECIPE FOR DEFECT PREVENTION Account for all triggers and trigger pairs in  requirements and design  in order to create more robust and higher quality devices Create design inspection checklists that incorporate each trigger and trigger pair Also design and inspect for the possibility of  masked behaviors  and  delayed effects  in the different operating spaces
WHEN CAN I START? NOW!
QUESTIONS?

More Related Content

PPTX
CS5032 Case study Ariane 5 launcher failure
PPTX
The complete guide for negative testing | David Tzemach
PPTX
Resilience and chaos engineering
PPTX
Twelve Factor - Designing for Change
PPTX
What is automation testing | David Tzemach
PPTX
Jack borden jb471909_testflight
PPTX
Automated Fault Tolerance Testing
PPTX
What is smoke testing
CS5032 Case study Ariane 5 launcher failure
The complete guide for negative testing | David Tzemach
Resilience and chaos engineering
Twelve Factor - Designing for Change
What is automation testing | David Tzemach
Jack borden jb471909_testflight
Automated Fault Tolerance Testing
What is smoke testing

What's hot (12)

DOCX
NAM Q4a 2011 UAT Strategy Document v1 0
DOC
Smoke testing
PPTX
Test Automation Architecture in Microservices
PPT
Testing 3: Types Of Tests That May Be Required
DOCX
Manual testing
PPTX
Sanity testing and smoke testing
PDF
5 types of uat testing
DOCX
Automation Testing
PDF
Software Reliability
PPTX
Techniques, Tips & Tools For Mobile App Testing
PPTX
Types of test tools
NAM Q4a 2011 UAT Strategy Document v1 0
Smoke testing
Test Automation Architecture in Microservices
Testing 3: Types Of Tests That May Be Required
Manual testing
Sanity testing and smoke testing
5 types of uat testing
Automation Testing
Software Reliability
Techniques, Tips & Tools For Mobile App Testing
Types of test tools
Ad

Similar to Using Test Triggers for Improved Defect Detection (20)

PPTX
Vlsi testing
PDF
DFT-Lecture regarding the JTAG, MBIST introduction to DFT
PPTX
1.Week1.pptx
PDF
VLSI testing and analysis
PDF
Design for Testability
PDF
2019 1 testing and verification of vlsi design_introduction
PDF
Nanometer Testing: Challenges and Solutions
PDF
Abraham q3 2008
PDF
Risk management and business protection with Coding Standardization & Static ...
PDF
Testing and Verification of Electronics Circuits : Introduction
PDF
40220140505007
PDF
Test Plan Development using Physics of Failure: The DfR Solutions Approach
PDF
Advancing VLSI Design Reliability: A Comprehensive Examination of Embedded De...
PDF
SOLUTION MANUAL OF WIRELESS COMMUNICATIONS BY THEODORE S RAPPAPORT
PDF
Software Defects and SW Reliability Assessment
PDF
Intro to burn in & ess quantification
PDF
Fault Detection in Mobile Communication Networks Using Data Mining Techniques...
PPT
Ch13 system testexecution
PDF
2019 5 testing and verification of vlsi design_fault_modeling
PDF
Counterfeit prevention & detection strategies, how to do it, when to do it.
Vlsi testing
DFT-Lecture regarding the JTAG, MBIST introduction to DFT
1.Week1.pptx
VLSI testing and analysis
Design for Testability
2019 1 testing and verification of vlsi design_introduction
Nanometer Testing: Challenges and Solutions
Abraham q3 2008
Risk management and business protection with Coding Standardization & Static ...
Testing and Verification of Electronics Circuits : Introduction
40220140505007
Test Plan Development using Physics of Failure: The DfR Solutions Approach
Advancing VLSI Design Reliability: A Comprehensive Examination of Embedded De...
SOLUTION MANUAL OF WIRELESS COMMUNICATIONS BY THEODORE S RAPPAPORT
Software Defects and SW Reliability Assessment
Intro to burn in & ess quantification
Fault Detection in Mobile Communication Networks Using Data Mining Techniques...
Ch13 system testexecution
2019 5 testing and verification of vlsi design_fault_modeling
Counterfeit prevention & detection strategies, how to do it, when to do it.
Ad

Recently uploaded (20)

PDF
NewMind AI Weekly Chronicles - August'25-Week II
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
PDF
Review of recent advances in non-invasive hemoglobin estimation
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
PPTX
Spectroscopy.pptx food analysis technology
PPTX
MYSQL Presentation for SQL database connectivity
PPTX
A Presentation on Artificial Intelligence
PPTX
Big Data Technologies - Introduction.pptx
PPTX
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
PPT
Teaching material agriculture food technology
PDF
Machine learning based COVID-19 study performance prediction
PDF
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
PDF
Electronic commerce courselecture one. Pdf
PDF
Advanced methodologies resolving dimensionality complications for autism neur...
DOCX
The AUB Centre for AI in Media Proposal.docx
PDF
Empathic Computing: Creating Shared Understanding
PDF
The Rise and Fall of 3GPP – Time for a Sabbatical?
PDF
Network Security Unit 5.pdf for BCA BBA.
PDF
Spectral efficient network and resource selection model in 5G networks
PDF
Assigned Numbers - 2025 - Bluetooth® Document
NewMind AI Weekly Chronicles - August'25-Week II
Diabetes mellitus diagnosis method based random forest with bat algorithm
Review of recent advances in non-invasive hemoglobin estimation
Mobile App Security Testing_ A Comprehensive Guide.pdf
Spectroscopy.pptx food analysis technology
MYSQL Presentation for SQL database connectivity
A Presentation on Artificial Intelligence
Big Data Technologies - Introduction.pptx
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
Teaching material agriculture food technology
Machine learning based COVID-19 study performance prediction
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
Electronic commerce courselecture one. Pdf
Advanced methodologies resolving dimensionality complications for autism neur...
The AUB Centre for AI in Media Proposal.docx
Empathic Computing: Creating Shared Understanding
The Rise and Fall of 3GPP – Time for a Sabbatical?
Network Security Unit 5.pdf for BCA BBA.
Spectral efficient network and resource selection model in 5G networks
Assigned Numbers - 2025 - Bluetooth® Document

Using Test Triggers for Improved Defect Detection

  • 1. Using Test Triggers for Improved Defect Detection Charles P. Schultz, ASQ CSQE Global Software Group United States - Florida IEEE LATW’03
  • 2. WHY DO YOU TEST? To demonstrate that the device meets its specification? - quality To find defects (non-conformance to specification) in the device as soon as possible after they are introduced?- cost To demonstrate that the device is fit to proceed (handoff) to the next stage of its development or evolution?- readiness
  • 3. WHAT ARE YOUR RESULTS? The specification does not adequately or fully describe the intended or required device? Many defects escape notice until very late in the development cycle when they are the most costly? Do some defects escape the entire product development cycle and reach the customer because they are not checked for by testing?
  • 4. SOME POSSIBLE REASONS... Specification are written mostly from the perspective of “normal” or “intended” use of the device Device capabilities are added incrementally so finding defects related to interactions gets “postponed” until late in the project Testers will sometimes think of “tricky” or “unusual” scenarios, but do not have a method for doing this regularly and consistently
  • 5. SOME ADDITIONAL CLUES... The attention of requirements, developers, and testers also tends to concentrate on the operation of the device once it has reached a static operating state Diagramming the possible operating spaces of a device may give some clues as to why some defects are missed and what can be done about it...
  • 6. DEVICE OPERATING SPACE Initialization Activate Post-Init Deactivate Pwrdn/Reset
  • 7. DEVICE OPERATING SPACE Attention is focused here Initialization Activate Post-Init Deactivate Pwrdn/Reset
  • 8. HOW CAN WE DO BETTER? The Orthogonal Defect Classification system defines a set of categories to classify how defects are triggered, which includes the following: CONFIGURATION STARTUP NORMAL RESTART
  • 9. WHY IS THIS SIGNIFICANT? These Defect Triggers map to the different regions of the Device Operating Space CONFIGURATION ALL STARTUP Initialization NORMAL Post-Initialization RESTART Powerdown/Reset
  • 10. HOW DOES THIS APPLY TO ME? Each test can be classified by which Defect Trigger(s) it uses. Mapping the test “coverage” of each trigger will reveal opportunities to improve the test set coverage of the Operating Space ability to find more defects Defining these additional tests may also reveal missing or ambiguous requirements
  • 11. TRIGGER COVERAGE MAP EXAMPLE 33% untested (12 of 36 possibilities) Boxes with low numbers may be undertested
  • 12. TEST EXAMPLE - CONFIGURATION SERIAL I/O Device works correctly at default Baud Rate Device cannot properly decode received date when operating at the lowest possible Baud Rate configuration Is device also being tested at high Baud Rate, or Baud Rates that use “unusual” clock multipliers?
  • 13. IS THAT ALL THERE IS? Further examination of device behavior and Defect Trigger test results reveals that some faults cannot be revealed by test cases using one trigger These faults may be masked by “repairs” that occur in different parts of the operating space Such a fault can only be detected by operating in the space where the defect is revealed, and without the presence of the repair
  • 14. FAULT MASKING EXAMPLE In order to detect Fault F , the device must go through a Restart, and then be operated during Initialization, before Repair R occurs F R Initialization Activate Post-Init Deactivate Pwrdn/Reset
  • 15. WHY ELSE USE TRIGGER PAIRS? Some faults are not masked, but their effects are not detectable until an operation occurs in a subsequent operating space Such a fault can only be detected by operating in the space that causes the defects effect, and then observing behavior in the space where the effect can be detected
  • 16. DELAYED FAULT EFFECT EXAMPLE Fault F creates Effect E which corrupts an Initialization parameter. The device must be used normally and then operated during Initialization to trigger E F E Initialization Activate Post-Init Deactivate Pwrdn/Reset
  • 17. ARE THESE DEFECTS SIGNIFICANT? Non-Normal, single trigger tests revealed 20% of the defects found in one set of features Trigger pairs detected an additional 23% of defects, which went undetected by the single trigger tests Under-tested Feature/Trigger pairs represent an additional opportunity to find more defects and/or provide more confidence in the product’s quality
  • 18. ARE THESE DEFECTS SIGNIFICANT? Non-Normal single trigger tests were over twice as efficient as Normal tests at finding defects (0.22 defects/test versus 0.10) Trigger pair test efficiency was similar, finding 0.26 defects/test A high density of defects indicates that more defects may still be present but undetected
  • 19. RECIPE FOR DEFECT TRIGGER TESTS Map tests to triggers and add tests to improve the coverage of under-utilized triggers for each functional area of the device Do the same for trigger pairs Create and use tests for new devices and features with the goal to provide good trigger coverage Start where you can make the biggest impact
  • 20. RECIPE FOR DEFECT PREVENTION Account for all triggers and trigger pairs in requirements and design in order to create more robust and higher quality devices Create design inspection checklists that incorporate each trigger and trigger pair Also design and inspect for the possibility of masked behaviors and delayed effects in the different operating spaces
  • 21. WHEN CAN I START? NOW!