Risk and Testing (2003)

Risk and Testing
extended after session

Presentation to Testing Master Class
27/28 March 2003
Neil Thompson

www.TiSCL.com
23 Oast House Crescent NeilT@TiSCL.com
Thompson Farnham, Surrey Direct phone 07000 NeilTh
information GU9 0NP (634584)
Systems England (UK) Direct fax 07000 NeilTF
Consulting Limited Phone & fax 01252 726900 (634583)
Some slides included with permission from and Paul Gerrard © Thompson
information
Systems
Consulting Limited

Agenda

• 1. What testers mean by risk
• 2. “Traditional” use of risk in testing
• 3. More recent contributions to thinking
• 4. Risk-Based Testing: Paul Gerrard (and I)
• 5. Next steps in RBT:
– end-to-end risk data model;
– automation
• 6. Refinements and ideas for future
© Thompson
Neil Thompson: Risk and Testing 27/28 Mar Slide 1 of 51 information
2003 Systems
Consulting Limited

1. What testers mean by risk
• Risk that software in live use will fail:
– software: could be Commercial Off The Shelf; packages such as
ERP; bespoke project; integrated programmes of multiple
systems...; industry-wide supply chains; any systems product
• Could include risk that later stages (higher levels)
of testing will be excessively disrupted by failures
• Chain of risks:
Error: Fault: Failure:
RISK mistake made by human RISK something wrong in a product RISK deviation of product from its
(eg spec-writing, (interim eg spec, expected* delivery or service
program-coding) final eg executable software) (doesn’t do what it should,
or does what it shouldn’t)
not all errors * “expected” may be as in spec,
not all faults
result in faults or spec may be wrong
result in failures
(verification & validation)

© Thompson
2003 Systems
Consulting Limited

Extra links (4,5) in the chain of risks?
• Steve Allott:
– distinguish faults in specifications from faults in the resulting program code

Link 2: Link 3: Link 4:
RISK Link 1: RISK something
RISK something RISK deviation of product from its
mistake made by human expected delivery or service
wrong in wrong in
a spec program (doesn’t do what it should,
code or does what it shouldn’t)

• Ed Kit (1995 book, Software Testing in the Real World):
– “error” reserved for its scientific use (but not always very useful in software testing?)

 Error Mistake:  (undefined):  Fault:  Failure:
RISK a human action that RISK incorrect results RISK an incorrect step, RISK an incorrect result
produces an incorrect process or data
in specifications
result (eg in spec-
writing, program- Could use Defect for this?
definition in a
computer program  Error:
amount by which
coding) (ie executable result is incorrect
software)

© Thompson
Neil Thompson: Risk and Testing 27/28 Mar Slide 2.x1 of 51 information
2003 Systems
Consulting Limited

Chain of risks could be up to 6 links?
• Adding in a distinction by John Musa (1998 book,
Software Reliability Engineering):
– not all deviations are failures (but this is just the “anomaly” concept?)
– (so the associated risks are in the testing process rather than development: that
an anomaly may not be noticed, or may be misinterpreted)

• A possible hybrid of all sources: (false alarm):
or Change Request,

 Anomaly:
or testware mistake

an unexpected result
during testing RISK OF MIS-
INTERPRETING
Note: this fits its usage
in inspections
RISK OF MISSING

 Mistake:  Defect:  Fault:  Failure:
RISK a human action that RISK incorrect results RISK an incorrect step, RISK an incorrect result
produces an incorrect in specifications process or data
result (eg in spec-
writing, program-
definition in a
computer program  Error:
amount by which
coding) RISK (ie executable result is incorrect
software)
Direct programming mistake

© Thompson
2003 Systems
Consulting Limited

Three types of software risk

Process Risk
Project Risk variances in planning and
resource constraints, estimation, shortfalls in
external interfaces, staffing, failure to track
supplier relationships, progress, lack of quality
contract restrictions assurance and
configuration management

Primarily a management Planning and the development
responsibility process are the main issues here.

Product Risk
lack of requirements
stability, complexity,
Testers are design quality, coding
mainly quality, non-functional
concerned with issues, test specifications.
Product Risk
Requirements risks are the most significant
risks reported in risk assessments.
© Thompson
2003 Systems
Consulting Limited

Risk management components

BAD THINGS WHICH
COULD HAPPEN, AND
PROBABILITY OF EACH

N E
W S
CONSEQUENCE OF
EACH BAD THING
WHICH COULD HAPPEN

© Thompson
2003 Systems
Consulting Limited

Symmetric view of
risk probability & consequence
PROBABILITY risk EXPOSURE =
probability
(likelihood) x
of bad thing
occurring
3 6 9 consequence

2 4 6
1 2 3
CONSEQUENCE (impact)
if bad thing
does occur

• This is how most people quantify risk
• Adding gives same rank as multiplying, but
less differentiation
© Thompson
2003 Systems
Consulting Limited

Risk management components

BAD THINGS WHICH
COULD HAPPEN, AND
PROBABILITY OF EACH

any other N E
W S
dimensions? CONSEQUENCE OF
EACH BAD THING
WHICH COULD HAPPEN

© Thompson
Neil Thompson: Risk and Testing 27/28 Mar Slide 4r of 51 information
2003 Systems
Consulting Limited

Do risks have any other dimensions?

• In addition to probability and consequence...
• Undetectability:
– difficulty of seeing a bad thing if it does happen
– eg insidious database corruption
• Urgency:
– advisability of looking for / preventing some bad
things before other bad things
– eg lack of requirements stability
• Any others?
© Thompson
2003 Systems
Consulting Limited

2. “Traditional” use of risk in testing

• Few authors & trainers in software testing now
miss an opportunity to link testing to “risk”
• In recent years, almost a mandatory mantra
• But it isn‟t new (what will be new is translating
the mantra into everyday doings!)
• Let‟s look at the major “traditional” authors:
– Hetzel
– Myers
– Beizer
– others
© Thompson
2003 Systems
Consulting Limited

Who wrote this and when?
• “Part of the art of testing is to know when to stop
testing”:
– some recent visionary / pragmatist?
– Myers? His eponymous 1979 book was the first on
testing?
– No, No and No!
– Fred Gruenberger in the original testing book, Program
Test Methods 1972-3, Ed. William C. Hetzel
• Also in this book (which is the proceedings of first-ever
testing conference)...
© Thompson
2003 Systems
Consulting Limited

Hetzel (Ed.) 1972-3
• Little / nothing explicitly on risk, but:
– reliability as a factor in quality; inability to cope with complexity of systems
– “the probability of being faulty is great”p255 (Jean-Claude Rault, CRL France)...
– “how to run the test for a given probability of error... number of random input
combinations before... considered „good‟”p258; sampling as a principle of testing

• Interestingly:
– “sampling as a principle should decrease in importance and be replaced by
hierarchical organization & logical reduction”p28 (William C. Hetzel)

• Other curiosities:
– ?source of Myers‟ triangle exercise p13 (ref. SYSTEM DESIGN
Dr. Richard Hamming, “Computers and Society”) COMPON‟T DESIGN
MODULE DESIGN

– the first “V-model”? p172 Outside-in design, inside-out testing
MODULE TEST
COMPONENT TEST
SYSTEM TEST
(Allan L. Scherr, IBM Poughkeepsie NY / his colleagues) CUSTOMER USE

© Thompson
2003 Systems
Consulting Limited

Myers 1976: Software Reliability -
Principles & Practices
• Again, “risk” not explicit, but principles are there:
– “reliability must be stated as a function of the severity of errors as well as their
frequency”; “software reliability is the probability that the software will execute
for a period of time without a failure, weighted by the cost to the user of each
failure”; “probability that a user will not enter a particular set of inputs that
leads to a failure”p7
– “if there is reason to believe that this set of test cases had a high probability of
uncovering all possible errors, then the tests have established some confidence
in the program‟s correctness”; “each test case used should provide should
provide a maximum yield on our investment... the probability that the test case
will expose a previously undetected error”p170, 176
– “if a reasonable estimate of [the number of remaining errors in a program] were
available during the testing stages, it would help to determine when to stop
testing”p329
– hazard function as a component of reliability modelsp330
© Thompson
2003 Systems
Consulting Limited

Myers 1979: The Art of Software Testing
• Risk is still not in the index, but more principles:
– “the earlier that errors are found, the lower are the costs of correcting... and the
higher is the probability of correcting the errors correctly”p18
– “what subset of all possible test cases has the highest probability of detecting
the most errors”p36
– tries to base completion criteria for each phase of testing on an estimate of the
number of errors originating in particular design processes, and during what
testing phases these errors are likely to be detectedp124
– testing adds value by increasing reliabilityp5
– revisits / updates the reliability models outlined in his 1976 book:
• those related to hardware reliability theory (reliability growth, Bayesian, Markov,
tailored per program)
• error seeding, statistical sampling theory
• simple intuitive (parallel independent testers, historic error data)
• complexity-based (composite design, code properties)
© Thompson
2003 Systems
Consulting Limited

Hetzel (1984)-8: The Complete Guide to
Software Testing
• Risk appears only once in the index, but is prominent:
– Testing principle #4p24: Testing Is Risk-Based
• amount of testing depends on risk of failure, or of missing a defect; so...
• use risk to decide number of cases, amount of emphasis, time & resources
• Other principles appear:
– testing measures software quality; want maximum confidence per unit cost via
maximum probability of finding defectsp255
– objectives of Testing In The Large include:p123
• are major failures unlikely?
• what level of quality is good enough?
• what amount of implementation risk is acceptable?
– System Testing should end when we have enough confidence that
Acceptance Testing is ready to startp134
© Thompson
2003 Systems
Consulting Limited

Beizer 1984: Software System Testing &
Quality Assurance
• Risk appears twice in index, but both insignificant
• However, some relevant principles are to be found:
– smartness in software production is ability to avoid past, present & future
bugsp2 (and bwgs?)
– now more than a dozen models/variations in software reliability theory: but all
far from reality; and all far from providing simple, pragmatic tools that can be
used to measure software developmentp292-293
– six specific criticisms: but if a theory were to overcome these then it would
probably be too complicated to be practicalp293-294
– a compromise may be possible in future, but instead for now, suggest go-live
when the system is considered to be useful, or at least sufficiently useful to
permit the risk of failurep295
– plotting and extrapolation of S-curves to assess when this point attainedp295-304

© Thompson
2003 Systems
Consulting Limited

Beizer (1983)-90:
Software Testing Techniques
• “Risk” word is indexed as though deliberate:
– a couple of occurrences are insignificant, but others:
• purpose of testing is not to prove anything but to reduce perceived risk [of
software not working] to an acceptable value (penultimate phase of attitude)
• testing not an act; is a mental discipline which results in low-risk software
without much testing effort (ultimate phase of attitude) p4
• accepting principles of statistical quality control (but perhaps not yet implementing,
because is not yet obvious how to, and in the case of small products, is dangerous) p6
• add test cases for transactions with high risksp135
• we risk release when confidence is high enoughp6
• Other occurrences of key principles, including:
– probability of failure due to hibernating bwgs* low enough to acceptp26
– importance of a bwg* depends on frequency, correction cost, [fix] installation cost
& consequencesp27
*bwg: ghost, spectre, bogey, hobgoblin, spirit of the night,any imaginary (?) thing that frightens a person (Welsh)
© Thompson
2003 Systems
Consulting Limited

Others
• The “traditional” period could be said to cover the
1970s and 1980s. A variety of views can be found:
– Edward Miller 1978, in Software Testing & Validation
Techniques (IEEE Tutorial):
• “except under very special situations [...], it is important to recognise that program
testing, if performed systematically, can serve to guarantee the absence of bugs”p4
• and/but(?) “a program is well tested when the program tester has an adequately high
level of confidence that there are no remaining “errors” that further testing would
uncover”p9 (italics by Neil Thompson!)

– DeMillo, McCracken, Martin & Passafiume 1987:
Software Testing & Evaluation
• “a technologically sound approach to testing will incorporate... evaluations of
software status into overall assessments of risk associated with the development and
eventual fielding of the system”p vii
© Thompson
2003 Systems
Consulting Limited

3. More recent contributions to
(risk use) thinking
• Traditional basis of testing on risk (although more
perceptive than some give credit for) is less than
satisfactory because:
– it tends to be “lip-service”, with no follow-through / practical application
– if there is follow-through, it involves merely using risk analysis as part of the
Testing Strategy (then that is shelved, and it‟s “heads down” from then on?)
• Contributions more recently from (for example):
– Ed Kit (Software Testing in the real world, 1995)
– Testing Maturity Model (Illinois Institute of Technology)
– Test Process Improvement® (Tim Koomen & Martin Pol)
– Testing Organisation MaturityTM questionnaire (Systeme Evolutif)
– Hans Schaefer‟s work
– Zen and the art of Object-Oriented Risk Management (Neil Thompson)
© Thompson
Neil Thompson: Risk and Testing 27/28 Mar Slide 16x of 51 information
2003 Systems
Consulting Limited

Kit 1995:
Software Testing in the real world
• Error-fault-failure chain extendedp18
• Clear statements on risk and risk managementp26:
– test the parts of the system whose failures would have
most serious consequencesp27
– frequent-use areas increase chances of failure foundp27
– focus on parts most likely to have errors in themp27
– risk is not only basis for test management decisions, is
basis for everyday test practitioner decisionsp28
• Risk management used in integration testp95
© Thompson
2003 Systems
Consulting Limited

Testing Maturity Model
• Five levels of increasing maturity, based loosely on decades of testing evolution
(eg in 1950s testing not even distinguished from debugging)
• Maturity goals and process areas for the five levels do not
include risk explicitly, although emphasis moves from
tactical to strategic (eg fault detection to prevention):
– in level 1, software released without adequate visibility of quality & risks
– in level 3, test strategy is determined using risk management techniques
– in level 4, software products are evaluated using quality criteria (relation to risk?)
– in level 5, costs & test effectiveness are continually improved (sampling quality)
• Strongly recommended are key practices & subpractices:
– (not yet available?)
• Little explicitly visible on risk; very process-oriented
© Thompson
2003 Systems
Consulting Limited

Test Process Improvement®
• Only one entry in index:
– risks and recommendations, substantiated with metrics (as part of
Reporting)
• risks indicated with regard to (parts of) the tested object
• risks can be (eg) delays, quality shortfalls

• But risks incorporated to some extent, eg:
– Test Strategy (differentiation in test depth depending on risks)

© Thompson
2003 Systems
Consulting Limited

Testing Organisation Maturity
(TOMTM) questionnaire

• Risk assessment is not only at the beginning:
– when development slips, a risk assessment is
conducted and a decision to squeeze, maintain or
extend test time may be made
– [for] tests that are descoped, the associated risks
are identified & understood

© Thompson
2003 Systems
Consulting Limited

Hans Schaefer’s work
• Squeeze on testing prioritise based on risk
• Consider possibility of stepwise release: :
– test most important functions first
– look for functions which can be delayed
• What is “important” in the potential release (key functions,
worst problems?)
– visibility (of function / characteristic)
– frequency of use
– possible cost of failure
• Where likely to be most problems?
– project history (new technology, methods, tools; numerous people, dispersed)
– product measures (areas complex, changed, needing optimising, faulty before)
© Thompson
2003 Systems
Consulting Limited

Zen and the Art of
Object-Oriented Risk Management
• Testing continues to lag development; but pessimism /
delay was currently unacceptable (deregulation, Euro, Y2k)
• Could use OO concepts to help testing:
– encapsulate risk information with tests (for effectiveness); and/or
– inherit tests to reuse (efficiency)
• Basic risk management:
– relationship to V-model (outline level)
– detail level: test specification

© Thompson
2003 Systems
Consulting Limited

Risk management & the V-model

Acceptance
testing

System
Risks that system(s) have testing
undetected defects
in them Integration
testing

Unit
testing
© Thompson
Neil Thompson: Risk and Testing 27/28 Mar Slide 22a of 51 information
2003 Systems
Consulting Limited

Risk management & the V-model

Risks that system(s) and business(es)
are not right and ready
for each other Acceptance
testing

System
Risks that system(s) have testing
undetected defects
in them Integration
testing

Unit
testing
© Thompson
Neil Thompson: Risk and Testing 27/28 Mar Slide 22b of 51 information
2003 Systems
Consulting Limited

Where and how testing manages risks:
first, at outline level

Level Risks
Service requirements
Acceptance Undetected errors damage
testing business
System specification
System Undetected errors waste user time
testing & damage confidence in
Acceptance testing
Interfaces don‟t match
Integration Undetected errors too late to fix
testing
Units don‟t work right
Unit Undetected errors won‟t be found
testing by later tests

© Thompson
2003 Systems
Consulting Limited

Where and how testing manages risks:
first, at outline level

Level Risks How managed
Service requirements Specify user-wanted tests against URS
Acceptance Undetected errors damage Script tests around user guide and user
testing business & operator training materials
System specification Use independent testers, functional &
System Undetected errors waste user time technical, to get fresh view
testing & damage confidence in Take last opportunity to do automated
Acceptance testing stress testing before env‟ts re-used
Interfaces don‟t match Use skills of designers before they move
Integration Undetected errors too late to fix away
testing Take last opportunity to exercise
interfaces singly
Units don‟t work right Use detailed knowledge of developers
Unit Undetected errors won‟t be found before they forget
testing by later tests Take last opportunity to exercise every
error message

© Thompson
2003 Systems
Consulting Limited

Second, at detail level:
risk management during test specification

Test specification
based on total
magnitude of risks
for all defects
imaginable
= ( Estimated
probability of
defect
occurring
x
Estimated
severity of
defect )
• To help decision-making during the “squeezing of testing”, it would be
useful to have recorded explicitly as part of the specification of each
test:
– the type of risk the set of tests is designed to minimise
– any specific risks at which a particular test or tests is aimed
• And this was one of the inputs to...

© Thompson
2003 Systems
Consulting Limited

4. Risk-Based (E-Business) Testing
• Advert!
– Artech House, 2002
– ISBN 1-58053-314-0
– reviews amazon.com & co.uk
• companion website
www.riskbasedtesting.com
– sample chapters
– Master Test Planning template
– comments from readers
(reviews, corrections)
With acknowledgements © Thompson
2003
to lead author Paul Gerrard Systems
Consulting Limited

Risk-Based E-Business Testing:
main themes
• Can define approximately 100 “product” risks threatening a typical e-
business system and its implementation and maintenance
• Test objectives can be derived almost directly as “inverse” of risks
• Usable reliability models are some way off (perhaps even
unattainable?) so better for now to work on basis of stakeholders‟
perceptions of risk
• Lists & explains techniques appropriate to each risk type
• Includes information on commercial and DIY tools
• Final chapters are on “making it happen”
• Go-live decision-making: when benefits “now” exceed risks “now”
• Written for e-business but principles are portable; extended to wider
tutorial for EuroSTAR 2002; following slides summarise key points
2003
Consulting Limited

Risks and test objectives -
examples

Risk Test Objective

The web site fails to function To demonstrate that the application functions
correctly on the user‟s client correctly on selected combinations of
operating system and browser operating systems and browser version
configuration. combinations.
Bank statement details To demonstrate that statement details
presented in the client presented in the client browser reconcile with
browser do not match records back-end legacy systems.
in the back-end legacy
banking systems.
Vulnerabilities that hackers To demonstrate through audit, scanning and
could exploit exist in the web ethical hacking that there are no security
site networking infrastructure. vulnerabilities in the web site networking
infrastructure.

© Thompson
2003 Systems
Consulting Limited

Generic test objectives
Test Objective Typical Test Stage
Demonstrate component meets requirements Component Testing
Demonstrate component is ready for reuse in larger Component Testing
sub-system
Demonstrate integrated components correctly Integration testing
assembled/combined and collaborate
Demonstrate system meets functional requirements Functional System
Testing
Demonstrate system meets non-functional requirements Non-Functional System
Testing
Demonstrate system meets industry regulation System or Acceptance
requirements Testing
Demonstrate supplier meets contractual obligations (Contract) Acceptance
Testing
Validate system meets business or user requirements (User) Acceptance
Testing
Demonstrate system, processes and people meet (User) Acceptance
business requirements Testing
© Thompson
2003 Systems
Consulting Limited

Master test planning

© Thompson
2003 Systems
Consulting Limited

Test Plans for each testing stage
(example)
eg for System Testing: Test 1 2 3 4 5 6 7 8 9 10 11 ...
Risk F1 125 Test objective F1
Risk U1 12 Test objective U1
Risk S1 100 Test objective S1
Risk S2 40 Test objective S2
Requirement F1
Requirement F2 Generic test objective G4
Requirement F3
Requirement N1 Generic test objective G5
Requirement N2 Test importance H L M H H H M M M L L ...
© Thompson
2003 Systems
Consulting Limited

Test Design:
target execution schedule (example)
ENVIRONMENTS TEAMS TESTERS
Test execution days
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 ...
Balance &
1

Retests & regression tests
transaction
reporting

Inter- 5 10
Partition for account 3 7 11
functional tests transfers 2
Payments 6 9
Direct debits
End-to-end
customer
scenarios

Partition for 4 8
disruptive non-functional tests
Comfortable completion date
Earliest completion date
© Thompson
2003 Systems
Consulting Limited

Plan to manage risk (and scope) during
test specification & execution
Quality

Quality Scope
Cost
Risk
Quality
best pair to
fine-tune

Time Time Cost
Scope
Scope Cost
Time
© Thompson
2003 Systems
Consulting Limited

Risk as the “inverse” of quality
Quality

Scope Time
Scope Time

Risk
© Thompson
2003 Systems
Consulting Limited

Managing risk during
test specification
initial scope set target go-live date
Scope Time by requirements set in advance

increasing decreasing
risk of faults risk as
introduced faults found

Risk
© Thompson
2003 Systems
Consulting Limited

Verification & validation as risk
management methods

© Thompson
2003 Systems
Consulting Limited

Risk management during
test specification: “micro-risks”

Test specification: Estimated Estimated RISKS TO
•for all defects probability x consequence = TEST AGAINST
imaginable...

© Thompson
Neil Thompson: Risk and Testing 27/28 Mar Slide 32ax of 51 information
2003 Systems
Consulting Limited

Risk management
clarifies during test execution

imaginable...

Test execution: Consequence =
•for each defect Probability
detected =1 x f (urgency,
importance)

© Thompson
Neil Thompson: Risk and Testing 27/28 Mar Slide 32bx of 51 information
2003 Systems
Consulting Limited

But clarity during test execution is
only close-range: fog ahead!

imaginable...

•for each failure Probability
detected… =1 x f (urgency,

{ }
importance)

=
REMAINING
•for all Estimated Estimated RISKS
anomalies as yet probability x consequence
undiscovered...

© Thompson
Neil Thompson: Risk and Testing 27/28 Mar Slide 32cx of 51 information
2003 Systems
Consulting Limited

Managing risk during test execution:
when has risk got low enough?


Risk
© Thompson
2003 Systems
Consulting Limited

Pragmatic approximation to risk
reduction: progress through “test+fix”
# anomalies
• Cumulative S-curves are
good because they:
Cumulative anomalies
– show several things at once
– facilitate extrapolation of trends Resolved
– are based on acknowledged Awaiting
theory and empirical data... fix Deferred

# Closed
tests date
Target tests run

Fail
Actual tests run Pass
date

© Thompson
2003 Systems
Consulting Limited

Cumulative S-curves: theoretical basis
(potential)
# failures
per day Software reliability growth model
depending on
operational
profile
Software
Hardware

Above curves based on Myers 1976: Software Reliability: Principles & Practices p10 date
cumulative
# Anomalies found
More than 1 Much less than 1 anomaly per test
anomaly per test
Much more than 1
anomaly per test, but
Tests run Late tests slower because

go-live date
only one visible at a time!
awaiting difficult and/or
lo-priority fixes
date
Early tests blocked by
hi-impact anomalies Middle tests fast and productive
© Thompson
2003 Systems
Consulting Limited

A possible variation on the
potential
software reliability growth model
# failures Possibility of knock-on errors included in

go-live date
Software Littlewood & Verrall 1973:
per day
A Bayesian Reliability Growth Model
for Computer Software (in IEEE Symposium)
Bad maintenance

Good maintenance

Good testing & maintenance: Bad testing & maintenance: date
convergence on stability divergence into instability
failures failures
TEST & TEST &
FIX HACK RETEST
RETEST
failures
failures
CHECK
AGAINST HAVE A
EXPECTED STROKE THINK AND
DIAGNOSE TALK TO FOLKS
RESULTS CHIN

© Thompson
2003 Systems
Consulting Limited

Cumulative S-curves: more theory
cumulative #

Actually there Anomalies found
are several
reliability growth
models, but: Tests run

go-live date
• the Rayleigh model is
part of hardware
reliability methodology date
and has been used
successfully in # per day
software reliability
during development Pattern of fault discovery Dunn & Ullman 1982 p61
and testing
Anomalies found
• its curve produces
the S-curve when

go-live date
accumulated

Tests run date
© Thompson
2003 Systems
Consulting Limited

Reliability theory more generally
# failures
per “t”
Software: exponential decay Myers 1976

Hardware: Poisson distribution Myers 1976

Both of our software execution time Σt
curves are members • shape
of the Weibull parameter “m”
# failures can take various • NB: models tend to
distribution:
per “t” values (only 1 & 2 use execution time
• has been used for shown here): Kan rather than elapsed
decades in hardware: 1995 p180 time (because removes
Kan 1995 p179 Rayleigh (m=2) test distortion, and uses
• two single-parameter operational profile ie
cases applied to how often used live)
software by Wagoner Exponential (m=1)
in 1970s: Dunn &
Ullman 1982 p318 execution time Σt
© Thompson
2003 Systems
Consulting Limited

Reliability models taxonomy
(Beizer, Dunn & Ullman, + Musa)
future addition to slide pack

© Thompson
2003 Systems
Consulting Limited

Reliability: problems with theories
(Beizer, 1984)
• Main problem is getting theories to match reality
• Several acknowledged shortcomings of many theories, eg:
– don‟t evaluate consequence (severity) of anomalies
– assume testing is like live (eg relatively few special cases)
– don‟t correct properly for stress-test effects, or code enhancements
– don‟t consider interactions between faults
– don‟t allow for debugging getting harder over time
• The science is moving on eg Wiley, Journal of Software Testing, Verification & Reliability but:
– a reliability theory that satisfied all the above would be complex
– would project managers use it, or would they go live anyway?
• So until these are resolved, let‟s turn to empirical data...
© Thompson
2003 Systems
Consulting Limited

Cumulative S-curves: empirical data
Observed empirically that faults plot takes characteristic shape Hetzel 1988 p210

#
Anomalies found Possible to use to roughly gauge
test time or
faults remaining Hetzel 1988 p210

Tests run

go-live date
date
S-curve also visible in Kit 1995: Software Testing in the Real World p135

The Japanese “Project Bankruptcy” study: Abe, Sakamura & Aiso 1979, in Beizer 1984
• analysed 23 projects, including application software & system software developments
• included new code, modifications to existing code, and combinations
• remarkable similarity across all projects for shape of test completion curve
• anomaly detection rates not significant (eg low could mean good software or bad testing)
• significant were (a) length of initial slow progress, and (b) shape of anomaly detection curve...

© Thompson
2003 Systems
Consulting Limited

Project bankruptcy study,
cumulative
# summarised by Beizer
Anomalies found (b) A secondary indicator was
start of integration stage

anomaly detection rate

planned go-live date
Derivative
(looks like rate
Tests run
Rayleigh “bankrupt”
curve)

failure
date success inevitable
Phase I............II...............III 100%
All the projects had 3 phases: Beizer 1984: Software System Testing & Quality Assurance, p297-300
(a) Duration of phases was primary indicator of project success or “bankruptcy”:
Ph I+II...............................72%.......................................126%
success “bankrupt”
failure inevitable

Ph I..15%................55%................97%

© Thompson
2003 Systems
Consulting Limited

S-curves & reliability:
summary of references
• Many of the original references are very old:
– illustrates the points that (a) the earliest testers seemed to be
“advanced” even at that outset, and (b) software reliability
seems not to have penetrated mainstream testing 30 years on!
– but: means these books & papers hard to obtain, so...
• Recommended “recent” references:
– Boris Beizer 1984 book: Software System Testing & Quality
Assurance (ISBN 0-442-21306-9)
– Stephen H. Kan 1995 book: Metrics & Models in Software
Quality Engineering (ISBN 0-201-63339-6)
– John Musa 1999 book: Software Reliability Engineering
(ISBN 0-07-913271-5)
© Thompson
2003 Systems
Consulting Limited

Progress through tests
• We are interested in two main aspects:
– can we manage the test execution to get complete before target
date?
– if not, can we do it for those tests of high (and medium?)
importance?
# tests # tests

Target tests passed
Target tests run

Actual tests
passed High
Fail
Actual tests run Pass
Medium
date date
Low

© Thompson
2003 Systems
Consulting Limited

Progress through
anomaly fixing and retesting
• Similarly, two main aspects:
– can we manage the workflow to get anomalies fixed and
retested before target date?
– if not, can we do it for those of material impact?
# anomalies # anomalies

Cumulative anomalies
Outstanding
material impact
Resolved
Awaiting
fix Deferred

Closed date
date

© Thompson
2003 Systems
Consulting Limited

Quantitative and qualitative risk
reduction from tests and retests
PROGRESS &
RESIDUAL RISK
UP RIGHT SIDE
OF W-MODEL

System Large-Scale Acceptance
Testing Integration Testing Testing
PROGRESS
THROUGH
H M Material
TESTS Fail impact
L
Pass
PROGRESS
THROUGH Awaiting fix Awaiting fix
Resolved Material
INCIDENT Deferred impact
FIXING Closed
& RETESTING
© Thompson
2003 Systems
Consulting Limited

…and also into regression testing

Retesting &
Specification Execution
regression
of tests of tests
testing
Difficulties, What’s left for:
Desired Time & Physical
coverage resource constraints squeeze on “complete” second allowance for
con- (environ- scripting & first run further
straints ments execution run runs
etc.) time

...

Testing Testing Test Test Execution: Execution: Regression
Strategy Plan Design Scripts First run Retest & 2nd run Testing
© Thompson
2003 Systems
Consulting Limited

From what we can report, to
what we would like to report

© Thompson
Neil Thompson: Risk and Testing 27/28 Mar Slide 37.1x of 51 information
2003 Systems
Consulting Limited

Risk-based reporting
Planned
start
end

all risks
„open‟ at
the start

Progress through the test plan
© Thompson
2003 Systems
Consulting Limited

Risk-based reporting
Planned
start today
end

residual
risks of
releasing
TODAY

Progress through the test plan
© Thompson
2003 Systems
Consulting Limited

Report not only risk, but also
scope, over time

how much
scope safely
delivered
so far?


Risk
© Thompson
Neil Thompson: Risk and Testing 27/28 Mar Slide 38.1x of 51 information
2003 Systems
Consulting Limited

Benefit & objectives based test reporting

Blocked NFR 1
Blocked func 1
Blocked func 2

Objective
Objective

Objective
Objective
Benefit
Benefit
Benefit
Benefit
Open
Closed
Risks

Open
Closed
Closed
Open
Open
Closed

Project objectives, hence benefits, available for release
© Thompson
Neil Thompson: Risk and Testing 27/28 Mar Slide 39ax of 51 information
2003 Systems
Consulting Limited

Benefit & objectives based test reporting

Objective
Objective
Objective
Objective
Objective
Benefit
Benefit

Benefit
Benefit
Benefit
Benefit
Open
Closed
Risks

Open
Closed
Closed
Open
Open
Closed

Project objectives, hence benefits, available for release
© Thompson
Neil Thompson: Risk and Testing 27/28 Mar Slide 39bx of 51 information
2003 Systems
Consulting Limited

Slippages and trade-offs: an example

• If Test Review Boards recommend delay, management
may demand a trade-off, “slip in a little of that
descoped functionality”

original first
target slip
scope

date

© Thompson
2003 Systems
Consulting Limited

Slippages and trade-offs: an example

• If Test Review Boards recommend delay, management
may demand a trade-off, “slip in a little of that
descoped functionality”
• This adds benefits but also new risks: more delay?
original first actual
target slip go-live
scope

date

© Thompson
2003 Systems
Consulting Limited

Tolerable risk-benefit balance:
another example
• Even if we resist temptation to trade off slippage
against scope, may still need to renegotiate the
tolerable level of risk balanced against benefits
original
target
(risk -
date
benefits)

original target net risk
date
© Thompson
2003 Systems
Consulting Limited

Tolerable risk-benefit balance:
another example
• Even if we resist temptation to trade off slippage
against scope, may still need to renegotiate the
tolerable level of risk balanced against benefits
original
actual
target
(risk - go-live
date
benefits)
“go for it”
margin

original target net risk
date
© Thompson
2003 Systems
Consulting Limited

5. Next steps in Risk-Based Testing
• End-to-end risk data model:
– is wanted to keep risk information linked to tests
– is under development by Paul Gerrard
– main reason is of course...
• Automation:
– clerical and spreadsheets really not enough
– want to keep risk to test link up-to-date through
descopes, reprioritisations etc
– Paul is working on an Access database for now
– any vendors with risk in their data models?
2003
Consulting Limited

6. Refinements and ideas for future
• Although almost universal, the simple multiplication of probability x
consequence can be troublingly over-simple: it might descope testing
for huge-impact risks which are very unlikely (avionics errors?!). So
use an asymmetric view?
• Some risks are from technology, and other risks are business risks, to
use of system. So distinguish “cause” risks from “effect” risks?
• Assessing perception of risks is a start, but can metrics give better
quantification? Metrics & Fault source analysis
• Reliability models a key part of testing theory in the 1970s, but still
not credibly usable? Reliability engineering
• Wider theoretical basis distinguishing risk from uncertainty:
Decision theory
© Thompson
2003 Systems
Consulting Limited

Asymmetric view of
risk probability & consequence
risk EXPOSURE =
• very high (red)
• high (orange)
PROBABILITY • high-ish (yellow)
(likelihood) • moderate (pale yellow)
• low (pale green)
• very low (green)
3 6 9
2 5 8
1 4 7
CONSEQUENCE
(impact)

• This is an arbitrary scheme;
others are possible, of course
© Thompson
2003 Systems
Consulting Limited

Distinguishing “cause” risk from
technology & “effect” risk to business)
probability • Each could be
exposure symmetric or
RISKS TO asymmetric
BUSINESS • Could weight:
(from faulty
documents • business risks
or software) consequence
higher for
Acceptance Testing
• technology risks
RISKS FROM higher for Unit &
TECHNOLOGY
(of faults being made Integration Testing
in documents
or software)
• equal for System
Testing

© Thompson
2003 Systems
Consulting Limited

But what about next time: metrics?

•for all defects
probability x consequence = TEST AGAINST
imaginable...


}
importance)

•for all
{
anomalies as yet
undiscovered... Estimated
probability x
Estimated
consequence
= REMAINING
RISKS

© Thompson
2003 Systems
Consulting Limited

How metrics can help

Risks to testing progress:
defects reported, fixed & retested by urgency
Number

Low Medium High

Risks to business:
defects reported, fixed & retested by importance
Number

Low Medium High

© Thompson
2003 Systems
Consulting Limited


•for all defects
imaginable...


}
importance)

•for all

undiscovered...
{
anomalies as yet
Estimated
probability x
Estimated
consequence
= REMAINING
RISKS

© Thompson
2003 Systems
Consulting Limited


Testing progress: Risks to testing progress:
tests executed by each day of
defects reported, fixed & retested by urgency
testing for a level (cumulative)
Number
Number

Time

Low Medium High

Testing productivity: Risks to business:
defects found by each day of defects reported, fixed & retested by importance
testing for a level (cumulative) Number
Number

Time

Low Medium High

© Thompson
2003 Systems
Consulting Limited


•for all defects
imaginable...


}
importance)

•for all
{
anomalies as yet
undiscovered...
Estimated
probability x
Estimated
consequence
= REMAINING
RISKS

© Thompson
Neil Thompson: Risk and Testing 27/28 Mar Slide 46c of 51 information
2003 Systems
Consulting Limited


Testing progress: Risks to testing progress: Fault source analysis:
tests executed by each day of
defects reported, fixed & retested by urgency source of faults by testing level detecting them
testing for a level (cumulative)
Number Require-
Number ments

Time
System
spec-
Low Medium High ification

Testing productivity: Risks to business:
faults found by each day of defects reported, fixed & retested by importance
testing for a level (cumulative) Number Design
Number

Time
Coding

Low Medium High Unit Integration System Acceptance

© Thompson
Neil Thompson: Risk and Testing 27/28 Mar Slide 47c of 51 information
2003 Systems
Consulting Limited

Fault Source Detected
early
Detected as
intended
Detected
late
Unacceptably
late

Analysis (Rehearsal of
Acceptance
Tests)

Requirements

System
specification
Phase of lifecycle
in which error made
which caused fault
Design

Coding

Unit Integration System & Acceptance Pilot Live
Testing Testing LSI Testing Testing Running

Where fault detected
© Thompson
2003 Systems
Consulting Limited

Reliability engineering
• A 1999 view of reliability (John D. Musa):
– “division between hardware & software reliability is
somewhat artificial... you may combine... to get system
reliability”p35
– to model software reliability, consider:
• fault introduction (development process, product characteristics)
• fault removal (failure discovery, quality of repair)
• environment (operational profiles)
– define necessary reliability
– execution time increases reliability
• Other books and sources
© Thompson
2003 Systems
Consulting Limited

Decision theory
• Decision theory is a body of knowledge and related
analytical techniques of different degrees of formality
designed to help a decision-maker choose among a set
of alternatives in light of their possible consequences.
Decision theory can apply to conditions of certainty,
risk or uncertainty
• Leads to consideration of utility value, game theory,
separating information from noise, etc
• Bayesian was in Myers 1976, but still being discussed
now as new & exciting (because of advances in
algorithms & computation?)
© Thompson
2003 Systems
Consulting Limited

Summary
• Risk has been part of testing for longer than many people think:
about 30 years?
• Main messages today:
– all testing should be based on risk
– risk is difficult to calculate, so use perceptions of
stakeholders, and broker consensus
– “enough” testing has been done when (benefits-
risks) of going live today is a positive quantity
• But there still seems to be far to go before we can use it
scientifically in testing on an everyday basis, and we‟re just
getting to grips with the art!
• Plenty of fun to come...
© Thompson
2003 Systems
Consulting Limited

Risk and Testing (2003)

More Related Content

Viewers also liked (19)

Similar to Risk and Testing (2003) (20)

More from Neil Thompson (15)

Recently uploaded (20)

Risk and Testing (2003)