metrics.ppt

Functional and Non-functional
Requirements
• System functional requirements may
specify error checking, recovery
features, and system failure protection
• System reliability and availability are
specified as part of the non-functional
requirements for the system.

System Reliability
Specification
• Hardware reliability
– probability a hardware component fails
• Software reliability
– probability a software component will produce an
incorrect output
– software does not wear out
– software can continue to operate after a bad result
• Operator reliability
– probability system user makes an error

Failure Probabilities
• If there are two independent components in a
system and the operation of the system
depends on them both then
P(S) = P(A) + P(B)
• If the components are replicated then the
probability of failure is
P(S) = P(A)n
meaning that all components fail at once

Functional Reliability
Requirements
• The system will check the all operator
inputs to see that they fall within their
required ranges.
• The system will check all disks for bad
blocks each time it is booted.
• The system must be implemented in
using a standard implementation of Ada.

Non-functional Reliability
Specification
• The required level of reliability must be
expressed quantitatively.
• Reliability is a dynamic system attribute.
• Source code reliability specifications are
meaningless (e.g. N faults/1000 LOC)
• An appropriate metric should be chosen
to specify the overall system reliability.

Hardware Reliability Metrics
• Hardware metrics are not suitable for
software since its metrics are based on
notion of component failure
• Software failures are often design
failures
• Often the system is available after the
failure has occurred
• Hardware components can wear out

Software Reliability Metrics
• Reliability metrics are units of measure for
system reliability
• System reliability is measured by counting the
number of operational failures and relating
these to demands made on the system at the
time of failure
• A long-term measurement program is
required to assess the reliability of critical
systems

Reliability Metrics - part 1
• Probability of Failure on Demand
(POFOD)
– POFOD = 0.001
– For one in every 1000 requests the service
fails per time unit
• Rate of Fault Occurrence (ROCOF)
– ROCOF = 0.02
– Two failures for each 100 operational time
units of operation

Reliability Metrics - part 2
• Mean Time to Failure (MTTF)
– average time between observed failures
(aka MTBF)
• Availability = MTBF / (MTBF+MTTR)
– MTBF = Mean Time Between Failure
– MTTR = Mean Time to Repair
• Reliability = MTBF / (1+MTBF)

Time Units
• Raw Execution Time
– non-stop system
• Calendar Time
– If the system has regular usage patterns
• Number of Transactions
– demand type transaction systems

Availability
• Measures the fraction of time system is
really available for use
• Takes repair and restart times into
account
• Relevant for non-stop continuously
running systems (e.g. traffic signal)

Probability of Failure on
Demand
• Probability system will fail when a service
request is made
• Useful when requests are made on an
intermittent or infrequent basis
• Appropriate for protection systems service
requests may be rare and consequences can
be serious if service is not delivered
• Relevant for many safety-critical systems with
exception handlers

Rate of Fault Occurrence
• Reflects rate of failure in the system
• Useful when system has to process a
large number of similar requests that
are relatively frequent
• Relevant for operating systems and
transaction processing systems

Mean Time to Failure
• Measures time between observable
system failures
• For stable systems MTTF = 1/ROCOF
• Relevant for systems when individual
transactions take lots of processing time
(e.g. CAD or WP systems)

Failure Consequences - part 1
• Reliability does not take consequences
into account
• Transient faults have no real
consequences but other faults might
cause data loss or corruption
• May be worthwhile to identify different
classes of failure, and use different
metrics for each

Failure Consequences - part 2
• When specifying reliability both the number of
failures and the consequences of each matter
• Failures with serious consequences are more
damaging than those where repair and
recovery is straightforward
• In some cases, different reliability
specifications may be defined for different
failure types

Failure Classification
• Transient - only occurs with certain inputs
• Permanent - occurs on all inputs
• Recoverable - system can recover without
operator help
• Unrecoverable - operator has to help
• Non-corrupting - failure does not corrupt
system state or data
• Corrupting - system state or data are altered

Building Reliability
Specification
• For each sub-system analyze
consequences of possible system
failures
• From system failure analysis partition
failure into appropriate classes
• For each class send out the appropriate
reliability metric

Examples
Failure Class Example Metric
Permanent
Non-corrupting
ATM fails to
operate with any
card, must restart to
correct
ROCOF = .0001
Time unit = days
Transient
Non-corrupting
Magnetic stripe
can't be read on
undamaged card
POFOD = .0001
Time unit =
transactions

Specification Validation
• It is impossible to empirically validate
high reliability specifications
• No database corruption really means
POFOD class < 1 in 200 million
• If each transaction takes 1 second to
verify, simulation of one day’s
transactions takes 3.5 days

Statistical Reliability Testing
• Test data used, needs to follow typical
software usage patterns
• Measuring numbers of errors needs to
be based on errors of omission (failing
to do the right thing) and errors of
commission (doing the wrong thing)

Difficulties with Statistical
Reliability Testing
• Uncertainty when creating the
operational profile
• High cost of generating the operational
profile
• Statistical uncertainty problems when
high reliabilities are specified

Safety Specification
• Each safety specification should be specified
separately
• These requirements should be based on
hazard and risk analysis
• Safety requirements usually apply to the
system as a whole rather than individual
components
• System safety is an an emergent system
property

Safety Life Cycle - part 1
• Concept and scope definition
• Hazard and risk analysis
• Safety requirements specification
– safety requirements derivation
– safety requirements allocation
• Planning and development
– safety related systems development
– external risk reduction facilities

Safety Life Cycle - part 2
• Deployment
– safety validation
– installation and commissioning
• Operation and maintenance
• System decommissioning

Safety Processes
• Hazard and risk analysis
– assess the hazards and risks associated with the
system
• Safety requirements specification
– specify system safety requirements
• Designation of safety-critical systems
– identify sub-systems whose incorrect operation
can compromise entire system safety
• Safety validation
– check overall system safety

Hazard Analysis Stages
• Hazard identification
– identify potential hazards that may arise
• Risk analysis and hazard classification
– assess risk associated with each hazard
• Hazard decomposition
– seek to discover potential root causes for each
hazard
• Risk reduction assessment
– describe how each hazard is to be taken into
account when system is designed

Fault-tree Analysis
• Hazard analysis method that starts with
an identified fault and works backwards
to the cause of the fault
• Can be used at all stages of hazard
analysis
• It is a top-down technique, that may be
combined with a bottom-up hazard
analysis techniques that start with
system failures that lead to hazards

Fault-tree Analysis Steps
• Identify hazard
• Identify potential causes of hazards
• Link combinations of alternative causes
using “or” or “and” symbols as
appropriate
• Continue process until “root” causes are
identified (result will be an and/or tree or
a logic circuit) the causes are the
“leaves”

How does it work?
• What would a fault tree look like for a
fault tree describing the causes for a
hazard like “data deleted”?

Risk Assessment
• Assess the hazard severity, hazard
probability, and accident probability
• Outcome of risk assessment is a statement of
acceptability
– Intolerable (can never occur)
– ALARP (as low as possible given cost and
schedule constraints)
– Acceptable (consequences are acceptable and no
extra cost should be incurred to reduce it further)

Risk Acceptability
• Determined by human, social, and
political considerations
• In most societies, the boundaries
between regions are pushed upwards
with time (meaning risk becomes less
acceptable)
• Risk assessment is always subjective
(what is acceptable to one person is
ALARP to another)

Risk Reduction
• System should be specified so that hazards do
not arise or result in an accident
• Hazard avoidance
– system designed so hazard can never arise during
normal operation
• Hazard detection and removal
– system designed so that hazards are detected and
neutralized before an accident can occur
• Damage limitation
– system designed to minimized accident
consequences

Security Specification
• Similar to safety specification
– not possible to specify quantitatively
– usually stated in “system shall not” terms
rather than “system shall” terms
• Differences
– no well-defined security life cycle yet
– security deals with generic threats rather
than system specific hazards

Security Specification Stages -
part 1
• Asset identification and evaluation
– data and programs identified with their level of
protection
– degree of protection depends on asset value
• Threat analysis and risk assessment
– security threats identified and risks associated with
each is estimated
• Threat assignment
– identified threats are related to assets so that asset
has a list of associated threats

Security Specification Stages -
part 2
• Technology analysis
– available security technologies and their
applicability against the threats
• Security requirements specification
– where appropriate these will identify the security
technologies that may be used to protect against
different threats to the system

Quantification
• One of the characteristics of a maturing
discipline is the replacement of art by
science.
• Early physics was dominated by
philosophical discussions with no
attempt to quantify things.
• Quantification was impossible until the
right questions were asked.

Quantification (Cont’d)
• Computer Science is slowly following
the quantification path.
• There is skepticism because so much of
what we want to quantify it tied to erratic
human behavior.

Software quantification
• Software Engineers are still counting
lines of code.
• This popular metric is highly inaccurate
when used to predict:
– costs
– resources
– schedules

Science begins with
quantification
• Physics needs measurements for time,
mass, etc.
• Thermodynamics needs measurements
for temperature.
• The “size” of software is not obvious.
• We need an objective measure of
software size.

Software quantification
• Lines of Code (LOC) is not a good measure
software size.
• In software testing we need a notion of size
when comparing two testing strategies.
• The number of tests should be normalized to
software size, for example:
– Strategy A needs 1.4 tests/unit size.

Asking the right questions
• When can we stop testing?
• How many bugs can we expect?
• Which testing technique is more effective?
• Are we testing hard or smart?
• Do we have a strong program or a weak test
suite?
• Currently, we are unable to answer these
questions satisfactorily.

Lessons from physics
• Measurements lead to Empirical Laws
which lead to Physical Laws.
• E.g., Kepler’s measurements of
planetary movement lead to Newton’s
Laws which lead to Modern Laws of
physics.

Lessons from physics (Cont’d)
• The metrics we are about to discuss
aim at getting empirical laws that relate
program size to:
– expected number of bugs
– expected number of tests required to find
bugs
– testing technique effectiveness

Metrics taxonomy
• Linguistic Metrics: Based on measuring
properties of program text without interpreting
what the text means.
– E.g., LOC.
• Structural Metrics: Based on structural
relations between the objects in a program.
– E.g., number of nodes and links in a control
flowgraph.

Lines of code (LOC)
• LOC is used as a measure of software
complexity.
• This metric is just as good as source listing
weight if we assume consistency w.r.t. paper
and font size.
• Makes as much sense (or nonsense) to say:
– “This is a 2 pound program”
• as it is to say:
– “This is a 100,000 line program.”

Lines of code paradox
• Paradox: If you unroll a loop, you reduce the
complexity of your software ...
• Studies show that there is a linear
relationship between LOC and error rates for
small programs (i.e., LOC < 100).
• The relationship becomes non-linear as
programs increases in size.

Halstead’s program length
program.
in the
objects)
(data
operands
distinct
of
number
the
=
n
operator.)
single
a
as
treated
are
end)
...
(begin
operators
(Paired
program.
in the
(keywords)
operators
distinct
of
number
the
=
n
n
log
n
+
n
log
n
=
H
2
1
2
2
2
1
2
1
LOC
Length
Program
:
WARNING 

Example of program length
48
7
log
7
+
9
log
9
=
H
1.0)
1,
x,
z,
pow,
0,
(y,
7
=
n
/)
(minus),
-
*,
=,
!
while,
(sign),
=,-
<,
(if,
9
=
n
2
2
2
1

if (y < 0)
pow = - y;
else
pow = y;
z = 1.0;
while (pow != 0) {
z = z * x;
pow = pow - 1;
}
if (y < 0)
z = 1.0 / z;

Example of program length
48
7
log
7
+
9
log
9
=
H
temp)
list,
k,
last,
N,
1,
(j,
7
=
n
if)
>,
[],
+,
-,
+,
+
<,
=,
(for,
9
=
n
2
2
2
1

for ( j=1; j<N; j++) {
last = N - j + 1;
for (k=1; k <last; k ++) {
if (list[k] > list[k+1]) {
temp = list[k];
list[k] = list[k+1];
list[k+1] = temp;
}
}
}

Halstead’s bug prediction
bugs
0.075
3000
7)
+
(9
log
31)
+
(25
=
B
bugs
0.049
3000
7)
+
(9
log
21)
+
(16
=
B
operands
of
number
total
the
=
N
operators
of
number
total
the
=
N
operands
distinct
of
number
the
=
n
operators
distinct
of
number
the
=
n
3000
)
n
+
(n
log
)
N
+
(N
=
B
2
2
2
1
2
1
2
1
2
2
1


t Example:
Bubble Sor
le:
tion Examp
Exponentia

How good are
Halstead’s metrics?
• The validity of the metric has been
confirmed experimentally many times,
independently, over a wide range of
programs and languages.
• Lipow compared actual to predicted bug
counts to within 8% over a range of
program sizes from 300 to 12,000
statements.

Structural metrics
• Linguistic complexity is ignored.
• Attention is focused on control-flow and
data-flow complexity.
• Structural metrics are based on the
properties of flowgraph models of
programs.

Cyclomatic complexity
• McCabe’s Cyclomatic complexity is
defined as: M = L - N + 2P
• L = number of links in the flowgraph
• N = number of nodes in the flowgraph
• P = number of disconnected parts of the
flowgraph.

Software testing
process metrics
• Bug tracking tools enable the extraction of
several useful metrics about the software and
the testing process.
• Test managers can see if any trends in the
data show areas that:
– may need more testing
– are on track for its scheduled release date
• Examples of software testing process metrics:
– Average number of bugs per tester per day
– Number of bugs found per module
– The ratio of Severity 1 bugs to Severity 4 bugs
– …

Example queries applied to a
bug tracking database
• What areas of the software have the most
bugs? The fewest bugs?
• How many resolved bugs are currently
assigned to John?
• Mary is leaving for vacation soon. How many
bugs does she have to fix before she leaves?
• Which tester has found the most bugs?
• What are the open Priority 1 bugs?

Example data plots
• Number of bugs versus:
– fixed bugs
– deferred bugs
– duplicate bugs
– non-bugs
• Number of bugs versus each major functional
area of the software:
– GUI
– documentation
– floating-point arithmetic
– etc

Example data plots (cont’d)
• Bugs opened versus date opened over time:
– This view can show:
• bugs opened each day
• cumulative opened bugs
• On the same plot we can plot resolved bugs,
closed bugs, etc to compare the trends.

You now know …
• … the importance of quantification
• … various software metrics
• … various software testing process
metrics and views

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 62
Planning and Monitoring the Process

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 63
Learning objectives
• Understand the purposes of planning
and monitoring
• Distinguish strategies from plans, and
understand their relation
• Understand the role of risks in planning
• Understand the potential role of tools in
monitoring a quality process
• Understand team organization as an
integral part of planning

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 64
What are Planning and
Monitoring?
• Planning:
– Scheduling activities (what steps? in what
order?)
– Allocating resources (who will do it?)
– Devising unambiguous milestones for
monitoring
• Monitoring: Judging progress against
the plan
– How are we doing?

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 65
Quality and Process
• Quality process: Set of activities and responsibilities
– focused primarily on ensuring adequate dependability
– concerned with project schedule or with product usability
• A framework for
– selecting and arranging activities
– considering interactions and trade-offs
• Follows the overall software process in which it is
embedded
– Example: waterfall software process ––> “V model”: unit
testing starts with implementation and finishes before
integration
– Example: XP and agile methods ––> emphasis on unit
testing and rapid iteration for acceptance testing by

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 66
Customer Requirements
Example Process: Cleanroom
Specification
Function Usage
Incremental
Development
Planning
Statistical test case
generation
Usage specifications
Formal Design
Correctness Verification
Functional specifications
Statistical testing
Source code Test cases
Quality Certification Model
MTTF statistics
Interfail times
Improvement Feedback

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 67
Customer Requirements
Example Process: Cleanroom
Specification
Function Usage
Incremental
Development
Planning
Statistical test case
generation
Usage specifications
Formal Design
Correctness Verification
Functional specifications
Statistical testing
Source code Test cases
Quality Certification Model
MTTF statistics
Interfail times
Improvement Feedback
Activities and
responsibilities
focused on quality
Integrated into an
overall
development
process

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 68
Define “Necessary”
Reliability
Requirements
and Architecture
Design and
Implementation
System Test and
Acceptance Test
Example Process: Software Reliability
Engineering Testing (SRET)
Development
Operational Profiles
Prepare
for Testing
Execute
tests
Interpret
Failure
Data

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 69
Define “Necessary”
Reliability
Requirements
and Architecture
Design and
Implementation
System Test and
Acceptance Test
Software Reliability Engineering Testing
(SRET)
Development
Operational Profiles
Prepare
for Testing
Execute
tests
Interpret
Failure
Data
Activities and
responsibilities
focused on quality
Integrated into an
overall
development
process

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 70
Example Process: Extreme Programming
(XP)
Generate User
Stories
Create Unit
Tests
Pair
Programming
+ unit testing
Create
Acceptance
Tests
Incremental
Release
pass
Next version
Review,
Refine,
prioritize
Acceptance
Testing
Passed all
unit tests
Passed all unit tests
Failed acceptance test

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 71
Extreme Programming (XP)
Generate User
Stories
Create Unit
Tests
Pair
Programming
+ unit testing
Create
Acceptance
Tests
Incremental
Release
pass
Next version
Review,
Refine,
prioritize
Acceptance
Testing
Passed all
unit tests
Passed all unit tests
Failed acceptance test
Activities and
responsibilities
focused on quality
Integrated into an
overall
development
process

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 72
Overall Organization
of a Quality Process
• Key principle of quality planning
– the cost of detecting and repairing a fault
increases as a function of time between
committing an error and detecting the
resultant faults
• therefore ...
– an efficient quality plan includes matched
sets of intermediate validation and
verification activities that detect most faults
within a short time of their introduction
• and ...

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 73
Verification Steps
for Intermediate Artifacts
• Internal consistency checks
– compliance with structuring rules that define “well-formed”
artifacts of that type
– a point of leverage: define syntactic and semantic rules
thoroughly and precisely enough that many common errors
result in detectable violations
• External consistency checks
– consistency with related artifacts
– Often: conformance to a “prior” or “higher-level” specification
• Generation of correctness conjectures
– Correctness conjectures: lay the groundwork for external
consistency checks of other work products
– Often: motivate refinement of the current product

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 74
Strategies vs Plans
Strategy Plan
Scope Organization Project
Structure
and content
based on
Organization structure,
experience and policy
over several projects
Standard structure
prescribed in
strategy
Evolves Slowly, with
organization and policy
changes
Quickly, adapting to
project needs

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 75
Test and Analysis Strategy
• Lessons of past experience
– an organizational asset built and refined over
time
• Body of explicit knowledge
– more valuable than islands of individual
competence
– amenable to improvement
– reduces vulnerability to organizational change
(e.g., loss of key individuals)
• Essential for
– avoiding recurring errors

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 76
Considerations in Fitting a
Strategy to an Organization
• Structure and size
– example
• Distinct quality groups in large organizations, overlapping of
roles in smaller organizations
• greater reliance on documents in large than small organizations
• Overall process
– example
• Cleanroom requires statistical testing and forbids unit testing
– fits with tight, formal specs and emphasis on reliability
• XP prescribes “test first” and pair programming
– fits with fluid specifications and rapid evolution
• Application domain
– example
• Safety critical domains may impose particular quality objectives
and require documentation for certification (e.g,RTCA/DO-

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 77
Elements of a Strategy
• Common quality requirements that
apply to all or most products
– unambiguous definition and measures
• Set of documents normally produced
during the quality process
– contents and relationships
• Activities prescribed by the overall
process
– standard tools and practices

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 78
Test and Analysis Plan
Answer the following questions:
• What quality activities will be carried
out?
• What are the dependencies among the
quality activities and between quality
and other development activities?
• What resources are needed and how
will they be allocated?
• How will both the process and the

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 79
Main Elements of a Plan
• Items and features to be verified
– Scope and target of the plan
• Activities and resources
– Constraints imposed by resources on
activities
• Approaches to be followed
– Methods and tools
• Criteria for evaluating results

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 80
Quality Goals
• Expressed as properties satisfied by the
product
– must include metrics to be monitored
during the project
– example: before entering acceptance
testing, the product must pass
comprehensive system testing with no
critical or severe failures
– not all details are available in the early
stages of development

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 81
Task Schedule
• Initially based on
– quality strategy
– past experience
• Breaks large tasks into subtasks
– refine as process advances
• Includes dependencies
– among quality activities
– between quality and development activities
• Guidelines and objectives:
– schedule activities for steady effort and continuous progress
and evaluation without delaying development activities
– schedule activities as early as possible
– increase process visibility (how do we know we’re on track?)

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 82
Sample Schedule
ID Task Name
1st quarter 2nd quarter 3rd quarter
1/7 2/4
1 Development framework
2 Requirements specifications
3 Architectural design
4
Detailed design of shopping
facility subsys.
5
Detailed design of
administrative biz logic
7
Sync and stabilize shopping
fac.
8
Admin biz logic code and
integration (including unit test)
9
Sync and stabilize
administrative biz logic
10 Design inspection
11
Inspection of requirements
specs.
12
Inspection of architectural
design
13
Inspection of det. Design of
shop. facilities
14
Inspection of detailed design
of admin logic
15 Code inspection
16
Inspection of shop. Fun. Core
code and unit tests
17
Inspection of admin. Biz. Log.
Code code and unit tests
18 Design tests
19 Design acceptance tests
20 Design system tests
21
Design shop fun subsystem
integration test
22
Design admin bix log
subsystem integration tests
23 Test execution
24 Exec integration tests
25 Exec system tests
26 Exec acceptance tests
6
Shopping fac code and
integration (incl unit test)

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 83
Schedule Risk
• critical path = chain of activities that must be completed
in sequence and that have maximum overall duration
– Schedule critical tasks and tasks that depend on critical tasks as
early as possible to
• provide schedule slack
• prevent delay in starting critical tasks
• critical dependence = task on a critical path scheduled
immediately after some other task on the critical path
– May occur with tasks outside the quality plan
(part of the project plan)
– Reduce critical dependences by decomposing tasks on critical
path, factoring out subtasks that can be performed earlier

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 84
Task name January Febrary March April May
CRITICAL SCHEDULE
Project start
Analysis and design
Code and integration
Design and execute
subsystem tests
Design and execute
system tests
Produce user
documentation
Product delivery
Reducing the Impact of
Critical Paths

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 85
UNLIMITED RESOURCES
Project start
Analysis and design
Design subsystem tests
Design system tests
Produce user
documentation
Execute subsystem
tests
Execute system tests
Product delivery
Critical Paths

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 86
LIMITED RESOURCES
Project start
Analysis and design
Design subsystem tests
Design system tests
Produce user
documentation
Execute subsystem
tests
Execute system tests
Product delivery
Critical Paths

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 88
Risk Planning
• Risks cannot be eliminated, but they
can be assessed, controlled, and
monitored
• Generic management risk
– personnel
– technology
– schedule
• Quality risk
– development

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 89
Personnel
Example Risks
• Loss of a staff
member
• Staff member
under-qualified for
task
Control Strategies
• cross training to avoid over-
dependence on individuals
• continuous education
• identification of skills gaps
early in project
• competitive compensation and
promotion policies and
rewarding work
• including training time in
project schedule

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 90
Technology
Example Risks
• High fault rate due
to unfamiliar COTS
component
interface
• Test and analysis
automation tools do
not meet
expectations
Control Strategies
• Anticipate and schedule extra
time for testing unfamiliar
interfaces.
• Invest training time for COTS
components and for training
with new tools
• Monitor, document, and
publicize common errors and
correct idioms.
• Introduce new tools in lower-
risk pilot projects or
prototyping exercises

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 91
Schedule
Example Risks
• Inadequate unit
testing leads to
unanticipated
expense and delays
in integration testing
• Difficulty of
scheduling meetings
makes inspection a
bottleneck in
development
Control Strategies
• Track and reward quality unit
testing as evidenced by low
fault densities in integration
• Set aside times in a weekly
schedule in which inspections
take precedence over other
meetings and work
• Try distributed and
asynchronous inspection
techniques, with a lower
frequency of face-to-face
inspection meetings

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 92
Development
Example Risks
• Poor quality
software delivered
to testing group
• Inadequate unit
test and analysis
before committing
to the code base
Control Strategies
• Provide early warning and
feedback
• Schedule inspection of design,
code and test suites
• Connect development and
inspection to the reward system
• Increase training through
inspection
• Require coverage or other
criteria at unit test level

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 93
Test Execution
Example Risks
• Execution costs higher
than planned
• Scarce resources
available for testing
Control Strategies
• Minimize parts that
require full system to be
executed
• Inspect architecture to
assess and improve
testability
• Increase intermediate
feedback
• Invest in scaffolding

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 94
Requirements
Example Risk
• High assurance
critical requirements
increase expense
and uncertainty
Control Strategies
• Compare planned testing
effort with former projects
with similar criticality level to
avoid underestimating testing
effort
• Balance test and analysis
• Isolate critical parts,
concerns and properties

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 95
Contingency Plan
• Part of the initial plan
– What could go wrong? How will we know, and
how will we recover?
• Evolves with the plan
• Derives from risk analysis
– Essential to consider risks explicitly and in
detail
• Defines actions in response to bad news
– Plan B at the ready (the sooner, the better)

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 96
Preliminary
plan
First
release
Second
release
Emergenc
y plan
Final
plan
…
Evolution of the Plan

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 97
Process Monitoring
• Identify deviations from the quality plan
as early as possible and take corrective
action
• Depends on a plan that is
– realistic
– well organized
– sufficiently detailed with clear,
unambiguous milestones and criteria
• A process is visible to the extent that it

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 98
0
20
40
60
80
100
120
140
160 1
3
5
7
9
Total
Critical
Severe
Moderate
Builds
Evaluate Aggregated Data by Analogy

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 99
Process Improvement
Monitoring and improvement within a
project or across multiple projects:
Orthogonal Defect Classification (ODC)
&Root Cause Analysis (RCA)

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 100
Orthogonal Defect
Classification (ODC)
• Accurate classification schema
– for very large projects
– to distill an unmanageable amount of
detailed information
• Two main steps
– Fault classification
• when faults are detected
• when faults are fixed
– Fault analysis

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 101
ODC Fault Classification
When faults are detected
• activity executed when the fault is revealed
• trigger that exposed the fault
• impact of the fault on the customer
When faults are fixed
• Target: entity fixed to remove the fault
• Type: type of the fault
• Source: origin of the faulty modules (in-house,
library, imported, outsourced)
• Age of the faulty element (new, old, rewritten, re-fixed
code)

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 102
ODC activities and triggers
• Review and Code Inspection
– Design Conformance:
– Logic/Flow
– Backward Compatibility
– Internal Document
– Lateral Compatibility
– Concurrency
– Language Dependency
– Side Effects
– Rare Situation
• Structural (White Box) Test
– Simple Path
– Complex Path
• Functional (Black box) Test
– Coverage
– Variation
– Sequencing
– Interaction
• System Test
– Workload/Stress
– Recovery/Exception
– Startup/Restart
– Hardware Configuration
– Software Configuration
– Blocked Test

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 103
ODC impact
• Installability
• Integrity/Security
• Performance
• Maintenance
• Serviceability
• Migration
• Documentation
• Usability
• Standards
• Reliability
• Accessibility
• Capability
• Requirements

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 104
ODC Fault Analysis
(example 1/4)
• Distribution of fault types versus activities
– Different quality activities target different classes of faults
– example:
• algorithmic faults are targeted primarily by unit testing.
– a high proportion of faults detected by unit testing should belong to
this class
• proportion of algorithmic faults found during unit testing
– unusually small
– larger than normal
 unit tests may not have been well designed
• proportion of algorithmic faults found during unit testing
unusually large
 integration testing may not focused strongly enough on
interface faults

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 105
ODC Fault Analysis
(example 2/4)
• Distribution of triggers over time during field test
– Faults corresponding to simple usage should arise early
during field test, while faults corresponding to complex
usage should arise late.
– The rate of disclosure of new faults should asymptotically
decrease
– Unexpected distributions of triggers over time may indicate
poor system or acceptance test
• Triggers that correspond to simple usage reveal many faults
late in acceptance testing
 The sample may not be representative of the user population
• Continuously growing faults during acceptance test
 System testing may have failed

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 106
ODC Fault Analysis
(example 3/4)
• Age distribution over target code
– Most faults should be located in new and rewritten code
– The proportion of faults in new and rewritten code with
respect to base and re-fixed code should gradually increase
– Different patterns
may indicate holes in the fault tracking and removal process
may indicate inadequate test and analysis that failed in
revealing faults early
– Example
• increase of faults located in base code after porting
 may indicate inadequate tests for portability

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 107
ODC Fault Analysis
(example 4/4)
• Distribution of fault classes over time
– The proportion of missing code faults
should gradually decrease
– The percentage of extraneous faults may
slowly increase, because missing
functionality should be revealed with use
• increasing number of missing faults
 may be a symptom of instability of the product
• sudden sharp increase in extraneous faults
 may indicate maintenance problems

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 108
Improving the Process
• Many classes of faults that occur frequently are
rooted in process and development flaws
– examples
• Shallow architectural design that does not take into account
resource allocation can lead to resource allocation faults
• Lack of experience with the development environment, which
leads to misunderstandings between analysts and
programmers on rare and exceptional cases, can result in faults
in exception handling.
• The occurrence of many such faults can be reduced
by modifying the process and environment
– examples
• Resource allocation faults resulting from shallow architectural
design can be reduced by introducing specific inspection tasks
• Faults attributable to inexperience with the development
environment can be reduced with focused training

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 109
Improving Current and Next
Processes
• Identifying weak aspects of a process
can be difficult
• Analysis of the fault history can help
software engineers build a feedback
mechanism to track relevant faults to
their root causes
– Sometimes information can be fed back
directly into the current product
development
– More often it helps software engineers

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 110
Root cause analysis (RCA)
• Technique for identifying and eliminating
process faults
– First developed in the nuclear power
industry; used in many fields.
• Four main steps
– What are the faults?
– When did faults occur? When, and when
were they found?
– Why did faults occur?

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 111
What are the faults?
• Identify a class of important faults
• Faults are categorized by
– severity = impact of the fault on the product
– Kind
• No fixed set of categories; Categories evolve
and adapt
• Goal:
– Identify the few most important classes of faults and
remove their causes
– Differs from ODC: Not trying to compare trends for
different classes of faults, but rather focusing on a

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 112
Fault Severity
Level Description Example
Critical The product is unusable The fault causes the program to crash
Severe Some product features
cannot be used, and there
is no workaround
The fault inhibits importing files
saved with a previous version of the
program, and there is no workaround
Moderate Some product features
require workarounds to
use, and reduce
efficiency, reliability, or
convenience and usability
The fault inhibits exporting in
Postscript format.
Postscript can be produced using the
printing facility, but with loss of
usability and efficiency
Cosmetic Minor inconvenience The fault limits the choice of colors
for customizing the graphical
interface, violating the specification
but causing only minor inconvenience

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 113
Pareto Distribution (80/20)
• Pareto rule (80/20)
– in many populations, a few (20%) are vital
and many (80%) are trivial
• Fault analysis
– 20% of the code is responsible for 80% of
the faults
• Faults tend to accumulate in a few modules
– identifying potentially faulty modules can improve the
cost effectiveness of fault detection
• Some classes of faults predominate
– removing the causes of a predominant class of faults

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 114
Why did faults occur?
• Core RCA step
– trace representative faults back to causes
– objective of identifying a “root” cause
• Iterative analysis
– explain the error that led to the fault
– explain the cause of that error
– explain the cause of that cause
– ...
• Rule of thumb

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 115
Example of fault tracing
• Tracing the causes of faults requires experience,
judgment, and knowledge of the development
process
• example
– most significant class of faults = memory leaks
– cause = forgetting to release memory in exception handlers
– cause = lack of information: “Programmers can't easily
determine what needs to be cleaned up in exception
handlers”
– cause = design error: “The resource management scheme
assumes normal flow of control”
– root problem = early design problem: “Exceptional conditions
were an afterthought dealt with late in design”

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 116
How could faults be
prevented?
• Many approaches depending on fault and process:
• From lightweight process changes
– example
• adding consideration of exceptional conditions to a design
inspection checklist
• To heavyweight changes:
– example
• making explicit consideration of exceptional conditions a part of
all requirements analysis and design steps
Goal is not perfection, but cost-effective
improvement

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 117
The Quality Team
• The quality plan must assign roles and
responsibilities to people
• assignment of responsibility occurs at
– strategic level
• test and analysis strategy
• structure of the organization
• external requirements (e.g., certification
agency)
– tactical level
• test and analysis plan

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 118
Roles and Responsibilities
at Tactical Level
• balance level of effort across time
• manage personal interactions
• ensure sufficient accountability that quality tasks are not
easily overlooked
• encourage objective judgment of quality
• prevent it from being subverted by schedule pressure
• foster shared commitment to quality among all team
members
• develop and communicate shared knowledge and values
regarding quality

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 119
Alternatives in Team Structure
• Conflicting pressures on choice of structure
– example
• autonomy to ensure objective assessment
• cooperation to meet overall project objectives
• Different structures of roles and responsibilities
– same individuals play roles of developer and tester
– most testing responsibility assigned to a distinct group
– some responsibility assigned to a distinct organization
• Distinguish
– oversight and accountability for approving a task
– responsibility for actually performing a task

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 120
Roles and responsibilities
pros and cons
• Same individuals play roles of developer and tester
– potential conflict between roles
• example
– a developer responsible for delivering a unit on schedule
– responsible for integration testing that could reveal faults that
delay delivery
– requires countermeasures to control risks from conflict
• Roles assigned to different individuals
– Potential conflict between individuals
• example
– developer and a tester who do not share motivation to deliver a
quality product on schedule
– requires countermeasures to control risks from conflict

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 121
Independent Testing Team
• Minimize risks of conflict between roles played by the
same individual
– Example
• project manager with schedule pressures cannot
– bypass quality activities or standards
– reallocate people from testing to development
– postpone quality activities until too late in the project
• Increases risk of conflict between goals of the
independent quality team and the developers
• Plan
– should include checks to ensure completion of quality
activities
– Example
• developers perform module testing
• independent quality team performs integration and system

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 122
Managing Communication
• Testing and development teams must share the goal
of shipping a high-quality product on schedule
– testing team
• must not be perceived as relieving developers from
responsibility for quality
• should not be completely oblivious to schedule pressure
• Independent quality teams require a mature
development process
– Test designers must
• work on sufficiently precise specifications
• execute tests in a controllable test environment
• Versions and configurations must be well defined
• Failures and faults must be suitably tracked and
monitored across versions

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 123
Testing within XP
• Full integration of quality activities with development
– Minimize communication and coordination overhead
– Developers take full responsibility for the quality of their work
– Technology and application expertise for quality tasks match
expertise available for development tasks
• Plan
– check that quality activities and objective assessment are
not easily tossed aside as deadlines loom
– example
• XP “test first” together with pair programming guard against
some of the inherent risks of mixing roles

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 124
Outsourcing Test and Analysis
• (Wrong) motivation
– testing is less technically demanding than development and
can be carried out by lower-paid and lower-skilled individuals
• Why wrong
– confuses test execution (straightforward) with analysis and
test design (as demanding as design and programming)
• A better motivation
– to maximize independence
• and possibly reduce cost as (only) a secondary effect
• The plan must define
– milestones and delivery for outsourced activities
– checks on the quality of delivery in both directions

(c) 2007 Mauro
Pezzè & Michal
Ch 20, slide 125
Summary
• Planning is necessary to
– order, provision, and coordinate quality
activities
• coordinate quality process with overall
development
• includes allocation of roles and responsibilities
– provide unambiguous milestones for
judging progress
• Process visibility is key
– ability to monitor quality and schedule at
each step

(c) 2007 Mauro
Pezzè & Michal
Ch 24, slide 126
Documenting Analysis and Test

(c) 2007 Mauro
Pezzè & Michal
Ch 24, slide 127
Learning objectives
• Understand the purposes and
importance of documentation
• Identify some key quality documents
and their relations
• Understand the structure and content of
key quality documents
• Appreciate needs and opportunities for
automatically generating and managing
documentation

(c) 2007 Mauro
Pezzè & Michal
Ch 24, slide 128
Why Produce Quality
Documentation?
• Monitor and assess the process
– For internal use (process visibility)
– For external authorities (certification,
auditing)
• Improve the process
– Maintain a body of knowledge reused
across projects
– Summarize and present data for process
improvement

(c) 2007 Mauro
Pezzè & Michal
Ch 24, slide 129
Major categories of
documents
• Planning documents
– describe the organization of the quality
process
– include organization strategies and project
plans
• Specification documents
– describe test suites and test cases
(as well as artifacts for other quality tasks)
– test design specifications, test case
specification, checklists, analysis

(c) 2007 Mauro
Pezzè & Michal
Ch 24, slide 130
Metadata
• Documents should include metadata to facilitate
management
– Approval: persons responsible for the document
– History of the document
– Table of Contents
– Summary: relevance and possible uses of the document
– Goals: purpose of the document– Who should read it, and
why?
– Required documents and references: reference to
documents and artifacts needed for understanding and
exploiting this document
– Glossary: technical terms used in the document

(c) 2007 Mauro
Pezzè & Michal
Ch 24, slide 133
Naming conventions
• Naming conventions help people
identify documents quickly
• A typical standard for document names
include keywords indicating
– general scope of the document (project
and part)
– kind of document (for example, test plan)
– specific document identity
– version

(c) 2007 Mauro
Pezzè & Michal
Ch 24, slide 134
W B XX – YY.ZZ
Project or product
(e.g., W for “web
presence”)
Sub-project
(e.g., “Business
logic”)Item
type Identifie
r Versio
n
WB 12 – 22.04
Might specify version 4 of document 12-
22 (quality monitoring procedures for
third-party software components) of web
presence project, business logic
subsystem.
Sample naming standard
example:

(c) 2007 Mauro
Pezzè & Michal
Ch 24, slide 135
Analysis and test strategy
• Strategy document describes quality guidelines for sets
of projects
(usually for an entire company or organization)
• Varies among organizations
• Few key elements:
common quality requirements across products
• May depend on business conditions - examples
– safety-critical software producer may need to satisfy minimum
dependability requirements defined by a certification authority
– embedded software department may need to ensure portability
across product lines
• Sets out requirements on other quality documents

(c) 2007 Mauro
Pezzè & Michal
Ch 24, slide 136
Analysis and Test Plan
• Standardized structure see next slide
• Overall quality plan comprises several individual plans
– Each individual plan indicates the items to be verified through
analysis or testing
– Example: documents to be inspected, code to be analyzed or
tested, ...
• May refer to the whole system or part of it
– Example: subsystem or a set of units
• May not address all aspects of quality activities
– Should indicate features to be verified and excluded
• Example: for a GUI– might deal only with functional properties and
not with usability (if a distinct team handles usability testing)
– Indication of excluded features is important
• omitted testing is a major cause of failure in large projects

(c) 2007 Mauro
Pezzè & Michal
Ch 24, slide 138
Test Design Specification
Documents
• Same purpose as other software design documentation:
– Guiding further development
– Preparing for maintenance
• Test design specification documents:
– describe complete test suites
– may be divided into
• unit, integration, system, acceptance suites (organize by granularity)
• functional, structural, performance suites (organized by objectives)
• ...
– include all the information needed for
• initial selection of test cases
• maintenance of the test suite over time
– identify features to be verified (cross-reference to specification or design
document
– include description of testing procedure and pass/fail criteria (references
to scaffolding and oracles)
– includes (logically) a list of test cases

(c) 2007 Mauro
Pezzè & Michal
Ch 24, slide 139
Test case specification
document
• Complete test design for individual test
case
• Defines
– test inputs
– required environmental conditions
– procedures for test execution
– expected outputs
• Indicates
– item to be tested (reference to design
document)
• Describes dependence on execution of

(c) 2007 Mauro
Pezzè & Michal
Ch 24, slide 140
Test and Analysis Reports
• Report test and analysis results
• Serve
– Developers
• identify open faults
• schedule fixes and revisions
– Test designers
• assess and refine their approach see chapter 20
• Prioritized list of open faults: the core of the fault
handling and repair procedure
• Failure reports must be
– consolidated and categorized to manage repair effort
systematically
– prioritized to properly allocate effort and handle all faults

(c) 2007 Mauro
Pezzè & Michal
Ch 24, slide 141
Summary reports and detailed
logs
• Summary reports track progress and status
– may be simple confirmation that build-and-test cycle ran
successfully
– may provide information to guide attention to trouble spots
• Include summary tables with
– executed test suites
– number of failures
– breakdown of failures into
• repeated from prior test execution,
• new failures
• test cases that previously failed but now execute correctly
• May be prescribed by a certifying authority

(c) 2007 Mauro
Pezzè & Michal
Ch 24, slide 142
Isn’t this a lot of work?
• Yes, but
– Everything produced by hand is actually
used
• Always know the purpose of a document.
Never expend effort on documents that are not
used.
– Parts can be automated
• Humans make and explain decisions. Let
machines do the rest.
• Designing effective quality

(c) 2007 Mauro
Pezzè & Michal
Ch 24, slide 143
Summary
• Mature software processes include
documentation standards for all
activities, including test and analysis
• Documentation can be inspected to
– verify progress against schedule and
quality goals
– identify problems
• Documentation supports process
visibility, monitoring, and

metrics.ppt

More Related Content

What's hot (20)

Similar to metrics.ppt (20)

Recently uploaded (20)

metrics.ppt