KI Measurement Program Starter Kit v3.1

KI MeasurementKI Measurement
Program Starter KitProgram Starter Kit

KI Measurement Pgm Starter Kit - 2Version 3.1© 2008 Kasse Initiatives, LLC
WelcomeWelcome
Wilkommen
Bienvenido
WelKom
Bienvenue
Bienvenuto
Velkommen
Tervetuloa Witamy
Huan Yín
ЌАΛΟΣ ΟΡΙΣΑΤΕ
ようこそ
Välkommen

AgendaAgenda
 Measurement – Is it Really Necessary?
 Metrics
 Goal – Question – Metric Paradigm
 Vision, Business Objectives and Measurement Objectives
 Measurement and Analysis
 Basic Measures
 Effectiveness of Processes
 Set of Organizational Processes
 Slightly More Advanced Measures
 Peer Reviews
 Test Coverage
 Quality Factors, Quality Criteria and Quality Metrics

Agenda - 2Agenda - 2
 Quantitative Project Management
 Path to Maturity Level 4
 Understanding Variation
 Measures and Analytic Techniques
 Descriptive Statistics
 Statistical Techniques
 Statistical Methods
 Causal Analysis Techniques
 Visual Display and other PresentationTechniques
 Summary

MeasurementMeasurement
Is It ReallyIs It Really
Necessary?Necessary?

At Project Start, Do YouAt Project Start, Do You
Know…?Know…?
 Can it be done?
 How long will it take?
 How much will it cost?
 How many people will it take?
 What is the risk?
 What are the tradeoffs?
 How many errors will there be?

What Do You Know Now?What Do You Know Now?
 How much does your current development
process cost?
 How much value does each piece of the
process add?
 What would the impact be of deleting,
modifying, adding a procedure to the process?
 What activities contribute the most to the final
product cost?
 Have you tried to improve the current
development process?
 What changes in cost/value resulted from that
improvement effort?

What Do You Know Now? - 2What Do You Know Now? - 2
 What processes represent the greatest potential
for return on improvement investment?
 How would you quantify the value of the
process improvement investment?
 Do you really want to know where the money is
going in your software development projects?
 What value do you think you are delivering to
your customers? Do they agree?
 How much is the knowledge of your costs and
the value delivered worth to you?

andand
MetricsMetrics

MetricsMetrics
 The term, ‘quality metric’, may be defined as a
measure of the extent or degree to which a
product possesses and exhibits a certain
(quality) or characteristic.
 Quality metrics deal with, for example, Number
of defects, or defects per thousand lines of code
– i.e., a measure of fitness for use

What Are MetricsWhat Are Metrics
 Quantitative measures of
 Process
 Product
 Cost
 Quality
 With the goals of
 Facilitating control
 Detecting deviations
 Identifying potential areas for improvement
 Determining if you are improving

(Taken from “Software Quality: How to Define, Measure, and Achieve It”,
Victor Basili, Department of Computer Science, University of Maryland)
Views of MetricsViews of Metrics
 Subjective
 No exact measurement
 An estimate of the degree a technique is applied
 A classification of a problem or experience
 An indicator
 Objective
 An absolute measure taken on the product or
process
time for development
number of lines of code

Views of Metrics - 2Views of Metrics - 2
 Product
 Measure of the actual developed product
lines of Source code
number of Documents
 Process
 Measure of the process model used for developing
the product
use of methodology

 Cost
 Expenditure of resources
staff months
capital investment
productivity
 Quality
 Value of the product
reliability
ease of use
maintainability

 Metrics can be used to measure:
 Status
Number of requirements
Number of hours spent on Quality Assurance
activities
Number of errors discovered by a customer
 Effectiveness
Effectiveness of Requirements Engineering
process
Effectiveness of Quality Assurance activities
Effectiveness of Peer Reviews

Metrics ConsiderationsMetrics Considerations
 Metrics are not free!
 Do not collect a metric unless you have:
a purpose/objective for collecting the metric
determined it is worth the cost of collecting it
 Use metrics as a tool not a weapon
 Use metrics as a tool for identifying and measuring
improvement activities
 Don’t use metrics to assign blame
 Metrics will change the behavior of those
required to collect them or the raw data that will
be used to derive the metrics

Goal Question MetricGoal Question Metric
(GQM)(GQM)
ParadigmParadigm

The Goal/Question/MetricThe Goal/Question/Metric
ParadigmParadigm
 The G/Q/M Paradigm is a well-known process
used to support development of a
measurement program.
 The process, regenerated by Basili, Rombach
and others, uses the goal/question/metric
framework as the structure for the
measurement process.
 Goals are issues of importance for the organization
 Questions define the issues in such a manner that
their answers indicate progress toward achieving the
Goals
 Metrics supply the data that provide the answers to
the Questions that indicate the status of efforts to
achieve the Goals

Issues of importance
to the organization
Characterize the Goals
(used to provide insight as to
the achievement of the goals)
Paradigm - 2Paradigm - 2
MetricsMetrics
QuestionsQuestions
GoalsGoals
Answer the Questions
(provide status and trends)
The Goal/Question/Metric
Framework is a commonly
Used structure for the
Measurement process
The Goal/Question/Metric
Framework is a commonly
Used structure for the
Measurement process

Paradigm - 3Paradigm - 3
Goal 2
Question 4 Question 5
Goal 1
Question 2Modularity
Question 1
Question 3
Metric 1 Metric 2 Metric 3
Metric 4

GQM MethodologyGQM Methodology
 Three High-Level Steps:
 Determine the Goal/Purpose/Objective to be
achieved (or Issue to be resolved)
 Develop questions which when answered will show
whether the goal/purpose/objective has been
achieved or the issue resolved
 Formulate quantitative answers to the questions
(these are the metrics you may want to collect)

G/Q/M Methodology - 2G/Q/M Methodology - 2
 Establish the goals of the data collection
 Develop a list of questions
 Specify the measures to answer the questions
 Collect the data
 Validate and analyze the data
 Apply the results to the project – Is the metric a
good indicator?
 Analyze measurement process for
improvement

Exercise 1Exercise 1
 Use the GQM Paradigm to develop measures
for one or more of these hard to quantify
Requirements Words

Vision,Vision,
Business Objectives,Business Objectives,
andand
Measurement ObjectivesMeasurement Objectives

VisionVision
 Where does senior management think the organization will
be in the next year, and in the next two to five years?
 What products will be in the mainstream?
 Who will the competitors be?
 Will there be collaborators or strategic alliance partners?
 What technology changes are expected and/or will be
required to support the vision?
 What does the organizational structure have to be to support
this vision?
 Who will the organization’s suppliers be?
 What must the organizational culture be to support this
vision?
 How will a Process Improvement Initiative support this
vision?

Business ObjectivesBusiness Objectives
 Examples of Business Objectives include:
 Reduce time to market
 Reduce system errors that are discovered by customers
 Improve delivery time
 Increase quality of products
 Find and fix software defects once and only once
 Reduce project risks
 Gain control of suppliers
 Improve service delivery
 Improve service availability and capacity
 Shorten find to fix repair rate

 An organization’s measurement objectives
might be:
 Reduce time to delivery to a specified percentage
 Reduce total lifecycle costs of new products by a
percentage
 Deliver specified functionality by a specified
increased percentage
 Improve prior levels of quality by reducing the
number of defects of type A that get shipped with the
product
 Improve prior customer satisfaction ratings by a
specified percentage compared to past ratings

Measurement and AnalysisMeasurement and Analysis
vs.vs.
Project Monitoring and ControlProject Monitoring and Control
 Understanding the organization’s business
objectives and the project’s information needs
based on those organization’s business
objectives as well as its own information needs
or project’s business objectives, is the first
major requirement for establishing the
organization’s measurement foundation
 Without this, measurement gets reduced to status
information that is normally collected through project
monitoring and control

and Analysisand Analysis

OverviewOverview
 A measurement initiative involves the following:
 Specifying the objectives of measurement and
analysis such that they are aligned with established
information needs and business objectives
 Defining the measures to be used, the data collection
process, the storage mechanisms, the analysis
processes, the reporting processes, and the
feedback processes

Sources of Information NeedsSources of Information Needs
 The CMMI provides us with some examples of sources
of information needs including:
 Project plans
 Monitoring of project performance
 Established management objectives at the organizational level
or project level
 Strategic plans
 Business plans
 Formal requirements or contractual obligations
 Recurring or other troublesome management or technical
problems
 Experiences of other projects or organizational entities
 External industry benchmarks
 Process improvement plans at the organizational and project
level

Sources of Information Needs - 2Sources of Information Needs - 2
 What is it about the project plans or technical
problems or experiences of other projects or
external industry benchmarks like CMMI appraisals
that suggests an information need?
 Have our ongoing project has not been meeting their
delivery dates?
 Have other projects have not been able to meet the
functionality promises that were made?
 Have technical problems that have reached production
caused significant rework and customer dissatisfaction?

 The initial focus for measurement activities is at
the project level, however, a measurement
capability may prove useful for addressing
organization- and/or enterprise-wide information
needs.
 Measurement activities should support
information needs at multiple levels including
the business, organizational unit, and project to
minimize re-work as the organization matures.
Project, OrganizationProject, Organization
and Business Focusand Business Focus

 While establishing measurement objectives, a
project/organization should:
 Document the purposes for which measurement and analysis
is done
 Specify the kinds of actions that may be taken based on the
results of the data analyses
 Continually ask the question – what value will this
measurement be to those people who will be asked to supply
the raw measurement data and who will receive the analyzed
results – “Why are we measuring this?”
 Maintain traceability of the proposed measurement objectives
to the information needs and business objectives
 Ensure business objectives are developed with clear
“WHYs” this measure will support the business and
quality goals of the organization (SEE NOTES)
Establish Measurement ObjectivesEstablish Measurement Objectives

 Example Measurement Objectives for either the
organization and/or the project to start with
include:
 Reduce time to delivery based on historical data
indicating late delivery
 Deliver specified functionality by a specified
increased percentage
 Improve prior levels of quality
 Improve levels of profit
 Improve prior customer satisfaction ratings
Establish MeasurementEstablish Measurement
Objectives - 2Objectives - 2

 Example Measurement Objectives for either the
organization and/or the project with more
emphasis on quantitative measures include:
 Reduce time to delivery to a specified percentage
 Reduce total lifecycle costs of new products by a
percentage
 Deliver specified functionality by a specified increased
percentage
 Improve prior levels of quality by reducing the number
of defects of type A that get shipped with the product
 Improve prior customer satisfaction ratings by a
specified percentage compared to past ratings
 Refer to Organizational Process Performance SP 1.3
Establish MeasurementEstablish Measurement
Objectives - 3Objectives - 3

Project’s Measurement ObjectivesProject’s Measurement Objectives
Organization’s
Measurement
Objectives
Customer
Demands
Competition
Demands
New Technologies
Opportunities
Past Project
Quality Defects
Quality Goals
Project’s
Measurement
Objectives
Given
Inherited

 Project Managers should develop their project’s
measurement objectives from their individual information
needs – not one objective for all projects
 Reduce open problem reports that come from the field when the
product is released through more and better conducted
Inspections and formal Unit Testing
 Increase defect detection found earlier in the product and
system lifecycle through Systems Test in order to reduce the
“Time to Delivery”
 Increase the number of Peer Reviews in order to reduce the
number of defects of Type A that has been shipped in previous
releases
 Reduce the number of maintenance releases to the field
through detection and removal of an increased percentage of
Major defects that reduces bottom-line profit
 Decrease the defect density of components, products and
systems in order to “Reduce the Cost of Poor Quality
Example: Project’sExample: Project’s

Base & Derived MeasuresBase & Derived Measures
 Base Measure
 A distinct property or characteristic of an
entity and the method for quantifying it.
 Derived Measure
 Data resulting from the mathematical function
of two or more base measures.

 Examples of commonly used base measures
 Estimates and actual measures of work product size
 Estimates and actual measures of effort and cost
 Estimates and actuals of environment resources
Base MeasuresBase Measures

 Define how data can and will be derived from
other measures
 Data may be generated from derived measures
which are based on combinations of data that were
collected for the defined basic measures
 Derived measures typically are expressed as
ratios, composite indices, or other aggregate
summary measures
 Derived measures are often more quantitatively
reliable and meaningfully interpretable than the
base measures used to construct them
 Moving from attribute (ordinal or interval data) to
continuous or ratio data – SEE NEXT SLIDE!
Derived MeasuresDerived Measures

Data TypesData Types
Interval data that has an
absolute zero
Ratio
Productivity
Defect Density
Preparation Rate
Cycle Time
Size
Test Hours
Data measured on a scale
that has equal intervals
IntervalContinuous Data
Severity ratings
Priority ratings
Customer Satisfaction ratings
High, Medium, or Low ratings
Categories or buckets of
data with ordering
Ordinal
Defect types
Language types
Customers
Document types
Categories or buckets of
data with no ordering
NominalAttribute or Categorical Data
ExamplesDescriptionTypes of Data
From SEI Designing Products and Processes Using Six Sigma
Basic Statistics Reference - 4

 Examples of commonly used derived measures
 Earned Value (actual cost of work performed
compared to the budgeted cost of work performed)
 Schedule Performance Index
 Cost Performance Index
 Defect density (Defects per Thousand Lines of Code)
 Peer review coverage
 Test or verification coverage
 Usability
 Reliability measures (e.g., mean time to failure)
 Quality measures (e.g., number of defects by
severity/total number of defects)
Commonly UsedCommonly Used
Derived MeasuresDerived Measures

Specify Data Collection andSpecify Data Collection and
Storage ProceduresStorage Procedures
 Specify how to collect and store the data for
each required measure
 Make explicit specifications of how, where, and
when the data will be collected
 Develop procedures for ensuring that the data
collected is valid data
 Ensure that the data is stored such that it is easily
accessed, retrieved, and restored as needed

Specify Analysis ProceduresSpecify Analysis Procedures
 Define the analysis procedures in advance
 Ensure that the results that will be fed back are
understandable and easily interpretable
 Collecting data for the sake of showing an assessor
the data is worthless
 Showing how it can be used to manage and control
the project is what counts

Specify Analysis Procedures - 2Specify Analysis Procedures - 2
Visual Display and Other Presentation Techniques
Bar Charts
Pie Charts
Radar Charts (Kiviat Diagrams)
Line Graphs
Scatter Diagrams
Check Lists
Interrelationship Diagraphs
SEE Examples of Techniques in Quantitative Management Section

Specify Analysis Procedures - 3Specify Analysis Procedures - 3
 Descriptive Statistics
 Mean (Average)
 Median
 Mode
 Distributions
 Central Tendency
 Extent of Variation

X
X X X
X
Data points vary, but as the data accumulates, it forms a distribution which occurs naturally.
Location Spread Shape
Distributions can vary in:
PROBABILITY DISTRIBUTIONS,
WHERE DO THEY COME FROM?

CollectCollect
Measurement DataMeasurement Data
 Collect the measurement data as defined, at the
points in the process that were agreed to,
according to the time scale established
 Generate data for derived measures
 Perform integrity checks as close to the source
of the data as possible

Analyze the Measurement DataAnalyze the Measurement Data
 Conduct the initial analyses
 Interpret the results and make preliminary conclusions
from explicitly stated criteria
 Conduct additional measurement and analyses
passes as necessary to gain confidence in the results
 Review the initial results with all stakeholders
 Prevents misunderstandings and rework
 Improve measurement definitions, data collection
procedures, analyses techniques as needed to ensure
meaningful results that support business objectives

Store the Measurement DataStore the Measurement Data
and Analyses Resultsand Analyses Results
 The stored information should contain or
reference the information needed to:
 Understand the measures
 Assess them for reasonableness and applicability
 The stored information should also:
 Enable the timely and cost effective future use of the
historical data and results
 Provide sufficient context for interpretation of the
data, measurement criteria, and analyses results

Communicate theCommunicate the
Measurement ResultsMeasurement Results
 Keep the relevant stakeholders up-to-date
about measurement results on a timely basis
 Follow up with those who need to know the
results
 Increases the likelihood that the reports will be used
 Assist the relevant stakeholders in
understanding and interpreting the
measurement results

GroupGroup
 Consider creating a measurement group that is
responsible for supporting the Measurement
and Analysis activities of multiple projects
 Typically the measurement group will support the
definition, collection, analysis, and presentation of
measures that address these measurement
objectives and support project estimation and
tracking

ToolsTools
 Incorporate tools used in performing
Measurement and Analysis activities such as:
 Statistical packages
 Database packages
 Spreadsheet programs
 Graphical or Visualization packages
 Packages that support data collection over networks
and the internet

TrainingTraining
 Provide training to all people who will perform or
support the Measurement and Analysis process
 Data collection, analyses, and reporting processes
 Measurement tools
 Goal-Question-Metric Paradigm
 How to establish measures
how to determine efficiency and effectiveness
 Quality factors measures (e.g., maintainability,
expandability)
 Basic and advanced statistical techniques

 In small teams, write down what you think the
organization’s Vision, Business Objectives and
Measurement Objectives are
 Compare the responses from each small team

Basic MeasuresBasic Measures

Basic MeasuresBasic Measures
 Estimate Size and/or Complexity - a relative
level of difficulty or complexity should be
assigned for each size attribute
 Examples of attributes to estimate for Systems
Engineering include:
 Number of logic gates
 Number of interfaces
 Examples of size measurements for Software
Engineering include:
 Function Points
 Lines of Code
 Number of requirements

Basic Measures - 2Basic Measures - 2
 Determine effort and cost
 Historical data or models are applied to planning parameters to
determine the project effort and cost based on the size and
complexity estimations
 Scaling data should also be applied to account for differing
sizes and complexity
 Establish the project’s schedule based on the size and
complexity estimations
 Include, or at least consider, infrastructure needs such
as critical computer resources
 Identify risks associated with the cost, resources,
schedule, and technical aspects of the project
 Control data (various forms of documentation) required
to support a project in all of its areas.

Basic Measures – 3Basic Measures – 3
 Identify the knowledge and skills needed to
perform the project according to the estimates
 Select and implement methods for providing the
necessary knowledge and skills
 Training (Internal and External)
 Mentoring
 Coaching
 On-the-job application of learned skills
 Monitor staffing needs – based on effort
required and the necessary knowledge and
skills to achieve the defined tasks

Staff Size
(Labor Category)
(Experience)
Months
2 4 6 8 10 12 14 16 18 20 22 24 26
Total
Lost
Added
Project Staff TurnoverProject Staff Turnover

 Involve relevant stakeholders throughout the
product lifecycle
 Track technical performance (Completion of
activities and milestones against the
schedule Example:
 Product components designed, constructed, unit
tested and integrated
 Compare actual milestones completed vs.
established commitments
 Monitor commitments and critical dependencies
against those documented in the project plan

 Track quality – Problems/Defects (open/closed
by product/activity)
 Problems and defects are direct contributors to the
amount of rework that must be performed—a
significant cost factor in development and
maintenance

 The number and frequency of problems and
defects in a product are inversely proportional
to its quality
 Problems and defects are among the few direct
measures of processes and products
 Tracking them provides objective insight into
trends in discovery rates, repairs, process and
product issues, and responsiveness to
customers
 The measures also provide the foundation for
quantifying several of the quality attributes —
maintainability, expandability, reliability,
correctness, completeness

 Problems and defects are direct contributors to
the amount of rework that must be performed—
a significant cost factor in development and
maintenance
 Knowledge of where and how the
problems/defects occur will support
improvement in methods of detection,
prevention, and prediction—all of which will
improve cost control

 As a class, develop a definition of rework

Effectiveness ofEffectiveness of
ProcessesProcesses

Effectiveness of ProcessesEffectiveness of Processes
 In addition to defining the processes that we
wish to follow on our project, we need to ensure
we are following them and we should be able to
determine if the processes are working for us
the way we expected them to
 How well are the processes working?

Efficiency and EffectivenessEfficiency and Effectiveness
Measures for RequirementsMeasures for Requirements
 Number of change requests per month
compared with the original number of
requirements for the project
 Critical change requests
 Intermediate change requests
 Nice to have change requests
 Time spent on change requests up until a Y/N
decision is given from the Senior Contract
group
 Number and size of critical change requests
that arise after the requirements phase has
been completed

Measures for Requirements - 2Measures for Requirements - 2
 Impact of the change requests on project
progress - effort spent on the change requests
versus the amount of effort to execute the
original project
 Actual cost of processing a change request
compared with budgeted or predicted costs
 Actually make the change
 Filling in the forms
 Impact Analysis
 Authorization
 Replanning

Measures for Requirements - 3Measures for Requirements - 3
 Rescheduling
 Re-negotiating commitments
 SQA effort
 SCM effort
 Test effort
 Number of change requests accepted versus
the total number of change requests during the
project’s lifetime
 Number of change requests accepted but not
implemented in a given time frame

Set of StandardSet of Standard
OrganizationalOrganizational
ProcessesProcesses

Importance of an OrganizationalImportance of an Organizational
View of ProcessesView of Processes
 Builds a common vocabulary
 Allows others to anticipate behavior and be
more proactive in their interactions
 Allows the organization to measure a controlled
set of processes to gain economy of scale
 Trends can be seen and predictability can be
achieved
 Process performance baselines can be
developed to support quantitative management
later

Organizational MeasurementOrganizational Measurement
RepositoryRepository
 Develop an organization measurement
repository - include:
 Product and process measures that are related to
the organization’s set of standard processes
 The related information needed to understand
and interpret the measurement data and asses it
for reasonableness and applicability
 Develop operational definitions for the
measures to specify the point in the process
where the data will be collected and for the
procedures for collecting valid data

Organizational MeasurementOrganizational Measurement
Repository - 2Repository - 2
 Examples of classes of commonly used
measures include:
 Size of work products (lines of code, function or
feature points, complexity)
 Effort and cost
 Actual measures of size, effort, and cost
 Quality measures
 Work product inspection coverage
 Test or verification coverage
 Reliability measures

Slightly MoreSlightly More
Advanced MeasuresAdvanced Measures

Peer ReviewsPeer Reviews

Defect TypesDefect Types
 A Major defect is one that could cause a failure or
unexpected result if uncorrected.
 For documents it is major if it could cause the user to make a
mistake.
 A Major Defect can have a negative impact on factors
such as:
 Cost
 Schedule
 Performance
 Quality
 Risk
 Customer Satisfaction
 Each organization must define for itself what a
major defect is in relation to Inspections and
Structured Walkthroughs

Defect Types - 2Defect Types - 2
 A Minor defect is one that won’t cause a failure
or unexpected result if uncorrected.
 Economically and/or strategically unimportant
to the organization
 No serious impact to the product
 Inconsistency in format
 Spelling or grammar in a project plan

Defect CorrectionDefect Correction
 A defect may be identified as Minor and turn out
to be Major
 Identify and correct Major defects FIRST, to
ensure the highest return of error correction
 Inspection should be used to increase the
probability that all of the defects are identified
and corrected to produce the highest quality
product

Defect ClassificationDefect Classification
 Once a defect is identified as Major or Minor, it
should be classified
 Categorizes the defect
 Provides the rationale for determining if a defect
exists
 Stratifies the defect data collected for better trend
analysis, causal analysis and process improvement

Classification ExamplesClassification Examples
 Logic (LO) – Some aspect of logic was omitted
or implemented incorrectly in the product
 Duplicate Logic
 Extreme Conditions Neglected
 Unnecessary Function
 Missing Condition Test
 Computational Problem (CP) – Some aspect of
an algorithm was incorrectly coded
 Interface (IF) – Some aspect of the software or
hardware interfaces does not function properly
 Example: Interface defects between two programs,
between two systems, or the interface between a
user and the system

Classification Examples - 2Classification Examples - 2
 Data Handling Problem (DH) – Some aspect of
data manipulation was handled incorrectly
 Quality Factors (QF) – Quality factors such as
reliability, maintainability, expandability or
interoperability are not defined or defined
incorrectly
 Verification and validation activities will not be able
to show the system exhibits the quality
characteristics that are required
 Process Failure (PF) – This defect is a direct
result of a failure in the product development
process

 Ambiguous (AM) – The statement can be
interpreted to mean more than one thing
 Requirements or specifications have uncertain or
multiple interpretations
 Incomplete Item (IC) – The statement or
description does not seem to consider all
aspects of the situation it attempts to describe
 Incorrect Item (IT) – The statement or
description is incorrect
 Missing Item (MI) – The statement or
description that must be included in the
document is missing

 Conflicting Items (CF) – Two or more
statements or descriptions conflict or contradict
each other.
 Redundant Items (RD) – The statement
repeats another statement and detracts from
clarity rather than enhancing it
 Illogical Item (IL) – The statement does not
make sense in reference to other statements
within the same document or other documents
to which it refers
 Non-Verifiable Item (NV) – The statement
(usually a requirement) or functional
description cannot be verified by any
reasonable testing method

 Unachievable Item (UA) – The statement
cannot be true in the reasonable lifetime of the
product
 Interoperability Problem (IP) – The product or
product component is not compatible with other
system products or product components
 Standards Conformance Problem (ST) – The
product or product component does not
conform to a standard, where conformance to a
particular standard is specified in the
requirements

Peer ReviewPeer Review
MeasuresMeasures
 Optimum Checking Rate (e.g., Number of
pages to be checked per hour)
 Logging Rate (e.g., Number of major and
defects logged per hour)
 Number of Major and Minor Defects
 Effectiveness - Number of Major Defects
found in this stage compared to the total
number of defects found so far

Peer ReviewPeer Review
Measures - 2Measures - 2
 Correct-Fix Rate – the percentage of edit
correction attempts which correctly fix a defect
and do not introduce any new defects
 Default: 83% five out of six correction attempts
 Fix-Fail-Rate – the percentage of edit correction
attempts which either fail to correct the defect or
introduce a new defect
 Default: 17% one out of six correction attempts

TestingTesting

Defects Discovered DuringDefects Discovered During
TestingTesting
 Effectiveness - Number of Major defects found
in a particular testing phase or instantiation of a
testing phase compared to the total number of
defects found during testing
 Number of defects projected to escape from the
current testing phase

TestTest
CoverageCoverage

Test Coverage TerminologyTest Coverage Terminology
 Code coverage analysis is the process of
 Finding areas of a program not exercised by a set of
test cases
 Creating additional test cases to increase coverage
 Determining a quantitative measure of code
coverage, which is an indirect measure of quality
 Code coverage analysis is sometimes called
test coverage analysis
 The terms are most often shortened to simple
code coverage or test coverage

Statement CoverageStatement Coverage
 Statement coverage measures whether each
executable statement is encountered
 Block coverage is the same as statement
coverage except that the unit of code measured
is each sequence of non-branching statements

Statement Coverage - 2Statement Coverage - 2
 Do-While loops are considered the same as
non-branching statements
 Statement coverage is completely insensitive
to the logical operators (| | and &&)
 Statement coverage cannot distinguish
consecutive “switch” labels

Decision CoverageDecision Coverage
 Decision coverage measures whether boolean
expressions tested in control structures (such
as if-statements or while-statements) evaluated
to both true and false
 The entire boolean expression is considered one
true-or-false predicate regardless of whether it
contains logical “and” or logical “or” operators
 A disadvantage of decision coverage is that this
measure branches within boolean expressions
which occur due to short-circuit operators

Decision Coverage - 2Decision Coverage - 2
If ( condition1 && (condition2 | | function1()))
statement1;
Else
statement2;
This measure could consider the control structure
completely exercised without a call to function1
The test expression is true when condition1 is true and
condition2 is true
The test expression is false when condition1 is false
The short circuit operators preclude a call to function1

Condition CoverageCondition Coverage
 Condition coverage measures the true or false
outcome of each boolean sub-expression
 Condition coverage is similar to decision
coverage but has better sensitivity to the control
flow
 However, full condition coverage does not
guarantee full decision coverage

Condition Coverage - 2Condition Coverage - 2
Bool f(bool e) {return false;}
Bool a[2] = {false, false};
If (f(a && b)) ….
If (a [int (a && b) ] ) …
If ((a && b) ? False : False) …
All three of the if-statements above branch false
regardless of the values of a and b
However, if you exercise this code with a and b
having all possible combinations of values,
condition coverage reports full coverage

Multiple Condition CoverageMultiple Condition Coverage
 Multiple condition coverage measures whether
every possible combination of boolean
subexpression occurs
 For languages with short circuit operators such
as C, C++, and Java, an advantage of multiple
condition coverage is that it requires very
thorough testing
 Multiple condition testing is similar to condition
testing

Multiple Condition Coverage - 2Multiple Condition Coverage - 2
 A disadvantage of this measure is that is can
be difficult to determine the minimum set of test
cases required
 This measure could also required a varied
number of test cases among conditions that
have similar complexity
a && b && (c | | (d && e))
((a | | b) && (c | | d) ) && e
To achieve full multiple condition coverage, the
first condition requires 6 test cases while the
second condition requires 11 test cases

Path CoveragePath Coverage
 Path coverage measures whether each of the
possible paths in each function have been
followed
 A path is a unique sequence of branches from the
function entry to the exit
 Possible combinations of logical conditions
 Loops introduce an unbounded number of paths
 Boundary-interior path testing considers two
possibilities for loops – zero repetitions and more
than zero
 Do-While loops consider two possibilities also – one
iteration and more than one iteration

Path Coverage - 2Path Coverage - 2
 Path coverage has the advantage of requiring
very thorough testing
 Path coverage has two disadvantages:
 The number of paths is exponential to the number of
branches – a function containing 10 if-statements
has 1024 paths to test – adding one more doubles
the count to 2048
 Many paths are impossible to exercise due to
relationships of data

 In small team come up with some workable
definitions of “Effectiveness of Processes” for:
 Configuration Management
 Supplier Management
 Quality Assurance
 Product Integration

Quality Factors,Quality Factors,
Quality Criteria,Quality Criteria,
andand
Quality MetricsQuality Metrics

User oriented view of
an aspect of product
quality
Software oriented
characteristics or
indicate
quality attributes
Quantitative measures
of characteristics
FACTOR
CRITERIONCRITERIONCRITERION
METRIC METRICMETRIC
Software Quality MetricsSoftware Quality Metrics
 Software Quality is described through a number of
factors (reliability, maintainability)
 Each factor has several attributes that describe it called
criteria
 Each criterion has associated with it several metrics
which taken together quantify the criterion

Correctness......................................Does the software comply with the requirements?
Efficiency..........................................How much resource is needed?
Expandability................................... How easy is it to expand the software?
Flexibility.......................................... How easy is it to change it?
Integrity.............................................How secure is it?
Interoperability.................................Does it interface easily?
Manageability...................................Is it easily managed?
Maintainability..................................How easy is it to repair?
Portability......................................... How easy is it to transport?
Usability............................................How easy is it to use?
Reliability..........................................How often will it fail?
Reusability........................................Is it reusable in other systems?
Safety................................................Does it prevent hazards?
Survivability..................................... Can it survive during failure?
Verifiability....................................... Is performance verification easy?
Software Quality
The “Ilities” of Software QualityThe “Ilities” of Software Quality

User’s Need for SoftwareUser’s Need for Software
QualityQuality
User’s Needs User’s Concerns Quality Factors
Functional
Performance
Change
Management
How secure is it?
How often will it fail?
Can it survive during failure
How easy is it to use?
How much is needed in the way of resources?
Does it comply with requirements?
Does it prevent hazards?
Does it interface easily?
How easy is it to repair?
How easy is it to expand?
How easy is it to change?
How easy is it to transport?
Is it reusable in other systems?
Is performance verification easy?
Is the software easily managed?
INTEGRITY
RELIABILITY
SURVIVABILITY
USABILITY
EFFICIENCY
CORRECTNESS
SAFETY
INTEROPERABILITY
MAINTAINABILITY
EXPANDABILITY
FLEXIBILITY
PORTABILITY
REUSABILITY
VERIFIABILITY
MANAGEABILITY

Software QualitySoftware Quality
FactorsFactors

Quality FactorsQuality Factors
Correctness
Efficiency
Expandability
Flexibility
Integrity
Interoperability
Maintainability
Manageability
 Portability
 Reliability
 Reusability
 Safety
 Survivability
 Usability
 Verifiability

Expandability
Modularity
Augmentability
Modularity Simplicity
Self -
Descriptiveness
Support
Generality
ExpandabilityExpandability
 Expandability deals with the aspects of software
maintenance, that is increasing the software’s
functionality or performance to meet new needs
 Adding a new type of deduction to an existing payroll
program
 Adding an interface to a new sensor

Expandability - 2Expandability - 2
 Fitness for use regarding expandability means
that the software was built to be open ended
making it easy to modify it to add new
capabilities

Maintainability
Modularity
Completeness
Consistency Simplicity
Self
Descriptiveness
Traceability
Visibility
Modularity
MaintainabilityMaintainability
 Maintainability deals with the ease of finding and fixing
errors
 Fitness for use regarding maintainability means that the
software is productive through the maintenance
lifecycle, covering error detection through the issue of a
new release

Portability
ModularityIndependence Support
Modularity Self
Descriptiveness
PortabilityPortability
 Portability deals with transporting the software
to execute on a host processor or operating
system different from the one for which it was
designed
 Recompiling a FORTRAN program on a different
computer
 Changing the operating system of an existing
computer

Portability - 2Portability - 2
 Fitness for use regarding portability means that
the software may be used on several different
operating systems or computers

Usability
Operability Training
UsabilityUsability
 Usability deals with the initial effort required to
learn, and the recurring effort to use the
functionality of the system

Usability - 2Usability - 2
 Usability can be enhanced or degraded by:
 The naturalness of the user interface
 The readability of documentation
 The number of keystrokes required for a given
command
 Fitness for use regarding usability means that
the software is easier to use than not to use

Quality CriteriaQuality Criteria

Quality CriteriaQuality Criteria
 Accuracy Achieving required precision in
calculations and outputs
 Anomaly Management Nondisruptive failure recovery
 Augmentability Ease of expansion in
functionality and data
 Autonomy Degree of decoupling from
execution environment
 Commonality Use of standards to achieve
interoperability
 Completeness All software is necessary and
sufficient
 Consistency Use of standards to achieve
uniformity
 Distributivity Geographical separation of
functions and data

Quality Criteria - 2Quality Criteria - 2
 Document Quality Access to complete
understandable information
 Efficiency of Comm. Economic use of
communication resources
 Efficiency of Processing Economic use of processing
resources
 Efficiency of Storage Economic use of storage
resources
 Functional Scope Range of applicability of a
function
 Generality Range of applicability of a
unit
 Independence Degree of decoupling from
support environment
 Modularity Orderliness of design and
Implementation

 Operability Ease of operating the software
 Safety Management Software design to avoid hazards
 Self-descriptiveness Understandability of design and
source code
 Simplicity Straightforward implementation of
functions
 Support Functionality supporting the
management of changes
 System accessibility Controlled access to software
and data
 System compatibility Ability of two or more systems to
work in harmony

 Traceability Ease of relating code to
requirements and
vice versa
 Training Provisions to learn how to use
the software
 Virtuality Logical implementation to
represent physical
components
 Visibility Insight into validity and
progress of development

Anomaly ManagementAnomaly Management
 The software is said to have Anomaly
Management built in if it can detect and recover
from such error conditions rather than disrupting
processing or halting
 The software should be designed for
survivability when faced with software or
hardware failure

Anomaly Management - 2Anomaly Management - 2
 Anomaly Management includes detection and
containment of, and recovery from:
 Improper input data
 Computational failures
 Hardware faults
 Device failures
 Communication errors
 Suggestions and questions for achieving
required levels of anomaly management:
 Does a documented requirements statement exist for
the error tolerance of input data?

 Is there a range for input values and is this checked?
 Are conflicting requests and illegal combinations
identified and checked?
 Is all input data available for processing and is it
checked before processing is begun?
 Is there a requirement for recovery from
computational failures?
 Are there alternative means to continue execution in
the presence of errors?

 Are loops and multiple index parameters range
tested before use?
 Are subscripts checked?
 Are critical output parameters checked before
processing?
 Is error checking information included in
communications messages?
 Do alternate communication routes exist in case
of failure of the main path?

Quality MetricsQuality Metrics

Quality Metrics ExamplesQuality Metrics Examples
(Reliability)(Reliability)
 Reliability
 Accuracy checks to see that the results produced by
software is within required accuracy tolerances
Do mathematical libraries exist for all
mathematical calculations to achieve the precision
requirements?
Count the number of different data representations
- the lower the count, the higher the probability of
achieving accuracy
Count the number of data representation
conversions - the lower the count, the higher the
probability of achieving accuracy

(Reliability) -(Reliability) - 22
 Reliability
 Anomaly Management checks if the system can
detect and recover from error conditions rather than
disrupting processing or halting?
determine if all input values accepted by a module
has a range of accepted values and if this is
checked before further processing
determine if all loop parameters are range tested
before execution
Do alternate communication paths exist in case of
failure of the main path?

(Reliability)(Reliability) - 3- 3
 Reliability - continued
 Simplicity can be measured using
McCabe’s cyclomatic complexity
counting minimum number of statements per
module, minimum number of module interfaces,
etc.
counting the number of Go To's
counting nesting levels beyond three
 A simple metric is to assess the number of errors per
delivered lines of code

(Portability)(Portability)
 Portability
 Independence
count number of references to underlying
operating system
count number of expressions dependent on word
size
count number of calls to software system library
routines

(Portability)(Portability) - 2- 2
 Portability - continued
 Modularity
count number of times local data is accessed from
outside the module where it resides
count number of times output data is not returned
to the calling unit
count number of times that units are not
separately compilable

 Self-descriptiveness
count the number of modules that are written
according to organization standards
examine the comments on global data definitions -
count deviations from standards
count the number of decision points and transfers
of control that do not have comments provided
count the number of Block and Indentation
Guidelines that have been violated

 Support
count the number of trouble reports closed before
Delivery
count how many modules are able to be tested
through automated testing techniques
Does a reuse library exist?
count the number or percentage of modules in the
library that are reused
Does a database of test software exist?

Quantitative ProjectQuantitative Project
ManagementManagement

When higher degrees of quality and
performance are demanded, the organization
and projects must determine if they have the
ability to improve the necessary processes to
satisfy the increased demands
Achieving the necessary quality and process
performance objectives requires stabilizing
the processes or subprocesses that
contribute most to the achievement of the
objectives
Assuming the technical requirements can be
met, the next decision is to determine if it is
cost effective
QuantitativeQuantitative
Management ConceptsManagement Concepts

Path to Maturity Level 4Path to Maturity Level 4

Why Is Early Consideration ofWhy Is Early Consideration of
Quantitative ManagementQuantitative Management
Important?Important?
 Measurements needed for performing
quantitative management may (or may not) be
different from measurements needed for
analysis performed with defined processes
 To perform quantitative management, analysis
of a history of measurement data is required
 Delaying consideration of measurement needs
for quantitative management will impact the
existing measurement program

Selecting the SubprocessesSelecting the Subprocesses
To Be Statistically ManagedTo Be Statistically Managed
 Criteria should be established to identify which
subprocesses are the main contributors to
achieving the identified quality and process
performance objectives and for which predictable
performance is important
 Identify the product and process attributes of the
selected subprocesses that will be measured and
controlled
 Defect density
 Cycle time
 Test coverage

Understanding VariationUnderstanding Variation
Understanding Variation
The Key to Managing Chaos
Donald J. Wheeler, SPC Press, 2000

Understanding VariationUnderstanding Variation
 Understanding variation is achieved by
collecting and analyzing process and product
measures so that special causes of variation
can be identified and addressed to achieve
predictable performance
 All characteristics of processes and products
display variation when measured over time
 Variation may be due to
 Natural or common causes
 Special or “assignable” causes of variation
 Understanding and controlling variation is the
essence of CMMI Maturity L4 & L5

Common Causes of VariationCommon Causes of Variation
 Common causes of variation
 Variation in process performance due to normal
interaction among the process components (people,
machines, material, environment, and methods)
 Characterized by a stable and consistent pattern of
measured values over time
 Variation due to common cause is random but will
vary within predictable bounds
 Unexpected results are extremely rare
 Predictable is synonymous with in control

Florac, W.A. & Carleton, A.D. Measuring the Software Process
Addison-Wesley, 1999
X X X
X
X
X X
X
x
X
X
X
X
X X
X
Variation in
Measured Values
Frequency
of
Measured
Values Time
X
X X
X
The Concept of ControlledThe Concept of Controlled
VariationVariation

Special Causes of VariationSpecial Causes of Variation
 Special or Assignable causes of variation
 Arise from events that are not part of the normal
process
 Represent sudden or persistent abnormal changes
due to one or more of the process components
inputs to the process
environment
process steps themselves
the way the process steps are executed
 Examples of assignable causes of variation include
inadequately trained people, tool failures, failures to
follow the process

X X X
X
X X X
X
x
X
X
X
X
X
X X
X X X
X
Variation in
Measured Values
Frequency
of
Measured
Values
Time
Concept of Uncontrolled orConcept of Uncontrolled or
Assignable Causes of VariationAssignable Causes of Variation

Process VariationProcess Variation
 Reducing process variation is an important
aspect to quantitative management:
 It is important to focus on subprocesses that can be
controlled to achieve a predictable performance
 Statistical process control is often better
focused on organizational areas such as
Product Lines where there is high similarity of
processes, than on the organization’s entire set
of products

MeasuresMeasures
and Analyticand Analytic
TechniquesTechniques

Select Measures and AnalyticSelect Measures and Analytic
 Specify the operational definitions of the measures,
their collection points in the subprocesses and how
the measures will be validated
 State specific target measures or ranges to be met
for each measured attribute of each selected
process
 Set up the organizational support environment to
support the collection and analysis of statistical
measures
 Identify the appropriate statistical analysis
techniques that are expected to be useful in
statistically managing the selected subprocesses

Select Measures andSelect Measures and
Analytic Techniques - 2Analytic Techniques - 2
 Examples of subprocess control measures include:
 Requirements volatility
 Ratios of estimated to measured values of the
planning parameters
 Coverage and efficiency of work product inspections
 Test coverage and efficiency
 Reliability, Maintainability, and Expandability
 Percentage of the total defects inserted or found in
the different stages of the lifecycle

Descriptive StatisticsDescriptive Statistics

Basic Statistical TermsBasic Statistical Terms
 Mean
 Median
 Mode
 Variance
 Central Tendency and Dispersion

MeanMean
 Suppose you were given five numbers and
asked to find the average or “mean” of those
five numbers
 1 4 5 8 2
 x = 1/5 (1 + 8 + 3 + 6 + 2) = 4
 Let xj = an individual value of x --> the mean of
any number of values x1, x2, …xn can be
represented by
 x = 1/n (x1 + x2 + x3 + x4 + ………xn) OR
 x = 1/n Σ i=1,n xi
 The mean is a measure of central tendency

MedianMedian
 The Median is another measure of central
tendency
 Sort the data by magnitude and the median is the
value in the middle
 There are as many numbers bigger than the median
value as there are smaller
 The median is often more illuminating than the
average where the occasional value might distort the
average

Median - 2Median - 2
 Given the following auction figures for buying a
house in thousands of dollars
185 190 145 220 1060 200 170
 Sorting the numbers yields
145 170 185 190 200 220 1060
 The Mean would be 310
 The Median would be 190 – a much better
indicator
 If there are an even number of values –
average the middle two values to get the
median

ModeMode
 The Mode is another measure of central
tendency. It means the most common number
in the data set
7 8 7 8 8 9 6 5 10 8
 Sorting the data into order yields
5 6 7 7 8 8 8 8 9 10
 If the numbers represented shoe sizes, the
shop owner would probably want to stock the
most commonly sold shoes than the average or
even median foot size of their customers

VarianceVariance
• Two shooters each fire ten shots into a separate target
• The shooter on the left has the tighter group but is off-
target
• The shooter on the right seems to be more on target
but is not as good a shot
More Precise
More Accurate

Variance - 2Variance - 2
 There are two separate concepts
 Accuracy – The distance between the process
average and the target
 Precision – The tightness of the grouping

Central TendencyCentral Tendency
and Dispersionand Dispersion
 Central tendency implies location, the balance
point or middle of a group of values
 Examples: mean, median
 Dispersion implies spread, the distance
between values or how much the values tend to
differ from one another
 Examples: Range, standard deviation

Continuous DistributionContinuous Distribution
x1 x2 x3 x4 x5 x6
x7
-∞ +∞
Spread

Statistical TechniquesStatistical Techniques

 Linear Regression Analysis – Used to define the
mathematical relationship between an output
variable (y) and one or more input variables (x)
 Regression models are used to predict the value of
the outcome or dependent variable (y) as a function
of the value of the input or independent variables (x)
 Logistic Regression – Used to predict a discrete
or attribute (y) outcome using either continuous or
discrete (x) factors
 Nominal
 Ordinal
 Binominal
Examples of Statistical TechniquesExamples of Statistical Techniques

 Monte Carlo Simulation (Allows modeling of
variables that are uncertain
 Can put in a range of values instead of a single value
 Analyzes simultaneous effects of many different
uncertain variables creating a more realistic analysis
 Establishes confidence levels for outcomes
 Process Model Simulation – Describe how
things must/should/could be done instead of
the process itself which describes what
really happens
 A rough anticipation of what the process will look
like
Examples of Statistical Techniques - 2Examples of Statistical Techniques - 2

Confidence and PredictionConfidence and Prediction
Intervals – ExampleIntervals – Example
Regression Depicting Confidence and Prediction
Intervals
320
300
280
260
240
220
200
Regression
U95%PI
L95%PI
U95%CI
L95% CI
Y-Data
2007 Carnegie Mellon University
o
ooo o o oo oo ooo o o oooo oo o ooo oo
30 32 33 34 35 36 37 38 39 40 41 42
X-Data

Statistical MethodsStatistical Methods

 Hypothesis Testing – Evaluate actual process
performance (mean and variation) relative to a
standard or specification to:
 Determine if differences exist between
processes
 Verify process improvements by comparing
before and after process performance baselines
(PPBs)

 Analysis of Variance (ANOVA) – Test for significant
differences on more than two group means and
estimate the 95% confidence interval of each group
mean
 Used together with Dummy Variable Regression
 Chi Square – Tests for significant differences with
attribute or categorical data
 Used together with Logistic Regression
 Used to verify that data fit into a particular distribution or
belong to a family of distributions
 Enables you to see if knowledge of one discrete factor is
useful in predicting a separate discrete outcome
 The presence of such a predictive relationship may be
utilized when developing a predictive model

Quantitative Data AnalysisQuantitative Data Analysis
Methods and ToolsMethods and Tools

There are a number of quantitative tools
considered to be applicable to statistical
process or quality control:
 Quantifying and Predicting Process Performance
Control Charts
Histograms
 Cause and Effect Relationships
Cause-and-effect (fishbone) diagrams
Pareto charts
Scatter diagrams
Interrelationship Diagraphs
Run charts
Check sheets
Bar charts
Force Field Diagram
Quantitative Data AnalysisQuantitative Data Analysis
Methods and ToolsMethods and Tools

Control charts – techniques for quantifying
process behavior
 Focuses attention on detecting and monitoring
process variation over time
 Distinguishes special from common causes of
variation, as a guide to local or management action
 Helps improve a process to perform consistently, and
predictably for higher quality, lower cost, and higher
effective capacity
Control ChartsControl Charts

Control Charts - 2Control Charts - 2
 Control Chart Characteristics
 Classical control charts have a centerline and control
limits on both sides of the centerline
 Both the centerline and the limits represent estimates
that are calculated from a set of observations
collected while the process is running
 The centerline and control limits cannot be assigned
arbitrarily as they are intended to show what the
process can actually do


Upper
Control
Limit
(UCL)
Lower
Control
Limit
(LCL)
Upper and
Lower
Control Limits
represent the
natural variation
In the process
Upper and
Lower
Control Limits
represent the
natural variation
In the process
METRIC:PROCESS CONTROL CHART TYPE:
Plotted points are either
individual measurements or the
means of small groups of
measurements
measurements
The chart is used for continuous
and time control of the process
and prevention of causes
The chart is analyzed using
standard Rules to define the
control status of the process
Data
relating to
the process
Data
relating to
the process
Center Line (CL)
(Mean of data used to
set up the chart)
Statistical Methods for Software Quality
Adrian Burr – Mal Owen, 1996
Common Cause Variation
Numerical data taken
in time sequence
in time sequence


Upper
Control
Limit
(UCL)
Lower
Control
Limit
(LCL)
Upper and
Lower
Control Limits
represent the
natural variation
In the process
Upper and
Lower
Control Limits
represent the
natural variation
In the process
METRIC:PROCESS CONTROL CHART TYPE:
A point above or below the
control lines suggests that the
measurement has a special
preventable or removable cause
A point above or below the
control lines suggests that the
measurement has a special
preventable or removable cause
measurements
measurements
Data
relating to
the process
Data
relating to
the process
Center Line (CL)
(Mean of data used to
set up the chart)
Statistical Methods for Software Quality
Adrian Burr – Mal Owen, 1996
Special Cause Variation
in time sequence
in time sequence

HistogramsHistograms
 Histograms – summarizes data from a process
that has been collected over a period of time,
and graphically present its frequency
distribution in bar form
 Show the frequencies of events that have occurred in
ways that make it easy to compare distributions and see
central tendencies
 Illustrates quickly the underlying distribution of the data
 Helps indicate if there has been a change in the process
 Provides useful information for predicting future
performance of the process
 Helps answer the question “Is the process capable of
meeting my customers requirements?”Florac, W.A. & Carleton, A.D. Measuring the Software Process

DetermineDetermine
SubprocessSubprocess
CapabilityCapability

Monitor Performance ofMonitor Performance of
Selected SubprocessesSelected Subprocesses
 Process capability is analyzed for those
subprocesses and those measured attributes
for which objectives have been set
 A capable process is one that is satisfying its
quality and process performance objectives
along with the customer requirements or
customer specifications and can be expected to
satisfy those objectives in the future
 Voice of the process
 Voice of the customer

Selected Subprocesses - 4Selected Subprocesses - 4
Customer
Requirements
Process
Within
Requirements
or Customer
Specifications
Process
Too Variable
Variation – what is the
variation or spread of
the data?
The Memory Jogger II A Pocket Guide of Tools
For Continuous Improvement & Effective Planning,
Michael Brassard & Diane Ritter, 1994

Stable Process
(Process Performance
Is Predictable)
Quality & Process
Performance
Meets Customer
Requirements
Selected Subprocesses - 4Selected Subprocesses - 4

Causal AnalysisCausal Analysis

Conduct Causal AnalysisConduct Causal Analysis
 Analyze defect data in the processes and
associated work products
 When a stable process does not meet its specified
product quality, service quality, or process
performance objectives
 During the task, if and when problems demand
additional meetings
 When a work product exhibits an unexpected
deviation from its requirements
 Analyze the selected defects and other
problems to determine their root causes

Conduct Causal Analysis - 2Conduct Causal Analysis - 2
 Examples of methods for determining causes
and other relationships that exist among critical
issues include:
 Cause and Effect (Fishbone Diagrams)
 Pareto analysis
 Scatter Diagrams
 Run charts
 Interrelationship Diagraphs
 Check Sheets
 Bar Charts
 Force Fields

Visual DisplayVisual Display
and otherand other
PresentationPresentation

Cause and EffectCause and Effect
Diagrams (Fishbone)Diagrams (Fishbone)
 Cause-and-effect (fishbone) diagrams
 Allows the project team to identify, explore, and
graphically display all of the possible causes related
to a problem to discover its root cause
 Helps the team to probe for, map, and prioritize a set
of factors that are thought to affect a particular
process, problem or outcome
 Helpful in eliciting and organizing information from
people who work within a process and know what
might be causing it to perform the way it does
 Focuses the project team on causes, not symptoms

Req’mts
Defects
Missing
Requirement
Incorrect
Requirement
Infeasible
Requirement
Vague
Requirement
Customer
Requirement Changed
Cause and Effect DiagramsCause and Effect Diagrams
(Fishbone)(Fishbone)

 Use the Fishbone or “Cause and Effect”
visualization technique to determine the most
significant causes for lack of adequate Quality
Assurance support in most large organizations

Pareto ChartsPareto Charts
 Pareto charts – special form of histogram or bar
chart
 Help focus investigations and solution finding by
ranking problems, causes, or actions in terms of their
amounts, frequencies of occurrence, or economic
consequences
 Based on the proven Pareto principle: 20% of the
sources cause 80% of any problem
 Helps prevent “shifting the problem” where the
“solution” removes some causes but worsens others

Pareto ChartsPareto Charts
 Percentage of Defects Detected During
System Testing by Phase Where Defect Was
Injected
50
25
20
5
0
10
20
30
40
50
60
Req'mts Design Code Test

Scatter DiagramsScatter Diagrams
 Scatter diagrams – display empirically observed
relationships between two process
characteristics
 A pattern in the plotted points may suggest that the
two factors are associated
 The scatter diagram does not predict cause and
effect relationships between two variables

Scatter DiagramsScatter Diagrams

 Run charts – specialized, time-sequenced
form of scatter diagram that can be used to
examine data quickly and informally for
trends or other patterns that occur over
time
 Monitors the performance of one or more
processes over time to detect trends, shifts, or
cycles
 Allows a team to compare a performance
measure before and after implementation of a
solution to measure its impact
 Tracks useful information for predicting trends
Run ChartsRun Charts

NumberofRequiredChangestoaModule
astheProjectApproachesSystemsTest
Syntax
Check
Desk
Check
Code
Review
Unit
Test
Integration
and Test
Systems
Test
Run Charts - 2Run Charts - 2

Interrelationship DiagraphsInterrelationship Diagraphs
 Interrelationship diagraphs – Allows a team to
systematically identify, analyze, and classify the
cause and effect relationships that exist among
critical issues
 Key drivers can become the heart of an effective
solution
 Encourages team members to think in multiple
directions
 Allows the key issues to emerge naturally rather than
be forced by a dominant member
 Allows a team to identify a root cause even when
credible data does not exist

What are the issues
relating
to traffic jams?
A- A- Auto
Accidents
In= 4  Out=1
B-B- Road
Construction
In= 0  Out= 2
D-D- Weather
Conditions
In=2  Out=3
F- F- Mechanical
Breakdown
In= 0  Out=2
C-C- Rush Hour
Traffic
In= 6  Out= 1
E-E- Cultural
Events
In= 2  Out= 2
InterrelationshipsInterrelationships
DiagraphDiagraph

Check SheetsCheck Sheets
 Check Sheets
 Allows a project team to systematically record and
compile data from historical sources or observations
as they happen
Patterns and trends can be clearly detected and
shown
 Builds, with each observation a clearer picture of the
facts as opposed to opinions of the team member
 Ensures that recordings are made consistently
 Makes patterns in the data become obvious quickly

Check Sheets - 2Check Sheets - 2
 Check Sheets - continued
 Must agree upon the definition of what is being
observed
 Data must be collected over a sufficient period of
time to be sure the data represents “typical” results
during a “typical” cycle for your business

Proof and Checking ErrorsProof and Checking Errors
Errors
Classification
Book Chapters
31 2 54 Total
Spelling
Punctuation
Missing Information
Redundancy
Technical Errors
Format Errors
Incomplete Concepts
Total
////
//
/
//
//
///
///
//
///
/
//
//
//
/
//
//
////
//
/
/
/
/
///
///
//
//
16
12
6
9
8
3
11 12 11 10 10 54

Bar ChartsBar Charts
 Bar Charts
 Similar to histograms but are not normally based on
measures of continuous variables or frequency
counts
 Bar charts are defined on discrete values
 Bar charts can display numerical value not just
counts or relative frequencies
Example: Bar charts can be used to display data
such as the total size, cost, or elapsed time
associated with individual entities

Bar Charts - 2Bar Charts - 2
 Bar Charts - continued
 Cell width is irrelevant and there are always gaps
between the cells
 The concepts of average and standard deviation
have no meaning for the independent variable in bar
charts that are defined on discrete scales
 Medians, modes, and ranges can be used

Bar ChartBar Chart
Injected
Found
Escaped
Reqts
analysis
Design Code Unit
test
Component
test
System
test
Customer
use
Defect Analysis
Software Activity
45
40
35
30
25
20
15
10
5
0
PercentofDefects

Force FieldsForce Fields
 Force Fields – Positives or Negatives of
Change
 Identifies the forces and factors in place that support
or work against the solution of an issue or problem
 Positives are reinforced – negatives are reduced or
eliminated
 Presents the “positives” and “negatives” of a
situation so that they are easily compared
 Forces people to think together about all the aspects
of making the desired change a permanent one
 Encourages honest reflection on the real underlying
roots of a problem and its solution
 Encourages people to agree about the relative
priority of factors on each side of the “balance sheet”

Force Fields - 2Force Fields - 2
FearofPublicSpeaking
Increases Self-Esteem ->
Helps career ->
Communicates ideas ->
Contributes to a plan/solution ->
Encourages others to speak ->
Helps others to change ->
Increases energy of group ->
Helps clarify speaker’s ideas by
getting feedback from others ->
Helps others to see new
perspective ->
<- Past Embarrassments
<- Afraid to make mistakes
<- Lack of knowledge on the topic
<- Afraid people will be indifferent
<- Afraid people will laugh
<- May forget what to say
<- Too revealing of personal
thoughts
<- Afraid of offending group
<- Fear that nervousness will show
<- Lack of confidence in personal
appearance
Driving Forces Restraining Forces

SummarySummary
 Evolving a Measurement Program for Systems /
Software Engineering Process Improvement
includes:
 Clearly defining the need for a measurement
program
 Establishing a measurement initiative with objectives
that are aligned with established information needs
and business objectives
 Ensuring basic measures are included for planning,
tracking, and taking corrective action as necessary
 Incorporating process effectiveness measures
 Establishing organizational standard processes

Summary - 2Summary - 2
 Establish and utilize measures such as peer review
measures, testing measures, and risk management
measures
 Evolve into project management based on a
quantitative understanding of the organization’s and
project’s defined processes

KI Measurement Program Starter Kit v3.1

More Related Content

What's hot (20)

Viewers also liked (12)

Similar to KI Measurement Program Starter Kit v3.1 (20)

KI Measurement Program Starter Kit v3.1

Editor's Notes