Can we do it better? Improving the maturity of software testing in test teams

1Dr. Vahid Garousi
Can we do it better? Improving the maturity of
software testing in test teams
Vahid Garousi
Associate Professor of Software Engineering
Information Technology Group (INF)
Wageningen University, Netherlands
www.vgarousi.com
www.wur.nl/inf
@vgarousi
A talk for:
Prague, Czech Republic
Oct. 5, 2018

2Dr. Vahid Garousi
My background
Education:
PhD in Software Engineering, Carleton University, Ottawa, Canada,
2006
(International) Work experience:
Associate Professor, Wageningen University, Netherlands, Since
2017
Associate Professor, Ankara, Turkey, 2014-2017
Associate Professor, University of Calgary, Canada, 2006-2014
Also actively providing Software Engineering Consulting
services since 2003

3Dr. Vahid Garousi
Can we do a better job in software testing?
 This usually boils to...
 Better test effectiveness: catching more defects by our test
activities and
 Better test efficiency: executing more and “better” tests
faster (in less time)
 Question: How do you improve
test effectiveness and test
efficiency in your company?

4Dr. Vahid Garousi
Outline
 Can we do a better job in software testing?
Brainstorming...
 Testing Maturity Model integrated (TMMi) model
 An overview of it
 Applying it for a client company and results
 General process for test maturity assessment (TMA) and
test process improvement (TPI)
 Overview of the other 58 testing maturity models “out
there!”

5Dr. Vahid Garousi
Can we do a better job in software testing?
 Many of us improve our team’s software testing
practices in ad-hoc manners, e.g., by reading
online sources, etc.
 But can we do it more systematically?
 The answer is Yes!
 Using the so-called test “maturity models”:
 Testing Maturity Model integrated (TMMi) model
 Test Process Improvement (TPI) model
 ...

6Dr. Vahid Garousi
Outline
Brainstorming...
 Applying it for a client company and results
there!”

7Dr. Vahid Garousi
Testing Maturity Model integrated (TMMi) model
 It is a “benchmark” which can consistently “grade” teams for their
testing maturity
(1) Initial
(2) Managed
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
(4) Measured
Test Measurement
Software Quality Evaluation
Advanced Peer Reviews
(3) Defined
Test Organization
Test Training Program
Test Lifecycle and Integration
Non-functional Testing
Peer Reviews
(5) Optimization
Defect Prevention
Test Process Optimization
Quality Control
Increase in
maturity
Increase in
maturity
Maturity levels
Process areas
(PAs)
More
maturity
Less
maturity
Developed by the
TMMi foundation

8Dr. Vahid Garousi
An overview of TMMi
 Structure:
(1) Initial
(2) Managed
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
(4) Measured
Test Measurement
Software Quality Evaluation
Advanced Peer Reviews
(3) Defined
Test Organization
Test Training Program
Test Lifecycle and Integration
Non-functional Testing
Peer Reviews
(5) Optimization
Defect Prevention
Test Process Optimization
Quality Control
Increase in
maturity
Increase in
maturity
Maturity levels
Process areas
(PAs)
Maturity level
Process Area (PA)
Specific Goal (SG)
Specific Practice (SP)
Has
several...
Has
several...
Has
several...
Examples next...

9Dr. Vahid Garousi
An overview of TMMi
 In total, for the 5 maturity levels, there are
16 Process Areas (PA), 50 Specific Goals
(SG), and 188 Specific Practices (SP).
Maturity level
Process Area (PA)
Specific Goal (SG)
Each SP is assessed:
1. Not applicable (N/A)
2. Not implemented (NI)
3. Partially implemented (PI)
4. Largely implemented (LI)
5. Fully implemented (FI)

10Dr. Vahid Garousi
An overview of TMMi
 For more details, see the full specification online...

11Dr. Vahid Garousi
An example application of TMMi
 Client: A large software company in Turkey, in 2016. Had a team of 45+ test
engineers, and about 500 software developers
 Results of the (unofficial) assessment:
 How we did it? Discussed next...
 Question: So which TMMi level is this company in?
60
47
11
0
13
6
6
0
1
5
4
16
6
10
0
10
20
30
40
50
60
70
80
90
Level 2 Level 3 Level 4 Level 5
#ofspecificpractices(SP's)
N/A
NI
PI
LI
FI
Not applicable (N/A)
Not implemented (NI)
Partially implemented (PI)
Largely implemented (LI)
Fully implemented (FI)

12Dr. Vahid Garousi
Application of TMMi: Howe was it done?
PA SG SP Explanation Supporting
artifacts
Score Opportunity for
TPI
PA 2.3-Test
monitoring
and control
SG 1-Monitor test
progress against
plan
SP 1.1 -Monitor test
planning parameters
Monitor the actual values of the
test planning parameters against
the test plan.
Sub practices:
- Monitor test progress against the
test schedule
- Monitor the test cost and
expended test effort
- Monitor the attributes of the test
work products and test tasks
- Monitor the knowledge and skills
of test staff
- Document the significant
deviations in the test planning
parameters.
The company has
test tracking
sheets;
Causal Analysis
and Resolution
Reports, task
control system
FI
SP 1.2-Monitor test
environment resources
provided and used
Monitor the actual usage of the
provided test environment
resources against the plan
The company has
Project test plan,
project progress
meetings
LI
Should improve
the monitoring
of the actual
usage of the
provided test
environment
resources
against the plan
SP 1.3-Monitor test
commitments
Monitor test commitments
achieved against those identified
in the test plan.
The company has
Project progress
meeting records
(internal), project
management
meetings
(external)
FI
,,, … …
Maturity level
Process Area (PA)
Specific Goal (SG)
Each SP is assessed:
1. Not applicable (N/A)
2. Not implemented (NI)
3. Partially implemented (PI)
4. Largely implemented (LI)
5. Fully implemented (FI)

13Dr. Vahid Garousi
An example application of TMMi
 Benefits to the client?
 Assessment using TMMi helped us to pinpoint areas for Test
Process Improvement (TPI), which itself led to several separate TPI
projects
 Some concrete examples of TPI for the client:
1. Separation of debugging from testing should be made more clear in test
policy documents
2. It is a good idea to document generic product risks in product documents
3. Test policy and test performance indicators and metrics could be updated
4. There is a need for more systematic risk-based testing
60
47
11
0
13
6
6
0
1
5
4
16
6
10
0
10
20
30
40
50
60
70
80
90
#ofspecificpractices(SP's) N/A
NI
PI
LI
FI

14Dr. Vahid Garousi
Outline
Brainstorming...
 Applying on for a client company and results
 General process for test maturity assessment (TMA)
and test process improvement (TPI)
there!”
 TMMi is not the only testing maturity model!

15Dr. Vahid Garousi
Process for test maturity assessment (TMA) and test process
improvement (TPI)

16Dr. Vahid Garousi
Outline
Brainstorming...
there!”
 TMMi is not the only testing maturity model!

17Dr. Vahid Garousi
So how many testing maturity models exist?
 We asked this question
when attempting to assess
the testing maturity of a
client
 Answer: We conducted a
systematic survey in 2015
and found 58 models
 Published the survey as a
paper
 Data from 181 sources
 51 sources from grey
literature (online
materials)
 130 sources from
academic literature
www.goo.gl/v2NLsV

18Dr. Vahid Garousi
Researchers and practitioners are involved
 Attention to this topic from both researchers and practitioners
has risen since the 1990s.

19Dr. Vahid Garousi
Looking at a few of those 58 testing maturity models

20Dr. Vahid Garousi
Looking at a few of those 58 testing maturity models

21Dr. Vahid Garousi
Evolution of testing maturity models

22Dr. Vahid Garousi
Summary and recommendations
 Many of us improve our team’s software testing
practices in ad-hoc manners, e.g., by reading online
sources, etc.
 But it makes sense to do it more systematically
 ...using the test maturity models (58 of them!)
 These models are like a “benchmark” which can
consistently “grade” teams for their testing maturity,
“across” the industry
60
47
11
0
13
6
6
0
1
5
4
16
6
10
0
10
20
30
40
50
60
70
80
90
#ofspecificpractices(SP's)
N/A
NI
PI
LI
FI

23Dr. Vahid Garousi
Outline
Brainstorming...
there!”
Questions / Answers

Can we do it better? Improving the maturity of software testing in test teams

More Related Content

Similar to Can we do it better? Improving the maturity of software testing in test teams (20)

More from Vəhid Gəruslu (20)

Recently uploaded (20)

Can we do it better? Improving the maturity of software testing in test teams