ISTQB Advance level syllabus 2019 Technical Test Analyst

ISTQB®
Certified Tester
Advanced Level Syllabus
Technical Test Analyst
Version 2019
International Software Testing Qualifications Board
Copyright Notice
This document may be copied in its entirety, or extracts made, if the source is acknowledged.

Certified Tester
Advanced Level Syllabus - Technical Test Analyst
International
Software Testing
Qualifications Board
Version 2019 V1.0 Page 2 of 58 October 18, 2019
© International Software Testing Qualifications Board
Copyright © International Software Testing Qualifications Board (hereinafter called ISTQB®).
Advanced Level Working Group: Graham Bath (vice-chair), Rex Black, Judy McKay, Kenji Onoshi, Mike
Smith (chair), Erik van Veenendaal

Certified Tester
International
Software Testing
Revision History
Version Date Remarks
2012 19 Oct 2012 GA release for 2012 version
2019 V1.0 18 Oct 2019 GA release for 2019 version

Certified Tester
International
Software Testing
Table of Contents
Revision History....................................................................................................................................... 3
Table of Contents .................................................................................................................................... 4
Acknowledgements ................................................................................................................................. 6
0. Introduction to this Syllabus............................................................................................................ 7
0.1 Purpose of this Syllabus......................................................................................................... 7
0.2 The Certified Tester Advanced Level in Software Testing..................................................... 7
0.3 Examinable Learning Objectives and Cognitive Levels of Knowledge.................................. 7
0.4 Expectations of Experience.................................................................................................... 7
0.5 The Advanced Level Technical Test Analyst Exam............................................................... 8
0.6 Entry Requirements for the Exam.......................................................................................... 8
0.7 Accreditation of Courses........................................................................................................ 8
0.8 Level of Syllabus Detail.......................................................................................................... 8
0.9 How this Syllabus is Organized.............................................................................................. 9
1. The Technical Test Analyst's Tasks in Risk-Based Testing - 30 mins. ........................................ 10
1.1 Introduction........................................................................................................................... 11
1.2 Risk-based Testing Tasks.................................................................................................... 11
1.2.1 Risk Identification............................................................................................................. 11
1.2.2 Risk Assessment ............................................................................................................. 11
1.2.3 Risk Mitigation.................................................................................................................. 12
2. White-box Test Techniques - 345 mins. ....................................................................................... 13
2.1 Introduction........................................................................................................................... 14
2.2 Statement Testing................................................................................................................ 14
2.3 Decision Testing................................................................................................................... 15
2.4 Modified Condition/Decision Coverage (MC/DC) Testing.................................................... 15
2.5 Multiple Condition Testing.................................................................................................... 16
2.6 Basis Path Testing ............................................................................................................... 17
2.7 API Testing........................................................................................................................... 18
2.8 Selecting a White-box Test Technique ................................................................................ 19
3. Analytical Techniques - 210 mins................................................................................................. 20
3.1 Introduction........................................................................................................................... 21
3.2 Static Analysis...................................................................................................................... 21
3.2.1 Control Flow Analysis ...................................................................................................... 21
3.2.2 Data Flow Analysis .......................................................................................................... 21
3.2.3 Using Static Analysis for Improving Maintainability ......................................................... 22
3.2.4 Call Graphs...................................................................................................................... 23
3.3 Dynamic Analysis................................................................................................................. 24
3.3.1 Overview.......................................................................................................................... 24
3.3.2 Detecting Memory Leaks................................................................................................. 25
3.3.3 Detecting Wild Pointers ................................................................................................... 25
3.3.4 Analysis of Performance Efficiency ................................................................................. 26
4. Quality Characteristics for Technical Testing - 345 mins. ........................................................... 27
4.1 Introduction........................................................................................................................... 28
4.2 General Planning Issues ...................................................................................................... 29
4.2.1 Stakeholder Requirements .............................................................................................. 29
4.2.2 Required Tool Acquisition and Training........................................................................... 29
4.2.3 Test Environment Requirements ..................................................................................... 30
4.2.4 Organizational Considerations......................................................................................... 30
4.2.5 Data Security Considerations .......................................................................................... 30
4.2.6 Risks and Typical Defects ............................................................................................... 30
4.3 Security Testing ................................................................................................................... 31
4.3.1 Reasons for Considering Security Testing ...................................................................... 31
4.3.2 Security Test Planning..................................................................................................... 31

Certified Tester
International
Software Testing
4.3.3 Security Test Specification .............................................................................................. 32
4.4 Reliability Testing ................................................................................................................. 33
4.4.1 Introduction ...................................................................................................................... 33
4.4.2 Measuring Software Maturity........................................................................................... 33
4.4.3 Fault Tolerance Testing ................................................................................................... 33
4.4.4 Recoverability Testing ..................................................................................................... 33
4.4.5 Availability Testing ........................................................................................................... 34
4.4.6 Reliability Test Planning .................................................................................................. 35
4.4.7 Reliability Test Specification............................................................................................ 35
4.5 Performance Efficiency Testing ........................................................................................... 35
4.5.1 Types of Performance Efficiency Testing ........................................................................ 35
4.5.2 Performance Efficiency Test Planning............................................................................. 36
4.5.3 Performance Efficiency Test Specification ...................................................................... 37
4.5.4 Quality Sub-characteristics of Performance Efficiency.................................................... 37
4.6 Maintainability Testing.......................................................................................................... 38
4.6.1 Static and Dynamic Maintainability Testing ..................................................................... 38
4.6.2 Maintainability Sub-characteristics .................................................................................. 39
4.7 Portability Testing................................................................................................................. 39
4.7.1 Introduction ...................................................................................................................... 39
4.7.2 Installability Testing ......................................................................................................... 40
4.7.3 Adaptability Testing ......................................................................................................... 40
4.7.4 Replaceability Testing...................................................................................................... 40
4.8 Compatibility Testing............................................................................................................ 41
4.8.1 Introduction ...................................................................................................................... 41
4.8.2 Co-existence Testing ....................................................................................................... 41
5. Reviews - 165 mins....................................................................................................................... 42
5.1 Technical Test Analyst Tasks in Reviews............................................................................ 43
5.2 Using Checklists in Reviews ................................................................................................ 43
5.2.1 Architectural Reviews ...................................................................................................... 44
5.2.2 Code Reviews.................................................................................................................. 44
6. Test Tools and Automation - 180 mins......................................................................................... 46
6.1 Defining the Test Automation Project .................................................................................. 47
6.1.1 Selecting the Automation Approach ................................................................................ 47
6.1.2 Modeling Business Processes for Automation ................................................................ 49
6.2 Specific Test Tools............................................................................................................... 50
6.2.1 Fault Seeding/Fault Injection Tools ................................................................................. 50
6.2.2 Performance Testing Tools.............................................................................................. 51
6.2.3 Tools for Web-Based Testing .......................................................................................... 51
6.2.4 Tools to Support Model-Based Testing ........................................................................... 52
6.2.5 Component Testing and Build Tools ............................................................................... 52
6.2.6 Tools to Support Mobile Application Testing ................................................................... 53
7. References.................................................................................................................................... 54
7.1 Standards............................................................................................................................. 54
7.2 ISTQB® Documents............................................................................................................. 54
7.3 Books ................................................................................................................................... 54
7.4 Other References................................................................................................................. 55
8. Appendix A: Quality Characteristics Overview ............................................................................. 56
9. Index ............................................................................................................................................. 57

Certified Tester
International
Software Testing
Acknowledgements
This document was produced by a core team from the International Software Testing Qualifications
Board Advanced Level Working Group: Graham Bath (vice-chair), Rex Black, Judy McKay, Kenji
Onoshi, Mike Smith (chair), Erik van Veenendaal
The following persons participated in the reviewing, commenting and balloting of this syllabus:
Dani Almog Andrew Archer Rex Black
Armin Born Sudeep Chatterjee Tibor Csöndes
Wim Decoutere Klaudia Dusser-Zieger Melinda Eckrich-Brájer
Peter Foldhazi David Frei Karol Frühauf
Jan Giesen Attila Gyuri Matthias Hamburg
Tamás Horváth N. Khimanand Jan te Kock
Attila Kovács Claire Lohr Rik Marselis
Marton Matyas Judy McKay Dénes Medzihradszky
Petr Neugebauer Ingvar Nordström Pálma Polyák
Meile Posthuma Stuart Reid Lloyd Roden
Adam Roman Jan Sabak Péter Sótér
Benjamin Timmermans Stephanie van Dijck Paul Weymouth
The core team thanks the review team and the National Boards for their suggestions and input.
This document was formally released by the General Assembly of the ISTQB® on 20 October 2019.

Certified Tester
International
Software Testing
0. Introduction to this Syllabus
0.1 Purpose of this Syllabus
This syllabus forms the basis for the International Software Testing Qualification at the Advanced Level
for the Technical Test Analyst. The ISTQB® provides this syllabus as follows:
1. To National Boards, to translate into their local language and to accredit training providers.
National Boards may adapt the syllabus to their particular language needs and modify the
references to adapt to their local publications.
2. To Exam Boards, to derive examination questions in their local language adapted to the learning
objectives for the syllabus.
3. To training providers, to produce courseware and determine appropriate teaching methods.
4. To certification candidates, to prepare for the exam (as part of a training course or
independently).
5. To the international software and systems engineering community, to advance the profession
of software and systems testing, and as a basis for books and articles.
The ISTQB® may allow other entities to use this syllabus for other purposes, provided they seek and
obtain prior written permission.
0.2 The Certified Tester Advanced Level in Software Testing
The Advanced Level qualification is comprised of three separate syllabi relating to the following roles:
• Test Manager
• Test Analyst
• Technical Test Analyst
The ISTQB® Advanced Level Overview 2019 is a separate document [ISTQB_AL_OVIEW] which
includes the following information:
• Business Outcomes
• Matrix showing traceability between business outcomes and learning objectives
• Summary
0.3 Examinable Learning Objectives and Cognitive Levels of Knowledge
The Learning Objectives support the Business Outcomes and are used to create the examination for
achieving the Advanced Technical Test Analyst Certification.
The knowledge levels of the specific learning objectives at K2, K3 and K4 levels are shown at the
beginning of each chapter and are classified as follows:
• K2: Understand
• K3: Apply
• K4: Analyze
0.4 Expectations of Experience
Some of the learning objectives for the Technical Test Analyst assume that basic experience is available
in the following areas:
• General programming concepts
• General concepts of system architectures

Certified Tester
International
Software Testing
0.5 The Advanced Level Technical Test Analyst Exam
The Advanced Level Technical Test Analyst exam will be based on this syllabus. Answers to exam
questions may require the use of materials based on more than one section of this syllabus. All sections
of the syllabus are examinable except for the introduction and the appendices. Standards, books and
other ISTQB® syllabi are included as references, but their content is not examinable beyond what is
summarized in this syllabus itself.
The format of the exam is multiple choice. There are 45 questions. To pass the exam, at least 65% of
the total points must be earned.
Exams may be taken as part of an accredited training course or taken independently (e.g., at an exam
center or in a public exam). Completion of an accredited training course is not a pre-requisite for the
exam.
0.6 Entry Requirements for the Exam
The Certified Tester Foundation Level certification shall be obtained before taking the Advanced Level
Technical Test Analyst certification exam.
0.7 Accreditation of Courses
An ISTQB® Member Board may accredit training providers whose course material follows this syllabus.
Training providers should obtain accreditation guidelines from the Member Board or body that performs
the accreditation. An accredited course is recognized as conforming to this syllabus, and is allowed to
have an ISTQB® exam as part of the course.
0.8 Level of Syllabus Detail
The level of detail in this syllabus allows internationally consistent courses and exams. In order to
achieve this goal, the syllabus consists of:
• General instructional objectives describing the intention of the Advanced Level Technical Test
Analyst
• A list of terms that students must be able to recall
• Learning objectives for each knowledge area, describing the cognitive learning outcome to be
achieved
• A description of the key concepts, including references to sources such as accepted literature
or standards
The syllabus content is not a description of the entire knowledge area; it reflects the level of detail to be
covered in Advanced Level training courses. It focuses on material that can apply to any software
projects, using any lifecycle. The syllabus does not contain any specific learning objectives relating to
any particular software development lifecycle or method, but it does discuss how these concepts apply
in Agile projects, other types of iterative and incremental lifecycles, and in sequential lifecycles.

Certified Tester
International
Software Testing
0.9 How this Syllabus is Organized
There are six chapters with examinable content. The top-level heading for each chapter specifies the
minimum time for the chapter; timing is not provided below chapter level. For accredited training courses,
the syllabus requires a minimum of 21 hours and 15 minutes of instruction, distributed across the six
chapters as follows:
• Chapter 1: The Technical Test Analyst's Tasks in Risk-Based Testing (30 minutes)
• Chapter 2: White-Box Testing (345 minutes)
• Chapter 3: Analytical Techniques (210 minutes)
• Chapter 4: Quality Characteristics for Technical Testing (345 minutes)
• Chapter 5: Reviews (165 minutes)
• Chapter 6: Test Tools and Automation (180 minutes)

Certified Tester
International
Software Testing
1. The Technical Test Analyst's Tasks in Risk-Based Testing
- 30 mins.
Keywords
product risk, risk assessment, risk identification, risk mitigation, risk-based testing
Learning Objectives for The Technical Test Analyst's Tasks in Risk-Based
Testing
1.2 Risk-based Testing Tasks
TTA-1.2.1 (K2) Summarize the generic risk factors that the Technical Test Analyst typically needs to
consider
TTA-1.2.2 (K2) Summarize the activities of the Technical Test Analyst within a risk-based approach
for testing activities

Certified Tester
International
Software Testing
1.1 Introduction
The Test Manager has overall responsibility for establishing and managing a risk-based testing strategy.
The Test Manager usually will request the involvement of the Technical Test Analyst to ensure the risk-
based approach is implemented correctly.
Technical Test Analysts work within the risk-based testing framework established by the Test Manager
for the project. They contribute their knowledge of the technical product risks that are inherent in the
project, such as risks related to security, system reliability and performance.
1.2 Risk-based Testing Tasks
Because of their particular technical expertise, Technical Test Analysts are actively involved in the
following risk-based testing tasks:
• Risk identification
• Risk assessment
• Risk mitigation
These tasks are performed iteratively throughout the project to deal with emerging product risks and
changing priorities, and to regularly evaluate and communicate risk status.
1.2.1 Risk Identification
By calling on the broadest possible sample of stakeholders, the risk identification process is most likely
to detect the largest possible number of significant risks. Because Technical Test Analysts possess
unique technical skills, they are particularly well-suited for conducting expert interviews, brainstorming
with co-workers and also analyzing the current and past experiences to determine where the likely areas
of product risk lie. In particular, Technical Test Analysts work closely with other stakeholders, such as
developers, architects, operations engineers, product owners, local support offices, and service desk
technicians, to determine areas of technical risk impacting the product and project. Involving other
stakeholders ensures that all views are considered and is typically facilitated by Test Managers.
Risks that might be identified by the Technical Test Analyst are typically based on the [ISO25010] quality
characteristics listed in Chapter 4, and include, for example:
• Performance efficiency (e.g., inability to achieve required response times under high load
conditions)
• Security (e.g., disclosure of sensitive data through security attacks)
• Reliability (e.g., application unable to meet availability specified in the Service Level Agreement)
1.2.2 Risk Assessment
While risk identification is about identifying as many pertinent risks as possible, risk assessment is the
study of those identified risks in order to categorize each risk and determine the likelihood and impact
associated with it. The likelihood of occurrence is usually interpreted as the probability that the potential
problem could exist in the system under test.
The Technical Test Analyst contributes to finding and understanding the potential technical product risk
for each risk item whereas the Test Analyst contributes to understanding the potential business impact
of the problem should it occur.
Project risks can impact the overall success of the project. Typically, the following generic project risks
need to be considered:
• Conflict between stakeholders regarding technical requirements

Certified Tester
International
Software Testing
• Communication problems resulting from the geographical distribution of the development
organization
• Tools and technology (including relevant skills)
• Time, resource and management pressure
• Lack of earlier quality assurance
• High change rates of technical requirements
Product risk factors may result in higher numbers of defects. Typically, the following generic product
risks need to be considered:
• Complexity of technology
• Complexity of code structure
• Amount of re-use compared to new code
• Large number of defects found relating to technical quality characteristics (defect history)
• Technical interface and integration issues
Given the available risk information, the Technical Test Analyst proposes an initial risk level according
to the guidelines established by the Test Manager. For example, the Test Manager may determine that
risks should be categorized with a value from 1 to 10, with 1 being highest risk. The initial value may be
modified by the Test Manager when all stakeholder views have been considered.
1.2.3 Risk Mitigation
During the project, Technical Test Analysts influence how testing responds to the identified risks. This
generally involves the following:
• Reducing risk by executing the most important tests (those addressing high risk areas) and by
putting into action appropriate mitigation and contingency measures as stated in the test plan
• Evaluating risks based on additional information gathered as the project unfolds, and using that
information to implement mitigation measures aimed at decreasing the likelihood or avoiding
the impact of those risks
The Technical Test Analyst will often cooperate with specialists in areas such as security and
performance to define risk mitigation measures and elements of the organizational test strategy.
Additional information can be obtained from ISTQB® Specialist syllabi, such as the Advanced Level
Security Testing syllabus [ISTQB_ALSEC_SYL] and the Foundation Level Performance Testing
syllabus [ISTQB_FLPT_SYL].

Certified Tester
International
Software Testing
2. White-box Test Techniques - 345 mins.
Keywords
API testing, atomic condition, control flow testing, cyclomatic complexity, decision testing, modified
condition/decision testing, multiple condition testing, path testing, short-circuiting, statement testing,
white-box test technique
Learning Objectives for White-Box Testing
Note: LOs 2.2.1, 2.3.1, 2.4.1, 2.5.1, and 2.6.1 refer to a “specification item”. This includes items such as
sections of code, requirements, user stories, use cases, and functional specifications.
2.2 Statement Testing
TTA-2.2.1 (K3) Write test cases for a given specification item by applying the Statement test technique
to achieve a defined level of coverage
2.3 Decision Testing
TTA-2.3.1 (K3) Write test cases for a given specification item by applying the Decision test technique
to achieve a defined level of coverage
2.4 Modified Condition/Decision Coverage (MC/DC) Testing
TTA-2.4.1 (K3) Write test cases by applying the Modified Condition/Decision Coverage (MC/DC) test
design technique to achieve a defined level of coverage
2.5 Multiple Condition Testing
TTA-2.5.1 (K3) Write test cases for a given specification item by applying the Multiple Condition test
technique to achieve a defined level of coverage
2.6 Basis Path Testing
TTA-2.6.1 (K3) Write test cases for a given specification item by applying McCabe’s Simplified
Baseline Method
2.7 API Testing
TTA-2.7.1 (K2) Understand the applicability of API testing and the kinds of defects it finds
2.8 Selecting a White-box Test Technique
TTA-2.8.1 (K4) Select an appropriate white-box test technique according to a given project situation

Certified Tester
International
Software Testing
2.1 Introduction
This chapter principally describes white-box test techniques. These techniques apply to code and other
structures, such as business process flow charts.
Each specific technique enables test cases to be derived systematically and focuses on a particular
aspect of the structure to be considered. The techniques provide coverage criteria which have to be
measured and associated with an objective defined by each project or organization. Achieving full
coverage does not mean that the entire set of tests is complete, but rather that the technique being used
no longer suggests any useful tests for the structure under consideration.
The following techniques are considered in this syllabus:
• Statement testing
• Decision testing
• Modified Condition/Decision Coverage (MC/DC) testing
• Multiple Condition testing
• Basis Path testing
• API testing
The Foundation Syllabus [ISTQB_FL_SYL] introduces Statement testing and Decision testing.
Statement testing exercises the executable statements in the code, whereas Decision testing exercises
the decisions in the code and tests the code that is executed based on the decision outcomes.
The MC/DC and Multiple Condition techniques listed above are based on decision predicates and
broadly find the same types of defects. No matter how complex a decision predicate may be, it will
evaluate to either TRUE or FALSE, which will determine the path taken through the code. A defect is
detected when the intended path is not taken because a decision predicate does not evaluate as
expected.
The first four techniques are successively more thorough (and Basis Path testing is more thorough than
Statement and Decision testing); more thorough techniques generally require more tests to be defined
in order to achieve their intended coverage and find more subtle defects.
Refer to [Bath14], [Beizer90], [Beizer95], [Copeland03], [McCabe96], [ISO29119] and [Koomen06].
2.2 Statement Testing
Statement testing exercises the executable statements in the code. Coverage is measured as the
number of statements executed by the tests divided by the total number of executable statements in the
test object, normally expressed as a percentage.
Applicability
This level of coverage should be considered as a minimum for all code being tested.
Limitations/Difficulties
Decisions are not considered. Even high percentages of statement coverage may not detect certain
defects in the code’s logic.

Certified Tester
International
Software Testing
2.3 Decision Testing
Decision testing exercises the decisions in the code and tests the code that is executed based on the
decision outcomes. To do this, the test cases follow the control flows that occur from a decision point
(e.g., for an IF statement, one for the true outcome and one for the false outcome; for a CASE statement,
test cases would be required for all the possible outcomes, including the default outcome).
Coverage is measured as the number of decision outcomes executed by the tests divided by the total
number of decision outcomes in the test object, normally expressed as a percentage.
Compared to the MC/DC and Multiple Condition techniques described below, decision testing considers
the entire decision as a whole and evaluates the TRUE and FALSE outcomes in separate test cases.
Applicability
This level of coverage should be considered when the code being tested is important or even critical
(see the table in Section 2.8).
Because it may require more test cases than testing only at the statement level, it may be problematic
when time is an issue. Decision testing does not consider the details of how a decision with multiple
conditions is made and may fail to detect defects caused by combinations of these conditions.
2.4 Modified Condition/Decision Coverage (MC/DC) Testing
Compared to Decision testing, which considers the entire decision as a whole and evaluates the TRUE
and FALSE outcomes in separate test cases, MC/DC testing considers how a decision is made when it
includes multiple conditions (otherwise it is simply decision testing).
Each decision predicate is made up of one or more simple atomic conditions, each of which evaluates
to a discrete Boolean value. These are logically combined to determine the final outcome of the decision.
This technique checks that each of the atomic conditions independently and correctly affects the
outcome of the overall decision.
This technique provides a stronger level of coverage than statement and decision coverage when there
are decisions containing multiple conditions. Assuming N unique, independent atomic conditions,
MC/DC can usually be achieved with N+1 unique test cases. MC/DC requires pairs of test cases that
show a single atomic condition can independently affect the result of a decision.
In the following example, a statement “If (A or B) and C, then ...” is considered.
A B C (A or B) and C
Test 1 TRUE FALSE TRUE TRUE
Test 2 FALSE TRUE TRUE TRUE
Test 3 FALSE FALSE TRUE FALSE
Test 4 TRUE FALSE FALSE FALSE
In Test 1, A is TRUE and the overall result is TRUE. If A is changed to FALSE (as in Test 3, holding the
other values unchanged) the result changes to FALSE, so showing that A can independently affect the
outcome of the decision.
In Test 2, B is TRUE and the overall result is TRUE. If B is changed to FALSE (as in Test 3, holding
other values unchanged) the result changes to FALSE, so showing that B can independently affect the

Certified Tester
International
Software Testing
In Test 1, C is TRUE and the overall result is TRUE. If C is changed to FALSE (as in Test 4, holding
other values unchanged) the result changes to FALSE, so showing that C can independently affect the
Note that unlike statement and decision testing techniques, there are no “defined levels of coverage” for
MC/DC; it is either achieved (i.e., 100%) or not.
Applicability
This technique is used in the aerospace industry and other industries for safety-critical systems. It is
used when testing software where a failure may cause a catastrophe.
Achieving MC/DC coverage may be complicated when there are multiple occurrences of a variable in a
decision with multiple conditions; when this occurs, the conditions may be “coupled”. Depending on the
decision, it may not be possible to vary the value of one condition such that it alone causes the decision
outcome to change. One approach to addressing this issue is to specify that only uncoupled atomic
conditions must be tested to the MC/DC level. The other approach is to analyze each decision in which
coupling occurs on a case-by-case basis.
Some programming languages and/or interpreters are designed such that they exhibit short-circuiting
behavior when evaluating a complex decision statement in the code. That is, the executing code may
not evaluate an entire expression if the final outcome of the evaluation can be determined after
evaluating only a portion of the expression. For example, if evaluating the decision “A and B”, there is
no reason to evaluate B if A has already been evaluated as FALSE. No value of B can change the final
result so the code may save execution time by not evaluating B. Short-circuiting may affect the ability
to attain MC/DC coverage since some required tests may not be achievable.
2.5 Multiple Condition Testing
In rare instances, it might be required to test all possible combinations of atomic conditions that a
decision may contain. This exhaustive level of testing is called multiple condition testing. The number of
required tests is dependent on the number of atomic conditions in the decision statement and can be
determined by calculating 2N where N is the number of uncoupled atomic conditions. Using the same
example as before, the following tests are required to achieve multiple condition coverage:
A B C (A or B) and C
Test 1 TRUE TRUE TRUE TRUE
Test 2 TRUE TRUE FALSE FALSE
Test 3 TRUE FALSE TRUE TRUE
Test 4 TRUE FALSE FALSE FALSE
Test 5 FALSE TRUE TRUE TRUE
Test 6 FALSE TRUE FALSE FALSE
Test 7 FALSE FALSE TRUE FALSE
Test 8 FALSE FALSE FALSE FALSE
Coverage is measured as the number of unique condition combinations executed by the tests divided
by the total number of condition combinations in the test object, normally expressed as a percentage.

Certified Tester
International
Software Testing
Applicability
This technique is used to test embedded software which is expected to run reliably without crashing for
long periods of time (e.g., telephone switches that are expected to last 30 years).
Because the number of test cases can be derived directly from a truth table containing all of the atomic
conditions, this level of coverage can easily be determined. However, the sheer number of test cases
required makes MC/DC coverage more feasible for most situations.
If the programming language uses short-circuiting, the number of actual test cases will often be reduced,
depending on the order and grouping of logical operations that are performed on the atomic conditions.
2.6 Basis Path Testing
Path testing in general consists of identifying paths through the code and then creating tests to cover
them. Conceptually, it would be useful to test every unique path through the system. In any non-trivial
system, however, the number of test cases could become excessively large due to the nature of looping
structures. In contrast, basis path testing can realistically be performed and follows the Simplified
Baseline Method developed by McCabe [McCabe96].
The technique is applied by the following steps:
1. Create a control flow graph for a given specification item (e.g., code or functional design
specification). Note that this may also be the first step in performing control flow analysis (see
section 3.2.1).
2. Select a baseline path through the code (not an exception path). This baseline path should be
the most important path to test – risk could be used to make this judgment.
3. Generate the second path by changing the outcome of the first decision on the path, while
keeping the maximum number of decision outcomes the same as the baseline path.
4. Generate the third path by again starting with the baseline path and changing the outcome of
the second decision on the path. When multiway decisions are encountered (e.g., a case
statement), each outcome of the decision should be exercised before moving on to the next
decision.
5. Generate further paths by changing each of the outcomes on the baseline path. When new
decisions are encountered, the most important outcome should be followed first.
6. Once all the decision outcomes on the baseline path have been covered, apply the same
approach to subsequent paths until all the decision outcomes in the specification item have
been exercised.
Applicability
The Simplified Baseline Method—as defined above—is often performed on mission critical software. It
is a good addition to the other methods covered in this chapter because it looks at paths through the
software rather than just at the way decisions are made.
When the syllabus was released, tool support for basis path testing was limited.
Coverage
The technique described above should ensure full coverage of all the linearly independent paths and
the number of paths should match the cyclomatic complexity for the code. Depending on the complexity
of the code, it may be useful to use a tool to check that full coverage of the basis set of paths has been
achieved. Coverage is measured as the number of linearly independent paths executed by the tests
divided by the total number of linearly independent paths in the test object, normally expressed as a

Certified Tester
International
Software Testing
percentage. Basis path testing provides more thorough testing than decision coverage, with a relatively
small increase in the number of tests [NIST96].
2.7 API Testing
An Application Programming Interface (API) is code which enables communication between different
processes, programs and/or systems. APIs are often utilized in a client/server relationship where one
process supplies some kind of functionality to other processes.
API Testing is a type of testing rather than a technique. In certain respects, API testing is quite similar
to testing a graphical user interface (GUI). The focus is on the evaluation of input values and returned
data.
Negative testing is often crucial when dealing with APIs. Programmers who use APIs to access services
external to their own code may try to use API interfaces in ways for which they were not intended. That
means that robust error handling is essential to avoid incorrect operation. Combinatorial testing of many
different interfaces may be required because APIs are often used in conjunction with other APIs, and
because a single interface may contain several parameters, where values of these may be combined in
many ways.
APIs frequently are loosely coupled, resulting in the very real possibility of lost transactions or timing
glitches. This necessitates thorough testing of the recovery and retry mechanisms. An organization
that provides an API interface must ensure that all services have a very high availability; this often
requires strict reliability testing by the API publisher as well as infrastructure support.
Applicability
API testing is becoming more important for testing systems of systems as the individual systems become
distributed or use remote processing as a way of off-loading some work to other processors. Examples
include:
• Operating systems calls
• Service-oriented architectures (SOA)
• Remote procedure calls (RPC)
• Web services
Software containerization [Burns18] results in the division of a software program into several containers
which communicate with each other using mechanisms such as those listed above. API testing should
also target these interfaces.
Testing an API directly usually requires a Technical Test Analyst to use specialized tools. Because
there is typically no direct graphical interface associated with an API, tools may be required to setup the
initial environment, marshal the data, invoke the API, and determine the result.
Coverage
API testing is a description of a type of testing; it does not denote any specific level of coverage. At a
minimum the API test should include making calls to the API with both realistic input values and
unexpected inputs for checking exception handling. More thorough API tests may ensure that callable
entities are exercised at least once or all possible calls are made at least once.
Types of Defects
The types of defects that can be found by testing APIs are quite disparate. Interface issues are common,
as are data handling issues, timing problems, loss of transactions and duplication of transactions.

Certified Tester
International
Software Testing
2.8 Selecting a White-box Test Technique
The context of the system under test will have an impact on its product risk and criticality levels ( see
below). These factors influence the required coverage metric (and hence the white-box test technique
to use) and the depth of coverage to be achieved. In general, the more critical the system and the higher
the product risk level, the more rigorous the coverage requirements and the higher the need for time
and resources to achieve the desired coverage.
Sometimes the required coverage metric may be derived from applicable standards that apply to the
software system. For example, if the software were to be used in an airborne environment, it may be
required to conform to standard DO-178C (in Europe, ED-12C.) This standard contains the following
five failure conditions:
A. Catastrophic: failure may cause lack of critical function needed to safely fly or land the
plane
B. Hazardous: failure may have a large negative impact on safety or performance efficiency
C. Major: failure is significant, but less serious than A or B
D. Minor: failure is noticeable, but with less impact than C
E. No effect: failure has no impact on safety
If the software system is categorized as level A, it must be tested to 100% MC/DC coverage. If it is level
B, it must be tested to 100% decision level coverage and MC/DC is optional. Level C requires 100%
statement coverage at a minimum.
Likewise, [IEC61508] is an international standard for the functional safety of programmable, electronic,
safety-related systems. This standard has been adapted in many different areas, including automotive,
rail, manufacturing, nuclear power plants, and machinery. Criticality is defined using a scale of Safety
Integrity Levels (SIL) where SIL1 is the least critical and SIL4 the most critical. The standard gives
recommendations for test coverage, as shown in the following table (note that the exact definitions for
each SIL and for the meaning of “recommended” and “highly recommended” is defined in the standard).
SIL 100% Statement
Coverage
100% Branch (Decision)
Coverage
100% MC/DC Coverage
1 Recommended Recommended Recommended
2 Highly recommended Recommended Recommended
3 Highly recommended Highly recommended Recommended
4 Highly recommended Highly recommended Highly recommended
In modern systems, it is rare that all processing will be done on a single system. API testing should be
instituted anytime some of the processing is going to be done remotely. The criticality of the system
should determine how much effort should be invested in API testing.

Certified Tester
International
Software Testing
3. Analytical Techniques - 210 mins.
Keywords
control flow analysis, cyclomatic complexity, data flow analysis, definition-use pair, dynamic analysis,
memory leak, pairwise integration testing, neighborhood integration testing, static analysis, wild pointer
Learning Objectives for Analytical Techniques
3.2 Static Analysis
TTA-3.2.1 (K3) Use control flow analysis to detect if code has any control flow anomalies
TTA-3.2.2 (K2) Explain how data flow analysis is used to detect if code has any data flow anomalies
TTA-3.2.3 (K3) Propose ways to improve the maintainability of code by applying static analysis
TTA-3.2.4 (K2) Explain the use of call graphs for establishing integration testing strategies
3.3 Dynamic Analysis
TTA-3.3.1 (K3) Apply dynamic analysis to achieve a specified goal

Certified Tester
International
Software Testing
3.1 Introduction
There are two types of analysis: static analysis and dynamic analysis.
Static analysis (Section 3.2) encompasses the analytical testing that can occur without executing the
software. Because the software is not executing, it is examined either by a tool or by a person to
determine if it will process correctly when it is executed. This static view of the software allows detailed
analysis without having to create the data and preconditions that would cause a scenario to be
exercised.
Note that the different forms of review which are relevant for the Technical Test Analyst are covered in
Chapter 5.
Dynamic analysis (Section 3.3) requires the actual execution of the code and is used to find faults which
are more easily detected when the code is executing (e.g., memory leaks). Dynamic analysis, as with
static analysis, may rely on tools or may rely on an individual monitoring the executing system watching
for such indicators as a rapid increase of memory use.
3.2 Static Analysis
The objective of static analysis is to detect actual or potential defects in code and system architecture
and to improve their maintainability. Static analysis is generally supported by tools.
3.2.1 Control Flow Analysis
Control flow analysis is the static technique where the steps followed through a program is analyzed,
either through the use of a control flow graph or a tool. There are a number of anomalies which can be
found in a system using this technique, including loops that are badly designed (e.g., having multiple
entry points), ambiguous targets of function calls in certain languages (e.g., Scheme), incorrect
sequencing of operations, etc.
Control flow analysis can be used to determine cyclomatic complexity. The cyclomatic complexity value
is a positive integer which represents the number of independent paths in a strongly connected graph.
Loops and iterations are ignored as soon as they have been traversed once. Each path, from entry to
exit, represents a unique path through the module. Each unique path should be tested.
The cyclomatic complexity value is generally used to understand the overall complexity of a module.
Thomas McCabe's theory [McCabe 76] was that the more complex the system, the harder it would be
to maintain and the more defects it would contain. Many studies over the years have noted this
correlation between complexity and the number of contained defects. The NIST (National Institute of
Standards and Technology) recommends a maximum complexity value of 10. Any module that is
measured with a higher complexity should be reviewed for possible division into multiple modules.
3.2.2 Data Flow Analysis
Data flow analysis covers a variety of techniques which gather information about the use of variables in
a system. The lifecycle of each variable is investigated, (i.e., where it is declared, defined, read,
evaluated and destroyed), since anomalies can occur during any of those operations or if the operations
are out of sequence.
One common technique is called definition-use notation where the lifecycle of each variable is split into
three different atomic actions:
• d: when the variable is declared, defined or initialized

Certified Tester
International
Software Testing
• u: when the variable is used or read in either a computation or a decision predicate
• k: when the variable is killed, destroyed or goes out of scope
A common alternative notation for d-u-k is: d (define) - r (reference or read) - u (undefine).
These three atomic actions are combined into pairs (“definition-use pairs”) to illustrate the data flow. For
example, a "du-path" represents a fragment of the code where a data variable is defined and then
subsequently used.
Possible data flow anomalies include performing the correct action on a variable at the wrong time or
carrying out an incorrect action on data in a variable. These anomalies include:
• Failing to assign a value to a variable before using it
• Taking an incorrect path due to an incorrect value in a control predicate
• Trying to use a variable after it is destroyed
• Referencing a variable when it is out of scope
• Declaring and destroying a variable without using it
• Redefining a variable before it has been used
• Failing to kill a dynamically allocated variable (causing a possible memory leak)
• Modifying a variable, which results in unexpected side effects (e.g., ripple effects when changing
a global variable without considering all uses of the variable)
The development language being used may guide the rules used in data flow analysis. Programming
languages may allow the programmer to perform certain actions with variables which are not illegal, but
may cause the system to behave differently than the programmer expected under certain circumstances.
For example, a variable might be defined twice without actually being used when a certain path is
followed. Data flow analysis will often label these uses "suspicious". While this may be a legal use of
the variable assignment capability, it can lead to future maintainability issues in the code.
Data flow testing "uses the control flow graph to explore the unreasonable things that can happen to
data" [Beizer90] and therefore finds different defects than control flow analysis. A Technical Test Analyst
should include this technique when planning testing since many of these defects cause intermittent
failures that are difficult to find while performing dynamic testing.
Data flow analysis is a static technique; it may miss some issues that occur as data is used in the runtime
system. For example, the static data variable may contain a pointer into a dynamically created array
that does not even exist until runtime. Multi-processor usage and pre-emptive multi-tasking may create
race conditions which will not be found by data flow or control flow analysis.
3.2.3 Using Static Analysis for Improving Maintainability
Static analysis can be applied in a number of ways to improve the maintainability of code, architecture
and web sites.
Poorly written, uncommented and unstructured code tends to be harder to maintain. It may require more
effort for developers to locate and analyze defects in the code, and the modification of the code to correct
a defect or add a new feature may result in further defects being introduced.
Static analysis is used with tool support to improve code maintainability by verifying compliance to
coding standards and guidelines. These standards and guidelines describe required coding practices
such as naming conventions, commenting, indentation and code modularization. Note that static
analysis tools generally raise warnings rather than detect errors. These warnings may be highlighted
even though the code may be syntactically correct.

Certified Tester
International
Software Testing
Static analysis tools can be applied to the code used for implementing web sites to check for possible
exposure to security vulnerabilities such as code injection, cookie security, cross-site scripting, resource
tampering and SQL code injection. Further details are provided in Section 4.3 and in the Advanced Level
Security Testing syllabus [ISTQB_ ALSEC_SYL].
Modular designs generally result in more maintainable code. Static analysis tools support the
development of modular code in the following ways:
• They search for repeated code. These sections of code may be candidates for refactoring into
modules (although the runtime overhead imposed by module calls may be an issue for real-time
systems).
• They generate metrics which are valuable indicators of code modularization. These include
measures of coupling and cohesion. A system that is to have good maintainability is more likely
to have a low measure of coupling (the degree to which modules rely on each other during
execution) and a high measure of cohesion (the degree to which a module is self-contained and
focused on a single task).
• They indicate, in object-oriented code, where derived objects may have too much or too little
visibility into parent classes.
• They highlight areas in code or architecture with a high level of structural complexity.
The maintenance of a web site can also be supported using static analysis tools. Here the objective is
to check if the tree-like structure of the site is well-balanced or if there is an imbalance that will lead to:
• More difficult testing tasks
• Increased maintenance workload
• Difficult navigation for the user
3.2.4 Call Graphs
Call graphs are a static representation of communication complexity. They are directed graphs in which
nodes represent program modules and edges represent communication among those modules.
Call graphs may be used in unit testing where different functions or methods call each other, in
integration and system testing when separate modules call each other, or in system integration testing
when separate systems call each other.
Call graphs can be used for the following purposes:
• Designing tests that call a specific module or system
• Establishing the number of locations within the software from where a module or system is
called
• Evaluating the structure of the code and of the system architecture
• Providing suggestions for the order of integration (e.g., pairwise and neighbourhood integration
as discussed below)
In the Foundation Level syllabus [ISTQB_FL_SYL], two different categories of integration testing were
discussed: incremental (top-down, bottom-up, etc.) and non-incremental (big bang). The incremental
methods were said to be preferred because they introduce code in increments thus making defect
isolation easier since the amount of code involved is limited.
In this Advanced syllabus, three more non-incremental methods using call graphs are introduced. These
may be preferable to incremental methods which likely will require additional builds to complete testing
and non-shippable code to be written to support the testing. These three methods are:
• Pairwise integration testing (not to be confused with the black-box test technique “pairwise
testing”), targets pairs of components that work together as seen in the call graph for integration
testing. While this method reduces the number of builds only by a small amount, it reduces the
amount of test harness code needed.

Certified Tester
International
Software Testing
• Neighborhood integration tests all of the nodes that connect to a given node as the basis for the
integration testing. All predecessor and successor nodes of a specific node in the call graph
are the basis for the test.
• McCabe's design predicate approach uses the theory of cyclomatic complexity as applied to a
call graph for modules. This requires the construction of a call graph that shows the different
ways that modules can call each other, including:
• Unconditional call: the call of one module to another always happens
• Conditional call: the call of one module to another sometimes happens
• Mutually exclusive conditional call: a module will call one (and only one) of a number of
different modules
• Iterative call: one module calls another at least once but may call it multiple times
• Iterative conditional call: one module can call another zero to many times
After creating the call graph, the integration complexity is calculated and tests are created to
cover the graph.
Refer to [Jorgensen07] for more information on using call graphs and neighborhood integration testing.
3.3 Dynamic Analysis
3.3.1 Overview
Dynamic analysis is used to detect failures where the symptoms are only visible when the code is
executed. For example, the possibility of memory leaks may be detectable by static analysis (finding
code that allocates but never frees memory), but a memory leak is readily apparent with dynamic
analysis.
Failures that are not immediately reproducible (intermittent) can have significant consequences on the
testing effort and on the ability to release or productively use software. Such failures may be caused by
memory or resource leaks, incorrect use of pointers and other corruptions (e.g., of the system stack)
[Kaner02]. Due to the nature of these failures, which may include the gradual worsening of system
performance or even system crashes, testing strategies must consider the risks associated with such
defects and, where appropriate, perform dynamic analysis to reduce them (typically by using tools).
Since these failures often are the most expensive failures to find and to correct, it is recommended to
start dynamic analysis early in the project.
Dynamic analysis may be applied to accomplish the following:
• Prevent failures from occurring by detecting memory leaks (see section 3.3.2) and wild pointers
(see section 3.3.3)
• Analyze system failures which cannot easily be reproduced
• Evaluate network behavior
• Improve system performance by providing information on runtime system behavior which can
be used to make informed changes
Dynamic analysis may be performed at any test level and requires technical and system skills to do the
following:
• Specify the testing objectives of dynamic analysis
• Determine the proper time to start and stop the analysis
• Analyze the results
During system testing, dynamic analysis tools can be used even if the Technical Test Analysts have
minimal technical skills; the tools utilized usually create comprehensive logs which can be analyzed by
those with the needed technical skills.

Certified Tester
International
Software Testing
3.3.2 Detecting Memory Leaks
A memory leak occurs when the areas of memory (RAM) available to a program are allocated by that
program but are not subsequently released when no longer needed. This memory area is left as
allocated and is not available for re-use. When this occurs frequently or in low memory situations, the
program may run out of usable memory. Historically, memory manipulation was the responsibility of the
programmer. Any dynamically allocated areas of memory had to be released by the allocating program
within the correct scope to avoid a memory leak. Many modern programming environments include
automatic or semi-automatic “garbage collection” where allocated memory is freed after use without the
programmer's direct intervention. Isolating memory leaks can be very difficult in cases where existing
allocated memory is freed by the automatic garbage collection.
Memory leaks cause problems which develop over time and may not be immediately obvious. This may
be the case if, for example, the software has been recently installed or the system restarted, which often
occurs during testing. For these reasons, the negative effects of memory leaks may first be noticed
when the program is in production.
The primary symptom of a memory leak is a steadily worsening of system response time which may
ultimately result in system failure. While such failures may be resolved by re-starting (re-booting) the
system, this may not always be practical or even possible.
Many dynamic analysis tools identify areas in the code where memory leaks occur so that they can be
corrected. Simple memory monitors can also be used to obtain a general impression of whether
available memory is declining over time, although a follow-up analysis would still be required to
determine the exact cause of the decline.
There are other types of leaks that also should be considered. Examples include file handles,
semaphores and connection pools for resources.
3.3.3 Detecting Wild Pointers
“Wild” pointers within a program are pointers that are no longer accurate and must not be used. For
example, a wild pointer may have “lost” the object or function to which it should be pointing or it does
not point to the area of memory intended (e.g., it points to an area that is beyond the allocated
boundaries of an array). When a program uses wild pointers, a variety of consequences may occur
including the following:
• The program may perform as expected. This may be the case where the wild pointer accesses
memory which is currently not used by the program and is notionally “free” and/or contains a
reasonable value.
• The program may crash. In this case the wild pointer may have caused a part of the memory
to be incorrectly used which is critical to the running of the program (e.g., the operating system).
• The program does not function correctly because objects required by the program cannot be
accessed. Under these conditions the program may continue to function, although an error
message may be issued.
• Data in the memory location may be corrupted by the pointer and incorrect values
subsequently used (this may also represent a security threat).
Note that any changes made to the program’s memory usage (e.g., a new build following a software
change) may trigger any of the four consequences listed above. This is particularly critical where initially
the program performs as expected despite the use of wild pointers, and then crashes unexpectedly
(perhaps even in production) following a software change. It is important to note that such failures are
often symptoms of an underlying defect (i.e., the wild pointer) (Refer to [Kaner02], “Lesson 74”). Tools
can help identify wild pointers as they are used by the program, irrespective of their impact on the
program’s execution. Some operating systems have built-in functions to check for memory access

Certified Tester
International
Software Testing
violations during runtime. For instance, the operating system may throw an exception when an
application tries to access a memory location that is outside of that application's allowed memory area.
3.3.4 Analysis of Performance Efficiency
Dynamic analysis is not just useful for detecting failures. With the dynamic analysis of program
performance, tools help identify performance efficiency bottlenecks and generate a wide range of
performance metrics which can be used by the developer to tune the system performance. For example,
information can be provided about the number of times a module is called during execution. Modules
which are frequently called would be likely candidates for performance enhancement.
By merging the information about the dynamic behavior of the software with information obtained from
call graphs during static analysis (see Section 3.2.4), the tester can also identify the modules which
might be candidates for detailed and extensive testing (e.g., modules which are frequently called and
have many interfaces).
Dynamic analysis of program performance is often done while conducting system tests, although it may
also be done when testing a single sub-system in earlier phases of testing using test harnesses. Further
details are provided in the Foundation Level Performance Testing syllabus [ISTQB_FLPT_SYL].

Certified Tester
International
Software Testing
4. Quality Characteristics for Technical Testing - 345 mins.
Keywords
accountability, adaptability, analyzability, authenticity, availability, capacity, co-existence, compatibility,
confidentiality, fault tolerance, installability, integrity, maintainability, maturity, modifiability, modularity,
non-repudiation, operational acceptance testing, operational profile, performance efficiency, portability,
quality characteristic, recoverability, reliability, reliability growth model, replaceability, resource
utilization, reusability, security, testability, time behavior
Learning Objectives for Quality Characteristics for Technical Testing
4.2 General Planning Issues
TTA-4.2.1 (K4) For a particular scenario, analyze the non-functional requirements and write the
respective sections of the test plan
TTA-4.2.2 (K3) Given a particular product risk, define the particular non-functional test type(s) which
are most appropriate
TTA-4.2.3 (K2) Understand and explain the stages in an application’s software development lifecycle
where non-functional tests should typically be applied
TTA-4.2.4 (K3) For a given scenario, define the types of defects you would expect to find by using the
different non-functional testing types
4.3 Security Testing
TTA-4.3.1 (K2) Explain the reasons for including security testing in a test approach
TTA-4.3.2 (K2) Explain the principal aspects to be considered in planning and specifying security tests
4.4 Reliability Testing
TTA-4.4.1 (K2) Explain the reasons for including reliability testing in a test approach
TTA-4.4.2 (K2) Explain the principal aspects to be considered in planning and specifying reliability
tests
4.5 Performance Efficiency Testing
TTA-4.5.1 (K2) Explain the reasons for including performance efficiency testing in a test approach
TTA-4.5.2 (K2) Explain the principal aspects to be considered in planning and specifying
performance efficiency tests
4.6 Maintainability Testing
TTA-4.6.1 (K2) Explain the reasons for including maintainability testing in a test approach
4.7 Portability Testing
TTA-4.7.1 (K2) Explain the reasons for including portability testing in a test approach
4.8 Compatibility Testing
TTA-4.8.1 (K2) Explain the reasons for including compatibility tests in a test approach

Certified Tester
International
Software Testing
4.1 Introduction
In general, the Technical Test Analyst focuses testing on "how" the product works, rather than the
functional aspects of "what" it does. These tests can take place at any test level. For example, during
component testing of real time and embedded systems, conducting performance efficiency
benchmarking and testing resource usage is important. During operational acceptance testing and
system test, testing for reliability aspects, such as recoverability, is appropriate. The tests at this level
are aimed at testing a specific system, i.e., combinations of hardware and software. The specific system
under test may include various servers, clients, databases, networks and other resources. Regardless
of the test level, testing should be performed according to the risk priorities and the available resources.
It should be noted that both dynamic testing and static testing (see Chapter 3) may be applied to test
the non-functional quality characteristics described in this chapter.
The description of product quality characteristics provided in ISO 25010 [ISO25010] is used as a guide
to describing the characteristics and their sub-characteristics. These are shown in the table below,
together with an indication of which characteristics/sub-characteristics are covered by the Test Analyst
and the Technical Test Analyst syllabi.
Characteristic Sub-Characteristics
Test Analyst
Technical
Test Analyst
Functional
suitability
Functional correctness, functional
appropriateness, functional completeness
X
Reliability Maturity, fault-tolerance, recoverability,
availability
X
Usability Appropriateness recognizability, learnability,
operability, user interface aesthetics, user error
protection, accessibility
X
Performance
efficiency
Time behavior, resource utilization, capacity X
Maintainability Analyzability, modifiability, testability,
modularity, reusability
X
Portability Adaptability, installability, replaceability X
Security Confidentiality, integrity, non-repudiation,
accountability, authenticity
X
Compatibility Co-existence X
Interoperability X
Note that a table is provided in Appendix A which compares the characteristics described in ISO 9126
(as used in version 2012 of this syllabus) with those in the newer ISO 25010.
For all of the quality characteristics and sub-characteristics discussed in this section, the typical risks
must be recognized so that an appropriate testing approach can be formed and documented. Quality
characteristic testing requires particular attention to lifecycle timing, required tools, required standards,
software and documentation availability and technical expertise. Without planning an approach to deal
with each characteristic and its unique testing needs, the tester may not have adequate planning,
preparation and test execution time built into the schedule [Bath14].
Some of this testing, e.g., performance efficiency testing, requires extensive planning, dedicated
equipment, specific tools, specialized testing skills and, in most cases, a significant amount of time.
Testing of the quality characteristics and sub-characteristics must be integrated into the overall testing
schedule with adequate resources allocated to the effort. Each of these test types has specific needs,
targets specific issues and may occur at different times during the software development lifecycle, as
discussed in the sections below.

Certified Tester
International
Software Testing
While the Test Manager will be concerned with compiling and reporting the summarized metric
information concerning quality characteristics and sub-characteristics, the Test Analyst or the Technical
Test Analyst (according to the table above) gathers the information for each metric.
Measurements of quality characteristics gathered in pre-production tests by the Technical Test Analyst
may form the basis for Service Level Agreements (SLAs) between the supplier and the stakeholders
(e.g., customers, operators) of the software system. In some cases, the tests may continue to be
executed after the software has entered production, often by a separate team or organization. This is
usually seen for performance efficiency and reliability testing which may show different results in the
production environment than in the testing environment.
4.2 General Planning Issues
Failure to plan for non-functional tests can put the success of an application at considerable risk. The
Technical Test Analyst may be requested by the Test Manager to identify the principal risks for the
relevant quality characteristics (see table in Section 4.1) and address any planning issues associated
with the proposed tests. This information may be used in creating the master test plan.
The following general factors are considered when performing these tasks:
• Stakeholder requirements
• Required tool acquisition and training
• Test environment requirements
• Organizational considerations
• Data security considerations
• Risks and typical defects
4.2.1 Stakeholder Requirements
Non-functional requirements are often poorly specified or even non-existent. At the planning stage,
Technical Test Analysts must be able to obtain expectation levels relating to technical quality
characteristics from affected stakeholders and evaluate the risks that these represent.
A common approach is to assume that if the customer is satisfied with the existing version of the system,
they will continue to be satisfied with new versions, as long as the achieved quality levels are maintained.
This enables the existing version of the system to be used as a benchmark. This can be a particularly
useful approach to adopt for some of the non-functional quality characteristics such as performance
efficiency, where stakeholders may find it difficult to specify their requirements.
It is advisable to obtain multiple viewpoints when capturing non-functional requirements. They must be
elicited from stakeholders such as customers, product owners, users, operations staff and maintenance
staff. If key stakeholders are excluded some requirements are likely to be missed. For more details
about capturing requirements, refer to the Advanced Test Manager syllabus [ISTQB_ALTM_SYL].
In Agile projects non-functional requirements may be stated as user stories or added to the functionality
specified in use cases as non-functional constraints.
4.2.2 Required Tool Acquisition and Training
Commercial tools or simulators are particularly relevant for performance efficiency and certain security
tests. Technical Test Analysts should estimate the costs and timescales involved for acquiring, learning
and implementing the tools. Where specialized tools are to be used, planning should account for the
learning curves for new tools and/or the cost of hiring external tool specialists.

Certified Tester
International
Software Testing
The development of a complex simulator may represent a development project in its own right and
should be planned as such. In particular the testing and documentation of the developed tool must be
accounted for in the schedule and resource plan. Sufficient budget and time should be planned for
upgrading and retesting the simulator as the simulated product changes. The planning for simulators to
be used in safety-critical applications must take into account the acceptance testing and possible
certification of the simulator by an independent body.
4.2.3 Test Environment Requirements
Many technical tests (e.g., security tests, performance efficiency tests) require a production-like test
environment in order to provide realistic measures. Depending on the size and complexity of the system
under test, this can have a significant impact on the planning and funding of the tests. Since the cost of
such environments may be high, the following alternatives may be considered:
• Using the production environment
• Using a scaled-down version of the system, taking care that the test results obtained are
sufficiently representative of the production system
• Using cloud-based resources as an alternative to acquiring the resources directly
• Using virtualized environments
The timing of such test executions must be planned carefully and it is quite likely that such tests can
only be executed at specific times (e.g., at low usage times).
4.2.4 Organizational Considerations
Technical tests may involve measuring the behavior of several components in a complete system (e.g.,
servers, databases, networks). If these components are distributed across a number of different sites
and organizations, the effort required to plan and co-ordinate the tests may be significant. For example,
certain software components may only be available for system testing at particular times of day or year,
or organizations may only offer support for testing for a limited number of days. Failing to confirm that
system components and staff (i.e., “borrowed” expertise) from other organizations are available “on call”
for testing purposes may result in severe disruption to the scheduled tests.
4.2.5 Data Security Considerations
Specific security measures implemented for a system should be taken into account at the test planning
stage to ensure that all testing activities are possible. For example, the use of data encryption may make
the creation of test data and the verification of results difficult.
Data protection policies and laws may preclude the generation of any required test data based on
production data (e.g., personal data, credit card data). Making test data anonymous is a non-trivial task
which must be planned for as part of the test implementation.
4.2.6 Risks and Typical Defects
Identifying and managing risks is a fundamental consideration for test planning (see Chapter 1). The
Technical Test Analyst identifies product risks by using knowledge of the typical types of defects to be
expected for a particular quality characteristic. This enables the types of testing required to address
those risks to be selected. These specific aspects are covered within the remaining sections of this
chapter which describe the individual quality characteristics.

Certified Tester
International
Software Testing
4.3 Security Testing
4.3.1 Reasons for Considering Security Testing
Security testing assesses a system's vulnerability to threats by attempting to compromise the system's
security policy. The following is a list of potential threats which should be explored during security testing:
• Unauthorized copying of applications or data.
• Unauthorized access control (e.g., ability to perform tasks for which the user does not have
rights). User rights, access and privileges are the focus of this testing. This information should
be available in the specifications for the system.
• Software which exhibits unintended side-effects when performing its intended function. For
example, a media player which correctly plays audio but does so by writing files out to
unencrypted temporary storage exhibits a side-effect which may be exploited by software
pirates.
• Code inserted into a web page which may be exercised by subsequent users (cross-site
scripting or XSS). This code may be malicious.
• Buffer overflow (buffer overrun) which may be caused by entering strings into a user interface
input field which are longer than the code can correctly handle. A buffer overflow vulnerability
represents an opportunity for running malicious code instructions.
• Denial of service, which prevents users from interacting with an application (e.g., by overloading
a web server with “nuisance” requests).
• The interception, mimicking and/or altering and subsequent relaying of communications (e.g.,
credit card transactions) by a third party such that a user remains unaware of that third party’s
presence (“Man in the Middle” attack)
• Breaking the encryption codes used to protect sensitive data.
• Logic bombs (sometimes called Easter Eggs), which may be maliciously inserted into code and
which activate only under certain conditions (e.g., on a specific date). When logic bombs
activate, they may perform malicious acts such as the deletion of files or formatting of disks.
4.3.2 Security Test Planning
In general the following aspects are of particular relevance when planning security tests:
• Because security issues can be introduced during the architecture, design and implementation
of the system, security testing may be scheduled for the unit, integration and system testing
levels. Due to the changing nature of security threats, security tests may also be scheduled on
a regular basis after the system has entered production. This is particularly true for dynamic
open architectures such as Internet of Things (IoT) where the production phase is characterized
by many updates to the software and hardware elements used.
• The test approaches proposed by the Technical Test Analyst may include reviews of the
architecture, design, and code, and the static analysis of code with security tools. These can be
effective in finding security issues that are easily missed during dynamic testing.
• The Technical Test Analyst may be called upon to design and perform certain security “attacks”
(see below) which require careful planning and coordination with stakeholders (including
security testing specialists). Other security tests may be performed in cooperation with
developers or with Test Analysts (e.g., testing user rights, access and privileges).
• An essential aspect of security test planning is obtaining approvals. For the Technical Test
Analyst, this means ensuring that explicit permission has been obtained from the Test Manager
to perform the planned security tests. Any additional, unplanned tests performed could appear
to be actual attacks and the person conducting those tests could be at risk for legal action. With
nothing in writing to show intent and authorization, the excuse "We were performing a security
test" may be difficult to explain convincingly.
• All security test planning should be coordinated with an organization’s Information Security
Officer if the organization has such a role.

Certified Tester
International
Software Testing
• It should be noted that improvements which may be made to the security of a system may affect
its performance efficiency or reliability. After making security improvements it is advisable to
consider the need for conducting performance efficiency or reliability tests (see Sections 4.4
and 4.5 below).
Individual standards may apply when conducting security test planning, such as [ISA/IEC 62443-3-2],
which applies to industrial automation and control systems.
The Advanced Level Security Testing syllabus [ISTQB_ALSEC_SYL] includes further details of key
security test plan elements.
4.3.3 Security Test Specification
Particular security tests may be grouped [Whittaker04] according to the origin of the security risk. These
include the following:
• User interface related - unauthorized access and malicious inputs
• File system related - access to sensitive data stored in files or repositories
• Operating system related - storage of sensitive information such as passwords in non-encrypted
form in memory which could be exposed when the system is crashed through malicious inputs
• External software related - interactions which may occur among external components that the
system utilizes. These may be at the network level (e.g., incorrect packets or messages passed) or
at the software component level (e.g., failure of a software component on which the software relies).
The ISO 25010 sub-characteristics of security [ISO25010] also provide a basis from which security tests
may be specified. These focus on the following aspects of security:
• Confidentiality – the degree to which a product or system ensures that data is accessible only to
those authorized to have access
• Integrity – the degree to which a system, product or component prevents unauthorized access to,
or modification of, computer programs or data
• Non-repudiation – the degree to which actions or events can be proven to have taken place so they
cannot be denied later
• Accountability – the degree to which the actions of an entity can be traced uniquely to the entity
• Authenticity – the degree to which the identity of a subject or resource can be proven to be the one
claimed
The following approach [Whittaker04] may be used to develop security tests:
• Gather information which may be useful in specifying tests, such as names of employees,
physical addresses, details regarding the internal networks, IP numbers, identity of software or
hardware used, and operating system version.
• Perform a vulnerability scan using widely available tools. Such tools are not used directly to
compromise the system(s), but to identify vulnerabilities that are, or that may result in, a breach
of security policy. Specific vulnerabilities can also be identified using information and checklists
such as those provided by the National Institute of Standards and Technology (NIST) [Web-1]
and the Open Web Application Security Project™ (OWASP) [Web-4].
• Develop “attack plans” (i.e., a plan of testing actions intended to compromise a particular
system’s security policy) using the gathered information. Several inputs via various interfaces
(e.g., user interface, file system) need to be specified in the attack plans to detect the most
severe security defects. The various “attacks” described in [Whittaker04] are a valuable source
of techniques developed specifically for security testing.
Note that attack plans may be developed for penetration tests (see [ISTQB_ALSEC_SYL]).
Security issues can also be exposed by reviews (see Chapter 5) and/or the use of static analysis tools
(see Section 3.2). Static analysis tools contain an extensive set of rules which are specific to security

Certified Tester
International
Software Testing
threats and against which the code is checked. For example, buffer overflow issues, caused by failure
to check buffer size before data assignment, can be found by the tool.
Section 3.2 (static analysis) and the Advanced Level Security Testing syllabus [ISTQB_ALSEC_SYL]
includes further details of security testing.
4.4 Reliability Testing
4.4.1 Introduction
The ISO 25010 classification of product quality characteristics defines the following sub-characteristics
of reliability:
• Maturity - the degree to which a component or system meets needs for reliability under normal
operation
• Fault tolerance - the capability of the software product to maintain a specified level of
performance in cases of software defects or of infringement of its specified interface
• Recoverability - the capability of the software product to re-establish a specified level of
performance and recover the data directly affected in case of failure
• Availability - the degree to which a component or system is operational and accessible when
required for use
4.4.2 Measuring Software Maturity
An objective of reliability testing is to monitor a statistical measure of software maturity over time and
compare this to a desired reliability goal which may be expressed as a Service Level Agreement (SLA).
The measures may take the form of a Mean Time Between Failures (MTBF), Mean Time To Repair
(MTTR) or any other form of failure intensity measurement (e.g., number of failures of a particular
severity occurring per week). These may be used as exit criteria (e.g., for production release).
Note that maturity in the reliability context should not be confused with maturity of the overall software
testing process, as discussed in the ISTQB® Advanced Level Test Manager syllabus
[ISTQB_ALTM_SYL].
4.4.3 Fault Tolerance Testing
In addition to the functional testing that evaluates the software’s tolerance to faults in terms of handling
unexpected input values (so-called negative tests), additional testing is needed to evaluate a system’s
tolerance to faults which occur externally to the application under test. Such faults are typically reported
by the operating system (e.g., disk full, process or service not available, file not found, memory not
available). Tests of fault tolerance at the system level may be supported by specific tools.
Note that the terms “robustness” and “error tolerance” are also commonly used when discussing fault
tolerance (see [ISTQB_GLOSSARY] for details).
4.4.4 Recoverability Testing
Further forms of reliability testing evaluate the software system’s ability to recover from hardware or
software failures in a predetermined manner which subsequently allows normal operations to be
resumed. Recoverability tests include Failover and Backup and Restore tests.
Failover tests are performed where the consequences of a software failure are so negative that specific
hardware and/or software measures have been implemented to ensure system operation even in the

Certified Tester
International
Software Testing
event of a failure. Failover tests may be applicable, for example, where the risk of financial losses is
extreme or where critical safety issues exist. Where failures may result from catastrophic events, this
form of recoverability testing may also be called “disaster recovery” testing.
Typical preventive measures for hardware failures might include load balancing across several
processors and clustering servers, processors or disks so that one can immediately take over from
another if it should fail (redundant systems). A typical software measure might be the implementation of
more than one independent instance of a software system (for example, an aircraft’s flight control
system) in so-called redundant dissimilar systems. Redundant systems are typically a combination of
software and hardware measures and may be called duplex, triplex or quadruplex systems, depending
on the number of independent instances (two, three or four respectively). The dissimilar aspect for the
software is achieved when the same software requirements are provided to two (or more) independent
and not connected development teams, with the objective of having the same services provided with
different software. This protects the redundant dissimilar systems in that a similar defective input is less
likely to have the same result. These measures taken to improve the recoverability of a system may
directly influence its reliability as well and should also be considered when performing reliability testing.
Failover testing is designed to explicitly test systems by simulating failure modes or actually causing
failures in a controlled environment. Following a failure, the failover mechanism is tested to ensure that
data is not lost or corrupted and that any agreed service levels are maintained (e.g., function availability
or response times).
Backup and Restore tests focus on the procedural measures set up to minimize the effects of a failure.
Such tests evaluate the procedures (usually documented in a manual) for taking different forms of
backup and for restoring that data if data loss or corruption should occur. Test cases are designed to
ensure that critical paths through each procedure are covered. Technical reviews may be performed to
“dry-run” these scenarios and validate the manuals against the actual procedures. Operational
acceptance testing exercises the scenarios in a production or production-like environment to validate
their actual use.
Measures for Backup and Restore tests may include the following:
• Time taken to perform different types of backup (e.g., full, incremental)
• Time taken to restore data
• Levels of guaranteed data backup (e.g., recovery of all data no more than 24 hours old, recovery
of specific transaction data no more than one hour old)
4.4.5 Availability Testing
Any system that has interfaces with other systems and/or processes (e.g., for receiving inputs) relies on
the availability of those interfaces to ensure overall operability.
Availability testing serves the following principal purposes:
• To establish whether required system components and processes are available (on demand or
continuously) and respond as expected to requests
• To provide measurements from which an overall level of availability can be obtained (often given
as a percentage of time in a SLA).
• To establish whether an overall system is ready for operation (e.g., as one of the criteria for
operational acceptance testing).
Availability testing is performed both before and after entering operational service, and is particularly
relevant for the following situations:
• Where systems are made up of other systems (i.e., systems of systems). Tests focus on the
availability of all individual component systems.

Certified Tester
International
Software Testing
• Where a system or service is sourced externally (e.g., from a third party supplier). Tests focus
on measuring availability levels to ensure that agreed service levels are upheld.
Availability may be measured using dedicated monitoring tools or by executing specific tests. Such tests
are typically automated and may run in parallel to normal operations provided they do not impact normal
operations (e.g., by reducing performance efficiency).
4.4.6 Reliability Test Planning
In general, the following aspects are of particular relevance when planning reliability tests:
• Reliability can continue to be monitored after the software has entered production. The
organization and staff responsible for operation of the software must be consulted when
gathering reliability requirements for test planning purposes.
• The Technical Test Analyst may select a reliability growth model which shows the expected
levels of reliability over time. A reliability growth model can provide useful information to the Test
Manager by enabling comparison of the expected and achieved reliability levels.
• Reliability tests should be performed in a production-like environment. The environment used
should remain as stable as possible to enable reliability trends to be monitored over time.
• Because reliability tests often require use of the entire system, reliability testing is most
commonly done as part of system testing. However, individual components can be subjected to
reliability testing as well as integrated sets of components. Detailed architecture, design and
code reviews can also be used to remove some of the risk of reliability issues occurring in the
implemented system.
• In order to produce test results that are statistically significant, reliability tests usually require
long execution times. This may make it difficult to schedule within other planned tests.
4.4.7 Reliability Test Specification
Reliability testing may take the form of a repeated set of predetermined tests. These may be tests
selected at random from a pool or test cases generated by a statistical model using random or pseudo-
random methods. Tests may also be based on patterns of use which are sometimes referred to as
“Operational Profiles” (see Section 4.5.3).
Where reliability tests are scheduled to run automatically in parallel to normal operations (e.g., to test
availability), they are generally specified to be as simple as possible to avoid possible negative impact
on the system performance efficiency.
Certain reliability tests may specify that memory-intensive actions be executed repeatedly so that
possible memory leaks can be detected.
4.5 Performance Efficiency Testing
4.5.1 Types of Performance Efficiency Testing
4.5.1.1 Load Testing
Load testing focuses on the ability of a system to handle increasing levels of anticipated realistic loads
resulting from the transaction requests generated by numbers of concurrent users or processes.
Average response times for users under different scenarios of typical use (operational profiles) can be
measured and analyzed. See also [Splaine01].

Certified Tester
International
Software Testing
4.5.1.2 Stress Testing
Stress testing focuses on the ability of a system or component to handle peak loads at or beyond the
limits of its anticipated or specified workloads, or with reduced availability of resources such as
available bandwidth. Performance levels should degrade slowly and predictably without failure as
stress levels are increased. In particular, the functional integrity of the system should be tested while
the system is under stress in order to find possible defects in functional processing or data
inconsistencies.
One possible objective of stress testing is to discover the limits at which a system actually fails so that
the “weakest link in the chain” can be determined. Stress testing allows additional capacity to be
added to the system in a timely manner (e.g., memory, CPU capability, database storage).
4.5.1.3 Scalability Testing
Scalability testing focuses on the ability of a system to meet future efficiency requirements, which may
be beyond those currently required. The objective of the tests is to determine the system’s ability to grow
(e.g., with more users, larger amounts of data stored) without reaching a point where the currently
specified performance requirements cannot be met or the system fails. Once the limits of scalability are
known, threshold values can be set and monitored in production to provide a warning of impending
problems. In addition, the production environment may be adjusted with appropriate amounts of
hardware to meet anticipated needs.
4.5.2 Performance Efficiency Test Planning
In addition to the general planning issues described in Section 4.2, the following factors can influence
the planning of performance efficiency tests:
• Depending on the test environment used and the software being tested, (see Section 4.2.3)
performance efficiency tests may require the entire system to be implemented before effective
testing can be done. In this case, performance efficiency testing is usually scheduled to occur
during system test. Other performance efficiency tests which can be performed effectively at
the component level may be scheduled during unit testing.
• In general, it is desirable to conduct initial performance efficiency tests as early as possible,
even if a production-like environment is not yet available. These early tests may find
performance efficiency problems (e.g., bottlenecks) and reduce project risk by avoiding time-
consuming corrections in the later stages of software development or production.
• Code reviews, in particular those which focus on database interaction, component interaction
and error handling, can identify performance efficiency issues (particularly regarding “wait and
retry” logic and inefficient queries) and should be scheduled early in the software development
lifecycle.
• The hardware, software and network bandwidth needed to run the performance efficiency tests
should be planned and budgeted. Needs depend primarily on the load to be generated, which
may be based on the number of virtual users to be simulated and the amount of network traffic
they are likely to generate. Failure to account for this may result in unrepresentative
performance measurements being taken. For example, verifying the scalability requirements of
a much-visited Internet site may require the simulation of hundreds of thousands of virtual users.
• Generating the required load for performance efficiency tests may have a significant influence
on hardware and tool acquisition costs. This must be considered in the planning of performance
efficiency tests to ensure that adequate funding is available.
• The costs of generating the load for performance efficiency tests may be minimized by renting
the required test infrastructure. This may involve, for example, renting “top-up” licenses for
performance tools or by using the services of a third party provider for meeting hardware needs
(e.g., cloud services). If this approach is taken, the available time for conducting the
performance efficiency tests may be limited and must therefore be carefully planned.

Certified Tester
International
Software Testing
• Care should be taken at the planning stage to ensure that the performance tool to be used
provides the required compatibility with the communications protocols used by the system under
test.
• Defects relating to performance efficiency often have significant impact on the system under
test. When performance efficiency requirements are imperative, it is often useful to conduct
performance efficiency tests on the critical components (via drivers and stubs) so testing can
start early in the lifecycle instead of waiting for system tests.
The Foundation Level Performance Testing syllabus [ISTQB_FLPT_SYL] includes further details of
performance test planning.
4.5.3 Performance Efficiency Test Specification
The specification of tests for different performance efficiency test types such as load and stress are
based on the definition of operational profiles. These represent distinct forms of user behavior when
interacting with an application. There may be multiple operational profiles for a given application.
The numbers of users per operational profile may be obtained by using monitoring tools (where the
actual or comparable application is already available) or by predicting usage. Such predictions may be
based on algorithms or provided by the business organization. These are especially important for
specifying the operational profile(s) to be used for scalability testing.
Operational profiles are the basis for the number and types of test cases to be used during performance
efficiency testing. These tests are often controlled by test tools that create "virtual" or simulated users
in quantities that will represent the profile under test (see Section 6.2.2).
The Foundation Level Performance Testing syllabus [ISTQB_FLPT_SYL] includes further details of
performance efficiency test design.
4.5.4 Quality Sub-characteristics of Performance Efficiency
The ISO 25010 classification of product quality characteristics includes the following sub-characteristics
of performance efficiency:
• Time behavior - the ability of a component or system to respond to user or system inputs within
a specified time and under specified conditions
• Resource utilization - the capability of the software product to use appropriate amounts and
types of resources
• Capacity - the maximum limit to which a particular parameter can be handled
4.5.4.1 Time Behavior
Time behavior focuses on the ability of a component or system to respond to user or system inputs
within a specified time and under specified conditions. Time behavior measurements vary according to
the objectives of the test. For individual software components, time behavior may be measured
according to CPU cycles, while for client-based systems time behavior may be measured according to
the time taken to respond to a particular user request. For systems whose architectures consist of
several components (e.g., clients, servers, databases) time behavior measurements are taken for
transactions between individual components so that “bottlenecks” can be identified.
4.5.4.2 Resource Utilization
Tests relating to resource utilization evaluate the usage of system resources (e.g., usage of memory,
disk capacity, network bandwidth, connections) against a predefined benchmark. These are compared
under both normal loads and stress situations, such as high levels of transaction and data volumes, to
determine if unnatural growth in usage is occurring.

Certified Tester
International
Software Testing
For example, for real-time embedded systems, memory usage (sometimes referred to as a “memory
footprint”) plays a significant role in performance efficiency testing. If the memory footprint exceeds the
allowed measure, the system may have insufficient memory needed to perform its tasks within the
specified time periods. This may slow down the system or even lead to a system crash.
Dynamic analysis may also be applied to the task of investigating resource utilization (see Section 3.3.4)
and detecting performance efficiency bottlenecks.
4.5.4.3 Capacity
The capacity of a system (including software and hardware) represents the maximum limit to which a
particular parameter can be handled. Capacity requirements are typically specified by technical and
operational stakeholders and may relate to parameters such as the maximum number of users that can
use an application at a given point of time, the maximum volume of data that can be transmitted per
second (i.e., bandwidth) and the maximum number of transactions that can be handled per second.
The approach to testing capacity limits is generally similar to the approach described in Sections 4.5.2
and 4.5.3 for testing performance efficiency. Operational profiles for capacity tests focus on generating
a load which exercises the particular limit. This may involve, for example, generating a load which
submits the system to the maximum amount of data transfer. Stress testing and scalability testing
approaches may also be applied to capacity to test system behavior beyond specified capacity limits
(see Sections 4.5.1.2 and 4.5.1.3 respectively).
4.6 Maintainability Testing
Software often spends substantially more of its lifetime being maintained than being developed.
Maintenance testing is performed to test the impact of changes to an operational system or its
environment. To ensure that the task of conducting maintenance is as efficient as possible,
maintainability testing is performed to measure the ease with which code can be analyzed, changed
and tested.
Typical maintainability objectives of affected stakeholders (e.g., the software owner or operator)
include:
• Minimizing the cost of owning or operating the software
• Minimizing downtime required for software maintenance
Maintainability tests should be included in a test approach where one or more of the following factors
apply:
• Software changes are likely after the software enters production (e.g., to correct defects or
introduce planned updates)
• The benefits of achieving maintainability objectives over the software development lifecycle
are considered by the affected stakeholders to outweigh the costs of performing the
maintainability tests and making any required changes
• The risks of poor software maintainability (e.g., long response times to defects reported by
users and/or customers) justify conducting maintainability tests
4.6.1 Static and Dynamic Maintainability Testing
Appropriate techniques for maintainability testing include static analysis and reviews as discussed in
Sections 3.2 and 5.2. Maintainability testing should be started as soon as the design documentation is
available and should continue throughout the code implementation effort. Since maintainability is built
into the code and the documentation for each individual code component, maintainability can be

Certified Tester
International
Software Testing
evaluated early in the software development lifecycle without having to wait for a completed and
running system.
Dynamic maintainability testing focuses on the documented procedures developed for maintaining a
particular application (e.g., for performing software upgrades). Selections of maintenance scenarios are
used as test cases to ensure the required service levels are attainable with the documented procedures.
This form of testing is particularly relevant where the underlying infrastructure is complex, and support
procedures may involve multiple departments/organizations. This form of testing may take place as part
of operational acceptance testing.
4.6.2 Maintainability Sub-characteristics
The maintainability of a system can be measured in terms of the effort required to diagnose problems
identified within a system (analyzability) and test the changed system (testability). Factors which
influence both analyzability and testability include the application of good programming practices (e.g.,
commenting, naming of variables, indentation), and the availability of technical documentation (e.g.,
system design specifications, interface specifications).
Other relevant quality sub-characteristics for maintainability [ISO25010] are:
• Modifiability - the degree to which a component or system can be effectively and efficiently
modified without introducing defects or degrading existing product quality
• Modularity – the degree to which a system, product or component is composed of discrete
components such that a change to one component has minimal impact on other components
• Reusability – the degree to which an asset can be used in more than one system, or in
building other assets
4.7 Portability Testing
4.7.1 Introduction
Portability tests in general relate to the degree to which a software component or system can be
transferred into its intended environment, either initially or from an existing environment.
[ISO25010] includes the following sub-characteristics of portability:
• Installability - the capability of the software product to be installed in a specified environment
• Adaptability - the degree to which a component or system can be adapted for different or
evolving hardware and software environments
• Replaceability - the capability of another software product to be used in place of the specified
software product for the same purpose in the same environment
Portability testing can start with the individual components (e.g., replaceability of a particular component
such as changing from one database management system to another) and then expand in scope as
more code becomes available. Installability may not be testable until all the components of the product
are functionally working.
Portability must be designed and built into the product and so must be considered early in the design
and architecture phases. Architecture and design reviews can be particularly productive for identifying
potential portability requirements and issues (e.g., dependency on a particular operating system).

Certified Tester
International
Software Testing
4.7.2 Installability Testing
Installability testing is performed on the software and written procedures used to install the software on
its target environment. This may include, for example, the software developed to install an operating
system onto a processor, or an installation “wizard” used to install a product onto a client PC.
Typical installability testing objectives include the following:
• Validating that the software can be successfully installed by following the instructions in an
installation manual (including the execution of any installation scripts), or by using an installation
wizard. This includes exercising installation options for different hardware/software
configurations and for various degrees of installation (e.g., initial or update).
• Testing whether failures which occur during installation (e.g., failure to load particular DLLs) are
dealt with by the installation software correctly without leaving the system in an undefined state
(e.g., partially installed software or incorrect system configurations)
• Testing whether a partial installation/de-installation can be completed
• Testing whether an installation wizard can successfully identify invalid hardware platforms or
operating system configurations
• Measuring whether the installation process can be completed within a specified number of
minutes or within a specified number of steps
• Validating that the software can be successfully downgraded or uninstalled
Functionality testing is normally performed after the installation test to detect any defects which may
have been introduced by the installation (e.g., incorrect configurations, functions not available). Usability
testing is normally performed in parallel with installability testing (e.g., to validate that users are provided
with understandable instructions and feedback/error messages during the installation).
4.7.3 Adaptability Testing
Adaptability testing checks whether a given application can function correctly in all intended target
environments (hardware, software, middleware, operating system, etc.). An adaptive system is therefore
an open system that is able to fit its behavior according to changes in its environment or in parts of the
system itself. Specifying tests for adaptability requires that combinations of the intended target
environments are identified, configured and available to the testing team. These environments are then
tested using a selection of functional test cases which exercise the various components present in the
environment.
Adaptability may relate to the ability of the software to be ported to various specified environments by
performing a predefined procedure. Tests may evaluate this procedure.
Adaptability tests may be performed in conjunction with installability tests and are typically followed by
functional tests to detect any defects which may have been introduced in adapting the software to a
different environment.
4.7.4 Replaceability Testing
Replaceability testing focuses on the ability of software components within a system to be exchanged
for others. This may be particularly relevant for systems which use commercial off-the-shelf (COTS)
software for specific system components.
Replaceability tests may be performed in parallel with functional integration tests where more than one
alternative component is available for integration into the complete system. Replaceability may be
evaluated by technical review or inspection at the architecture and design levels, where the emphasis
is placed on the clear definition of interfaces to potential replaceable components.

Certified Tester
International
Software Testing
4.8 Compatibility Testing
4.8.1 Introduction
Compatibility testing considers the following aspects [ISO25010]:
• Co-existence - the degree to which a test item can function satisfactorily alongside other
independent products in a shared environment. This is described below.
• Interoperability - the degree to which a system exchanges information with other systems or
components. This is described in the ISTQB® Advanced Level Test Analyst
[ISTQB_ALTA_SYL] syllabus.
4.8.2 Co-existence Testing
Computer systems which are not related to each other are said to co-exist when they can run in the
same environment (e.g., on the same hardware) without affecting each other's behavior (e.g., resource
conflicts). Tests for co-existence should be performed when new or upgraded software will be rolled out
into environments which already contain installed applications.
Co-existence problems may arise when the application is tested in an environment where it is the only
installed application (where incompatibility issues are not detectable) and then deployed onto another
environment (e.g., production) which also runs other applications.
Typical objectives of co-existence testing include:
• Evaluation of possible adverse impact on functionality when applications are loaded in the same
environment (e.g., conflicting resource usage when a server runs multiple applications)
• Evaluation of the impact to any application resulting from the deployment of operating system
fixes and upgrades
Co-existence issues should be analyzed when planning the targeted production environment but the
actual tests are normally performed after system testing has been successfully completed.

Certified Tester
International
Software Testing
5. Reviews - 165 mins.
Keywords
anti-pattern
Learning Objectives for Reviews
5.1 Technical Test Analyst Tasks in Reviews
TTA 5.1.1 (K2) Explain why review preparation is important for the Technical Test Analyst
5.2 Using Checklists in Reviews
TTA 5.2.1 (K4) Analyze an architectural design and identify problems according to a checklist provided
in the syllabus
TTA 5.2.2 (K4) Analyze a section of code or pseudo-code and identify problems according to a
checklist provided in the syllabus

Certified Tester
International
Software Testing
5.1 Technical Test Analyst Tasks in Reviews
Technical Test Analysts must be active participants in the technical review process, providing their
unique views. All review participants should have formal review training to better understand their
respective roles and must be committed to the benefits of a well-conducted technical review. This
includes maintaining a constructive working relationship with the authors when describing and
discussing review comments. For a complete description of technical reviews, including numerous
review checklists, see [Wiegers02]. Technical Test Analysts normally participate in technical reviews
and inspections where they bring an operational (behavioral) viewpoint that may be missed by
developers. In addition, Technical Test Analysts play an important role in the definition, application, and
maintenance of review checklists and defect severity information.
Regardless of the type of review being performed, the Technical Test Analyst must be allowed adequate
time to prepare. This includes time to review the work product, time to check cross-referenced
documentation to verify consistency, and time to determine what might be missing from the work
product. Without adequate preparation time, the review can become an editing exercise rather than a
true review. A good review includes understanding what is written, determining what is missing, and
verifying that the described product is consistent with other products that are either already developed
or are in development. For example, when reviewing an integration level test plan, the Technical Test
Analyst must also consider the items that are being integrated. Are they ready for integration? Are
there dependencies that must be documented? Is there data available to test the integration points? A
review is not isolated to the work product being reviewed. It must also consider the interaction of that
item with the others in the system.
5.2 Using Checklists in Reviews
Checklists are used during reviews to remind the participants to verify specific points during the review.
Checklists can also help to de-personalize the review, e.g., "this is the same checklist we use for every
review, and we are not targeting only your work product." Checklists can be generic and used for all
reviews or focused on specific quality characteristics or areas. For example,
a generic checklist might verify the proper usage of the terms "shall" and "should", verify proper
formatting and similar conformance items. A targeted checklist might concentrate on security issues or
performance efficiency issues.
The most useful checklists are those gradually developed by an individual organization, because they
reflect:
• The nature of the product
• The local development environment
o Staff
o Tools
o Priorities
• History of previous successes and defects
• Particular issues (e.g., performance efficiency, security)
Checklists should be customized for the organization and perhaps for the particular project. The
checklists provided in this chapter are meant only to serve as examples.
Some organizations extend the usual notion of a software checklist to include “anti-patterns” that refer
to common errors, poor techniques, and other ineffective practices. The term derives from the popular
concept of “design patterns” which are reusable solutions to common problems that have been shown
to be effective in practical situations [Gamma94]. An anti-pattern, then, is a commonly made error, often
implemented as an expedient short-cut.

Certified Tester
International
Software Testing
It is important to remember that if a requirement is not testable, meaning that it is not defined in such a
way that the Technical Test Analyst can determine how to test it, then it is a defect. For example, a
requirement that states “The software should be fast” cannot be tested. How can the Technical Test
Analyst determine if the software is fast? If, instead, the requirement said “The software must provide
a maximum response time of three seconds under specific load conditions”, then the testability of this
requirement is substantially better assuming the “specific load conditions” (e.g., number of concurrent
users, activities performed by the users) are defined. It is also an overarching requirement because this
one requirement could easily spawn many individual test cases in a non-trivial application. Traceability
from this requirement to the test cases is also critical because if the requirement should change, all the
test cases will need to be reviewed and updated as needed.
5.2.1 Architectural Reviews
Software architecture consists of the fundamental organization of a system, embodied in its
components, their relationships to each other and the environment, and the principles governing its
design and evolution. [ISO42010], [Bass03].
Checklists1 used for architecture reviews could, for example, include verification of the proper
implementation of the following items, which are quoted from [Web-2]:
• “Connection pooling - reducing the execution time overhead associated with establishing
database connections by establishing a shared pool of connections
• Load balancing – spreading the load evenly between a set of resources
• Distributed processing
• Caching – using a local copy of data to reduce access time
• Lazy instantiation
• Transaction concurrency
• Process isolation between Online Transactional Processing (OLTP) and Online Analytical
Processing (OLAP)
• Replication of data”
5.2.2 Code Reviews
Checklists for code reviews are necessarily very detailed, and, as with checklists for architecture
reviews, are most useful when they are language, project and company-specific. The inclusion of code-
level anti-patterns is helpful, particularly for less experienced software developers.
Checklists1 used for code reviews could include the following items:.
1. Structure
• Does the code completely and correctly implement the design?
• Does the code conform to any pertinent coding standards?
• Is the code well-structured, consistent in style, and consistently formatted?
• Are there any uncalled or unneeded procedures or any unreachable code?
• Are there any leftover stubs or test routines in the code?
• Can any code be replaced by calls to external reusable components or library functions?
• Are there any blocks of repeated code that could be condensed into a single procedure?
• Is storage use efficient?
• Are symbolics used rather than “magic number” constants or string constants?
• Are any modules excessively complex and should be restructured or split into multiple
modules?
1 The exam question will provide a subset of the checklist with which to answer the question

Certified Tester
International
Software Testing
2. Documentation
• Is the code clearly and adequately documented with an easy-to-maintain commenting style?
• Are all comments consistent with the code?
• Does the documentation conform to applicable standards?
3. Variables
• Are all variables properly defined with meaningful, consistent, and clear names?
• Are there any redundant or unused variables?
4. Arithmetic Operations
• Does the code avoid comparing floating-point numbers for equality?
• Does the code systematically prevent rounding errors?
• Does the code avoid additions and subtractions on numbers with greatly different magnitudes?
• Are divisors tested for zero or noise?
5. Loops and Branches
• Are all loops, branches, and logic constructs complete, correct, and properly nested?
• Are the most common cases tested first in IF-ELSEIF chains?
• Are all cases covered in an IF-ELSEIF or CASE block, including ELSE or DEFAULT clauses?
• Does every case statement have a default?
• Are loop termination conditions obvious and invariably achievable?
• Are indices or subscripts properly initialized, just prior to the loop?
• Can any statements that are enclosed within loops be placed outside the loops?
• Does the code in the loop avoid manipulating the index variable or using it upon exit from the
loop?
6. Defensive Programming
• Are indices, pointers, and subscripts tested against array, record, or file bounds?
• Are imported data and input arguments tested for validity and completeness?
• Are all output variables assigned?
• Is the correct data element operated on in each statement?
• Is every memory allocation released?
• Are timeouts or error traps used for external device access?
• Are files checked for existence before attempting to access them?
• Are all files and devices left in the correct state upon program termination?

Certified Tester
International
Software Testing
6. Test Tools and Automation - 180 mins.
Keywords
capture/playback, data-driven testing, debugging, emulator, fault seeding, hyperlink, keyword-driven
testing, performance efficiency, simulator, test execution, test management
Learning Objectives for Test Tools and Automation
6.1 Defining the Test Automation Project
TTA-6.1.1 (K2) Summarize the activities that the Technical Test Analyst performs when setting up a
test automation project
TTA-6.1.2 (K2) Summarize the differences between data-driven and keyword-driven automation
TTA-6.1.3 (K2) Summarize common technical issues that cause automation projects to fail to
achieve the planned return on investment
TTA-6.1.4 (K3) Construct keywords based on a given business process
6.2 Specific Test Tools
TTA-6.2.1 (K2) Summarize the purpose of tools for fault seeding and fault injection
TTA-6.2.2 (K2) Summarize the main characteristics and implementation issues for performance
testing tools
TTA-6.2.3 (K2) Explain the general purpose of tools used for web-based testing
TTA-6.2.4 (K2) Explain how tools support the practice of model-based testing
TTA-6.2.5 (K2) Outline the purpose of tools used to support component testing and the build process
TTA-6.2.6 (K2) Outline the purpose of tools used to support mobile application testing

Certified Tester
International
Software Testing
6.1 Defining the Test Automation Project
In order to be cost-effective, test tools (and particularly those which support test execution), must be
carefully architected and designed. Implementing a test execution automation strategy without a solid
architecture usually results in a tool set that is costly to maintain, insufficient for the purpose and unable
to achieve the target return on investment.
A test automation project should be considered a software development project. This includes the need
for architecture documentation, detailed design documentation, design and code reviews, component
and component integration testing, as well as final system testing. Testing can be needlessly delayed
or complicated when unstable or inaccurate test automation code is used.
There are multiple tasks that the Technical Test Analyst can perform regarding test execution
automation. These include:
• Determining who will be responsible for the test execution (possibly in coordination with a Test
Manager)
• Selecting the appropriate tool for the organization, timeline, skills of the team, and maintenance
requirements (note this could mean deciding to create a tool to use rather than acquiring one)
• Defining the interface requirements between the automation tool and other tools such as the
test management, defect management and tools used for continuous integration
• Developing any adapters which may be required to create an interface between the test
execution tool and the software under test
• Selecting the automation approach, i.e., keyword-driven or data-driven (see Section 6.1.1
below)
• Working with the Test Manager to estimate the cost of the implementation, including training. In
Agile projects this aspect would typically be discussed and agreed in project/sprint planning
meetings with the whole team.
• Scheduling the automation project and allocating the time for maintenance
• Training the Test Analysts and Business Analysts to use and supply data for the automation
• Determining how and when the automated tests will be executed
• Determining how the automated test results will be combined with the manual test results
In projects with a strong emphasis on test automation, a Test Automation Engineer may be tasked with
many of these activities (see the Advanced Level Test Automation Engineer syllabus
[ISTQB_ALTAE_SYL] for details). Certain organizational tasks may be taken on by a Test Manager
according to project needs and preferences. In Agile projects the assignment of these tasks to roles is
typically more flexible and less formal.
These activities and the resulting decisions will influence the scalability and maintainability of the
automation solution. Sufficient time must be spent researching the options, investigating available tools
and technologies and understanding the future plans for the organization.
6.1.1 Selecting the Automation Approach
This section considers the following factors which impact the test automation approach:
• Automating through the GUI
• Applying a data-driven approach
• Applying a keyword-driven approach
• Handling software failures
• Considering system state

Certified Tester
International
Software Testing
The Advanced Level Test Automation Engineer syllabus [ISTQB_ALTAE_SYL] includes further details
on selecting an automation approach.
6.1.1.1 Automating through the GUI
Test automation is not limited to testing through the GUI. Tools exist to help automate testing at the API
level, through a Command Line Interface (CLI) and other interface points in the software under test.
One of the first decisions the Technical Test Analyst must make is determining the most effective
interface to be accessed to automate the testing. General test execution tools require the development
of adapters to these interfaces. Planning shall consider the effort for the adapter development.
One of the difficulties of testing through the GUI is the tendency for the GUI to change as the software
evolves. Depending on the way the test automation code is designed, this can result in a significant
maintenance burden. For example, using the capture/playback capability of a test automation tool may
result in automated test cases (often called test scripts) that no longer run as desired if the GUI changes.
This is because the recorded script captures interactions with the graphical objects when the tester
executes the software manually. If the objects being accessed change, the recorded scripts may also
need updating to reflect those changes.
Capture/playback tools may be used as a convenient starting point for developing automation scripts.
The tester records a test session and recorded script is then modified to improve maintainability (e.g.,
by replacing sections in the recorded script with reusable functions).
6.1.1.2 Applying a Data-driven Approach
Depending on the software being tested, the data used for each test may be different although the
executed test steps are virtually identical (e.g., testing error handling for an input field by entering
multiple invalid values and checking the error returned for each). It is inefficient to develop and maintain
an automated test script for each of these values to be tested. A common technical solution to this
problem is to move the data from the scripts to an external store such as a spreadsheet or a database.
Functions are written to access the specific data for each execution of the test script, which enables a
single script to work through a set of test data that supplies the input values and expected result values
(e.g., a value shown in a text field or an error message). This approach is called data-driven.
When using this approach, in addition to the test scripts that process the supplied data, a harness and
infrastructure are needed to support the execution of the script or set of scripts. The actual data held in
the spreadsheet or database is created by Test Analysts who are familiar with the business function of
the software. In Agile projects the business representative (e.g., Product Owner) may also be involved
in defining data, in particular for acceptance tests. This division of labor allows those responsible for
developing test scripts (e.g., the Technical Test Analyst) to focus on the implementation of intelligent
automation scripts while the Test Analyst maintains ownership of the actual test. In most cases, the Test
Analyst will be responsible for executing the test scripts once the automation is implemented and tested.
6.1.1.3 Applying a Keyword-driven Approach
Another approach, called keyword- or action word-driven, goes a step further by also separating the
action to be performed on the supplied data from the test script [Buwalda01]. In order to accomplish this
further separation, a high-level meta language is created which is descriptive rather than directly
executable. Each statement of this language describes a full or partial business process of the domain
that may require testing. For example, business process keywords could include "Login", "CreateUser",
and "DeleteUser". A keyword describes a high-level action that will be performed in the application
domain. Lower level actions which denote interaction with the software interface itself, such as:
“ClickButton”, “SelectFromList”, or “TraverseTree” may also be defined and may be used to test GUI
capabilities that do not neatly fit into business process keywords.

Certified Tester
International
Software Testing
Once the keywords and data to be used have been defined, the test automator (e.g., Technical Test
Analyst or Test Automation Engineer) translates the business process keywords and lower level actions
into test automation code. The keywords and actions, along with the data to be used, may be stored in
spreadsheets or entered using specific tools which support keyword-driven test automation. The test
automation framework implements the keyword as a set of one or more executable functions or scripts.
Tools read test cases written with keywords and call the appropriate test functions or scripts which
implement them. The executables are implemented in a highly modular manner to enable easy mapping
to specific keywords. Programming skills are needed to implement these modular scripts.
This separation of the knowledge of the business logic from the actual programming required to
implement the test automation scripts provides the most effective use of the test resources. The
Technical Test Analyst, in the role as the test automator, can effectively apply programming skills without
having to become a domain expert across many areas of the business.
Separating the code from the changeable data helps to insulate the automation from changes, improving
the overall maintainability of the code and improving the return on the automation investment.
6.1.1.4 Handling Software Failures
In any test automation design, it is important to anticipate and handle software failures. If a failure
occurs, the test automator must determine what the software should do. Should the failure be recorded
and the tests continue? Should the tests be terminated? Can the failure be handled with a specific
action (such as clicking a button in a dialog box) or perhaps by adding a delay in the test? Unhandled
software failures may corrupt subsequent test results as well as causing a problem with the test that
was executing when the failure occurred.
6.1.1.5 Considering System State
It is also important to consider the state of the system at the start and end of the tests. It may be
necessary to ensure the system is returned to a pre-defined state after the test execution is completed.
This will allow a suite of automated tests to be run repeatedly without manual intervention to reset the
system to a known state. To do this, the test automation may have to, for example, delete the data it
created or alter the status of records in a database. The automation framework should ensure that a
proper termination has been accomplished at the end of the tests (i.e., logging out after the tests
complete).
6.1.2 Modeling Business Processes for Automation
In order to implement a keyword-driven approach for test automation, the business processes to be
tested must be modeled in the high-level keyword language. It is important that the language is intuitive
to its users who are likely to be the Test Analysts working on the project or, in the case of Agile projects,
the business representative (e.g., Product Owner).
Keywords are generally used to represent high-level business interactions with a system. For example,
“Cancel_Order” may require checking the existence of the order, verifying the access rights of the
person requesting the cancellation, displaying the order to be cancelled and requesting confirmation of
the cancellation. Sequences of keywords (e.g., “Login”, “Select_Order”, “Cancel_Order”), and the
relevant test data are used by the Test Analyst to specify test cases. The following is a simple keyword-
driven input table that could be used to test the ability of the software to add, reset and delete user
accounts:
Keyword User Password Result
Add_User User1 Pass1 User added message
Add_User @Rec34 @Rec35 User added message
Reset_Password User1 Welcome Password reset confirmation message

Certified Tester
International
Software Testing
Keyword User Password Result
Delete_User User1 Invalid username/password message
Add_User User3 Pass3 User added message
Delete_User User2 User not found message
The automation script that uses this table would look for the input values to be used by the automation
script. For example, when it gets to the row with the keyword “Delete_User”, only the user name is
required. To add a new user both user name and password are required. Input values may also be
referenced from a data store as shown with the second "Add_User" keyword where a reference to the
data is entered rather than the data itself providing more flexibility to access data that may be changing
as the tests execute. This allows data-driven techniques to be combined with the keyword scheme.
Issues to consider include the following:
• The more granular the keywords, the more specific the scenarios that can be covered, but the
high-level language may become more complex to maintain.
• Allowing Test Analysts to specify low-level actions ("ClickButton", "SelectFromList", etc.) makes
the keyword tests much more capable of handling different situations. However, because these
actions are tied directly to the GUI, it also may cause the tests to require more maintenance
when changes occur.
• Use of aggregated keywords may simplify development but complicate maintenance. For
example, there may be six different keywords that collectively create a record. Should a keyword
that actually calls all six keywords consecutively be created to simplify that action?
• No matter how much analysis goes into the keyword language, there will often be times when
new and different keywords will be needed. There are two separate domains to a keyword (i.e.,
the business logic behind it and the automation functionality to execute it). Therefore, a process
must be created to deal with both domains.
Keyword-based test automation can significantly reduce the maintenance costs of test automation, but
it is more costly, more difficult to develop, and takes more time to design correctly in order to gain the
expected return on investment.
The Advanced Test Automation Engineer syllabus [ISTQB_ALTAE_SYL] includes further details of
modelling business processes for automation.
6.2 Specific Test Tools
This section contains overview information on tools that are likely to be used by a Technical Test Analyst
beyond what is discussed in the Foundation Level syllabus [ISTQB_FL_SYL].
Note that detailed information about tools is provided by the following ISTQB® syllabi:
• Mobile Application Testing [ISTQB_FLMAT_SYL]
• Performance Testing [ISTQB_FLPT_SYL]
• Model-Based Testing [ISTQB_FLMBT_SYL]
• Test Automation Engineer [ISTQB_ALTAE_SYL]
6.2.1 Fault Seeding/Fault Injection Tools
Fault seeding tools actually modify the code under test (possibly using predefined algorithms) in order
to check the coverage achieved by specified tests. When applied in a systematic way this enables the
quality of the tests (i.e., their ability to detect the inserted defects) to be evaluated and, where necessary,
improved.
Fault injection tools deliberately supply incorrect inputs to the software to ensure the software can cope
with the fault. The inputs are injected to disrupt the normal execution flow of the code and enable test

Certified Tester
International
Software Testing
coverage to be extended (e.g., to cover more negative test conditions and test error handling
mechanisms).
Both of these types of tools are generally used by the Technical Test Analyst, but may also be used by
the developer when testing newly developed code.
6.2.2 Performance Testing Tools
Performance testing tools have the following main functions:
• Generating load
• Providing measurement, monitoring, visualization and analysis of the system response to a
given load
• Giving insights into the resource behavior of system and network components
Load generation is performed by implementing a pre-defined operational profile (see Section 4.5.3) as
a script. The script may initially be captured for a single user (possibly using a capture/playback tool)
and is then implemented for the specified operational profile using the performance test tool. This
implementation must take into account the variation of data per transaction (or sets of transactions).
Performance tools generate a load by simulating large numbers of multiple users (“virtual” users)
following their designated operational profiles to accomplish tasks including generating specific volumes
of input data. In comparison with individual test execution automation scripts, many performance testing
scripts reproduce user interaction with the system at the communications protocol level and not by
simulating user interaction via a graphical user interface. This usually reduces the number of separate
"sessions" needed during the testing. Some load generation tools can also drive the application using
its user interface to more closely measure response time while the system is under load.
A wide range of measurements are taken by a performance test tool to enable analysis during or after
execution of the test. Typical metrics taken and reports provided include:
• Number of simulated users throughout the test
• Number and type of transactions generated by the simulated users and the arrival rate of the
transactions
• Response times to particular transaction requests made by the users
• Reports and graphs of load against response times
• Reports on resource usage (e.g., usage over time with minimum and maximum values)
Significant factors to consider in the implementation of performance test tools include:
• The hardware and network bandwidth required to generate the load
• The compatibility of the tool with the communications protocol used by the system under test
• The flexibility of the tool to allow different operational profiles to be easily implemented
• The monitoring, analysis and reporting facilities required
Performance test tools are typically acquired rather than developed in-house due to the effort required
to develop them. It may, however, be appropriate to develop a specific performance tool if technical
restrictions prevent an available product being used, or if the load profile and facilities to be provided
are relatively simple. Further details of performance testing tools are provided in the Foundation Level
Performance Testing syllabus [ISTQB_FLPT_SYL].
6.2.3 Tools for Web-Based Testing
A variety of open source and commercial specialized tools are available for web testing. The following
list shows the purpose of some of the common web-based testing tools:
• Hyperlink test tools are used to scan and check that no broken or missing hyperlinks are present
on a web site

Certified Tester
International
Software Testing
• HTML and XML checkers are tools which check compliance to the HTML and XML standards
of the pages that are created by a web site
• Load simulators to test how the server will react when large numbers of users connect
• Lightweight automation execution tools that work with different browsers
• Tools to scan through the server, checking for orphaned (unlinked) files
• HTML specific spell checkers
• Cascading Style Sheet (CSS) checking tools
• Tools to check for standards violations e.g., Section 508 accessibility standards in the U.S. or
M/376 in Europe
• Tools that find a variety of security issues
The following are good sources of open source web testing tools
• The World Wide Web Consortium (W3C) [Web-3] This organization sets standards for the
Internet and supplies a variety of tools to check for errors against those standards.
• The Web Hypertext Application Technology Working Group (WHATWG) [Web-5]. This
organization sets HTML standards. They have a tool which performs HTML validation [Web-6].
Some tools that include a web spider engine can also provide information on the size of the pages and
on the time necessary to download them, and on whether a page is present or not (e.g., HTTP error
404). This provides useful information for the developer, the webmaster and the tester.
Test Analysts and Technical Test Analysts use these tools primarily during system testing.
6.2.4 Tools to Support Model-Based Testing
Model-Based Testing (MBT) is a technique whereby a formal model such as a finite state machine is
used to describe the intended execution-time behavior of a software-controlled system. Commercial
MBT tools (see [Utting07]) often provide an engine that allows a user to “execute” the model. Interesting
threads of execution can be saved and used as test cases. Other executable models such as Petri Nets
and Statecharts also support MBT.
MBT models (and tools) can be used to generate large sets of distinct execution threads. MBT tools
also can help reduce the very large number of possible paths that can be generated in a model. Testing
using these tools can provide a different view of the software to be tested. This can result in the
discovery of defects that might have been missed by functional testing.
Further details of model-based testing tools are provided in the Foundation Level Model-Based Testing
syllabus [ISTQB_FLMBT_SYL].
6.2.5 Component Testing and Build Tools
While component testing and build automation tools are developer tools, in many instances, they are
used and maintained by Technical Test Analysts, especially in the context of Agile development.
Component testing tools are often specific to the language that is used for programming a module. For
example, if Java was used as the programming language, JUnit might be used to automate the unit
testing. Many other languages have their own special test tools; these are collectively called xUnit
frameworks. Such a framework generates test objects for each class that is created, thus simplifying
the tasks that the programmer needs to do when automating the component testing.
Debugging tools facilitate manual component testing at a very low level, allowing developers and
Technical Test Analysts to change variable values during execution and step through the code line by
line while testing. Debugging tools are also used to help the developer isolate and identify problems in
the code when a failure is reported by the test team.

Certified Tester
International
Software Testing
Build automation tools often allow a new build to be automatically triggered any time a component is
changed. After the build is completed, other tools automatically execute the component tests. This level
of automation around the build process is usually seen in a continuous integration environment.
When set up correctly, this set of tools can have a very positive effect on the quality of builds being
released into testing. Should a change made by a programmer introduce regression defects into the
build, it will usually cause some of the automated tests to fail, triggering immediate investigation into the
cause of the failures before the build is released into the test environment.
6.2.6 Tools to Support Mobile Application Testing
Emulators and simulators are frequently used tools to support the testing of mobile applications.
6.2.6.1 Simulators
A mobile simulator models the mobile platform’s runtime environment. Applications tested on a simulator
are compiled into a dedicated version, which works in the simulator but not on a real device. Simulators
are sometimes used as replacements for real devices in testing. However, the application tested on a
simulator differs from the application that will be distributed.
6.2.6.2 Emulators
A mobile emulator models the hardware and utilizes the same runtime environment as the physical
hardware. Applications compiled to be deployed and tested on an emulator could also be used by the
real device.
Emulators are often used to reduce the cost of test environments by replacing real devices. However,
an emulator cannot fully replace a device because the emulator may behave in a different manner than
the mobile device it tries to mimic. In addition, some features may not be supported such as (multi)touch,
accelerometer, and others. This is partly caused by limitations of the platform used to run the emulator.
6.2.6.3 Common Aspects
Simulators and emulators are useful in the early stage of development as these typically integrate with
development environments and allow quick deployment, testing, and monitoring of applications. Using
an emulator or simulator requires launching it, installing the necessary app on it and then testing the
app as if it were on the actual device. Each mobile operating system development environment typically
comes with its own bundled emulator and simulator. Third party emulators and simulators are also
available.
Usually emulators and simulators allow the setting of various usage parameters. These settings might
include network emulation at different speeds, signal strengths and packet losses, changing the
orientation, generating interrupts, and GPS location data. Some of these settings can be very useful
because they can be difficult or costly to replicate with real devices, such as global GPS positions or
signal strengths.
The Foundation Level Mobile Application Testing syllabus [ISTQB_FLMAT_SYL] includes further
details.

Certified Tester
International
Software Testing
7. References
7.1 Standards
The following standards are mentioned in these respective chapters.
• [RTCA DO-178C/ED-12C]: Software Considerations in Airborne Systems and Equipment
Certification, RTCA/EUROCAE ED12C. 2013.
Chapter 2
• [ISO9126] ISO/IEC 9126-1:2001, Software Engineering – Software Product Quality
Chapter 4
• [ISO25010] ISO/IEC 25010 (2014) Systems and software engineering – Systems and
software Quality Requirements and Evaluation (SQuaRE) System and software quality models
Chapters 2 and 4
• [ISO29119] ISO/IEC/IEEE 29119-4 International Standard for Software and Systems
Engineering - Software Testing Part 4: Test techniques. 2015
Chapter 2
• [ISO42010] ISO/IEC/IEEE 42010:2011
Systems and software engineering - Architecture description
Chapter 5
• [IEC61508] IEC 61508-5 (2010) Functional Safety of Electrical/Electronic/Programmable
Electronic Safety-related Systems, Part 5: Examples of methods for the determination of safety
integrity levels
Chapter 2
7.2 ISTQB® Documents
• [ISTQB_AL_OVIEW] Advanced Level Overview, Version 2019
• [ISTQB_ALSEC_SYL] Advanced Level Security Testing Syllabus, Version 2016
• [ISTQB_ALTAE_SYL] Advanced Level Test Automation Engineer Syllabus, Version 2017
• [ISTQB_FL_SYL] Foundation Level Syllabus, Version 2018
• [ISTQB_FLPT_SYL] Foundation Level Performance Testing Syllabus, Version 2018
• [ISTQB_FLMBT_SYL] Foundation Level Model-Based Testing Syllabus, Version 2015
• [ISTQB_ALTA_SYL] Advanced Level Test Analyst Syllabus, Version 2019
• [ISTQB_ALTM_SYL] Advanced Level Test Manager Syllabus, Version 2012
• [ISTQB_FLMAT_SYL] Foundation Level Mobile Application Testing Syllabus, 2019
• [ISTQB_GLOSSARY] Glossary of Terms used in Software Testing, Version 3.2, 2019
7.3 Books
[Bass03] Len Bass, Paul Clements, Rick Kazman “Software Architecture in Practice (2nd edition)”,
Addison-Wesley 2003, ISBN 0-321-15495-9
[Bath14] Graham Bath, Judy McKay, “The Software Test Engineer’s Handbook (2nd edition)”, Rocky
Nook, 2014, ISBN 978-1-933952-24-6

Certified Tester
International
Software Testing
[Beizer90] Boris Beizer, "Software Testing Techniques Second Edition", International Thomson
Computer Press, 1990, ISBN 1-8503-2880-3
[Beizer95] Boris Beizer, “Black-box Testing”, John Wiley & Sons, 1995, ISBN 0-471-12094-4
[Burns18] Brendan Burns, “Designing Distributed Systems: Patterns and Paradigms for Scalable,
Reliable Services”, O’Reilly, 2018, ISBN 13: 978-1491983645
[Buwalda01]: Hans Buwalda, “Integrated Test Design and Automation”, Addison-Wesley Longman,
2001, ISBN 0-201-73725-6
[Copeland03]: Lee Copeland, “A Practitioner's Guide to Software Test Design”, Artech House, 2003,
ISBN 1-58053-791-X
[Gamma94] Design Patterns: Elements of Reusable Object-Oriented Software, Addison-Wesley, 1994,
ISBN 0-201-63361-2
[Jorgensen07]: Paul C. Jorgensen, “Software Testing, a Craftsman’s Approach third edition”, CRC
press, 2007, ISBN-13:978-0-8493-7475-3
[Kaner02]: Cem Kaner, James Bach, Bret Pettichord; “Lessons Learned in Software Testing”; Wiley,
2002, ISBN: 0-471-08112-4
[Koomen06]: Tim Koomen, Leo van der Aalst, Bart Broekman, Michael Vroon, "TMap Next for result-
driven testing"; UTN Publishers, 2006, ISBN: 90-72194-79-9
[McCabe76] Thomas J. McCabe, "A Complexity Measure", IEEE Transactions on Software Engineering,
Vol. SE-2, No. 4, December 1976. PP 308-320
[McCabe96] Arthur H. Watson and Thomas J. McCabe. "Structured Testing: A Testing Methodology Using
the Cyclomatic Complexity Metric" (PDF), 1996, NIST Special Publication 500-235.
[NIST96] Arthur H. Watson and Thomas J. McCabe, "Structured Testing: A Testing Methodology Using
the Cyclomatic Complexity Metric", NIST Special Publication 500-235, Prepared under NIST
Contract 43NANB517266, September 1996.
[Splaine01]: Steven Splaine, Stefan P. Jaskiel, “The Web-Testing Handbook”, STQE Publishing, 2001,
ISBN 0-970-43630-0
[Utting07] Mark Utting, Bruno Legeard, “Practical Model-Based Testing: A Tools Approach”, Morgan-
Kaufmann, 2007, ISBN: 978-0-12-372501-1
[Whittaker04]: James Whittaker and Herbert Thompson, “How to Break Software Security”, Pearson /
Addison-Wesley, 2004, ISBN 0-321-19433-0
[Wiegers02] Karl Wiegers, "Peer Reviews in Software: A Practical Guide", Addison-Wesley, 2002, ISBN
0-201-73485-0
7.4 Other References
The following references point to information available on the Internet. Even though these references
were checked at the time of publication of this Advanced Level Syllabus, the ISTQB® cannot be held
responsible if the references are not available anymore.
[Web-1] http://www.nist.gov NIST National Institute of Standards and Technology,
[Web-2] http://www.codeproject.com/KB/architecture/SWArchitectureReview.aspx
[Web-3] http://www.W3C.org
[Web-4] https://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project
[Web-5] https://whatwg.org
[Web-6] https://whatwg.org/validator/
Chapter 4: [Web-1] [Web-4]
Chapter 5: [Web-2]
Chapter 6: [Web-3] [Web-5] [Web-6]

Certified Tester
International
Software Testing
8. Appendix A: Quality Characteristics Overview
The following table compares the quality characteristics described in ISO 9126 (as used in the 2012
version of the Technical Test Analyst syllabus) with those in the newer [ISO25010] (as used in the 2019
version of the syllabus). Only the characteristics which are relevant for the Technical Test Analyst are
shown.
ISO/IEC 25010 ISO/IEC 9126-1 Notes
Performance efficiency Efficiency
Time behaviour Time behaviour
Resource utilization Resource utilization
Capacity New sub-characteristic
Compatibility New characteristic
Co-existence Co-existence Moved from Portability
Interoperability Moved from Functionality (Test Analyst)
Reliability Reliability
Maturity Maturity
Availability New sub-characteristic
Fault tolerance Fault tolerance
Recoverability Recoverability
Security Security No previous sub-characteristics
Confidentiality New sub-characteristic
Integrity New sub-characteristic
Non-repudiation New sub-characteristic
Accountability New sub-characteristic
Authenticity New sub-characteristic
Maintainability Maintainability
Modularity New sub-characteristic
Reusability New sub-characteristic
Analysability Analysability
Modifiability Stability Combines Changeability and Stability
Changeability
Testability Testability
Portability Portability
Adaptability Adaptability
Installability Installability
Co-existence Moved to Compatibility
Replaceability Replaceability
Compliance Deleted in 25010

Certified Tester
International
Software Testing
9. Index
action word-driven, 48
adaptability, 27
adaptability testing, 40
analyzability, 27
anti-pattern, 43
Application Programming Interface (API), 18
architectural reviews, 44
atomic condition, 13, 15
attack, 32
backup and restore, 34
benchmark, 29
capture/playback, 46
capture/playback tool, 48
client/server, 18
code reviews, 44
co-existence, 27
co-existence/compatibility testing, 41
cohesion, 23
control flow analysis, 20, 21
control flow coverage, 15
control flow graph, 21
control flow testing, 13
coupling, 23
cyclomatic complexity, 13, 20, 21
data flow analysis, 20, 21
data security considerations, 30
data-driven, 48
data-driven testing, 46
decision predicates, 14
decision testing, 13
definition-use pairs, 22
du-path, 22
dynamic analysis, 20, 24
memory leaks, 25
overview, 24
performance efficiency, 26
wild pointers, 25
dynamic maintainability testing, 39
emulator, 53
failover, 33, 34
fault seeding, 46
installability, 27
installability, 40
keyword-driven, 46, 48
load testing, 35
maintainability, 22, 27
maintainability testing, 38
master test plan, 29
maturity, 27
MC/DC, 15
MC/DC testing, 15
McCabe's design predicate, 24
memory leak, 20
metrics
performance, 26
modified condition/decision coverage
(MC/DC), 15
MTBF, 33
MTTR, 33
multiple condition coverage, 16
multiple condition testing, 13
neighborhood integration testing, 20, 24
operational acceptance test, 27
operational acceptance testing, 34
operational profile, 27, 35, 37
organizational considerations, 30
pairwise integration testing, 20, 23
path testing, 13
performance efficiency, 27
performance efficiency test planning, 36
performance efficiency test specification, 37
performance efficiency testing, 35
portability, 27
portability testing, 39, 41
product quality characteristics, 28
product risk, 10
quality attributes for technical testing, 27
recoverability, 27
recoverability testing, 33
redundant dissimilar software, 34
reliability, 27
reliability growth model, 27
reliability test planning, 35
reliability test specification, 35
reliability testing, 33
remote procedure calls (RPC), 18
replaceability, 27
replaceability testing, 40
required tooling, 29
resource utilization, 27
reviews, 42
checklists, 43
risk analysis, 10
risk assessment, 10, 11
risk identification, 10, 11
risk level, 10
risk mitigation, 10, 12
risk-based testing, 10
robustness, 27
Safety Integrity Level (SIL), 19
scalability testing, 36
security

Certified Tester
International
Software Testing
buffer overflow, 31
cross-site scripting, 31
denial of service, 31
logic bombs, 31
man in the middle, 31
security test planning, 31
security testing, 31
service-oriented architectures (SOA), 18
short-circuiting, 13, 16
Simplified Baseline Method, 17
simulator, 53
simulators, 29
stakeholder requirements, 29
standards
DO-178B, 19
ED-12C, 19
IEC 61508, 19
ISO 25010, 37
ISO 9126, 39
statement testing, 13
static analysis, 20, 21, 22
call graph, 23
tools, 22
stress testing, 36
test automation project, 47
test environment, 30
test for robustness, 33, 34
test of resource utilization, 37
test tools, 50
build automation, 53
component testing, 52
debugging, 46, 52
fault injection, 50
fault seeding, 50
hyperlink verification, 46, 51
mobile testing, 53
model-based testing, 52
performance, 51
unit testing, 52
web tools, 51
virtual users, 51
white-box technique, 13
wild pointer, 20

ISTQB Advance level syllabus 2019 Technical Test Analyst

More Related Content

What's hot (17)

Similar to ISTQB Advance level syllabus 2019 Technical Test Analyst (20)

More from Neeraj Kumar Singh (20)

Recently uploaded (20)

ISTQB Advance level syllabus 2019 Technical Test Analyst