![1. INTRODUCTION
We need to know what we are trying to test, the inputs, the results that should be
produced by those inputs, and how we actually get ready for and run the tests.
In this section we are looking at three things: test conditions, test cases and test
procedures (or scripts) – they are described in the sections below. Each is specified in
its own document, according to the Test Documentation Standard [IEEE829].
Test conditions are documented in a Test Design Specification. We will look at how to
choose test conditions and prioritize them.
Test cases are documented in a Test Case Specification. We will look at how to write a
good test case, showing clear traceability to the test basis (e.g. the requirement
specification) as well as to test conditions.
Test procedures are documented (as you may expect) in a Test Procedure Specification
(also known as a test script or a manual test script). We will look at how to translate
test cases into test procedures relevant to the knowledge of the tester who will be
executing the test, and we will look at how to produce a test execution schedule, using
prioritization and technical and logical dependencies.
In this section, look for the definitions of the glossary terms: test case, test case
specification, test condition, test data, test procedure specification, test script and
traceability.](https://image.slidesharecdn.com/presentation1-171211083503/85/IDENTIFYING-TEST-CONDITIONS-AND-DESIGNING-TEST-CASES-2-320.jpg)
![3. TEST ANALYSIS: IDENTIFYING TEST CONDITIONS
Test analysis is the process of looking at something that can be used to derive test
information. This basis for the tests is called the 'test basis'. It could be a system
requirement, a technical specification, the code itself (for structural testing), or a business
process. Sometimes tests can be based on an experienced user's knowledge of the system,
which may not be documented. The test basis includes whatever the tests are based on.
From a testing perspective, we look at the test basis in order to see what could be tested –
these are the test conditions. A test condition is simply something that we could test. If we
are looking to measure coverage of code decisions (branches), then the test basis would be
the code itself, and the list of test conditions would be the decision outcomes (True and
False). If we have a requirements specification, the table of contents can be our initial list of
test conditions.
A good way to understand requirements better is to try to define tests to meet those
requirements, as pointed out by [Hetzel, 1988].For example, if we are testing a customer
management and marketing system for a mobile phone company, we might have test
conditions that are related to a marketing campaign, such as age of customer (pre-teen,
teenager, young adult, mature), gender, postcode or zip code, and purchasing preference
(pay-as-you-go or contract). A particular advertising campaign could be aimed at male
teenaged customers in the mid-west of the USA on pay-as-you-go, for example.](https://image.slidesharecdn.com/presentation1-171211083503/85/IDENTIFYING-TEST-CONDITIONS-AND-DESIGNING-TEST-CASES-4-320.jpg)
![3. Continue ~
Testing experts use different names to represent the basic idea of 'a list of things that we
could test'. For example, Marick refers to 'test requirements' as things that should be tested.
Although it is not intended to imply that everything must be tested, it is too easily
interpreted in that way. [Marick, 1994] In contrast, Hutcheson talks about the 'test inventory'
as a list of things that could be tested [Hutcheson, 2003]; Craig talks about 'test objectives'
as broad categories of things to test and 'test inventories' as the actual list of things that
need to be tested [Craig, 2002]. These authors are all referring to what the ISTQB glossary
calls a test condition.
A testing technique helps us select a good set of tests from the total number of all possible
tests for a given system. Different techniques offer different ways of looking at the software
under test, possibly challenging assumptions made about it. Each technique provides a set of
rules or guidelines for the tester to follow in identifying test conditions and test cases.
The test conditions that are chosen will depend on the test strategy or detailed test
approach. For example, they might be based on risk, models of the system, likely failures,
compliance requirements, expert advice or heuristics. The word 'heuristic' comes from the
same Greek root as eureka, which means 'I find'. A heuristic is a way of directing your
attention, a common sense rule useful in solving a problem.
Test conditions should be able to be linked back to their sources in the test basis – this is
called traceability.](https://image.slidesharecdn.com/presentation1-171211083503/85/IDENTIFYING-TEST-CONDITIONS-AND-DESIGNING-TEST-CASES-5-320.jpg)
![4. TEST DESIGN: SPECIFYING TEST CASES
Test conditions can be rather vague, covering quite a large range of possibilities as we saw
with our mobile phone company example (e.g. a teenager in the mid-west), or a test
condition may be more specific (e.g. a particular male customer on pay-as-you-go with less
than $10 credit). However when we come to make a test case, we are required to be very
specific; in fact we now need exact and detailed specific inputs, not general descriptions (e.g.
Jim Green, age 17, living in Grand Rapids, Michigan, with credit of $8.64, expected result:
add to Q4 marketing campaign). Note that one test case covers a number of conditions
(teenager, male, mid-west area, pay-as-you-go, and credit of less than $10).
Should these detailed test cases be written down? They can be formally documented, as we
will describe below. However, it is possible to test without documenting at the test-case
level. If you give an experienced user acceptance tester with a strong business background a
list of high-level test conditions, they could probably do a good job of testing. But if you gave
the same list to a new starter who didn't know the system at all, they would probably be
lost, so they would benefit from having more detailed test cases.
One of the most important aspects of a test is that it assesses that the system does what it is
supposed to do. Copeland says 'At its core, testing is the process of comparing "what is" with
"what ought to be" '. [Copeland, 2003] If we simply put in some inputs and think 'that was
fun, I guess the system is probably OK because it didn't crash', then are we actually testing
it? We don't think so. You have observed that the system does what the system does – this is
not a test. Boris Beizer refers to this as 'kiddie testing' [Beizer, 1990]. We may not know what
the right answer is in detail every time, and we can still get some benefit from this approach
at times, but it isn't really testing.](https://image.slidesharecdn.com/presentation1-171211083503/85/IDENTIFYING-TEST-CONDITIONS-AND-DESIGNING-TEST-CASES-7-320.jpg)
IDENTIFYING TEST CONDITIONS AND DESIGNING TEST CASES
- 1. IDENTIFYING TEST CONDITIONS AND DESIGNING TEST CASES Test design techniques by Nathania Adisya Permata Information System Study Program Science and Technology Faculty State Islamic University Sultan Syarif Kasim
- 2. 1. INTRODUCTION We need to know what we are trying to test, the inputs, the results that should be produced by those inputs, and how we actually get ready for and run the tests. In this section we are looking at three things: test conditions, test cases and test procedures (or scripts) – they are described in the sections below. Each is specified in its own document, according to the Test Documentation Standard [IEEE829]. Test conditions are documented in a Test Design Specification. We will look at how to choose test conditions and prioritize them. Test cases are documented in a Test Case Specification. We will look at how to write a good test case, showing clear traceability to the test basis (e.g. the requirement specification) as well as to test conditions. Test procedures are documented (as you may expect) in a Test Procedure Specification (also known as a test script or a manual test script). We will look at how to translate test cases into test procedures relevant to the knowledge of the tester who will be executing the test, and we will look at how to produce a test execution schedule, using prioritization and technical and logical dependencies. In this section, look for the definitions of the glossary terms: test case, test case specification, test condition, test data, test procedure specification, test script and traceability.
- 3. 2. FORMALITY OF TEST DOCUMENTATION Testing may be performed with varying degrees of formality. Very formal testing would have extensive documentation which is well controlled, and would expect the documented detail of the tests to include the exact and specific input and expected outcome of the test. Very informal testing may have no documentation at all, or only notes kept by individual testers, but we'd still expect the testers to have in their minds and notes some idea of what they intended to test and what they expected the outcome to be. Most people are probably somewhere in between! The right level of formality for you depends on your context: a commercial safety-critical application has very different needs than a one-off application to be used by only a few people for a short time. The level of formality is also influenced by your organization – its culture, the people working there, how mature the development process is, how mature the testing process is, etc. The thoroughness of your test documentation may also depend on your time constraints; under excessive deadline pressure, keeping good documentation may be compromised. In this chapter we will describe a fairly formal documentation style. If this is not appropriate for you, you might adopt a less formal approach, but you will be aware of how to increase formality if you need to.
- 4. 3. TEST ANALYSIS: IDENTIFYING TEST CONDITIONS Test analysis is the process of looking at something that can be used to derive test information. This basis for the tests is called the 'test basis'. It could be a system requirement, a technical specification, the code itself (for structural testing), or a business process. Sometimes tests can be based on an experienced user's knowledge of the system, which may not be documented. The test basis includes whatever the tests are based on. From a testing perspective, we look at the test basis in order to see what could be tested – these are the test conditions. A test condition is simply something that we could test. If we are looking to measure coverage of code decisions (branches), then the test basis would be the code itself, and the list of test conditions would be the decision outcomes (True and False). If we have a requirements specification, the table of contents can be our initial list of test conditions. A good way to understand requirements better is to try to define tests to meet those requirements, as pointed out by [Hetzel, 1988].For example, if we are testing a customer management and marketing system for a mobile phone company, we might have test conditions that are related to a marketing campaign, such as age of customer (pre-teen, teenager, young adult, mature), gender, postcode or zip code, and purchasing preference (pay-as-you-go or contract). A particular advertising campaign could be aimed at male teenaged customers in the mid-west of the USA on pay-as-you-go, for example.
- 5. 3. Continue ~ Testing experts use different names to represent the basic idea of 'a list of things that we could test'. For example, Marick refers to 'test requirements' as things that should be tested. Although it is not intended to imply that everything must be tested, it is too easily interpreted in that way. [Marick, 1994] In contrast, Hutcheson talks about the 'test inventory' as a list of things that could be tested [Hutcheson, 2003]; Craig talks about 'test objectives' as broad categories of things to test and 'test inventories' as the actual list of things that need to be tested [Craig, 2002]. These authors are all referring to what the ISTQB glossary calls a test condition. A testing technique helps us select a good set of tests from the total number of all possible tests for a given system. Different techniques offer different ways of looking at the software under test, possibly challenging assumptions made about it. Each technique provides a set of rules or guidelines for the tester to follow in identifying test conditions and test cases. The test conditions that are chosen will depend on the test strategy or detailed test approach. For example, they might be based on risk, models of the system, likely failures, compliance requirements, expert advice or heuristics. The word 'heuristic' comes from the same Greek root as eureka, which means 'I find'. A heuristic is a way of directing your attention, a common sense rule useful in solving a problem. Test conditions should be able to be linked back to their sources in the test basis – this is called traceability.
- 6. 3. Continue ~ Why is traceability important? Consider these examples: • The requirements for a given function or feature have changed. Some of the fields now have different ranges that can be entered. Which tests were looking at those boundaries? They now need to be changed. How many tests will actually be affected by this change in the requirements? These questions can be answered easily if the requirements can easily be traced to the tests. • A set of tests that has run OK in the past has started to have serious prob lems. What functionality do these tests actually exercise? Traceability between the tests and the requirement being tested enables the functions or features affected to be identified more easily. • Before delivering a new release, we want to know whether or not we have tested all of the specified requirements in the requirements specification. We have the list of the tests that have passed – was every requirement tested? Note that spending some extra time now, while identifying test conditions, doesn't take very long, as we are only listing things that we could test. This is a good investment of our time – we don't want to spend time implementing poor tests! Test conditions can be identified for test data as well as for test inputs and test outcomes, for example, different types of record, different distribution of types of record within a file or database, different sizes of records or fields in a record. The test data should be designed to represent the most important types of data, i.e. the most important data conditions. Test conditions are documented in the IEEE 829 document called a Test Design Specification, shown below.
- 7. 4. TEST DESIGN: SPECIFYING TEST CASES Test conditions can be rather vague, covering quite a large range of possibilities as we saw with our mobile phone company example (e.g. a teenager in the mid-west), or a test condition may be more specific (e.g. a particular male customer on pay-as-you-go with less than $10 credit). However when we come to make a test case, we are required to be very specific; in fact we now need exact and detailed specific inputs, not general descriptions (e.g. Jim Green, age 17, living in Grand Rapids, Michigan, with credit of $8.64, expected result: add to Q4 marketing campaign). Note that one test case covers a number of conditions (teenager, male, mid-west area, pay-as-you-go, and credit of less than $10). Should these detailed test cases be written down? They can be formally documented, as we will describe below. However, it is possible to test without documenting at the test-case level. If you give an experienced user acceptance tester with a strong business background a list of high-level test conditions, they could probably do a good job of testing. But if you gave the same list to a new starter who didn't know the system at all, they would probably be lost, so they would benefit from having more detailed test cases. One of the most important aspects of a test is that it assesses that the system does what it is supposed to do. Copeland says 'At its core, testing is the process of comparing "what is" with "what ought to be" '. [Copeland, 2003] If we simply put in some inputs and think 'that was fun, I guess the system is probably OK because it didn't crash', then are we actually testing it? We don't think so. You have observed that the system does what the system does – this is not a test. Boris Beizer refers to this as 'kiddie testing' [Beizer, 1990]. We may not know what the right answer is in detail every time, and we can still get some benefit from this approach at times, but it isn't really testing.
- 8. 4. Continue ~ What if we don't decide on the expected results before we run a test? We can still look at what the system produces and would probably notice if something was wildly wrong. However, we would probably not notice small differences in calculations, or results that seemed to look OK (i.e. are plausible). So we would conclude that the test had passed, when in fact the software has not given the correct result. Small differences in one calculation can add up to something very major later on, for example if results are multiplied by a large factor. Ideally expected results should be predicted before the test is run – then your assessment of whether or not the software did the right thing will be more objective. For a few applications it may not be possible to predict or know exactly what an expected result should be; we can only do a 'reasonableness check'. In this case we have a 'partial oracle' – we know when something is very wrong, but would probably have to accept something that looked reasonable. An example is when a system has been written to calculate something where it may not be possible to manually produce expected results in a reasonable timescale because the calculations are so complex. In addition to the expected results, the test case also specifies the environment and other things that must be in place before the test can be run (the pre-conditions) and any things that should apply after the test completes (the postconditions).
- 9. 5. TEST IMPLEMENTATION: SPECIFYING TEST PROCEDURES OR SCRIPTS The next step is to group the test cases in a sensible way for executing them and to specify the sequential steps that need to be done to run the test. For example, a set of simple tests that cover the breadth of the system may form a regression suite, or all of the tests that explore the working of a given functionality or feature in depth may be grouped to be run together. Some test cases may need to be run in a particular sequence. For example, a test may create a new customer record, amend that newly created record and then delete it. These tests need to be run in the correct order, or they won't test what they are meant to test. The document that describes the steps to be taken in running a set of tests (and specifies the executable order of the tests) is called a test procedure in IEEE 829, and is often also referred to as a test script. It could be called a manual test script for tests that are intended to be run manually rather than using a test execution tool. Test script is also used to describe the instructions to a test execution tool. An automation script is written in a programming language that the tool can interpret. (This is an automated test procedure.) See Chapter 6 for more information on this and other types of testing tools. The test procedures, or test scripts, are then formed into a test execution schedule that specifies which procedures are to be run first – a kind of super-script. The test schedule would say when a given script should be run and by whom. The logical and technical dependencies between the scripts would also be taken into account when scheduling the scripts. For example, a regression script may always be the first to be run when a new release of the software arrives, as a smoke test or sanity check.
- 10. 5. Continue ~ Returning to our example of the mobile phone company's marketing campaign, we may have some tests to set up customers of different types on the database. It may be sensible to run all of the setup for a group of tests first. So our first test procedure would entail setting up a number of customers, including Jim Green, on the database. Test procedure DB15: Set up customers for marketing campaign Y. Step 1: Open database with write privilege Step 2: Set up customer Bob Flounders male, 62, Hudsonville, contract Step 3: Set up customer Jim Green male, 17, Grand Rapids, pay-as-you-go, $8.64 Step 4: ... We may then have another test procedure to do with the marketing campaign: Test procedure MC03: Special offers for low-credit teenagers Step 1: Get details for Jim Green from database Step 2: Send text message offering double credit Step 3: Jim Green requests $20 credit, $40 credited