Chapter 1 Fundamentals of Testing
- 1. CHAPTER 1 FUNDAMENTALS OF TESTING Z E T R Y A N S A T R I A P R O G R A M S T U D I S 1 S I S T E M I N F O R M A S I U N I V E R S I T A S I S L A M N E G E R I S U L T A N S Y A R I F K A S I M R I A U
- 2. FUNDAMENTALS OF TESTING • In this chapter, we will introduce you to the fundamentals of testing: why testing is needed; its limitations, objectives and purpose; the principles behind testing; the process that testers follow; and some of the psychological factors that testers must consider in their work. By reading this chapter you'll gain an understanding of the fundamentals of testing and be able to describe those fundamentals.
- 3. 1.1 WHY IS TESTING NECESSARY 1. Describe, with examples, the way in which a defect in software can cause harm to a person, to the environment or to a company. (K2) 2. Distinguish between the root cause of a defect and its effects. (K2) 3. Give reasons why testing is necessary by giving examples. (K2) 4. Describe why testing is part of quality assurance and give examples of how testing contributes to higher quality. (K2) 5. Recall the terms 'mistake', 'defect', 'fault', 'failure' and the corresponding terms 'error' and 'bug'. (K1) 6. Explain the fundamental principles in testing. (K2)
- 4. INTRODUCTION • In this section, we're going to kick off the book with a discussion on why testing matters. We'll describe and illustrate how software defects or bugs can cause problems for people, the environment or a company. We'll draw important distinctions between defects, their root causes and their effects. We'll explain why testing is necessary to find these defects, how testing promotes quality, and how testing fits into quality assurance. In this section, we will also introduce some fundamental principles of testing. • As we go through this section, watch for the Syllabus terms bug, defect, error, failure, fault, mistake, quality, risk, software, testing and exhaustive testing. You'll these terms defined in the glossary.
- 5. INTRODUCTION • You may be asking 'what is testing?' and we'll look more closely at the definition of testing in Section 1.2. For the moment, let's adopt a simple everyday-life usage: 'when we are testing something we are checking whether it is OK'. We'll need to refine that when we define software testing later on. Let's start by considering why testing is needed. Testing is necessary because we all make mis- takes. Some of those mistakes are unimportant, but some of them are expensive or dangerous. We need to check everything and anything we produce because things can always go wrong - humans make mistakes all the time – it is what we do best! • Because we should assume our work contains mistakes, we all need to check our own work. However, some mistakes come from bad assumptions and blind spots, so we might make the same mistakes when we check our own work as we made when we did it. So we may not notice the flaws in what we have done. Ideally, we should get someone else to check our work – another person is more likely to spot the flaws.
- 6. SOFTWARE SYSTEMS CONTEXT Some of the problems we encounter when using software are quite trivial, but others can be costly and damaging - with loss of money, time or business reputation - and even may result in injury or death. For example, suppose a user interface has typographical defects. Does this matter? It may be trivial, but it could have a significant effect, depending on the website and the defect: • If my personal family-tree website has my maternal grandmother's maiden name spelt wrong, my mother gets annoyed and I have to put up with some family teasing, but I can fix it easily and only the family see it (probably). • If the company website has some spelling mistakes in the text, potential customers may be put off the company as it looks unprofessional. • If a software program miscalculates pesticide application quantities, the effect could be very significant: suppose a decimal point is wrongly placed so that the application rate is 10 times too large. The farmer or gardener uses more pesticide than needed, which raises his costs, has environmental impacts on wildlife and water supplies and has health and safety impact for the farmer, gardener, family and workforce, livestock and pets. There may also be consequent loss of trust in and business for the company and possible legal costs and fines for causing the environmental and health problems.
- 7. CAUSES OF SOFTWARE DEFECTS When we think about what might go wrong we have to consider defects and failures arising from: • errors in the specification, design and implementation of the software and system; • errors in use of the system; • environmental conditions; • intentional damage; • potential consequences of earlier errors, intentional damage, defects and failures.
- 8. WHAT IS QUALITY? • Projects aim to deliver software to specification. For the project to deliver what the customer needs requires a correct specification. Additionally, the delivered system must meet the specification. This is known as validation ('is this the right specification?') and verification ('is the system correct to specification?'). Of course, as well as wanting the right software system built correctly, the customer wants the project to be within budget and timescale – it should arrive when they need it and not cost too much. • The ISTQB glossary definition covers not just the specified requirements but also user and customer needs and expectations. It is important that the project team, the customers and any other project stakeholders set and agree expectations. We need to understand what the customers understand by quality and what their expectations are. What we as software developers and testers may see as quality – that the software meets its defined specification, is technically excellent and has few bugs in it – may not provide a quality solution for our customers. Furthermore, if our customers find they have spent more money than they wanted or that the software doesn't help them carry out their tasks, they won't be impressed by the technical excellence of the solution. If the customer wants a cheap car for a 'run- about' and has a small budget then an expensive sports car or a military tank are not quality solutions, however well built they are.
- 9. WHAT IS ROOT CAUSE ANALYSIS? • If your testing is confined to software, you might look at these and say, 'These are not software problems, so they don't concern us!' So, as software testers we might confine ourselves to reporting the printer driver failure. However, our remit as testers may be beyond the software; we might have a remit to look at a whole system including hardware and firmware. Additionally, even if our remit is software, we might want to consider how software might help people prevent or resolve problems; we may look beyond this view. The software could provide a user interface which helps the user anticipate when paper or ink is getting low. It could provide simple step-by-step instructions to help the users change the cartridges or replenish paper. It could provide a high temperature warning so that the environment can be managed. As testers, we want not just to think and report on defects but, with the rest of the project team, think about any potential causes of failures`.
- 10. DEFINING SOFTWARE TESTING First, let's look at testing as a process : 1. Process 2. All life cycle activities 3. Both static and dynamic 4. Planning 5. Preparation 6. Evaluation 7. Software products and related work products
- 11. WHEN CAN WE MEET OUR TEST OBJECTIVES? We mentioned above that testing can have different goals and objectives, which often include: 1. finding defects; 2. gaining confidence in and providing information about the level of quality; 3. preventing defects.
- 12. TEST PLANNING AND CONTROL • During test planning, we make sure we understand the goals and objectives of the customers, stakeholders, and the project, and the risks which testing is intended to address. This will give us what is sometimes called the mission of testing or the test assignment. Based on this understanding, we set the goals and objectives for the testing itself, and derive an approach and plan for the tests, including specification of test activities. To help us we may have organization or program test policies and a test strategy.
- 13. TEST PLANNING AND CONTROL Test planning has the following major tasks, given approxi- mately in order, which help us build a test plan: 1. Determine the scope and risks and identify the objectives of testing 2. Determine the test approach (techniques, test items, coverage, identifying and interfacing with the teams involved in testing, testware) 3. Implement the test policy and/or the test strategy 4. Determine the required test resources (e.g. people, test environment, PCs) 5. Schedule test analysis and design tasks, test implementation, execution and evaluation 6. Determine the exit criteria
- 14. TEST ANALYSIS AND DESIGN Test analysis and design is the activity where general testing objectives are trans- formed into tangible test conditions and test designs. During test analysis and design, we take general testing objectives identified during planning and build test designs and test procedures (scripts). Test analysis and design has the following major tasks, in approximately the following order: 1. Review the test basis 2. Identify test conditions 3. Design the tests 4. Evaluate testability of the requirements and system 5. Design the test environment set-up and identify any required infrastructure and tools
- 15. TEST IMPLEMENTATION AND EXECUTION Test implementation and execution have the following major tasks, in approximately the following order: • Implementation: 1. Develop and prioritize our test cases 2. Create test suites from the test cases for efficient test execution 3. Implement and verify the environment • Execution