Software Testing and Debugging
Download as PPTX, PDF5 likes5,372 views
This document provides an overview of software testing fundamentals. It discusses that software testing is a critical part of quality assurance and aims to identify errors by exercising a program with sample inputs and comparing actual outcomes to expected outcomes. There are different types of testing such as white box testing, which tests internal program structure, and black box testing, which tests external functionality without knowledge of internal structure. The document also outlines testing strategies, techniques for different types of tests, and the importance of testing in the software development process.
Software Testing and Debugging
- 1. Thanks to M8034 @ Peter Lo 2006 1 Software Testing 2 Software Testing Fundamentals ■ Software Testing is a critical element of software quality assurance and represents the ultimate review of specification, design and coding. ■ It concerned with the actively identifying errors in software ■ Testing of software is a means of measuring or assessing the software to determine its quality. ■ Testing is dynamic assessment of the software ◆ Sample input ◆ Actual outcome compare with expected outcome Testing Objectives ■ Software Testing is the process of exercising a program with the specific intent of finding errors prior to delivery to the end user. ■ It is the process of checking to see if software matches its specification for specific cases called Test Case. ■ A Good Test Case is one that has a high probability of finding an as yet undiscovered error. ■ A Successful Test is one that uncovers an as yet undiscoverederror. Testing vs. Debugging ■ Testing is different from debugging ■ Debugging is removal of defects in the software, a correction process ■ Testing is an assessment process ■ Testing consumes 40-50% of the development effort 3 4 Lecture 6 BSIT 6th Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006
- 2. 5 What can Testing Show? Errors Requirements Conformance Performance An indication of quality 6 Who Tests the Software? Independent TesterDeveloper Understands the system but, will test "gently" and, is driven by "delivery" Must learn about the system, but, will attempt to break it and, is driven by quality Testing Paradox ■ To gain confidence, a successful test is one that the software does as in the functional spec. ■ To reveal error, a successful test is one that finds an error. ■ In practice, a mixture of Defect-revealing and Correct-operation tests are used. Developer performs Constructive Actions Tester performs Destructive Actions Information Flow in Testing ■ Two classes of input are provided to the test process: ◆ Software Configuration: includes Software Requirements Specification, Design Specification, and source code. ◆ Test Configuration: includes Test Plan and Procedure, any testing tools that are to be used, and test cases and their expected results. 7 8 Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006
- 3. 9 Necessary Conditions for Testing ■ A controlled/observed environment, because tests must be exactly reproducible ◆ Sample Input – test uses only small sample input (limitation) ◆ Predicted Result – the results of a test ideally should be predictable ■ Actual output must be able to compare with the expectedoutput 10 Attributes of a “Good Test” ■ A good test has a high probability of finding an error. ◆ The tester must understand the software and attempt to develop a mental picture of how the software might fail. ■ A good test is not redundant. ◆ Testing time and resources are limited. ◆ There is no point in conducting a test that has the same purpose as another test. ■ A good test should be neither too simple nor too complex. ◆ Side effect of attempting to combine a series of tests into one test case is that this approach may mask errors. Attribute of Testability? ■ Operability — It operates cleanly ■ Observability — The results of each test case are readily observed ■ Controllability — The degree to which testing can be automated and optimized ■ Decomposability — Testing can be targeted ■ Simplicity — Reduce complex architecture and logic to simplify tests ■ Stability — Few changes are requested during testing ■ Understandability — The purpose of the system is clear to the evaluator Software Testing Technique ■ White Box Testing, or Structure Testing is derived directly from the implementation of a module and able to test all the implemented code ■ Black Box Testing, or Functional Testing is able to test any functionality is missing from the implementation. Methods White-box Testing Black-box Testing 11 12 Strategies Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006
- 4. 13 White Box Testing Technique ■ White box testing is a test case design method that uses the control structure of the procedural design to derive test cases. ... our goal is to ensure that all statements and conditions have been executed at least once ... 14 White Box Testing Technique ■ White Box Testing of software is predicated on close examination of procedural detail. ■ Logical paths through the software are tested by providing test cases that exercise specific sets of conditions and / or loops. ■ The status of the program may be examined at various points to determine if the expected or asserted status corresponds to the actual status. Process of White Box Testing ■ Tests are derived from an examination of the source code for the modules of the program. ■ These are fed as input to the implementation, and the execution traces are used to determine if there is sufficient coverage of the program source code Benefit of White Box Testing ■ Using white box testing methods, the software engineer can derive test cases that: ◆ Guarantee that all independent paths within a module have been exercised at least once; ◆ Exercise all logical decisions on their true or false sides; ◆ Execute all loops at their boundaries and within their operational bounds ; and ◆ Exercise internal data structures to ensure their validity. 15 16 Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006
- 5. 29 30 Exhaustive Testing ■ There are 1014 possible paths! If we execute one test per millisecond, it would take 3,170 years to test this program!! loop < 20 X Selective Testing loop < 20 X Selected path 31 32 Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006
- 6. 33 34 Condition Testing ■ Simple condition is a Boolean variable or relational expression ■ Condition testing is a test case design method that exercises the logical conditions contained in a program module, and therefore focuses on testing each condition in the program. Data Flow Testing ■ The Data flow testing method selects test paths of a program according to the locations of definitions and uses of variables in the program 36 Loop Testing ■ Loop testing is a white-box testing technique that focuses exclusively on the validity of loop constructs ■ Four classes of loops: 1. Simple loops 2. Concatenated loops 3. Nested loops 4. Unstructured loops 35 Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006
- 7. 37 Test Cases for Simple Loops ■ Where n is the maximum number of allowable passes through the loop: ◆ Skip the loop entirely ◆ Only one pass through the loop ◆ Two passes through the loop ◆ m passes through the loop where m<n ◆ n-1, n, n+1 passes through the loop Simple loop 38 Test Cases for Nested Loops ■ Start at the innermost loops. Set all other loops to minimum values ■ Conduct simple loop tests for the innermost loop while holding the outer loops at their minimum iteration parameter values. Add other tests for out-of -range or excluded values ■ Work outward, conducting tests for the next loop but keeping all other outer loops at minimum values and other nested loops to "typical" values ■ Continue until all loops have been tested Nested Loops Test Cases for Concatenated Loops ■ If each of the loops is independent of the others, perform simple loop tests for each loop ■ If the loops are dependent, apply the nested loop tests Concatenated Loops 40 Test Cases for Unstructured Loops ■ Whenever possible, redesign this class of loops to reflect the structured programming constructs Unstructured Loops39 Thanks to M8034 @ Peter Lo 2006Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006
- 8. 41 Black Box Testing ■ Black Box Testing focus on the functional requirements of the software, i.e. derives sets of input conditions that will fully exercise all functional requirements for a program. ■ Black box is based upon the specification of a module rather than the implementation of the module. Requirements EventsInput Output 42 Black Box Testing ■ Black box testing attempts to find errors in the following categories: ◆ Incorrect or missing functions ◆ Interface errors ◆ Errors in data structures or external databases access ◆ Performance errors ◆ Initialization and termination errors. Process of Black Box Testing Random Testing ■ Input is generated at random and submitted to the program and corresponding output is then compared. 43 44 Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006
- 9. 53 Comparison Testing ■ All the versions are executed in parallel with a real-time comparison of results to ensure consistency. ◆ If the output from each version is the same, then it is assumed that all implementations are correct. ◆ If the output is different, each of the applications is investigated to determine if a defect in one or more versions is responsible for the difference. 54 Automated Testing Tools ■ CodeAuditors ■ Assertion Processors ■ Test File Generators ■ Test Data Generators ■ Test Verifiers ■ Output Comparators Code Auditors ■ These special-purpose filters are used to check the quality of software to ensure that it meets minimum coding standards. 55 56 Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006
- 10. 61 Testing Strategy Unit Test ■ A testing strategy must always incorporate test planning, test case design, text execution, and the resultant data collection and evaluation Integration Test Validation Test System Test 62 Test Case Design "Bugs lurk in corners and congregate at boundaries ..." Boris Beizer OBJECTIVE CRITERIA CONSTRAINT to uncover errors in a complete manner with a minimum of effort and time Verification and Validation ■ Verification – Set of activities that ensure that software correctly implements a specific function ◆ Are we building the project right? ■ Validation – Set of activities that ensure that the software that has been built is traceable to customerrequirements ◆ Are we building the right product? Generic Characteristics of Software Testing Strategies ■ Testing beings at the module level and works toward the integration of the entire system ■ Different testing techniques are appropriate at different points in time ■ Testing is conducted by the software developer and an Independent Test Group (ITG) ■ Debugging must be accommodated in any testing strategy 63 64 Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006
- 11. M8034 @ Peter Lo 2006 65 Software Testing Strategy 66 Software Testing Strategy ■ A strategy for software testing moves outward along the spiral. ◆ Unit Testing: Concentrates on each unit of the software as implemented in the source code. ◆ Integration Testing: Focus on the design and the construction of the software architecture. ◆ Validation Testing: Requirements established as part of software requirement analysis are validated against the software that has been constructed. ◆ System Testing: The software and other system elements are tested as a whole. M8034 @ Peter Lo 2006 67 Testing Direction Software Testing Direction ■ Unit Tests ◆ Focuses on each module and makes heavy use of white box testing ■ Integration Tests ◆ Focuses on the design and construction of software architecture; black box testing is most prevalent with limited white box testing. ■ High-order Tests ◆ Conduct validation and system tests. Makes use of black box testing exclusively (such as Validation Test, System Test, Alpha and Beta Test, and other Specialized Testing). 68 Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006
- 12. 69 Unit Testing Module to be Tested Test Cases Results Software Engineer 70 Unit Testing ■ Unit testing focuses on the results from coding. ■ Each module is tested in turn and in isolation from the others. ■ Using the detail design description as a guide, important control paths are tested to uncover errors within the boundary of the module. ■ Uses white-box techniques. Interface Local Data Structures Boundary Conditions Independent Paths Error Handling Paths Module to be Tested Test Cases Unit Test Environment Module Stub Stub Driver RESULTS Interface Local Data Structures Boundary Conditions Independent Paths Error Handling Paths Test Cases Unit Testing Procedures ■ Module is not a stand-alone program, driver & stub software must be developed for each unit test. ■ A driver is a program that accepts test case data, passes such data to the module, and prints the relevant results. ■ Stubs serve to replace modules that are subordinate the module to be tested. ■ A stub or "dummy subprogram" uses the subordinate module's interface, may do nominal data manipulation, prints verification of entry, and returns. ■ Drivers and stubs also represent overhead 71 72 Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006
- 13. 73 Unit Test Considerations ■ The module interface is tested to ensure that information properly flows into and out of the program unit under test. ■ The local data structure is examined to ensure that data stored temporarily maintains its integrity during all steps in an algorithm's execution. ■ Boundary conditions are tested to ensure that the module operates properly at boundaries established to limit or restrict processing. ■ All independent paths (basis paths) through the control structure are exercised to ensure that all statements in a module have been executed at least once. ■ And finally, all error-handling paths are tested. 74 Integration Testing ■ A technique for constructing the program structure while at the same time conducting tests to uncover tests to uncover errors associated with interfacing ■ Objective is combining unit-tested modules and build a program structure that has been dictated by design. ■ Integration testing should be done incrementally. ■ It can be done top-down, bottom-up or in bi-directional. Top-down Integration ■ Modules are integrated by moving downward through the control hierarchy, beginning with main control module. ■ Subordinate modules are incorporated into the structure in either a depth-first or breadth- first manner. Top module is tested with stubs As new modules are integrated, some subset of tests is re-run A C D E B F G Stubs are replaced one at a time, "depth first" Procedure ■ The main control module is used as a test driver and stubs are substituted for all modules directly subordinate to the main control module ■ Subordinate stubs are replaced one at a time with actual modules ■ Tests are conducted as each module is integrated ■ On the completion of each set of tests, another stub is replaced with the real module ■ Regression testing may be conducted to ensure that new errors have not been introduced 75 76Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006
- 14. M8034 @ Peter Lo 2006 77 Example ■ For the program structure, the following test cases may be derived if top-down integration is conducted: ◆ Test case 1: Modules A and B are integrated ◆ Test case 2: Modules A, B and C are integrated ◆ Test case 3: Modules A., B, C and D are integrated (etc.) 78 Problem of Top-Down Testing ■ Inadequate testing at upper levels when data flows at low levels in the hierarchy are required ■ Delay many test until stubs are replaced with actual modules; but this can lead to difficulties in determining the cause of errors and tends to violate the highly constrained nature of the top- down approach ■ Develop stubs that perform limited functions that simulate the actual module; but this can lead to significantoverhead Bottom-Up Integration Testing ■ This integration process begins construction and testing with atomic modules. ■ Because modules are integrated from the bottom up, processing required for modules subordinate to a given level is always available and the need for stubs is eliminated. Drivers are replaced one at a time, "depth first" Worker modules are grouped into builds and integrated A B C D E F G cluster Procedure ■ Low-level modules are combined into clusters that perform a specific software sub-function ■ A driver is written to coordinate test case input and output ■ The cluster is tested ■ Drivers are removed and clusters are combined moving upward in the program structure 79 80 Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006
- 15. M8034 @ Peter Lo 2006 81 Example ■ Test case 1: Modules E and F are integrated ■ Test case 2: Modules E, F and G are integrated ■ Test case 3: Modules E., F, G and H are integrated ■ Test case 4: Modules E., F, G, H and C are integrated (etc.) ■ Drivers are used all round. 82 Validation Testing ■ Ensuring software functions can be reasonably expected by the customer. ■ Achieve through a series of black tests that demonstrate conformity with requirements. ■ A test plan outlines the classes of tests to be conducted, and a test procedure defines specific test cases that will be used in an attempt to uncover errors in conformity with requirements. ■ Validation testing begins, driven by the validation criteria that were elicited during requirement capture. ■ A series of acceptance tests are conducted with the end users Validation Testing ■ After the developers and the independent testers have satisfied, the end users carry out acceptance tests, which are part of the validation testing. ■ These occur in two stages. ◆ Alpha testing ◆ Is conducted at the developer's site by a customer ◆ The developer would supervise ◆ Is conducted in a controlled environment ◆ Beta testing ◆ Is conducted at one or more customer sites by the end user of the software ◆ The developer is generally not present ◆ Is conducted in a "live" environment System Testing ■ System testing is a series of different tests whose primary purpose is to fully exercise the computer-based system. ■ System testing focuses on those from system engineering. ■ Test the entire computer-based system. ■ One main concern is the interfaces between software, hardware and human components. ■ Kind of System Testing ◆ Recovery ◆ Security ◆ Stress ◆ Performance 83 84 Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006
- 16. 85 Recovery Testing ■ A system test that forces software to fail in a variety of ways and verifies that recovery is properly performed. ■ If recovery is automatic, re-initialization, check- pointing mechanisms, data recovery, and restart are each evaluated for correctness. ■ If recovery is manual, the mean time to repair is evaluated to determine whether it is within acceptablelimits. 86 Security Testing ■ Security testing attempts to verify that protection mechanisms built into a system will in fact protect it from improper penetration. ■ Particularly important to a computer-based system that manages sensitive information or is capable of causing actions that can improperly harm individuals when targeted. Stress Testing ■ Stress Testing is designed to confront programs with abnormal situations where unusual quantity frequency, or volume of resources are demanded ■ A variation is called sensitivity testing; ◆ Attempts to uncover data combinations within valid input classes that may cause instability or improper processing Performance Testing ■ Test the run-time performance of software within the context of an integrated system. ■ Extra instrumentation can monitor execution intervals, log events as they occur, and sample machine states on a regular basis ■ Use of instrumentation can uncover situations that lead to degradation and possible system failure 87 88 Thanks to M8034 @ Peter Lo 2006Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006
- 17. 89 Debugging ■ Debugging will be the process that results in the removal of the error after the execution of a test case. ■ Its objective is to remove the defects uncovered by tests. ■ Because the cause may not be directly linked to the symptom, it may be necessary to enumerate hypotheses explicitly and then design new test cases to allow confirmation or rejection of the hypothesized cause. 90 The Debugging Process Test Cases Results Debugging Identified Causes Corrections Regression Tests New test Cases Suspected Causes What is Bug? ■ A bug is a part of a program that, if executed in the right state, will cause the system to deviate from its specification (or cause the system to deviate from the behavior desired by the user). Characteristics of Bugs ■ The symptom and the cause may be geographically remote ■ The symptom may disappear when another error is corrected ■ The symptom may actually be caused by non-errors ■ The symptom may be caused by a human error that is not easily traced ■ It may be difficult to accurately reproduce input conditions ■ The symptom may be intermittent. This is particularly common in embedded systems that couple hardware and software inextricably ■ The symptom may be due to causes that are distributed across a number of tasks running on different processors 91 92 Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006
- 18. 93 Debugging Techniques Brute Force / Testing Backtracking Cause Elimination 94 Debugging Approaches – Brute Force ■ Probably the most common and least efficient method for isolating the cause of a software error. ■ The program is loaded with run-time traces, and WRITE statements, and hopefully some information will be produced that will indicated a clue to the cause of an error. Debugging Approaches – Backtracking ■ Fairly common in small programs. ■ Starting from where the symptom has been uncovered, backtrack manually until the site of the cause is found. ■ Unfortunately, as the number of source code lines increases, the number of potential backward paths may become unmanageably large. Debugging Approaches – Cause Elimination ■ Data related to the error occurrence is organized to isolate potential causes. ■ A "cause hypothesis" is devised and the above data are used to prove or disapprove the hypothesis. ■ Alternatively, a list of all possible causes is developed and tests are conducted to eliminate each. ■ If the initial tests indicate that a particular cause hypothesis shows promise, the data are refined in a attempt to isolate the bug. 95 96 Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006
- 19. 101 Debugging Effort Time required to diagnose the symptom and determine the cause Time required to correct the error and conduct regression tests 102 Consequences of Bugs infectious damage catastrophic extreme serious disturbing mild annoying Bug Type Bug Categories: function-related bugs, system-related bugs, data bugs, coding bugs, design bugs, documentation bugs, standards violations, etc. Debugging Tools ■ Debuggingcompilers ■ Dynamic debugging aides ("tracers") ■ Automatic test case generators ■ Memory dumps Debugging: Final Thoughts ■ Don't run off half-cocked, think about the symptom you're seeing. ■ Use tools (e.g., dynamic debugger) to gain more insight. ■ If at an impasse, get help from someone else. ■ Be absolutely sure to conduct regression tests when you do "fix" the bug. 103 104 Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006 Thanks to M8034 @ Peter Lo 2006