345_12325_SE492_20112012_1__1_1_Lect 3 Software Testing Fundamentals .ppt
- 1. Lecture 3 Software Testing Fundamentals Prof. Mostafa Abdel Aziem Mostafa
- 2. Contents 3.1 General Testing Principles (The Seven Key Principles) 3.2 Fundamental Test Process 3.3 Psychology of Testing
- 3. 3.1 General Testing Principles 1.Testing shows presence of Defects 2.Exhaustive Testing is Impossible! 3.Early Testing 4.Defect Clustering 5.The Pesticide Paradox 6.Testing is Context Dependent 7.Absence of Errors Fallacy (The Seven Key Principles)
- 4. General Testing Principles 1.Testing shows the presence of Defects ( it is not a ghost. It is a fact ) • We test to find Faults (as known as Defects) • As we find more defects, the probability of undiscovered defects remaining in a system reduces ( decreasing nature). • However Testing cannot prove that there are no defects present The Seven Key Principles
- 5. Why Testing is necessary • “The probability of the existence of more errors in a section of a program is proportional to the number of errors already found in that program” • “Do not plan a test effort under the tacit assumption that no errors will be found” • “A good test is one that has a high probability of detecting an as yet undiscovered error” • “A successful test is one that detects an as-yet undiscovered error” Myers 2004 Testing Pearls of Wisdom
- 6. General Testing Principles 2.Exhaustive Testing is Impossible! • We have learned that we cannot test everything (i.e. all combinations of inputs and pre-conditions). • That is we must Prioritise our testing effort using a Risk Based Approach. The Seven Key Principles
- 7. Why Testing is Necessary System has 20 screens Average 4 menus / screen Average 3 options / menu Average of 10 fields / screen 2 types of input per field Around 100 possible values Approximate total for exhaustive testing 20 x 4 x 3 x 10 x 2 x 100 = 480,000 tests Test length = 1 sec then test duration = 17.7 days Test length = 10 sec then test duration = 34 weeks Test length = 1 min then test duration = 4 years Test length = 10 mins then test duration = 40 years! Why don’t we test everything ? It is not a matter of time. But,time is money ( salary is taken by hour. So second is valuable for software houses)
- 8. Urgency of Equilibrium • If you test too little, the probability of software failure increases • If you try to test too much, the development cost becomes unaffordable • So, we need to conduct some sort of equilibrium Cost of Testing Number of Missed Bugs Over Testing Amount of Testing Under Testing Testing Equilibrium Q u a n t i t y
- 9. General Testing Principles 3.Early testing • Testing activities should start as early as possible in the development life cycle • These activities should be focused on defined objectives – outlined in the Test Strategy • Remember from our Definition of Testing, that Testing doesn’t start once the code has been written! The Seven Key Principles
- 10. 0 10 20 30 40 50 60 70 80 90 100 Reqs Des Code Unit Accept Use Cost Cost of Fault Correction Relative Multiples
- 11. General Testing Principles 4.Defect Clustering • Defects are not evenly distributed in a system • They are ‘clustered’ • In other words, most defects found during testing are usually confined to a small number of modules ( 80% of uncovered errors focused in 20% modules of the whole application) “Pareto law” • Similarly, most operational failures of a system are usually confined to a small number of modules • An important consideration in test prioritisation! The Seven Key Principles
- 12. General Testing Principles 5.The Pesticide Paradox ( الحشرية المبيدات مفارقة ) • Testing identifies bugs, and programmers respond to fix them • As bugs are eliminated by the programmers, the software improves • As software improves the effectiveness of previous tests erodes (تآكل ) • Therefore we must learn, create and use new tests based on new techniques to catch new bugs ( i.e. It is not a matter of repetition. It is a matter of learning and improving) • N.B It's called the "pesticide paradox" after the agricultural phenomenon, where bugs such as the boll weevil build up tolerance to pesticides, leaving you with the choice of ever-more powerful pesticides followed by ever-more powerful bugs or an altogether different approach.’ – Beizer 1995 The Seven Key Principles
- 13. General Testing Principles 6.Testing is Context (background)Dependent • Testing is done differently in different contexts • For example, safety-critical software is tested differently from an e-commerce site • Whilst, Testing can be 50% of development costs, in NASA's Apollo program (it was 80% testing) • 3 to 10 failures per thousand lines of code (KLOC) typical for commercial software • 1 to 3 failures per KLOC typical for industrial software • 0.01 failures per KLOC for NASA Shuttle code! • Also different industries impose different testing standards The Seven Key Principles
- 14. General Testing Principles 7.Absence of Errors Fallacy ( )مغالطة • If we build a system and, in doing so, find and fix defects .... It doesn’t make it a good system • Even after defects have been resolved, it may still be unusable and/or does not fulfil the users’ needs and expectations • Visit the following site: http://www.guru99.com/software-testing-seven-principles.h tml The Seven Key Principles
- 15. 3.2 Fundamental Test Process The five stages of the fundamental test process • Test Planning and Control • Test Analysis and Design • Test Implementation and Execution • Evaluating Exit Criteria and Reporting • Test Closure Activities
- 16. Fundamental Test Process • The process always starts with planning and ends with test closure activities • Each phase may have to be executed a number of times in order to fulfil exit or completion criteria • Although logically sequential, the activities in the process may overlap or take place concurrently Fundamental Test Process
- 17. Fundamental Test Process Test Planning • Specifies how the test strategy and project Test Plan A document describing the scope, approach, resources and schedule of intended test activities apply to the software under test • Principally: – verify the mission, – define the Test objectives – Specify the Test Activities required to meet the mission and objectives 1.Test Planning and Control
- 18. Fundamental Test Process Test Planning (continued) • Major Tasks are :- – Identify the objectives of testing – Determine Scope – Determine the Test Approach – Determine the required test resources – Implement the test policy and/or the test strategy – Schedule test analysis and design tasks – Schedule test implementation, execution and evaluation – Determine the Exit Criteria • More on Test Planning in Test Management section Test Planning and Control
- 19. Fundamental Test Process Test Control • The ongoing activity of comparing actual progress against the plan • Reporting status, including deviations from the plan • Taking actions necessary to meet the mission and objectives of the project • Test Planning takes into account the feedback from monitoring and control activities. • Major Tasks are :- – measure and analyse results – Monitor and document progress, test coverage and exit criteria – initiate corrective actions – make decisions • More on Test Control in Test Management section Test Planning and Control
- 20. Fundamental Test Process • General testing objectives are transformed into tangible Test Conditions(An item or event of a component or system that could be verified by one or more test cases, e.g. a function, transaction, feature, quality attribute, or structural element ) and Test Cases(A set of input values, execution preconditions, expected results and execution Post-conditions, developed for a particular objective or test condition, such as to exercise a particular program path or to verify compliance with a specific requirement). • Tests should be designed using the test design techniques selected in the test planning activity • Major tasks are: – Review the Test Basis - in doing so evaluate testability of Test Basis and Test Object(s) – From Analysis of Test Basis and Test Items, identify and prioritize Test Conditions and associated Test Data – Test Conditions and associated Test Data are documented in a Test Design Specification – Design and prioritize the Test Cases – Identify Test Data required to support Test Cases – Design the test environment set-up – Identify any required infrastructure and tools 2. Test Analysis and Design
- 21. Fundamental Test Process • Each Test Case is specified in terms of :- – its objective – the initial state of the system – the input – the expected result. Test Analysis and Design
- 22. Fundamental Test Process • “The act of designing tests is one of the most effective error prevention mechanisms known ... The thought process that must take place to create useful tests can discover and eliminate problems at every stage of development.“ Beizer 1983 Testing Pearl of Wisdom
- 23. Fundamental Test Process • Test Cases are combined in a particular order to form Test Procedures Specifications (a.k.a Test Scripts) A document specifying a sequence of actions for the execution of a test. Also known as test script or manual test script • Test Data is created • The test environment is created • Major tasks are: Create Test Scripts and Data – Develop, implement and prioritise Test Cases – Create the Test Scripts – Create test data – Preparing test harnesses ( dedicated resources) – Write automated test scripts – Create test suites from the test cases for efficient test execution Check the Environment – Verify that the test environment has been set up correctly 3. Test Implementation and Execution
- 24. Fundamental Test Process • Major tasks (continued): Execute the Tests – Execute the Test Cases (manually or through tools) according to the planned sequence. – Log ( record ) the outcome of test execution – Test execution records should uniquely identify the versions of the software under test, test tools and Testware – It should be possible to establish that all testing has been carried out by reference to the test records. Test Implementation and Execution
- 25. Fundamental Test Process • Major tasks (continued): Check the Results – Compare actual results with expected results – Report discrepancies as Incidents – Analyse Incidents to establish Root cause – Repeat, as necessary, test activities as result of action taken for each discrepancy – The test coverage levels achieved for those measures specified as test completion criteria should be recorded. Test Implementation and Execution
- 26. Fundamental Test Process • “carefully inspect the results of each test” Myers - 2004 Testing Pearl of Wisdom Ref: Myers, The Art of Software Testing, J Wiley and Sons, 1979
- 27. Fundamental Test Process • Exit criteria ensures that the testing of the application is completed and ready. • Test execution is assessed against the objectives defined in Test Planning • This should be done for each Test Level (i.e. test stage) A group of test activities that are organized and managed together. • Major tasks are: – Check test logs against the exit criteria specified in Test Planning – If the exit criteria has not been met • Assess if more tests are needed • Assess which test activities may need to be repeated – Assess if the exit criteria specified should be changed – Produce a test summary report for stakeholders review 4. Evaluating Exit Criteria and Reporting
- 28. Fundamental Test Process 1. All the planned requirements must be met 2. All the high Priority bugs should be closed 3. All the test cases should be executed 4. If the scheduled time out is arrived 5. Test manager must sign off the release Note: All these parameters can be met by percentages( not 100 %) How to measure exit criteria ?
- 29. Fix component test plan and repeat Fix test design and repeat Fix test design and repeat Fix component or test cases/scripts and repeat 5 Phases of the Fundamental Test Process Fundamental Test Process Test Planning and Control Test Analysis and Design Test Implementation and Execution Evaluating Exit Criteria and Reporting Test Closure Activities
- 30. 3.3 Psychology of Testing • Historically testing was viewed as showing the system meets its requirements • This has evolved to a stage where testing is performed with the primary aim of finding faults rather than proving correctness. It is perceived as a destructive ( negative ) process • Seeking to find failures (the right approach) can be viewed as criticism of the product and/or its author • But looking for failures as constructive! ( useful) – Time can be saved – Risks reduced – Costs reduced – Skills improved The Testing approach
- 31. Psychology of Testing • It is important that the Objectives of testing are clearly understood as humans will moderate their behaviour accordingly (however sub-consciously )بوعي:- – “If testing is showing the system meets its requirements then I will just produce tests that show this.” – “If testing is aimed at finding faults then I will be measured on this so I will put effort into designing tests that are more likely to find faults.” • The Testing approach is different from a Developer’s The Testing approach
- 32. Psychology of Testing • “Testing is an extremely creative and intellectually ( )فكري challenging task” • “Tests must be written for invalid and unexpected, as well as valid and expected, input conditions” Myers - 1979 Testing Pearl of Wisdom
- 33. Psychology of Testing • A Tester needs: – good communication skills – good observation skills – people handling skills – Curiosity ( testing activities interest ) – patience – reliability – Thoroughness ( carefulness) – an inquisitive nature ()فضولي – attention to detail – creativity in terms of identifying likely faults – Experience • However as with most other disciplines an effective test team will need a mix of skills so it is difficult to generalise Traits of Good Testers
- 34. Psychology of Testing • The relationship between a Developer and a Tester is not normally an easy one because:- – testers point out problems with software – developers like to think their software is perfect – testers are perceived as delaying the project by finding faults in the system – when the development slips, testers normally have to work long hours to test the product, which in turn can cause anger • It is important that they work together • It is also important that they have mutual respect for each other. • Collaboration is the right approach – we work to a common goal! • Communicate findings objectively, not subjectively Developer vs Tester Relationship
- 35. Psychology of Testing • The right approach could enable Developers to test the code • However, passing this responsibility to trained and professional testing resources has many benefits (such as higher defect find rates) • Authors tend to bring across assumptions they have made when developing the software. They are less likely to write tests to show faults in their own software (human nature) • With testing performed by independent testers, testing effort is focused and not compromised by development effort and bias • It is generally believed that objective independent testing is more effective Independent testing ( (االختبار حيادية
- 36. Psychology of Testing • There are several levels of Independence (from Low to High): – Tests designed by the person(s) who wrote the software under test – Tests designed by another person(s) (e.g. from the development team). – Tests designed by a person(s) from a different organizational group (e.g. an independent test team). – Tests designed by a person(s) from a different organization or company (e.g. outsourcing to an in-house or external test specialist organisation) Independent testing
Editor's Notes
- #3: Remember these!
- #5: START – a string of Pearls! The Art of Software Testing, Second Edition by Glenford J. Myers John Wiley & Sons © 2004
- #6: Last bullet - More on Risk Based approach in topic 5
- #7: END - So clearly you can’t test everything!
- #10: The graph shows the relative cost of correcting software faults (or making other software changes) as a function of the phase in which the corrections or changes are made. The longer we leave it, the more expensive it gets! If a software requirements error is detected and corrected during the planning and requirements phase, its correction is a relatively simple matter of updating the requirements specification. If the same error is not corrected until the maintenance phase, the correction involves a much larger inventory of specifications, code, user and maintenance manuals, and training material. Further, late corrections involve a much more formal change approval and control process, and a much more extensive activity to revalidate the correction. These factors combine to make the error typically 100 times more expensive to correct in the maintenance phase on large projects than in the requirements phase. The total economic impact of leaving faults to be found after the software has become operational is actually even larger, because of the added operational costs incurred by the fault. Reference: Adapted from Barry H. Boehm, Software Engineering Economics, Prentice-Hall PTR, 1981, page 40
- #11: END I.E Don’t test evenly – focus on key areas!
- #12: END - So the message is – Don’t get complacent with your testing. Improve! Learn and Improve! This principle is pertinent to UAT – UAT shouldn’t be just a re-run of System Test cases (as is sometimes the case). Instead it should have a fresh business user perspective. Real end users too. ‘All test techniques have built-in assumptions about the nature of bugs. Every test technique targets a different set of bugs. If programmers respond to testing and bug information by reducing or eliminating such bugs, it follows that as their software improves, the effectiveness of previous tests is eroded. That is, your tests wear out and you'll have to learn, create, and use new tests based on new techniques to catch their new bugs. It's called the "pesticide paradox" after the agricultural phenomenon, where bugs such as the boll weevil build up tolerance to pesticides, leaving you with the choice of ever-more powerful pesticides followed by ever-more powerful bugs or an altogether different approach.’ Black-Box Testing: Techniques for Functional Testing of Software and Systems by Boris Beizer John Wiley & Sons © 1995
- #14: END - What do you think is meant be this? It means that unless the ultimate users of the system have been involved in the testing process at some stage then it is likely that they won’t get what they really want. E.g. you must involve end-users and operational staff in Acceptance testing and preferably earlier (UCT). The screens/web pages may seem error free but if they have been designed in a way that does not support the business process the system will be unusable.
- #15: Important – learn these! Testing is an activity performed in parallel with software development and one that consists of its own phases of planning, specifying, executing, recording and ongoing maintenance of tests. We have to build or acquire tests before we can run them and careful planning and analysis of the test objectives and requirements must be done before we design and build tests. The timing emphasis is on getting most of the test planning and specification work completed before the software is designed and coded. The starting trigger to begin test specification and design is the availability of a specification for the software under test. For example, the customer or the customer’s test agent should begin planning and specifying requirements-based acceptance tests as soon as the system’s requirements have been documented. As the system is being functionally designed at the high level the test team should begin planning and specifying design-based tests for execution during system testing. As the system is being internally designed at the low level the test team should begin planning and specifying design-based tests for execution during integration testing. Test plans and objectives for each level of testing come first, then test design, then test implementation, and, finally, we can execute and evaluate. Reference: William C. Hetzel, The Complete Guide to Software Testing, Collins, 1985, p 31, 39
- #16: END - We’ll talk about at each Stage, then show how they all fit together
- #17: Stage 1
- #19: Again more on Monitoring and Control in Section 5
- #20: Per the Syllabus (2007 version), activities are: Reviewing the test basis (such as requirements, architecture, design, interfaces). Evaluating testability of the test basis and test objects. Identifying and prioritizing test conditions based on analysis of test items, the specification, behaviour and structure. Designing and prioritizing test cases. Identifying necessary test data to support the test conditions and test cases. Designing the test environment set-up and identifying any required infrastructure and tools. 1st sub-bullet - Evaluate testability of the requirements and system – stress this as it is in Mock an people forget it! Penultimate sub-bullet: Test environment – specify logical design only, I would expect. System Architects transforms this into a physical design. END - More on Test Conditions, Cases and Procedures in Test Design Techniques (Section 4) From the Glossary: test item: The individual element to be tested. There usually is one test object and many test items. See also test object. test object: The component or system to be tested. See also test item. test design specification: A document specifying the test conditions (coverage items) for a test item, the detailed test approach and identifying the associated high level test cases
- #23: From the 2007 Syllabus, main activities are: Developing, implementing and prioritizing test cases. Developing and prioritizing test procedures, creating test data and, optionally, preparing test harnesses and writing automated test scripts. Creating test suites from the test procedures for efficient test execution. Verifying that the test environment has been set up correctly.
- #24: From the 2007 Syllabus, main activities are: Executing test procedures either manually or by using test execution tools, according to the planned sequence. Logging the outcome of test execution and recording the identities and versions of the software under test, test tools and testware.
- #25: From the 2007 Syllabus, main activities are: Comparing actual results with expected results. Reporting discrepancies as incidents and analyzing them in order to establish their cause (e.g. a defect in the code, in specified test data, in the test document, or a mistake in the way the test was executed). Repeating test activities as a result of action taken for each discrepancy. For example, re-execution of a test that previously failed in order to confirm a fix (confirmation testing), execution of a corrected test and/or execution of tests in order to ensure that defects have not been introduced in unchanged areas of the software or that defect fixing did not uncover other defects (regression testing)..
- #26: END – an important pearl. It is easy to miss errors because we assume they are not there – don’t just check for what you expect to find! Ref: The Art of Software Testing, Second Edition by Glenford J. Myers John Wiley & Sons © 2004
- #27: Stage 4 Glossary (test level): A group of test activities that are organized and managed together. A test level is linked to the responsibilities in a project. Examples of test levels are component test, integration test, system test and acceptance test. [After TMap]
- #28: START - And finally, Stage 5
- #29: START - This diagram shows the sequence of the 5 activities and the relationship between them. There are loops in the process as defects are found. During the Test Implementation stage we often have to re-execute tests following defect fixes as well as run regression tests (hence the internal loop) and may need to repair faulty tests and re-execute them (hence the loop back to the previous stage). It is during the ‘Evaluating Exit Criteria and Reporting’ stage that we may need to consider designing more tests (hence the loop back to the previous 2 stages).
- #30: And now some Psychology!
- #32: END - If you don’t believe this to be the case, then your probably not doing it right!
- #33: Maybe also perseverance?
- #35: First bullet – in theory we could train developers to have the same mindset required for testers. In practice this is risky and experience warns us otherwise Last bullet - Objective independent testing is generally believed to be more effective (remember this) but can cost more in the short term due to the knowledge transfer and communication requirements between development and testing teams. In the longer term this increased cost in failure prevention MAY be outweighed by the possible future cost of failure correction. END – This slide might suggest that Developers only develop and Testers Test – is that a good approach? No, we shouldn’t let developers completely off the hook – We require thorough Development testing effort (Component and maybe Component Integration Testing). Objective independent testing is generally believed to be more effective but can cost more in the short term due to the knowledge transfer and communication requirements between development and testing teams In the longer term this increased cost in failure prevention may be outweighed by the possible future cost of failure correction.
- #36: END – We cover this again at the start of Topic 5, under Test Organisations! Maybe show Mark’s witty slides about Testers/Developers brains??