![Popular Software Estimation Methods
• Sizing Methods
– Source Lines of Code (SLOC)
– Function Points Analysis
– Use Case Points
• Effort Estimation Methods
– Expert judgment/experience
– Productivity index
– COCOMO, SEER-SIM, SLIM models
• Software testing estimation methods
– Using a test distribution percentage
– Test Case Points Analysis (by Cognizant Technology Solutions) [2]
• Mainly based on the number of steps
6](https://image.slidesharecdn.com/vunguyen-tcpa-111212052957-phpapp02/85/Test-Case-Point-Analysis-6-320.jpg)
![Motivation
• Testing accounts for up to 50% of project effort [1]
• Current problems
– estimates are done for the whole project rather than
testing specific
– lack of reliable methods designed for estimating size and
effort of software testing
– vague definitions of testing productivity
• due to the lack of a size measure for software testing
• There are needs of
– accurately estimating effort of testing activities
– measuring size and productivity of testing activities
– measuring effectiveness and efficiency of software
testing
7](https://image.slidesharecdn.com/vunguyen-tcpa-111212052957-phpapp02/85/Test-Case-Point-Analysis-7-320.jpg)
![Tools Support by QASymphony [3]
• qTest – Quality Management Suite
– Manage requirements and test cases
– Plan testing activities and resources
– Monitor and control testing activities
– Perform measurement, analysis, and reporting
• Test case point analysis
• Effort estimation
• qTrace – defect and scenario recording tool
– Record defect
– Record user scenarios
29](https://image.slidesharecdn.com/vunguyen-tcpa-111212052957-phpapp02/85/Test-Case-Point-Analysis-29-320.jpg)
![References
• [1] Y. Yang, Q. Li, M. Li, Q. Wang, An empirical analysis on distribution patterns of software
maintenance effort, International Conference on Software Maintenance, 2008, pp. 456-459
• [2] N. Patel, M. Govindrajan, S. Maharana, S. Ramdas, “Test Case Point Analysis”, Cognizant
Technology Solutions, White Paper, 2001
• [3] QASymphony: www.qasymphony.com](https://image.slidesharecdn.com/vunguyen-tcpa-111212052957-phpapp02/85/Test-Case-Point-Analysis-33-320.jpg)
Test Case Point Analysis
- 2. Agenda • Objectives • Background and Motivation • Existing Estimation Methods • Test Case Point Analysis • Effort Estimation Methods using TCP • Conclusion and Future Work 2
- 3. Objectives • Discuss importance of software estimation, especially for testing projects • Discuss existing methods for estimating testing activities • Introduce an approach to estimating testing projects – Test Case Point Analysis – Effort Estimation using TCP 3
- 4. Agenda • Objectives • Background and Motivation • Existing Estimation Methods • Test Case Point Analysis • Effort Estimation Methods using TCP • Conclusion and Future Work 4
- 5. Software Estimation and Its Importance • Software estimation – process of determining the cost, time, staff, and other related attributes of software projects, often before work is performed • Estimation is important for the success or failure of software projects • Provides inputs for – Making investment decisions – Budget and staff allocation – Tradeoff and risk analysis – Project planning – Stakeholder negotiation – etc. 5
- 6. Popular Software Estimation Methods • Sizing Methods – Source Lines of Code (SLOC) – Function Points Analysis – Use Case Points • Effort Estimation Methods – Expert judgment/experience – Productivity index – COCOMO, SEER-SIM, SLIM models • Software testing estimation methods – Using a test distribution percentage – Test Case Points Analysis (by Cognizant Technology Solutions) [2] • Mainly based on the number of steps 6
- 7. Motivation • Testing accounts for up to 50% of project effort [1] • Current problems – estimates are done for the whole project rather than testing specific – lack of reliable methods designed for estimating size and effort of software testing – vague definitions of testing productivity • due to the lack of a size measure for software testing • There are needs of – accurately estimating effort of testing activities – measuring size and productivity of testing activities – measuring effectiveness and efficiency of software testing 7
- 8. Motivation (cont’d) • Our aim to – Defining a method for estimating the size of testing activities – Defining methods to estimate testing effort, schedule, and staff accurately using this size measure – Using this size measure as a basis for measuring • productivity • effectiveness • other testing metrics 8
- 9. Agenda • Objectives • Background and Motivation • Existing Estimation Methods • Test Case Point Analysis • Effort Estimation Methods using TCP • Conclusion and Future Work 9
- 10. Test Case Point Analysis • Principles – Size must reflect the mass and complexity of the testing project – Size should correlate with testing effort – Test case point is measured using test cases as main input – Test case complexity is based on • Checkpoints • Precondition • Test Data • Type of test case 10
- 11. Test Case Point Analysis (cont’d) • Process overview Project Attributes Test Case Estimate Estimated Test Case Test Case Points Testing Effort Point Analysis Effort 11
- 12. Test Case Point Analysis (cont’d) • TCPA estimates the size of testing projects using test cases as input • Steps 1. Measure test case complexity • Count checkpoints • Measure the complexity of precondition • Measure the complexity of data 2. Adjust Test Case Point by type of test • Consider 11 types of test case o Each type has a weight o Adjust TCP using the weight • Use functional test cases as baseline with the weight of 1.0 12
- 13. Checkpoint • Checkpoint – Is the condition in which the tester verifies whether the result produced by the target function matches the expected criterion – One test case consists of one or many checkpoints • Counting rule 1 One checkpoint is counted as one Test Case Point 13
- 14. Test Case Precondition • Precondition – specifies the condition to execute the test case – mainly affects the cost to execute the test case – may be related to data prepared for the test case • Four levels of complexity – None – Low – Medium – High 14
- 15. Test Case Precondition (cont’d) • None – The precondition is not applicable or important to execute the test case – Or, the precondition is just reused from the previous test case to continue the current test case • Low – The condition for executing the test case is available with some simple modifications required – Or, some simple set-up steps are needed • Medium – Some explicit preparation is needed to execute the test case – The condition for executing is available with some additional modifications required – Or, some additional set-up steps are needed • High – Heavy hardware and/or software configurations are needed to execute the test case 15
- 16. Test Case Precondition (cont’d) • Counting rule 2A: Counting Unadjusted Test Case Points for Precondition: Each complexity level of precondition is assigned a number of Test Case Points Based on our survey of 18 senior QA engineers 16
- 17. Test Data • Test Data – used to execute the test case – can be generated at the test case execution time, sourced from previous tests, or generated by test scripts – Test data is test case specific, or general to a group of test cases, or for the whole system • Four levels of complexity – None – Low – Medium – High 17
- 18. Test Data (cont’d) • None – No test data preparation is needed • Low – Simple test data is needed and can be created during the test case execution time – Or, the test case uses a slightly modified version of existing test data and requires little or no effort to modify the test data • Medium – Test data is deliberately prepared in advance with extra effort to ensure its completeness, comprehensiveness, and consistency • High – Test data is prepared in advance with considerable effort to ensure its completeness, comprehensiveness, and consistency – This could include using support tools to generate data and a database to store and manage test data – Scripts may be required to generate test data 18
- 19. Test Data (cont’d) • Counting rule 2B: Counting Unadjusted Test Case Points for Test Data: Each complexity level of Test Data is assigned a number of Test Case Points Based on our survey of 18 senior QA engineers 19
- 20. Adjust Test Case Point • Test Case Point counted till this point is considered Unadjusted Test Case Point (UTCP) • UTCP is adjusted by considering types of test case – Each type of test case is assigned a weight – Adjusted Test Case Point (ATCP): n ATCP = ∑UTCPi * Wi i=1 • UTCPi - the number of UTCP counted for the test case ith. • Wi - the weight of the test case ith, taking into account its test type 20
- 21. Weight by Type of Test Based on our survey of 18 senior QA engineers 21
- 22. Weight by Type of Test (cont’d) 22
- 23. Test Case Point Analysis - Summary • Process overview Count Checkpoints Determine Test Case UTCP Adjust with TCP Precondition Test Type Complexity Determine Test Data Complexity 23
- 24. Agenda • Objectives • Background and Motivation • Existing Estimation Methods • Test Case Point Analysis • Effort Estimation Methods using TCP • Conclusion and Future Work 24
- 25. Estimate Testing Effort • Estimate testing effort using TCP • Test effort distribution, four phases – Test Planning – Test Analysis and Design Each of these phases may be performed multiple times – Test Execution – Test Tracking and Reporting • Effort estimation methods – Productivity index – Regression 25
- 26. Estimate Testing Effort (cont’d) • When estimating TCP, assume that each phase is performed ONCE • Effort is estimated dependent on how many times a phase is performed 26
- 27. Productivity Index • Effort is computed using productivity index of completed project • Productivity index is measured as person-month per TCP Effort = TCP * Productivity Index Simple method 27
- 28. Regression • Estimate effort of new projects using size and effort of completed projects 100 90 Regression 80 formula 70 y = 0.072x + 1.640 Effort (PM) 60 50 40 30 20 10 0 0 100 200 300 400 500 600 700 800 900 1000 Adjusted TCP 28
- 29. Tools Support by QASymphony [3] • qTest – Quality Management Suite – Manage requirements and test cases – Plan testing activities and resources – Monitor and control testing activities – Perform measurement, analysis, and reporting • Test case point analysis • Effort estimation • qTrace – defect and scenario recording tool – Record defect – Record user scenarios 29
- 30. Conclusion and Future Work • It is important to estimate size and effort of testing projects • Test Case Point Analysis (TCPA) was proposed to estimate size. Elements include – Checkpoints – Precondition – Test Data – Test Type • Simple two methods for estimating effort were introduced – Productivity index – Regression 30
- 31. Conclusion and Future Work • Advantages of TCPA – Easy to implement – Reflecting real complexity of test cases – Independent with the number of steps • Limitations and future work – Need empirical validations – Recalibrate constants used in the method – Support only manual testing – Lack of support for performance testing 31
- 32. Thank You
- 33. References • [1] Y. Yang, Q. Li, M. Li, Q. Wang, An empirical analysis on distribution patterns of software maintenance effort, International Conference on Software Maintenance, 2008, pp. 456-459 • [2] N. Patel, M. Govindrajan, S. Maharana, S. Ramdas, “Test Case Point Analysis”, Cognizant Technology Solutions, White Paper, 2001 • [3] QASymphony: www.qasymphony.com