Simple is Not Necessarily Better: Why Software Productivity Factors Can Lead to Bad Estimates
1 like567 views
The document discusses the risks associated with using simple software productivity factors for estimating software development costs, highlighting that this method can lead to significant estimation errors. Key risks include linear extrapolation neglecting diseconomies of scale, ignoring system effects, and flawed adjustment factors. The authors propose improvements through parametric estimating relationships and incorporating cumulative reuse in size estimates to mitigate these issues.
Simple is Not Necessarily Better: Why Software Productivity Factors Can Lead to Bad Estimates
- 1. Presented at 2007 ISPA-SCEA Joint Conference Technomics 201 12th Street South Suite 612, West Tower Arlington, VA 22202 Voice (703) 412-0602 5290 Overpass Road Suite 206 Santa Barbara, CA 93111 Voice (805) 964-9894 Simple is Not Necessarily Better: Why Software Productivity Factors Can Lead to Bad Estimates 2007 ISPA-SCEA Joint Conference New Orleans, LA Simple is Not Necessarily Better: Why Software Productivity Factors Can Lead to Bad Estimates 2007 ISPA-SCEA Joint Conference New Orleans, LA Michael A. Gallo Paul Hardin
- 2. Presented at 2007 ISPA-SCEA Joint Conference Slide 1Technomics The problem…The problem… The use of simple productivity factors to estimate software development cost injects unnecessary risk into the program. SIZE ESTIMATE (ESLOC) SIZE ESTIMATE (ESLOC) X PRODUCTIVITY FACTOR (HRS/ESLOC) PRODUCTIVITY FACTOR (HRS/ESLOC) = ESTIMATED EFFORT (HRS) ESTIMATED EFFORT (HRS) SOFTWARE COST ESTIMATING USING THE PRODUCTIVITY METHOD • ESLOC (Equivalent New Source Lines of Code) is a weighted sum of New, Modified, Reused, etc. The weights used for modified and reused, etc are typically less than 1 which implies that the cost of this code is less than the cost of new code. • In almost all cases, ESLOC ≤ Delivered SLOC (DSLOC) • Based on either an analogy to a similar completed project development or based on an average of productivities of several analogous projects • Frequently reflects only ‘core’ software development activities (Design, Code, Unit Test) • Method for computing ESLOC may differ from method used to compute estimated ESLOC • Estimated effort will reflect the set of activities included in the productivity factor • Additional activities (e.g. Requirements, System Integration & Test, etc) are estimated as a factor of the ‘core’ software development estimate
- 3. Presented at 2007 ISPA-SCEA Joint Conference Slide 2Technomics Risks Associated With Productivity FactorsRisks Associated With Productivity Factors 1. Linear extrapolation fails to account for diseconomies of scale 2. Error can be exacerbated when the estimate is treated discretely rather than as a whole system 3. May neglect to properly count ESLOC for incremental developments 4. Leads to erroneous use of adjustment factors to account for missing software development activities
- 4. Presented at 2007 ISPA-SCEA Joint Conference Slide 3Technomics Effort Size New SW Project The potential for significant estimating error exists when a software estimate is built based upon the productivity of an analogous software development project. 1. A productivity based approach starts with an analogous project that (hopefully) is completed. The analyst derives a productivity by taking the ratio of actual effort incurred to size of the software developed. Size is usually a weighted sum of new, modified, and reused software, for example, ESLOC. Productivity = Total Effort/ESLOC 2. The estimate of the new software development project uses the productivity derived from the analogous project in #1. This is a linear extrapolation (represented by the blue dashed line) from the analogous project. New SW Project Effort = New SW Project ESLOC x (Productivity) Completed Project Our “Analogy” 3. Technomics research shows a continuation of diseconomy of scale for weapon system software development. That means the effort to develop software increases non‐linearly as the size of the software grows. New SW Project Effort = k* New SW Project ESLOC Z 4. As the size of the new project continues to move away from the size of the analogous program, the non‐ linear estimate will continue to diverge from the productivity based estimate. The gap between the two estimates is the potential error that exists in the estimate by not accounting for diseconomy of scale. Risk #1: Linear ExtrapolationRisk #1: Linear Extrapolation Diseconomies of scale remains prevalent in DoD software projects – Effort = a*Sizeb; b>1 Productivity methods use a linear relationship – Effort = a * Size1 Impact: a potentially large underestimation of effort when size of project is substantially larger than its analogy Simple productivity factors fail to reflect diseconomies of scale Potential Error Arising from the Use of a Linearly Extrapolated Productivity Analogy to Estimate SW Development Effort 1.0X 1.1X 1.2X 1.3X 1.4X 1.5X 1.6X 1.7X 1X 4X 7X 10X Ratio (New Project SIZE / Analagous Project SIZE) Ratio(Non-LinearEstimate/LinearEstimate) b = 1.20 b = 1.15 b = 1.10 b = 1.05 b = 1.02 Trend in DoD Weapon System Software Size Size measured in Source Lines of Code 1.4M0.80M0.40M0.24M 34M 16M 13M 2.8M 1.7M F-14D F/A-18 A/B F/A-18 C/D C-17 F-22 F/A-18 E/F Joint Strike Fighter DD(X) Future Combat System Sources: General Accounting Office Defense Contract Management Agency Software Productivity Consortium F‐22 Program Website
- 5. Presented at 2007 ISPA-SCEA Joint Conference Slide 4Technomics Risk #2: Ignoring System EffectsRisk #2: Ignoring System Effects The whole (i.e. system) is greater than the sum of its parts (i.e. components and sub-systems) Existing DoD databases have perpetuated this problem – Systems are broken up and sanitized to protect proprietary data – Little if any insight into the number of development organizations by system – Few (if any) total system data points Hrs Size Diseconomies of scale trend Diseconomies of scale trend ErrorError Discrete Component Size Total System Size Σ Discrete Effort Estimates Without System Effect System Effort Estimate Discrete Effort Estimates with System Effect Discrete Effort Estimates Without System Effect Current DoD data and analyses miss the ‘big picture’ Tier 1Tier 1 SubSub--ContractorContractor Tier 2Tier 2 SubSub--ContractorContractor DisplayDisplay SystemSystem ContractorContractor EWEW EWEW System of SystemsSystem of Systems ContractorContractor IntegratedIntegrated AvionicsAvionics DisplayDisplay 107 - 106 106 - 105 105 - 104 105 - 103 Increasing Product Size (SLOC) Increasing Level of Integration Prevalence of Data Collected
- 6. Presented at 2007 ISPA-SCEA Joint Conference Slide 5Technomics Risk #3: Incremental DevelopmentRisk #3: Incremental Development Incremental development looks suspiciously like pre- planned product improvement – Successive deliveries of software are built upon pre-existing base of software – Cost to build, integrate and test the latest product build is a function of the product’s cumulative size Estimates should incorporate pre- existing base in the ESLOC computation Incremental Development 1 1 2 1 2 3 1 2 3 4 Increments DeliveredSize Increment 1 Increment 2 Increment 3 Increment 4 ESLOC computations must include cumulative reuse Pre-Planned Product Improvement 1 1 2 1 2 3 1 2 3 4 Production Versions DeliveredSize Version 1.0 Version 2.0 Version 3.0 Version 4.0
- 7. Presented at 2007 ISPA-SCEA Joint Conference Slide 6Technomics Risk #4: Flawed Adjustment FactorsRisk #4: Flawed Adjustment Factors Developer’s definition of software effort may not align with cost analyst’s standardized definition of effort It is common practice to apply simple (fixed) factors to add or remove software development activities However, distribution of activities changes as the size of the software changes –Result: too much (or too little) addition or removal of effort Application of fixed factors increases estimating error System Reqt's System Reqt's System Reqt's System Reqt's System Reqt's System Reqt's System Reqt's System Design System Design System Design System Design System Design System Design System Design SW Reqt'sSW Reqt'sSW Reqt'sSW Reqt'sSW Reqt'sSW Reqt'sSW Reqt's Prototyping Prototyping Prototyping Prototyping Prototyping Prototyping Prototyping Prelim Des Prelim Des Prelim Des Prelim Des Prelim Des Prelim Des Prelim Des Detailed Design Detailed Design Detailed Design Detailed Design Detailed Design Detailed Design Detailed Design C&UT C&UT C&UT C&UT C&UT C&UTC&UT SW I&T SW I&T SW I&T SW I&T SW I&T SW I&T SW I&T System I&T System I&T System I&T System I&T System I&T System I&T System I&T Certification Certification Certification Certification Certification Certification Certification CM CM CM CM CM CM CM QA QA QA QA QA QA QA SW PM SW PM SW PM SW PM SW PM SW PM SW PM Data Data Data Data Data Data Data 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 10,000 50,000 100,000 500,000 1,000,000 5,000,000 10,000,000 Equivalent New SLOC %ofTotalEffort Standardized Software Cost Estimate Contractor A Standard SW Development Activities X Contractor B Standard SW Development Activities Activity to remove from the SW cost estimate Activity to add to the SW cost estimate
- 8. Presented at 2007 ISPA-SCEA Joint Conference Slide 7Technomics How to Mitigate These RisksHow to Mitigate These Risks Use parametric estimating relationships to add/remove activities Flawed Adjustment Factors Include ‘Carryover’ in ESLOC computation Incremental Sizing Derive equations at the system level; specify equations below system level System Effect Use proper parametric estimating relationships Linear Extrapolation Our ApproachProblem A new Technomics model (VERA) implements these approaches
- 9. Presented at 2007 ISPA-SCEA Joint Conference Slide 8Technomics Final ThoughtFinal Thought Source: “Engineering of Systems for Navy Interoperability”, Version 5.8 dated April 2003 by Eric Honour, Honourcode, Inc. http://jax.acqconf.com/Presentations/Power%20Points/SE‐Interop.pdf
- 10. Presented at 2007 ISPA-SCEA Joint Conference Slide 9Technomics Technomics POCsTechnomics POCs Mike Gallo, Senior Cost Analyst Phone (571) 366-1405 E-mail: mgallo@technomics.net Paul Hardin, Senior Cost Analyst Phone: (571) 366-1407 E-mail: phardin@technomics.net Fax: (703) 412-0600 Web Site: www.technomics.net
- 12. K Size Effort Graph of Effort = K * SIZEb b>1, Effort increasing at increasing rate 0 < b < 1, Effort increasing at decreasing rate b = 1, Effort increasing at constant rate ‐1 < b < 0, Effort decreasing at decreasing rate b <‐ 1, Effort decreasing at increasing rate Hyperlink from Slide 2
- 13. Effort Size New SW Project The potential for significant estimating error exists when a software estimate is built based upon the productivity of an analogous software development project. 1. A productivity based approach starts with an analogous project that (hopefully) is completed. The analyst derives a productivity by taking the ratio of actual effort incurred to size of the software developed. Size is usually a weighted sum of new, modified, and reused software, for example, ESLOC. Productivity = Total Effort/ESLOC 2. The estimate of the new software development project uses the productivity derived from the analogous project in #1. This is a linear extrapolation (represented by the blue dashed line) from the analogous project. New SW Project Effort = New SW Project ESLOC x (Productivity) Completed Project Our “Analogy” 3. Technomics research shows a continuation of diseconomy of scale for weapon system software development. That means the effort to develop software increases non‐linearly as the size of the software grows. New SW Project Effort = k* New SW Project ESLOC Z 4. As the size of the new project continues to move away from the size of the analogous program, the non‐ linear estimate will continue to diverge from the productivity based estimate. The gap between the two estimates is the potential error that exists in the estimate by not accounting for diseconomy of scale. Hyperlink from Slide 3
- 14. Hyperlink from Slide 3 Potential Error Arising from the Use of a Linearly Extrapolated Productivity Analogy to Estimate SW Development Effort 1.0X 1.1X 1.2X 1.3X 1.4X 1.5X 1.6X 1.7X 1X 4X 7X 10X Ratio (New Project SIZE / Analagous Project SIZE) Ratio(Non-LinearEstimate/LinearEstimate) b = 1.20 b = 1.15 b = 1.10 b = 1.05 b = 1.02
- 15. Hyperlink from Slide 3 Trend in DoD Weapon System Software Size Size measured in Source Lines of Code 1.4M0.80M0.40M0.24M 34M 16M 13M 2.8M 1.7M F-14D F/A-18 A/B F/A-18 C/D C-17 F-22 F/A-18 E/F Joint Strike Fighter DD(X) Future Combat System Sources: General Accounting Office Defense Contract Management Agency Software Productivity Consortium F‐22 Program Website
- 16. Hrs Size Diseconomies of scale trend Diseconomies of scale trend ErrorError Discrete Component Size Total System Size Σ Discrete Effort Estimates Without System Effect System Effort Estimate Discrete Effort Estimates with System Effect Discrete Effort Estimates Without System Effect Hyperlink from Slide 4
- 17. Tier 1Tier 1 SubSub--ContractorContractor Tier 2Tier 2 SubSub--ContractorContractor DisplayDisplay SystemSystem ContractorContractor EWEW EWEW System of SystemsSystem of Systems ContractorContractor IntegratedIntegrated AvionicsAvionics DisplayDisplay 107 - 106 106 - 105 105 - 104 105 - 103 Increasing Product Size (SLOC) Increasing Level of Integration Prevalence of Data Collected Hyperlink from Slide 4
- 18. Pre-Planned Product Improvement 1 1 2 1 2 3 1 2 3 4 Production Versions DeliveredSize Version 1.0 Version 2.0 Version 3.0 Version 4.0
- 20. Hyperlink from Slide 6 Standardized Software Cost Estimate Contractor A Standard SW Development Activities X Contractor B Standard SW Development Activities Activity to remove from the SW cost estimate Activity to add to the SW cost estimate
- 21. System Reqt's System Reqt's System Reqt's System Reqt's System Reqt's System Reqt's System Reqt's System Design System Design System Design System Design System Design System Design System Design SW Reqt'sSW Reqt'sSW Reqt'sSW Reqt'sSW Reqt'sSW Reqt'sSW Reqt's Prototyping Prototyping Prototyping Prototyping Prototyping Prototyping Prototyping Prelim Des Prelim Des Prelim Des Prelim Des Prelim Des Prelim Des Prelim Des Detailed Design Detailed Design Detailed Design Detailed Design Detailed Design Detailed Design Detailed Design C&UT C&UT C&UT C&UT C&UT C&UTC&UT SW I&T SW I&T SW I&T SW I&T SW I&T SW I&T SW I&T System I&T System I&T System I&T System I&T System I&T System I&T System I&T Certification Certification Certification Certification Certification Certification Certification CM CM CM CM CM CM CM QA QA QA QA QA QA QA SW PM SW PM SW PM SW PM SW PM SW PM SW PM Data Data Data Data Data Data Data 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 10,000 50,000 100,000 500,000 1,000,000 5,000,000 10,000,000 Equivalent New SLOC %ofTotalEffort Hyperlink from Slide 6