Benchmarking As a Tool for Optimising Software Development Performance
1 like403 views
QuantiMetrics provides benchmarking services to help clients improve software development performance. Their approach involves collecting project data from clients and comparing key metrics like costs, productivity and quality to industry benchmarks. This identifies areas for improvement. Case studies showed benchmarking helped a telecom company reduce cycle times with minimal productivity loss, and an insurance company lower costs by 54% while improving quality. Critical success factors include executive sponsorship, process maturity, collecting comparable data, ensuring confidentiality and taking a holistic view of interrelated factors.
Benchmarking As a Tool for Optimising Software Development Performance
- 1. Measure. Optimize. Deliver. Phone +1.610.644.2856 Measuring Success Benchmarking as a Tool for Optimising Software Development Performance Bram Meyerson QuantiMetrics, a DCG Partner
- 2. ©2013 David Consulting Group and QuantiMetrics QuantiMetrics • A two decade track record as an independent trusted advisor • Established in 1992 and remained independent • Acquired Research Services Benchmarking (ex: Butler Cox) from CSC in 2000 • Owns the largest validated and calibrated benchmark database covering project and application data that spans thousands of projects from hundreds of organisations • Retained by world-class organisations and consultancies to help them evaluate suppliers and to improve their in-house efficiencies • Provides client-centric service … underpinned by trust, confidentiality and teamwork • Offices in London, Wiesbaden, Johannesburg 2
- 3. ©2013 David Consulting Group and QuantiMetrics Proven Capabilities and a Focus on Ongoing Innovation • Fact-based IT executive advisory and benchmark provider • Optimisation of software development and support processes • Due diligence and professional review of portfolio, programs and project budgets and plans • Software application sizing (in-house development, outsourced applications and vendor-supplied packages) • Evaluation of 3rd party software development services • Commercial software delivery and support dispute advisory 3
- 4. ©2013 David Consulting Group and QuantiMetrics Agenda • Why benchmark? • Our unique approach to benchmarking • Examples of how benchmarking has underpinned performance improvement • Critical success factors for benchmarking 4
- 5. ©2013 David Consulting Group and QuantiMetrics Why benchmark? • Provides an objective 3rd party scorecard • Provides the cross-industry breadth perspective. Most IT executives have a depth of MIS about their project environment • Acts as a catalyst for change by setting realistic improvement targets • Provides an accurate mechanism to calibrate project estimation models. • Focuses attention on measurement and improvement 5
- 6. ©2013 David Consulting Group and QuantiMetrics 6 Our Unique Approach to Benchmarking: Process Applications Development Project portfolio Proposed or disputed Project Standard project data (current) QPeP Benchmark Report Calibrated client baseline QuantiMetrics Validated Project Database QEsT Risk Report High Moderate Acceptable <-------------- Estimation Scenarios --------------à Efficiency Staffing IndexCost per FTE Unit cost Elapsed months All testing errors Operational errors Schedule conformance Budget conformance Client Minimum Maximum Median Top quartile Top decile IT Development – Summary Performance Efficiency Staffing IndexCost per FTE Unit cost Elapsed months All testing errors Operational errors Schedule conformance Budget conformance Client Minimum Maximum Median Top quartile Top decile IT Development – Summary Performance Identifies good practices and successes – but also highlights inefficiencies and problems in methodology, staffing, planning, productivity, cost or capability across sizes, types & technologies – and proposes improvement actions Identifies probability of successful delivery against time, budget & quality targets – and proposes alternative scenarios with higher likelihood Project characteristics
- 7. ©2013 David Consulting Group and QuantiMetrics • By project, programme, group of projects and organisation • The impact of size on project performance (finding the “sweet spot”) • Project type • Development technologies • The impact of time pressure on project performance (finding the ideal time-box) • Industry sector 7 Our Unique Approach to Benchmarking: Key Focus on Normalization
- 8. ©2013 David Consulting Group and QuantiMetrics 8 Base Data • Classification data • Size data • Resource data • Testing errors • Operational errors • Qualitative data Outcomes • Productivity • Duration • Unit Cost • Test Errors • Reliability • Time Slip • Effort Overrun Drivers • Efficiency • Staffing Style • Technical Quality • Planning Style Context • Technology • Cost • Project Size Raw Metrics Derived Metrics Influence Assist in interpreting Our Unique Approach to Benchmarking: Appropriate Level of Metrics
- 9. ©2013 David Consulting Group and QuantiMetrics Good performance for Reliability & Effort Overrun; poorest performance in Productivity, Unit Cost, & Test Errors. The more the blue- lined (Customer) footprint approaches the outer edge of the polygon, the better the performance; the closer to the origin (centre), the worse the performance Note: Project sensitive cost benchmarks are UK- based and blended where necessary to take account of use of external and off- shore resources Productivity Duration Unit cost Test errorsReliability Time slip Effort overrun Minimum Maximum Median Top quartile PEP member Bottom quartile OUTCOMES Our Unique Approach to Benchmarking: Graphical Reporting
- 10. ©2013 David Consulting Group and QuantiMetrics Excellent budget conformance, weak underlying efficiency and typical staffing levels. Efficiency Staffing QualityPlan conformance Budget conformance Minimum Maximum Median Top quartile PEP member Bottom quartile DRIVERS The more the blue- lined (Customer) footprint approaches the outer edge of the polygon the better the performance, the closer to the origin (centre) the worse the performance Our Unique Approach to Benchmarking: Graphical Reporting
- 11. ©2013 David Consulting Group and QuantiMetrics Technology gearing is typical, but projects are much smaller than typical, and staff costs close to the top 25%. Gearing examines the power of the toolset / languages (i.e. function points per 1,000 lines of code) Gearing FTE costSize Minimum Maximum Median Top quartile PEP member Bottom quartile CONTEXT These costs are fully loaded. Benchmark rates quoted here are global, dominantly based on internal staff plus increasing contribution of off-shore resourcing The more the blue- lined (Customer) footprint approaches the outer edge of the polygon, the better the performance; the closer to the origin (centre), the worse the performance Our Unique Approach to Benchmarking: Graphical Reporting
- 12. ©2013 David Consulting Group and QuantiMetrics Cost-driven approach – high use of low-cost, off-shore suppliers whose productivity is low; time-driven characteristic largely accounts for this, but also contributes to high efficiency … productivity and efficiency are both better than previous Staffing Style Efficiency Current projects Previous projects Current Agile Previous Agile Current not-Agile Lean Time driven UnproductiveStretched H H L LStaff cost Productivity Current projects Previous projects Current Agile Previous Agile Current not-Agile Economic Skill driven Cost driven Uneconomic H H L L Our Unique Approach to Benchmarking: Trade-offs
- 13. ©2013 David Consulting Group and QuantiMetrics Agenda • Why benchmark? • Our unique approach to benchmarking • Examples of how benchmarking has underpinned performance improvement • Critical success factors for benchmarking 13
- 14. ©2013 David Consulting Group and QuantiMetrics Case Study 1: Large Telco • Leading communications company, serving the needs of customers in almost 200 countries • Mandated to become more cost effective and embarked on an aggressive performance transformation initiative. The company participated in benchmarking to underpin their transformation journey • Summary of achievements: – Benchmarking reports have been used as a catalyst to speed up, sharpen costs and improve reliability of software delivery – Benchmarking deliverables identified the impact of the adoption of various Agile practices and assisted in identifying which were/were not making a difference – Dramatically shortened cycle times, with small effort tradeoffs – Quality of software products also improved 14
- 15. ©2013 David Consulting Group and QuantiMetrics Case Study 1: Large Telco Relative to equivalent benchmarks, cycle time reduced dramatically, offset by small productivity loss 15 Achieved by: • Increasing adoption of time boxing (and other elements of Agile practice) • Increasing proportion of outsourcing to a small set of suppliers • The decision to go for speed was more of a concern than productivity, as simultaneous use of off- shore cheaper resources would more than compensate. • The client could not continue to do things the same old way; to remain competitive in an era of deregulation required being smart Relativetobenchmark Time
- 16. ©2013 David Consulting Group and QuantiMetrics • Leading Life, Pensions and Investment company • Non measurable achievements: benchmarking reports have been used as a catalyst to speed up, sharpen costs and improve reliability of software delivery. • Summary of achievements: – Improved quality of software products – 54% reduction in project costs ($/fp) – More disciplined project management – More accurate schedule and cost estimation – Better technical quality (fewer errors) Case Study 2: Large Insurance Company
- 17. ©2013 David Consulting Group and QuantiMetrics Case Study 2: Large Insurance Company Significant productivity and quality improvements 17 0% 2% 4% 6% 8% 10% 12% 14% 16% 18% Proportionofmembers Function points perstaff-month IQR Year 3Year 1 Inter-quartile range All testing errors Earlyoperationerrors Current proje Previous proj Current Agile Previous Agil Current not-A Ineffective testing Poor quality approach Good quality approach Effective testing H H L L
- 18. ©2013 David Consulting Group and QuantiMetrics Technical Quality Efficiency Previous projects Current projects Current Agile Previous Agile Current not-Agile Quality deferred Quality led Quality laggedQuality ignorant H H L L Focus on Process Leads to Getting it Right the First Time 18
- 19. ©2013 David Consulting Group and QuantiMetrics Critical Success Factors • Sponsorship • Maturity • Comparability • Confidentiality • Taking a systemic view 19
- 20. ©2013 David Consulting Group and QuantiMetrics High Productivity (fp/sm) Low resource cost ($/FTE) Low Unit cost (R/fp) Effective Testing (reducing Fault rework) High technology gearing (Kloc/fp) Process Efficiency Quality Process (approach) High requirements churn Right Staffing (approach to Staffing levels) Planning Efficiency Effective requirements management Higher skill levels Right project size (fp) Excessive business (time) demands Late business scope creep High speed Lean Processes Reuse “Efficeincy over Utelisation” A Systems-thinking View on Project Performance
- 21. ©2013 David Consulting Group and QuantiMetrics Contact Us Alan Cameron, DCG-SMS, Europe Email: a.cameron@dcg-sms.com Phone: +44 843 2895174 Mobile: +44 7825 271344 http://www.dcg-sms.com 21 @DavidConsultGrp /DavidConsultGrp /company/David-Consulting-Group Mike Harris, DCG, United States Email: m.harris@davidconsultinggroup.com Phone: +1 (610) 644-2856 Mobile: +1 (484) 753-3935 http://www.davidconsultinggroup.com Measure. Optimize. Deliver. Phone +1.610.644.2856