Metrics based software supplier selection - Best practice used in the largest Dutch telecom company
Download as PPTX, PDF0 likes500 views
The document presents a metrics-based software supplier selection best practice employed by KPN, the largest Dutch telecom company, focusing on achieving cost reduction and improved supplier accountability. Key phases include data collection from historical projects, compliance and reality assessments, and the application of various productivity and quality metrics to evaluate suppliers. The findings emphasize the importance of transparency and adherence to compliance rules during the selection process, along with recommendations for penalty points for violations.
Metrics based software supplier selection - Best practice used in the largest Dutch telecom company
- 2. Metrics Based Software Supplier Selection Best practice used in the largest Dutch telecom company Hans Kuijpers Harold van Heeringen Assisi, October 2012
- 3. Introduction Harold van Heeringen Hans Kuijpers Sizing, Estimating & Control Program Assurance & Methods Harold.van.heeringen@sogeti.nl hans.tm.kuijpers@kpn,.com @haroldveendam @_hanskuijpers @Sogeti_SEC Sogeti Nederland BV: KPN Nederland: Senior Consultant Software Metrics Manager Metrics Desk ISBSG: President Certified Scope Manager NESMA: Board Member QSM: Special Interest Group Agile NESMA: Working Group Chair COSMIC NESMA: Working Group Chair Benchmarking COSMIC: IAC Member Metrics Based Software Supplier Selection Pagina 2
- 4. Agenda • Introduction and Context • Phases and Timeline • The Model • Results • Conclusions & Recommendations Metrics Based Software Supplier Selection -3-
- 5. Introduction and Context Revenues: €13.000m .Why supplier selection? KPN Board EBITDA: € 5.100m Employees: 31.000 • Consolidation # suppliers • Cost reduction Consumer Business Corporate KPN • Supplier acts as SI & MSP Market Market Market NetCo E-Plus Belgium • 5 Year investment • SPM is a best practice at KPN Customer Fixed Mobile Wholesale IT Operations Experience Generic & OSS Domains BSS Traditional Problem: no more competition between suppliers instrument needed to avoid excessive cost Unit of Work pricing Metrics Based Software Supplier Selection -4-
- 6. Agenda • Introduction and Context • Phases and Timeline • The Model • Results • Conclusions & Recommendations Metrics Based Software Supplier Selection -5-
- 7. Phases and Timeline Why Productivity metric added? • Objective selection criteria • Supplier willingness to show their transparency • Basis for productivity baseline • Insight in quality level • Negotiations for year on year cost reduction • Relation to continous improvement steps Metrics Based Software Supplier Selection -6-
- 8. Requested project information • Data of 6 historical projects, max 3 KPN projects • In scope of current technology domain • Range 300 – 1000 FP • Sizing method NESMA 2.1 or IFPUG 4.x • DCF must be completely filled out • No other template is allowed In BAFO phase suppliers should show evidence of the size and productivity figures by releasing FPA-reports, Data Collection Forms and/or insight in their administrative systems. Metrics Based Software Supplier Selection -7-
- 9. Historical Project Data form (1) Metrics Based Software Supplier Selection -8-
- 10. Historical Project Data form (2) Per data field requirements are mentioned in the template. Metrics Based Software Supplier Selection -9-
- 11. Agenda • Introduction and Context • Phases and Timeline • The Model • Results • Conclusions & Recommendations Metrics Based Software Supplier Selection - 10 -
- 12. The Model Characteristics: • Degree of openess and compliancy • Completeness and cohesion of submitted data • Productivity benchmark against each other and industry • Delivered Quality • During the RFP phase the data will be considered as correct, but will be checked on reality The 3 test criteria: A. Compliancy value (10%) B. Reality value (30%) C. Productivity - Quality value (60%) Metrics Based Software Supplier Selection - 11 -
- 13. Used metrics and benchmarks Project Delivery Rate (PDR) = spent project effort related to function point (h/FP) Productivity Index (PI) = metric from QSM, derived from size, duration and effort Quality: delivered defects per FP Benchmarks: • PI against the QSM Business trend line • PDR against ISBSG Repository • Adjusted = normalised to Construction+Test activities Metrics Based Software Supplier Selection - 12 -
- 14. Compliancy value (10%) Suppliers start with 10 points The compliancy value is substracted with 2 points for each violation of rule: a)Range 300 – 1000 Function Points b)Method NESMA 2.1 or IFPUG 4.x c) Each field of “Historical Project Data”-form must be filled out Maximum value = 10, minimum value = 0 Metrics Based Software Supplier Selection - 13 -
- 15. Reality value (30%) PI vs. Functional size (FP) 35 Unrealistic projects are discarded Unrealistic from further analysis: 30 • PI > +2 sigma (95%) 25 • PDR < P25 ISBSG (best in class 20 projects) PI 15 The reality value is substracted with 10 Supplier A Supplier B 2 points for each unrealistic project 5 Supplier C Supplier D Supplier E QSM Business Maximum value = 10 0 Avg. Line Style 2 Sigma Line Style 50 150 250 350 450 550 650 750 850 950 Minimum value = 0 Effective FP Functional Size (FP) Metrics Based Software Supplier Selection - 14 -
- 16. Productivity - Quality value (60%) Productivity - Quality value = (Points PI score * 0,5) + (Points PDR score * 0,3) + (Points Quality score * 0,2) ID PDR (h/FP) PDR ISBSG median PDR score ID Defects/FP Quality score 7 5,9 8,6 -2,7 15 41,7 8 6,0 8,6 -2,6 18 13,9 9 6,9 8,6 -1,7 21 66,7 11 6,2 8,6 -2,4 22 4,0 12 7,3 8,6 -1,3 23 10,0 Average: -2,1 Median 13,9 The lowest average most points The lowest median most points For PI and PDR the average of the distance to the benchmark value is determined For the quality the median is dertermined The highest average most points Metrics Based Software Supplier Selection - 15 -
- 17. Agenda • Introduction and Context • Phases and Timeline • The Model • Results • Conclusions & Recommendations Metrics Based Software Supplier Selection - 16 -
- 18. Results of Compliancy (1) Projects discarded: • Projects on going (4) • Project sized in COSMIC (1) Result: 1 supplier has 3 violations, the others 5 or more Blank crucial fields • Defect data Supplier Compliancy Value • Effort data Supplier A 0 • Dates Supplier B 0 Supplier C 4 Other violations Supplier D 0 • Primary Language (example English) Supplier E 0 Metrics Based Software Supplier Selection - 17 -
- 19. Results of Compliancy (2) Metrics Based Software Supplier Selection - 18 -
- 20. Results of Reality Projects unrealistic: • 3 according to PI • 1 according to PDR Unrealistic Unrealistic Discarded for further analysis projects PI projects PDR Supplier criterion criterion Reality Value Supplier A 1 1 6 Supplier B 1 0 8 Supplier C 0 0 10 Supplier D 0 0 10 Supplier E 1 0 8 Metrics Based Software Supplier Selection - 19 -
- 21. Results of Productivity / Quality Rank Points PDR Rank Points Supplier PI score PI score PI score Supplier score PDR score PDR score Supplier A 3,9 2 8 Supplier A -3,2 1 10 Supplier B 5,0 1 10 + Supplier B -2,1 2 8 + Supplier C 3,4 3 6 Supplier C 16,6 4 4 Supplier D 3,0 5 2 Supplier D 6,2 3 6 Supplier E 3,2 4 4 Supplier E 18,3 5 2 Quality Rank Points Points Points Points Productivity/ Supplier Score Quality score Quality score Supplier PI score PDR score Quality score Quality value Supplier A 3,1 1 10 Supplier A 8 10 10 9,0 Supplier B 13,9 2 8 Supplier B 10 8 8 9,0 Supplier C 52,6 3 6 = Supplier C 6 4 6 5,4 Supplier D 1000,0 5 2 Supplier D 2 6 2 3,2 Supplier E 94,6 4 4 Supplier E 4 2 4 3,4 weight 50% 30% 20% Metrics Based Software Supplier Selection - 20 -
- 22. Results of Total Assessment Recommendation from Metrics Desk: Supplier B and A score best in the model Compliancy Reality Productivity/ Total Supplier value value Quality value Points Rank Supplier A 0 6 9,0 7,2 2 Supplier B 0 8 9,0 7,8 1 Supplier C 4 10 5,4 6,6 3 Supplier D 0 10 3,2 4,9 4 Supplier E 0 8 3,4 4,4 5 weight 10% 30% 60% However suppliers B and C were selected for the next BAFO phase Metrics Based Software Supplier Selection - 21 -
- 23. Findings BAFO phase Metrics Desk investigated the provided project data of the selected suppliers B + C: • Size • Dates • Hours • Defects Supplier B: • Resistance: confidentiality clause with clients • Client site visit Supplier C: • Size measurement by junior not certified measurers Metrics Based Software Supplier Selection - 22 -
- 24. Agenda • Introduction and Context • Phases and Timeline • The Model • Results • Conclusions & Recommendations Metrics Based Software Supplier Selection - 23 -
- 25. Conclusions and recommendations • The productivity assessment influenced the total outcome significantly • The assessment and discussions afterwards gave insight in: Conclusions: • Transparency and CMMI level • The results are used in the negotiations phase to maximize the baseline value • Make sure the parties understand: • the purpose of the assessment • the use of the “Historical Project Data” form • that the disclosed data will be validated and should not be confidental • the consequences of violating the governance rules (e.g. penalty points) Recommendations: • Because of many violations of the compliancy rules, consider 1 penalty point per violation • Construct model beforehand, but don’t communicate the model with suppliers • Bring site visits when offered. This gives extra information next to the productivity validation Metrics Based Software Supplier Selection - 24 -
- 26. Productivity: don’t trust it, check it Harold van Heeringen Hans Kuijpers Sizing, Estimating & Control Software Metrics Consultant Harold.van.heeringen@sogeti.nl hans.tm.kuijpers@kpn,.com @haroldveendam @_hanskuijpers @Sogeti_SEC Metrics Based Software Supplier Selection - 25 -
- 27. Back-up sheets Metrics Based Software Supplier Selection - 26 -
- 28. Productivity index (PI) 500 FP Effort 17 h/FP Productivity Index=18 14 h/FP Productivity Index=16 10 h/FP 9 h/FP 7 h/FP Duration - 27 -