James.taylor
- 1. Getting What you Paid for: Acquisition Risk Analysis James Taylor Omnia Paratus Corporation Used with Permission 0
- 2. Lamborghini Reventon $1.6M Kia Rio $12,145 Omnia Paratus Corporation 1
- 3. A recent poll showed that 68% of companies experienced financial losses directly related to Supply Chain disruptions  Most of the financial impacts were related to supplier performance that did not meet demand requirements, and delayed, damaged or misdirected shipments.  The majority of companies polled are in the early stages or have yet to think about integrating Risk Management into their Supply Chain  Supplier related risks are most often identified after contract award is a program issue, not a risk: – Supplier shipment delay – Supplier capacity exceeded – Unable to meet technical requirements – 1st Article or Flight Test failures – Parts damaged during shipment or rejected during quality inspection  Schedule delays caused by supplier performance can have an equal or greater financial impact to a program’s bottom line, but are often overshadowed budget impacts due to supplier cost overruns Omnia Paratus Corporation 2
- 4. Procurement analysis exists in varying degrees, although the most commonly used practice is Best Value Analysis (BVA) Supplier w/ Existing New Incumbent Capabilities Supplier Score based on savings to program ~$300K Technical Score based on: - New Capability to Supplier - Technical deviations req’d based on proposal - 1st Article & Flight Req’d based on Customer reqt’s  BVA considers not only cost, but other quantifiable and non-quantifiable factors supporting an investment decision – Utilizes weighting scales for analyzing “True” program value of supplier bids – Can include, but is not limited to, performance, producibility, reliability, maintainability, and supportability enhancements – Intended to select the source offering the greatest overall benefit in response to the requirement Omnia Paratus Corporation 3
- 5. While a BVA attempts to provide insight to the least risky procurement, it does not specifically address or integrate impacts of risk Design changes resulting from Technical deviations resulted in Technical Deviations cost growth to original required by Supplier purchase order Impact from production delays of 4 weeks Program Award fee lost resulting in Supplier delays cause IMS Orders placed were milestones to be missed against original design and fulfilled by Incumbent suppler Fees required to get Incumbent supplier operations back up and $390K overrun was a direct minimize delay of loss of company profit deliveries to Customer  Performing a sample analysis of a new supplier shows the potential risks that may be incurred when basing decisions on costs alone – Technical requirements outside current capabilities – Schedule impact due from potential delay of 1st Article Testing or Flight Test Requirements – Cost growth due to technical deviation’s required – Impact to Operations due to late supplier deliveries Omnia Paratus Corporation 4
- 6. Despite the value-add it offers to improving the acquisition process, risk management is seldom considered or implemented  Risk Management Is Not Difficult, Is Well Documented, and Training Materials Are Readily Available. So Why Is It “Hard”? – Seldom seems urgent  It’s a “lower right quadrant” activity High Urgency  It’s often overcome by events  It’s someone else’s job – Requires careful thought Low  People think it’s “easy” because it’s not difficult  Fail to distinguish between perception and reality Low High  Skip the analysis and solve the wrong problem Importance  Determining what can be controlled, influenced, or changed – Team participation Risk Management  Part of the culture  Common understanding  Training, support, and reinforcement Omnia Paratus Corporation 5
- 7. Risks that impact large scale programs often directly originate from supplier performance throughout all phases of a program  The main challenges that confront these programs include: – Program Requirements – Technical requirements can have a tendency to increase unknown potential for inhibiting program success or realization of full award fee are often over looked while developing scope of work and supplier selection process. – Source Selection Analysis - A review of industry standard Best Value Analysis or Lowest Cost Alternative approach reveals gaps in several areas of Procurement Analysis skewing the perception of results derived, inherently selecting a supplier who may not adequately meet program requirements. – Risk Identification – Most risks identified often have root causes stemming from supplier performance and/or capabilities, these risks tend to be identified after procurement award. – Risk Impact Analysis – Procurement analysis does not provide insight into the potential cost and schedule impacts associated with selecting a supplier, impacts that could affect the success or ability of program operations, award fee and sustaining customer contracts. – Mitigation Analysis – Developing a mitigation plan after a problem has occurred limits a program’s mitigation options resulting in exuberant mitigation costs that extend beyond program office and/or client budget. Omnia Paratus Corporation 6
- 8. Integrating Risk Management directly into the acquisition analysis process is seamless and beneficial Request for Proposal Proposal Evaluation Contract Award Development Description  Develop and issue RFPs based  Evaluate proposals using a  Analyze potential risk impact on a standard format; collect clear and structured evaluation and mitigation posture relating vendor proposals mechanism and methodology to supplier proposal’s  Assess program requirements  RFP responses collection  Supplier proposal risk analysis  Assess supplier capabilities Activities  Technical and contractual /  Identify potential mitigation plans  Assessment methodology & procedural evaluation  Assess mitigation posture criteria development  Risk identification  Integration of risk impact into  Define risk parameters  Risk ratings defined program budget and schedule  Detailed functional and technical  Technical, contractual /  Pre-Mitigation Analysis Outcome requirements procedural and commercial  Post-Mitigation Analysis and  Evaluation methodology evaluation of proposals Effectiveness  Scoring model for RFP evaluation  List of potential risks and  Program level risk adjusted  Vendor bidders list risk ratings within individual cost and schedule analysis proposals  Supplier selection  RFP document  Risk rating scales & categories  Contract Award Compare supplier risk profile at Assess program requirements, Provide insight into where supplier proposal evaluation completion to supplier capabilities and evaluation risks affect the program and uncover determine outstanding risk criteria to establish RFP their true impacts. exposure. Omnia Paratus Corporation 7
- 9. Acquisition Risk Analysis follows a typical Risk Management process and can be tailored to program specific needs 1 Risk 2 Risk Risk 4 Risk Applying 3 5 Planning Identification Analysis Mitigation Risk Results Risk Identificat • Creating standard risk • Collecting SME input, • Conduct probabilistic • Identifying, evaluating, • Using cumulative risk ion terms and definitions best practices, assessments of risk and selecting scores as a vehicle for metrics, etc., to impacts strategies to reduce ranking alternatives • Developing processes, identify possible risks risk criteria, and scoring approach for risk • Documenting risk • Developing an data into a central actionable risk repository mitigation plan, with sound rationale (if applicable) Identifies risks that Sets risk parameters for Examines risks to Develops measures for Uses results of risk may impact cost, effective risk analysis determine impacts by keeping risk at analysis in assessing schedule, or and across alternatives acceptable levels alternatives performance Omnia Paratus Corporation 8
- 10. Building qualitative definitions for risk ratings enables an objective, not subjective quantification of proposal risks  The first step in Risk Planning involves defining standard risk terminology using many different resources, but should be robust and accurate enough to reflect the program’s overall risk tolerance  Developing a robust, well-defined risk analysis process will produce results that reflect program needs; effectively communicates these needs across stakeholders; reduce personal bias in determining relative risk importance; and uncover potential impacts of great importance to the program  Risk criteria is not normative, and are based on available resources, time constraints, and amount/type of information solicited in generating the criteria Considerations for Defining Risk Criteria Performance The analyst needs to ensure that the units of measure for risk impact reflect program needs. Critical program drivers might include life-cycle Metrics costs or metrics reliability (e.g., Mean Time Between Failure). If impact thresholds are expressed in percentages (for example) the analyst needs to review those figures within the context of the overall Order of Magnitude anticipated program cost—e.g., 10% of $5 million program is substantially different than 10% of a $5 billion program. The analyst needs to ensure that there is an adequate number of thresholds for evaluating the risk impact. The more levels of consequence Level of Impact that are utilized, the greater the insight into the need for increased data fidelity/quantity of data the analyst would need to consider. Qualitative data is represented by probability values ranging from “Very Unlikely” to “Very Likely” , using stakeholder input to assess and weigh the importance of benefits, cost, and risk in relationship to the total analysis or evaluation of benefit, risk, and cost factors that cannot be quantified. Qualitative vs. Quantitative Quantitative data is represented by consequence values using linear numerical probabilities (e.g., 0.1, 0.3, 0.5, 0.7, and 0.9), nonlinear numerical probabilities (e.g., 0.05, 0.1, 0.2, 0.4, 0.8), cost estimates, metrics, ranges, or percentages. Omnia Paratus Corporation 9
- 11. A Supplier Management Maturity module is used to determine if supplier capabilities comply with program requirements 1 – Minimal Capability 2 – Functional 3 – High Performance 4 – Best in Class  Framework used to assess each IPTs risk  Minimal understanding or  Dedicated resources do not  Applied use of specific Regular training conducted to enhance experience in applying basic exist processes/tools skills and capabilities management maturity in terms of People, Process, People concepts and principles  Limited to individuals who may  Dedicated team resources  Dedicated organizational resources  Minimal understanding of principles or language have had little or no formal training  In-house core experts, formally trained in basic skills All staff is informed and capable of applying mid- to advanced concepts Technology, and Governance  Inconsistent application of  Common processes defined  Formal processes integrated into Qualitative/quantitative analysis  The optimal goal of the risk enhancement effort is concepts or principles  Formal risk decision-making and formally documented  Process effectiveness limited to different areas of the program methodologies employed with emphasis on valid and reliable data sources to select supplier cable of bidding based on their Process body does not exist  Risk reporting and/or metrics is a dedicated team  Qualitative analysis based on  Active allocation and management of budgets  Metrics used and reported, with consistent feedback for improvement “Best in Class” maturity minimal or does not exist ill-defined rating system  Metrics collected  External stakeholders actively  Formal, documented process  Established decision-body  Key internal stakeholders actively participate in process does not exist forum participate in process Integration into organizational processes and decision-making NOTIONAL • No structured application • Customizable solutions tailored • Integrated set of tools and  State-of-the-art tools and Technology  Management and tracking tools to program methodologies methodologies not in use • Few repeatable technological  Centralized data environment  Distributed data environment that  Analysis not performed solutions in place managed by dedicated team provides access to all program resources resources  Standardized and automated reporting capabilities • Minimal awareness  Upper management • Accepted as a program  Integral to informed decision-making by  Minimal upper management encourages, but does not management function upper management Governance involvement require use  Benefits recognized and expected  Active use encouraged and rewarded  Tendency to continue with  Application varied through out  Upper management requires  Part of the organizational philosophy to existing processes even in the program tracking and reporting achieving program success face of potential failure • Process may be viewed as  Focus on mitigation effectiveness  Top-down commitment by leadership  Dedicated resources do not exist additional overhead with versus reporting and status variable benefits tracking Average People Process Tech Gov’t Metrics Quality Capacity Rating Supplier A 3 4 4 1 2 2 1 2.42  Supplier’s will be scored based on the Supplier B 2 3 - 3 NOTIONAL 3 1 1 1.85 maturity model to establish a list of Supplier C 3 3 4 2 3 3 2 2.85 qualified suppliers for proposal solicitation Supplier D 3 3 4 4 3 3 2 3.14 Omnia Paratus Corporation 10
- 12. Once proposals have been received a technical evaluation should be conducted for clarifications prior to identifying supplier risks Proposal Clarification Items Ref. Issue Topic Area Questions Answer Given Resolved  Is the team available as indicated in the Proposal? P. 60 Staffing  Additionally, a further elaboration on local staff/ qualifications is required  All questions to a  A further elaboration on the timeline is required, especially on the 9 P. 46 Bidder should be days timeframe proposed for defining the architecture and standards compiled and sent General  A show-case of the deliverables must be provided in a formal email / Approach  How realistic are the assumptions on available data and fax Organization staff?  All Bidders should Assumptions  What is the impact on costs and timeline of a provision of a P. 18 be given ample and documentation in other languages? the same amount of Phase 0: Kick-Off Alignment  How will Bidder ensure the alignment of the PM tools to the P. 34 time to respond to Organization’s PMO? clarifications  How will the automatic “systems” feed work (XML). Is it a P. 36- requested requirement? 38  How will any incompatibilities of import functions affect the Phase 1: Baseline Tools suggested timeline?  What if the Organization wants to use other EA tools, like ARIS?  Further elaboration on “GEAS” layers required. A mapping to the Organization framework layers is needed as well  Explanation of the rationale for selecting Singapore, Australia and P. 43 Phase 2: Best Canada Approach Practice Review  Is the access to those countries only via databases/benchmarks or “real” contact? Phase 3: Definition  No reference given, on how the Organization’s imperatives will be P. 45 of Architecture & Imperatives ensure – therefore, a further elaboration on the approach is required Standards Omnia Paratus Corporation 11
- 13. The next step to the risk assessment process is to identify risks that may impact a program’s cost, schedule, or technical performance……  The risk analysis focuses on risks that affect each supplier’s ability to ensure program success (i.e., ongoing, uninterrupted support to the operational forces). Risk  Although identification of risk relies on the skill, experience, and insight of subject matter experts and Identificat risk personnel, the methods and tools for initiating the identification of risk may vary ion  As such utilizing risk categories such as these bolsters risk identification… Budget Suppler Metrics Schedule Scope of Work Constraints Supplier Performance “Common” Risks Lessons Learned Capabilities Metrics Material Testing & Qualification Resources Availability Integration Requirements Sensitivity Program Components Management Analysis Requirements Omnia Paratus Corporation 12
- 14. …each risk is then reviewed against a set of Program-specific impact definitions, with the highest rating used to determine overall risk severity Cost ( % over of cost Level Technical Supplier Delivery Schedule target) Disastrous Existing technology does not exists >5 month slip in MRP requirements > 6.00% Severe Existing technology exists, but has not been proven 4-5 month slip in MRP requirements 5.01 – 6.00% Critical Supplier has never built component before, Flight Test required 3-4 month slip in MRP requirements 4.01 – 5.00% Substantial Supplier has never built component before, 1st article required 2-3 month slip in MRP requirements 3.01 – 4.00% Significant Supplier has never built component before 1-2 month slip in MRP requirements 2.01 – 3.00% High Supplier built similar component, Flight Test required < 1 month slip in MRP requirements 1.01 – 2.00% Moderate Supplier has built a component similar in nature, 1st article required Risk erodes 100% of schedule margin .76 – 1.00% Medium Supplier has built identical component for other Programs Risk erodes 51 – 75% of schedule margin .51 - .75% Low Supplier has previously built component Risk erodes 26 – 50% of schedule margin .26 - .50% Minimal Incumbent supplier Risk erodes < 25% of schedule margin < .25% Technical = High 3 Schedule = Critical 1 Cost = High 2 Consequence Rating = Critical Omnia Paratus Corporation 13
- 15. Risks are then integrated into the suppliers proposal to determine impacts of risks based on the risk ratings identified  The risk database transposes risk rating into cost and schedule impacts in quantifiable dollar and days  WBS and IMS task are later used to integrate supplier risk adjusted proposal into the Program’s Budget and IMS Schedule Risk WBS IMS Risk Cost Impact Risk Description Prob. Conseq. Impact # Impact Task ID Rating ($K's) (Days) Lack of supplier staff to support delivery 1 1.2.1.3 3 8 31.00 53 requirements $7,500 88 2 2.3.1.6 71 CCB is a development item 5 9 71.25 $8,500 118 Supplier X's On Time Delivery rating 3 4.3.5.1.1 2 10 28.50 101 subpar $9,500 150 Requirements of flux capacitor increase 4 6.3.3.7 the need for Supplier X System Test & 4 6 32.25 88,97 Eval staff and resources $5,500 52 Lack of technical capability in power 5 1.3.1.5 generation / storage hardware may cost / 1 3 3.30 70 schedule over runs $2,500 18 1.3.1.2 Tooling re-use approach may not be 6 4 10 71.25 1.3.3.15 67,97 compatible with new technology $9,500 150 Omnia Paratus Corporation 14
- 16. Monte Carlo Analysis is used in calculating risks associated with supplier cost impacts to program budget….  This chart example utilized Crystal Ball to perform Monte Carlo, multiple types of software exists and can be used for this analysis  Pick a Confidence Level based on program maturity and requirements  This example highlights the 75th Percentile – 75% of costs are below the line, 25% of costs are above – The 75% CL is $49,448 Omnia Paratus Corporation 15
- 17. ….as well as supplier risks that have the potential for disrupting a program’s Operations schedule.  The analysis example utilized SCRAM to run the schedule risks analysis  Confidence Level (CL) picked for cost is also used for schedule to determine how risk may impact a supplier’s delivery schedule Omnia Paratus Corporation 16
- 18. Schedule & Cost Risk Assessment Module (SCRAM) is a MS Project add-in and FREE for use on NASA projects.  SCRAM’s capabilities compare with that of Pertmaster, @Risk and Risk++  Extremely user friendly and reliable  Customizable aspects not available with other tools  Compatible will all MS Office Products Omnia Paratus Corporation 17
- 19. Once Monte Carlo simulations are complete and confidence levels selected, supplier bids can be compared based on potential risk impacts Supplier A Supplier B Supplier C Proposed $ 494,958 $ 465,135 $ 514,453 10% $ 507,645 $ 512,622 $ 547,565 20% $ 509,542 $ 523,002 $ 555,345 30% $ 510,834 $ 530,892 $ 561,676 40% $ 512,016 $ 537,855 $ 567,689 50% $ 513,090 $ 544,29 $ 573,385 60% $ 514,241 $ 551,640 $ 579,471 70% $ 517,449 $ 552,673 $ 585,912 80% $ 520,923 $ 565,114 $ 594,321 $500,000,000 $600,000,000 $700,000,000 90% $ 523,007 $ 582,310 $ 607,304 100% $ 629,484 $ 589,685 $ 711,107 Supplier B Confidence Level Chosen Supplier A Supplier C Omnia Paratus Corporation 18
- 20. Side-by-side risk exposure calculations provides leadership with comparative insights into supplier potential costs impacts Analysis will lead to a cumulative assessment of the total risk exposure All costs reported at the and the potential impact to program 90% confidence interval. budget. $39,100 Potential Risk $650,000 Potential Risk Impact $179K Potential Risk Impact $99K Program Cost (in thousands) $600,000 Impact $25K $550,000 $500,000 $450,000 $400,000 Component $350,000 budget $500K $300,000 $250,000 $497,042 $482,331 $200,000 $427,436 Supplier Initial Cost $150,000 Risk Exposure $100,000 $50,000 $0 Supplier A Risk Adjusted Bid Supplier B Risk Adjusted Bid Supplier C Risk Adjusted Bid Omnia Paratus Corporation 19
- 21. Risk adjusted delivery schedules are then compared to determine the potential risk impact to program’s operations Supplier A Supplier B Supplier C 9 8 4.5 8 4 7 7 .5 6 6 3 5 5 .5 4 4 Illustrative Illustrative 2 Illustrative 3 3 .5 2 2 1 1 1 0.5 0 0 0 5/3/10 5/7/10 5/11/10 5/15/10 6/2/10 6/6/10 6/10/10 6/14/10 6/18/10 6/22/10 7/7/10 7/11/10 7/15/10 7/19/10 7/23/10 7/27/10 7/31/10 8/4/10 8/8/10 Supplier’s Initial Proposal Supplier A B C Proposed Schedule 04/18/2010 04/04/2010 04/21/2010 Risk Adjusted Schedule 05/13/2010 07/18/2012 08/21/2012 Material Resource Planning requirement for this component is 04/25/2010 Omnia Paratus Corporation 20
- 22. The risk-adjusted cost and schedule results are then compared for awarding contracts on a risk averse path Supplier’s Initial Proposal Supplier’s Risk Adjusted Proposal Supplier A B C Supplier A B C Risk Adjusted Cost $ 523,007 $ 582,310 $ 607,304 Proposed Cost $ 497,042 $ 482,331 $ 427,436 Risk Adjusted Proposed Schedule 04/18/2010 04/04/2010 04/21/2010 05/13/2010 07/18/2012 08/21/2012 Schedule  Initial review of Supplier bids would indicate “C” as the supplier of choice  Based on Supplier B & C delivery metrics and potential risk, schedule impact could result in more than a 3 months past MRP requirements  Based on supplier C’s lack of technical capabilities and schedule risk to operations, risk impact could equate to ~$100K over component budget  Based on risk adjusted Cost & Schedule proposal analysis supplier “A” should receive program consensus for contract award based on least amount of risk exposure to the program Omnia Paratus Corporation 21
- 23. Given these potential benefits, a few key considerations are worth noting  In order to have a successful portfolio risk management process, it’s important that the constituent components of the program have sufficiently mature supply chain management and risk management processes.  Integration of acquisition risk analysis into a program’s budget and schedule is necessary to capture the magnitude of potential program risk impact by a single supplier  Identifying risks within a proposal enables forward looking program management that can be streamlined into existing risk database’s for future risk management planning and mitigation.  Qualitative risk analysis provides enhanced proposal evidentiary support and solid justification for awarding contracts  The success of a supply chain risk management program requires the consistent and active support of program leadership in order to be successful. Omnia Paratus Corporation 22
- 24. For more information on how acquisition risk analysis can be applied to your specific challenges, please:  Contact: James Taylor Huntsville, Alabama 310-462-6878 James.Taylor@omniaparatus.com Omnia Paratus Corporation 23