Big data meets evm (submitted)
4 likes6,901 views
The document discusses the integration of big data with earned value management to improve predictive and prescriptive forecasting for program performance. It emphasizes the limitations of traditional forecasting methods, advocating for the use of descriptive, predictive, and prescriptive analytics to enhance decision-making and identify underlying drivers of performance variance. The author suggests employing advanced statistical techniques such as time series analysis and principal component analysis to reveal correlations and improve future estimates at completion.
Big data meets evm (submitted)
- 1. + Big Data Meets Earned Value Management We have lots data. How can we use it to make predictive and prescriptive forecasts of future performance to increase Probability of Program Success? Glen B. Alleman Thomas J. Coonce
- 2. +The Killer Question For Every Manager Of A Complex, High Risk Program Is … 2 … How Can I See An Unanticipated Estimate At Completion (EAC) Coming Before It’s Too Late? “What’s in Your Estimate at Completion?”, Pat Barker and Roberta Tomasini, Defense AT&L, March-April, 2014
- 3. +Here’s WHY We Need Better Ways To Forecast Estimate At Complete … 3 … the root cause starts on day one, with a less than credible PMB. 42% 29% 21% 0% 10% 20% 30% 40% 50% 60% From Phase B Start From PDR From CDR DevelopmentCostGrowth 29% 23% 19% 0% 10% 20% 30% 40% 50% 60% From Phase B Start From PDR From CDR PhaseB/C/DScheduleGrowth
- 4. +Three Types Of Data Are Available In Big Data Repositories n Descriptive – looking in the past we can learn what happened, but it’s too late to take corrective action. n Predictive – using past performance we can answer the question what will happen if we do nothing but do the same as we’ve done in the past. n Prescriptive – past performance data used to make predictions and suggest decision options to take advantage of the predictions 4 Prescriptive analytics not only anticipates what will happen and when it will happen, but why it will happen.
- 5. +Descriptive Analytics n Descriptive Analytics – condensing big data into smaller, useful nuggets of information. n Most raw Earned Value data is not suitable for human consumption since it is reported by WBS without the connectivity to the product or programmatic topology n Descriptive data summarizes what happened in the past, many times 45 days in the past. n Correlations between WBS elements not defined nor correlations between risk, technical performance or Systems Engineering attributes – MOE, MOP, KPP† The EVM repositories provide the raw material for Descriptive Analytics through the IPMR (DI-MGMT-81861) submittals 5 † The Defense Acquisition Guide defines how to apply Measures of Effectiveness, Measures of Performance, Technical Performance Measures, and Key Performance Parameters to assess program performance
- 6. +DAU Gold Card’s EAC Formula Uses Predictive Analytics, But … n Past variances are wiped out with “Cumulative to Date” data n No adjustment for risk n Not statistically corrected for past performance 6
- 7. +Prescriptive Analytics n Is a type of Predictive Analytics n Used when we need to prescribe an action so leadership can take the data and act. n Predictive analytics doesn’t predict one future outcome – but Multiple outcomes based on the decision makers actions. n Prescriptive analytics requires a predictive model with two additional components: n Actionable data. n Feedback system that tracks the outcome produced by the action taken.. 7
- 8. +Prescriptive Analytics Is The Foundation For Corrective Actions n Prescriptive Analytics is about making decisions based on data. n Prescriptive analytics requires a predictive model with two components: n Actionable data n Feedback from those actions n Prescriptive models predict the possible consequences based on different choices of action. 8 Milestones are rocks on the side of the road.The Roman Milestone was a measure back to Rome.You only know that distance after you pass the milestone.
- 9. +There Is Untapped Value In An Earned Value Data Repository n Most data is of little value at the detail level since it is uncorrelated in the reporting process n Making correlations between cause and effect is difficult for humans, but statistical process algorithms can do this for us n With correlated data in hand, we can start generating descriptive analytics n But drivers of variance are not visible in the repository n Variances from past can be calculated, but not used in future forecasts n There is no built-in mechanism to see patterns in the data n Standard tools produce linear, non-statistical, non-risk adjusted forecasts To extract this value we need to overcome some limitations in today’s repositories 9
- 10. +All Programmatic Forecasting Is Probabilistic, Driven By Underlying Statistical Processes If we make forecasts about program performance that are not statistically and risk adjusted – we’re gonna get wet. 10
- 11. +Schedule, Related Cost And Technical Elements Are Probabilistic The IMS doesn’t help us much either,since the correlative drivers are themselves non-linear stochastic processes 11 A Stochastic process is a collection of random variables used to represent the evolution of some random value or system, over time.
- 12. +The Ability To Forecast Future Performance Starts With A Tool That Provides… n Forecasting of future performance, using time series of the past using Autoregressive Integrated Moving Average (ARIMA) algorithm n Confidence intervals of these forecasts for past performance n Correlation between the time series elements (CPI, SPI,WBS element) n Deeper correlations between these Earned Value elements as risk retirement, increase effectiveness and performance and any other recorded measure of the program. 12 http://cran.us.r-project.org/ The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data. - John Tukey
- 13. +A Quick Look At Where We’re Going Starting with Forecasting CPI/SPI n We have a time series of CPI, SPI, in the repository n What’s possible behaviors in the future can we discover from the past behavior? n The R code on the top answers that in 4 lines. If we want to credibly forecast the future with the past,we’ll need better tools.We’ve got the data,just need to use it 13
- 14. 14 § The Units of Measures for Earned Value Management are Dollars § Cumulative indices wipe out all the variances § Forecasts of future performance are not statistically adjusted § There is no correlative information drivers of variances § None of these forecasts use the risk register to adjust their value
- 15. +Since ARIMA Is A Well Traveled Path, We Need More and Better Tools n The Earned Value Management Performance measures need to be connected to: n Risk retirement and buy down status Technical Performance Measure compliance n Measures of Effectiveness and Measures of Performance n Work Breakdown Structure correlations for each work activity n Correlations between performance and work performed is available in the repository n We’re missing the tool to reveal these correlations, drivers, and corrective actions to keep the program GREEN To provide better forecasts of EAC,we need more data. CPI/SPI needs to be augmented with technical program data 15
- 16. +We Need More Power To See Into The Future And Take Corrective Actions 16 We need more power Mr. Scott I’m given her all she’s got ‘Captain
- 17. +Principal Component Analysis (PCA) Gets More Power from our data 17 Principal component analysis (PCA) is a statistical procedure that uses orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components. We want to convert a larger set of program performance variables – SPI, CPI, Risk Retirements,TPM, MOE, MOP, KPP, Staffing, and others, into a small set of drivers of variance. PCA can provide visibility to what are the connections between EAC growth and the source of that growth in a statistically sound manner, not currently available with IPMR reporting using CPI/SPI
- 18. +What Can PCA Tell Us? n If data lies in high dimensional space (more than just CPI/SPI), then large amount of data is required to learn distributions or decision rules. n For each WBS element 9 dimensions (CPI, SPI,WBS, TPM, MOE, MOP, KPP, Risk, Staffing Profiles). n Each dimension has 36 levels (36 months of data). n We could produce a 9 dimension scatter plot for the 36 months of data and it’d look like a big blob. n We need to know what are the drivers in this Blob of data? With “all” the data in a single place – which it is not – we need a way to reduce the dimensionality to provide analysis 18
- 19. +From 2 Dimensions (SPI/CPI) to 8 Dimensions and Back Again n Two components, for example – SPI and CPI n Discover the correlation between these two data samples n Locate in the individual samples the time the drivers started impacting the program n Extend this to 8 dimensions n Similar to Joint Confidence Level, but with actual data 19 PCi=a1X1 + a2X2 + a3X3 + … + a8K8
- 20. +Program Performance Dimensions PCA data can be simple 2 dimensional – CPI/SPI or more complex and represent other “attributes” driving EAC 20 Variable Information that mat drive Unanticipated EAC CPI/SPI CPI for program, time phased by reporting period TPM Technical Performance Measures, with control bands as program moves left to right.These can be any measure technical compliance § Weight § Throughput § Information Assurance validation § Any of the JROC KPPs Risk Risk § Risk retirement buy down plan § Risk handling and planned reduction Margin Cost and schedule margin burn down to plan
- 21. +Call to Action for increased visibility to Unanticipated EAC Growth using BIG Data 21 Normalize data in the Central Repository in preparation for analysis Apply ARIMA to normalized data to forecast CPI, SPI, and Calculated EAC Adjust ARIMA parameters using past performance compliance Integrate external data with EV repository data to build correlations the EAC forecasts Apply Principal Component Analysis (PCA) to identify correlated drivers of EAC growth