SDM_Task-A_25.07.2013_AKC
- 1. Project Report on evaluating OpStrat through Statistics - By Anwesh Kumar Chakraborty Title: An Exploratory Parametric Analysis of Production Cost Improvement in the Aerospace Industry Rationale: The journal is about parametric analysis of production cost improvement in aerospace industry. Following are the reasons why the same has been selected for review- I come from operations and maintenance background wherein I have been closely associated with various cost improvement initiatives. Thus I find the topic of this study falling under the purview of my current and future areas of interest. The study has been conducted by a technical expert and because of my technical background I found the journal academically enriching Of late I have been studying the methodology of statistical decision making and this paper will provide me some insights into statistical studies that are conducted prior to the decision making stage I also believe that this study will provide a practical approach to my understanding of hypothesis development, parametric tests and result interpretation. Objective: The following are the objectives of this paper- To investigate whether parametric analysis is a feasible approach to estimating cost improvement rates To investigate whether cost improvement should be estimated without reference to product type, i.e., by using characteristics of the product and production environment that apply to a range of product types Hypotheses discussed: The null hypothesis (H0 – as specified in the paper) & alternate hypothesis (HA – as deduced) tested in this paper are as follows: H0: The coefficients of the restricted (product specific subsets) and unrestricted (unspecified product subsets) models are equal, i.e. they are the same model. HA: The coefficients of the restricted and unrestricted models are not equal, i.e. they are not the same model. Other null hypotheses (Hnew01 & Hnew02) & alternate hypotheses (HnewA1 & HnewA2)that can be tested/ developed through the analyses done in this paper are as follows: Hnew01: Cost improvement rates are affected only by the process HAnew1: Cost improvement rates are not only affected by the process i.e. other factors also affect the cost. Hnew02: Complex products have greater scope of cost improvement that simpler products HAnew2: Complex products have lesser scope of cost improvement that simpler products
- 2. Project Report on evaluating OpStrat through Statistics - By Anwesh Kumar Chakraborty Introduction of paper: In the field of cost analysis, the terms “learning” and “cost improvement” refer to the observed reductions in unit cost that occur during a production run for a complex end item. For many years, cost analysts have fit downward sloping curves to plots of unit production costs, and have used the slopes to characterize the rate of cost improvement. Cost analysts in training spend much time learning about these curves, including how to properly plot average lot costs, and the differences between the unit and cumulative average computational approaches. This training is useful for answering such questions as, “What will be the cost of the next production lot (or unit) to be produced on an established production line?” In this case, there is an empirical observation of the relevant improvement rate, and it is assumed that this rate will continue to be applicable to the given production facility with its known tooling, workers, and management (AFSC, 1986, 7–96). In addition, cost analysts working for established manufacturers often will have an extensive corporate cost experience on which to draw, and when a new product, or new model of a product, is to be produced they will have an expectation for unit cost improvement in the various factory departments (Teplitz, 1991, 48). To date, this is essentially the recommended curve selection approach—to extend the observed historical cost improvement experience on the same product or a very similar product (AFSC, 1986, 7–80). The underlying assumption of these methods is that similar products produced using similar production methods should have the same or nearly the same cost improvement experience. Unfortunately, there are common situations in which methods based on these assumptions are difficult to apply. These situations include the production of new products when there is little corporate experience to draw upon, or the setup of new production facilities. Even more problematic is a task encountered by cost analysts for government procurement agencies and aerospace contractors. Here they often must provide an estimate for a budget or corporate cost position on a prospective contract even before the request for proposals has been released. Later, during source selection, another related issue is encountered when bidders’ estimates are being evaluated by the government. Is the improvement rate used in the proposal reasonable? Or is the bidder trying to “buy-in” with an overly optimistic rate?2 As a result, the assumed improvement rates can become a source of controversy. Program advocates on one side press for “affordable” cost estimates (higher rates of learning), while on the other side independent reviewers and higher headquarters want “conservative” estimates (lower rates) in order to avoid overruns. The cost analyst is in the centre of this discussion and faced with justifying one cost improvement rate that is acceptable to all. It is a difficult position, and it results from the fact that cost improvement remains essentially an empirical phenomenon. Current cost improvement “theory” can be more accurately described as a set of conjectures about the sources and effects of the various factors that influence the production process. But it is not objectively predictive. If it were, the theory would yield quantitative techniques that analysts could apply to their estimates. Such techniques could then be used to calculate “most probable,” “conservative,” or “optimistic” cost improvement rates, along with associated statistical confidence levels. This paper addresses cost improvement from the perspective of government and prime contractor cost estimators who are faced with the task of specifying defensible cost improvement rates for future production programs. Much of this experience applies to the defence aerospace business and their products (space vehicles, missiles, aircraft, avionics, etc.). In this arena, analysis is often performed on programs that will not enter into production for years. They may still be in concept exploration or engineering development, with the production contractor yet to be identified. For these programs, the practical application of the concept of cost improvement is not well-supported by the descriptive nature of the underlying theory. In short, there is a deficit of cost estimating methodology in this area. After some further discussion and definition of the problem, this paper will describe the results of an exploratory parametric analysis of cost improvement for a number of aerospace products, which indicate the possibility of developing a parametric approach to the issue of choosing a cost improvement slope. Such an approach, based on broad assumptions about the sources of cost improvement, could provide cost analysts with a better basis for estimating cost improvement rates in analytic environments like that of the defence aerospace industry. Research questions (objective of the paper): The objective of the research described here was to investigate whether parametric analysis is a feasible approach to estimating cost improvement rates. If so, the “cost improvement
- 3. Project Report on evaluating OpStrat through Statistics - By Anwesh Kumar Chakraborty relationships,” or CIRs, resulting from such an analysis could provide a useful addition to the cost analysis toolbox. With traditional CERs to estimate unit costs and CIRs to predict the rate of improvement of those costs during production, cost analysts would have a more complete set of analytic tools with which to conduct production estimates. The following research questions were used to pursue the research objective: •Can a theory of cost improvement causality be incorporated into the design of CIRs? •What independent variables can be used in CIRs? •Are independent variables that are related to product type (e.g., satellite, aircraft, avionics, missiles) as predictive as those related to the production environment (level of automation, design stability, etc.)? As pointed out in the above discussion of problems associated with the analogy methodology, the wide variation within product type classes makes it worthwhile to investigate whether cost improvement should be estimated without reference to product type, i.e., by using characteristics of the product and production environment that apply to a range of product types. A parametric analysis such as the one envisioned in this research would serve to test this hypothesis. Insights gained (from the perspective of statistics): Following were the major points observed- To pursue the research objective it was necessary to compile data on cost improvement experienced by a number of diverse aerospace products. This was because the research did not assume that cost improvement is best estimated within the bounds of individual product classes The missile, avionics, and aircraft production data were obtained in the form of average lot costs. In each case, curves of the form Y = Axb were fit to the cumulative average plot points using logarithmic transformations and ordinary least squares regression. Data representing a number of potential predictive parameters were also collected where one couldn’t specifically define the causal factor owing to subjectivity Independent variables were chosen based on their causal relationship with cost improvement rate Relationship of cost with the predictive parameters considered were assumed to be linear to establish a direct association (to make regression analysis possible). Hypotheses Development and Critical analysis (scope of the task A): The scope of this task includes the null hypothesis being tested in this paper as well as other null hypotheses that could be developed based on this paper. The null hypothesis (H0 – as specified in the paper) & alternate hypothesis (HA – as deduced) tested in this paper are as follows: H0: The coefficients of the restricted (product specific subsets) and unrestricted (unspecified product subsets) models are equal, i.e. they are the same model. HA: The coefficients of the restricted and unrestricted models are not equal, i.e. they are not the same model. Other null hypotheses (Hnew01 & Hnew02) & alternate hypotheses (HnewA1 & HnewA2)that can be tested/ developed through the analyses done in this paper are as follows: Hnew01: Cost improvement rates are affected only by the process HnewA1: Cost improvement rates are not only affected by the process i.e. other factors also affect the cost. Hnew02: Complex products have greater scope of cost improvement that simpler products HnewA2: Complex products have lesser scope of cost improvement that simpler products Comment on the methodology followed: The major steps were as follows: Collection of data Selection of independent parameters which predicted the causal factors of cost improvement
- 4. Project Report on evaluating OpStrat through Statistics - By Anwesh Kumar Chakraborty A pairwise analysis was performed to examine the strength of the direct linear correlations between potential independent variables and the dependent variable Regression Analysis of the Specified Model Regression Analysis of a Non-Product-Specific Model Chow Test on Product Data Subsets Regression Analysis of Product-Specific Models The methodology followed shows how the pairwise analysis was done to examine the strength of the direct linear correlations between potential independent variables and the dependent variable. Regression analysis of specified model was done to find which parameters were better predictors of the cost improvement and whether the model was a feasible one to validate the null hypothesis Regression Analysis of a Non-Product-Specific Model helped to improve the model as multicollinearity of independent variables was removed. Chow test was performed to check whether the specified model was fitting with non-product specific models and they were similar in nature i.e. their coefficients were equal. Lastly the Regression Analysis of Product-Specific Models was done to check the causality of cost improvement rate and the predictive factors. Alternative statistical tool suggestion: The best way to check the association is through regression analysis. No other tool could be used for that. Though other better predictive parameters could be found out to check the causality of CIR. Finding from the suggested tool: Not checked Discussion on findings: Not applicable Lesson learnt: The purpose of the research was to investigate the feasibility of predicting cost improvement in future production programs using parametric methods. Toward this end, three research questions were posed. • Can a theory of causality be incorporated into the design of CIRs? While this is an unavoidably subjective question, it is a necessary one when investigating whether statistically-based parametric relationships can be applied to an empirical phenomenon that has not been previously modeled. Any statistical relationship can only be said to have predictive qualities to the extent that it incorporates a theory of causality that makes sense to the users of the relationship. This research proposed that the potential for production cost improvement can be encompassed by three dimensions; product scale, product complexity, and design stability. This scheme seemed to have predictive power with respect to cost improvement rates, both for the models based on the pooled data and for the product-specific models. The fact that this was observed in a relatively small dataset indicates that future research efforts using more extensive and representative data could find success in modelling more detailed descriptive theories. • What independent variables can be used in CIRs? In CERs, the validity of using technical and performance parameters to predict the development or production costs of products is usually not questioned. However, at this early stage of thinking about CIRs, it is worthwhile asking whether “cost improvement drivers” can be found. In this research, although the use of weight to represent product scale was straightforward, trying to exemplify product complexity and design stability was more problematic. Dollars- per-pound was settled on as the surrogate for complexity, while the average lot size was used for design stability. While the analysis showed that there was a degree of statistical correlation between these parameters and the cost improvement rates of diverse aerospace products, whether or not they actually represent the desired aspects of the causality theory remains a subjective analytic judgment. As research and consensus-building moves forward on how to embody cost improvement in statistical relationships, a key area of discussion and agreement clearly must be on the identification of appropriate parameters.
- 5. Project Report on evaluating OpStrat through Statistics - By Anwesh Kumar Chakraborty • Are independent variables related to product type (e.g., satellite, aircraft, avionics, missiles) as predictive as those related to the production environment (level of automation, design stability, etc.)? When this question was posed at the outset of the research it was anticipated that CIRs would look like the Specified Model shown in Figure 5. This model incorporated a combination of technical parameters and product-related indicator variables. It was assumed that if the indicator variables were not significant, then this would be evidence that cost improvement could be modelled without regard to product type. The analysis showed, however, that all the variables in the Specified Model were significant, albeit multicollinear. The exclusion of indicator variables in the “non-product specific” model shown in Figure 6 reduced the multicollinearity, but the technical variables alone did not result in a close fit to the data. At this point, the results were inconclusive about the value of product type as a predictor of cost improvement. The results of the Chow Test conducted on the product-specific subsets of the data indicated that neither the Figure 5 nor the Figure 6 models might be appropriate. It indicated that the products could be independent in terms of the intercepts and slopes of the models to be fit to their data. The relatively good fit statistics of the product-specific models shown in Figures 7 through 10 provided support to this hypothesis, although these models were based on fewer observations than is desirable. The research question was intended to address the value of product-related indicator variables, and the analysis results must be considered inconclusive with respect to this issue. However, this inconclusiveness led to additional analysis providing indications that product type is indeed an important element of cost improvement analysis, not as a stratifying variable, but as a subject for separate analysis. LIMITATIONS OF THE RESEARCH The limitations of this work derive chiefly from its exploratory nature. In any cost research, data collection and normalization are the most time consuming tasks, and it was in these areas that the effort described here was chiefly constrained. In order to streamline the process of dataset development, previously conducted studies were drawn on to compile data on 31 production programs; six spacecraft, nine tactical missiles, 11 avionics systems, and five fighter aircraft. A more conclusive effort clearly would require a broader range of data in each product area. Despite the diverse sources, efforts were made to insure consistency of the data across product types in terms of computational approach (cumulative average) and the use of a single base year for the dollar related parameters (FY2002). However, only limited information was available to insure that the content of the unit costs to which the improvement curves were fit was consistent in terms of the treatment of direct and indirect costs, overhead, and so on. Future research efforts on this topic need to address the content of each data point in detail, as well as the strength and consistency of the cost improvement trends in each program. Finally, future efforts should go beyond the limited number of independent parameters available in this study for prediction of cost improvement rates. Parts counts, numbers of workers per shift, levels of automation, numbers of engineering design changes, and so on could be useful parameters to investigate. CONCLUSIONS This exploratory analysis was aimed at investigating whether a parametric approach to predicting production cost improvement is feasible. Keeping in mind the previously discussed limitations of the research, it appears that CIRs would represent an improved methodology for estimating cost improvement for future programs. The model specification and parameters investigated in this research resulted in significant fits to the cost improvement data, particularly when the data was segregated by product type. Furthermore, although the product-specific models were hampered by fewer degrees of freedom, the improvement in the fit statistics for these models compared with the pooled data was significant. The research indicates that formulating cost improvement models within the context of aerospace product types would probably yield the best results. An additional advantage of this approach is that it allows for the use of functional specifications and parameters that would be appropriate to the production environment, technology, or physical characteristics of the product. Nevertheless, the research did not demonstrate that non-product-specific models are infeasible. It may be that more general models, especially those that combine product types that have much in common, may have merit. Techniques such as the Chow Test could assist researchers in identifying products that should be modelled jointly. It
- 6. Project Report on evaluating OpStrat through Statistics - By Anwesh Kumar Chakraborty is through more parametric analysis of the unit cost improvement phenomenon that cost analysts will be able to move beyond the current subjective methodologies to a more theory-based, quantitative treatment. Cost research aimed at extending the work begun in this study should focus on the following: • Discussion and formulation of an improved theory of cost improvement that can be incorporated into CIRs • The compilation of a larger and more consistent database of aerospace programs that have experienced cost improvement, along with a wider array of possible descriptive parameters is clearly needed to support more definitive parametric analysis of cost improvement. • Formulation of CIRs to provide better predictors of future cost improvement than the current approaches used by the cost analysis community