A Comparative analysis of Software Reliability Growth Models using defect data of Closed and Open Source Software
- 1. A Comparative analysis of Software Reliability Growth Models using defect data of Closed and Open Source Software. Najeeb Ullah Maurizio Morisio Antonio Vetro’ POLITECNICO DI TORINO
- 2. Software Reliability • Probability of Failure free operations of software for specified period of time in a specified environment • IEEE Std 1633-2008 IEEE Recommended Practice in Software Reliability
- 3. Software Reliability Model • SRM is a mathematical expression that specifies the general form of the software failure process as a function of factors such as fault introduction, fault removal, and the operational environment. • IEEE Std 1633-2008 IEEE Recommended Practice in Software Reliability
- 4. White Box Approaches • Analyze the structure of software. • Derive reliability estimates on the basis of the relationship between components of the software and their interactions. • Architecture Based Models.
- 5. Black Box Approaches • Treats software as an entity and predict reliability in later phases i.e. testing and operation • Ignore the structure and interdependencies of the components of software • Early Prediction Models, SRGMs, Input Domain Based Models, and Hybrid Black Box Models.
- 6. Software Reliability Growth Models • Assume that reliability grows after a defect has been detected and fixed. • Can be applied to guide the test board in their decision of whether to stop or continue the testing. • Divided into Concave and S-Shaped. time Reliability Reliability time
- 7. Goals and Research Questions • Determine a good model in terms of fitting and prediction, among selected 8 SRGM. • Study SRGM models with both industrial and Open source data. – RQ1: Which SRGM models fit best? – RQ2: Which SRGM models are good predictors? – RQ3: A model with good fit is also a good predictor?
- 8. Methodology and Models Evaluation Metrics (RQ1) • Non Linear Regression used for Models fitting. • Goodness of Fit (i.e. R2 ): – indicates how successful the fit is. – R2 takes a value between 0 and 1, inclusive. – The closer the R2 value is to one, the better the fit. – Rank models for each data set on R2 value – Selected Threshold (i.e. Categorization) : R2 > 0.90 RQ1: Which SRGM models fit best?
- 9. Methodology and Models Evaluation Metrics (RQ2) • Models are fitted to two-third data points of each data set and predict the remaining. • Theil’s Statistics (i.e. TS: Accuracy) – average deviation percentage over all data points. – The closer TS is to zero, the better the prediction accuracy of the model. – Rank models for each data set on TS value. – Selected Threshold (i.e. Categorization) : TS < 10% RQ2: Which SRGM models are good predictors?
- 10. Methodology and Models Evaluation Metrics (RQ2) • Predictive Relative Error (i.e. PRE: Correctness) – ratio between the error difference (actual versus predicted) and the predicted number of defects – Rank models for each data set on PRE – Selected threshold (i.e. Categorization) : ± 10% of total no of actual defects
- 11. Methodology and Models Evaluation Metrics (RQ3) • Best Fitted and Best Predictor Model: (RQ3) – Count no of best fitted and best predictor model for each data set. – R2 > 0.90 AND TS < 10% – R2 > 0.90 AND PRE within ± 10% of total no of actual defects. RQ3: A model with good fit is also a good predictor?
- 12. Selected SRGM Models • Eight Models have been selected for this study due to their prevalence among SRM. Model Type Musa Okumoto Concave Inflection S-Shaped S-Shaped Goel Okumoto Concave Delayed S-Shaped S-Shaped Logistic S-Shaped Yamada Exponential Concave Gompertz S-Shaped Generalized Goel Concave
- 13. Data Collection (Industrial datasets) • Searched papers on IEEE Explorer, ACM Digital Library and in four journals, i.e. Journal of Information and Software Technology, the Journal of System and Software, IEEE software and IEEE Transaction on Reliability. • Strings used: – Software failure rate, Software failure intensity, Software failure Data sets, Software failure rate and Reliability, Software failure intensity and Reliability. • 2100 Papers, Relevant 19 papers, contained failure data sets on 32 projects (22 from system test, 10 from operation) .
- 14. Data Collection (OSS datasets) C++ Standard Library V4.1.2 V4.1.3 V4.1.4 V4.2.0 V4.2.1 V4.2.2 V4.2.3 V5.0.0 JUDDI V0.9 V2.0 V3.0 V3.1.0 GNOME V2.0 V2.2 V2.4 Apache V2.0.35 V2.0.36 V2.0.39
- 15. Models Categorization Results (RQ1) On R2 Box Plots of fitting (R2) values
- 16. Models Ranking Results (RQ1) on R2 No of DS fitted by each Model Ranking on Best Fitting-R2 RQ1: Which SRGM models fit best?
- 17. Models Categorization Results (RQ2) On TS & PRE Box Plots of Prediction Correctness (PRE) values Box Plots of Prediction Accuracy (TS) values
- 18. Models Ranking Results (RQ2) Ranking on best prediction: TS RQ2: Which SRGM models are good predictors? Ranking on best prediction: PRE
- 19. Results for RQ3 on Models Categorization Basis Fitting and prediction capability of models
- 20. Threat to Validity • No hypothesis test in methodology used for answering RQs • The choice of thresholds is not grounded in the literature. • Provide ranking of the models to identify the best model for each type of dataset and metric.
- 21. Conclusions • Results – Musa Okumoto model is the best one in fitting and prediction in industrial Datasets. – Inflection S-Shaped achieved very good results with respect to the metrics thresholds we adopted. – Gompertz model applied better followed by Inflection S-Shaped model in the case of OSS. • Observations – Models which have good performances with system test datasets also good performances with field defect data. – Best models in OSS are different from the best in industrial datasets
- 22. Questions ? A Comparative analysis of Software Reliability Growth Models using defect data of Closed and Open Source Software. Najeeb Ullah , Maurizio Morisio, Antonio Vetro’ name.surname@polito.it