Reliability and Quality - Predicting post-release defects using pre-release field testing results
0 likes783 views
The document discusses the importance of field testing in predicting post-release defects in software applications, emphasizing metrics such as time to first failure (TTFF), failure accumulation rating (FAR), and overall failure rating (OFR) to assess reliability more accurately than traditional metrics like mean time between failure (MTBF) and average usage time (AVT). A case study involving 18 software versions demonstrated that these new metrics provide faster and more precise predictions of post-release defects. The proposed metrics complement existing measures and can predict defects effectively within a shorter timeframe.
Reliability and Quality - Predicting post-release defects using pre-release field testing results
- 1. PREDICTING POST-RELEASE DEFECTS USING PRE-RELEASE FIELD TESTING RESULTS Foutse Khomh, Brian Chan, Ying Zou Anand Sinha, Dave Dietz
- 2. FIELD TESTING CYCLE Field testing is important to improve the quality of 2 an application before release.
- 3. MEAN TIME BETWEEN FAILURE Mean Time Between Failures (MTBF) is frequently used to gauge the reliability of the application. Applications with a low MTBF are undesirable 3 since they would have a higher number of defects
- 4. AVERAGE USAGE TIME AVT is the average time that a user actively uses the application. The AVT can be longer than the period of field testing. A longer AVT indicates that an application is 4 reliable and a user tends to use the application longer.
- 5. PROBLEM STATEMENT MTBF and AVT cannot capture the whole pattern of failure occurrences in the field testing of an application. 5 The reliability of A and B is very different.
- 6. METRICS We propose three metrics that capture additional patterns of failure occurrences: TTFF: the average length of usage time before the occurrence of the first failure, FAR: the failure accumulation rating to gauge the spread of failures to the majority of users, and OFR: the overall failure ratio that captures daily rates of failures. 6
- 7. AVERAGE TIME TO FIRST FAILURE (TTFF) VersionA % of users reporting failures 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0 7 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Days
- 8. AVERAGE TIME TO FIRST FAILURE (TTFF) VersionA VersionB % of users reporting failures 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0 8 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Days
- 9. AVERAGE TIME TO FIRST FAILURE (TTFF) reporting failures VersionA VersionB % of users 0.5 0.4 0.3 0.2 0.1 0 1 2 3 4 5 6 7 8 9 1011121314 Days TTFF produces high scores for applications where the majority of users experience the 9 first failure late.
- 10. AVERAGE TIME TO FIRST FAILURE (TTFF) VersionA VersionB 0.45 % of users reporting failures 0.4 0.35 0.3 TTFFB = 3.56 0.25 0.2 0.15 TTFFA = 6.11 0.1 0.05 0 10 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Days
- 11. FAILURE ACCUMULATION RATING (FAR) 1 0.9 0.8 0.7 % of users reporting 0.6 0.5 VersionA 0.4 0.3 0.2 0.1 0 11 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Number of unique failures
- 12. FAILURE ACCUMULATION RATING (FAR) 1 0.9 0.8 0.7 % of users reporting 0.6 0.5 VersionA 0.4 VersionB 0.3 0.2 0.1 0 12 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Number of unique failures
- 13. FAILURE ACCUMULATION RATING (FAR) 1 % of users reporting 0.8 0.6 0.4 0.2 0 1 3 5 7 9 11 13 Number of unique failures The FAR metric produces high scores for 13 applications where the majority of users report a very low numbers of failures.
- 14. FAILURE ACCUMULATION RATING (FAR) 1 0.9 0.8 0.7 FARB = 4.97 % of users reporting 0.6 0.5 VersionA 0.4 FARA = 6.97 VersionB 0.3 0.2 0.1 0 14 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Number of unique failures
- 15. OVERALL FAILURE RATING (OFR) VersionA 0.35 % of users reporting failures 0.3 0.25 0.2 0.15 0.1 0.05 0 15 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Days
- 16. OVERALL FAILURE RATING (OFR) VersionA VersionB 0.35 % of users reporting failures 0.3 0.25 0.2 0.15 0.1 0.05 0 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Days
- 17. OVERALL FAILURE RATING (OFR) % of users reporting VersionA VersionB 0.4 0.3 0.2 failures 0.1 0 1 3 5 7 9 11 13 Days The OFR metric produces high scores for 17 applications with fewer users reporting failures overall.
- 18. OVERALL FAILURE RATING (OFR) VersionA VersionB OFRB = 0.78 0.35 % of users reporting failures 0.3 OFRA = 0.93 0.25 0.2 0.15 0.1 0.05 0 18 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Days
- 19. CASE STUDY We analyze 18 versions of an enterprise software application Overall 2,546 users were involved in the field testing The testing period lasted 30 days 19
- 20. SPEARMAN CORRELATION OF THE METRICS TTFF FAR OFR AVT MTBF TTFF 1 0.09 -0.08 -0.31 -0.08 FAR 0.09 1 0.07 0.33 -0.24 OFR -0.08 0.07 1 0.39 -0.54 AVT -0.31 0.33 0.39 1 -0.3 MTBF -0.08 -0.24 -0.54 -0.3 1 20
- 21. INDEPENDENCY AMONG PROPOSED METRICS 1 0.8 0.6 0.4 0.2 TTFF FAR 0 PC1 PC2 PC3 PC4 OFR -0.2 MTBF -0.4 -0.6 -0.8 21 -1
- 22. PREDICTIVE POWER FOR POST-RELEASE DEFECTS 0.14 0.12 0.1 Marginal R-square square 0.08 6 months 1 year 0.06 2 years 0.04 0.02 0 22 TTFF FAR OFR AVT MTBF Metrics
- 23. PRECISION OF PREDICTIONS WITH ALL FIVE METRICS 100 90 80 70 60 6 months 50 Precision (%) 1 year 40 2 years 30 20 10 0 23 5 10 15 20 25 30 Number of testing days
- 24. CONCLUSION TTFF, FAR, and OFR complement the traditional MTBF and AVT in predicting the number of post- release defects Provide faster predictions of the number of post- release defects with good precision within just 5 days of a pre-release testing period It takes MTBF up to 25 days to predict the number of post-release defects 24
- 25. 25