Assessing the Reliability of a Human Estimator
Download as PPT, PDF1 like403 views
The document summarizes research on building predictive models to estimate software project efforts based on human estimators' demographics. A web survey collected estimators' demographics and estimates. Models were constructed using demographics as attributes to predict estimates. Some models achieved up to 78% accuracy. Attribute reduction improved accuracy. Decision trees were created to distinguish over-estimators from best estimators based on demographics.
![Assessing the Reliability of a Human Estimator http:// nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop Gary D. Boetticher Nazim Lokhandwala Univ. of Houston - Clear Lake, Houston, TX, USA [email_address] [email_address]](https://image.slidesharecdn.com/assessing-the-reliability-of-a-human-estimator-29765/85/Assessing-the-Reliability-of-a-Human-Estimator-1-320.jpg)
Assessing the Reliability of a Human Estimator
- 1. Assessing the Reliability of a Human Estimator http:// nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop Gary D. Boetticher Nazim Lokhandwala Univ. of Houston - Clear Lake, Houston, TX, USA [email_address] [email_address]
- 2. Current Configuration of PROMISE Repository Defect Prediction – 18 http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop Others - 9 Effort Estimation - 9
- 3. Research vs. Reality according to Jörgensen TSE ’07: 300+ software est. papers, 76 journals, 15+ Years http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop JSS ’04: Compendium of expert estimation studies 52 26 19 7 Misc. 46 21 22 3 Human 74 41 32 1 ML 255 70 137 48 Algorithm Total 00-04 89-99 -89 68% Algorithm 20% ML 12% Human 72% Kitchenham 02 100% Hill 00 84% Jørgensen 97 86% Paynter 96 62% Heemstra 91 89% Hihn 91 Human Paper 82% Human 18% Formal
- 4. Research vs. Reality How to resolve? http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop Researchers coerce/entice/exhort/nudge practitioners Practitioners ignore researchers Researchers meet practitioners where they are COCOMO
- 5. Statement of Problem How do human demographics affect human-based estimation? Can predictive models be constructed using human demographics? http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop
- 6. PROMISE 2006 Addressed the problem using Genetic Programs and non-linear regression (up to 5 th order) models Produced some accurate(77 – 93%) models, GP solutions got lengthy: The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop http://nas.cl.uh.edu/boetticher/publications.html (( MgmtGCourses ^ ((( Log ((( TotLangExp / ( TotLangExp / ( TechGCourses * HWPMExp ))) - ( TechGCourses * HWPMExp )) - (( Sin ( MgmtGCourses ^ ( Sin (( TechGCourses * HWPMExp ) - ( MgmtGCourses ^ ((( Log ( HWPMExp ^ ( TotLangExp / ( TechGCourses * HWPMExp )))) - ( Abs ( Log (( TotLangExp / ( TechGCourses * HWPMExp )) - (( Sin (( Sin ( Abs ( TechUGCourses / MgmtGCourses ))) - ( TotLangExp / ( MgmtGCourses ^ ((( Log ((( TotLangExp / ( HWPMExp / SWProjEstExp )) - ( Sin ( TotLangExp / ( TotLangExp / (( MgmtGCourses ^ (( Log ( TechGCourses * HWPMExp )) - ( Sin ( Abs ( Log (( HWPMExp / SWProjEstExp ) - ( TechGCourses * HWPMExp ))))))) + (( Sin ( TechGCourses * HWPMExp )) - ( Sin ( TechUGCourses / MgmtGCourses )))))))) - ( Sin ( TechUGCourses / MgmtGCourses )))) - ( TechGCourses * HWPMExp )) - ( Sin ( TechUGCourses / MgmtGCourses ))))))) - ( HWPMExp / SWProjEstExp )))))) - ( Sin ( TechUGCourses / MgmtGCourses )))))))) - (( Sin ( Abs ( Log (( TotLangExp / ( TechGCourses * HWPMExp )) - (( Sin (( Sin ( Abs ( Log ( HWPMExp ^ ( TotLangExp / ( TechGCourses * HWPMExp )))))) - ( TechGCourses * HWPMExp ))) - ( HWPMExp / SWProjEstExp )))))) - ( Sin ( TechUGCourses / MgmtGCourses )))))) - ( TotLangExp / ( TechGCourses * HWPMExp ))) - ( Sin ( TechUGCourses / MgmtGCourses )))) + ( TotLangExp / ( TechGCourses * HWPMExp))) So for 2007…
- 7. PROMISE 2007 The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop http://nas.cl.uh.edu/boetticher/publications.html Larger sample set. 2006 PROMISE 122 samples 2007 PROMISE 178 samples Many learners. 51 classifiers, 4142 experimental trials Attribute analysis. Simpler models. Focus is on classifiers Human readable models
- 8. Strategy Create a Web-based survey Users demographics Users Estimate software components Feedback Users Build models: demographics estimates http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop
- 9. The Survey (2001 -2005) http:// nas.cl.uh.edu/boetticher/EffortEstimationSurvey.html http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop Demographics Personal Academic Background Work Experience Domain Experience
- 10. Ecommerce: Competitive Procurement Buyer Admin Buyer 1 Buyer n ... Buyer Software Distribution Server Supplier 1 Supplier 2 Supplier n : Supplier Software http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop
- 11. Sample Estimation Screenshots http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop
- 12. Feedback to Users http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop
- 13. User Demographics - 1 http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop Average age: 31.43 148 males, 30 females 1% Ph.D., 24% Master, 72% Bach., 5% High School 25 countries: 42% India, 32% U.S., 6% Romania, 4% Vietnam.
- 14. User Demographics - 2 http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop 5.3856 28 3.6629 Process Industry 4.4391 25 1.4382 Procurement & Billing Domain Experience 5.3856 28 3.6692 Software Projects 4.4390 25 1.4382 Hardware Projects No. of Projects estimated 2.4757 15 1.6967 Software Project Manager 3.0633 25 1.0169 Hardware Project Manager Years of Experience as a Std. Dev. Max. Ave. Years
- 15. Data preprocessing & Experiments http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop 178 Samples WEKA: 51 Classifiers, 4 seeds, 10-fold Attribute Reduction: 2 configs. Remove outliers: Estimate > 10 * Actual or Estimate < 0.1*Actual 163 Extract: 25 Worst under -estimators 25 Best estimators 25 Worst over -estimators
- 16. Results: Under vs. Best http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop Ave. Accuracy 48.22% 64% VFI 64% ThresSel 64% Logistic 68% J48 76% PART Accuracy Classifier Y Y Y Total Lang Exp. Y Y Y Total Workshops Y Total Conferences Y Tech Undergrad Courses Y Y Software Proj. Mgmt Exp. Y Y Level of College Y Y Y Y # of Hardware Proj. Est. Y Y Y Mgmt Undergrad Crses Y Mgmt Grad. Courses Y Y Y Y Hardware Project Management Exp. Y Y Y Domain Exp. VFI Thresh. PART Logistic J48 Demographic Evaluator Classifier 68% Logistic/ Logistic 70% VFI / VFI 74% PART/J48 74% J48/J48 74% LogitBoost/J48 74% Bagging/J48 76% ThresholdSel/ ThresholdSel 78% ADTree/Part Accuracy Class./Eval.
- 17. Under vs. Best: Attribute Reduction http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop Y Y Total Lang Experience Y Y Total Workshops Y Y Total Conferences Y Y Tech Undergrad Crses Y Y Soft. Proj. Mgmt Exp. Y Y Y Level of College Y # of Software Proj. Est. Y Y Y Y # of Hardware Proj. Est. Y Y Y Mgmt Undergrad Crses Y Y Y Mgmt Grad. Courses Y Y Y Y Hardware Proj. Mgmt Exp. Y Y Y Y Domain Experience VFI Thresh PART Logistic J48 Demographic Evaluator Classifier 68% Logistic / Logistic 70% VFI / VFI 74% PART / J48 74% ADTree / J48 74% PART/ PART 74% J48/ PART 76% ADTree/ ThreshSel Accuracy Class / Eval
- 18. Under vs. Best: Attribute Reduction http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop Domain Exp <= 3 | No Of Hardware Proj Estimated <= 4 | | Hardware Proj Mgmt Exp <= 1 | | | MgmtUGCourses <= 0: BEST (23.0/8.0) | | | MgmtUGCourses > 0: UNDER (13.0/1.0) | | Hard. Proj Mgmt Exp > 1: BEST (5.0) | No Of Hard. Proj Est. > 4: UNDER (5.0) Domain Exp > 3: BEST (4.0) J48 Rule: 74% Accuracy BEST UNDER <-- classified as 21 4 | BEST 9 16 | UNDER
- 19. Results: Best vs. Over http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop Ave. Accuracy 42.86% 60% Ridor 60% ThresholdSel 60% RandComm 62% Decorate 66% RndTree Accuracy Classifier Y Y Total Lang Experience Y Total Workshops Y Y Total Conferences Y Tech Undergrad Courses Y Y Soft. Proj. Mgmt Exp. Y # of Software Proj. Est. Y Mgmt Undergrad Crses Y Y Y Mgmt Grad. Courses Y Y Y Hard. Proj Mgmt Exp. Threshold Selector Ridor Rnd Comm Demographic 62% ADTree / ThresholdSel 66% ThresholdSel / ThreshSel 72% Rand. Comm./ RandComm 80% IB1 / Ridor Accuracy Class/ Eval
- 20. Experiment: Best vs. Over http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop 62% Ridor Ridor 62% ThresholdSel Ridor 64% Ridor ThresholdSel 66% ThresholdSel NNge 72% Decorate PART 72% Decorate NNge 72% Decorate RndComm 74% Decorate RandomForest 74% Decorate IBk 74% Decorate IB1 80% RndComm RandomTree 80% RndComm RndComm Accuracy Evaluator Classifier Y Y Y Total Lang Experience Y Total Workshops Y Tech Undergrad Courses Y Y Tech Grad Courses Y Software Proj. Mgmt Exp. Y Y Procurement Industry Exp Y Level of College Y # of Hardware Proj. Est. Y Y Mgmt Undergrad Courses Y Mgmt Grad. Courses Y Y Y Hard. Proj Mgmt Exp Y Domain Experience Thresh Ridor Rand Comm. Decorate Demographic
- 21. Experiment: Best vs. Over TechUGCourses < 45.5 | Hardware Proj Mgmt Exp < 6 | | No Of Hardware Proj Estimated < 4.5 | | | No Of Hardware Proj Estimated < 3 | | | | TechUGCourses < 23 | | | | | Hardware Proj Mgmt Exp < 0.75 | | | | | | TechUGCourses < 18 | | | | | | | Hardware Proj Mgmt Exp < 0.13 | | | | | | | | TechUGCourses < 0.5 | | | | | | | | | TechUGCourses < -1 : F (1/0) | | | | | | | | | TechUGCourses >= -1 | | | | | | | | | | Degree < 3.5 : A (4/0) | | | | | | | | | | Degree >= 3.5 : A (5/2) | | | | | | | | TechUGCourses >= 0.5 | | | | | | | | | TechUGCourses < 5.5 | | | | | | | | | | Degree < 3.5 : F (5/0) | | | | | | | | | | Degree >= 3.5 | | | | | | | | | | | TechUGCrses < 2 : A (1/0) | | | | | | | | | | | TechUGCrses >= 2 : F (1/0) | | | | | | | | | TechUGCrses >= 5.5 | | | | | | | | | | Degree < 3.5 | | | | | | | | | | | TechUGCrs < 10.5 : A (3/0) | | | | | | | | | | | TechUGCrses >= 10.5 | | | | | | | | | | | | TechUGCrs<12.5 : F (3/0) | | | | | | | | | | | | TechUGCrses >= 12.5 | | | | | | | | | | | | | TechUGCrs<16: A (2/0) | | | | | | | | | | | | | TechUGCrs>15 : A (2/1) | | | | | | | | | | Degree >= 3.5 : F (1/0) | | | | | | | HardProjMgmt Exp >= 0.13 : A (2/0) | | | | | | TechUGCourses >= 18 : A (2/0) | | | | | Hard Proj Mgmt Exp >= 0.75 : F (1/0) | | | | TechUGCourses >= 23 : F (5/0) | | | No Of Hardware Proj Est >= 3 : F (1/0) | | No Of Hardware Proj Est >= 4.5 : A (5/0) | Hardware Proj Mgmt Exp >= 6 : F (4/0) TechUGCrses >= 45.5 : A (2/0) The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop BEST OVER <-- classified as 23 2 | BEST 8 17 | OVER
- 22. Conclusions Very Good accuracy rates, especially after attribute reduction Bridges expert and model groups http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop
- 23. http://nas.cl.uh.edu/boetticher/publications.html Questions? The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop
- 24. http://nas.cl.uh.edu/boetticher/publications.html Thank You ! The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop
- 25. References Jorgensen, M., “A review of studies on Expert Estimation of Software Development Effort,” Journal of Systems and Software, 2004. J ø rgensen, Shepperd, A Systematic Review of Software Development Cost Estimation Studies, IEEE Transactions on Software Engineering, 33, 1, January, 2007, Pp. 33-53. The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop http://nas.cl.uh.edu/boetticher/publications.html