ICSE12 SEE.ppt
1 like409 views
The document summarizes a five-day empirical software engineering school held in Montreal. It provided 44 graduate students from 9 countries hands-on training in experiment design, mining software repositories, and building prediction models from collected data. The school used a learn-by-doing approach through example studies, labs analyzing real data sets, and feedback from lecturers and keynote speakers. Participants gained experience planning experiments, collecting and analyzing various types of data to build models and draw conclusions, with the goal of providing skills often missing from traditional university curricula. Feedback suggested expanding certain labs and tutorials while guidelines could help with appropriate research conduct.
![Lab Script Example
#UNPAIRED ANALYSIS
#Analysis of single experiment
#Mann-Whitney
attach(tbn)
wilcox.test(Correct[Fit=="yes"],Correct[Fit=="no"],paired=FALSE,alternative="greater")
attach(ttrento)
wilcox.test(Correct[Fit=="yes"],Correct[Fit=="no"],paired=FALSE,alternative="greater")
attach(tphd)
wilcox.test(Correct[Fit=="yes"],Correct[Fit=="no"],paired=FALSE,alternative="greater")
#All data
attach(t)
wilcox.test(Correct[Fit=="yes"],Correct[Fit=="no"],paired=FALSE,alternative="greater")
#Exercises:
# 1) perform a two-tailed test
# 2) can t-test be applied instead of Wilcoxon test? test for data normality using the Wilk-Shapiro test
# 3) repeat the analysis using the t-test?
# 4) repeat the analysis for the time dependent variable](https://image.slidesharecdn.com/see12-ppt-130412090548-phpapp02/85/ICSE12-SEE-ppt-21-320.jpg)
ICSE12 SEE.ppt
- 1. Five Days of Empirical Software Engineering: The PASED Experience Massimiliano Di Penta Giuliano Antoniol Daniel M. Germán Yann-Gaël Guéhéneuc Bram Adams
- 2. Motivation Empirical background important for graduate students Courses on statistics insufficient to provide such a background (Most) University curricula could not afford do have specific courses Some exceptions (there are others for sure!): • Easterbrook’s CSC2130: Empirical Research Methods in Software Engineering at the University of Toronto (2009) • Herbsleb’s 08-803: Empirical Methods for Socio-Technical Research at CMU (2010) • Dewayne E. Perry course, Univ. of Texas at Austin
- 3. So students need that!
- 4. General Info About the School • Ecole Polytechnique de Montréal, June 2011 • Funded by MITACS • low fee for students $250 all included • 44 participants, 9 countries and 25 different institutions • More on http://pased.soccerlab.polymtl.ca
- 5. Learning Objectives 1. plan and conduct software engineering experiments with human subjects and collect related data 2. plan and conduct software engineering studies involving the mining of data from (un)structured software repositories 3. build prediction and classification models from the collected data, and to use these models
- 6. Challenges Choosing Combining Topics theory and practice Dealing with heterogeneous participants
- 7. What Topics?
- 8. What Topics? Planning the study
- 9. What Topics? Getting the data Planning the study
- 10. What Topics? Getting the data Planning the Analyzing study results
- 11. Only 5 days.... ...that’s too much!
- 12. The Approach • Learn by example and by doing format • Experiment design principles and statistics introduced by presenting cases from studies in literature • Practical application of theoretical concepts during labs • Course material and laboratory packages available online, including course videos
- 13. School Content Mining Exp. Text Statistical Predictor AM Software Design mining Analysis Models Archives Keynote Keynote Keynote Keynote Keynote PM Hands Hands Hands Hands Hands on on on on on lab lab lab lab lab
- 15. from the school Running example I • Use of UML Stereotypes in comprehension and maintenance tasks • Filippo Ricca, Massimiliano Di Penta, Marco Torchiano, Paolo Tonella, Mariano Ceccato: How Developers' Experience and Ability Influence Web Application Comprehension Tasks Supported by UML Stereotypes: A Series of Four Experiments. IEEE Trans. Software Eng. 36(1): 96-118 (2010) • Filippo Ricca, Massimiliano Di Penta, Marco Torchiano, Paolo Tonella, Mariano Ceccato: The Role of Experience and Ability in Comprehension Tasks Supported by UML Stereotypes. ICSE 2007: 375-384 • In the following briefly referred as “Conallen”
- 16. from the school Experiment Design Group 1 Group 2 Group 3 Group 4 Lab 1 Sys A Sys A Sys B Sys B Lab 2 Sys B Sys B Sys A Sys A
- 17. from the school Data format: example
- 18. from the school Boxplots: Conallen
- 19. from the Paired analysis: example school ID F.Conallen F.UML >wilcox.test(F.Conallen,F.UML, paired=TRUE, T20 0.74 0.74 alternative="greater") T21 0.74 0.51 T22 0.7 0.29 T24 0.88 0.62 Wilcoxon signed rank test with continuity correction T25 0.75 0.8 T26 0.66 0.39 T27 0.35 0.51 data: F.Conallen and F.UML T28 0.62 0.59 T29 0.57 0.68 V = 138, p-value = 0.04354 T30 0.73 0.43 alternative hypothesis: true location shift is greater than 0 T32 0.74 0.56 • Must have data in a paired format • or you can use a proper R script • Need to remove subjects that took part to one lab only • For parametric statistics just replace wilcox.test with t.test
- 20. Hands on Labs • Mining software repository challenge: extract interesting facts from git • Experiment design: groups working together on designing a study • Data analysis: text mining, analyze working data sets of previous experiments and build bug predictors • Working data sets from previous experiments, PROMISE data sets • Tools: R and Weka
- 21. Lab Script Example #UNPAIRED ANALYSIS #Analysis of single experiment #Mann-Whitney attach(tbn) wilcox.test(Correct[Fit=="yes"],Correct[Fit=="no"],paired=FALSE,alternative="greater") attach(ttrento) wilcox.test(Correct[Fit=="yes"],Correct[Fit=="no"],paired=FALSE,alternative="greater") attach(tphd) wilcox.test(Correct[Fit=="yes"],Correct[Fit=="no"],paired=FALSE,alternative="greater") #All data attach(t) wilcox.test(Correct[Fit=="yes"],Correct[Fit=="no"],paired=FALSE,alternative="greater") #Exercises: # 1) perform a two-tailed test # 2) can t-test be applied instead of Wilcoxon test? test for data normality using the Wilk-Shapiro test # 3) repeat the analysis using the t-test? # 4) repeat the analysis for the time dependent variable
- 22. Calibrating Courses to Participants’ Profiles Excellent 1 Excellent 4 Good 21 Good 20 Basic 23 Basic 18 None 1 None 2 0 10 20 30 0 10 20 30 Statistical Analyses Empirical Sw Engineering Excellent 4 Excellent 1 Good 16 Good 12 Basic 24 Basic 25 None 1 None 7 0 10 20 30 0 10 20 30 Mining Sw repositories Machine learning
- 23. Feedbacks Longer Guidelines on Labs what not to do Tutorials How to write on tools empirical papers
- 24. Acknowledgments • Lecturers (other than the paper authors): • Ahmed E. Hassan, Queen’s University, Canada • Andrian Marcus, Wayne State University, USA • Keynote Speakers • Gail Murphy, University of British Columbia, USA • Prem Devanbu, UC Davis, USA • Alain Picard, Benchmark Consulting Services Inc., Canada • Maria Codipiero, Peter Colligan, Kal Murtaia, SAP, Canada • Marc-André Decoste, Google Montréal, Canada • Student volunteers • The attendees! • MITACS (http://www.mitacs.ca/)
- 25. Conclusions