Beyond Surveys: Analyzing Software Development Artifacts to Assess Teaching Efforts
1 like151 views
The document discusses an undergraduate software engineering capstone course at the Hasso Plattner Institute, focusing on managing large software projects and exploring the impact of different development methodologies like Scrum and Kanban. It emphasizes the need for assessing curriculum changes through the analysis of software development artifacts rather than traditional surveys, revealing mixed success in implementing Kanban. Feedback indicates that while students appreciated Kanban, it led to fewer significant changes in developer engagement and project dynamics.
![“
Background
Course Focus
You will learn how to manage a long-running software
project with a large number of developers. [1]
2[1] https://hpi.de/plattner/teaching/archive/winter-term-201718/softwaretechnik-ii.html
”
An undergraduate software engineering capstone course](https://image.slidesharecdn.com/fie18matthies-181005220532/85/Beyond-Surveys-Analyzing-Software-Development-Artifacts-to-Assess-Teaching-Efforts-2-320.jpg)
![“
Background
Course Focus
You will learn how to manage a long-running software
project with a large number of developers. [1]
3[1] https://hpi.de/plattner/teaching/archive/winter-term-201718/softwaretechnik-ii.html
■ All participants (in teams) jointly develop a software
■ Self-organizing teams
■ Collaboration > technical skills
”
An undergraduate software engineering capstone course](https://image.slidesharecdn.com/fie18matthies-181005220532/85/Beyond-Surveys-Analyzing-Software-Development-Artifacts-to-Assess-Teaching-Efforts-3-320.jpg)
![“
Background
Course Focus
You will learn how to manage a long-running software
project with a large number of developers. [1]
4[1] https://hpi.de/plattner/teaching/archive/winter-term-201718/softwaretechnik-ii.html
■ All participants (in teams) jointly develop a software
■ Self-organizing teams
■ Collaboration > technical skills
■ Project work & intro exercises & lectures & tutors
■ Learn and apply agile methods
■ Real-world scenario, real development tools
”
An undergraduate software engineering capstone course](https://image.slidesharecdn.com/fie18matthies-181005220532/85/Beyond-Surveys-Analyzing-Software-Development-Artifacts-to-Assess-Teaching-Efforts-4-320.jpg)
![Challenges & Ideas
■ Course employed Scrum
■ Structured, prescriptive
■ Good for starting [2]
■ Kanban gaining popularity in industry
5
Evaluate and adapt the course over time
[2] V. Mahnic, “From Scrum to Kanban: Introducing Lean
Principles to a Software Engineering Capstone Course”](https://image.slidesharecdn.com/fie18matthies-181005220532/85/Beyond-Surveys-Analyzing-Software-Development-Artifacts-to-Assess-Teaching-Efforts-5-320.jpg)
![Challenges & Ideas
■ Course employed Scrum
■ Structured, prescriptive
■ Good for starting [2]
■ Kanban gaining popularity in industry
■ Idea: Update course and project!
■ Employ Scrum Sprints first, then switch to Kanban
■ Kanban for “finishing touches”
6
Evaluate and adapt the employed development methodology
[2] V. Mahnic, “From Scrum to Kanban: Introducing Lean
Principles to a Software Engineering Capstone Course”](https://image.slidesharecdn.com/fie18matthies-181005220532/85/Beyond-Surveys-Analyzing-Software-Development-Artifacts-to-Assess-Teaching-Efforts-6-320.jpg)
Beyond Surveys: Analyzing Software Development Artifacts to Assess Teaching Efforts
- 1. Hasso Plattner Institute University of Potsdam, Germany christoph.matthies@hpi.de @chrisma0 Beyond Surveys: Analyzing Software Development Artifacts to Assess Teaching Efforts Christoph Matthies, Ralf Teusner, Guenter Hesse ’18, San Jose, CA, October 2018
- 2. “ Background Course Focus You will learn how to manage a long-running software project with a large number of developers. [1] 2[1] https://hpi.de/plattner/teaching/archive/winter-term-201718/softwaretechnik-ii.html ” An undergraduate software engineering capstone course
- 3. “ Background Course Focus You will learn how to manage a long-running software project with a large number of developers. [1] 3[1] https://hpi.de/plattner/teaching/archive/winter-term-201718/softwaretechnik-ii.html ■ All participants (in teams) jointly develop a software ■ Self-organizing teams ■ Collaboration > technical skills ” An undergraduate software engineering capstone course
- 4. “ Background Course Focus You will learn how to manage a long-running software project with a large number of developers. [1] 4[1] https://hpi.de/plattner/teaching/archive/winter-term-201718/softwaretechnik-ii.html ■ All participants (in teams) jointly develop a software ■ Self-organizing teams ■ Collaboration > technical skills ■ Project work & intro exercises & lectures & tutors ■ Learn and apply agile methods ■ Real-world scenario, real development tools ” An undergraduate software engineering capstone course
- 5. Challenges & Ideas ■ Course employed Scrum ■ Structured, prescriptive ■ Good for starting [2] ■ Kanban gaining popularity in industry 5 Evaluate and adapt the course over time [2] V. Mahnic, “From Scrum to Kanban: Introducing Lean Principles to a Software Engineering Capstone Course”
- 6. Challenges & Ideas ■ Course employed Scrum ■ Structured, prescriptive ■ Good for starting [2] ■ Kanban gaining popularity in industry ■ Idea: Update course and project! ■ Employ Scrum Sprints first, then switch to Kanban ■ Kanban for “finishing touches” 6 Evaluate and adapt the employed development methodology [2] V. Mahnic, “From Scrum to Kanban: Introducing Lean Principles to a Software Engineering Capstone Course”
- 7. Challenges & Ideas 7 Scrum: the usual agile development process Scrum key ideas ■ Sprints: timeboxed iterations ■ Planning and estimation ■ Review and retrospectives ■ Prescriptive process
- 8. Challenges & Ideas 8 Kanban: the new kid on the block Kanban key ideas ■ Visualize work items on Kanban board ■ “Pull” them through the process ■ Limit work-in-progress ■ Handle bottlenecks
- 9. Research Question 9 Goals of the research and background How can we gauge (the effect of curriculum) changes in student behavior during project work?
- 10. Research Question ■ Usual approach: end-of-term surveys 10 Goals of the research and background How can we gauge (the effect of curriculum) changes in student behavior during project work?
- 11. ■ Performed regularly after end of course (before grades) ■ Allows student feedback on the course ■ Standardized questions, overwhelmingly positive responses → Hard to gauge changes in curriculum design End-of-term Surveys 11 General satisfaction indicators of the last 5 years
- 12. Research Question ■ Usual approach: end-of-term surveys 12 Goals of the research and background How can we gauge (the effect of curriculum) changes in student behavior during project work?
- 13. Research Question ■ Usual approach: end-of-term surveys ■ Programming project provides unique opportunity ■ Developers regularly produce software artifacts ■ GitHub: Version control system, issue tracker 13 Goals of the research and background How can we gauge (the effect of curriculum) changes in student behavior during project work?
- 14. ■ Collected data from the last 5 course instalments ■ Last 3 course iterations introduced Kanban ■ The 2 before these used only Scrum Development Artifact Collection 14 Comparing development artifacts over course instalments
- 15. ■ Collected data from the last 5 course instalments ■ Last 3 course iterations introduced Kanban ■ The 2 before these used only Scrum Crawled GitHub APIs and extracted artifacts Development Artifact Collection 15 Comparing development artifacts over course instalments commits tickets / user stories
- 16. Development Artifact Analysis Kanban usage had some noticeable effects 16 Gaining insights into team behaviors
- 17. Development Artifact Analysis Kanban usage had some noticeable effects ■ Higher mean amount of non-comment events ■ Assigning labels (status, priority) & developers (responsibility) → More interaction with issues 17 Gaining insights into team behaviors
- 18. Development Artifact Analysis Kanban usage had some noticeable effects ■ Higher mean amount of non-comment events ■ Assigning labels (status, priority) & developers (responsibility) → More interaction with issues ■ Commits towards end of Sprint higher in Kanban ■ Scrum (planning) vs Kanban (dynamic) → Better ability to adapt to changes 18 Gaining insights into team behaviors
- 19. Development Artifact Analysis Key development artifacts measures did not change significantly ■ Mean amount of commits & touched files ■ Mean line changes per commit ■ Mean amount of unique issues referenced ■ Mean issues closed, mean comments 19 Gaining insights into team behaviors
- 20. Development Artifact Analysis Hypotheses regarding changes in artifacts were violated 20 Gaining insights into team behaviors
- 21. Development Artifact Analysis Hypotheses regarding changes in artifacts were violated ■ Similar percentage of issues opened and closed by same person ■ No dedicated Product Owner role → Expected higher engagement of entire team 21 Gaining insights into team behaviors
- 22. Development Artifact Analysis Hypotheses regarding changes in artifacts were violated ■ Similar percentage of issues opened and closed by same person ■ No dedicated Product Owner role → Expected higher engagement of entire team ■ No change in user story length ■ No need to estimate, focus on throughput → Expected smaller user stories 22 Gaining insights into team behaviors
- 23. Kanban Survey Survey in 2017/18 course instalment (N=18, 5 point Likert scale) 23 Asking the questions that actually matter
- 24. Kanban Survey Survey in 2017/18 course instalment (N=18, 5 point Likert scale) ■ “Was the Kanban sprint more useful and productive than another Scrum sprint?” ■ Yes!, mean 4.08 24 Asking the questions that actually matter
- 25. Kanban Survey Survey in 2017/18 course instalment (N=18, 5 point Likert scale) ■ “Was the Kanban sprint more useful and productive than another Scrum sprint?” ■ Yes!, mean 4.08 ■ “Did you adapt your workflow?” ■ Yes., 3.83 25 Asking the questions that actually matter
- 26. Kanban Survey Survey in 2017/18 course instalment (N=18, 5 point Likert scale) ■ “Was the Kanban sprint more useful and productive than another Scrum sprint?” ■ Yes!, mean 4.08 ■ “Did you adapt your workflow?” ■ Yes., 3.83 ■ “Biggest (dis)advantages of Kanban?” (free text) ■ Advantages: Efficiency & Autonomy ■ Drawbacks: Only work on small stories, uneven task distribution 26 Asking the questions that actually matter
- 27. Kanban Survey Survey in 2017/18 course instalment (N=18, 5 point Likert scale) ■ “How did user stories change from using Scrum to Kanban?” ■ More bug-oriented (11 mentions) ■ Shorter (11 mentions) ■ With more detailed requirements (8 mentions) 27 Asking the questions that actually matter
- 28. Kanban Survey Survey in 2017/18 course instalment (N=18, 5 point Likert scale) ■ “How did user stories change from using Scrum to Kanban?” ■ More bug-oriented (11 mentions) ■ Shorter (11 mentions) ■ With more detailed requirements (8 mentions) ■ “Would you recommend using Kanban to next year’s participants?” ■ YES!, mean 4.33 28 Asking the questions that actually matter
- 29. Summary & Conclusion ■ Kanban introduction was liked by students, but w/ mixed success 29 Take-away messages christoph.matthies@hpi.de @chrisma0
- 30. Summary & Conclusion ■ Kanban introduction was liked by students, but w/ mixed success ■ Development artifacts represent another dimension of analysis ■ Beyond the perceptions of students ■ Based on data naturally produced, high “response rate” 30 Take-away messages christoph.matthies@hpi.de @chrisma0
- 31. Summary & Conclusion ■ Kanban introduction was liked by students, but w/ mixed success ■ Development artifacts represent another dimension of analysis ■ Beyond the perceptions of students ■ Based on data naturally produced, high “response rate” ■ Analysis allowed finding those areas where expectations are… ■ Confirmed ■ Violated! (even more interesting) 31 Take-away messages christoph.matthies@hpi.de @chrisma0
- 32. Summary & Conclusion ■ Kanban introduction was liked by students, but w/ mixed success ■ Development artifacts represent another dimension of analysis ■ Beyond the perceptions of students ■ Based on data naturally produced, high “response rate” ■ Analysis allowed finding those areas where expectations are… ■ Confirmed ■ Violated! (even more interesting) → Opportunity for conversation and improvement 32 Take-away messages christoph.matthies@hpi.de @chrisma0
- 33. Image Credits 33 In order of appearance ■ Archaeologist by Gan Khoon Lay from the Noun Project (CC BY 3.0 US) ■ Mortar Board by Mike Chum from the Noun Project (CC BY 3.0 US) ■ Target by Arthur Shlain from the Noun Project (CC BY 3.0 US) ■ Process by Laymik from the Noun Project (CC BY 3.0 US) ■ Questions by Gregor Cresnar from the Noun Project (CC BY 3.0 US) ■ Data collection by H Alberto Gongora from the Noun Project (CC BY 3.0 US) ■ Search Code by icon 54 from the Noun Project (CC BY 3.0 US) ■ Clipboard by David from the Noun Project (CC BY 3.0 US) ■ Idea by Gilbert Bages from the Noun Project (CC BY 3.0 US)