Effective course evaluation
- 1. Effective Course Evaluation Summary report from interviews among senior academics and student representatives Improving response rates Improving student feedback Improving turnaround time Improving survey administration Improving the student experience
- 2. Effective Course Evaluation Reality is: There is a problem! Its 2012 IT /process improvement have had a go Where do institutions go from here?
- 3. Improving response rates – from the report Many universities seeking feedback on courses and lecturers via surveys struggle to achieve a meaningful response from students, which is partly due to students being inundated with requests to complete surveys. In-class student involvement in survey administration can increase commitment as they are stakeholders in the process. “A key issue facing universities in relation to gaining effective course feedback from students is around achieving significant response rates to any questionnaire-based approach. Students are increasingly overloaded with surveys - from the National Student Survey downwards - so you need a good reason to encourage participation.” Professor Glenn Burgess, Pro-Vice-Chancellor of Learning and Teaching, University of Hull
- 4. Improving response rates – from the report Group work: Why is obtaining a high response rate important?
- 5. Survey Best Practice rates – from our experience Improving response Is it a key challenge to achieve or maintain high response rates? 81% Strongly agree 12% Neutral 6% Disagree Population: 150 university administration staff at 70 institutions (UK) 5
- 6. Improving response rates – from our experience Why is obtaining a high response rate important? Survey Sample General student satisfaction survey Sample university student population The University Online survey 20% response rate Draw conclusions
- 7. Improving response rates – from our experience Why is obtaining a high response rate important? Survey Sample Course/module/seminar evaluation 10 – 30 students The Course 20% response rate Statistically valid?? Useful organisationally??
- 8. Survey Best Practice In-class Most likely to complete Satisfied Dissatisfied +30% higher response with in-class over email/VLE online surveys 8
- 9. Survey Best Practice Out of class Most likely to complete Satisfied Dissatisfied “UK Students: The most distracted by social media in the world” - Sept 2011 Telegraph 9
- 10. Improving student feedback – from the report Student representatives have indicated that students are not effectively engaged in the feedback process and, for some, providing feedback can even be intimidating. Universities need to work harder at feeding back to students the actions they will be taking as a result of input provided for course and lecturer evaluation surveys. “Students need to provide feedback, then the universities need to take that on board and do what they can with it, and then provide clear feedback to the students on the actions and outcomes” Alex Smith, former Academic Affairs Officer, University of Leicester Students‟ Union, and member of Quality Assurance Agency Board of Directors.
- 11. Improving student feedback – from the report Group work: How would you improve engagement in student feedback
- 12. Improving student feedback – from our experience Manage feedback „buy in‟ for all stakeholders Student Lecturer School T&L Executive
- 13. Improving turnaround time – from the report Ideally students want the opportunity to express their views on course improvements at a time that their feedback benefits them directly. End-of-module evaluation is a particular stumbling block in the provision of feedback to students - and feedback can be slow - but moving to mid-module evaluation can help to improve the process. “Students should be able to express how their course could be improved while it impacts on them.” Alex Bols, Head of Education and Quality, National Union of Students
- 14. Improving turnaround time – from the report Group work: How would you improve turn around time for module feedback
- 15. Improving turnaround time – from the report What survey methodologies are used at your institution? 9% Paper only 9% Online only 12% Hand-Key & Online 3% Hand tabulation 77% Paper & online Population: 150 university administration staff at 70 institutions 15
- 16. Improving turnaround time – from our experience Advantages Instant feedback Low deployment costs High response rate Easy for administration Easy for participant Location independent Onlin Paper Lower risk Multi-delivery modes e Delayed feedback Higher deployment costs Low response rate More administration More effort for Sources of error participant Environmental concerns Higher risk Disadvantages
- 17. Improving turnaround time – from our experience Advantages Instant feedback Low deployment costs High response rate Easy for administration Easy for participant Location independent Lower risk Multi-delivery modes Out of In class class Delayed feedback Higher deployment costs Low response rate More administration More effort for Sources of error participant Environmental concerns Higher risk Disadvantages
- 18. Improving survey administration– from the report Universities need to embrace new technologies to improve turnaround time - but effective feedback can be gained via a combination of paper and online surveys. “There should be a unified approach that includes core questions and specifies the way the feedback is processed - which allows benchmarking and consistency. A common set of core questions should be owned institutionally, with subject areas able to select from a bank of optional additional questions.” Professor Alan Speight, Pro-Vice-Chancellor (Student Experience and Academic Quality Enhancement), University of Swansea, and member of Quality Assurance Agency Board of Directors.
- 19. Improving survey administration– from the report Group work: (Module/Course level) Academic led vs. centralised module surveys – which is best?
- 20. Improving survey administration– from our experience Who manages the survey process at your organisation? 50% 65% 65% Centrally 47% Department or curriculum area 47% 50% Individual instructors Population: 150 university administration staff at 70 institutions 20
- 21. Improving survey administration– from our experience Who has the main responsibility for analysis and reporting at your organisation? 15% 59% Centrally 59% 38% 35% Department or curriculum area 38% Individual instructors 35% 15% Institutional Research Population: 150 university administration staff at 70 institutions 21
- 22. Efficiency and costadministration– from our experience Improving survey savings Individual Academic with 5 courses = 1 day per semester (Survey Design: 2hrs / Deployment: 1hr / Data Capture: 1hr / Analysis: 4 hrs) 22
- 23. Efficiency and costadministration– from our experience Improving survey savings Department with 50 academics = 52 days per semester (Individual survey admin: 50 days / Aggregate departmental analysis 2 days) 23
- 24. Efficiency and costadministration– from our experience Improving survey savings Arts Business Engineering School Law School Languages Sciences Average University (academic run survey admin): 20 departments / 1000 instructors = 1040 days per semester 24
- 25. Efficiency and costadministration– from our experience Improving survey savings Best practice example: London School of Economics 23 Departments /1000+ staff / 9000 students Complete survey administration 15 days per semester (Deployment: 4 days / Data Capture: 10 days / Reporting: 1 day) 25
- 26. Improving the student experience – from the report Universities should establish a more consistent (centralised) approach to survey administration - including a standard set of survey questions - to enable effective benchmarking at course and institutional level. However, individual departments should have the flexibility to include bespoke questions for particular courses. “With the National Student Survey, and the introduction of „Key Information Sets‟, it is vital that there is an effective course and lecturer feedback method so there can be constant improvements that will determine a student‟s opinion of their institution based on direct experience.” Professor David Coates, Dean of School of Life Sciences‟ Learning and Teaching Division, University of Dundee.
- 27. Improving the sutdent experience– from the report Group work: (Module/Course level) How can module evaluation enhance the student experience
- 28. Improving the student experience – from our experience Centralise d survey unit (e.g. NSS, PTES, PRES, module etc) Common survey/evaluation data gathering (standardise format, reports etc) Define a core set of questions for course evaluations Optional questions for departments but aim to use as few questionnaires as possible Use course evaluation data as a means to improve the student experience
- 29. Summary – from the report “universities will be expected to publish online summary reports of student surveys of lecture courses, aiding choice and stimulating competition between the best academics” Higher Education White Paper, June 2011 Effective course evaluation is necessary for universities to provide a clear evidence base to demonstrate their „value‟ to students.
- 30. Summary – from our experience Use in class time (paper or online) Target 50% response to maintain confidence and credibility of results Avoid incentivising students to complete surveys Avoid withholding grades Turnaround feedback to all stakeholders before the course finishes Tie results to action planning and improvements Gain stakeholder buy-in as the key to participate - „Myth of Survey Fatigue‟ Use „you said – we did‟ reporting Brand the process of giving feedback Ask students to evaluate their own effort on the course Reinforce the concept of anonymity to get honest feedback
Editor's Notes
- #6: Amend green thingAmend arguments
- #9: Amend green thingAmend arguments
- #10: Amend green thingAmend arguments
- #16: Amend green thingAmend arguments
- #21: Amend green thingAmend arguments
- #22: Amend green thingAmend arguments
- #23: Amend green thingAmend arguments
- #24: Amend green thingAmend arguments
- #25: Amend green thingAmend arguments
- #26: Amend green thingAmend arguments