Evaluation tips and tools from 40+ Technology Enhance Learning projects
Download as PPT, PDF3 likes1,094 views
The document discusses the importance and methods of evaluation in technology-enhanced learning (TEL) projects, emphasizing stakeholder involvement and evidence generation to inform future practices. It outlines various evaluation approaches, stages, and tools that can be utilized to assess curriculum design and enhancements in assessment and feedback practices. Additionally, it reflects on the OCTEL experience, highlighting participant engagement and outcomes.
Evaluation tips and tools from 40+ Technology Enhance Learning projects
- 1. Evaluation tips and tools from 40+ TEL projects Dr Rachel A Harris rachel@inspire-research.co.uk ocTEL week 10 webinar 20th June 2013
- 2. Inspire Research Ltd Session outline • Why evaluate? • Approaches used in Jisc Curriculum Design & Delivery and Assessment & Feedback programmes • Some methods and tools. • Thinking about evaluation design. • Reflecting on the ocTEL experience – ALT, Tutors, Participants!
- 3. Inspire Research Ltd Why evaluate? 1. Involve stakeholders – enable them to express needs and provide feedback. • Students are “the best quality controllers. Give them every opportunity to feedback and report where improvements can be made” (eBiolabs). 1. Inform the ongoing development of an innovation. • Identifying if a change is working as expected, or if scaling up or out to other disciplines is beneficial. • Understanding why TEL innovations do or don’t work. 1. Build evaluation capacity and skills.
- 4. Inspire Research Ltd Even more reasons to evaluate 4. Generate evidence of what works to inform the wider sector. • identifying "what can be learned from this program's experiences and results to inform future efforts" (Patton, 1997). • providing a “powerful lever in changing others' behaviour by changing their knowledge of what is possible and effective” (Draper & Nicol, 2006). 4. Provide evidence to support local embedding or transfer to other disciplines. • “These efficiency savings, and our ability to quantify them, have been useful … to make the case to fully embed and sustain these services going forward” (Cascade project).
- 6. Inspire Research Ltd Evaluation approaches wordle What we can learn from the Curriculum Design projects about evaluation Evaluation synthesis: Learning the lessons through evaluation and research, a story of variety in scope, approach and application
- 7. Inspire Research Ltd Initial evaluation stages • Envision potential impact /achievements: – potential indicators of impact/achievement, – credible evidence of impact, from different stakeholders’ perspectives. • Baseline: – first set of evaluation data – review the current context – identify existing practice – determine stakeholders’ attitudes – clarify challenges (and identify barriers)
- 8. Inspire Research Ltd A multitude of approaches • Action research (Atelier-D, Duckling, KUBE) • Independent int/external evaluator (Cascade) • Appreciative inquiry (ESCAPE) Appreciative Inquiry & Fish bone analysis for forward planning eAFFECT project’s Appreciative Inquiry Workshop Script • Balanced scorecard (COWL) • Formative evaluation (MAC) • Micro & macro data sources (Springboard TV) • Rich qualitative methods (ISCC) • Multi-method (PiP) A model for evaluating technology-supported curriculum design and approval initiatives
- 9. Inspire Research Ltd Assessment & Feedback Programme The programme focuses on “large-scale changes in assessment and feedback practice, supported by technology “with the aim of enhancing the learning and teaching process and delivering efficiencies and quality improvements”. – Pedagogic enhancements – Workload efficiencies – Building the business case
- 10. Inspire Research Ltd Methods & Tools for Evaluating Assessment Innovations Assessment Experience Questionnaire AEQ Gibbs (2007) FASTECH Baseline Survey of Technology -in-use Evaluation of Electronic Voting Systems Student Questionnaire EEVS Learning & Assessment Analytics (CETIS) Self-evaluation of written feedback for teachers Questionnaire Feedback Analysis Chart for Tutors (FACT) S creencast overview Think-aloud participant observation technique Time motion studies (Cascade evaluation report) Map my programmeAssessment patterns HE Academy Assessment Audit Tool HEA Employability audit tool
- 11. Inspire Research Ltd Other tools and methods?
- 12. Inspire Research Ltd Evaluation design
- 13. Inspire Research Ltd Designing your evaluation (I) 1. What do you want to happen as a result of your TEL project activities/course? (Inputs Outputs Outcomes & Impact) 2. Who and/or what might your activities impact on? (Stakeholders) 3. How will you know you have achieved your intended outcomes? Or What indicators could you use to demonstrate what you have achieved?
- 14. Inspire Research Ltd Designing your evaluation (II) • How will you find out whether you have achieved your intended outcomes? (What evidence, How collected and When?)
- 15. Inspire Research Ltd Any questions?
- 16. Inspire Research Ltd Reflecting on the ocTEL experience
- 17. Inspire Research Ltd ALT’s view of ocTEL?ALT’s view of ocTEL?
- 18. Inspire Research Ltd The tutor perspectiveThe tutor perspective
- 19. Inspire Research Ltd The tutor perspectiveThe tutor perspective • Supporting discussions • Supporting different ‘MOOC’ spaces • Lurkers? • Engaging with ocTEL • Knowledge generation within MOOCs • Risk-taking and fun
- 20. Inspire Research Ltd What was your experience?
- 21. Inspire Research Ltd Some highlights – ocTEL in numbers • Registrations: 903 on 3rd April • Email list: 712 members at minimum on 6th April, now 915 members • Discussion: 1,326 forum posts, 574 blog posts • Webinar participation: 102 max, 17 min • Tutors: 34 volunteered, 20 contributed to at least one week.
- 22. Inspire Research Ltd What were your highlights?
- 23. Inspire Research Ltd A big thank you!
Editor's Notes
- #5: Patton, M.Q. (1997) Utilization-focused evaluation: the new century text. 3rd Edition London: Sage Publications Ltd. Draper, S. & Nicol, D. (2006) Transformation in e-learning. Short paper delivered at ALT-C 2006
- #6: http://collections.infocollections.org/ukedu/en/d/Jgpimse/3.1.html
- #7: Curriculum Delivery projects set out to “transform how they deliver and support learning across a curriculum area through the effective use of technology”. http://jiscdesignstudio.pbworks.com/w/page/40379712/Transforming Curriculum Delivery through Technology Curriculum Design projects developed innovative technology-supported approaches to curriculum design, approval and review. Emphasis on gathering and using evidence Identifying what works Adding value by generating evidence http://jiscdesignstudio.pbworks.com/w/page/40489793/Institutional%20Approaches%20to%20Curriculum%20Design http://www.wordle.net/show/wrdl/6282822/Evaluation_approaches_from_CDD_projects An overview on the design studio of what we can learn from the curriculum design projects about evaluation http://jiscdesignstudio.pbworks.com/w/page/61872351/What%20the%20Curriculum%20Design%20projects%20tell%20us%20about%20evaluation
- #8: The traditional view would be that by describing the current position, the baseline can “provide a sound basis on which the success of the project could later be evaluated” (Cascade). However, a baseline study can also be seen as the start of the essential project management requirement “to be aware of all factors which may impact positively or negatively on the effectiveness of the project” (KUBE).
- #9: Atelier-D Achieving Transformation, Enhanced Learning & Innovation through Educational Resources in Design, Open University Investigated web technologies for developing a virtual design studio space to support student learning throughout the Design programme of the Open University. CASCADE University of Oxford Implemented new strategies to improve curriculum delivery models to allow the University of Oxford’s Department for Continuing Education to respond more flexibly to stakeholders’ needs. COWL Coventry Online Writing Laboratory, Coventry University Developed and extended the pedagogy, reach and diversity of academic writing services, through a technology-enhanced teaching and learning environment. DUCKLING Delivering University Curricula: Knowledge, Learning & Innovation Gains, University of Leicester Developed delivery, presentation and assessment processes to enhance the work-based learning experience of students studying remotely. ESCAPE Effecting Sustainable Change in Assessment Practice & Experience, University of Hertfordshire Responding to National and Institutional concerns regarding assessment and feedback, the project worked with two Schools to develop assessment for learning activities to enhance the assessment experience for learners and staff. ISCC Information Spaces for Creative Conversations, Middlesex University and City University Addressed a recurrent problem in design education of students sometimes being disengaged from key creative conversations, a problem that can be exacerbated by learning technologies. KUBE Kingston Uplift for Business Education, Kingston College Set out to enhance the learning experience of students studying on higher-level business education programmes delivered at Kingston College on behalf of Kingston University. MAC Making Assessment Count, University of Westminster Enhanced curriculum delivery through the development of an innovative assessment feedback process, eReflect. Springboard TV : An internet TV station to enrich teaching & learning, College of West Anglia Set out to address challenges associated with recruitment, learner satisfaction, engagement, progression and employability by designing an innovative learner journey delivered in a simulated TV production and broadcast environment.
- #10: The programme call notes the focus “is on large-scale changes in assessment and feedback practice, supported by technology, with the aim of enhancing the learning and teaching process and delivering efficiencies and quality improvements”. Projects will be expected to address (at least some of) these areas in their evaluation activities. This will feed into the development of an Evaluation and Synthesis Framework,
- #11: AEQ http://www.fastech.ac.uk/index.php/resources/research-toolkit/aeq Tech-in-use http://www.fastech.ac.uk/index.php/resources/research-toolkit/baseline-survey Electronic voting questionnaire http://jiscdesignstudio.pbworks.com/w/page/48734953/EEVS Project CETIS Analytics Series http://publications.cetis.ac.uk/c/analytics Self-evaluation of written feedback for teachers http://www.open.ac.uk/fast/pdfs/SelfEval%20feedback%20practice.pdf Developed by the Formative Assessment in Science Teaching (FAST) project. Feedback Analysis Chart for Tutors (FACT) Screencast overview http://www.screencast.com/t/fsELBHbWuy8C Think-aloud participant observation technique Cotton D, Gresty, K. Reflecting on the think-aloud method for evaluating e-learning. British Journal of Educational Technology, Vol. 37 No 1, 2006. p 45-54 Time motion studies http://www.mapmyprogramme.com/ http://jiscdesignstudio.pbworks.com/w/page/30631817/ESCAPE%20-%20Assessment%20timelines HE Academy Assessment audit tool http://www.bioscience.heacademy.ac.uk/ftp/resources/audit/assessment.pdf HEA Employability audit http://www.bioscience.heacademy.ac.uk/ftp/resources/audit/employability.pdf
- #13: http://xkcd.com/568/
- #14: Remembering that it may not be possible to evaluate the full range of project activities. (How might you investigate why the impact occurred?)
- #15: http://freshspectrum.com/how-to-not-really-evaluate-in-3-cartoons/
- #16: Graphic downloaded from the http://freshspectrum.com/settling-for-the-next-best-thing/