CAA 2012 Closing Address

Editor's Notes

• #14: Assessment, as we all know, lies at the heart of the heart of the learning experience. Over the last couple of years JISC recognised the importance of this area in it being a key driver to change in institutions, as well the enormous potential technology has to transform assessment practices, proven by previous programmes of work including the Curriculum Design and Curriculum Delivery programmes which have recently completed. We have recently launched a new guide Effective Assessment in a Digital Age, which showcases some of the excellent work that has gone before, as well as run a number of national workshops taking this work into institutions. This programme builds on all of this past work, aiming to take us forward in new ways. Strand A: Institutional change. These projects are redesigning assessment and feedback processes and practices, making best use of technology to deliver significant change at programme, school or institutional level.(15 months – 2 years funded, £100-£200k per project)Strand B: Evidence and evaluation. You’ll hear more about these projects today in the next session. These projects are evaluating assessment-and feedback-related innovations which are already underway in a faculty or institution, and report on lessons for the sector.(6 months – 2 years, £20k per project) Strand C: Technology transfer. These projects are packageing up technology innovations in assessment and feedback for re-use (with associated processes and practice), and supporting its transfer to two or more named external institutions.(9 months – 2 years, £40k per project)Support projectA Support and Synthesis project led by JISC infoNet and in partnership with JISC-CETIS is co-ordinating a support and synthesis team to provide support to projects across all strands through a range of mechanisms including webinars, access to sector expertise and peer support through CAMEL networks.
• #15: These aims are reflected in the diagram which represents the goals of the 8 institutional change projects. Some will be addressing one of these but all are related....At the heart is the aim of enhancing learning and teaching practice, through assessment for learning - meeting students’ needs for well-timed, meaningful, authentic assessment experiences, backed up by feedback that increases their understanding of the topic. Better understanding of the criteria, goals and standards before an assignment is also involved, hence a focus on feed forward in some project outlines. Alongside the development of more integrated strategies, policies and processes, through technology acting as a catalyst for change across institutional strategies and processes, with the associated potential overhaul of design, review and approval processes. At the very least, e-learning strategies should be built into the institutions assessment strategy. One project has in its main aims to develop that institutional vision for assessment.Increased efficiency: Three aspects are focused on – a) Pedagogical gains from more efficient working practices e.g. Technology enhanced feedback, or greater opportunities for regular and frequent testing even in large-group contexts. b) Time saving. E.g through data being reused effectively to inform an institutional continuous quality improvement and from a learner perspective provide them with a better overview of the progress leading to increased self regulation. More productive use of staff and student time is perhaps the most frequently cited efficiency gain from the use of technology (FASTECH, InterACT, iTEAM, e-AFFECT ) but c) institutional quality assurance may also gain.Improved student learning: In line with principles for good assessment and feedback as developed by Nicol and McFarlane, Chickering and Gamson, Gibbs and Simpson, the quality of student learning is linked directly to the way assessment, and particularly feedback, are designed. InterACT focuses on the value of engaging students in an active response to feedback: ‘The system shifts responsibility to the student to engage with the feedback in a meaningful way and the outcome measures of time will determine the staff effort invested against measures of satisfaction, quality of reflection, progress through modules and attrition rates.’ Dialogue with self, peers and tutors is seen as a key ingredient and redesigns of feedback often make use of the interactive but asynchronous potential of technology-mediated communication. With greater alignment of assessment designs with employer and student needs (Collaborate, FAST, iTEAM), student satisfaction with assessment and feedback is also seen as likely to improve.
• #16: This is just a flavour of the technologies being used to enhance practice and process across the programme.These include general learning technologies, including the VLE, as well as specific assessment and feedback technologies.
• #17: Projects have identified a range of challenges and potential benefits for technology enhancement, ranging from 2 who have a clear focus on the timeliness and quality feedback, plus ensuring opportunities for reflection on, dialogue about, and engagement with feedback, looking at innovative ‘longitudinal’ ‘feedforward’ systems. Others are looking more at assessment management, and how technology can enhance and make more efficient process of submission, marking etc . Others are exploring design issues such as the timing and sequencing of assignments, with clearer articulation of criteria, goals and standardsAll change projects have been asked to identify and place at the heart of their change projects educational principles around good assessment and feedback practice, e.g. REAP.
• #19: Why did we do this? At the programme level: It is a way of validating the entire rationale for the programme – we want to know that there is a job that needs doing so, as far as we are concerned, it is great to identify that there are significant areas that need improvement. We want to know that we are addressing issues that are important to the sector as a whole rather than working in niche areas of limited wider application. We want to know that we aren’t reinventing the wheel and that the projects are building on earlier work. We want to develop a shared understanding about our aims and direction of travel.Note importance of ongoing dialogue with groups such as L&T Experts to continue to achieve these goals over the life of the project.
• #20: 1st bullet point repeated as it is important for strategic projects like these with major transformational goals to know where they sit in the wider landscape.Requirement to do this work based on experience from many other JISC projects that the exercise is useful.Reports back from A&F projects show that the dialogue itself is useful – in helping define the status quo and the issues, stakeholders are already beginning to be engaged and ready to take ownership of the solutions.And maybe most importantly, it is a way of capturing the state of play at a point in time, and providing a baseline measure from which change against the baseline can be evidenced. Although ongoing processAs a way of promoting discussion around the issues, it is also a means of creating a state of change-readiness.
• #21: This is just a collection of the different approaches taken to this exercise, and the breadth of evidence captured. These reflect the different areas explored, from strategy and policy as represented in formal documentation and processes, assessment and feedback processes, and practice in terms of course level evaluations, and the voices of staff and students. Projects were given guidance around capturing evidence relating to strategy and policy, process, infrastructure, stakeholders Illustrates richness of evidence base.Individuals and small groups identifiable in many cases hence anonymity of synthesis report.List can act as a source of ideas for anyone reviewing practice in their own institution and seeking to compare their findings with our summary.Some of these artefacts may also be useful to revisit as measures of change e.g. shifts in language used in formal documents may reflect cultural change.
• #22: These example wordles of the text of a couple of reports give the flavour of a couple of the most distinctive differences in approaches taken by projects:Some of the baselines show a lot of emphasis on formal/summative assessment and the processes and procedures that surround this.In this example words like assignment/submission predominate …Another project noted that the term summative assessment is rarely used in institutional documentation because it tends to be assumed that all assessment is summative unless otherwise stated.
• #23: Another group of reports reflect institutions that are focusing more on the practice of assessment and feedback, with more of a direct focus on enhancing how learners engage with their feedback. These reflect projects hat the project teams believe to be a little more change-ready. Their baselines show more emphasis on feedback – in this example learning and assessment appear in equal measure.We should not however over-emphasise the differences between the two.Some of the institutions that are concentrating on formal processes are very concerned with the learner experience and achieving parity of learner experience across the institution.Some of the institutions that are basing their work on the premise that developmental feedback is more valuable to the learning experience than summative assessment are nonetheless basing their project ambitions on pockets of good practice (either internally or elsewhere) that are far from being mainstream as yet.
• #24: We believe the baseline picture we have drawn is a reasonable summary of the state of play across the wider landscape.This rich picture from one of the projects at IOE sums up their view of the terrain. The map has at its heart the student experience province – student experience links to policy with a road as their concerns relate to each other. Teaching staff province less well defined, but strongly linked to student experience, less well linked to policy as staff interpretations vary. External exampiners province is uclear and on the fringes of the mainland.It described the map of the territory as follows: ‘The sector practice includes two provinces which are strongly defined and these are the technological innovation in assessment and feedback and the good practice guidelines which draw on wider research and theorising of assessment for learning. The overall sector practice is less well defined and is represented on the map as a large area of uncertainty and danger in the current climate of retrenchment and funding cuts with high mountain ranges and sea monsters.’If time permits allow a few mins discussion on whether people recognise this picture and agree that good/innovative practice and mainstream practice are still separate territories.
• #25: Given the time we have today I’m going to share just a few of the headline findings in three areas – strategy and policy, stakeholder engagement and practice, and if we have time I’d like to to consider these findings on your tables to see if they reflect your experiences. 1. Despite the central part played by assessment and feedback in the learning and teaching process, there is little evidence of institutions being strongly directive in terms of their strategy and policy steer in these areas.Institutions appear more likely to have assessment ‘frameworks’ and ‘policies’ than top-level strategies and these are often mainly aimed at guiding the development of strategy and policy at a more devolved (school/faculty/college) level. There are some excellent examples across the sector where institutional strategy puts assessment at the heart of learning but joined-up approaches (e.g. institutional strategy, TEL, A&F, Learning spaces) are by no means commonplace.Formal documents tend to be quite procedural rather than emphasising the developmental aspects of A&F. In this sense they seem to lag behind the thinking that underpins most of the projects.2. The programme is sending a strong message that sound educational principles need to underpin A&F practice. The principles that most projects identify as being relevant to their work e.g. REAP, NUS, Gibbs and Simpson (2004) tend to differ from those that are described in formal documents which are again concerned more with procedure and consistency.3. The fact that ultimate responsibility for A&F practice is highly devolved leads to considerable variation in practice as borne out by testimony from both staff (including external examiners) and students in the reviews. This is a major concern for senior managers who feel that lack of parity in the student experience may be a source of complaint/litigation in future. There are series of related concerns discussed in the report about the ability to support learners’ longitudinal development.
• #26: Regarding stakeholder engagement, these three themes emerged from the reports:1. Learner engagement is a major feature of many of the projects in the programme but was noticeably absent from the descriptions of baseline practice. There was a similar gap around the role of student induction or training in terms of preparing learners to get the most from assessment and feedback. Engagement with small groups of learners is often seen as adequate rather than widespread engagement (repeated for each cohort) being embedded practice.2. The projects made many interesting observations about A&F in relation to employability.The difference between the institutional emphasis on summative assessment and the more formative ways in which professionals develop throughout their careers (including through extensive use of peer review) was noted.One project went so far as to challenge received wisdom that greater definition and clarity in relation to marks and grading is necessarily a good thing: ‘What emerged was the notion of business clients as assessors, and the critical notion that in a business scenario you do not know exactly what you will be assessed on. Indeed, it could be said to be part of the assessment itself, in a business context, to work out exactly what is expected from you based on the client and the information that they may or may not be providing.This raises the challenging question: Are we being too specific in detailing exactly how students get marks from our assessments? Should part of the assessment be the task of working out which are the more crucial parts of the assessment itself? 3. In a webinar to discuss the baseline findings some projects expressed surprise at how ‘clunky’ their admin processes appeared as a result of their reviews. It is clear that the diversity of admin processes within each institution (devolved responsibility meaning each faculty or school does things its own way) is a barrier to getting the most out of the IT systems that support A&F. It is also notable that although admin staff are identified as stakeholders they tend to be the people that ‘we haven’t got round to talking to yet’. Given the dual emphasis of the programme on effectiveness and efficiency it is evident that admin processes need to be a significant focus for improvement.
• #27: 1. Although the reviews identified a large variety of assessment types in use it appears that traditional forms such as essays/exams still predominate. Project approaches vary from trying to introduce greater awareness of different forms of assessment to trying to improve practice in relation to the most commonly used types. Both staff and students need to be convinced of the benefits of peer-to-peer learning.2. Along with consistency, the timeliness of feedback appears to be a key issue across the board. Many schools/departments do set deadlines for marking and the return of feedback but there remain issues with the overall assessment timetable and whether students receive feedback in time for it to be useful in preparing for the next assignment. Where this does occur there is substantial evidence that students do appreciate and act upon feedback.3. The extent to which feedback supports the ongoing development of the individual learner by feeding forward into their future learning depends on the quality of the feedback given and whether it is developmental in nature rather than simply justifying a grade.In some cases curriculum design presents structural issues that can provide barriers to this type of development arising from modularisation and from distributed teaching teams that inhibit continuity in terms of establishing a relationship with the student or a shared context for the assessment task. A number of the projects in the programme are looking at ways to better facilitate the joining up of the overall learning experience. One project is looking to develop the concept of an 'assessment career’ building upon a range of previous research including work on ipsative assessment whereby feedback acknowledges progress against the learner's previous performance regardless of achievement, another is aiming for process change to provide a longitudinal overview of feedback provided and student reflection on previous assignments and a further project is focusing on whole programme transformation to support coherent approaches to student learning through assessment design focused at the level of the degree programme rather than the module.
• #29: JISC Assessment pagesProvides:Background to JISC’s activities with technology enhanced assessment Link to the latest JISC publication on assessment – Effective assessment in a digital age Information on previous projects and activities in the assessment spaceResources including activities to run with stakeholder groups developed as part of a series of national workshops – including overview of the leading educational principles around good assessment and feedback practice.
• #30: A range of different types of tools, case studies, models, learning design, lessons learned etc which support teams in designing, developing and delivering curriculum in their institutions.
• #31: As part of the assessment and feedback programme and the excellent work of Ros Smith and the synthesis team we’ve now compiled a set of assessment and feedback pages, taking you through the key themes around assessment and feedback, which will provide a framework for compiling and making sense of project outputs as they arise.
• #33: Go to ‘View’ menu > ‘Header and Footer…’ to edit the footers on this slide (click ‘Apply’ to change only the currently selected slide, or ‘Apply to All’ to change the footers on all slides.