Developmental evaluations for institutional impact

Editor's Notes

• #3: Need to say at the outset that I was not starting from scratch. OCSLD has long history of conducting research which changes things within the institution. In 2000, Oxford Brookes redesigned its policies and procedures for managing breaches of academic regulations.  Whilst the changes apply to any of a number of forms of misconduct, the most common way that students do not comply is by plagiarism and by copying from fellow students which is often called collusion.  The changes stress that student plagiarism can only be managed by taking an integrated or holistic approach to ensuring that students know what they must do, that they have the skills necessary to comply, that assessment tasks are designed to discourage finding or faking rather than doing the task, and that breaches when they occur are managed consistently and defensibly.A key aspect of the Brookes holistic approach is to refer all possible breaches to a specialist officer, termed an Academic Conduct officer or ACO.  The ACO investigates a case, interviews the student and awards one of the designated penalties (based on an agreed tariff for different levels of plagiarism).  The ACO model is now well embedded at Brookes and has been replicated around the UK and internationally..
• #4: In both these examples, the research came first, then began the work of disseminating and trying to make it make a difference at Brookes and elsewhere.
• #5: Note this is research and knowledge transfer….We argue that OCSLD has the expertise to position Brookes to make a substantial contribution to pedagogic research nationally and internationally. Currently we have a number of small active research groups who are working independently within their specialist fields. The task is to bring these together into core themes and to build capacity around these core themes to enable us to be a position to undertake large scale projects of national importance and enhance the REF submission in 2012.(2) We make a case for using our research expertise to contribute to improvements to the student experience at Brookes. The White Paper recognises that teaching and learning practices at Brookes must be based on current research. Currently OCSLD has great strengths in conducting both primary investigations of the student experience and knowledge transfer practices and this expertise is drawn on by other universities and sector wide organisations. The task is to put this expertise to good use for Brookes to ensure that our practices here are informed by our own local investigations as well as a wider range of research.(3) We fully support the White Paper’s emphasis on the professionalisation of teaching and the need for continuing professional development for academic staff. We argue that scholarship of teaching should be a core element of the emerging continuing professional development framework and one of the ways in which Brookes ensures links between teaching and research. Currently OCSLD supports the development of researchers within our own group and nationally. The task is to support the development of pedagogic researchers within Brookes, both as part of their CPD and in order to build sustainable, ongoing capacity for (1) and (2) above.
• #6: In the section on ‘evidence based policy development’… it says this about evaluating pedagogic developments…The focus on impact assessment comes from the PVC. The final part comes from me. This says something about our approach …‘Sustained and iterative’‘assessing the process as well as the end product’These came from evaluation approaches that we’ve been involved in, which we thought would have real potential for institutional research….
• #8: The AI process has a number of stages:Discovery: Appreciating the best of what is. time to search for, highlight and illuminate those factors that give life to the organisation. Find moments of excellence and discuss the factors and forces that made them possible.Dream: Envisioning what could be. creating new, positive alternatives using words, phrases and stories that illustrate the organisation at its best.Design: Co constructing what could be: designing the future through dialogue, share discoveries and possibilities. Get to a point where everyone can say ‘this is an ideal or vision that we value and should aspire to’.Destiny: Sustaining what will be. constructing the future through innovation and action by including ever increasing numbers of participants in the conversations. Steps in the Emerge Project
• #9: So, we approached our commitment to conducting institutional research, already favourin approaches to research which are….Other research approaches are available… this is important to say. Not everyone buys into our approach and you need to know where other people are coming from.Other purposes of evaluation might be forAccountabilityKnowledge creationas well as ours fordevelopment
• #11: These are the RQs that the uni wanted us to do.Really it was about checking if the new SSCs were OK
• #12: This is a lot of data.There are several routes through this data. We’ve used it to tell a number of different stories about student support. We’ve written a number of internal reports and submitted academic research papers. Briefly, in this session, I’m going to tell you just one little story about student help seeking…..
• #13: High satisfaction… backed up by some lovely open comments from survey respondents who have seen SSCs.we could have stopped here.
• #15: For all 11 issues the percentages of students in the survey who selected Academic Advisers or SSCs as preferred support sources has increased compared with the 2010 survey, showing that students are increasingly recognising these as appropriate places to go to when they need advice and information. What the literature saysThe assumption has been made that students go to the people who provide support they are satisfied with. If you know what makes students satisfied, you can understand their help-seeking behaviour. The problem with this is that satisfaction seems to be linked to individual’s interpersonal skills, expressed in terms such a ‘warmth’, and not on their approaches to giving support (Mottarella, Fritzsche, & Cerabino, 2004).
• #16: The reports drew on individual stories of support needs over the student lifecycle and explored the reasons behind the choices students make when seeking advice and information. These stories make use of pseudonyms. Each student has confirmed the contents of their story for use in the project.Stories provide insights into the effects of our practices. They work on the affective domain and complement observational and statistical data. They bring together many diverse aspects of a situation and often demand more ‘joined-up thinking’ and represent the lived experience of the situation
• #17: Stories provide insights into the effects of our practices. They work on the affective domain and complement observational and statistical data. They bring together many diverse aspects of a situation and often demand more ‘joined-up thinking’ and represent the lived experience of the situation
• #18: Stories provide insights into the effects of our practices. They work on the affective domain and complement observational and statistical data. They bring together many diverse aspects of a situation and often demand more ‘joined-up thinking’ and represent the lived experience of the situation
• #19: Research on help-seeking within the classroom has identified three main reasons forstudent failure to seek help: help-seeking is perceived as a dependent behavior whichconflicts with the need for autonomy (Ryan, Shim, & Patrick, 2005); help-seeking isperceived as evidence of incompetence and, therefore, a threat to ability perceptions(Butler, 1998); and the mistaken belief that help seeking will not be effective (i.e.,there is no competent and willing helper who can provide assistance or seeking helpwill take to long) (Ryan et al., 2005)
• #20: Understanding why students don’t seek help…Open coding resulted in the creation of 12 categories. Each response was allocated up to two codes by the principal investigator. See Table 9Although the numbers are small, students’ own words provide a vivid insight into their strength of feeling about not being able to access support when they need it e.g.It appears that students are trying BUT find that that issues are not RESOLVED or not RELEVANT
• #21: Research on help-seeking within the classroom has identified three main reasons forstudent failure to seek help: help-seeking is perceived as a dependent behavior whichconflicts with the need for autonomy (Ryan, Shim, & Patrick, 2005); help-seeking isperceived as evidence of incompetence and, therefore, a threat to ability perceptions(Butler, 1998); and the mistaken belief that help seeking will not be effective (i.e.,there is no competent and willing helper who can provide assistance or seeking helpwill take to long) (Ryan et al., 2005)
• #22: Remember there is high satisfaction when help is accessed
• #24: Can we get students to change their behaviours to take advantage of our new support structures? Can we modify their help seeking behaviours? Probably not, but understanding their help-seeking behavious can help us plan our support interventions.
• #25: This is a long way from saying the new system is working well because of high satisfaction rates!
• #26: Stories provide insights into the effects of our practices. They work on the affective domain and complement observational and statistical data. They bring together many diverse aspects of a situation and often demand more ‘joined-up thinking’ and represent the lived experience of the situation
• #27: Stories provide insights into the effects of our practices. They work on the affective domain and complement observational and statistical data. They bring together many diverse aspects of a situation and often demand more ‘joined-up thinking’ and represent the lived experience of the situation
• #30: Future projects I’d really like to doAn appreciate inquiry for our next Student Experience StrategyCreate a student research panel