Southampton: lecture on TEF
Download as PPTX, PDF2 likes6,546 views
The document discusses concerns regarding the quality of education at UK universities, emphasizing the need for a Teaching Excellence Framework (TEF) to address issues of dissatisfaction among students and employers. It critiques the proposed metrics for TEF, arguing they may not accurately measure teaching quality and could create adverse incentives. The document also suggests alternative approaches to balance the focus on teaching and research within higher education institutions.
Southampton: lecture on TEF
- 1. What’s wrong with our Universities, and will the Teaching Excellence Framework put it right? Dorothy V. M. Bishop Professor of Developmental Neuropsychology University of Oxford @deevybee http://deevybee.blogspot.co.uk/ cdbu.org.uk Distinguished Lecture Series University of Southampton 1st February 2017 http://www.slideshare.net/deevybishop/reading-list-whats-wrong-with-our-universities
- 2. • Case made by government for introducing TEF • Responses to TEF consultation • Validity of proposed TEF metrics • Cost-benefit analysis of TEF Alternative approaches to restore balance of teaching and research Structure of talk
- 3. Case made by government for introducing TEF
- 4. Case set out in Green Paper in Nov 2015 • Interested parties invited to comment • Questions did not cover whether a Teaching Excellence Framework was needed – this was manifesto commitment* to be implemented Manifesto: “We will ensure that universities deliver the best possible value for money to students: we will introduce a framework to recognise universities offering the highest teaching quality; encourage universities to offer more two- year courses; and require more data to be openly available to potential students so that they can make decisions informed by the career paths of past graduates”
- 5. Claim 1: Teaching is ‘patchy’ and ‘lamentable’ 9th September 2015 ”I hear this when I talk to worried parents, such as the physics teacher whose son dropped out at the start of year two of a humanities programme at a prestigious London university, having barely set eyes on his tutor. Her other son, by contrast, studying engineering at Bristol, saw the system at its best: he was worked off his feet, with plenty of support and mostly excellent teaching. This patchiness in the student experience within and between institutions cannot continue. There is extraordinary teaching that deserves greater recognition. And there is lamentable teaching that must be driven out of our system. It damages the reputation of UK higher education and I am determined to address it.” https://www.gov.uk/government/speeches/higher-education-fulfilling-our-potential Jo Johnson Minister for Universities and Science
- 6. Is teaching ‘patchy’ and ‘lamentable’? “In the NSS 2015 survey, two thirds of providers are performing well below their peers on at least one aspect of the student experience; and 44% of providers are performing well below their peers on at least one aspect of the teaching, assessment and feedback part of the student experience.” Jo Johnson Challenged to give evidence of ‘lamentable’ teaching by Select Committee: Distribution of responses to item 22: "Overall I am satisfied with the quality of the course" See: http://deevybee.blogspot.co.uk/2015/12/a-lamentable-performance-by-jo-johnson.html
- 7. Claim 2: Students are dissatisfied Introduction: The transparency challenge, point 15: “Students are also concerned about value for money, with one third of undergraduates paying higher fees in England believing their course represents very poor or poor value for money.” Cites this report: http://www.hepi.ac.uk/2015/06/04/2015-academic-experience-survey/
- 8. Are students dissatisfied? England Scotland
- 9. Are students dissatisfied? England Scotland “Perceptions of value for money have diverged as a result of the increase in the full-time undergraduate fee cap to £9,000 in 2012 for students from England …. Only 7% of students from England on the higher fees feel they receive ‘very good’ value for money – the figure for students from Scotland who remain there to study is five times higher (35%).”
- 10. But some problems noted, e.g. : ‘One third (33%) of students describe the information they received before starting their course as “accurate”, which has been a consistent finding over the last three years’ Are students dissatisfied?
- 11. Claim 3: Employers are dissatisfied Introduction: The productivity challenge: “Too many organisations find it hard to recruit the skilled people they need; this poses serious risks to the competitiveness, financial health and even survival of many businesses.” Cites this report: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/3737 69/14.11.11._EPS_2014_-_Main_Report_full_V2.pdf
- 12. Are employers dissatisfied? “Very few said that their recruits lacked basic numeracy or literacy, or that they had had a poor education (fewer than five per cent in every case).”
- 13. Claim 4: Teaching is neglected relative to research “There are weak incentives on Higher Education Providers to increase the standard of teaching in the higher education sector.” “Because many universities see their reputation, their standing in prestigious international league tables and their marginal funding as being principally determined by scholarly output, teaching has regrettably been allowed to become something of a poor cousin to research in parts of our system" Proposed solution: more league tables, this time for teaching N.B. League tables NOT needed to ‘drive out poor teaching’ – that requires Quality Assurance Agency: focus on poor-performing minority
- 14. Responses to TEF consultation
- 15. 618 respondees • 136 Higher Education Institutions • 26 alternative providers • 22 further education colleges • 78 student unions • + individuals and organisations with a stake in the sector https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/556355/ TEF_Year_2_specification.pdf Responses to TEF consultation (Green Paper)
- 16. Responses to TEF consultation http://cdbu.org.uk/perils-of-ignoring-consultation-on-the-higher-education-and-research-
- 17. Distribution of responses by type of respondent would be interesting, so I requested raw data from BIS…… No wonder it took BIS so long to produce their report: Appears they don’t use computers!
- 18. “Any model of a teaching excellence framework (TEF) must provide a true measure of teaching excellence. … a system that relies on crude or reductive metrics would be misleading to students and could create adverse incentives for institutions. A TEF must be focused on teaching (or learning/educational) excellence alone. It must not be conflated with wider policy issues, such as widening participation. If a TEF is associated with an ability to increase tuition fees beyond inflation, students would be forced to choose between quality, as measured by a TEF, and affordability. “ https://www.admin.cam.ac.uk/reporter/2015-16/weekly/6412/GreenPaperResponse- Cambridge-Jan16.pdf Response by the University of Cambridge
- 20. Main change between Green Paper and White Paper: TEF disappeared from the Higher Education Bill TEF going ahead without Government approval: does not need legislation
- 21. Proposed TEF metrics Key NSS: National Student Survey ILR: Individualised Learner Record (FE) DLHE: Destinations of Leavers from Higher Education survey • Some changes made since Green Paper • To be interpreted in light of qualitative information about ‘context’ https://www.gov.uk/government/uploads/system/uploads/attach ment_data/file/556355/TEF_Year_2_specification.pdf
- 22. “Outcomes in Year Two will not be associated with differential fee uplifts for providers in England – rather, all those achieving a rating of Bronze, Silver and Gold will receive the full inflationary uplift. However, these awards will be used from Year Three onwards to inform differentiated fees”
- 23. Validity of proposed TEF metrics
- 24. The NSS scales Teaching on my course 1 - Staff are good at explaining things. 2 - Staff have made the subject interesting. 3 - Staff are enthusiastic about what they are teaching. 4 - The course is intellectually stimulating. Benchmark factors: Subject of Study, Age on Entry, Ethnicity, Sex, Disability 5 - The criteria used in marking have been clear in advance. 6 - Assessment arrangements and marking have been fair. 7 - Feedback on my work has been prompt. 8 - I have received detailed comments on my work. 9 - Feedback on my work has helped me clarify things I did not understand Assessment and Feedback ’. 10 - I have received sufficient advice and support with my studies. 11 - I have been able to contact staff when I needed to. 12 - Good advice was available when I needed to make study choices. Academic Support
- 25. Problems with NSS as a metric 1. It doesn’t measure teaching quality
- 26. Response by Royal Statistical Society
- 27. Response by Royal Statistical Society
- 28. Response by Royal Statistical Society ”Anecdotally, we have heard of institutions explicitly `dumbing down‟ programmes so as to result in higher NSS scores. A new TEF needs to recognise this and mitigate against it. One goal of higher education is to produce highly educated people of use to the society of the future and the NSS inadvertently encourages the opposite.”
- 29. “I do not think student satisfaction is an accurate proxy for teaching quality,” said Professor Husbands, a former director of the UCL Institute of Education who was named chair of the TEF panel by the Department for Business, Energy and Industrial Strategy in August 2016. Professor Husbands drew attention to US studies that show that student satisfaction scores are heavily influenced by non-academic factors, including the sex or ethnicity of a lecturer. “Student satisfaction seems to be driven by the physical attractiveness of academics rather than anything else,” he said of the US research findings. He added that the TEF panel would “not draw policy from a single data point” and that “all data” are “flawed” in some respect. However, he went on, the challenge was to recognise and understand the flaws, and to learn how the information could be used effectively.
- 30. Problems with NSS as a metric 1. It doesn’t measure teaching quality 2. It doesn’t discriminate between institutions
- 31. Response by Royal Statistical Society “It is not clear that it is possible to discriminate the vast majority of HE institutions on the overall NSS satisfaction scores, let alone when they are broken down into smaller subgroups." Cites ONS: “Teaching Excellence Framework: Review of Data Sources --- Interim Report” Rank ordered HEIs Line shows mean score, vertical bar shows confidence interval
- 32. Problems with NSS as a metric 1. It doesn’t measure teaching quality 2. It doesn’t discriminate between institutions 3. Success is defined relative to all providers i.e. norm-referenced rather than criterion-referenced
- 33. Proportion rating ‘very dissatisfied’ or ‘dissatisfied’ Problems with norm-referenced measure Worst 10%, cutoff .13
- 34. Proportion rating ‘very dissatisfied’ or ‘dissatisfied’ Problems with norm-referenced measure Suppose the worst-performing institutions either improve or go out of business
- 35. Proportion rating ‘very dissatisfied’ or ‘dissatisfied’ Problems with norm-referenced measure Worst 10%, cutoff .105 Cutoff now moves: however good they are, some institutions will always be in the bottom 10%
- 36. Problems with norm-referenced measure • Measure only as good as the data it is based on • What if substantial N students don’t comply? • What if institutions likely to do poorly opt out? • Benchmarking raises statistical problems: very difficult to compare institutions fairly when different sizes; unclear how best to estimate standard errors David Draper and Mark Gittoes. Statistical Analysis of Performance Indicators in UK Higher Education. Journal of the Royal Statistical Society. Series A (Statistics in Society), Vol. 167, No. 3 (2004), pp. 449-474
- 37. Cost-benefit analysis of TEF
- 38. Cost-benefit analysis of TEF https://deevybee.blogspot.co.uk/2016/07/cost-benefit-analysis-of-teaching.html = Universities/HEIs https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/5 28005/bis-16-295-he-research-bill-detailed-impact-assessment.pdf Estimated as £53K per institution on average
- 39. Cost-benefit analysis of TEF: Benefits Table 14: Benefits to HEIs in real terms • Projections crucially dependent on correctly predicting student N and inflation rate • Benefits of TEF vs no TEF purely down to fact that fees will be fixed if no TEF http://deevybee.blogspot.co.uk/2016/07/cost-benefit-analysis-of-teaching.html
- 40. Cost-benefit analysis of TEF Why not allow everyone to increase fees in line with inflation provided they have a satisfactory Quality Assurance rating? “whilst QA is a good starting point, reliance on QA alone and in the longer-term will not enable significant differentiation of teaching quality to help inform student decisions and encourage institutions to improve their teaching quality.” (p. 37). One consequence (and one suspects one purpose) of TEF is to facilitate the division into institutional sheep and goats, followed by starvation of the goats. (Bishopblog analysis, July 2016)
- 41. ’…the metrics are flawed. This is not renegade opinion, but rather the overwhelming view of those actually involved in higher education…. Yet the government has us over a barrel. It has linked the TEF to tuition fees and, potentially, our ability to recruit international students.’
- 42. What about costs to academic staff? Bring in substantial grant income Publish in ‘high impact’ journals Demonstrate impact beyond academia Manage a research group REF demands Be available for students at all times Provide detailed and prompt feedback Give interesting lectures with enthusiasm Ensure students go on to get good jobs TEF demands Can one person do all of these? As well as being good ‘academic citizen’?
- 43. Alternative approaches to restore balance of teaching and research 1. Include student learning in the impact section of REF Professor Roger Brown http://cdbu.org.uk/reflections-on-the-green-paper-tef/
- 44. Alternative approaches to restore balance of teaching and research https://www.theguardian.com/higher-education-network/2016/nov/08/tef-dump-the- pointless-metrics-and-take-a-hard-look-at-casualisation 2.
- 45. House of Lords Debate, 18th Jan 2017 https://hansard.parliament.uk/Lords/2017-01-18/debates/BE1D3D6A-6AF6-4D6A-870A- 472D27EA2622/HigherEducationAndResearchBill#contribution-06CF9C5B-303A-4EFE-8887-74B3E0B825CE These are not perfect measures. We are on a journey, and I look forward to these metrics being revised and replaced by superior metrics in the future. They are not as bad as we have heard in some of the caricatures of them, and in my experience, if we wait until we have a perfect indicator and then start using it, we will have a very long wait. If we use the indicators that we have, however imperfect, people then work hard to improve them. That is the spirit with which we should approach the TEF today Before the noble Lord sits down, will he explain what consolation he will offer to those institutions which are put out of business, at worst, while we perfect the metric that is being used in this case?
- 46. The final word? https://hansard.parliament.uk/Lords/2017-01-18/debates/BE1D3D6A-6AF6-4D6A-870A- 472D27EA2622/HigherEducationAndResearchBill#contribution-06CF9C5B-303A-4EFE-8887-74B3E0B825CE My Lords, the noble Lord, Lord Willetts, said that we are embarking on a journey, which indeed we are, but I feel that the car in which we will travel does not yet have all the component parts. I therefore wonder if, when we have concluded all our debates, rather than going full speed ahead into a TEF for everybody who wants to participate, we should have some pilots. In that way the metrics could be amended quite properly before everybody else embarks on the journey with us.
- 47. What’s wrong with our Universities, and will the Teaching Excellence Framework put it right? Dorothy V. M. Bishop Professor of Developmental Neuropsychology University of Oxford @deevybee http://deevybee.blogspot.co.uk/ cdbu.org.uk http://www.slideshare.net/deevybishop/reading-list-whats-wrong-with-our-universities
- 48. Lowest-scoring item: ‘Feedback on my work has been prompt’ http://dictionary.cambridge.org/dictionary/english/prompt