Presentations morning session 22 January 2018 HEFCE open event “Using data to increase learning gains and teaching excellence”

Editor's Notes

• #12: Stanislav Petrov, in 2017 1983 Soviet nuclear false alarm incident On 26 September 1983, just three weeks after the Soviet military had shot down Korean Air Lines Flight 007, Petrov was the duty officer at the command center for the Oko nuclear early-warning system when the system reported that a missile had been launched from the United States, followed by up to five more. Petrov judged the reports to be a false alarm,[1] and his decision to disobey orders, against Soviet military protocol,[2] is credited with having prevented an erroneous retaliatory nuclear attack on the United States and its NATO allies that could have resulted in large-scale nuclear war. Investigation later confirmed that the Soviet satellite warning system had indeed malfunctioned.[3]
• #18: Add reference
• #19: Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
• #22: The raw gain as a value of gain is inaccurate due to the difference between scores being less reliable than scores themselves i.e., raw gain represents compound error of pre-test and post-test. The raw change in scores between pre-test and post-test assumes linear relationship between test scores and ability. However, learning gain on a simple test will be higher for a low-ability student and lower for a high-ability student, whereas the raw learning gain on a difficult test will be higher for a high-ability student and lower for a low-ability student. The true gain is based on a linear regression procedure where the true gain for each individual is the difference between group mean pre-test score and group mean post-test score assuming the reliability of estimates is satisfactory i.e., the pre-test post-test variances and reliability are equal (Lord, 1956, 1958). However, when these assumptions are violated the reliability of raw gain scores is actually high and therefore use of raw gain scores produces would be a more accurate representation of a gain that true gain (Overall & Woodward, 1975; Zimmerman & Williams, 1982). Building on linear regression procedure for measuring true gain, the residual gain was proposed as a measure of change (Cronbach & Furby, 1970). The computation of residual gain is compatible with the raw gain. The advantage of this computation is that it removes the change from the posttest scores that are predicted from the pretest (Linn & Slinde, 1977). Although, residual gain allows to identify individuals that showed more or less than expected gain (i.e., are superior of inferior learners) residuals do not represent change as such, they only represent what was not predicted linearly (Baird, 1988). As they are residuals (essentially deviations) half of students will by default be above the mean and half bellow which makes judgement on the effectiveness of learning inappropriate (Pike, 1992). Similar, computation of learning gain is basefree measure of change (Tucker, Damarin, & Messick, 1966), but as they suggested in their work, basefree measure of change was directed at correlational research into understanding why certain individual demonstrate above average change. The basefree measure of gain was also criticised for incorrect attribution of the change scores in their computations (Bond, 1979; Cronbach & Furby, 1970) and as such was not used in the research on gains (for the exception of few studies in 70s and early 80s). Further attempt to compute gain was to compute normalised gain (Hake, 1998). The normalised gain is the most widely used measure of learning gain. The main advantages of using normalised gain is that it solves the problem of ceiling effect and bias towards strong students (high pre-testers) by using the difference between the maximum test score and pre-test score as denominator. Thus, normalised gain demonstrates realised gain to the maximum of possible gain. In addition average normalised gain for a group can be computed using either individual scores or group means for pre-test and post-test (Bao, 2006). Although in both cases the same principle is used, the two methods may yell different results for the same sample due to asymmetrical distribution of differences between low and high scoring pre-testers on their post-tests or scoring lower on the post-test than in the pre-test (for a review see Bao, 2006). In addition the normalised gain cannot be computed for individuals who scored absolute maximum on the pre-test scores or the average normalised gain cannot be computed using individual scores if any one person scored maximum on the pre-test (Marx & Cummings, 2007). Hoping that as a result of intervention/course/teaching it is more likely that students will demonstrate positive gain and score higher on the post-test than on pre-test. However that is not always the case and in the instances of negative gain or when maximum possible score was obtained at the pre-test the normalised learning gain cannot represent a meaningful learning gain or lack of learning. In cases where post-test scores are lower than pre-test scores computation of normalised change is more meaningful (Marx & Cummings, 2007). The normalised change has advantage over normalised gain in cases of negative gain by using analogous computation where observed loss is the ratio of possible maximum loss. However this method does not apply to the students who scored possible maximum or possible minimum on both pre-test and post-test (Marx & Cummings, 2007). In addition, normalised change ranges from -1 to +1 and removes low pre-test scores bias. Thus, students who score 90% on a pre-test can obtain a change ranging from -1 to +1 and the same is the case for students who scored 50% or 20%. As such, it is much easier and more intuitive to interpret normalised change rather than normalised gain. Marx and Cummings (2007) further argued that normalised change is suitable for computation if the pre-test and post-test when they are not the same. However it is important to understand that when averaging gains or losses to a group, both gains and losses are relative only to the maximum possible gain i.e., the result will show more gain than loss. All previous methods for computing change were aiming to address the issues posed by raw gain e.g., measurement errors, regression to the mean, bias towards high scores in the pre-tests. To further improve accuracy in computing gain in educational settings, Rasch model was used. The Rasch model is based on probability and the assumption is that scores obtained at the pre-test and post-test for each student are a combination of that student’s ability and difficulty of the test. In cases when the research design involves comparison between two or three groups, the most commonly used methods for computing learning gains from pre-test and post-test scores are: analysis of variance (ANOVA) and analysis of covariance (ANCOVA). The advantage of ANOVA and ANCOVA is in that they both reduce error variance and increase the power of the test (Sörbom, 1976). There are four ways of data analysis commonly used in mixed research. The preferred method is to use ANCOVA on pretest and posttest scores or ANOVA on raw gain scores. The two least favourable analyses are ANOVA on residuals and repeated measures ANOVA (Dimitrov & Rumrill Jr, 2003). In ANCOVA pretest scores are used as covariates of postest scores because that reduces error variance by adjusting posttest means to the pretest. However, ANCOVA will produce reliable results when the assumptions of linear relationship, randomization and homogeneity of variances are met. Same assumption should be met for ANOVA on raw gains analysis. However ANCOVA is more powerful than ANOVA and more flexible on the assumption of linear relationship and as such ANCOVA will also produce more accurate computation of gain if the relationship between pretest and posttest has quadratic or cubic component. In all, all four ways of data analysis have limitations, but results obtained from ANCOVA seems to be most accurate for the comparison of learning gains between two or more groups.
• #29: Assessment = already collected metrics as opposed to new measures
• #30: Level 1 – Grade: repeated measures on students and tell us about students learning trajectory Level 2 – student: between students variations Level 3 – Course: between course variation
• #31: 10 - modules
• #32: 10 - modules
• #35: Student-level intercept-slope correlation - Students with low initial achievements show high learning gains that students with initial high achievements. Module level intercept-slope correlation Students in modules with the low initial achievements show higher learning gains than students in courses with the high initial achievements VPC – is variance partition coefficient. VPC module shows variance that can be attributed to the differences between modules. VPC student shows the mount of variance that can be attributed to the differences between students, and VPC TMA shows variance within each student across assessments. In this last one we can see that it is almost double in Social Science hence further supports that assessments play important role.
• #36: Gender: male-female learning gains gap (male reference group) Other-white gain gap Mixed-White gain gap Asian-White gain gap Black-white gain gap HE qualification - A levels gain gap lower that A levels - A levels gain gap No formal qualification – A levels gain gap PG qualification - A levels gain gap
• #39: On average students showed improvement in standardised grades, although this was only significant for University 1
• #40: University 1 variance was mainly nested between students University 2 had more variance at the departmental level and within students The negative learning gains seen in some students in University 2 does not imply that these students are losing knowledge or ability per se. However it highlights the complexity of factors that have to be taken into account when using students’ academic performance as a proxy for learning gains, such as ‘assessment difficulty’ and ‘learning design’ (Rienties & Toetenel, 2016). We need to understand these if we are to use academic performance as a proxy for learning gains within Higher Education.
• #43: We set out to see if multilevel modelling of assessment data could be a valid and useful proxy for learning gain Aggregate learning gains estimates can result in misleading estimates of students’ learning gains on a discipline or degree level. (or show no gain hiding a more complex picture – University 2) Multilevel modelling is a more accurate method in comparison with simple linear models when estimating students’ learning gains. The simple models are not able to detect differences between modules when looking at the department and degree level performance, whereas multilevel modelling can.
• #44: The HEFCE funded programme of which this project is a part wants to devise a sector wide APPROACH (not method) and characterisation of learning gain, and explore possible proxies. The contribution of the ABC project is that if we are to use LG research for optimising learning (LAK definition) then we need to use a methodology that allows observation of differences in patterns of gain between departments, courses or modules. We have seen that there are subject level differences and so our research supports the introduction of a subject level TEF. Ideally, we’d like to see LG not just used to judge institutions but integrated into curriculum design. Designing for learning is becoming more learner centred – moving from ’what am I going to teach?’ to ‘How are students going to learn?’. A LG methodology such as this could provide data to support course teams to make evidence based decisions (complementing student satisfaction data). For example, do similar modules with different types of learning activities produce different LG trajectories? We know from the field of Learning Design that visualisation is important for course team making decisions, the multi level modelling approach provide data visualisation. Some questions still to answer….
• #48: Needing to be robust,
• #50: The original purpose of the alumni interviews was to assess the contribution of the GAs to employability. The interview schedule asked questions about personal development, employability and work readiness.   We also asked them some general questions about what they though they gained from their experience at Brookes and prompted specifically for ABC types of learning gains.
• #52: we had become interested through the project in trajectories and so for this presentation I’ve looked and listened again at the interviews of 4 students who were on an upward trajectory. We didn’t select alumni interviews on the basis of their academic progress or trajectory and unsurprisingly, only 1 downward trajectory person volunteered for interview, and that was because he had beef with Brookes. The interviewees were mostly high achievers. Out of the 12, only one got a 2:2, rest for good degrees. With a greater proportion than you would expect having chosen a combined honours degree and/or a year’s placement.
• #53: Emotional language used to describe these points. Interviewees able to identify them. Express the scale of the impact they made on them in their choice of words. For example, for A5, Psychology, 3 years, 2:1, but did internship with a social enterprise while at Brookes, and that turned into employment in 3rd year, initially one day a week and then promotion to 2 days a week in a leadership role.. “I’m one of those people who can’t sit still, I literally feel I must keep learning.” No
• #54: 2 examples from each interview set
• #55: So what changed? The ability to critically reflect on own study strategies, A self awareness of own strengths An ability to see a plan for own career or life goals. This awareness came from Course and study (A12) Placement experience (A8) Or extra curricular work experience (A5) For 12, this turning point was about an awareness of own study strategies, partly a result of friends having left uni while she was on placement. Having always been driven and determined, she now set her sights on a goal (good grades) and discovered how to achieve it. For A8 this turning point was about an awareness of where own interests lie within a broad discipline and the ability to follow this new self knowledge by choosing more relevant modules, which she enjoyed and did better in, and so saw an increase in grades. For A5 this turning point was about finding a place where you belong and you are doing something you are good at. Being in a position to recognize own team leadership skills and put them into practice (here through social enterprise work experience) “Finding where you belong and what you’re good at and what you like. “I found where I belong if that makes sense and that’s where my drive towards thinking well actually I quite like HR, because I did a bit of HR and I did a bit of corporate relations, those were my two like job titles in the enterprise. And I also discovered that I, although I never take on a front facing leadership role, I often find myself being a leader within a group.”
• #56: Another common feature of what changed for these 3 upward trajectory students was a confidence in working and learning independently For A12, The ability to study successfully independently. More than to do this successfully, but actually to revel in it. For A12, returning from placement and finding friends had all gone, started to study independently and enjoy that challenge and see it having a beneficial effect on marks. Coming back from placement, all my friends left, ‘self realization of helping myself…. I was really able to kind of understand how much I was able to gain and achieve by just working independently. Yeah and I really cherished my alone time, for me to explore things on my own and discover things.” (12) For A8, returning from a placement where had been supervised by a ‘hands-off’ manager who had largely allowed her to pursue her own interests. Now applying this new independence to study and assignment tasks. For A5 it was about personal maturity (independence?) “I definitely, when I started Brookes I was a completely different person to the girl who left Brookes. I think, I mean I think I’ve always been fairly mature for my age and I never really felt right in my shoes from that. But when I left Brookes I felt like I fitted in my boots a lot better, like I still wasn’t completely there, but I found my niche, if that makes sense. “ For A7 it was independence of choice I think being given the freedom to choose your own topic within a broader topic, because then you’re really getting down to something you’re interested in and from the psychology perspective you had like a role in the choice making that it seems more like your decision, so you’re more motivated to complete it.