How learning gains and Quality Assurance are (mis)Aligned: An Interactive Workshop
Download as PPTX, PDF0 likes223 views
The document outlines an interactive workshop led by Prof. Bart Rienties and Dr. Jekaterina Rogaten that focuses on the concept of learning gains and its alignment with quality assurance in higher education. The agenda includes discussions on the definitions and usefulness of learning gains, types of learning gains (affective, behavioral, cognitive), and their measurement across various studies. It emphasizes the need for consistent and validated approaches to measuring learning gains to improve educational outcomes and teaching practices.
![What students think they gain?
I think I am more openly critical
(in the positive sense)
Day to day when I have my book I have very different
approach from recording my notes for example
[in my new job], there will reports and
planning to be drawn and I think that this will
be an aspect of my job where I can say yes
the OU study and discipline I’ve received
from the OU has actually contributed to that.
I observe things better, work into deeper and
work on the whole picture rather than narrow.
I think more logically and more ‘why did that
happen, why did that happen’, there is more
questioning, instead of just to accept things.
I am much better at time management, I am much more
organised now and planning things in advance.
now I say, ‘you know what, I can do that in future’.
I feel more confident and I am happier
because I am doing something I have always
wanted to be doing and something that
interests me
I think I can go confidently to
speak what I learned. But
even to a job that isn’t directly
related to this subject area. I
could talk about my
experiences, my time
management, team working,
computer skills as I feel much
more confident, I can say,
‘actually I have done this’.
Which was one of the
reasons I wanted to a degree.](https://image.slidesharecdn.com/learninggainsinqaahe23092020-200922143728/85/How-learning-gains-and-Quality-Assurance-are-mis-Aligned-An-Interactive-Workshop-35-320.jpg)
![Do grades matter?
How well do your grades represent your progress?
probably in the same way that many other people when
they look at their own assignment results and exam results
…. I feel that I am doing fairly well but I’d always like to
improve myself to my results.
I get quite upset when I get around 70s
… because I am putting so much effort I want my grades to reflect it.
They usually go up. But it is Marginal. 5 marks across all the TMAs
that’s the variance, it just varies very slightly
Even if it is 1-2 marks I say what did I do differently and
I go back to tutor to see what did I do differently. What
happened, what caused it?
Well there are questions with the text books, exercises. So
if I get correct answer, I know I am doing fine. When I say
correct answer that’s not the end product that’s the whole
answer check through it
“I suppose you could say… the skills you learn, like group work, presenting and being able to talk to people…
I would say the main way that you think about [achievement], it’s just the grade because… that’s what is going
on your CV… and affect what job you get. … I’d say the skills you learn as well as becoming an all-rounded person
are quite important as well”.](https://image.slidesharecdn.com/learninggainsinqaahe23092020-200922143728/85/How-learning-gains-and-Quality-Assurance-are-mis-Aligned-An-Interactive-Workshop-36-320.jpg)
How learning gains and Quality Assurance are (mis)Aligned: An Interactive Workshop
- 1. 1 Thank you for joining this interactive workshop While we are getting ready for our interactive workshop: 1.Can you indicate in the Zoom chat who you are? 2.What made you join this session on learning gains? 3.What do you hope to achieve after this interactive workshop?
- 2. How learning gains and Quality Assurance are (mis)Aligned: An Interactive Workshop Prof Bart Rienties, Head of Academic Professional Development, Open University UK Dr Jekaterina Rogaten, Teaching and Learning co-ordinator at the University of the Arts, London Milton Keynes/Online: 23-09-2020 Workshop 3 INQAAHE
- 3. Prof Bart Rienties Open University Dr Jekaterina Rogaten University of the Arts Dr Simon Cross Open University Dr Ian Scott Oxford Brookes Prof Ian Kinchin University of Surrey Prof Denise Whitelock Open University Prof Allison Littlejohn Open University Prof Rhona Sharpe University of Surrey Dr Simon Lygo-Baker University of Surrey Dr George Roberts Oxford Brookes
- 4. 4 Thank you for joining this interactive workshop While we are getting ready for our interactive workshop: 1.Can you indicate in the Zoom chat who you are? (or if you are brave turn on your mic and briefly share your thoughts)? 2.What made you join this session on learning gains? 3.What do you hope to achieve after this interactive workshop?
- 5. 5 Agenda for today 1500-1515 Welcome by INQAAHE President and getting to know each other 1515-1600 What is this concept of learning gains, and why might it be useful for your institution? • Brief definition of learning gains • Interactive discussion in two breakout rooms • Reporting of the findings/discussions from the two breakout rooms, and our own reflections 1600-1700 How could learning gains be used for quality assurance? • Brief definition of learning gains and quality assurance • Interactive discussion in two breakout rooms • Take a short break • Reporting of the findings/discussions from the two breakout rooms, and our own reflections 1700-1730 How could learning gains be useful in your context • Open discussion • Lessons-learned • Best practice • What have we learned from ABC learning gains project
- 6. 6 A brief definition of learning gains McGrath, C. H., Guerin, B., Harte, E., Frearson, M., & Manville, C. (2015). Learning gain in higher education. www.rand.org/t/rr996/ As indicated in the RAND report by McGrath et al. (2015, xi), learning gains are defined as: “as the ‘distance travelled’, or the difference between the skills, competencies, content knowledge and personal development demonstrated by students at two points in time. This allows for a comparison of academic abilities and how participation in higher education has contributed to such intellectual development.”
- 7. 7 Agenda for today 1500-1515 Welcome by INQAAHE President and getting to know each other 1515-1600 What is this concept of learning gains, and why might it be useful for your institution? •Brief definition of learning gains •What types of learning gains would be useful? •Is your institution currently using concepts like learning gain? •Why might learning gains be useful for your institution? Why not?
- 8. 8 Agenda for today 1500-1515 Welcome by INQAAHE President and getting to know each other 1515-1600 What is this concept of learning gains, and why might it be useful for your institution? • Brief definition of learning gains • Interactive discussion in two breakout rooms • Reporting of the findings/discussions from the two breakout rooms, and our own reflections 1600-1700 How could learning gains be used for quality assurance? • Brief definition of learning gains and quality assurance • Interactive discussion in two breakout rooms • Take a short break • Reporting of the findings/discussions from the two breakout rooms, and our own reflections 1700-1730 How could learning gains be useful in your context • Open discussion • Lessons-learned • Best practice • What have we learned from ABC learning gains project
- 9. How are learning gains measured: a systematic analysis 52 studies selected: 42000+ students The concept of learning gain is primarily used to examine the effect of any particular educational ‘intervention’ There is a gradual increase in studies examining learning gains all across the world All learning gains can be classified into ABC 53% 16% 21% 10% Behaviour-Cognitive Learning Gains Affective-Behaviour-Cognitive Learning Gains Cognitive Learning Gains Rogaten, J., Rienties, B, Sharpe, R., Cross, S., Whitelock, D., Lygo-Baker, S., Littlejohn, A. (2018). Reviewing affective, behavioural, and cognitive learning gains in higher education. Assessment and Evaluation in Higher Education, 44(3), 321-337. Affective-Cognitive Learning Gains Year Numberofstudies
- 10. What type of learning gains are there Affective learning gains: • Attitude • Confidence • Enjoyment • Enthusiasm for a topic • Feeling comfortable with complex ideas • Interest in a topic • Motivation • Satisfaction • Self-efficacy Cognitive learning gains: • Students’ ability to evaluate and create knowledge • Analytical ability • Autonomous cognition • Critical thinking • Ethical thinking • Creative and higher order thinking Discipline specific skills • Knowledge and understanding of the topic, • Oral and written communication • Problem solving • Scientific reasoning • Statistical and research kills/knowledge Behavioural learning gains: •Ability to work independently •Applied conceptual understanding •Effort and engagement •Leadership skills •Team/group working skills •Practical competence •Resource management •Responsibility •Preparation skills •Time management skills Rogaten, J., Rienties, B, Sharpe, R., Cross, S., Whitelock, D., Lygo-Baker, S., Littlejohn, A. (2018). Reviewing affective, behavioural, and cognitive learning gains in higher education. Assessment and Evaluation in Higher Education, 44(3), 321-337.
- 11. Affective learning gains Affective learning gains were measured in 19 studies (e.g., Moorer, 2009; Strayhorn, 2010) comprising 28 student samples totalling 3,333 higher education students. Self-reported affective learning gains there were mainly studies that reported relatively high learning gains of > 40%, ranging from 39- 98% Rogaten, J., Rienties, B, Sharpe, R., Cross, S., Whitelock, D., Lygo-Baker, S., Littlejohn, A. (2018). Reviewing affective, behavioural, and cognitive learning gains in higher education. Assessment and Evaluation in Higher Education, 44(3), 321-337.
- 12. Behavioural learning gains Behavioural learning gains were measured in 13 studies (e.g., Casem, 2006; Varsavsky et al., 2014) comprising 23 student samples totalling 4,268 higher education students. With the exception of one study (Stolk and Martello 2015), the remaining 12 studies used a cross- sectional design for measuring behavioural learning gains Rogaten, J., Rienties, B, Sharpe, R., Cross, S., Whitelock, D., Lygo-Baker, S., Littlejohn, A. (2018). Reviewing affective, behavioural, and cognitive learning gains in higher education. Assessment and Evaluation in
- 13. Cognitive learning gains Cognitive learning gains were measured in 22 studies, comprising 39 student samples, totalling 18,024 higher education students. Pre-post testing was used in four studies, and two studies used a form of pre-post testing through reflection all totalling to seven student samples. Only in one sample (Stolk and Martello 2015) did students report lower cognitive ability at the post- test than at the pre-test, but the difference was not significant. Rogaten, J., Rienties, B, Sharpe, R., Cross, S., Whitelock, D., Lygo-Baker, S., Littlejohn, A. (2018). Reviewing affective, behavioural, and cognitive learning gains in higher education. Assessment and Evaluation in
- 14. Cognitive learning gains Follow-up analyses of treatment studies which compared a change in curricular or module design (treatment) enhanced students learning in comparison to traditional lectures (control) found that students performed better in the treatment condition <g> = 0.39 than in control condition <g> = 0.26. Again a wide range of learning gains were found, whereby average normalised learning gains g ranged from -.20 to .81, Rogaten, J., Rienties, B, Sharpe, R., Cross, S., Whitelock, D., Lygo-Baker, S., Littlejohn, A. (2018). Reviewing affective, behavioural, and cognitive learning gains in higher education. Assessment and Evaluation in
- 15. 15 Learning gains concepts widely used but have different meanings in different contexts In summary • Following a robust and thorough coding and analysis of affective, behavioural, and cognitive learning gains across 52 empirical learning gain studies, our systematic review found that most studies focussed on cognitive learning gains, mostly using pre-post objective testing. • Studies examining a combination of affective, behavioural, and cognitive learning gains, but in their majority they employed cross-sectional design, and mostly relied on self-reported retrospective estimates of learning gains, with obvious potential limitations • 52 studies reported a vast range of learning gains, highlighting on the one hand the inherent and well-recognised complexity of higher education teaching practice, and on the other hand the methodological complexities of defining what actually constitutes a learning gain, and how this could potentially be measured appropriately. • The reported magnitude of learning gains amongst these 52 studies was surprisingly diverse and wide ranging. • There is a lack of consistency in the ways in which learning gains are currently measured and reported. These inconsistencies and limitations hamper effective comparisons of learning gains and teaching excellence. • As highlighted by Everson (2016) we need to make it clear to teachers and students what we mean when measuring value added. • We recommend a greater emphasis on longitudinal measurement of learning gains using validated approaches. Rogaten, J., Rienties, B, Sharpe, R., Cross, S., Whitelock, D., Lygo-Baker, S., Littlejohn, A. (2018). Reviewing affective, behavioural, and cognitive learning gains in higher education. Assessment and Evaluation in Higher Education, 44(3), 321-337.
- 16. 16 Agenda for today 1500-1515 Welcome by INQAAHE President and getting to know each other 1515-1600 What is this concept of learning gains, and why might it be useful for your institution? • Brief definition of learning gains • Interactive discussion in two breakout rooms • Reporting of the findings/discussions from the two breakout rooms, and our own reflections 1600-1700 How could learning gains be used for quality assurance? • Brief definition of learning gains and quality assurance • Interactive discussion in two breakout rooms • Take a short break • Reporting of the findings/discussions from the two breakout rooms, and our own reflections 1700-1730 How could learning gains be useful in your context • Open discussion • Lessons-learned • Best practice • What have we learned from ABC learning gains project
- 17. 17 Interactive discussion in two breakout rooms 1600-1700 How could learning gains be used for quality assurance? •Brief definition of learning gains and quality assurance •Is your institution using learning gains at present for quality assurance •Take a short break •Reporting of the findings/discussions from the two breakout rooms, and our own reflections
- 18. 18 Agenda for today 1500-1515 Welcome by INQAAHE President and getting to know each other 1515-1600 What is this concept of learning gains, and why might it be useful for your institution? • Brief definition of learning gains • Interactive discussion in two breakout rooms • Reporting of the findings/discussions from the two breakout rooms, and our own reflections 1600-1700 How could learning gains be used for quality assurance? • Brief definition of learning gains and quality assurance • Interactive discussion in two breakout rooms • Take a short break • Reporting of the findings/discussions from the two breakout rooms, and our own reflections 1700-1730 How could learning gains be useful in your context • Open discussion • Lessons-learned • Best practice • What have we learned from ABC learning gains project
- 19. 19 Agenda for today 1500-1515 Welcome by INQAAHE President and getting to know each other 1515-1600 What is this concept of learning gains, and why might it be useful for your institution? • Brief definition of learning gains • Interactive discussion in two breakout rooms • Reporting of the findings/discussions from the two breakout rooms, and our own reflections 1600-1700 How could learning gains be used for quality assurance? • Brief definition of learning gains and quality assurance • Interactive discussion in two breakout rooms • Take a short break • Reporting of the findings/discussions from the two breakout rooms, and our own reflections 1700-1730 How could learning gains be useful in your context • Open discussion • Lessons-learned • Best practice • What have we learned from ABC learning gains project
- 20. 20 Downloads 1. Rogaten, Jekaterina; Clow, Doug; Edwards, Chris; Gaved, Mark and Rienties, Bart (2020). Are Assessment Practices Well Aligned Over Time? A Big Data Exploration. In: Bearman, M; Dawson, P; Ajjawi, R.; Tai, J. and Boud, D. eds. Re-imagining University Assessment in a Digital World. The Enabling Power of Assessment, 7. Cham: Springer, pp. 147–164. •2. Rogaten, Jekaterina and Rienties, Bart (2020). A critical review of learning gains on its methods and approaches. In: Hughes, Christina and Tight, Malcolm eds. International Perspectives on Higher Education Research. Emerald, (In Press). •3. Rogaten, Jekaterina; Rienties, Bart; Sharpe, Rhona; Cross, Simon; Whitelock, Denise; Lygo-Baker, Simon and Littlejohn, Allison (2019). Reviewing affective, behavioural, and cognitive learning gains in higher education. Assessment & Evaluation in Higher Education, 44(3) pp. 321–337. •4. Rogaten, Jekaterina and Rienties, Bart (2018). Which first-year students are making most learning gains in STEM subjects? Higher Education Pedagogies, 3(1) pp. 161–172. •5. Rogaten, Jekaterina; Whitelock, Denise and Rienties, Bart (2016). Assessing learning gains. In: Technology Enhanced Assessment, Communications in Computer and Information Science, pp. 117–132.
- 21. How learning gains and Quality Assurance are (mis)Aligned: An Interactive Workshop Prof Bart Rienties, Head of Academic Professional Development, Open University UK Bart.Rienties@open.ac.uk Dr Jekaterina Rogaten, Teaching and Learning co-ordinator at the University of the Arts, London j.rogaten@fashion.arts.ac.uk Milton Keynes/Online: 23-09-2020 Workshop 3 INQAAHE
- 22. • Affective: 1 cycle of data from OU, 1 cycle from OB • Behavioural: 1 cycle of data from OU, 1 cycle from US • Cognitive: 2 cycles of data from OU, 1 cycle from US and OB • Communication with OFS with types of data collected OU OB US data Grades and demographics data Grades and demographics data Grades and demographics data File size 1.45GB 5,59 MB 213 MB +26.78 MB Number of students 166,722 2,653 (21 – 241 per department) 25,825 (171 – 4276) Number of qualifications/departments 246 18 21
- 23. Affective learning gains • Using student satisfaction data for proxies for affective learning gains was not an appropriate approach. • First, there was a lack of consistent data over time for sufficiently large numbers of students. • Second, substantial variation in student satisfaction rates across modules, so changes in measured affective learning gains are more likely to arise from differences in sequences of modules Third, those who completed the student satisfaction surveys were not representative for the wider student population. Fourth, when comparing the approaches across the institutions, the lack of standardisation of student satisfaction approaches, constructs, and items made it impossible to compare potential differences in learning gains across institutions over time.
- 24. Behavioural learning gains • Engagement data from VLE not good proxy for behaviour learning gains • First of all, engagement of students in a respective module is strongly dependent by the learning design. • Second, even if proxies for engagement could be identified, our research showed that the types of engagement will heavily be influenced by the type of learning design Third, related research looking at fine-grained analyses of what students are actually studying, and when, showed substantial variation in engagement and successful learning approaches In other words, our longitudinal analyses showed that our LMS proxies of engagement were not effective for understanding how students made behavioural learning gains over time.
- 25. Estimating learning trajectories Level 1 Level 2 Level 3 Grade1 Student1 Grade3 Grade1Grade2Grade3Grade1Grade2Grade3Grade2 Student2 Student3 Course1 Course2 Grade1Grade2Grade3 Student4 Grade1Grade2Grade3 Student5 Course3
- 26. Cognitive learning gains • Cognitive learning gains were measured in five ways: • 1. Cognitive learning gains within modules • 2. Cognitive learning gains from first to second module • 3. Cognitive learning gains within a qualification • 4. Cognitive learning gains across different qualifications • 5. Cognitive learning gains between institutions Rogaten, J., Rienties, B., & Whitelock, D. (2017). Assessing Learning Gains. In D. Joosten-ten Brinke & M. Laanpere (Eds.), Technology Enhanced Assessment. TEA 2016. Communications in Computer and
- 27. Cognitive learning gains • Cognitive learning gains were measured in five ways: • 1. Cognitive learning gains within modules • 2. Cognitive learning gains from first to second module • 3. Cognitive learning gains within a qualification • 4. Cognitive learning gains across different qualifications • 5. Cognitive learning gains between institutions Rogaten, J., & Rienties, B. (2018). Which first-year students are making most learning gains in STEM subjects? Higher Education Pedagogies, 3(1), 161-172. doi: 10.1080/23752696.2018.1484671.
- 28. Cognitive learning gains • Cognitive learning gains were measured in five ways: • 1. Cognitive learning gains within modules • 2. Cognitive learning gains from first to second module • 3. Cognitive learning gains within a qualification • 4. Cognitive learning gains across different qualifications • 5. Cognitive learning gains between institutions Rogaten, J., Clow, D., Edwards, C., Gaved, M., Rienties, B. (2020). Are assessment practices well aligned over time? A big data exploration. In M. Bearman, P. Dawson, R. Ajjawi, J. Tai, D. Boud (Eds). Re-
- 29. Cognitive learning gains • Cognitive learning gains were measured in five ways: • 1. Cognitive learning gains within modules • 2. Cognitive learning gains from first to second module • 3. Cognitive learning gains within a qualification • 4. Cognitive learning gains across different qualifications • 5. Cognitive learning gains between institutions Rienties, B., Rogaten, J., Nguyen, Q., Edwards, C., Gaved, M., Holt, D., . . . Ullmann, T. (2017). Scholarly insight Spring 2017: a Data wrangler perspective. Milton Keynes: Open University UK.
- 30. Cognitive learning gains • Cognitive learning gains were measured in five ways: • 1. Cognitive learning gains within modules • 2. Cognitive learning gains from first to second module • 3. Cognitive learning gains within a qualification • 4. Cognitive learning gains across different qualifications • 5. Cognitive learning gains between institutions Rienties, B., Rogaten, J., Nguyen, Q., Edwards, C., Gaved, M., Holt, D., . . . Ullmann, T. (2017). Scholarly insight Spring 2017: a Data wrangler perspective. Milton Keynes: Open University UK.
- 31. Cognitive learning gains • Cognitive learning gains were measured in five ways: • 1. Cognitive learning gains within modules • 2. Cognitive learning gains from first to second module • 3. Cognitive learning gains within a qualification • 4. Cognitive learning gains across different qualifications • 5. Cognitive learning gains between institutions Rienties, B., Rogaten, J., Nguyen, Q., Edwards, C., Gaved, M., Holt, D., . . . Ullmann, T. (2017). Scholarly insight Spring 2017: a Data wrangler perspective. Milton Keynes: Open University UK.
- 32. Cognitive learning gains • Cognitive learning gains were measured in five ways: • 1. Cognitive learning gains within modules • 2. Cognitive learning gains from first to second module • 3. Cognitive learning gains within a qualification • 4. Cognitive learning gains across different qualifications • 5. Cognitive learning gains between institutions Rienties, B., Rogaten, J., Nguyen, Q., Edwards, C., Gaved, M., Holt, D., . . . Ullmann, T. (2017). Scholarly insight Spring 2017: a Data wrangler perspective. Milton Keynes: Open University UK.
- 33. Cognitive learning gains • Cognitive learning gains were measured in five ways: • 1. Cognitive learning gains within modules • 2. Cognitive learning gains from first to second module • 3. Cognitive learning gains within a qualification • 4. Cognitive learning gains across different qualifications • 5. Cognitive learning gains between institutions Rienties, B., Rogaten, J., Nguyen, Q., Edwards, C., Gaved, M., Holt, D., . . . Ullmann, T. (2017). Scholarly insight Spring 2017: a Data wrangler perspective. Milton Keynes: Open University UK.
- 34. Cognitive learning gains • Cognitive learning gains were measured in five ways: • 1. Cognitive learning gains within modules • 2. Cognitive learning gains from first to second module • 3. Cognitive learning gains within a qualification • 4. Cognitive learning gains across different qualifications • 5. Cognitive learning gains between institutions Rienties, B., Rogaten, J., Nguyen, Q., Edwards, C., Gaved, M., Holt, D., . . . Ullmann, T. (2017). Scholarly insight Spring 2017: a Data wrangler perspective. Milton Keynes: Open University UK. The proportion of variance due to the differences OU OB US Level 3: Between qualifications 12% 8% 22% Level 2: Between students 45% 67% 22%* Level 1 Between modules (i.e., within-student level between modules any one student completed) 43% 25% 56% Number of students (n) 18329 1990 1547 Table 1 Proportion of variance explained by qualification, student characteristics, and across modules (OU, OB, US)
- 35. What students think they gain? I think I am more openly critical (in the positive sense) Day to day when I have my book I have very different approach from recording my notes for example [in my new job], there will reports and planning to be drawn and I think that this will be an aspect of my job where I can say yes the OU study and discipline I’ve received from the OU has actually contributed to that. I observe things better, work into deeper and work on the whole picture rather than narrow. I think more logically and more ‘why did that happen, why did that happen’, there is more questioning, instead of just to accept things. I am much better at time management, I am much more organised now and planning things in advance. now I say, ‘you know what, I can do that in future’. I feel more confident and I am happier because I am doing something I have always wanted to be doing and something that interests me I think I can go confidently to speak what I learned. But even to a job that isn’t directly related to this subject area. I could talk about my experiences, my time management, team working, computer skills as I feel much more confident, I can say, ‘actually I have done this’. Which was one of the reasons I wanted to a degree.
- 36. Do grades matter? How well do your grades represent your progress? probably in the same way that many other people when they look at their own assignment results and exam results …. I feel that I am doing fairly well but I’d always like to improve myself to my results. I get quite upset when I get around 70s … because I am putting so much effort I want my grades to reflect it. They usually go up. But it is Marginal. 5 marks across all the TMAs that’s the variance, it just varies very slightly Even if it is 1-2 marks I say what did I do differently and I go back to tutor to see what did I do differently. What happened, what caused it? Well there are questions with the text books, exercises. So if I get correct answer, I know I am doing fine. When I say correct answer that’s not the end product that’s the whole answer check through it “I suppose you could say… the skills you learn, like group work, presenting and being able to talk to people… I would say the main way that you think about [achievement], it’s just the grade because… that’s what is going on your CV… and affect what job you get. … I’d say the skills you learn as well as becoming an all-rounded person are quite important as well”.
Editor's Notes
- #2: Start with Bart By having this start-up screen hopefully we can start to prime why people are here, and we will repeat this for late-comers and people who are slow typers. By the way if a slide is pink, this means interactivity for participants . When a slide is white it is us talking
- #4: Bart and Jekaterina briefly introduce themselves
- #5: Start with Bart By having this start-up screen hopefully we can start to prime why people are here, and we will repeat this for late-comers and people who are slow typers. By the way if a slide is pink, this means interactivity for participants . When a slide is white it is us talking
- #6: Bart
- #7: Bart
- #8: Bart and Jekaterina in two different groups
- #9: So what have we learned from the two breakout rooms?
- #10: Jekaterina: I think it would be useful once we have heard from the workshop participants about their thoughts that we briefly present the findings from our systematic literature review. When I say briefly I would say 5-10 minutes max. Would this be ok for you?
- #12: Jekaterina: I think it would be useful once we have heard from the workshop participants about their thoughts that we briefly present the findings from our systematic literature review. When I say briefly I would say 5-10 minutes max. Would this be ok for you?
- #17: Bart
- #18: Bart and Jekaterina in two different groups
- #19: So what have we learned from the second breakout session?
- #20: So what have we learned from the second breakout session?
- #22: Jekaterina: I mainly hope that we can learn from the participants. However, if the discussions fall flat we can afterwards share some of the findings that I have copy/pasted afterwards. Not sure if we should go through all the slides as I am keen to keep the interactive workshop format, but hey
- #24: Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #25: Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #26: Level 1 – Grade: repeated measures on students and tell us about students learning trajectory Level 2 – student: between students variations Level 3 – Course: between course variation
- #27: Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #28: Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #29: Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #30: Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #31: Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #32: Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #33: Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #34: Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #35: Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)