Lessons learned from 200K students and 2 GB of learning gains data.
Download as PPTX, PDF1 like1,647 views
The document outlines the findings of the ABC project, which analyzed learning gains data from over 200,000 students across multiple studies focused on affective, behavioral, and cognitive learning outcomes. It highlights that traditional proxies for measuring learning gains, such as student satisfaction and engagement data, may not accurately reflect true learning achievements. The research calls for more reliable methods to assess and compare learning gains across different educational contexts and institutions.
![What students think they gain?
I think I am more openly critical
(in the positive sense)
Day to day when I have my book I have very different
approach from recording my notes for example
[in my new job], there will reports and
planning to be drawn and I think that this will
be an aspect of my job where I can say yes
the OU study and discipline I’ve received
from the OU has actually contributed to that.
I observe things better, work into deeper and
work on the whole picture rather than narrow.
I think more logically and more ‘why did that
happen, why did that happen’, there is more
questioning, instead of just to accept things.
I am much better at time management, I am much more
organised now and planning things in advance.
now I say, ‘you know what, I can do that in future’.
I feel more confident and I am happier
because I am doing something I have always
wanted to be doing and something that
interests me
I think I can go confidently to
speak what I learned. But
even to a job that isn’t directly
related to this subject area. I
could talk about my
experiences, my time
management, team working,
computer skills as I feel much
more confident, I can say,
‘actually I have done this’.
Which was one of the
reasons I wanted to a degree.](https://image.slidesharecdn.com/legacyrienties26092018-180927092141/85/Lessons-learned-from-200K-students-and-2-GB-of-learning-gains-data-28-320.jpg)
![Do grades matter?
How well do your grades represent your progress?
probably in the same way that many other people when
they look at their own assignment results and exam results
…. I feel that I am doing fairly well but I’d always like to
improve myself to my results.
I get quite upset when I get around 70s
… because I am putting so much effort I want my grades to reflect it.
They usually go up. But it is Marginal. 5 marks across all the TMAs
that’s the variance, it just varies very slightly
Even if it is 1-2 marks I say what did I do differently and
I go back to tutor to see what did I do differently. What
happened, what caused it?
Well there are questions with the text books, exercises. So
if I get correct answer, I know I am doing fine. When I say
correct answer that’s not the end product that’s the whole
answer check through it
“I suppose you could say… the skills you learn, like group work, presenting and being able to talk to people…
I would say the main way that you think about [achievement], it’s just the grade because… that’s what is going
on your CV… and affect what job you get. … I’d say the skills you learn as well as becoming an all-rounded person
are quite important as well”.](https://image.slidesharecdn.com/legacyrienties26092018-180927092141/85/Lessons-learned-from-200K-students-and-2-GB-of-learning-gains-data-29-320.jpg)
Lessons learned from 200K students and 2 GB of learning gains data.
- 1. Lessons learned from 200K students and 2 GB of learning gains data https://twitter.com/LearningGains https://abclearninggains.com/ The results of the ABC project were made possible due to Simon Cross, Ceri Hitching, Ian Kinchin, Simon Lygo-Baker, Allison Littlejohn, Jekaterina Rogaten, Bart Rienties, George Roberts, Ian Scott, Rhona Sharpe, Steve Warburton, and Denise Whitelock. Please contract bart.rienties@open.ac.uk if you want to know more about ABC learning gains
- 2. Research Team Prof Bart Rienties Open University Dr Jekaterina Rogaten Open University Dr Simon Cross Open University Dr Ian Scott Oxford Brookes Prof Ian Kinchin University of Surrey Prof Denise Whitelock Open University Prof Allison Littlejohn Open University Prof Rhona Sharpe University of Surrey Dr Simon Lygo-Baker University of Surrey Dr George Roberts Oxford Brookes
- 3. Agenda for today 1. What have we learned from systematic literature review of 52 studies focussed on ABC learning gains (n = 42000+ students)? 2. What have we learned from 200K students and 2 GB of learning gains data in terms of ABC learning gains?
- 5. How are learning gains measured: a systematic analysis 52 studies selected: 42000+ students The concept of learning gain is primarily used to examine the effect of any particular educational ‘intervention’ There is a gradual increase in studies examining learning gains all across the world All learning gains can be classified into ABC 53% 16% 21% 10% Behaviour-Cognitive Learning Gains Affective-Behaviour-Cognitive Learning Gains Cognitive Learning Gains Rogaten, J., Rienties, B, Cross, S., Whitelock, D., Sharpe, R., Lygo-Baker, S., Littlejohn, A. (2018). Reviewing affective, behavioural, and cognitive learning gains in higher education. Affective-Cognitive Learning Gains Year Numberofstudies
- 6. What type of learning gains are there Affective learning gains: • Attitude • Confidence • Enjoyment • Enthusiasm for a topic • Feeling comfortable with complex ideas • Interest in a topic • Motivation • Satisfaction • Self-efficacy Cognitive learning gains: • Students’ ability to evaluate and create knowledge • Analytical ability • Autonomous cognition • Critical thinking • Ethical thinking • Creative and higher order thinking Discipline specific skills • Knowledge and understanding of the topic, • Oral and written communication • Problem solving • Scientific reasoning • Statistical and research kills/knowledge Behavioural learning gains: •Ability to work independently •Applied conceptual understanding •Effort and engagement •Leadership skills •Team/group working skills •Practical competence •Resource management •Responsibility •Preparation skills •Time management skills Rogaten, J., Rienties, B, Cross, S., Whitelock, D., Sharpe, R., Lygo-Baker, S., Littlejohn, A. (2018). Reviewing affective, behavioural, and cognitive learning gains in higher education. Assessment & Evaluation in
- 7. Affective learning gains Affective learning gains were measured in 19 studies (e.g., Moorer, 2009; Strayhorn, 2010) comprising 28 student samples totalling 3,333 higher education students. Self-reported affective learning gains there were mainly studies that reported relatively high learning gains of > 40%, ranging from 39- 98% Rogaten, J., Rienties, B, Cross, S., Whitelock, D., Sharpe, R., Lygo-Baker, S., Littlejohn, A. (2018). Reviewing affective, behavioural, and cognitive learning gains in higher education. Assessment & Evaluation in
- 8. Behavioural learning gains Behavioural learning gains were measured in 13 studies (e.g., Casem, 2006; Varsavsky et al., 2014) comprising 23 student samples totalling 4,268 higher education students. With the exception of one study (Stolk and Martello 2015), the remaining 12 studies used a cross- sectional design for measuring behavioural learning gains Rogaten, J., Rienties, B, Cross, S., Whitelock, D., Sharpe, R., Lygo-Baker, S., Littlejohn, A. (2018). Reviewing affective, behavioural, and cognitive learning gains in higher education. Assessment & Evaluation in
- 9. Cognitive learning gains Cognitive learning gains were measured in 22 studies, comprising 39 student samples, totalling 18,024 higher education students. Pre-post testing was used in four studies, and two studies used a form of pre-post testing through reflection all totalling to seven student samples. Only in one sample (Stolk and Martello 2015) did students report lower cognitive ability at the post- test than at the pre-test, but the difference was not significant. Rogaten, J., Rienties, B, Cross, S., Whitelock, D., Sharpe, R., Lygo-Baker, S., Littlejohn, A. (2018). Reviewing affective, behavioural, and cognitive learning gains in higher education. Assessment & Evaluation in
- 10. Cognitive learning gains Follow-up analyses of treatment studies which compared a change in curricular or module design (treatment) enhanced students learning in comparison to traditional lectures (control) found that students performed better in the treatment condition <g> = 0.39 than in control condition <g> = 0.26. Again a wide range of learning gains were found, whereby average normalised learning gains g ranged from -.20 to .81, Rogaten, J., Rienties, B, Cross, S., Whitelock, D., Sharpe, R., Lygo-Baker, S., Littlejohn, A. (2018). Reviewing affective, behavioural, and cognitive learning gains in higher education. Assessment & Evaluation in
- 13. 2. What have we learned from 200K students and 2 GB of learning gains data in terms of ABC learning gains? • Affective: 1 cycle of data from OU, 1 cycle from OB • Behavioural: 1 cycle of data from OU, 1 cycle from US • Cognitive: 2 cycles of data from OU, 1 cycle from US and OB • Communication with OFS with types of data collected OU OB US data Grades and demographics data Grades and demographics data Grades and demographics data File size 1.45GB 5,59 MB 213 MB +26.78 MB Number of students 166,722 2,653 (21 – 241 per department) 25,825 (171 – 4276) Number of qualifications/departments 246 18 21
- 14. Affective learning gains Using student satisfaction data for proxies for affective learning gains was not an appropriate approach. First, there was a lack of consistent data over time for sufficiently large numbers of students. Second, substantial variation in student satisfaction rates across modules, so changes in measured affective learning gains are more likely to arise from differences in sequences of modules using student satisfaction data for proxies for affective learning gains was not an appropriate approach. Although there are several studies highlighting the importance of affective learning gains12-14 Third, those who completed the student satisfaction surveys were not representative for the wider student population. Fourth, when comparing the approaches across the institutions, the lack of standardisation of student satisfaction approaches, constructs, and items made it impossible to compare potential differences in learning gains across institutions over time.
- 15. Behavioural learning gains Engagement data from VLE not good proxy for behaviour learning gains First of all, engagement of students in a respective module is strongly dependent by the learning design. Second, even if proxies for engagement could be identified, our research showed that the types of engagement will heavily be influenced by the type of learning design using student satisfaction data for proxies for affective learning gains was not an appropriate approach. Although there are several studies highlighting the importance of affective learning gains12-14 Third, related research looking at fine-grained analyses of what students are actually studying, and when, showed substantial variation in engagement and successful learning approaches In other words, our longitudinal analyses showed that our LMS proxies of engagement were not effective for understanding how students made behavioural learning gains over time.
- 16. Using Grades as proxies for cognitive learning gains David Boud44: ”The most problematic feature of current marking practice is that it is not possible to associate any reported mark with what a student can or cannot do. The meaning of the mark is not described in terms of the standards to be reached as articulated in the stated learning outcomes. Outside its immediate context, it is not clear what meaning should be attached to a mark. Marks act as obscuring devices”.
- 17. Estimating learning trajectories Level 1 Level 2 Level 3 Grade1 Student1 Grade3 Grade1Grade2Grade3Grade1Grade2Grade3Grade2 Student2 Student3 Course1 Course2 Grade1Grade2Grade3 Student4 Grade1Grade2Grade3 Student5 Course3
- 18. Cognitive learning gains Cognitive learning gains were measured in five ways: 1. Cognitive learning gains within modules 2. Cognitive learning gains from first to second module 3. Cognitive learning gains within a qualification 4. Cognitive learning gains across different qualifications 5. Cognitive learning gains between institutions Rogaten, J., Rienties, B., & Whitelock, D. (2017). Assessing Learning Gains. In D. Joosten-ten Brinke & M. Laanpere (Eds.), Technology Enhanced Assessment. TEA 2016. Communications in Computer and
- 19. Cognitive learning gains Cognitive learning gains were measured in five ways: 1. Cognitive learning gains within modules 2. Cognitive learning gains from first to second module 3. Cognitive learning gains within a qualification 4. Cognitive learning gains across different qualifications 5. Cognitive learning gains between institutions Rogaten, J., & Rienties, B. (2018). Which first-year students are making most learning gains in STEM subjects? Higher Education Pedagogies, 3(1), 161-172. doi: 10.1080/23752696.2018.1484671.
- 20. Cognitive learning gains Cognitive learning gains were measured in five ways: 1. Cognitive learning gains within modules 2. Cognitive learning gains from first to second module 3. Cognitive learning gains within a qualification 4. Cognitive learning gains across different qualifications 5. Cognitive learning gains between institutions Rogaten, J., Clow, D., Edwards, C., Gaved, M., Rienties, B. (Accepted with minor revision: 12-07-2018). Do we need to re-imagine university assessment in a digital world? A big data exploration. Margaret
- 21. Cognitive learning gains Cognitive learning gains were measured in five ways: 1. Cognitive learning gains within modules 2. Cognitive learning gains from first to second module 3. Cognitive learning gains within a qualification 4. Cognitive learning gains across different qualifications 5. Cognitive learning gains between institutions Rienties, B., Rogaten, J., Nguyen, Q., Edwards, C., Gaved, M., Holt, D., . . . Ullmann, T. (2017). Scholarly insight Spring 2017: a Data wrangler perspective. Milton Keynes: Open University UK.
- 22. Cognitive learning gains Cognitive learning gains were measured in five ways: 1. Cognitive learning gains within modules 2. Cognitive learning gains from first to second module 3. Cognitive learning gains within a qualification 4. Cognitive learning gains across different qualifications 5. Cognitive learning gains between institutions Rienties, B., Rogaten, J., Nguyen, Q., Edwards, C., Gaved, M., Holt, D., . . . Ullmann, T. (2017). Scholarly insight Spring 2017: a Data wrangler perspective. Milton Keynes: Open University UK.
- 23. Cognitive learning gains Cognitive learning gains were measured in five ways: 1. Cognitive learning gains within modules 2. Cognitive learning gains from first to second module 3. Cognitive learning gains within a qualification 4. Cognitive learning gains across different qualifications 5. Cognitive learning gains between institutions Rienties, B., Rogaten, J., Nguyen, Q., Edwards, C., Gaved, M., Holt, D., . . . Ullmann, T. (2017). Scholarly insight Spring 2017: a Data wrangler perspective. Milton Keynes: Open University UK.
- 24. Cognitive learning gains Cognitive learning gains were measured in five ways: 1. Cognitive learning gains within modules 2. Cognitive learning gains from first to second module 3. Cognitive learning gains within a qualification 4. Cognitive learning gains across different qualifications 5. Cognitive learning gains between institutions Rienties, B., Rogaten, J., Nguyen, Q., Edwards, C., Gaved, M., Holt, D., . . . Ullmann, T. (2017). Scholarly insight Spring 2017: a Data wrangler perspective. Milton Keynes: Open University UK.
- 25. Cognitive learning gains Cognitive learning gains were measured in five ways: 1. Cognitive learning gains within modules 2. Cognitive learning gains from first to second module 3. Cognitive learning gains within a qualification 4. Cognitive learning gains across different qualifications 5. Cognitive learning gains between institutions Rienties, B., Rogaten, J., Nguyen, Q., Edwards, C., Gaved, M., Holt, D., . . . Ullmann, T. (2017). Scholarly insight Spring 2017: a Data wrangler perspective. Milton Keynes: Open University UK.
- 26. Cognitive learning gains Cognitive learning gains were measured in five ways: 1. Cognitive learning gains within modules 2. Cognitive learning gains from first to second module 3. Cognitive learning gains within a qualification 4. Cognitive learning gains across different qualifications 5. Cognitive learning gains between institutions Rienties, B., Rogaten, J., Nguyen, Q., Edwards, C., Gaved, M., Holt, D., . . . Ullmann, T. (2017). Scholarly insight Spring 2017: a Data wrangler perspective. Milton Keynes: Open University UK. The proportion of variance due to differences OU OB US Level 3: Between qualifications 12% 8% 22% Level 2: Between students 45% 67% 22%* Level 1 Between modules (i.e., within- student level between modules any one student completed) 43% 25% 56% Number of students (n) 18329 1990 1547 Table 1 Proportion of variance explained by qualification, student characteristics, and across modules (OU, OB, US)
- 27. Our qualitative data shows a different picture using student satisfaction data for proxies for affective learning gains was not an appropriate approach. Although there are several studies highlighting the importance of affective learning gains12-14
- 28. What students think they gain? I think I am more openly critical (in the positive sense) Day to day when I have my book I have very different approach from recording my notes for example [in my new job], there will reports and planning to be drawn and I think that this will be an aspect of my job where I can say yes the OU study and discipline I’ve received from the OU has actually contributed to that. I observe things better, work into deeper and work on the whole picture rather than narrow. I think more logically and more ‘why did that happen, why did that happen’, there is more questioning, instead of just to accept things. I am much better at time management, I am much more organised now and planning things in advance. now I say, ‘you know what, I can do that in future’. I feel more confident and I am happier because I am doing something I have always wanted to be doing and something that interests me I think I can go confidently to speak what I learned. But even to a job that isn’t directly related to this subject area. I could talk about my experiences, my time management, team working, computer skills as I feel much more confident, I can say, ‘actually I have done this’. Which was one of the reasons I wanted to a degree.
- 29. Do grades matter? How well do your grades represent your progress? probably in the same way that many other people when they look at their own assignment results and exam results …. I feel that I am doing fairly well but I’d always like to improve myself to my results. I get quite upset when I get around 70s … because I am putting so much effort I want my grades to reflect it. They usually go up. But it is Marginal. 5 marks across all the TMAs that’s the variance, it just varies very slightly Even if it is 1-2 marks I say what did I do differently and I go back to tutor to see what did I do differently. What happened, what caused it? Well there are questions with the text books, exercises. So if I get correct answer, I know I am doing fine. When I say correct answer that’s not the end product that’s the whole answer check through it “I suppose you could say… the skills you learn, like group work, presenting and being able to talk to people… I would say the main way that you think about [achievement], it’s just the grade because… that’s what is going on your CV… and affect what job you get. … I’d say the skills you learn as well as becoming an all-rounded person are quite important as well”.
- 32. Implications for practice • Substantial freedom for students to select “unique” pathways: some programmes and qualifications have relatively fixed and structured pathways. Other programmes and qualifications offer students wide and far reaching freedom to choose (one qualification had 84 potential pathways to complete a degree). However, institutions provide limited to no structural support which pathways would fit students’ needs and abilities. Recommendation 1: Institutions needs to improve how we communicate to our students which modules fit with their needs and abilities, and be more explicit about successful pathways for students to obtain a qualification. Rogaten, J., Clow, D., Edwards, C., Gaved, M., Rienties, B. (Accepted with minor revision: 12-07-2018). Do we need to re-imagine university assessment in a digital world? A big data exploration. Margaret Bearman, Phillip Dawson, Rola Ajjawi, Joanna Tai, David Boud (Eds). Re-imagining University Assessment in a Digital World. Springer.
- 33. Implications for practice • Alignment of modules within a qualification: students experience substantially different learning designs, assessment practices, and workload fluctuations when transitioning from one module to another. Recommendation 2: Institutions need to improve how we communicate and manage the students’ expectations of the learning designs and assessment practices from one module to another. Recommendation 3: In the longer term, it would be beneficial to align the module designs across a qualification based upon evidence-based practice and what works, thereby allowing smooth transitions from one module to another in a qualification. Rogaten, J., Clow, D., Edwards, C., Gaved, M., Rienties, B. (Accepted with minor revision: 12-07-2018). Do we need to re-imagine university assessment in a digital world? A big data exploration. Margaret Bearman, Phillip Dawson, Rola
- 34. Implications for practice • Alignment of marking within and across modules within and across qualifications: “embedded expectations”, norms and practice influence marking practices. Across some qualifications there appears to be a widespread deliberate approach of making early assessment relatively easy, both within modules (particularly the first assignment) and within qualifications (particularly the first module). This approach is intended to reduce drop-out, but may have unintended consequences. • Furthermore, given that in most modules teachers are marking relatively small numbers of students, potential misalignments might be present which may not be immediately apparent when just looking at average grades and the normal distribution curves. • Another potential explanation is the increasing difficulty of the material being assessed may not be completely accounted for in the marks awarded. Final-year-equivalent modules rightly contain much more difficult material than entry modules. Rogaten, J., Clow, D., Edwards, C., Gaved, M., Rienties, B. (Accepted with minor revision: 12-07-2018). Do we need to re-imagine university assessment in a digital world? A big data exploration. Margaret Bearman, Phillip Dawson, Rola
- 35. Implications for practice Recommendation 4: It is essential that grades are aligned not only within a module but also across a qualification. For exam boards we recommend to include cross- checks of previous performance of students (e.g., correlation analyses) and longitudinal analyses of historical data to determine whether previously successful students were again successful, and whether they maintained a successful learning journey after a respective module. Recommendation 5: We recommend that clearer guidelines and grade descriptors across a qualification are developed, which are clearly communicated to staff and students. Recommendation 6: Given that many students follow modules from different qualifications, it is important to develop coherent university-wide grade descriptors and align marking across qualifications. Rogaten, J., Clow, D., Edwards, C., Gaved, M., Rienties, B. (Accepted with minor revision: 12-07-2018). Do we need to re-imagine university assessment in a digital world? A big data exploration. Margaret Bearman, Phillip Dawson, Rola
- 36. Lessons learned from 200K students and 2 GB of learning gains data https://twitter.com/LearningGains https://abclearninggains.com/ The results of the ABC project were made possible due to Simon Cross, Ceri Hitching, Ian Kinchin, Simon Lygo-Baker, Allison Littlejohn, Jekaterina Rogaten, Bart Rienties, George Roberts, Ian Scott, Rhona Sharpe, Steve Warburton, and Denise Whitelock. Please contract bart.rienties@open.ac.uk if you want to know more about ABC learning gains
Editor's Notes
- #6: Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #8: Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #9: Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #10: Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #11: Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #15: Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #16: Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #18: Level 1 – Grade: repeated measures on students and tell us about students learning trajectory Level 2 – student: between students variations Level 3 – Course: between course variation
- #19: Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #20: Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #21: Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #22: Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #23: Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #24: Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #25: Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #26: Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #27: Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #28: Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #33: Add reference
- #34: Add reference
- #35: Add reference
- #36: Add reference