Moving through MOOCs: Pedagogy, Learning and Patterns of Engagement
Download as PPTX, PDF8 likes6,422 views
The document examines learner engagement patterns in MOOCs using data from multiple universities, identifying four main engagement categories: sampling, auditing, disengaging, and completing learners. It highlights that engagement patterns may vary across different MOOCs and suggests that pedagogical changes can significantly influence learner engagement. Additionally, it discusses strategies to improve learning and support for different learner profiles, emphasizing the need for adaptive course features.
![10
Typical engagement profiles
These profiles apply to an eight-week course
● Samplers visit only briefly
[1, 0, 0, 0, 0, 0, 0, 0] – 1 means they visited content
● Strong Starters do first assessment
[9, 1, 0, 0, 0, 0, 0, 0] – 9 means they visited content and did assessment on time
● Returners come back in Week 2 [9, 9, 0, 0, 0, 0, 0, 0]
● Mid-way Dropouts
[9, 9, 9, 4, 1, 1, 0, 0] – 4 means they submitted assessment late
● Nearly There drop out near the end
[11, 11, 9, 11, 9, 9, 0, 0] – 11 means full engagement, 8 means submission on time
● Late Completers finish
[5, 5, 5, 5, 5, 9, 9, 9] – 5 means they viewed content and submitted late
● Keen Completers do almost everything [11, 11, 9, 9, 11, 11, 9, 9]
OU FL study](https://image.slidesharecdn.com/movingthroughmoocs-150916130934-lva1-app6892/85/Moving-through-MOOCs-Pedagogy-Learning-and-Patterns-of-Engagement-10-320.jpg)
Moving through MOOCs: Pedagogy, Learning and Patterns of Engagement
- 1. Moving through MOOCs: Pedagogy, Learning and Patterns of Engagement Rebecca Ferguson, Doug Clow (OU) Russell Beale, Alison J Cooper (Birmingham) Neil Morris (Leeds) Siân Bayne, Amy Woodgate (Edinburgh)
- 2. Current context Students seek not merely access, but access to success “ John Daniel, 2012 % complete from: www.katyjordan.com/MOOCproject ”
- 3. Patterns of engagement: Coursera ● Sampling learners explored some course materials ● Auditing learners watched most videos, but completed assessments rarely, if at all ● Disengaging learners completed assessments at the start of the course and then reduced their engagement ● Completing learners completed most assessments MOOC designers can apply this simple and scalable categorization to target interventions and develop adaptive course features “ ” Coursera study Kizilcec, R., Piech, C., and Schneider, E., 2013. Deconstructing disengagement: analyzing learner subpopulations in massive open online courses. LAK13
- 4. 4 Replication Open University FutureLearn data Replication MOOC1 MOOC2 MOOC3 MOOC4 Subject area Physical sciences Life sciences Arts Business M 51% 39% 32% 35% F 48% 61% 67% 65% Participants 5,069 3,238 16,118 9,778 Fully participating 1,548 684 3,616 1,416 Participation rate 31% 21% 22% 14%
- 5. 5 Calculating an activity profile Replicating the method ● T = on track (3) undertook the assessment on time ● B = behind (2) submitted the assessment late ● A = auditing (1) engaged with content but not assessment ● O = out (0) did not participate Replication
- 6. 6 Replication Identifying dissimilarity between engagement patterns Assigned numerical value to each label ● On track = 3 ● Behind = 2 ● Auditing = 1 ● Out = 0 Calculated L1 norm for each engagement pattern Used that as the basis for one-dimensional k-means clustering Repeated clustering 100 times and selected solution with highest likelihood Focused on extracting four clusters Replication
- 7. 7 Replication Coursera and FutureLearn results were different ● Sampling learners explored some course materials ● Auditing learners watched most videos, but completed assessments rarely, if at all ● Disengaging learners completed assessments at the start of the course and then reduced their engagement ● Completing learners completed most assessments √ √ x x They also differed when we tried ●Different values for k ●A one-dimensional approach ●Running k means directly on engagement profiles Replication
- 8. 8 FutureLearn is different Conversation is a central feature Sharples, M., & Ferguson, R. (2014). Innovative Pedagogy at Massive Scale: Teaching and Learning in MOOCs. ECTEL 2014.
- 9. 9 Revising the numeric values OU FL study 1 only visited content (for example, video, audio, text) 2 commented but visited no new content 3 visited content and commented 4 did the assessment late and did nothing else that week 5 visited content and did the assessment late 6 did the assessment late, commented, but visited no new content 7 visited content, commented, late assessment 8 assessment early or on time, but nothing else that week 9 visited content and completed assessment early / on time 10 assessment early or on time, commented, but visited no new content 11 visited, posted, completed assessment early / on time
- 10. 10 Typical engagement profiles These profiles apply to an eight-week course ● Samplers visit only briefly [1, 0, 0, 0, 0, 0, 0, 0] – 1 means they visited content ● Strong Starters do first assessment [9, 1, 0, 0, 0, 0, 0, 0] – 9 means they visited content and did assessment on time ● Returners come back in Week 2 [9, 9, 0, 0, 0, 0, 0, 0] ● Mid-way Dropouts [9, 9, 9, 4, 1, 1, 0, 0] – 4 means they submitted assessment late ● Nearly There drop out near the end [11, 11, 9, 11, 9, 9, 0, 0] – 11 means full engagement, 8 means submission on time ● Late Completers finish [5, 5, 5, 5, 5, 9, 9, 9] – 5 means they viewed content and submitted late ● Keen Completers do almost everything [11, 11, 9, 9, 11, 11, 9, 9] OU FL study
- 11. 11 Samplers & Strong starters Samplers (1, 0, 0, 0, 0, 0, 0, 0) ● The largest group in all MOOCs ● Typically accounted for 37% – 39% of learners ● Visited the materials, but only briefly ● Active in a small number of weeks ● 25% – 40% joined after Week 1 ● Very few Samplers posted comments (6% – 15%) ● Almost no Samplers submitted any assessment Strong starters (9, 1, 0, 0, 0, 0, 0, 0) ●All Strong Starters submitted the first assignment ●Engagement dropped off sharply after that ●A little over a third of them posted comments ●Typically posted fewer than four comments OU FL study
- 12. 12 Returners & Mid-way dropouts Returners (9, 9, 0, 0, 0, 0, 0, 0) ● Completed the assessment in the first week ● Completed the assessment in the second week ● Then dropped out ● Over 97% completed those two assessments, although some submittted late ● No Returner explored all course steps ● Average amount of steps visited varied (23% – 47%) Mid-way dropouts (9, 9, 9, 4, 1, 1, 0, 0) ●A much smaller cluster (6% of learners on MOOC1, 7% on MOOC4) ●These learners completed three or four assessments ●They dropped out around halfway through the course ●Mid-way dropouts visited about half the steps on the course ●Just under half posted comments ●Posted just over six comments on average OU FL study
- 13. 13 Nearly There Nearly there (11, 11, 9, 11, 9, 9, 0, 0) ● Another small cluster (5% – 6% of learners) ● Consistently completed assessments ● Dropped out just before the end of the course ● Visited around 80% of the course ● Submitted assignments consistently (>90%) and typically on time until Week 5 ● Activity then declined steeply ● Few completed the final assessment ● None completed the final assessment on time OU FL study
- 14. 14 Late completers & keen completers Late completers (5, 5, 5, 5, 5, 9, 9, 9) ● Submitted the final assessment ● Submitted most other assessments ● However, either submitted late or missed some assessments ● Each week, more than 94% of this cluster submitted their assessments ● More than three-quarters submitted the final assessment on time (78% – 90%) ● Around 40% of them posted comments (76% did so on MOOC3) Keen completers (11, 11, 9, 9, 11, 11, 9, 9) ●Accounted for 7% – 23% of learners ●All Keen Completers submitted all assessments ●More than 80% of these were submitted on time ●Typically, Keen Completers visited more than 90% of course content ●Over two-thirds contributed comments (68% – 73%) ●Mean number of comments varied from 21 to 54 OU FL study
- 15. 15 Cross-university dataset FutureLearn data from four universities Cross-university Name Duration University Discipline Active learners LongMOOC1 8 OU Hard science 5,069 LongMOOC2 7 Edinburgh Hard science 10,136 TalkMOOC3 6 Edinburgh Politics 6,141 ShortMOOC4 3 Birmingham Medical science 6,839 ShortMOOC5 3 Leeds Medical science 4,756
- 16. 16 Values for k used in this study Different values were used for the three study phases Cross-university Name Phase 1 Phase 2 Phase 3 LongMOOC1 7 – – LongMOOC2 7 – – TalkMOOC3 – 7 3 ShortMOOC4 – 7 4 ShortMOOC5 – 7 5 Phase 1: Best-fit value for k aligned with OU study Phase 2: Testing k=7 where this was not the best fit Phase 3: Most suitable value for k in each set of data
- 17. 17 Phase two: k ≠ 7 Why k≠7 in Talk MOOC3 Phase two The absence of assessment in TalkMOOC3 limited its coding profile 1 only visited content (for example, video, audio, text) 2 commented but visited no new content 3 visited content and commented 4 did the assessment late and did nothing else that week 5 visited content and did the assessment late 6 did the assessment late, commented, but visited no new content 7 visited content, commented, late assessment 8 assessment early or on time, but nothing else that week 9 visited content and completed assessment early / on time 10 assessment early or on time, commented, but visited no new content 11 visited, posted, completed assessment early / on time
- 18. 18 Phase two: k ≠ 7 Why k≠7 in ShortMOOC4 and ShortMOOC5 Phase two Three clusters are indistinguishable in a three-week MOOC Returners who come back in Week 2 Mid-way Dropouts who drop out mid-course Nearly There who drop out near the end With only three opportunities for late submission, there are no Late Completers (who typically submit assessment late five times) The three-week course design means other clusters emerge, such as: Surgers concentrate their effort after the first week of a three-week course Improvers fall behind in Week 1, begin to catch up in Week 2 and complete on time
- 19. 19 Phase three: suitable values for k TalkMOOC3: k=3 Phase three Quiet (1, 0, 0, 0, 0, 0) ● The largest cluster ● Visit a quarter of course steps ● Do not comment in first week ● Only 7% comment at all ● Only 9% engage with second half of course Contributors (3, 1, 1, 0, 0, 0) ● 19% of cohort ● Visit 38% of course steps ● Every cluster member posts in first week of course ● Half do not comment again Consistent engagers (3, 3, 3, 3, 1, 1) ● 11% of cohort ● Visit 82% of course steps ● Engage throughout course ● Every cluster member posts a comment ● 95% contribute more than three comments ● 7% contribute more than 100 comments
- 20. 20 Phase three: suitable values for k ShortMOOC4: k=4 Phase three Very weak starters (2, 1, 0) ● The largest cluster ● Visit 20% of steps ● 20% do not engage in first week Strong starters (truncated) (10, 1, 0) ● 17% of cohort ● Submit week 1 assessment ● Do not submit another assessment ● Almost half post comment Returners (truncated) (3, 3, 3, 3, 1, 1) ● Most submit week 1 assessment ● All submit week 2 assessment ● Half submit at least one comment Keen completers (truncated) (9, 9, 9) ● Visit more than 90% of steps ● Submit work on time ● Engage throughout
- 21. 21 Phase three: suitable values for k ShortMOOC5: k=5 Phase three Samplers (truncated) (1, 0, 0) ● Visit few steps ● Includes many latecomers (>25%) ● Very few submit assessment Strong starters (truncated) (9, 1, 0) ● Submit week 1 assessment ● Do not submit another assessment Returners (truncated) (8, 8, 2) ● Most submit week 1 assessment ● All submit week 2 assessment Keen completers (truncated) (9, 9, 9) ● Visit more than 90% of steps ● Submit work on time ● Engage throughout Improvers (5, 6, 9) ● Activity increases each week ● Final assessment submitte on time
- 22. 22 Learning design and pedagogy ● The Coursera Study suggested that MOOC designers would be able to apply the four engagement patterns they had identified ‘to target interventions and develop adaptive course features’ ● These subsequent studies show that this is not necessarily the case –engagement patterns are not consistent across MOOCs ● Changes to the basic pedagogic elements of a course are associated with shifts in patterns of engagement. ● Shifts in pedagogic approach can change the elements of a course that can be regarded as key ● Changes to some elements of learning design can change learners’ patterns of engagement with a MOOC
- 23. 23 Shorter courses ● Reducing course length does not necessarily increase engagement ● Many learners do not approach a three-week course in the same way as an eight-week course ● Many focus their attention on later weeks and may miss out the content and activities in the first week ● A three-week course offers limited opportunities to get ahead of, or behind, the cohort ● It is possible to dip in at different points without losing the sense of being a cohort member
- 24. 24 Improving learning and learning environments Closing the loop ● Previews of course material would allow Samplers to make a more informed decision about whether to join the course ● Sign-up pages could draw attention to the problems experienced by those who are out of step with the cohort ● Discussion steps for latecomers could support those who fall behind at the start ● Prompts might encourage flagging learners to return and register for a subsequent presentation ● Bridges between course weeks could indicate links and point learners forward
- 25. 25 View these slides at www.slideshare.net/R3beccaF Rebecca Ferguson @R3beccaF Doug Clow @dougclow