Wsu guest lecture_2015_czlv_v3
Download as PPTX, PDF1 like83 views
The document discusses surveys conducted to evaluate learners' experiences in MOOCs offered by UNSW Australia. It finds that surveys embedded within course activities, like pre-course surveys, obtain higher response rates than stand-alone surveys. In-video surveys are more effective than those outside videos at capturing engagement. Across four MOOCs, common motivations were personal growth, topic interest, and job relevance. Course B learners primarily aimed to develop teaching strategies. Engagement varied across activities, with videos and forums most used. Paying learners reported higher perceptions of video usefulness than non-paying learners.
Wsu guest lecture_2015_czlv_v3
- 1. Evaluating the ‘Student’ Experience in Massive Open Online Courses (MOOCs) Catherine Zhao & Lorenzo Vigentini Learning and Teaching Unit, UNSW Australia
- 2. In this presentation - • Background – Previous work on Student learning Experience & UNSW MOOCs • Methods - the surveys for evaluation • Findings - o Surveys - fit-for-purpose? o Learners – what’s it like to experience a MOOC? • Current work & Challenges
- 3. A Model of the Student Learning Experience Worth of Experience Future Past Motivation Experience Perception Engagement Aspirations Expectations Personal development ‘match’ or ‘fit’ with context/LE Direct future benefits Vigentini, L., and Zhao, C. Revisiting end-of-semester evaluation: the questions, the instrument and the process.,” in AAIR : turning silver into gold, Melbourne, 2014.
- 4. Background – Why UNSW joined the ‘bandwagon’
- 5. Background- What it takes to design and deliver a MOOC
- 6. Five known facts about MOOCs Massive enrollments Dramatic dropouts Unconventional learners (Relatively) limited design/delivery options Engagement and the learner experience The rise of ‘certificates’
- 8. Background - Four MOOCs we look at A B C D Disciplines Engineering Education Science Medicine Duration (weeks) 9 6 9 8 Target Group Engineers Teachers High school students/ teachers General public Course Design All-at-once All-at-once Sequential Semi-sequential Course Delivery All-at-once Staggered Staggered Staggered Videos 110 224 98 58 Quizzes 10 22 42 2 Assignments 7 3 2 2 Forums 54 105 63 64 Registrants (Active learners*) 32928 (59.96%) 28864 (62.59%) 22761 (46.67%) 13185 (35.44%) *Active learner: those who appear at least once during the course MOOCs – A = engineering, B = education, C = physics, D = medicine
- 9. Background- What & When Surveys Evaluation Surveys Deployment A B C D Demographic survey Prior to course commencement Pre-course survey OR First course activity In-video surveys (IVS) During the course Quick evaluation (Video) Quick evaluation (Module) Quick Question survey Post-course survey End of course Present in MOOC MOOCs – A = engineering, B = education, C = physics, D = medicine
- 10. Methods - Overview of the Pre-Course survey questions Common A B C D Course-specific A B C D Reasons/Purpose of Joining Background-Gender, Ethnicity Prior Professional Experience/Knowledge English Proficiency Intended Effort Prior MOOC Experience Familiarisation with Topics Geographic Location Academic Qualification Contributing Elements of ‘Teaching Online’ Industry of Employment Defining ‘Completion' Preventing Factors for Course Completion Learning Preference Expected Outcomes (course related) Achievement Goal Oriented Present in the course MOOCs – A = engineering, B = education, C = physics, D = medicine
- 11. Common A B C D Course-specific A B C D Course Satisfaction Proportion of Completed Course Activities Course Features Evaluating Peer Assessment Actual Effort (Probing) Self-Regulated Learning Desired Topics Satisfaction on Personal Development (Intend to) Apply Course Topics Considering Credit-Bearing Courses Methods- Overview of the Post-course survey questions MOOCs – A = engineering, B = education, C = physics, D = medicine
- 12. Methods– Pre- & Post-Course Survey Sample Sizes A B C D # Active Learners 19708 18024 10576 4673 Required sample size* 2141 2119 1957 1587 Pre- & Post- Course survey Pre-course survey sample 2684 4490 264 401 response rate 13.62% 24.91% 2.50% 8.58% Margin of error 1.76% 1.28% 5.96% 4.68% Post-course survey sample 638 313 126 71 response rate 3.24% 1.74% 1.19% 1.52% Margin of error 3.82% 5.49% 8.68% 11.54% Coursera Demographic Survey # of registrants 32928 28864 22487 13185 Demographic Survey Responses 2774 2873 2720 1466 Margin of error 1.20% 2.20% 1.52% 2.41% *Required sample size is calculated under the condition of CL = 95%, and margin of error = 2%
- 13. Methods – Patterns of participation in Pre-& Post- Course surveys Counts of responses in the four MOOCs’ pre-course surveys (logarithmic scale) over 9 weeks. 1 10 100 1000 10000 W1 W2 W3 W4 W5 W6 W7 W8 W9 Countofresponses A Pre Post 1 10 100 1000 10000 W1 W2 W3 W4 W5 W6 W7 W8 W9 B Pre Post 1 10 100 1000 10000 W1 W2 W3 W4 W5 W6 W7 W8 W9 Countofresponses C Pre Post 1 10 100 1000 10000 W1 W2 W3 W4 W5 W6 W7 W8 W9 D Pre Post
- 14. Methods- Collection tools Course A IVS (In-video survey) Course C Quick Evaluation MOOCs – A = engineering, B = education, C = physics, D = medicine
- 15. Research Questions 1) Are the tools used robust enough to characterize the learners’ experience in MOOC? 2) Are there substantial differences in the modes of gathering feedback? 3) Is it possible to identify key elements of the learners’ MOOC experiences? 4) What is the relevance of ‘teaching’ questions for the online experience?
- 16. Results – Are the surveys robust? • Accuracy • A B C D Required sample size (CI=95%,margin of error 2%) 2141 2119 1957 1587 Pre-course survey sample 2684 4490 264 401 Margin of error 1.76% 1.28% 5.96% 4.68% Post-course survey sample 638 313 126 71 Margin of error 3.82% 5.49% 8.68% 11.54%
- 17. Results – Are the surveys robust? (Cont’d) A B C D Motivation Block 0.78 0.86 0.65 Course experience Block 0.90 0.94 0.89 0.92 IVS items 0.82 Confidence 0.84 Satisfaction of personal gain 0.91 Satisfaction of course materials and personal gain 0.92 (probing) self-regulated learning skills 0.90 • Reliability – Cronbach’s alpha MOOCs – A = engineering, B = education, C = physics, D = medicine
- 18. Results – Are the surveys robust? (Cont’d) • For the motivation block that is <.7 (D) Reason items Cronbach's Alpha if Item Deleted I am taking the course out of general interest, curiosity, or enjoyment. .722 I am interested in taking a course from this particular institution. .616 I am interested in learning from the expert researchers and clinicians delivering the course. .636 I want to connect with other students interested in this topic. .589 The course supports my current academic program. .565 The course is aligned with my current job responsibilities or company's line-of-business. .588 The skills gained from this course may be useful for obtaining a new job. .571 MOOCs – A = engineering, B = education, C = physics, D = medicine
- 19. Results – Comparison of Evaluation Approaches • The reasons for adopting different approaches 1. Technical constraints 2. Accommodating various requirements put forward by academic leads 3. Test different tools/ways to evaluate • Two key sets of approaches we are comparing - 1. First course activity in A & B vs. Pre-course survey in C & D 2. In-Video Surveys (IVS) vs. Quick Evaluation (QE) MOOCs – A = engineering, B = education, C = physics, D = medicine
- 20. Results – Effectiveness of Pre-course survey • ‘Embedded’ (= first course activity) is more effective than ‘stand-alone’ • Higher response rate • measures how much learners want to engage A B C D 13.62% 24.91% 2.50% 8.58% MOOCs – A = engineering, B = education, C = physics, D = medicine
- 21. 1 10 100 1000 10000 100000 W1 W2 W3 W4 W5 W6 W7 W8 W9 A - IVS INTSE Views of lecture videos INTSE # Participation in IVQs 1 10 100 1000 10000 100000 W1 W2 W3 W4 W5 W6 W7 W8 W9 C - Quick Evaluation P2P Views of lecture videos P2P # Participation in Quick Evaluation Results – In-Video-Surveys vs. Quick Evaluation Inside the video is more effective than outside the video. MOOCs – A = engineering, B = education, C = physics, D = medicine
- 22. Our learners are… A B C D Ave. Age (SD) 34.37 (11.37) 39.91 (12.39) 35.9 (15) 37.35 (13.84) Gender ratio (M/F) 4/1 1/1.2 2.6/1 1/1.2 Full-time Employed 57% 63% 59% 59% Full-time Studying 41% 30% 39% 38% Bachelors and Above 74% 82% 63% 64% Prior Experience 41.51% 57.20% - 79.25% MOOCs – A = engineering, B = education, C = physics, D = medicine
- 23. Results – motivations I A C D M(SD) M(SD) M(SD) For personal growth and enrichment 3.23 (.76) 2.10(.72) General interest in topic 3.14 (.74) 2.23(.72) 3.23(.92) Relevant to job 2.63 (1.01) 1.6(.76) 2.71(1.38) For fun and challenge 2.58 (.96) 1.42(.63) Earn a certificate/statement of accomplishment 2.42 (1.03) 1.55 (.70) Course offered by prestigious university/professor 2.17 (1.03) 1.91(.77) 3.06(.97) For career change 2.07 (1.06) 1.32(.59) 1.92(1.41) Experience an online course 1.81 (.93) 1.21(.51) Relevant to school or degree program 1.75 (1.00) 1.74(.83) 2.56(1.37) To improve my English skills 1.66 (.95) 1.52(.71) Relevant to academic research 1.65 (.98) 2.37(.86) Meet new people 1.52 (.76) 1.65(.70) 1.96(1.06) Take with colleagues/friends 1.32 (.68) 1.93(.75) 1=Not at all important, 2 = somewhat important, 3 = very important, 4 = extremely important
- 24. Results – Motivations II Reasons Item loading A1 C2 D3 I II III VI I II III VI I II Experience an online course1,2 0.7 0.7 Meet new people1,2,3 0.7 0.8 0.6 To improve my English skills1,2 0.7 0.2 0.6 Take with colleagues/friends1,2 0.7 0.6 Course offered by prestigious University/Professor1,2,3 0.6 0.8 0.8 For fun and challenge1,2 0.8 0.7 Personal growth&enrichment1,2 0.8 0.8 General interest in topic1,2,3 0.7 0.8 0.5 Relevant to academic research1,2 0.9 0.8 Relevant to academic study1,2,3 0.9 0.8 0.8 Relevant to job1,2,3 0.8 0.7 0.8 Earn a certificate1,2 0.6 For career change1,2,3 0.5 0.8 0.7 Items with loading lower than 0.5 are removed. Superscripts 1, 2, 3 represent occurrence in A, C, & D respective. MOOCs – A = engineering, B = education, C = physics, D = medicine
- 25. Results – Course B Specifics Intent & Goals
- 26. Results – Course B Specifics (Cont’d) 2.20% 2.77% 3.72% 3.77% 8.76% 14.17% 30.43% 34.17% I want to develop my knowledge and understanding of a specific concept I want to exchange ideas and learn from colleagues I am interested in looking at the course design I am curious about the topic I want to complete the course and obtain recognition and certification for my work I want to use my knowledge to develop a personal online teaching strategy I want to develop my knowledge and understanding of the overall topic I want to develop an online learning design that I can use in my own teaching (N = 4083) • Overview of Learners’ Intents
- 27. Results – Course B Specifics (Cont’d) • Intent is not sig. correlated to learners’ behaviours • Data Mining (K-NN, Naïve Bayes) using learners’ characteristics (incl. behaviours) indicate 30% predictive accuracy for performance class (no grade, pass, distinction).
- 28. Results – Course B Specifics (cont’d) Activity by intent
- 29. Results – Course B Specifics (Cont’d) Week 2 Week 4 Week 7 Assessment scores vs. Intents
- 30. 0 10000 20000 30000 40000 50000 60000 Announcement Times Site Access Video Views & Downloads Announcements superimposed on User Video and Site Access Activity Results – Course B Specifics (Cont’d)
- 31. Results – The same in Course D
- 32. Results – Engagement with Forum A B C D MOOCs – A = engineering, B = education, C = physics, D = medicine
- 34. Results – Engagement with Quizzes A B C D MOOCs – A = engineering, B = education, C = physics, D = medicine
- 35. Results – Engagement with Videos A B C D MOOCs – A = engineering, B = education, C = physics, D = medicine
- 36. Results – Engagement III Comparing paying & non- paying learners F t df Sig. 2-tails Mean Diff’ CI=95% Interval of diff' Lower Upper Q1 I find this lecture useful. 33.66 -4.9 9092 <0.05 -0.11 -0.16 -0.07 Q2 I understand the content of this lecture. 24.81 -3.89 9045 <0.05 -0.09 -0.13 -0.04 Q3 I’d like to explore other modules of this course. 43.62 -5.16 8896 <0.05 -0.11 -0.15 -0.07 Results of F-test suggest unequal variance hence the t-test results assumed unequal variance are used. (Using Course A as an example – in-video survey items) MOOCs – A = engineering, B = education, C = physics, D = medicine
- 37. Results – Learners’ Course Experience Course experience items (mapped on new CATEI) A B C I II I II I II Overall, I am satisfied with the quality of this course. 0.9 0.7 0.6 The material of the course was presented in an engaging manner. 0.9 0.8 0.8 Overall, this course met my expectations. 0.9 0.8 0.6 Examples, illustrations or real-world cases were used effectively to explain things. 0.9 0.8 0.7 The course encouraged my interest in this field of study. 0.8 0.8 The goals and requirements of the course were made clear to me. 0.8 0.8 0.7 Overall I have improved the knowledge/skills I will need. 0.8 0.8 0.8 Quizzes helped me to evaluate my progress effectively. 0.6 0.6 0.8 Interacting in the forums helped me to clarify things I did not understand. 0.9 0.9 0.6 Items with loading lower than 0.5 are removed. MOOCs – A = engineering, B = education, C = physics, D = medicine
- 38. Course experience Items D I II Depth of content 0.8 Course materials 0.8 Lecture videos 0.8 Clarity of explanations 0.8 Course content 0.8 Knowledgeability 0.8 Instructor/teaching staff 0.8 Overall satisfaction 0.7 In-video quizzes 0.7 Presentation skills 0.5 Assessments 0.9 Discussion forums 0.6 Extraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser Normalization. a. Rotation converged in 3 iterations. Results – Learners’ Course Experience (Cont’d) MOOCs – A = engineering, B = education, C = physics, D = medicine
- 39. Results – the Worth of Course Experience R2 F Sig. A 0.17 1.22 0.26 C 0.79 0.72 0.72 D 0.39 1.08 0.38 • Motivation cannot predict satisfaction LinearRegression(motivations) -> ‘How satisfied am I’ MOOCs – A = engineering, B = education, C = physics, D = medicine
- 40. Aspects (Question Block) A coefficient Sig' Course Experience Overall, this course met my expectations 0.54 0.00 The material of the course was presented in an engaging manner 0.19 0.00 The course encouraged my interest in its field of study 0.11 0.00 Examples, illustrations or real-world cases were used effectively to explain things 0.08 0.01 The goals and requirements of the course were made clear to me 0.07 0.02 Aspects (Question Block B coefficient Sig' Course Experience Overall this course met my expectations 0.57 0.00 Satisfaction Provided an opportunity for fun and challenge 0.10 0.03 What can predict satisfaction ‘How satisfied am I’ <= LinearRegression(multiple items) Results – the Worth of Course Experience (Cont’d)
- 41. Results – the Worth of Course Experience (Cont’d) Aspects (Question Block) C coefficient Sig' Overall I have improved the knowledge/skills I will need 0.63 0.00 Course Experience The course encouraged my interest in this field of study 0.61 0.01 The material of the course was presented in an engaging manner 0.32 0.02 The course encouraged my interest in this field of study 0.36 0.02 The experimental tasks were useful to apply the theory 0.30 0.02 LinearRegression(multiple items) ->‘How satisfied am I’ Aspects (Question Block) D coefficient Sig' Self-regulation I am confident that I could deal efficiently with unexpected events. 0.87 0.00 I can remain calm when facing difficulties because I can rely on my coping abilities. 0.62 0.01 I can solve most problems if I invest the necessary effort. 0.35 0.02 I can usually handle whatever comes my way 0.32 0.05 Course Experience Discussion forums 0.42 0.01 In-video quizzes 0.45 0.03 Depth of content 0.67 0.03 I found the material interesting. 0.31 0.03 Motivation I am taking the course out of general interest, curiosity, or enjoyment. 0.28 0.03
- 42. Conclusions 1. HBR paper*: two types of learners – ‘career builders’ and ‘education seekers’. 2. Motivation weakly related to engagement or satisfaction 3. Some relations between behaviors (engagement) and performance 4. ‘Embedded’ evaluation tools are more effective 5. Mapping of ‘online’ learning experience * Zhenghao, C., Alcorn, B., Christensen, G., Eriksson, N., Koller, D., & Emanuel, E. J. (2015). Who’s Benefiting from MOOCs, and Why. Retrieved October 29, 2015, from https://hbr.org/2015/09/whos-benefiting-from- moocs-and-why
- 43. Challenges What are the characteristics of MOOCs that scaffolds learning? The intricacy of the combination of the factors which contribute to the worth of course experience. The matter of Agency.
Editor's Notes
- #3: Student experience is too broad, includes campus life and you may get questions Services are supporting it, but focus is on learning experience Be prepared
- #7: Bridging slide for everyone
- #8: Give them context Tell them what A, B, C.. Explain multiple iterations 3x intse 2x ltto 2x p2p 1x Pmed HL next in line
- #16: CATEI Is eval of teaching for management, so we test some questions
- #17: Say its relatively small
- #21: Call it how keen they are to do stuff in the mooc instead. NEED to run an ANOVA on N responses Factors/levels: Credit (y/n) x course (4) x timing (1st week/later)
- #28: Correlated, NOT related suggesting low/weak association Separate things! Otherwise you have no idea what it is
- #29: Question, what’s the best way of presenting/delivering content? Left is bigger pic of activity for the course: patterns of access for the course as a whole: distinct access people added up for class % of activity per row timeline for each module with spikes each week (traffic) another representation including everything for a module, then you break it down BOTTOM is goal setting activity done all at the start Right is splitting by intents activity by intents over time (they are the same)
- #30: Need to explain a bit Learners more likely to participate at the beginning regardless of intents Too many dimensions
- #31: Example of how ‘teaching’ drives activity Whenever you have an announcement spikes of activity follow (LTTO)
- #33: Breaking down by activity type over multiple courses.
- #38: Very dangerous No performance difference in free Scrap this, otherwise you need the figure for 4 courses We will edit later… weeks
- #40: This is important, highlight mapping to the new CATEI
- #41: Where do you say that it is course B? explain
- #42: Deceiving Motivation predicts satisfaction not the other way Why highlight NS? What is C? large R not sure about result
- #43: Same here These are sig, but no highlight Relatively large
- #44: Same swap of vars large
- #45: Harvard business review paper Is it supported? Then explain 2 explain more 3. Give more examples