CHI (Computer Human Interaction) 2019 enhancing online problems through instructor centered tools for randomized experiments
Download as PPTX, PDF1 like509 views
The document discusses the development and implementation of an instructor-centered tool called DynamicProblem for conducting A/B experiments in educational settings. It aims to ease the programming burden on instructors, allowing them to collect data on instructional approaches while providing elaborate messages to aid student learning. Insights from deploying the tool with faculty members highlighted its potential to improve student experiences and foster a practical research environment.
![Enhancing Online Problems Through Instructor-
Centered Tools for Randomized Experiments
Joseph Jay Williams
University of Toronto Computer Science ( Nat. U. of Singapore)
www.josephjaywilliams.com/papers, tiny.cc/icepdf
Anna Rafferty, Andrew Ang, Dustin Tingley, Walter Lasecki, Juho Kim
[I’m originally from the Caribbean,
Trinidad and Tobago]
Postdoc at U of T (www.josephjaywilliams.com/postdoc)
Computer Science PhD positions to do Education Research
CHI 2019 subcommittee on “Learning/Families” (Amy Ogan &
I are SCs)](https://image.slidesharecdn.com/chi2019enhancingonlineproblemsthroughinstructor-centeredtoolsforrandomizedexperiments-180424161903/85/CHI-Computer-Human-Interaction-2019-enhancing-online-problems-through-instructor-centered-tools-for-randomized-experiments-1-320.jpg)
![Observations from Deployment with 3 Instructors
• Lowered Barriers: “not aware of any tools that do this
sort of thing”, “even if I found one, wouldn’t have the
technical expertise to incorporate it in my course”
• Reflection on pedagogy: “I never really seriously
considered [testing] multiple versions as we are now
doing. So even if we don't get any significant data, that
will have been a benefit in my mind”
• Making research practical: “a valuable tool. Putting in
the hands of the teacher to understand how their
students learn. Not just in broad terms, but specifically
in their course”….](https://image.slidesharecdn.com/chi2019enhancingonlineproblemsthroughinstructor-centeredtoolsforrandomizedexperiments-180424161903/85/CHI-Computer-Human-Interaction-2019-enhancing-online-problems-through-instructor-centered-tools-for-randomized-experiments-13-320.jpg)
CHI (Computer Human Interaction) 2019 enhancing online problems through instructor centered tools for randomized experiments
- 1. Enhancing Online Problems Through Instructor- Centered Tools for Randomized Experiments Joseph Jay Williams University of Toronto Computer Science ( Nat. U. of Singapore) www.josephjaywilliams.com/papers, tiny.cc/icepdf Anna Rafferty, Andrew Ang, Dustin Tingley, Walter Lasecki, Juho Kim [I’m originally from the Caribbean, Trinidad and Tobago] Postdoc at U of T (www.josephjaywilliams.com/postdoc) Computer Science PhD positions to do Education Research CHI 2019 subcommittee on “Learning/Families” (Amy Ogan & I are SCs)
- 2. How Can We Help Instructors Conduct A/B Experiments? • Opportunity: Collect data about alternative instructional approaches, instead of relying on intuition • Obstacle: Time and effort to program experiments • Elaboration Messages in online problems x = matrix(rnorm(m*n),m,n) What is the standard error? Answer: A z-score is defined as the number of standard deviations a specific point is away from the mean. Elaboration Messages: Explanations, Hints, Learning Tips
- 3. Related Work • Technology for A/B Experimentation (Optimizely, edX, ASSISTments) (Heffernan & Heffernan, 2014) • Involving instructors in research (Barab & Squire, 2014) • Elaboration messages in online problems (Shute, 2008)
- 4. Overview • Design goals for instructor-centered experimentation • DynamicProblem, an end-user tool (on-campus courses & MOOCs) • Reinforcement learning for dynamic experimentation • Insights from deployment with 3 instructors
- 5. Goals for Instructor Centered Experimentation • 1. Deploy experiments and obtain data with minimal programming – Provide end-user plug-in, DynamicProblem • 2. Use data for practical improvement – Use reinforcement learning to automatically give more effective conditions to future students
- 6. DynamicProblem Plug-In for Courses • Embed into any Learning Management System (e.g. Canvas) or MOOC, via Learning Tools Interoperability Standard
- 7. Student View of DynamicProblem Linda is training for a marathon, which is a race that is 26 miles long. Her average training time for the 26 miles is 208 minutes, but the day of the marathon she was x minutes faster than her average time. What was Linda's running speed for the marathon in miles per minute? Elaboration Message Linda's speed is the distance she ran divided by the time it took. The distance Linda ran was 26 miles. The time it took her was 208 – x. Linda's speed was 26/(208 - x) 26/(208 - x) How helpful was the above information for your learning? Completely Perfectly Unhelpful Helpful 0 1 2 3 4 5 6 7 8 9 10 A B ACM Learning @ Scale 2016
- 8. Instructor View of DynamicProblem
- 12. Data Dashboard (Instructor 3) Learning Tip Probability of Message
- 13. Observations from Deployment with 3 Instructors • Lowered Barriers: “not aware of any tools that do this sort of thing”, “even if I found one, wouldn’t have the technical expertise to incorporate it in my course” • Reflection on pedagogy: “I never really seriously considered [testing] multiple versions as we are now doing. So even if we don't get any significant data, that will have been a benefit in my mind” • Making research practical: “a valuable tool. Putting in the hands of the teacher to understand how their students learn. Not just in broad terms, but specifically in their course”….
- 14. 2. Use Data For Practical Improvement • Instructor concerns: – Experiments advance researchers’ goals, but do not directly help their students – Ethics of giving students unhelpful conditions • Approach of Dynamic Experimentation: – Analyze data in real-time – Give higher-rated messages to future students
- 15. Model Action a Dynamic Experimentation: Exploration vs Exploitation • Multi-Armed Bandit (Reinforcement Learning) A Reward R Policy Elaboration Message A The probability is 3/7 * 5/8, because the number of cookies is changing. Rating How helpful was the above information for your learning? 0 1 2 3 4 5 6 7 8 9 10 A B 70% 30% (Probability of Message being Helpful) (0 to 10 Rating by Student) Elaboration Message B The number of cookies is changing.. Randomized Probability Matching (Thompson Sampling)
- 16. Dynamically Weighted Randomization • 50/50 probability of assignment • 60/40 • 70/30 • … 100/0 • Probability a student assigned to a message = • Probability of that message being highest rated
- 18. Observations from Deployment with 3 Instructors • Directly helping students: “improved the experience of many of the students by giving them answers that are more helpful… the earlier ones can help improve the experience of the later students. That’s pretty neat”
- 19. Student Perceptions? • Students weren’t surprised by, and appreciated the approach: • “I assume companies are always A/B testing on me” • “now the data I provide can help other people learn”
- 20. Limitations & Future Work • Conduct better experiments – You can use the hosted tool: www.josephjaywilliams.com/dynamicproblem github.com/kunanit/adaptive-mooclet-lti • Go beyond subjective student judgments • Instructor-researcher collaboration • Personalization of Elaboration Messages –MOOClet WebService for dynamic A/B testing (www.josephjaywilliams.com/mooclet)
- 21. Review • Design goals for instructor experimentation • DynamicProblem, an end-user tool (Use it at www.josephjaywilliams.com/dynamicproblem) • Multi-armed bandits for dynamic experimentation • Insights from deployment with 3 instructors • Limitations & Future Work • PhD students can do education at U of T Comp Sci • www.josephjaywilliams.com/postdoc • Learning/Families subcommittee for CHI 2019 (Amy Ogan & I are SCs)
- 22. Thank you! • National University of Singapore Information Systems & Analytics • Harvard VPAL Research Group