2015 EDM Leopard for Adaptive Tutoring Evaluation
1 like539 views
The paper critiques the current evaluation methods for adaptive tutoring systems, arguing that classification accuracy metrics often do not accurately reflect their effectiveness. It proposes a new paradigm called LEOPARD (Learner Effort-Outcomes Paradigm) to assess both the effort expended by the system and the outcomes achieved by students without requiring randomized controlled trials. The authors stress the need for better evaluation metrics that correlate more meaningfully with real learning gains.
![Bleu [Papineni et al ‘01]:
machine translation systems
Rouge [Lin et al ’02]:
automatic summarization systems
Paradise [Walker et al ’99]:
spoken dialogue systems
15](https://image.slidesharecdn.com/2015edmleopard-151027151538-lva1-app6891/85/2015-EDM-Leopard-for-Adaptive-Tutoring-Evaluation-15-320.jpg)
![Other fields verify that automatic metrics
correlated with the target behavior
[Eg.: Callison-Burch et al ’06]
19](https://image.slidesharecdn.com/2015edmleopard-151027151538-lva1-app6891/85/2015-EDM-Leopard-for-Adaptive-Tutoring-Evaluation-19-320.jpg)
![Surprisingly, in spite of all of the evidence
against using classification evaluation
metrics, their use is still very widespread in
the adaptive literature*
Can we do better?
* ExpOppNeed [Lee & Brunskill] is an exception
21](https://image.slidesharecdn.com/2015edmleopard-151027151538-lva1-app6891/85/2015-EDM-Leopard-for-Adaptive-Tutoring-Evaluation-21-320.jpg)
2015 EDM Leopard for Adaptive Tutoring Evaluation
- 1. Your model is predictive but is it useful?− Theoretical and Empirical Considerations of a New Paradigm for Adaptive Tutoring Evaluation José P. González-Brenes, Pearson Yun Huang, University of Pittsburgh 1
- 2. Main point of the paper: We are not evaluating student models correctly New paradigm, Leopard, may help 2
- 3. 3
- 4. Why are we here? 4
- 5. 5
- 6. Educational Data Mining Data Mining = ? 6
- 7. Should we just publish at KDD*? *or other data mining venue 7
- 8. Claim: Educational Data Mining helps learners 8
- 9. 9 Is our research helping learners?
- 10. Adaptive Intelligent Tutoring Systems: Systems that teach and adapt content to humans teach collect evidence 10
- 11. Paper writing: Researchers quantify the improvements of the systems compared Not a purely academic pursuit: Superintendents to choose between alternative technology Teachers choose between systems 11
- 12. Randomized Controlled Trials may measure the time students spent on tutoring, and their performance on post-tests 12
- 13. Difficulties of Randomized Controlled Trials: • IRB approval • experimental design by an expert • recruiting (and often payment!) of enough participants to achieve statistical power • data analysis 13
- 14. How do other A.I. disciplines do it? 14
- 15. Bleu [Papineni et al ‘01]: machine translation systems Rouge [Lin et al ’02]: automatic summarization systems Paradise [Walker et al ’99]: spoken dialogue systems 15
- 16. Using automatic metrics can be very positive: • Cheaper experimentation • Faster comparisons • Competitions that accelerate progress 16
- 17. Automatic metrics do not replace RCTs 17
- 18. What does the Educational Data Mining community do? Evaluate the student model using classification accuracy metrics like RMSE, AUC of ROC, accuracy… (Literature reviews by Pardos, Pelánek, … ) 18
- 19. Other fields verify that automatic metrics correlated with the target behavior [Eg.: Callison-Burch et al ’06] 19
- 20. Ironically, we have a growing body of evidence that classification evaluation metrics are a BAD way to evaluate adaptive tutors Read Baker and Beck papers on limitations / problems of these evaluation metrics 20
- 21. Surprisingly, in spite of all of the evidence against using classification evaluation metrics, their use is still very widespread in the adaptive literature* Can we do better? * ExpOppNeed [Lee & Brunskill] is an exception 21
- 22. The rest of this talk: • Leopard Paradigm – Teal – White • Meta-evaluation • Discussion 22
- 23. Leopard 23
- 24. • Leopard: Learner Effort-Outcomes Paradigm • Leopard quantifies the effort and outcomes of students in adaptive tutoring 24
- 25. • Effort: Quantifies how much practice the adaptive tutor gives to students. Eg., number of items assigned to students, amount of time… • Outcome: Quantifies the performance of students after adaptive tutoring 25
- 26. • Measuring effort and outcomes is not novel by itself (e.g, RCT) • Leopard’s contribution is measuring both without a randomized control trial • White and Teal are metrics that operationalize Leopard 26
- 27. White: Whole Intelligent Tutoring system Evaluation White performs a counterfactual simulation (“What Would the Tutor Do?”) to estimate how much practice students receive 27
- 28. 28 Design desiderata: Evaluation metric should be easy to use Same, or similar input than conventional metrics
- 29. 29 Bob Alice skill q 1 student u .5 Bob s1 s1 Bob Alice s1 .7 Bob s13 .7 s1 .8 6 1 yt 0 4 1 .6 1 1 Bob 2 3 4 Bob .7 2 .9 1Bob .5 t 1 Alice s1 1 0 s1 s1 s1 0 s1 s1 1 ct+1 .6 Alice 0 .4 .9 ˆ actual performance predicted performance cu,q,t+1 yu,q,tˆ
- 30. 30 Bob Alice skill q 1 student u .5 Bob s1 s1 Bob Alice s1 .7 Bob s13 .7 s1 .8 6 1 yt 0 4 1 .6 1 1 Bob 2 3 4 Bob .7 2 .9 1Bob .5 t 1 Alice s1 1 0 s1 s1 s1 0 s1 s1 1 ct+1 .6 Alice 0 .4 .9 2/3 4/5 score score effort= effort= ˆ actual performance predicted performance cu,q,t+1 yu,q,tˆ Alternatively, we can model error
- 31. Future direction: Present aggregate results 31
- 32. Q/ What if student does not achieve target performance? A: “Visible” imputation 32
- 34. Compare: • Conventional classification metrics • Leopard metrics (White) 34
- 35. Datasets: • Data from a middle-school Math commercial tutor – 1.2 million observations – 25,000 students – Item to skill mapping: • Coarse: 27 skills • Fine: 90 skills • (Other item-to-skill model not reported) • Synthetic data 35
- 36. Assessing an evaluation metric with real student data is difficult because we often do not know the ground truth Insight: Use data that we know a priori its behavior in an adaptive tutor 36
- 37. For adaptive tutoring to be able to optimize when to stop instruction, the student performance should increase with repeated practice (the learning curve should be increasing) Decreasing /flat learning curve = bad data 37
- 38. 38
- 39. 39
- 40. Procedure: 1. Select skills with decreasing/flat learning curve (aka bad data) 2. Train a student model on those skills 3. Compare classification metrics with Leopard 40
- 41. F1 AUC Score Effort Bad student model .79 .85 Majority class 0 .50 41
- 42. F1 AUC Score Effort Bad student model .79 .85 .18 10.1 Majority class 0 .50 .18 11.2 42
- 43. What does this mean? • High accuracy models may not be useful for adaptive tutoring • We need to change how we report results in adaptive tutoring 43
- 44. Solutions • Report classification accuracy averaged over skills (for models with 1 skill per item) ✖ Not useful for comparing or discovering different skill models • Report as “difficulty” baseline ✖ Experiments suggest that models with baseline performance can be useful • Use Leopard 44
- 45. Let’s use all data, and pick an item-to-skill mapping: AUC Score Effort Coarse (27 skills) .69 Fine (90 skills) .74 45
- 46. Let’s use all data, and pick an item-to-skill mapping: AUC Score Effort Coarse (27 skills) .69 .41 55.7 Fine (90 skills) .74 .36 88.1 46
- 47. With synthetic data we can use Teal as the ground truth We generate 500 synthetic datasets with known Knowledge Tracing Parameters Which metrics correlate best to the truth? 47
- 48. 48
- 49. 49
- 50. Discussion 50
- 51. 51 In EDM 2014 we proposed “FAST” toolkit for Knowledge Tracing with Features
- 52. “FAST model improves 25% AUC of ROC”
- 54. 54
- 55. Input Teal White Knowledge Tracing Family parameters Student’s correct or incorrect response Sequence length Student models’ prediction of correct/ incorrect Target Probability of correct Target Probability of correct 55