Automatic Question Generation for Evidence-based Online Courseware Engineering
- 1. INNOVATIVE EDUCATIONAL COMPUTING LAB Pedagogical Question Generation towards Automatic Online Courseware Engineering Machi Shimmei, Noboru Matsuda Dept of Computer Science North Carolina State University
- 2. INNOVATIVE EDUCATIONAL COMPUTING LAB Introduction • Learning by doing on MOOC – Answering questions facilitates learning when combined with watching videos • Test-enhanced learning – Answering questions (any kind) facilitate learning • Spaced repetition – Repetitive exposure to answering questions strengthen learning 2 7/27/22
- 3. INNOVATIVE EDUCATIONAL COMPUTING LAB Gap and Need • Creating questions is challenging • Creating effective questions is even more challenging • A pragmatic technology for automated question generation is desired 3 7/27/22
- 4. INNOVATIVE EDUCATIONAL COMPUTING LAB Solution: QUADL • QUestion generation with an Application of Deep Learning • Goal – Given a didactic text and a learning objective, generate a question that is suitable to achieve the learning objective. • Type of question: Verbatim question – The answer can be found in the didactic text 4 7/27/22
- 5. INNOVATIVE EDUCATIONAL COMPUTING LAB QUADL: Example • Learning objective (LO): Describe metabolic pathways as stepwise chemical transformations either requiring or releasing energy; and recognize conserved themes in these pathways. • Sentence (S): Among the main pathways of the cell are photosynthesis and cellular respiration, although there are a variety of alternative pathways such as fermentation. • Question (Q): Along with photosynthesis , what are the main pathways of the cell ? – Answer: cellular respiration 7/27/22 5
- 6. INNOVATIVE EDUCATIONAL COMPUTING LAB QUADL: Architecture 6 Sentence (S) Learning Objective (LO) Answer Prediction Verbatim Question (Q) S <Index_start, Index_end> (S is suitable with a predicted answer) S <0, 0> (S is not suitable for LO) Question Conversion QG-net BERT = Non-Target Sentence = Target sentence 7/27/22
- 7. INNOVATIVE EDUCATIONAL COMPUTING LAB Generating Training Data from OLI • Made exhaustive combination of LO and corresponding S • Instructors tagged answer index for randomly sampled 395: <LO, S<Is, Ie>> – 30% of the pairs are not-target (i.e.,<Is=0, Ie=0>) • 345 pairs for training; 50 pairs for testing 7 7/27/22
- 8. INNOVATIVE EDUCATIONAL COMPUTING LAB QUADL: Preliminary Evaluation • Trained Answer Prediction model on data from Open Learning Initiative • Used pre-trained QG-Net for Question Conversion • Had Amazon Mechanical Turkers (AMT) / instructors forecast the usefulness of generated questions 8 7/27/22
- 9. INNOVATIVE EDUCATIONAL COMPUTING LAB Measure: • Participants judged <LO, S<Is, Ie>, Q> (1) Is it adequate to convert sentence S into a question whose answer is token <Is, Ie> to attain the learning objective LO? (2) Is the question Q suitable for attaining the learning objectives LO? 9 7/27/22
- 10. INNOVATIVE EDUCATIONAL COMPUTING LAB Results: Answer Prediction (by AMT) • Is it adequate (not) to convert S<Is, Ie> into Q? 10 7/27/22
- 11. INNOVATIVE EDUCATIONAL COMPUTING LAB Results: Question Conversion 11 All S with <Is, Ie> output from the Answer Prediction model • Does Q help students attain the learning goal? 7/27/22
- 12. INNOVATIVE EDUCATIONAL COMPUTING LAB Results: Question Conversion 12 All S with <Is, Ie> output from the Answer Prediction model Only S with <Is, Ie> judged as adequate by participants • Does Q help students attain the learning goal? 7/27/22
- 13. INNOVATIVE EDUCATIONAL COMPUTING LAB Examples 7/27/22 13 A participant judged both a target term and a question were adequate : A participant judged a target term was adequate, but the question was not adequate:
- 14. INNOVATIVE EDUCATIONAL COMPUTING LAB QUADL: Future Work • We proposed QUADL for generating questions that are aligned with the given learning objective. As far as we are aware, there have been no studies that aim to generate questions that are suitable for attaining the learning objectives. • Improve model performance – Alternative Question Conversion model – Extremely low data regime • Efficacy study – Measure learning outcome with authentic students @ OLI 14 7/27/22