Pythia Reloaded: An Intelligent Unit Testing-Based Code Grader for Education
0 likes561 views
Pythia Reloaded is an open-source unit testing framework designed for automatic code assessment in educational contexts, providing intelligent feedback to learners. It operates in three layers: system, task, and problem, facilitating safe execution and evaluation of learners' code against various tests. The platform aims to improve feedback quality and enhance the learning experience by offering detailed insights on code performance and common errors.
![Pythia Reloaded: An Intelligent
Unit Testing-Based Code Grader for Education
Dr Sébastien Combéfis 1,2 Alexis Paques 2
1École Centrale des Arts et Métiers (ECAM)
2Computer Science and IT in Education (CSITEd)
July 14, 2015
[CHESE 2015, Baltimore, MD, USA]](https://image.slidesharecdn.com/combefis-chese2015-slides-150714204211-lva1-app6891/85/Pythia-Reloaded-An-Intelligent-Unit-Testing-Based-Code-Grader-for-Education-1-320.jpg)
![Configuration-based definition
{
"q1": {
"argc": 2,
"predefined": {
"argv": [{
"data": "(10, 5)",
"feedback": {
"10": "Have you summed the 2nd parameter?",
"5": "Have you summed the 1st parameter?"
}
}, {
"data": "(7, 15)"
}, {
"data": "(-1, 2)",
"feedback": {
"*": "Have considered negative parameter?",
}
}, {
"data": "(12, 0)"
}]
},
"random": {
"n": 10,
"args": ["int(-20,20)", "int(-20,20)"]
},
"code": "def sum(a, b):n return a + b"
}
} 13](https://image.slidesharecdn.com/combefis-chese2015-slides-150714204211-lva1-app6891/85/Pythia-Reloaded-An-Intelligent-Unit-Testing-Based-Code-Grader-for-Education-13-320.jpg)
Pythia Reloaded: An Intelligent Unit Testing-Based Code Grader for Education
- 1. Pythia Reloaded: An Intelligent Unit Testing-Based Code Grader for Education Dr Sébastien Combéfis 1,2 Alexis Paques 2 1École Centrale des Arts et Métiers (ECAM) 2Computer Science and IT in Education (CSITEd) July 14, 2015 [CHESE 2015, Baltimore, MD, USA]
- 2. Ce(tte) œuvre est mise à disposition selon les termes de la Licence Creative Commons Attribution – Pas d’Utilisation Commerciale – Pas de Modification 4.0 International.
- 3. Context Automatic assessment of codes Programming learning platforms, MOOCs, higher education courses, competitions... Two major kinds of code assessment Unit testing frameworks Competition graders Platform for automatic assessment and “intelligent” feedback Suited for education and that helps learners 3
- 4. Pythia framework Released as an open-source project http://www.pythia-project.org A unit testing framework... Functional tests on learners’ code ...and a competition grader Sandboxed environment with execution constraints Prototyped in 2012, used for a MOOC in Spring 2014 4
- 5. Architecture The Pythia framework consists in three layers System layer Safe execution of jobs Task layer Feedbacks Problem layer Context and input/output structure Problem layer Task layer System layer 5
- 6. System layer Front-ends communicate with the outside world Pools manage the safe execution of jobs Front-end Front-end Front-end Queue Pool Pool 6
- 7. Job execution Sandbox uses UML with trimmed-down version of ArchLinux The VM boots in under one second A job has an environment and a task filesystem Constraints can be added to the VM (time, memory and output) { "environment": "python", "taskfs": "hello-world-python.sfs", "limits": { "time": 60, "memory": 32, "disk": 50, "output": 1024 } } 7
- 8. Problem example A problem combines several tasks and adds a context 8
- 9. Pythia LMS 9
- 10. Unit Testing-Based Grading Different kinds of tasks are possible Examine stdout, evaluate time complexity, run unit tests... Unit testing is for developers Feedback is related to the specification of the function Feedback for learners is related to the problem to be solved 10
- 11. Main executable I 1 Preprocess Integrates code snippets from learners into skeletons 2 Compile Analyses statically the code and compiles it 3 Generate Generates tests sets and saves them to a file 11
- 12. Main executable II 4 Execute Executes learner’s code against tests sets, generates data 5 Postprocess Analyses generated data, and produces analysis results 6 Feedback Generates feedbacks 12
- 13. Configuration-based definition { "q1": { "argc": 2, "predefined": { "argv": [{ "data": "(10, 5)", "feedback": { "10": "Have you summed the 2nd parameter?", "5": "Have you summed the 1st parameter?" } }, { "data": "(7, 15)" }, { "data": "(-1, 2)", "feedback": { "*": "Have considered negative parameter?", } }, { "data": "(12, 0)" }] }, "random": { "n": 10, "args": ["int(-20,20)", "int(-20,20)"] }, "code": "def sum(a, b):n return a + b" } } 13
- 14. Importance of feedback Predefined tests must cover errors often made by learners Feedback should be related to the problem being solved “Your code failed for the input a = 10, b = 5. The expected result is 15 and your code produced 10. Have you summed the 2nd parameter?” 14
- 15. Visual feedback Visual feedback to be interpreted by the learner 15
- 16. Conclusion Pythia is an open-source platform containing a unit testing-based grader specifically designed for education Ongoing work Definition and specification of new kinds of tasks Development of Pythia Studio and Pythia LMS Future work Explore use of other kinds of code checks Mining code to identify common errors to extract feedback 16