Boosting individual feedback with AutoFeedback
0 likes239 views
The document presents 'autofeedback,' an automated code feedback system developed for computer science courses at Aston University to enhance student learning and feedback efficiency. It addresses challenges in manual code feedback, such as delays and inconsistencies, and is built using Docker for robust performance and scalability. Future plans include supporting additional programming languages and improving feedback mechanisms for students.
Boosting individual feedback with AutoFeedback
- 1. Boosting individual feedback with AutoFeedback April 29th , 2021
- 2. Who am I? ● Lecturer in Computer Science at Aston University ● I teach ○ Y1 Java Programming ○ Y2 Team Project ● I do research on software engineering (i.e. making better software): ○ Automating software testing ○ Creating better notations to describe what the system should do ● I am one of the Directors at Beautiful Canoe: ○ Not-for-profit social enterprise at Aston (students develop projects here!) ○ https://beautifulcanoe.com/
- 3. Motivation for automated code feedback in CS1OOP ● CS1OOP runs through the whole year: ○ Term 1 - intro to programming with Processing (https://processing.org/) ○ Term 2 - object-oriented programming with Java ○ Labs supported by 1 staff member for each 20-30 students ● Hands-on labs each week: ○ Students need to know if they need to fix their code ○ Lecturers need to know what parts of the lab are perceived as difficult ● Manual code feedback limitations: ○ Manually going through hundreds of submissions each week would delay feedback ○ Late feedback may not be useful! ● Automation helps students self-diagnose and staff give faster feedback: ○ The tool can check your code 24/7, and be 100% consistent ○ You can share the results with staff, to identify issues faster
- 4. Challenges involved ● Submitted code may have bugs - they shouldn't break the server! ● The platform needs to integrate with real tools ○ Professionals use IDEs like Eclipse or IntelliJ to develop their programs ○ Commercial solutions for automated feedback tend to force students to use web-based code editors, which are not as powerful as the above IDEs ● Submission should be easy for first-time programmers ○ Shouldn't have to learn Git just to send your first program ● The platform needs to support multiple feedback loops: ○ Based on feedback, students will refine their code and re-submit ○ When stuck, students will share their results with staff ○ Based on student feedback, lecturers will refine automated checks and lab worksheets ○ The platform is continuously improved based on the student experience
- 5. AutoFeedback - our house-grown solution ● We did not find any commercial option which ticked all these boxes ● We made our own: AutoFeedback ● Piloted in term 2 of CS1OOP this year ● Processed more than 18,000 submissions! AutoFeedback is open source - get the code at https://gitlab.com/autofeedback/
- 6. AutoFeedback - how does a student submit code? 1. Download the starting code from our virtual learning environment (Blackboard) 2. Open the code with your IDE (e.g. Eclipse or IntelliJ) 3. Go through the lab worksheet 4. Right-click on a special file and click on "Submit to AutoFeedback" (two clicks!)
- 7. AutoFeedback - how does a student see feedback? (I) ● The submission process will ZIP your code and send it to AutoFeedback ○ AF queues submissions: if we get many at once, you will just wait a bit longer! ● It also opens a browser tab that will eventually refresh with the results ● Keep working on the feedback and resubmitting! ● Ask staff if something is unclear!
- 8. AutoFeedback - how does a student see feedback? (II) ● Each lab has its own collection of tests, with: ○ Marks for passing the test ○ Feedback if passed ○ Feedback if failed ● Colors: ○ Passed: green ○ Crashed: yellow ○ Failed: green ● AF uses popular testing tools: ○ Maven ○ JUnit 5 ○ Mockito ○ JavaParser ○ TestFX
- 9. AutoFeedback - how is it made? ● AutoFeedback is a system, not just one tool! ● AF has two main parts: ○ The AutoFeedback website - written in PHP using the Laravel web framework ■ Runs 24/7 inside a server in Aston, with automated backups and system upgrades ○ The AutoFeedback Maven plugin - written in Java ■ Runs from the IDE in your computer - submits your code and opens the browser tab
- 10. AutoFeedback - how is the website made? (I) ● Remember the risks around letting buggy code crash our server? ● To protect against that, we use "containerization" with Docker: ○ Each container is effectively running as if they were on a separate computer ○ We can impose CPU/disk/memory limits on containers - each submission gets: ■ Half a CPU core, for up to 10 minutes ■ 100MB of disk space - gets erased automatically after marking ■ 512MB of memory ● We use containers for all the other parts of the AF website, too ○ Containers ensure the website is set up exactly the same everywhere ○ No more "but it works on my machine!" :-)
- 11. AutoFeedback - how is the website made? (II) ● Our Docker containers: ○ nginx is the web server - it receives HTTP(S) connections and serves static files (CSS, JS, images) directly. ○ "app" is the application server - it renders web pages and schedules jobs to process student code into a queue. ○ The "default-worker"s process non-Java background jobs coming from "app" (e.g. marking or importing Blackboard users). ○ The "java-worker"s run student code under CPU/RAM/disk limits. ○ We use MariaDB for storing data, and Redis for storing push notifications and the job queue. ○ Laravel Echo delivers push notifications.
- 12. AutoFeedback - how is the website kept up to date? ● We use the Git version control system to track all our changes to AF ● The AF code is hosted in Gitlab: all changes undergo automated tests ○ This is known as Continuous Integration (CI) ● If the code passes all tests, it is automatically deployed in the server: ○ This is known as Continuous Delivery (CD) ○ CD prevents breaking the server due to a mistake doing some manual upgrade step
- 13. AutoFeedback - what is next? ● Supporting other languages besides Java: ○ Planned redesign around the Kubernetes orchestration platform ○ Could support anything that produces a JUnit XML file (there are testing tools like that for PHP, JavaScript, Python…) ● Improving the feedback: ○ Allowing students to ask questions in AutoFeedback if it's not clear enough ○ Integrating videos on top of text for feedback ● Showing lecturers the most common mistakes students make ● Supporting "micro-assessments" with code written from the website ○ Good for first-time programmers doing very short programs ○ May be useful for the first few weeks!