Do-it-yourself
e-Exams
Mathew Hillier
Monash University
Scott Grant
Monash University
ASCILITE Conference 25-28 Nov 2018 – Geelong, Australia
TRANSFORMINGEXAMS.COM
A Scalable Examination Platform for BYOD Invigilated Assessment
A key motivation - The gap
Real world of work Exams
2
World Economic Forum – How will digital change your working world.
https://agenda.weforum.org/wp-content/uploads/rtr2m8vm1-628x330.jpg
Exams at Monash Caufield in 2015 (mathew.hilier[at]monash.edu)
70,000 student university.
We are faced with a growing disconnect between
the way high stakes testing is conducted using pen
on paper exams and students’ everyday experiences
of study, work and life, let alone the future!
4IR
21C
A key idea …
SAMR Theory - Puentedura, R. (2012). Thinking About Change in Learning and Technology. Presentation
given September 25, 2012 at the 1st Global Mobile Learning Conference, Al Ain, UAE.
http://www.hippasus.com/rrpweblog/archives/2012/04/10/iPad_Intro.pdf
Study context – broader e-Exam project
Start > > > > > > > > > > > > > > Future >
Get Ready Phase 1 Phase 2 Phase 3 Phase 4 Phase 5
Institutional
approvals,
research
ethics,
hardware and
infrastructure
Paper
equivalent
small scale.
Post-paper
small to
medium.
Medium to
large scale.
Whitelisted
and logged
Internet
Open but fully
logged
Internet
Basic doc
exams to
begin!
Expanding the
app and media
landscape.
Adding the
power of an
LMS.
Network
BYOD exam.
Network
mixed mode
BYOD exam.
Crawling Walking Running Jumping Flying!
4
http://ta.vu/e-exam-roadmap
Extension work:
An offline e-learning platform
see moleap.org
The project has
progressed!
Moodle resistant to
network outages!
This paper!
Pedagogical Affordances
PDF
Scratch SDK
Exam doc Video
LMS quiz
Specialist applications
Sims
Authentic assessments. It is a platform, not an app!
Research Questions for spreadsheet pilot (phase 2.5)
Could a spreadsheet be used as a small-scale, do-it-yourself exam delivery system?
In particular, guided by literature, could we:
• Utilise standard and common spreadsheet software (leverage existing tools with
minimal barriers to use by teachers and students)
• Enable automated or semi-automated marking of selected response style questions
and short text responses without manual data entry (to realise work efficiencies of
technology and minimise errors)
• Not rely on a live network during the exam (to minimise the chance of disruption)
• Be hack proof - the student portion of the system must not to have answers ‘hidden’
within it.
Would students accept this approach?
In particular, ease of use, perceive it to be reliable and secure.
Doing it…
Room set up:
 Paper 'e-Exam quick start guide' and post-exam survey.
 A power socket was provided for each typist.
 Hand-writers were given a paper copy of the exam questions and response booklets. These
were available to typists upon request.
Both typists and hand-writers sat in the same room.
1) Students enter the room and seated.
2) Typists were given an e-Exam USB stick containing the questions.
3) Students start their computer with a USB stick progressing to the e-Exam desktop. A desktop
background image provides a visual check that all have booted from the correct USB.
4) Students enter their student ID and name into the e-Exam Starter screen.
5) All students now wait.
6) Invigilator announces the start of the exam and students click the ‘Start Exam’ button.
7) Auto-save occurs every 2 minutes.
8) At the end of the exam, the student uses File>Save one last time and Exit the Office suite.
9) When done they shut down the computer.
10) Students return the USB sticks.
11) Students complete the post-exam survey before leaving the room.
e-Exam workflow using spreadsheets (offline mode)
Post-exam: assessment
8.
Responses
retrieved
from USBs.
2. Create master USB
(load exam, configure, test)
Pre-exam: prepare exam
4. USBs sent to
exam venue
OR
USBs can be
recycled next
exam.
Linux Live
USB.
Libre Office
Spreadsheet
5. Pre-session:
Student laptop
setup &
practice.
3. USBs duplicated
1. Teacher creates spreadsheets
(exam and matching gradebook)
7. Collect
USBs
10. Marking
undertaken.
Checked by teacher.
9.
Responses
collated with
RDBmerge
6. Exam venue:
a. Students enter room
b.Given USB
c. Start laptop from USB
d.Do exam
e. Finalise and shutdown
f. Return USB
g. Leave room
11.
Return
feedback
to student.
NOTE!!!
NEW!!!Our
robust online
Moodle version
halves USB
handselling
Start up
Select
LOTE
Enter student info.
Spread sheet
The system then opens the
question and response
document automatically.
Language tools Language tools available
according to LOTE selection
Respond in designated cells
(other cells are locked).
Participation
Intro to Chinese (first year).
22 students at pre-exam practice.
16 typed the exam.
7 females and 9 males.
Caveat: sample was small and not random -
descriptive of these groups only.
Pre and post response trends
Strongly Disagree 1 2 3 4 5 Strongly Agree
Caveat: Small, not random samples – descriptive of these groups only.
1 2 3 4 5
Written instructions were easy to follow
It was easy to start my computer using the e-Exam USB stick
I can use the e-Exam system just as well as my own laptop system
It was easy to use the office suite (word processor/spread sheet)
It was easy to answer multiple-choice questions in the e-Exam…
Overall, I feel the e-Exam System is easy to use
I feel the e-Exam System is reliable against technical failures
I feel the e-Exam System is secure against cheating
I now feel relaxed about using the e-Exam system for my exam
I would recommend the e-Exam System to others
I would like to use a computer for exams in the future
Pre Post
Sign Test
n/s
0.03
n/s
n/s
n/s
n/s
n/s
n/s
n/s
n/s
n/s
Study Process
1) Two weeks prior: practice session + pre-survey.
2) Exam day: In-class, graded, supervised assessment task + post survey.
Summary - Key Findings
a) The offline Spreadsheet based e-Exam was rated well by the typists:
4+ out of 5.
b) The spreadsheet worked - each student completed successfully.
Responses were auto saved to USB each two minutes, retrieved following exam and
processed using a second ‘gradebook’ spreadsheet.
c) Marking was more efficient due to some auto marking and readable responses.
Possible next steps for spreadsheet version…
i) Trials in units (subjects) that would best take advantage of a spreadsheet
For example: accounting, finance, maths, statistics, science.
ii) Further development to streamline the workflow with respect to setup and
retrieval and marking.
iii) However project focus has shifted to the now proven ‘Robust’ online Moodle
approach!
Thank you
For further information contact:
Mathew.Hillier@gmail.com
TransformingExams.com
Presentations at ASCILITE2018:
1. Do-it-yourself e-Exams [8] Mon 11:30am D2.212
2. Towards authentic e-exams at scale – Robust networked Moodle [51]
Following this session! Same Room! Mon 12noon D2.212
3. Integrated mixed reality spatial learning analytics into secure
electronic exams [183]. Tue 11:20am D2.212

More Related Content

PDF
Computer application in Secondary education (Mind Map)
PDF
Computer application in secondary education
PPTX
Recommendation to my school head for institutional development
PPT
M4 - Computing - Week 1
PDF
CS Syllabus 2014
DOCX
Semi-Detailed Lesson Plan - Input and Output Devices of Computers
DOCX
Lesson plan Output Devices
DOCX
School technology survey
Computer application in Secondary education (Mind Map)
Computer application in secondary education
Recommendation to my school head for institutional development
M4 - Computing - Week 1
CS Syllabus 2014
Semi-Detailed Lesson Plan - Input and Output Devices of Computers
Lesson plan Output Devices
School technology survey

What's hot (6)

PPT
Jing Enhanced West Point Bridge Builder
PDF
Report on science computer assignment
PPTX
Smart Academic Timetable Reminder Mobile Application - First Slide
PPTX
PDF
ElectronicFinal Christy Twilight
PPTX
Bto130 Tour
Jing Enhanced West Point Bridge Builder
Report on science computer assignment
Smart Academic Timetable Reminder Mobile Application - First Slide
ElectronicFinal Christy Twilight
Bto130 Tour
Ad

Similar to ASCILITE 2018: Do-it-yourself e-Exams (using spreadsheets) (20)

PPTX
ASCILITE 2018: Towards authentic e-Exams at scale: robust networked Moodle
PPTX
Occe2018: Student experiences with a bring your own laptop e-Exam system in p...
PPTX
Quiz Buzz.pptx
PPTX
online exninition system ppt
DOCX
Online Examination System Report
PPTX
Engaging learners in computer-based summative exams: Reflections on a partici...
PPTX
Occe2018: writing e-exams in pre-university college
PPTX
LIJI ppt new
PDF
ASCILITE2015_post_e-exam_survey_slides
PPTX
Online Exam System
PPTX
Offline Marking of Practical Exams using OSCEMark
PDF
online learning and examination website
PDF
M-Learning Enabled Secure Exam Management Systems (MLESEMS)
PDF
onlineexaminationsystem-140315133558-phpapp01.pdf
PPTX
Online examination system
PPTX
Onlineexaminationsystem 140315133558-phpapp01
PPTX
Online Exam
PPT
online examination system
PPT
Online Examination
PPT
Online examination system
ASCILITE 2018: Towards authentic e-Exams at scale: robust networked Moodle
Occe2018: Student experiences with a bring your own laptop e-Exam system in p...
Quiz Buzz.pptx
online exninition system ppt
Online Examination System Report
Engaging learners in computer-based summative exams: Reflections on a partici...
Occe2018: writing e-exams in pre-university college
LIJI ppt new
ASCILITE2015_post_e-exam_survey_slides
Online Exam System
Offline Marking of Practical Exams using OSCEMark
online learning and examination website
M-Learning Enabled Secure Exam Management Systems (MLESEMS)
onlineexaminationsystem-140315133558-phpapp01.pdf
Online examination system
Onlineexaminationsystem 140315133558-phpapp01
Online Exam
online examination system
Online Examination
Online examination system
Ad

More from mathewhillier (7)

PPTX
ASCILITE 2018: Integrating mixed reality spatial learning analytics into secu...
PPTX
Moodle quiz: towards post-paper e-assessment
PPTX
occe2020 - e-exams: technology problems and its impact on student experience
PPTX
Opportunities and Challenges for Assessment in XR: Transforming Assessment
PDF
ASCILITE2015_possible_e-exam_futures
PPTX
Approximating volume using slabs
PPT
e-Learning and e-assessment examples
ASCILITE 2018: Integrating mixed reality spatial learning analytics into secu...
Moodle quiz: towards post-paper e-assessment
occe2020 - e-exams: technology problems and its impact on student experience
Opportunities and Challenges for Assessment in XR: Transforming Assessment
ASCILITE2015_possible_e-exam_futures
Approximating volume using slabs
e-Learning and e-assessment examples

Recently uploaded (20)

PDF
FORM 1 BIOLOGY MIND MAPS and their schemes
PDF
LEARNERS WITH ADDITIONAL NEEDS ProfEd Topic
PDF
Journal of Dental Science - UDMY (2020).pdf
PDF
Skin Care and Cosmetic Ingredients Dictionary ( PDFDrive ).pdf
PDF
Myanmar Dental Journal, The Journal of the Myanmar Dental Association (2013).pdf
PDF
Literature_Review_methods_ BRACU_MKT426 course material
PDF
BP 704 T. NOVEL DRUG DELIVERY SYSTEMS (UNIT 2).pdf
PDF
Climate and Adaptation MCQs class 7 from chatgpt
PDF
CISA (Certified Information Systems Auditor) Domain-Wise Summary.pdf
PDF
Complications of Minimal Access-Surgery.pdf
PPTX
Education and Perspectives of Education.pptx
PDF
Τίμαιος είναι φιλοσοφικός διάλογος του Πλάτωνα
PDF
Empowerment Technology for Senior High School Guide
PDF
FOISHS ANNUAL IMPLEMENTATION PLAN 2025.pdf
PPTX
DRUGS USED FOR HORMONAL DISORDER, SUPPLIMENTATION, CONTRACEPTION, & MEDICAL T...
PPTX
What’s under the hood: Parsing standardized learning content for AI
PDF
BP 505 T. PHARMACEUTICAL JURISPRUDENCE (UNIT 2).pdf
PPTX
Module on health assessment of CHN. pptx
PDF
IP : I ; Unit I : Preformulation Studies
PPTX
ELIAS-SEZIURE AND EPilepsy semmioan session.pptx
FORM 1 BIOLOGY MIND MAPS and their schemes
LEARNERS WITH ADDITIONAL NEEDS ProfEd Topic
Journal of Dental Science - UDMY (2020).pdf
Skin Care and Cosmetic Ingredients Dictionary ( PDFDrive ).pdf
Myanmar Dental Journal, The Journal of the Myanmar Dental Association (2013).pdf
Literature_Review_methods_ BRACU_MKT426 course material
BP 704 T. NOVEL DRUG DELIVERY SYSTEMS (UNIT 2).pdf
Climate and Adaptation MCQs class 7 from chatgpt
CISA (Certified Information Systems Auditor) Domain-Wise Summary.pdf
Complications of Minimal Access-Surgery.pdf
Education and Perspectives of Education.pptx
Τίμαιος είναι φιλοσοφικός διάλογος του Πλάτωνα
Empowerment Technology for Senior High School Guide
FOISHS ANNUAL IMPLEMENTATION PLAN 2025.pdf
DRUGS USED FOR HORMONAL DISORDER, SUPPLIMENTATION, CONTRACEPTION, & MEDICAL T...
What’s under the hood: Parsing standardized learning content for AI
BP 505 T. PHARMACEUTICAL JURISPRUDENCE (UNIT 2).pdf
Module on health assessment of CHN. pptx
IP : I ; Unit I : Preformulation Studies
ELIAS-SEZIURE AND EPilepsy semmioan session.pptx

ASCILITE 2018: Do-it-yourself e-Exams (using spreadsheets)

  • 1. Do-it-yourself e-Exams Mathew Hillier Monash University Scott Grant Monash University ASCILITE Conference 25-28 Nov 2018 – Geelong, Australia TRANSFORMINGEXAMS.COM A Scalable Examination Platform for BYOD Invigilated Assessment
  • 2. A key motivation - The gap Real world of work Exams 2 World Economic Forum – How will digital change your working world. https://agenda.weforum.org/wp-content/uploads/rtr2m8vm1-628x330.jpg Exams at Monash Caufield in 2015 (mathew.hilier[at]monash.edu) 70,000 student university. We are faced with a growing disconnect between the way high stakes testing is conducted using pen on paper exams and students’ everyday experiences of study, work and life, let alone the future! 4IR 21C
  • 3. A key idea … SAMR Theory - Puentedura, R. (2012). Thinking About Change in Learning and Technology. Presentation given September 25, 2012 at the 1st Global Mobile Learning Conference, Al Ain, UAE. http://www.hippasus.com/rrpweblog/archives/2012/04/10/iPad_Intro.pdf
  • 4. Study context – broader e-Exam project Start > > > > > > > > > > > > > > Future > Get Ready Phase 1 Phase 2 Phase 3 Phase 4 Phase 5 Institutional approvals, research ethics, hardware and infrastructure Paper equivalent small scale. Post-paper small to medium. Medium to large scale. Whitelisted and logged Internet Open but fully logged Internet Basic doc exams to begin! Expanding the app and media landscape. Adding the power of an LMS. Network BYOD exam. Network mixed mode BYOD exam. Crawling Walking Running Jumping Flying! 4 http://ta.vu/e-exam-roadmap Extension work: An offline e-learning platform see moleap.org The project has progressed! Moodle resistant to network outages! This paper!
  • 5. Pedagogical Affordances PDF Scratch SDK Exam doc Video LMS quiz Specialist applications Sims Authentic assessments. It is a platform, not an app!
  • 6. Research Questions for spreadsheet pilot (phase 2.5) Could a spreadsheet be used as a small-scale, do-it-yourself exam delivery system? In particular, guided by literature, could we: • Utilise standard and common spreadsheet software (leverage existing tools with minimal barriers to use by teachers and students) • Enable automated or semi-automated marking of selected response style questions and short text responses without manual data entry (to realise work efficiencies of technology and minimise errors) • Not rely on a live network during the exam (to minimise the chance of disruption) • Be hack proof - the student portion of the system must not to have answers ‘hidden’ within it. Would students accept this approach? In particular, ease of use, perceive it to be reliable and secure.
  • 7. Doing it… Room set up:  Paper 'e-Exam quick start guide' and post-exam survey.  A power socket was provided for each typist.  Hand-writers were given a paper copy of the exam questions and response booklets. These were available to typists upon request. Both typists and hand-writers sat in the same room. 1) Students enter the room and seated. 2) Typists were given an e-Exam USB stick containing the questions. 3) Students start their computer with a USB stick progressing to the e-Exam desktop. A desktop background image provides a visual check that all have booted from the correct USB. 4) Students enter their student ID and name into the e-Exam Starter screen. 5) All students now wait. 6) Invigilator announces the start of the exam and students click the ‘Start Exam’ button. 7) Auto-save occurs every 2 minutes. 8) At the end of the exam, the student uses File>Save one last time and Exit the Office suite. 9) When done they shut down the computer. 10) Students return the USB sticks. 11) Students complete the post-exam survey before leaving the room.
  • 8. e-Exam workflow using spreadsheets (offline mode) Post-exam: assessment 8. Responses retrieved from USBs. 2. Create master USB (load exam, configure, test) Pre-exam: prepare exam 4. USBs sent to exam venue OR USBs can be recycled next exam. Linux Live USB. Libre Office Spreadsheet 5. Pre-session: Student laptop setup & practice. 3. USBs duplicated 1. Teacher creates spreadsheets (exam and matching gradebook) 7. Collect USBs 10. Marking undertaken. Checked by teacher. 9. Responses collated with RDBmerge 6. Exam venue: a. Students enter room b.Given USB c. Start laptop from USB d.Do exam e. Finalise and shutdown f. Return USB g. Leave room 11. Return feedback to student. NOTE!!! NEW!!!Our robust online Moodle version halves USB handselling
  • 10. Spread sheet The system then opens the question and response document automatically.
  • 11. Language tools Language tools available according to LOTE selection Respond in designated cells (other cells are locked).
  • 12. Participation Intro to Chinese (first year). 22 students at pre-exam practice. 16 typed the exam. 7 females and 9 males. Caveat: sample was small and not random - descriptive of these groups only.
  • 13. Pre and post response trends Strongly Disagree 1 2 3 4 5 Strongly Agree Caveat: Small, not random samples – descriptive of these groups only. 1 2 3 4 5 Written instructions were easy to follow It was easy to start my computer using the e-Exam USB stick I can use the e-Exam system just as well as my own laptop system It was easy to use the office suite (word processor/spread sheet) It was easy to answer multiple-choice questions in the e-Exam… Overall, I feel the e-Exam System is easy to use I feel the e-Exam System is reliable against technical failures I feel the e-Exam System is secure against cheating I now feel relaxed about using the e-Exam system for my exam I would recommend the e-Exam System to others I would like to use a computer for exams in the future Pre Post Sign Test n/s 0.03 n/s n/s n/s n/s n/s n/s n/s n/s n/s
  • 14. Study Process 1) Two weeks prior: practice session + pre-survey. 2) Exam day: In-class, graded, supervised assessment task + post survey.
  • 15. Summary - Key Findings a) The offline Spreadsheet based e-Exam was rated well by the typists: 4+ out of 5. b) The spreadsheet worked - each student completed successfully. Responses were auto saved to USB each two minutes, retrieved following exam and processed using a second ‘gradebook’ spreadsheet. c) Marking was more efficient due to some auto marking and readable responses. Possible next steps for spreadsheet version… i) Trials in units (subjects) that would best take advantage of a spreadsheet For example: accounting, finance, maths, statistics, science. ii) Further development to streamline the workflow with respect to setup and retrieval and marking. iii) However project focus has shifted to the now proven ‘Robust’ online Moodle approach!
  • 16. Thank you For further information contact: Mathew.Hillier@gmail.com TransformingExams.com Presentations at ASCILITE2018: 1. Do-it-yourself e-Exams [8] Mon 11:30am D2.212 2. Towards authentic e-exams at scale – Robust networked Moodle [51] Following this session! Same Room! Mon 12noon D2.212 3. Integrated mixed reality spatial learning analytics into secure electronic exams [183]. Tue 11:20am D2.212

Editor's Notes

  • #2: ABSTRACT Electronic examination and assessment systems pose a challenge in both finding and/or creating a solution that is effective and practical from a real world and software engineering perspective. Any proposed solution must be flexible to allow for a range of assessments, taking into account the requirements and expectations of multiple stakeholders such as student, teachers, administrators and technical support. This includes being easy to use, open enough to prevent vendor lock-in, practical to implement, secure and efficient to support. In this presentation, we explore the current incarnation of an e-Exam platform, in particular its approach of using purely open source components, the mechanics of the system and design decisions taken in developing a modular and automated foundation for a systematic electronic learning and assessment approach. This, together with a Bring Your Own Device strategy for hardware provision demands a high level of common hardware compatibility in an ever-changing technology landscape. This talk describes general experiences with e-exams such as advantages, challenges, preparing a good e-exam and tips on the implementation
  • #4: Puentedura, R. (2012). Thinking About Change in Learning and Technology. Presentation given September 25, 2012 at the 1st Global Mobile Learning Conference, Al Ain, UAE. http://www.hippasus.com/rrpweblog/archives/2012/04/10/iPad_Intro.pdf
  • #5: Spare slide
  • #6: Example of a question that requires a very complex response, which could not be posed without a computer
  • #9: How your system works, in a nutshell. A diagram or photo to illustrate the critical element of the technological implementation
  • #10: Illustration of the technological implementation of the workflow for running an e-exam on ‘robust’ Moodle infrastructure. Exam content resides on a server. USB sticks serve as a standardised operating and application suite. USBs are immediately reusable without manual processing*. System is resistant to network outages and uses secure browser with keys for access to server. Responses are auto saved every minute to server or in case of network outage saved to USB in an encrypted file. Once the exam starts the quiz content is cached to the browser, from then onwards the network is not required for successful exam continuance and completion. USBs can be used in a generic manner but can also be customised with additional software tools or to pre-load large media files. The above workflow was used successfully for e-Exam trials in two units in semester 1 2018. Each unit had a sequence of a practice session followed by two or three exams. Our trials used WIFI dongles to expand the range of compatible laptops as development of the system continued to integrate more drivers. The last trials in the sequence did not need the drivers because we had expanded the compatibility or students borrowed compatible laptops. e-Exam USBs are configurable for multiple use cases. This includes the use of local files in fully offline mode, the use of the robust online mode via SEB, or the use of online whitelisted internet based resources. The e-Exam USB can be used to provide a unified examination environment across BYO laptops, institution owned laptops and computers including computer labs provided boot from USB is enabled. * A full USB auto reset feature is still under development that will be applicable when response submission is entirely handled via Moodle where a live network connection exists at the end of the exam.