Detecting and addressing contract cheating in online assessment

Detecting and addressing
contract cheating in online
assessment
Phillip (Phill) Dawson
Centre for Research in Assessment and
Digital Learning (CRADLE)
Deakin University, Melbourne, Australia

Disclaimer
• I support AfL & AI
• I think cheating is a symptom of
broader problems
• I think universities have a
responsibility to try to prevent
and detect cheating
• I receive research funding from
ed tech companies
• My mum helped me contract
cheat in year four

5 things to
take from this
presentation
1: Cheating is a big problem
2: There are no quick fixes or perfect
solutions
3: We need to balance academic
integrity and assessment security
4: There are some approaches that
might help
5: Assessment design trumps
assessment security
@phillipdawson

1: Cheating is a big problem
• Students don’t just find
cheating, it finds them
• A small but significant
proportion of students
cheat
• Only a tiny fraction of
cheating gets detected

…research paper…
…hate stats…
…due tomorrow…
Amigud, A. (2020). Cheaters on Twitter: an analysis of engagement approaches of contract cheating services.
Studies in Higher Education, 45(3), 692-705. doi:10.1080/03075079.2018.1564258

Australia, extrapolated from
Bretag et al 2018
~80k total
avg. ~2k per uni
UK

2: There are no quick fixes or perfect solutions
• Harsh restrictions won’t
stop cheating
• Technology won’t stop
cheating
• Blind faith won’t stop
cheating
• Cheating can occur in any
task type

Exams won’t save us.
‘Third party cheating’ is likely more common in
exams than assignments.
‘Third party cheating’ is likely detected less often in
exams than assignments.
(Harper et al, 2020)
(Based on comparisons of self-reported student cheating and educator detection rates – so take with caution)
Harper, R., Bretag, T., & Rundle, K. (2020). Detecting contract cheating: examining the role of assessment type. Higher Education Research & Development, 1-16. doi:10.1080/07294360.2020.1724899

Remote proctoring might help, but vendors won’t let
me test it out.
Treat their claims with caution.
(Dawson, 2021)
Dawson, P. (2021). Defending Assessment Security in a Digital World: Preventing E-Cheating and Supporting Academic Integrity in Higher Education. Routledge.

“If we trust students not to cheat then cheating
rates will go down” sounds good but evidence is
thin.
(e.g. honor codes help, but only explain a small
amount of the variance in cheating rates)
(McCabe et al 2002)
McCabe, D. L., Treviño, L. K., & Butterfield, K. D. (2002, 2002/06/01). Honor Codes and Other Contextual Influences on Academic Integrity: A Replication and Extension to Modified Honor Code
Settings. Research in Higher Education, 43(3), 357-378. https://doi.org/10.1023/A:1014893102151

Banning essays won’t save us.
Authentic assessment won’t save us.
Contract cheating can occur with any task type.
(Ellis et al, 2019)
Ellis, C., van Haeringen, K., Harper, R., Bretag, T., Zucker, I., McBride, S., . . . Saddiqui, S. (2019). Does authentic assessment assure academic integrity? Evidence from contract cheating data. Higher Education
Research & Development, 1-16. doi:10.1080/07294360.2019.1680956

3: We need to balance academic
integrity and assessment security
• Academic integrity is positive,
educative and values-based
• Assessment security is
adversarial, punitive and
evidence-based
• In tension, but not a dichotomy

Fundamental values
of academic integrity
• Honesty
• Trust
• Fairness
• Respect
• Responsibility
• Courage

Assessment security:
“measures taken to harden assessment against
attempts to cheat. This includes approaches to
detect and evidence attempts to cheat, as well as
measures to make cheating more difficult.”
(Dawson, 2021)
Dawson, P. (2021). Defending Assessment Security in a Digital World: Preventing E-Cheating and Supporting Academic Integrity in Higher Education. Routledge.

Addressing cheating requires…
Academic Integrity
• Trusting
• Educative
• Proactive
Think ‘crime prevention’
Assessment security
• Detecting
• Punitive
• Proactive or reactive
‘policing’ or ‘surveillance’
It’s a balance, not a dichotomy

4: There are some
approaches that might work
• Talking with students
• Designing tasks students say they are less likely
to cheat in
• Trying to detect cheating in take-home tasks
• Vivas/dialogues
• Remote proctoring
• Authentic assessment restrictions
• Random audits
• Programmatic assessment

Talk with students.
Contract cheating sites lie and even blackmail.
(Rowland et al, 2018; Yorke, et al in press)
Rowland, S., Slade, C., Wong, K.-S., & Whiting, B. (2018). ‘Just turn to us’: the persuasive features of contract cheating websites. Assessment & Evaluation in Higher Education, 43(4), 652-665.
doi:10.1080/02602938.2017.1391948
Yorke, J., Sefcik, L., & Veeran-Colton, T. (in press). Contract cheating and blackmail: a risky business? Studies in Higher Education. doi:10.1080/03075079.2020.1730313

Talk with students.
“You don’t always get what you pay for.”
(Sutherland-Smith & Dullaghan)
Sutherland-Smith, W., & Dullaghan, K. (2019). You don’t always get what you pay for: User experiences of engaging with contract cheating sites. Assessment & Evaluation in Higher Education, 44(8), 1148-1162.
doi:10.1080/02602938.2019.1576028

0 5 10 15 20 25 30 35 40
Short turnaround time
Heavily weighted task
Series of small graded tasks
Research, analysis and thinking skills
No ‘right’ answer
Integrate knowledge/skills vital to programme
Relevant professional skills
Major part of nested task
Small part of nested task
In-class task
Personalised and unique
Viva
Reflection on practicum
Listen to students.
Students’ perceptions of the likelihood of contract cheating (%)
Bretag, T., Harper, R., Burton, M., Ellis, C., Newton, P., van Haeringen, K.,
et al. (2019). Contract cheating and assessment design: exploring the
relationship. Assessment & Evaluation in Higher Education, 44(5), 676-691.

Rundle, K., Curtis, G., & Clare, J. (2020). Why students choose not to cheat. In T.
Bretag (Ed.), A Research Agenda for Academic Integrity. Edward Elgar Publishing.

Listen to professional cheaters.

Can markers detect contract cheating?
(Dawson & Sutherland-Smith, 2018)
Can training improve detection rates?
(Dawson & Sutherland-Smith, 2019)
Can software improve detection rates?
(Dawson, Sutherland-Smith & Ricksen, in press)
Dawson, P., & Sutherland-Smith, W. (2018). Can markers detect contract cheating? Results from a pilot study. Assessment & Evaluation in Higher Education, 43(2), 286-293. doi:10.1080/02602938.2017.1336746
Dawson, P., & Sutherland-Smith, W. (2019). Can training improve marker accuracy at detecting contract cheating? A multi-disciplinary pre-post study. Assessment & Evaluation in Higher Education, 44(5), 715-725.
doi:10.1080/02602938.2018.1531109
Dawson, P., Sutherland-Smith, W., & Ricksen, M. (in press, published online 2019). Can software improve marker accuracy at detecting contract cheating? A pilot study of the Turnitin authorship investigate alpha.
Assessment & Evaluation in Higher Education, 1-10. doi:10.1080/02602938.2019.1662884

0 20 40 60 80 100
Trained to look for it
Looking for it
Not looking for it
Accuracy at detecting contract cheating (%)
e.g. Lines 2016; Medway et al 2018
Lines, L. (2016). Ghostwriters guaranteeing grades?
The quality of online ghostwriting services available to
tertiary students in Australia. Teaching in Higher
Education, 1-26. doi:10.1080/13562517.2016.1198759
Medway, D., Roper, S., & Gillooly, L. (2018). Contract
cheating in UK higher education: A covert investigation
of essay mills. British Educational Research Journal,
44(3), 393-418. doi:10.1002/berj.3335

0 20 40 60 80 100
Trained to look for it
Looking for it
Not looking for it
False positive rate (%)
e.g. Lines 2016; Medway et al 2018
Lines, L. (2016). Ghostwriters guaranteeing grades?
The quality of online ghostwriting services available to
tertiary students in Australia. Teaching in Higher
Education, 1-26. doi:10.1080/13562517.2016.1198759
Medway, D., Roper, S., & Gillooly, L. (2018). Contract
cheating in UK higher education: A covert investigation
of essay mills. British Educational Research Journal,
44(3), 393-418. doi:10.1002/berj.3335

Potential generic indicators of contract cheating
reflection done poorly
metadata
unusual mistakes
not using materials from class
reads as something by a generalist
strange formatting
does not address question
wrong task type

Viva had 100% detection rate
(Needs replication before we publish but it’s a
good sign)

Consider the pros and cons of
remote proctoring – and
demand evidence
Do students really hate
proctoring so much?
Is proctoring actually an effective
anti-cheating measure?
Dawson, P. (2021). Defending Assessment Security in a Digital World: Preventing E-Cheating and Supporting Academic Integrity in
Higher Education. Routledge.

Authentic restrictions:
Authentic assessment is great but it
doesn’t stop cheating. Authentic
restrictions might.
Restrictions need to be enforced, and
therefore make assessment harder to
secure.
Authentic restrictions reduce the
‘attack surface’.
Allowing students tools, collaboration
and/or information reduces the options
for cheating.
https://teaching.unsw.edu.au/academic-integrity/case-studies
David Kellermann UNSW @DrKellermann

Programmatic assessment security:
‘Cheat-proofing’ every act of
assessment is probably impossible
and definitely a bad idea.
Focus on securing those acts of
assessment that matter to the
degree program outcomes.
Focus on developing academic
integrity in the others.

Random audit:
‘Cheat-proofing’ every act of
assessment is probably
impossible and definitely a bad
idea.
Consider random audit of
individual students’ work.
The possibility of an audit is
associated with more honest
behavior in other contexts (e.g.
tax)

5: Assessment
design trumps
assessment
security
It’s only worth securing reliable
and valid assessment of learning

5 things to
take from this
presentation
Cheating is a big problem
There are no quick fixes or perfect
solutions
We need to balance academic integrity
and assessment security
There are some approaches that might
help
Assessment design trumps assessment
security
@phillipdawson

Bonus content:
Seven proposed
standards for
assessment
security

Coverage
across a
programme
How much of a degree should be
secured?
Securing everything, nothing, or in a
haphazard way is unacceptable
Securing the major task in each
unit/module is commonplace but we
can do better
Best practice is programmatic

Authentication
How do we ensure the student is who they
say they are?
Unacceptable to not attempt this (for the
tasks we are securing)
Minimum is to use photo ID or something
else like stylometrics or biometrics – but this
just establishes who they are right now
Aspirational is to dialogue with students
about the work

Control of
circumstances
How can we be sure the task was done
in the circumstances we designed for?
Unacceptable to not attempt to control
(for the tasks we are securing)
Some tasks require a combination of
proctoring, lockdown or text matching
to be secure
At an aspirational level we should try
non-text and/or semantic matching

Difficulty to
cheat metrics
We need to know how hard tasks are to cheat in
Unacceptable to just assume tasks are hard to
cheat in without any evidence
(for the tasks we are securing)
At a minimum, educators should complete tasks
themselves and consider how to cheat
As an aspiration we should get specialists to audit
tasks for how difficult they are to cheat

Detection
accuracy
metrics
We need to know if our detection
approaches work
Unacceptable to just assume we are
detecting all cases
At a minimum, approaches should be
based on research into detection
As an aspiration we should validate that
detection approaches actually work

Proof metrics
We need to be able to prove cases of
cheating
Unacceptable to not keep records about
proof rates, or to only rarely prove cases
As a minimum, records need to be kept
about proof rates, and proven cases should
be relatively common
As an aspiration, proof rates should be
externally audited and benchmarked

Prevalence
metrics
We need to know approximate rates of
undetected, detected and proven cheating
in a cohort
Unacceptable to not attempt to compare
rates of proven cases with expected rates
At a minimum, rates of proven cases should
be benchmarked against prevalence data
from similar programmes
As an aspiration, existing research is used to
calculate prevalence of unproven cheating

Detecting and addressing contract cheating in online assessment

More Related Content

Similar to Detecting and addressing contract cheating in online assessment (20)

Recently uploaded (20)

Detecting and addressing contract cheating in online assessment

Editor's Notes