Rethinking our approach to plagiarism
Download as PPTX, PDF5 likes1,442 views
This document summarizes a comparison of two plagiarism detection tools: TurnItIn and Urkund. It outlines criteria for evaluation such as cost, functionality, matching ability, turnaround time, and support. Data is presented on usage from 2014-2017 showing Urkund requiring less support tickets and having faster turnaround. While TurnItIn has more functionality, Urkund has a lower cost. Based on the evaluation, the next steps outlined are to terminate TurnItIn in June 2017 and develop an audio feedback plugin, while negotiating a better price for Urkund by increasing its user base.
Rethinking our approach to plagiarism
- 1. @glynnmark Rethinking our approach plagiarism Dr Mark Glynn Dr Shadi Karazi
- 2. @glynnmark #heie Outline 1. Background 2. Criteria 3. Text Matching Evaluation 4. Next steps
- 3. @glynnmark #heie Text matching evaluation What are we trying to achieve?
- 4. @glynnmark #heie Overview of use of TurnItIn Year Originality reports Grade Mark 2014 26678 3107 2015 31003 4216 2016 31735 5897 2017 23019 3614
- 5. @glynnmark #heie Criteria Cost Functionality Matching ability Turnaround time Support
- 6. @glynnmark #heie Support required Both systems produce a similarity score 23019 reports generated through submissions made to TurnItIn. In that time we had 52 tickets to our helpdesk related to TurnItIn In the same period we ran 55000+ submissions through Urkund and we had 4 tickets.
- 7. @glynnmark #heie Turn around time In consistent and unreliable Plugin related
- 8. @glynnmark #heie Matching Ability 90% of the documents analysed had a difference in similarity scores of less than 10% (with 80% of the documents having less than 5% difference in similarity score). The differences in the score can be due to three main factors. Similarity Score 75 – 100% 50 – 74% 25 – 49% 0 – 24% % of Reports 1.65% 7.67% 27.5% 67.98%
- 9. @glynnmark #heie Functionality Feature TurnItIn Moodle + Urkund Annotation* Can the lecturer annotate a student’s submission x x Marking Guides Structured feedback on specified criteria with a marks range for each student x Rubrics In Built Rubrics x x Group work x Similarity reports x x Peer Review x x Rubric / Marking guides reports** Can the lecturer get a report providing a breakdown of all of the marks and comments for each criteria within a rubric/marking guide given to the entire class x Audio Feedback x Extensions Can the lecturer grant deadline extensions to one or more students easily x
- 10. @glynnmark #heie Cost Year Cost 2014 £15,000 2015 £23,302 2016 £25,247 Urkund - £9158
- 11. @glynnmark #heie Next steps We terminate TurnItIn in June 2017. We have developed an audio plugin to facilitate audio feedback similar to TurnItIn. We get more people buying Urkund to build the database and help us negotiate a better price.
Editor's Notes
- #4: Students enoy the use of Moodle in one class –will demand it in others
- #7: Both TurnItIn and Urkund are integrated in Loop via plugin. Both systems produce a similarity score which is produced as part of the originality report and fed back to Loop. For the duration of the evaluation there were 23019 reports generated through submissions made to TurnItIn. In that time we had 52 tickets to our helpdesk related to TurnItIn. In the same period we ran 55000+ submissions through Urkund and we had 4 tickets.
- #9: Without exception TurnItIn produced a higher similarity score for each submission, however the 90% of the documents analysed had a difference in similarity scores of less than 10% (with 80% of the documents having less than 5% difference in similarity score). The differences in the score can be due to three main factors: Each system uses their own algorithm which leads to slightly different results. This is manifested by TurnItIn counting “matches” in documents with a smaller number of words relative to what Urkund classifies as a “match” TurnItIn has a larger dataset to chose from as it has a larger customer base particularly in higher education in the UK therefore would have more documents at their disposal Furthermore TurnItIn analyses relatively more journal databases that are not openly available.
- #10: While both TurnItIn and Urkund offer similar services i.e. text matching, it is important to note that TurnItIn also provides feedback functionality. Key features are illustrated in Table 3 below. The feedback feature within TurnItIn is an excellent addition to the product but currently is only used by 89 (8.5%) of our academic staff members for feedback to students. The majority of the staff use it just for similarity scores. It is worth noting that while Urkund does not have feedback functionality Loop does and there will be significant improvements in the next version of Loop scheduled for upgrade in June 2017. Therefore the table below includes Loop in the analysis as well. The question now becomes is the better functionality in TurnItIn for feedback worth the additional year on year cost
- #11: The final factor considered in this evaluation is cost. TurnItIn cost €23,406 in 2014, €40247 in 2015 and €33497 in 2016. The actual invoice is quoted in sterling and while our rate was increased from 2014 to 2015 (£23,302) the rate remained stable for 2016 (£25,247) but fluctuations in currency strength resulted in a cost saving. The cost for Urkund is €10,720 per annum and that is the normal ongoing price i.e. not a special introductory offer. The cost saving between TurnItIn and Urkund is €22,777 per annum. This difference can be put towards improving the feedback functionality of Loop to match if not exceed the current capabilities of TurnItIn as a once off cost.