Introduction to CrossCheck

Editor's Notes

• #3: But before I do that I’d like to give you a little bit of background on CrossRef - what CrossRef is, and why we are running an originality screening service.
• #4: CrossRef was founded fifteen years ago to solve the problem of broken links. The web is all about links, but links break. This is annoying if you’re browsing the web and want to follow an interesting link, but in the context of scholarly publishing it becomes more than annoying - if you can’t follow a citation from one paper to another you’re being hampered in your research. CItation linking is one of the greatest benefits of online publishing, but it really does need to be reliable
• #5: ...and publishers were finding that web sites changed, content moved, and links that they had put into their articles stopped working. So they started a multi-publisher initiative to solve this problem of broken links.
• #6: This is done using the DOI - the Digital Object Identifier, which I’m sure many of you are familiar with. A CrossRef DOI is simply a unique identifier for a piece of content. Once assigned, it doesn’t change. It is to all intents and purposes a meaningless number, but it allows that piece of content to be located on the web.
• #7: And it works like this: publishers use CrossRef DOIs to link to content, usually from the references at the end of articles. Users click on those DOI-based links and are referred via the CrossRef database to the cited article at it’s correct location on the web. If content moves the publisher only has to update the CrossRef database once, and all of the publishers that are linking to their content using CrossRef DOIs will be redirected to the content in its new location.
• #8: And nowadays CrossRef DOIs are also used in many other scenarios - basically whenever someone wants to reliably and persistently link to a piece of content they can do so using CrossRef DOIs. Here are a couple of examples of people tweeting links to articles using CrossRef DOIs...
• #9: A few numbers for you to give some idea of how CrossRef has grown in the fifteen years since its launch... Books are the fastest growing at the moment - most publishers have assigned DOIs to their journals and journal archives, but more and more are now starting to assign them to their books, and to register their book metadata with CrossRef. Publishers are also registering components – over 1m plus over 1m datasets.
• #10: Every month there are around 100 million clicks on CrossRef DOI links, so 100 million citations resolved to content.
• #11: And a little about us as an organisation. *UK and Boston, MA *Publishers of all sizes, subjects, and nationalities - you can see the list on our website if you’re interested. *One of the reasons we achieve so much with such a small staff is that we’re lucky to be supported by a network of working groups and committees.
• #12: For a bit more background, this is the CrossRef mission statement. We’re a not-for profit membership association here to support the scholarly communications industry. As you can see our mission is about more than just reference linking, even though linking is still the main thing that we do for publishers. And as such we have developed other services to meet the needs of our membership - originality screening being one of them.
• #13: So let’s look at how much of a problem plagiarism is in scholarly publishing. It’s certainly not a new problem, but is it one that’s getting worse? It has never been easier to search across vast amounts of content in online publications and databases. And with more content being produced than ever before it’s much harder for reviewers to have thoroughly read everything in their field.
• #14: I’d like to touch on a few studies that have attempted to quantify levels of plagiarism in research and publishing. Going back a few years to Nature in 2005, this survey of just over 3000 US scientists asked about “questionable research practices” and 1.4% of respondents admitted to using another’s ideas without obtaining permission or giving due credit”.
• #15: This is another piece from Nature, this time in June 2008, and this particular survey flipped things around and looked at misconduct witnessed or suspected by colleagues but going unreported. 2000 or so scientists responded, and 164 of them reported 201 cases of observed misconduct. 36% of those cases were plagiarism.
• #16: And finally a study that has been reported and commented on in a number of places. A team led by Harold Garner and now based at the Virginia Bioinformatics Institute analysed the contents of Medline using a piece of text comparison software called eTBLAST. By comparing texts against each other computationally they identified pairs of articles with high levels of similarity.
• #17: Where the software threw up matches they went on to manually compare the documents - as you can see here the areas of this article that appear in an earlier article have been highlighted. And they have written up their findings in each case in the deja vu database.
• #18: You can access the database with their findings at the URL I’ve listed here. At the time that this article appeared in Science in May of 2009 year there had been at least 48 retractions of suspicious papers that were flagged to editors as a result of this project.
• #19: The team went on to contact the authors and journals involved and documented their responses - some of them were written up in Science in March 2009.
• #20: This is a quote from one of the authors whose work was plagiarised. READ SLOWLY. He is referring to plagiarism detection software used in schools, which I’ll come back to a little later.
• #21: So we started to develop CrossCheck. We ran a pilot towards the end of 2007 and the start of 2008 with 7 major publishers and a technology partner iParadigms, and on successful conclusion of the pilot we launched the live CrossCheck service in June 2008. So it’s been running now for around 8 years.
• #22: The first thing that I always say when I talk about CrossCheck is that although we call it a plagiarism detection service, it doesn’t actually detect plagiarism.
• #23: Here’s another definition of plagiarism, this time from an online dictionary. I won’t read this one out to you, but the last sentence is where I would ask you to focus - plagiarism is usually distinguished from imitation by its dishonest intention.
• #24: A machine cannot detect plagiarism. A machine can look at written text and tell you where it matches other written text, but it cannot tell you why that text matches, and this is critical. It takes a human being with a certain amount of domain expertise to analyse the results of any computer programme in order to determine the intent of the author or authors.
• #25: There are legitimate reasons why text might be the same in two documents. Here’s a bibliography section which will almost-certainly be repeated in numerous places. A mathematical proof might be repeated in order to be extended, and so on... a human can spot some of these examples very easily whereas a piece of software cannot.
• #26: So let’s take a look at how CrossCheck actually works by starting with a simple overview. You have a manuscript or document that you want to check for originality. You submit it to the CrossCheck system, where it is broken down and analysed, then compared against a database of other scholarly publications. Wherever overlapping or similar passages of text are found, they are highlighted in a report for an editor to take a look at.
• #28: To effectively screen research material you need to compare it with other research material, and most of that is in publications that are on many different publisher platforms and often behind access control. So even if you find a match using Google Scholar you will still need to go to the publisher’s website to see the abstract, which may or may not contain the matching text. If it doesn’t, you need to get access to the full text, which may or may not involve paying, and so on and so forth.
• #29: This is where CrossRef is able to bring something new to a service such as this. We already have working relationships with over 5000 publishers and societies, and can circumvent the need for so many bilateral agreements to create a comprehensive database of content.
• #30: And this is precisely what we’ve done - we’ve facilitated the indexing of full text content from CrossRef publishers who join CrossCheck, and with iParadigms have put this content into a database to screen documents against.
• #31: Just to talk about our technology partner for this project for a moment. Several technology providers were considered when we were planning CrossCheck. The one that we decided to work with is an organisation called iParadigms, who are based in Northern California. Their proven technology is probably best known as powering the Turnitin plagiarism screening tools for higher education. Turnitin is used widely in the UK and USA and I believe in several other European countries. Since 2003 it has also been available in the form of a commercial product called iThenticate. And it’s the iThenticate system that is made available to CrossCheck participants.
• #32: So to look at the process in a little more detail: you submit your manuscript to the iThenticate system, and it is by default checked against three databases of content. It is checked against web content - iThenticate indexes web pages in much the same way as a search engine, but with the added advantage that they keep an archive of web pages going back eight years. The manuscript is checked against the CrossCheck database, which contains the content from all of the participating CrossCheck publishers. And it’s also checked against a growing repository of online and offline content that iThenticate is gathering and indexing, including datbases from Gale and Ebsco, and sites such as PubMed and Arxiv.org. And as before, matches retrieved by comparison with these databases are pulled into a report for an editor to examine in more detail.
• #33: This is the screen that you see when you’ve uploaded one or more manuscripts to iThenticate. You can see the article titles on the left, author and date processed on the right. The Report column with the square buttons beneath tells you what percentage of text within the manuscript has been found to match text in other documents. Explain highlighting and thresholds.
• #34: And you get to this, which is the first of four different report manipulations available - this one is called the Similarity Report: Manuscript on left, matches on right from highest to lowest. Scroll up and down to compare. URLs (plus date) or citation depending on database. Links. Ability to exclude a match if you know it’s not relevant. Click on the left to see side by side report Show link to Document Viewer and touch on report view
• #35: Here you can see the two matching pieces of content side by side: Glance feature, full source view Importantly - entire article or piece of content on the right. Scroll up and down and have opportunity to establish the context. This is another benefit of using CrossCheck. We negotiated with iThenticate to allow users to see the complete article where there is matching text. If you use the commercial ithenticate product directly you only see snippets of text either side of the match. But we feel that it’s important for publishers and editors to be able to see more than that.
• #36: You might have spotted in the previous examples that the technology isn’t just looking for word for word matches. The way that it breaks the text down allows it to spot passages of text with word substitutions, so it is looking for similar as well as identical text. In this example you can see that some of the words have been very subtly substituted or moved but the technology still picks them up.
• #37: Some additional features that you should be aware of: it’s possible to exclude certain things to help reduce background noise. You can opt to exclude anything that’s included in quotation marks. You can exclude the reference section, and you can choose not to be shown any matches below a certain number of words - so perhaps strings of fewer than 25 words would not be shown. You should be aware that the first two of these features work on fairly hard and fast rules, so there need to be opening and closing quotation marks for a quote to be spotted and excluded, and the exclude bibliography feature relies on there being a recognisable section heading for the references to identify it at the end of a document. So some documents will slip through these filters.
• #38: Limitations: photos or images graphs and tables formulae Text only
• #39: Also copy and paste
• #40: Three obvious places where you might want to do plagiarism screening 1) On submission 2) At some defined point in the review and editorial process - - and obviously this is a massive over-simplification not in the least because this process varies widely form publisher to publisher 3) Just prior to acceptance And we have CrossCheck members taking each of these approaches - trend moving towards on submission.
• #41: Cost of processing manuscript goes up as you go through the process - if you find a problem later in the process it’s probably going to cost you more to deal with it than if you spot it on submission, where there’s the possibility of a simple rejection with advice on correcting errors. However, number of manuscripts goes down as you go through the process, so less direct cost of screening documents - there is a per-document fee for checking manuscripts, albeit a fairly small one starting at 75 cents per doc.
• #42: The main manuscript tracking systems have all integrated or are in the process of integrating iThenticate so that you can submit manuscripts directly as part of your existing workflow... ...Important to note that none of these systems are dictating when in the process you do the check - they have all left it very open and up to the publisher or user to decide at which point the checking should be done.
• #43: The progress of CrossCheck to date. Very comprehensive database - can see list of titles on our website. Just to emphasise really that they are publishers of all sizes, of multiple nationalities and representing many disciplines.
• #44: This graph shows how many documents have been run through iThenticate month on month.
• #45: I’d just like to remind you that if you are using CrossCheck you’re welcome to - in fact we would encourage you to let your authors and readers know that you are screening for originality. We have CrossCheck logos that you can download from our website to put on your journals and submission pages to provide some reassurance to honest authors and some warning to the less honest...
• #47: When you are fairly certain you’ve got a case of plagiarism on your hands, there are a number of places that offer advice on how to deal with it. The Office of Research integrity has advice on handling misconduct, although obviously it’s quite US-centric. The committee on publication ethics - COPE - has some very useful flowcharts that I would recommend taking a look at if you’re not already familiar with them. And the CSE white paper on Promoting Integrity in Scientific Journal Publishing has a section on Responding to Research Misconduct.
• #48: If you’ve had to deal with a case of plagiarism in one of your publications I don’t need to tell you about the repercussions that come with such situations and how important it is that cases are dealt with sensitively and carefully. You’ve potentially got legal costs, and the staff costs of following paper trails and contacting everyone involved, plus the costs of retractions or corrections. There’s also the cost to the reputation of not just an author or authors, but also departments, institutions, publications, publishers, etc
• #49: And that’s exactly why the CrossCheck service has been created - to help publishers screen for originality before it gets published and is spotted by a reader or indeed by a project such as the Deja Vu database. Once plagiarised content is published and out there it becomes quite a messy problem to fix. With CrossCheck we’re providing tools that will hopefully mean any problems are found and sorted out much earlier on in the process. And ultimately it’s about maintaining the integrity of your publications, and adding value through the publishing process.