John Rolfe - Editor Case Study: Implementing a Journal Data Policy
1 like241 views
The document discusses the editor's concern about authors being unable to replicate or verify the data and results in their published articles. It outlines some of the potential issues this can cause, such as questions about data quality, bias, and inability to conduct meta-analyses using the original data. The Australian Journal of Agricultural and Resource Economics addresses this by having a clear policy requiring authors to submit major datasets and code used in their analysis for replication purposes. Exceptions can be made for confidential data, but authors must document their data sources and processing methods thoroughly. The policy aims to encourage replicable research while balancing authors' ownership over proprietary data. Compliance currently relies on author cooperation rather than enforcement.
John Rolfe - Editor Case Study: Implementing a Journal Data Policy
- 1. Data Management Professor John Rolfe Editor-in-Chief, Australian Journal of Agricultural and Resource Economics
- 2. What does an editor worry about? • Not enough articles • Too many articles • Can’t find reviewers • Reviews are limited/conflicting • Associate Editors are slow/don’t provide good advice • Nobody understands the maths/concepts • There won’t be enough for the next issue
- 3. What are editor nightmares? • The impact factor will plummet while I am in charge • I reject the author/article that goes on win the Nobel prize • I publish material plagiarised from three other journals • The data / analysis in an article is shonky • Results get questioned and authors can’t find the data or replicate their analysis
- 4. The data management nightmare • Insomniac editors worry about things like: – Authors will be quizzed about findings and asked to show data/replicate results • Not all authors are organised/multiple model runs/ data managed by previous research assistant • Authors can be snarky about requests to verify – Authors will be suspected of bias • Increasing pressure to publish / cutting corners – Authors invited to supply data retrospectively • Past data often needed for meta analysis, retrospective fitting to new statistical techniques • Not all authors are cooperative
- 5. Perhaps the editor should go back to sleep? • Isn’t this really just a problem for the author? • Four key reasons why this is important – Quality issue • Need to be able to substantiate results – Copyright issue • Copyright held by society/journal/publisher, not author – Academic standards • Results should be replicable by other scholars – Potential reuse • Source data often requested for meta-analysis
- 6. Soothing editor nerves ….. • Policies on data storage and documentation – Make it explicit to authors about their responsibilities • Many authors possessive about data – Provide a mechanism to hold copies of data and syntax for analysis by Journal • In reserve – no automatic access • In appendices to article • Open access
- 7. The Australian Journal of Agricultural and Resource Economics • Has a policy on data storage and access • Clearly identified in Guidelines to Authors – Second section on Editorial and Content Consideration • Aims and Scope • Editorial Review and Acceptance • Data Storage and Documentation
- 8. Data Storage and Documentation • Analyses published in AJARE are expected to be replicable by other scholars. To this end, authors are expected to document their data sources, models, and estimation procedures as thoroughly as possible, and to make the data used available to others for replication purposes. Our policy is to encourage authors to submit major databases and code used to conduct the data analyses in their paper. Please read the Data and Code Availability Policy. When submitting your paper, follow the instructions on ScholarOne to upload these files. If an exception to this rule is desired, for example due to confidentiality of data, this should be noted in the article, and the reasons for seeking an exception should be explicitly noted in the cover letter at time of submission.
- 9. Key messages • Make authors aware of responsibilities • Detail what has to be done • Compliance is not mandatory Analyses published in AJARE are expected to be replicable by other scholars. To this end, authors are expected to document their data sources, models, and estimation procedures as thoroughly as possible, and to make the data used available to others for replication purposes. Our policy is to encourage authors to submit major databases and code used to conduct the data analyses in their paper. Please read the Data and Code Availability Policy. When submitting your paper, follow the instructions on ScholarOne to upload these files. If an exception to this rule is desired, for example due to confidentiality of data, this should be noted in the article, and the reasons for seeking an exception should be explicitly noted in the cover letter at time of submission.
- 10. Data and code availability policy • 2 page policy, available from guidelines – Preamble • Aligns policy with other journals such as the American Economic Review, the Journal of Political Economy and the Economic Record. – Objectives • encourages authors of a submitted paper to submit major datasets and code used to conduct the data analysis in their paper • building up a repository of data so that authors may increase the number of citations of their work, thus enhancing the reputation of both their work and the journal • If a reviewer requests access to data or code, normally these should be made available for the reviewing process
- 11. Data and code availability policy 2 • Confidential datasets • We do not expect confidential (or partly restricted) datasets to be provided. If uncertain, authors need to check with their data source whether or not it is legal to make the data available on the internet. If not, a data appendix or files containing codes and instructions on data cleaning and manipulation may still be uploaded through ScholarOne Manuscripts to enable interested investigators to replicate the analysis after obtaining the data for themselves. • Author-generated datasets • We recognise that, in some instances (such as once-off surveys, interviews or experiments), researchers have invested significant effort in collecting proprietary data and it may be detrimental to their research program to publish this data at the present time. In these cases the features of the data should be welldescribed within the paper. The authors may upload a file describing their data source with instructions on data cleaning and manipulation so that in future, when the data become available, replication may be undertaken.
- 12. Data and code availability policy 3 • Links in the accepted paper • The online published article will include a link to the online data and code material. A footnote in the printed version of the paper should indicate that the data is available with the online version with a link to the online journal homepage (http://wileyonlinelibrary.com/journal/ajare). • Copyright • With regard to copyright, the terms of the signed copyright form that authors of accepted papers need to provide apply to supporting information as well as the article itself. Among these terms is a statement that authors have obtained all necessary permissions to use any of the material presented, which includes datasets.
- 13. How well is it working ? • Data uploads not currently enforced • Authors tend to argue data is confidential • Reminders could be more explicit • Unclear if stricter enforcement would discourage submissions – Of good quality articles (a concern) – Of poor quality articles (a bonus)
- 14. Challenges looking forward • QA likely to become bigger issue for journals • Holding data and syntax is key step • Level of access to be given • In reserve – Achieves QA but not academic sharing • In appendices to article – achieves QA and sharing but concerns about ownership • Open access – Larger concerns about loss of ownership • Should data lodgement be mandatory?