Transparency and reproducibility in research
Download as PPTX, PDF0 likes207 views
The document discusses the importance of transparency and reproducibility in research, particularly regarding the requirement for data and methods to be clearly accessible to validate research findings. It highlights the challenges faced by science, including fraud and publication bias, and outlines various transparency policies from journals requiring authors to provide replicable data. The document also emphasizes the need for detailed protocols and data sharing to enhance the credibility and reliability of social science research.
![Transparency and
reproducibility in research
Louise Corti and Diarmuid McDonnell
UK Data Service
UK Data Archive, University of Essex
Copyright © [year] UK Data Service. Created by [Organisation], [Institution]
ESS Summer school: An introduction to using big data in
the social sciences
20-24 July 2020
University of Essex, Colchester](https://image.slidesharecdn.com/essbigdata20transparency-reproducibiltypresentation-201111122033/85/Transparency-and-reproducibility-in-research-1-320.jpg)
Transparency and reproducibility in research
- 1. Transparency and reproducibility in research Louise Corti and Diarmuid McDonnell UK Data Service UK Data Archive, University of Essex Copyright © [year] UK Data Service. Created by [Organisation], [Institution] ESS Summer school: An introduction to using big data in the social sciences 20-24 July 2020 University of Essex, Colchester
- 2. Issues • The reproducibility agenda • What does transparency mean for big data? • Where to publish data and code • Reproducing a research output
- 3. Science is Under Attack • Big challenges to science • Fraud and mistakes • Publication bias • Political pressures • Raw data availability
- 4. Protecting the Legitimacy of Research Transparency is now becoming an obligation Professional ethics guidance Funders require public access to data Journal guidelines Transparency in methods Data access statements Pre-analysis plans Robust data citation
- 6. Journal / Publisher Data Policies • Most science journals require a Data Availability Statement • Ans some have formalised data replication policies • E.g. Science, Nature, PLOS ONE “PLOS ONE will not consider a study if the conclusions depend solely on the analysis of proprietary data … the paper must include an analysis of public data that validates the conclusions so others can reproduce the analysis.” • e.g. BioMed Central open data statement • Digital Object Identifiers (DOIs) usually required
- 7. Social science: journal transparency • Replication policies exist for psychology, economics and political science journals The Journal of Development Economics Quarterly Journal of Economics Quarterly Journal of Political Science • Options: • Some require a preregistered hypothesis/design • Authors supply enough information that the exact analysis can be replicated - raw data (anonymized if necessary), data collection protocols, computer programs and scripts • Editors required to replicate the study • Journal may have own repository, e.g. Dataverse https://dataverse.harvard.edu/dataverse/ajps
- 8. The American Economic Review It is the policy of the American Economic Review to publish papers only if the data used in the analysis are clearly and precisely documented and are readily available to any researcher for purposes of replication. Authors of accepted papers that contain empirical work, simulations, or experimental work must provide to the Review, prior to publication, the data, programs, and other details of the computations sufficient to permit replication. These will be posted on the AER Web site. The Editor should be notified at the time of submission if the data used in a paper are proprietary or if, for some other reason, the requirements above cannot be met. As soon as possible after acceptance, authors are expected to send their data, programs, and sufficient details to permit replication, in electronic form, to the AER office. Data Availability Policy
- 9. The case of political science journals • Data Access and Research Transparency (DA-RT): Joint Statement by Political Science Journal Editors “Journal editors commit their respective journals to the principles of data access and research transparency, and to implementing policies requiring authors to make as accessible as possible the empirical foundation and logic of inquiry of evidence-based research” http://media.wix.com/ugd/fa8393_da017d3fed824cf587932534c860ea25.pdf • 26 journals signed up by 2016 • Some established rules for implementation
- 10. DA-RT initiative 1. Data Transparency - whereby researchers publicize the data they use as evidence. Includes primary data collected by researchers and secondary sources they may use in their work 2. Analytic Transparency - where researchers publicize how they measure, code, interpret, and analyze that data 3. Process (or Production) Transparency - where researchers explain how they came to choose their research design, and why they used particular sets of data, theories, and methods
- 11. Good research relies on visibility of process Data Sharing Research Transparency Data used to support an evidence-based claim Data made available for secondary analysis
- 12. Replication Data Policy Guidelines for Accepted Articles: The manuscript will not be published unless the first footnote explicitly states where the data used in the study can be obtained for purposes of replication and any sources that funded the research. All replication files must be stored on the AJPS Data Archive on Dataverse.
- 13. How to prepare a replication file: https://ajpsblogging.files.wordpress.com/2015/03/ajps- guide-for-replic-materials-1-0.pdf
- 14. Requirements for replication files Readme File - names of all files in the study plus description Analysis Dataset(s) – each Study must include one or more files containing the data required to reproduce all tables, figures, and other analytic results reported in the article Code book Information and software commands to reconstruct an analysis dataset from another source Syntax for new derived variables Access instructions Code for the analysis
- 15. Data intensive research? • In analyses based upon highly data-intensive procedures not necessary to provide full contents of each replicated dataset – instead full set of relevant results and software command files Stata .do files, R command scripts, text files Comment statements should be used extensively Links to data sources used (even if being continually updated) Software version used e.g. R version 3.1.0
- 16. Example: climate models • Models based on mathematical representations of the climate system • Expressed as code or syntax, run on powerful computers Confidence in models: • Model that are based on established physical laws • mass, energy and momentum • wealth of natural environmental observations • The ability of models to simulate important aspects of the current climate • Evaluation undertaken by organised multi-model comparisons; weather forecasts
- 17. How Reliable Are the Models Used to Make Projections of Future Climate Change?
- 18. Questions to ask when trying to attempt replication • Where might you get the data from? URL or a DOI? • Is there any access issues with reusing the data? • How many separate data files might you need? • Can you get raw or derived/processed data? • Can you link the data types easily? • Which variables are needed? Can these be regrouped or recoded? • Which variables were omitted from the models? Do you need them in the dataset • What about weights for data – item non-response? • Is the syntax/code clear?
- 19. Data, replication data and evidence • Research data underpinning a paper still often not available - even on on request. Sociology lagging behind • Formally publishing a replication dataset provides an opportunity to be reproducible! • Publishing the whole dataset from a research study brings added advantages for the creator– longevity, robust citation and proper attribution
- 22. Data papers