OSFair2017 Training | Increasing Research Transparency using the Open Science Framework
1 like378 views
The document discusses the importance of transparency and openness in scientific research through the Open Science Framework (OSF) to enhance reproducibility and integrity. It highlights challenges in data sharing and provides recommendations for improving research workflows, such as preregistration and open access. The OSF supports various features that promote collaboration and the sharing of data and research methods to facilitate metascience.
OSFair2017 Training | Increasing Research Transparency using the Open Science Framework
- 1. cos.io | osf.io Jennifer Freeman Smith, PhD Transparency and Openness Training Coordinator Increasing Research Transparency Using the Open Science Framework
- 2. Improving Openness, Integrity, and Reproducibility of Scientific Research MISSION
- 4. Everything we do is free and open. COS Strategic Plan: https://osf.io/x2w9h
- 6. How does transparency improve science?
- 8. SOURCES OF ISSUES IN REPRODUCIBILITY • Methodological, statistical, and reporting practices • Structural and organizational practices • Rarely, intentional scientific misconduct
- 9. WHAT IS REPRODUCIBILITY? Computation Reproducibility: If we took your data and code/analysis scripts and reran it, we can reproduce the numbers/graphs in your paper Methods Reproducibility: We have enough information to rerun the experiment or survey the way it was originally conducted Results Reproducibility/Replicability: We use your exact methods and analyses, but collect new data, and we get the same statistical conclusion
- 10. If we seek to facilitate reproducibility, replicability, extension, and reuse…
- 11. We need to move beyond description of outcomes to description of process or, better, sharing actual process.
- 12. Open Access Open Data Open Materials Open Data Cleaning Scripts Open … Open Workflow
- 13. OPEN WORKFLOW • Increases process transparency • Increases accountability • Facilitates reproducibility • Facilitates metascience • Fosters collaboration • Fosters inclusivity • Fosters innovation • Protects against lock-in: Open + Accessible
- 14. It takes some effort to organize your research to be reproducible…the principal beneficiary is generally the author herself. Jon Claerbout Making Scientific Contributions Reproducible http://sepwww.stanford.edu/oldsep/matt/join/redoc/web/iris.html “ ”
- 16. OPEN SCIENCE FRAMEWORK http://osf.io Free, open source scientific commons
- 18. Put data, materials, and code on the OSF
- 22. Why open your workflow?
- 23. WHY OPEN YOUR WORKFLOW? • Improve reproducibility and replicability • Increased efficiency • Increases reuse and extension of knowledge • Public data can be combined with private data • Can influence scientists, entrepreneurs, policymakers, citizens
- 24. OPEN DATA CHALLENGES • How do we make our data accessible, understandable, reusable? • Which repository should I choose? • Who owns the data? Do I have a copyright on the raw data I collected? • If I reuse data from someone else, do I have to offer them co-authorship? • How should privacy issues be addressed?
- 25. SHARING IS A CONTINUUM • Data underlying just results reported in a paper • Data underlying publication + information about other variables collected • Data underlying publication + embargo on full dataset • All data collected for that study
- 27. GUIDs make sharing simple Arnold BF, van der Laan MJ, Hubbard AE, Steel C, Kubofcik J, Hamlin KL, et al. (2017) https://doi.org/10.1371/journal.pntd.0005616
- 28. File downloads Forks See the Impact
- 29. NEXT STEPS 1.Build a test project on the OSF 2.Document from the beginning - or even right now 3.Talk to your collaborators – What is our data management plan? – What/when will we share?
- 30. Jennifer Freeman Smith, PhD Center for Open Science Charlottesville, VA, USA jennifer@cos.io @jfsmith434 Find this presentation at https://osf.io/ncdpa/ QUESTIONS AND COMMENTS
- 33. POSITIVE RESULTS BY DISCIPLINE Fanelli D (2010) “Positive” Results Increase Down the Hierarchy of the Sciences. PLOS ONE 5(4): e10068. doi:10.1371/journal.pone.001 0068 http://journals.plos.org/ploso ne/article?id=10.1371/journal .pone.0010068
- 34. RESEARCHER DEGREES OF FREEDOM All data processing and analytical choices made after seeing and interacting with your data • Should I collect more data? • Which observations should I exclude? • Which conditions should I compare? • What should be my main DV? • Should I look for an interaction effect?
- 35. FALSE POSITIVE INFLATION Simmons, Nelson, & Simonsohn (2011)
- 36. EXPLORATORY VS. CONFIRMATORY ANALYSES Exploratory • Interested in exploring possible patterns/relationships in data to develop hypotheses Confirmatory • Have a specific hypothesis you want to test Preregistered analysis plans clarify which results are exploratory and which are confirmatory
- 37. Other ways the OSF supports open workflows
- 38. Preprints decouple publication and evaluation, allowing for the rapid dissemination of content.
- 39. OSF Preprints
- 41. Free, open (meta)dataset of research activity across the research lifecycle ~40M records from ~163 sources http://share.osf.io
- 43. PRE-REGISTRATION Documenting your research plan in a read-only public repository before you conduct the study. Pre-registration helps reduce the “file drawer effect” by increasing discoverability of unpublished studies.
- 44. PRE-REGISTRATION Benefits of pre-registering your study depend on how much information you include. At a minimum a preregistration should include the “what” of the study: • Research question • Population and sample size • General design • Variables you’ll be collecting, or dataset you’ll be using
- 45. PRE-ANALYSIS PLAN Details the analyses planned for hypothesis testing: Sample size Data processing and cleaning procedures Exclusion criteria Statistical analyses Including a pre-analysis plan in your pre-registration helps improve study accuracy and replicability by guarding against unintended false positive inflation.