Pfeiffer "Enhancing Open Science: Assessing Tools & Charting Progress"
0 likes175 views
The document discusses enhancing open science through various tools and initiatives that aim to shift research culture towards greater transparency, rigor, and collaboration. It highlights the importance of addressing current challenges, such as publication bias and perception gaps within the research community, by implementing rewards and training for desired behaviors. Central to the approach is the use of open scholarship practices, such as registered reports and open-source project management tools, to improve research quality and foster a supportive environment for innovative practices.
Pfeiffer "Enhancing Open Science: Assessing Tools & Charting Progress"
- 1. Enhancing Open Science: Assessing Tools & Charting Progress
- 2. Systems Approach to Research Culture Change COS Theory of Change Make it Required Make it Normative Make it Possible Make it Rewarding Make it Easy
- 3. Year of Open Science Catalyzing Change “The principle and practice of making research products and processes available to all, while respecting diverse cultures, maintaining security and privacy, and fostering collaborations, reproducibility, and equity.” Policy Requirements Grants/Funding TOPS Training & Education Repositories, PIDs,
- 4. Initiatives to Improve Addressing current challenges COS initiatives to bring Solutions: ● Open Scholarship Survey ○ Understand community perceptions and attitudes ● Badges ○ Reward desired behaviors ● Registered reports & Preregistration ○ Improve quality and rigor ○ Remove publication bias ● Open Research Infrastructure ○ NIH GREI ○ NSF POSE No one else is doing it Can I trust the results? No reward for doing it I don’t know how I don’t have the tools
- 5. Open Scholarship Survey Addressing current challenges ● Which open scholarship practices in a community are most ripe for advancement by identifying attitude/behavior gaps, where adoption of a practice lags behind the community’s support for it. ● Attitude/perception gaps, where researchers may be underestimating the support for a practice in their community. ● Key blockers to open scholarship adoption, which could be addressed by more targeted training, better incentives, or more awareness of a community’s norms. No one else is doing it Innova tion Early Adop tion Mainstream Standard Diffusion of Innovations; Rogers, 1963
- 6. Attitude Perception Perception gap in open scholarship
- 7. Attitude Action Action gap in open scholarship
- 8. Attitude Perception Perception gap (Pluralistic ignorance) Perception gap offers opportunities for change Opportunity for intervention
- 9. Incentives for individual success are focused on getting it published, not getting it right.” Nosek, Spies, & Motyl (2012)
- 10. A dysfunctional reward system Incentives for novel, positive, tidy outcomes Selective reporting Questionable research practices Minimal transparency or sharing No replication of studies Low credibility published literature No self-correction process Research waste, slow progress poor return on investment
- 11. Solutions
- 12. Improving the reward system Incentives for rigor and high quality methodology Complete reporting Transparent research practices More transparency or sharing More replication of studies More credible published literature Self-correction process Less waste, more progress more return on investment
- 13. Signals of Trust Improve quality and rigor Badges ● Signal good practices ● Set new community norms Registered Reports & Preregistration ● Remove publication bias ● Improve research quality ● Peer review prior to data collection ● Over 200 journals use RR format ● Null findings get published Can I trust the results? Kidwell et al, 2016 http://cos.io/rr, Committee Chair: Chris Chambers. Nosek & Lakens, 2014 No reward for doing it
- 14. Registered Reports associated with higher quality Soderberg et al., 2021
- 15. Adoption of Registered Reports Chambers & Tzavella, 2021
- 16. Changing culture with Registered Reports Peer review Authors Journals Funders
- 17. Perceived rewards for sharing data, materials, and code
- 18. Make it Possible OSF is a free, open source project management and collaboration tool that supports researchers throughout the entire research lifecycle - from planning, to conducting, to reporting, and discovering research. I don’t have the tools
- 19. Make it Possible OSF is a free, open source project management and collaboration tool that supports researchers throughout the entire research lifecycle - from planning, to conducting, to reporting, and discovering research. data data metadata I don’t know how badges metadata data metadata
- 20. Get in Touch! Want to learn more? Nici@cos.io Cos.io OSF.io 0000-0001-8335-6018