Towards reproducibility and maximally-open data
- 1. Towards reproducible research and maximally-open data Pablo Bernabeu OSCG Open Scholarship Prize Competition, 14th May 2021
- 2. Psycholinguistics > Conceptual processing Reanalysis of data from Hutchison et al. (2013) How does the brain process the meaning of words? Statistical regularities in language as well as perceptual, motor and emotional information. How does this process vary in different contexts and across different people? 2
- 3. Open information at each stage of research Design: preregistration of theoretical background and methodological protocol. Development: procedural issues (corrections, errors, other changes) that bear on the final materials. Completion: data collection software, raw data, processed data, final data set, analysis code and results. S h a r e d i n m y P h D S h a r e d i n m y M A 3 Application of open science My research Community: open-source code workshops and software
- 4. Open data Bernabeu et al. (2017), Bernabeu (2018) • All materials from the completion stage: experiment administration software, raw data, processed data, final data sets, analysis code and results. • Development stage proceedings reported. • Readme files describing the data sets and linking to resources such as data dashboards. 4
- 5. Maximally-open data Bernabeu et al. (2017), Bernabeu (2018) • R-based web applications open to scientists and the general public. • Easy visualization of the variance and inspection of procedural aspects such as trimming, adjustments, changes. Quicker usage of the data (see blog post). 5
- 6. Preregistration, power analysis and open data Chen et al. (2018) • Prereg.: https://psyarxiv.com/t2pjv/ • Video demonstration of the lab procedure: https://osf.io/h36wr/ • Data: https://osf.io/waf48/ 6
- 7. Preregistration and open data Bernabeu et al. (2021) • Specification of the theoretical background and the methodological protocol for a forthcoming study. • Integration of FAIR data from several, large-sample studies • Large, secondary data = valuable alternative to small, noisy samples (see Loken & Gelman, 2017). 7
- 8. Power analysis: How many participants required? For next study, the preregistration will include a power analysis based on two large- sample pilots (combined, FAIR data sets), using power curves based on Monte Carlo simulations (simr R package). Preliminary curves below (pending more simulations for a greater accuracy). Y axis = power for a certain effect; X axis = 1 to 312 participants. 8
- 9. R workshops Workshops and presentations on data visualisation and analysis in R since 2018, mostly in the context of a fellowship from the Software Sustainability Institute. • http://pablobernabeu.github.io/#workshops • https://github.com/pablobernabeu/Data-is-present Blogging Several blog posts on psycholinguistic research, open science and statistics. http://pablobernabeu.github.io/blog 9
- 10. More open-source web applications for research and teaching Experimental data simulation WebVTT caption transcription https://github.com/pablobernabeu/Experimental-data-simulation https://github.com/pablobernabeu/VTT-Transcription-App 10
- 11. Concluding thoughts • Attainable for early-career researcher: individual and community applications Some win-win benefits • Open science is a framework, rather than an all-or-nothing result. Design stage: my preregistrations could be even more precise (see Bakker et al., 2020). Development stage: my procedures could be even more open. Completion stage: my materials could be even more easily accessible. • Let’s not eschew studies that report adjustments or errors. If errare humanum est, by definition (see Bakker et al., 2020), how many spotless studies should there naturally be in journals? • Reward structures (e.g., promotion) still often prioritise number of publications. 11
- 12. References Bakker, M., Veldkamp, C. L., van Assen, M. A., Crompvoets, E. A., Ong, H. H., Nosek, B. A., Soderberg, C. K., Mellor, D., & Wicherts, J. M. (2020). Ensuring the quality and specificity of preregistrations. PLoS Biology, 18(12), e3000937. https://doi.org/10.1371/journal.pbio.3000937 Bernabeu, P. (2018). Dutch modality exclusivity norms for 336 properties and 411 concepts. PsyArXiv. https://psyarxiv.com/s2c5h Bernabeu, P., Lynott, D., & Connell, L. (2021). Preregistration: The interplay between linguistic and embodied systems in conceptual processing. OSF. https://osf.io/ftydw/ Bernabeu, P., Willems, R. M., & Louwerse, M. M. (2017). Modality switch effects emerge early and increase throughout conceptual processing: evidence from ERPs. In G. Gunzelmann, A. Howes, T. Tenbrink, & E. J. Davelaar (Eds.), Proceedings of the 39th Annual Conference of the Cognitive Science Society (pp. 1629-1634). Austin, TX: Cognitive Science Society. https://mindmodeling.org/cogsci2017/papers/0318 Chen, S., Szabelska, A., Chartier, C. R., Kekecs, Z., Lynott, D., Bernabeu, P., … Schmidt, K. (2018). Investigating object orientation effects across 14 languages. PsyArXiv. https://doi.org/10.31234/osf.io/t2pjv/ Hutchison, K. A., Balota, D. A., Neely, J. H., Cortese, M. J., Cohen-Shikora, E. R., Tse, C.-S., Yap, M. J., Bengson, J. J., Niemeyer, D., & Buchanan, E. (2013). The semantic priming project. Behavior Research Methods, 45, 1099–1114. https://doi.org/10.3758/s13428-012-0304-z Loken, E., & Gelman, A. (2017). Measurement error and the replication crisis. Science, 355(6325), 584-585. https://doi.org/10.1126/science.aal3618 12
- 13. Thank you to OSCG, the sponsors and the audience! Also, thank you to my mentors and everyone else who has contributed to my research. 13