Reward, reproducibility and recognition in research - the case for going Open

Editor's Notes

• #9: The citation distributions are so skewed that up to 75% of the articles in any given journal had lower citation counts than the journal's average number.  
• #16: Unsurprisingly this position raised a small storm of protest (an example is here). This was so sustained that four days later a clarification was issued, which did not include the word ‘parasites’.
• #25: The nine circles of scientific hell (with apologies to Dante and xkcd)‏
• #28: Introduction The recent release of data from the largest clinical trial of psychotherapy treatments for chronic fatigue syndrome (CFS), the ‘PACE-Trial’, has triggered a perfect storm of patient anger and professional defensiveness. The data were only released after a protracted freedom of information case brought by a patient with CFS. A tribunal ordered the lead author’s institution to release their data. Upon release, re-analysis showed that the levels of improvement and recovery observed in the released data were much lower than the levels reported in the published report (White et al., 2011a) and other related publications. The released data showed that the effectiveness of cognitive behavioural therapy (CBT) and graded exercise therapy (GET), in comparison to standard medical care (SMC) and adaptive pacing therapy (APT), fell by almost two-thirds. Patient groups and independent experts have remarked that without data access, the medical establishment would have been left to accept the outcomes from the PACE-Trial, as robust evidence that CBT and GET are effective treatments for CFS. Instead, patients are calling for the wider scientific community to investigate their claim that the PACE-Trial authors overstated the benefits of CBT and GET. This editorial considers the ramifications of this unfolding story for patients with CFS, and its impact on the science of clinical trials of psycho-behavioural therapies.    
• #29: More than 70% of researchers have tried and failed to reproduce another scientist's experiments, and more than half have failed to reproduce their own experiments. Those are some of the telling figures that emerged from Nature's survey of 1,576 researchers who took a brief online questionnaire on reproducibility in research. The data reveal sometimes-contradictory attitudes towards reproducibility. Although 52% of those surveyed agree that there is a significant 'crisis' of reproducibility, less than 31% think that failure to reproduce published results means that the result is probably wrong, and most say that they still trust the published literature.
• #31: Correlation between impact factor and retraction index. The 2010 journal impact factor (37) is plotted against the retraction index as a measure of the frequency of retracted articles from 2001 to 2010 (see text for details). Journals analyzed were Cell, EMBO Journal, FEMS Microbiology Letters, Infection and Immunity, Journal of Bacteriology, Journal of Biological Chemistry, Journal of Experimental Medicine, Journal of Immunology, Journal of Infectious Diseases, Journal of Virology, Lancet, Microbial Pathogenesis, Molecular Microbiology, Nature, New England Journal of Medicine, PNAS, and Science.
• #32: Worse in some areas than others Replication crisis in psychology and biomedical science Sociology, economics, climate science also vulnerable [according to Smaldino] Poor research design and data analysis encourage false-positive findings. Such poor methods persist despite perennial calls for improvement, suggesting that they result from something more than just misunderstanding. The persistence of poor methods results partly from incentives that favour them, leading to the natural selection of bad science. This dynamic requires no conscious strategizing—no deliberate cheating nor loafing— by scientists, only that publication is a principal factor for career advancement. Some normative methods of analysis have almost certainly been selected to further publication instead of discovery. In order to improve the culture of science, a shift must be made away from correcting misunderstandings and towards rewarding understanding. We support this argument with empirical evidence and computational modelling. We first present a 60-year meta-analysis of statistical power in the behavioural sciences and show that power has not improved despite repeated demonstrations of the necessity of increasing power. To demonstrate the logical consequences of structural incentives, we then present a dynamic model of scientific communities in which competing laboratories investigate novel or previously published hypotheses using culturally transmitted research methods. As in the real world, successful labs produce more ‘progeny,’ such that their methods are more often copied and their students are more likely to start labs of their own. Selection for high output leads to poorer methods and increasingly high false discovery rates. We additionally show that replication slows but does not stop the process of methodological deterioration. Improving the quality of research requires change at the institutional level.
• #33: When researchers were asked how the challenges in research have influenced their careers, 65% said they had considered quitting research, and 15% that they had actually quit. Around one-third felt that they had been judged solely on the number of papers they had published, and another one-third said that they had published a paper they were not proud of. And 16% said they had cut corners in research. (Readers could choose more than one answer.)   When asked to choose the biggest challenge facing early-career scientists, 44% of some 12,000 respondents overwhelmingly picked ‘the fight for funding’. This result aligns closely with the answers of the 3,000-plus people who responded to Nature’s 2016 salary survey, just under half of whom ranked ‘competition for funding’ as the biggest challenge to their career progression. The next biggest challenges identified in the reader poll, ‘lack of work–life balance’ and ‘progression judged too heavily on publication record’ received just under one-fifth of the total vote each.
• #36: https://www.flickr.com/photos/veggiesosage/8129414104/in/photolist-donnA9-6sfvqZ-8rWsjR-roCJtM-dRMUe7-5JE3DX-5bY9jD-5gCtmQ-Gzywtf-fANgPy-6LGtL-6AdqKh-8qZicQ-dJdJE8-48RRrG-5fe5e3-8jCedo-8JFhMX-dRMUrS-7zHQEm-axsb6-aRDZtV-4F6gVx-cyfdM7-9hY5Qz-61tJsZ-2iJ7VY-5frH4J-axsdg-5q8Lko-h942Yo-6YomAZ-2EygiV-tEsAP-3UN759-dCCU93-6YW14K-apE2wJ-4SuG87-3vppp-d95MMw-845Gx-6LGte-8uqdXH-jYQBF6-dRMTZu-4xMeSv-q3xvwc-4mb63q-aqdnhZ
• #44: Wellcome Trust   Van den Eynden, Veerle et al. (2016) Towards Open Research: practices, experiences, barriers and opportunities. Wellcome Trust. https://dx.doi.org/10.6084/m9.figshare.4055448 investigates researchers’ attitudes and behaviour towards open research, examining the sharing and reuse of research data, code, and open access publications, in order to identify practical actions the Wellcome Trust can take to remove or mitigate barriers and maximise the opportunities for practising open science. 70% Wellcome Trust authors publishing OA Half are making data available With two-fifth of researchers generating code in their research, less than half of them also make it available for access and use by others.
• #46: FORCE11 (Future Of Research Communications and eScholarship) is a community of scholars, librarians, archivists, publishers and research funders that has arisen organically to help facilitate the change toward improved knowledge creation and sharing. Individually and collectively, we aim to bring about a change in modern scholarly communications through the effective use of information technology. SPARC (the Scholarly Publishing and Academic Resources Coalition) works to enable the open sharing of research outputs and educational materials in order to democratize access to knowledge, accelerate discovery, and increase the return on our investment in research and education.  OpenCon is aimed at young researchers and is a platform for the next generation to learn about Open Access, Open Education, and Open Data, develop critical skills, and catalyze action toward a more open system for sharing the world’s information Enabling Open Scholarship is an organisation for universities and research institutions worldwide. The organisation is both an information service and a forum for raising and discussing issues around the mission of modern universities and research institutions, particularly with regard to the creation, dissemination and preservation of research findings. The Centre for Open Science began with a single project and is now a team of 50 people supporting a much larger collection
 of communities that are producing tools and services to align scientific practices with scientific values.  
• #47: FORCE11 (Future Of Research Communications and eScholarship) is a community of scholars, librarians, archivists, publishers and research funders that has arisen organically to help facilitate the change toward improved knowledge creation and sharing. Individually and collectively, we aim to bring about a change in modern scholarly communications through the effective use of information technology. SPARC (the Scholarly Publishing and Academic Resources Coalition) works to enable the open sharing of research outputs and educational materials in order to democratize access to knowledge, accelerate discovery, and increase the return on our investment in research and education.  OpenCon is aimed at young researchers and is a platform for the next generation to learn about Open Access, Open Education, and Open Data, develop critical skills, and catalyze action toward a more open system for sharing the world’s information Enabling Open Scholarship is an organisation for universities and research institutions worldwide. The organisation is both an information service and a forum for raising and discussing issues around the mission of modern universities and research institutions, particularly with regard to the creation, dissemination and preservation of research findings. The Centre for Open Science began with a single project and is now a team of 50 people supporting a much larger collection
 of communities that are producing tools and services to align scientific practices with scientific values.  
• #48: FORCE11 (Future Of Research Communications and eScholarship) is a community of scholars, librarians, archivists, publishers and research funders that has arisen organically to help facilitate the change toward improved knowledge creation and sharing. Individually and collectively, we aim to bring about a change in modern scholarly communications through the effective use of information technology. SPARC (the Scholarly Publishing and Academic Resources Coalition) works to enable the open sharing of research outputs and educational materials in order to democratize access to knowledge, accelerate discovery, and increase the return on our investment in research and education.  OpenCon is aimed at young researchers and is a platform for the next generation to learn about Open Access, Open Education, and Open Data, develop critical skills, and catalyze action toward a more open system for sharing the world’s information Enabling Open Scholarship is an organisation for universities and research institutions worldwide. The organisation is both an information service and a forum for raising and discussing issues around the mission of modern universities and research institutions, particularly with regard to the creation, dissemination and preservation of research findings. The Centre for Open Science began with a single project and is now a team of 50 people supporting a much larger collection
 of communities that are producing tools and services to align scientific practices with scientific values.  
• #49: FORCE11 (Future Of Research Communications and eScholarship) is a community of scholars, librarians, archivists, publishers and research funders that has arisen organically to help facilitate the change toward improved knowledge creation and sharing. Individually and collectively, we aim to bring about a change in modern scholarly communications through the effective use of information technology. SPARC (the Scholarly Publishing and Academic Resources Coalition) works to enable the open sharing of research outputs and educational materials in order to democratize access to knowledge, accelerate discovery, and increase the return on our investment in research and education.  OpenCon is aimed at young researchers and is a platform for the next generation to learn about Open Access, Open Education, and Open Data, develop critical skills, and catalyze action toward a more open system for sharing the world’s information Enabling Open Scholarship is an organisation for universities and research institutions worldwide. The organisation is both an information service and a forum for raising and discussing issues around the mission of modern universities and research institutions, particularly with regard to the creation, dissemination and preservation of research findings. The Centre for Open Science began with a single project and is now a team of 50 people supporting a much larger collection
 of communities that are producing tools and services to align scientific practices with scientific values.  
• #56: Publishes all outputs of the research cycle, including: project proposals, data, methods, workflows, software, project reports and research articles together on a single collaborative platform, with the most transparent, open and public peer-review process. http://riojournal.com/
• #57: Publishes all outputs of the research cycle, including: project proposals, data, methods, workflows, software, project reports and research articles together on a single collaborative platform, with the most transparent, open and public peer-review process. http://riojournal.com/
• #58: Publishes all outputs of the research cycle, including: project proposals, data, methods, workflows, software, project reports and research articles together on a single collaborative platform, with the most transparent, open and public peer-review process. http://riojournal.com/
• #59: F1000Research uses an author-led process, publishing all scientific research within a few days. Open, invited peer review of articles is conducted after publication, focusing on scientific soundness rather than novelty or impact. https://f1000research.com/
• #60: PLOS ONE new collection to highlight studies that present inconclusive, null findings or demonstrate failed replications of other published work.  http://blogs.plos.org/everyone/2015/02/25/positively-negative-new-plos-one-collection-focusing-negative-null-inconclusive-results/
• #61: PLOS ONE new collection to highlight studies that present inconclusive, null findings or demonstrate failed replications of other published work.  http://blogs.plos.org/everyone/2015/02/25/positively-negative-new-plos-one-collection-focusing-negative-null-inconclusive-results/
• #70: NIH: “We want to know if interim research products can incease the rigor XXXX