Is the current measure of excellence perverting Science? A Data deluge is coming, it is time to act
1 like44 views
The document discusses the challenges surrounding scientific reproducibility, highlighting that a significant percentage of researchers struggle to replicate experiments, both others' and their own. It emphasizes the need for better practices, collaboration, and the integration of data and methodologies in addressing the reproducibility crisis, particularly in the context of upcoming megascience projects. The paper calls for a shift in the current scientific culture to prioritize reproducibility alongside productivity in the research process.
![METRICS
... “Science is being
killed by numerical
ranking,”[...] Ranking
systems lures scientists
into pursuing high
rankings first and good
science second
Productivity seems to prevail
over Discovery](https://image.slidesharecdn.com/lourdesverdesmontenegromotivationfinal-190408202640/85/Is-the-current-measure-of-excellence-perverting-Science-A-Data-deluge-is-coming-it-is-time-to-act-17-320.jpg)
![METRICS
... “Science is being
killed by numerical
ranking,”[...] Ranking
systems lures scientists
into pursuing high
rankings first and good
science second
Reproducibility
crisis](https://image.slidesharecdn.com/lourdesverdesmontenegromotivationfinal-190408202640/85/Is-the-current-measure-of-excellence-perverting-Science-A-Data-deluge-is-coming-it-is-time-to-act-18-320.jpg)
Is the current measure of excellence perverting Science? A Data deluge is coming, it is time to act
- 1. Is the current measure of excellence perverting Science? A Data deluge is coming, it is time to act Lourdes Verdes-Montenegro Instituto de Astrofísica de Andalucía (IAA-CSIC) Session Organiser: William Garnier (SKAO) (Submitter and Manager) Theme #3 Science policy and transformation of research practice
- 2. ANY PROBLEM WITH THE SCIENTIFIC METHOD?? Scientific Reproducibility is a fundamental principle of the Scientific Method, a process established in the 17th century that marked the beginning of modern science and laid the foundations for the Philosophy of Science. We all agree, “reproducibility is great!”, right? Then… What is the problem?
- 3. ANY PROBLEM WITH THE SCIENTIFIC METHOD?? Scientific Reproducibility is a fundamental principle of the Scientific Method, a process established in the 17th century that marked the beginning of modern science and laid the foundations for the Philosophy of Science. We all agree, “reproducibility is great!”, right? Then… What is the problem? Paradoxically: • part of the scientific community claims that reproducibility is already achieved (because they have a section describing their methods in their papers, or because they share their data) • the remainder mostly consider it a utopy
- 4. Questionnaire on reproducibility (1500 scientists) • 70% of researchers have tried and failed to reproduce another scientist's experiments • > 50% have failed to reproduce their own ones! • Chemistry: 90% (60%) • Biology: 80% (60%) • Physics and engineering: 70% (50%) • Medicine: 70% (60%) • Earth and environmental science: 60% (40%) ANY PROBLEM WITH THE SCIENTIFIC METHOD?? ACTUALLY, YES! 2016
- 5. Questionnaire on reproducibility (1500 scientists) • 70% of researchers have tried and failed to reproduce another scientist's experiments • > 50% have failed to reproduce their own ones! • Chemistry: 90% (60%) • Biology: 80% (60%) • Physics and engineering: 70% (50%) • Medicine: 70% (60%) • Earth and environmental science: 60% (40%) ANY PROBLEM WITH THE SCIENTIFIC METHOD?? ACTUALLY, YES! 2016 Aha! So you don’t empathise?
- 6. Questionnaire on reproducibility (1500 scientists) • 70% of researchers have tried and failed to reproduce another scientist's experiments • > 50% have failed to reproduce their own ones! • Chemistry: 90% (60%) • Biology: 80% (60%) • Physics and engineering: 70% (50%) • Medicine: 70% (60%) • Earth and environmental science: 60% (40%) ANY PROBLEM WITH THE SCIENTIFIC METHOD?? ACTUALLY, YES! Maybe with this? 2016
- 7. • Reproducibility (theoretical definition): An experiment/study is reproducible if an external researcher could repeat the same procedures and confirm the results using the same set up, input data and methods • Reproducibility (in practice): input data, methods, set up parameters, output data and results, and the computational environment, together with details on the context and links between the pieces of the experiment. UHMMM,WHAT DOYOU MEAN BY REPRODUCIBILITY? Moving from narratives (last 300 yrs) to the actual output of research
- 8. • Reproducibility (theoretical definition): An experiment/study is reproducible if an external researcher could repeat the same procedures and confirm the results using the same set up, input data and methods • Reproducibility (in practice): input data, methods, set up parameters, output data and results, and the computational environment, together with details on the context and links between the pieces of the experiment. UHMMM,WHAT DOYOU MEAN BY REPRODUCIBILITY? BTW: Reproducibility is not the aim, is the mean
- 9. PERSPECTIVES
- 10. PERSPECTIVES Data to the desktop: “individual scientist” • I have the best code, which I know how to use and can do special things • I do not trust any “pipeline” that you made • partly because I know better how to do it • partly because I read the news and there is a reproducibility crisis • well, and I can hardly reproduce the results of my own papers some years later... • In general I want full control of the software and of the computational environment
- 11. PERSPECTIVES Computation to data, providers perspective: Data Centres Mandatory in the Era of Megascience infrastructures
- 12. PERSPECTIVES Computation to data, providers perspective: Data Centres • We need to install your software in our platform. Can we trust it? Can we run it? Environment, dependencies, etc • Hey, we are offering services to the community, computation + tools. We would be grateful if you allow us to share it with other users (with proper credit) • Mmmm, sharing is great, but, putting the software in the platform is not enough: you need to provide the context for people to be able to rerun the software on the same or other data Mandatory in the Era of Megascience infrastructures
- 13. PERSPECTIVES Large alliances of scientists to develop Key Science Projects Mandatory (as well) to analyse the data deluge from this next generation of facilities
- 14. PERSPECTIVES Large alliances of scientists to develop Key Science Projects • We have tools to generate Advanced Data Products, and we will put them there where the storage and computation is (Data Centres) • But... we put effort on it, what would we gain if we make the *additional effort* to make it reusable? If we make it, then we will pave the way to competitors • Well, maybe we will share in 4 yrs time (PhD typical time) Mandatory (as well) to analyse the data deluge from this next generation of facilities
- 15. PERSPECTIVES Publishers • Will we need different profiles of referees to evaluate the scientific discussion together with the data quality and the methods (aka. Reproducibility)? • If the data and the methods (tools) will be in Data Centres, will our referees need to become a “user” of the Data Centres to be able to validate a paper? • Will we be able to engage so many referees as may be needed? • Will we need to validate the data, the tools, and the scientific analysis separetely? The challenge of going “beyond the PDF”
- 16. PERSPECTIVES Policy makers / funding agencies • How to measure reproducibility? • How to weight it and/or aggregate with other indicators? • Is it affordable / sustainable? Reproducibility as a key element of Open Science
- 17. METRICS ... “Science is being killed by numerical ranking,”[...] Ranking systems lures scientists into pursuing high rankings first and good science second Productivity seems to prevail over Discovery
- 18. METRICS ... “Science is being killed by numerical ranking,”[...] Ranking systems lures scientists into pursuing high rankings first and good science second Reproducibility crisis
- 19. SKA1: 197 dishes + 125.000 dipoles SKA: 2500 dishes + 500.000 dipoles THE SQUARE KILOMETRE ARRAY Ø 1000 scientists & engineers from > 270 institutions, > 20 countries. Ø 11 Member countries
- 20. SKA1: 197 dishes + 125.000 dipoles SKA: 2500 dishes + 500.000 dipoles THE SQUARE KILOMETRE ARRAY Ø 1000 scientists & engineers from > 270 institutions, > 20 countries. Ø 11 Member countries
- 21. THE SQUARE KILOMETRE ARRAY The Challenge: Extraction of scientific knowledge • Direct delivery to end users is unfeasible • International distributed scientific teams SKA Regional Centres will provide access to SKA data products, tools and processing power to generate and analyse Advanced Data Products
- 22. AREWE READY? • We are in a race to exploit ever larger datasets: in our quest for “efficiency” we risk forgetting about reproducibility • Unless we are ready to change the way in which we, the scientists, work, there is no guarantee that the quality of Science will improve.
- 23. AREWE READY? The era of Big Data is beginning across sciences Today is the time to ask what kind of Research mega-science infrastructures want to do in tomorrow’s future Are we ready to take up that challenge? • We are in a race to exploit ever larger datasets: in our quest for “efficiency” we risk forgetting about reproducibility • Unless we are ready to change the way in which we, the scientists, work, there is no guarantee that the quality of Science will improve.