Towards Reproducible Science: a few building blocks from my personal experience

Editor's Notes

• #2: Cambiar la licencia por la que aplique.
• #3: Experiments are central to empirical science, they are the foundation in which experimental sciences are built and improved. They allow to verify the hypothesis defined according to the scientific method. Convince the reader (other scientists) that the conclusions of an study are correct. For that, and for supporting the growth of science, the must be a repeatable process. (both by him/herself and by other scientists).
• #4: In last decades there has been an evolution in the way experimental science is conducted, adding computational resources for solving scientific problems. We have moved from a paradigm in which experiments were mainly conducted on laboratories or in nature, also referred to as in vitro or in vivo science To a paradigm in which simulations and mathematical models executed over computational resources, are used for obtaining scientific insights, also referred to as computational science or in silico science. Computational experiments complement rather than substitute classical experiments.
• #5: In both cases, either in classical or computational experimental science, experiments must be a repeatable process For trusting the scientific results And for allowing the development of incremental research.
• #8: In this context, a definition of which kind od repeatability we are looking for, and how we plan to do it, must be provided. The first thing that we have to do, is to define how we are going to take care of the object of interest, which can be done in 2 main ways Preservation: the act of isolating the object preventing any interaction that could damage it. Conservation: the set of actions for studying the object and its associated features, allowing a supervised or restricted use of it. The processes allow to prolong the life of the object.
• #9: Once a plan for taking care of the object have been stated, we have as well two ways for obtaining a repetition of the it: A replication: an copy of the original object which is as close as possible to the original A reproduction: an object that expose or mimic a certain set of features in the same way as the original one In this work we explore how conservation techniques can be applied for experimental science reproducibility For achieving this conservation and reproducibility…
• #10: Any scientific experiment can be divided into three main components DATA: the phenomena we study from nature, light from stars, genomes from plants or animals, reports in social science, etc. SCIENTIFIC PROCEDURE: the set of steps that have to be performed in order to obtain the results of the experiment. EQUIPMENT: the set of tools that are required by scientists in order to capture, process and interpret the desired data. From telescopes to microscopes, petri dishes or bunsen burners, there is a wide range of tools depending on the scientific domain. All these components… __________________________
• #11: All these components have a counterpart in the Computational Science world. DATA is often represented by means of tables in data bases, structured files, or even web services providing data. The SCIENTIFIC PROCEDURE can be defined by the source code written on a given language or by the descriptions of a set of invocations of different tools. … and in last decades, as Scientific Workflows, which have emerge as a paradigm for formally defining the set of data transformations to perform the scientific procedure of a computational experiment. Finally, the EQUIPMENT of a computational experiment is defined by the of hardware and software resources that are required to execute the experiment. Some initiatives have ….
• #25: In our platform the users login with an ORCID ID.
• #26: We capture bibliographic data and information related to the description of the protocol like purpose, applications, advantages, limitations, etc.
• #27: We capture a set of metadata for representing the sample, one of them is the name of the organism; and the name of the organism come from …
• #28: And in the case of the reagents we capture the reagents from PubChem API
• #29: the users can draw their workflows, describe each step or instruction and capture additional information as equipment, reagent, kits, software that participate in each step, also the users can include alerts messages, etc.
• #32: All these components have a counterpart in the Computational Science world. DATA is often represented by means of tables in data bases, structured files, or even web services providing data. The SCIENTIFIC PROCEDURE can be defined by the source code written on a given language or by the descriptions of a set of invocations of different tools. … and in last decades, as Scientific Workflows, which have emerge as a paradigm for formally defining the set of data transformations to perform the scientific procedure of a computational experiment. Finally, the EQUIPMENT of a computational experiment is defined by the of hardware and software resources that are required to execute the experiment. Some initiatives have ….
• #33: Some initiatives have been proposed to target the reproducibility issues of the different components of experiments in computational science.
• #34: DATA Examples: RDA, Open Provenance Mode, MIBBI, VCR…
• #35: Some initiatives have been proposed to target the reproducibility issues of the parts of computational experiments. SC. PROCEDURE Examples: Taverna, Pegasus, WINGS, Galaxy, SCUFL WMS and their related WF languages are a way of encapsulating an preserving the scientific procedure in computational experiments Platforms such as myExperiment allow its sharing and reproducibility
• #36: Finally, we found that there was a lack of approaches targeting the computational equipment by the time we started this work. Most of the work done in the area by that time, focused on sharing virtual machine images, as a way of providing exact copies of the execution environment During the time of this work, some other initiatives have appear targeting this problem, as we will discuss later. ------------------------------------------------- EQUIPMENT There is a lack of initiatives in this aspect Some projects have aimed to approach it during the time of this work. Most of them focus on the use of VM -> BLACK BOXES (here we should motivate the need of exposing the knowledge about the execution environment for increasing the reproducibility) Examples: CernVM, ReproZip, TIMBUS NOTE: LINK THIS ONE WITH THE FOLLOWING SLIDE ABOUT THE OPEN RESEARCH PROBLEMS
• #37: To share your research materials (RO as a social object) To facilitate reproducibility and reuse of methods To be recognized and cited (even for constituent resources) To preserve results and prevent decay (curation of workflow definition; using provenance for partial rerun) Middleware
• #39: All these components have a counterpart in the Computational Science world. DATA is often represented by means of tables in data bases, structured files, or even web services providing data. The SCIENTIFIC PROCEDURE can be defined by the source code written on a given language or by the descriptions of a set of invocations of different tools. … and in last decades, as Scientific Workflows, which have emerge as a paradigm for formally defining the set of data transformations to perform the scientific procedure of a computational experiment. Finally, the EQUIPMENT of a computational experiment is defined by the of hardware and software resources that are required to execute the experiment. Some initiatives have ….
• #40: The firs open problem we identified is that… ____________________________________ Open Research Problem 1: Computational Infrastructures are usually a predefined element of a Computational Scientific Workflow. The majority of computational scientists develop their experiments with an already existing infrastructure in mind, thus not considering its definition as part of the experiment. Open Research Problem 2: Execution Environments are poorly described, or even not described at all, when describing the results of an experiment. Often, the infrastructure used in the evaluation process is summarized explaining briefly its hardware overall capabilities and the basic software stack. This lack of information compromises the conservation and reproducibility of the experiment. Open Research Problem 3: Current approaches for Computational Scientific Experiments conservation and reproducibility take into account only the compu-tational process of the experiment (scientific procedure) and the data used and produced, but not the execution environment.
• #41: Open Research Problem 1: Computational Infrastructures are usually a predefined element of a Computational Scientific Workflow. The majority of computational scientists develop their experiments with an already existing infrastructure in mind, thus not considering its definition as part of the experiment.
• #42: Open Research Problem 2: Execution Environments are poorly described, or even not described at all, when describing the results of an experiment. Often, the infrastructure used in the evaluation process is summarized explaining briefly its hardware overall capabilities and the basic software stack. This lack of information compromises the conservation and reproducibility of the experiment.
• #43: Open Research Problem 3: Current approaches for Computational Scientific Experiments conservation and reproducibility take into account only the computational process of the experiment (scientific procedure) and the data used and produced, but not the execution environment. Based on this study, in this work, we focus on the aspects related to the reproducibility of the computational EQUIPMENT of a scientific experiment defined as a computational scientific workflows.
• #44: That is, a set of modes for annotating the original environment, and that can be used for specifying and reproducing a new equivalent using cloud solutions
• #45: As a result of this process, we developed the WICUS ontology network, which is composed…
• #46: The first ontology is the workflow execution environment, which introduces the concept of workflow… Using this ontology we can describe the structure of a workflow, such as the ones depicted on this figure, which describes 3 workflows belonging to the Pegasus WMS, represented by the different figures and colors. Here we see how each of workflows is composed by a set of subworkflows, each one of them related to different execution requirement, as well as the requirement defined by the WMS (pegasus in this case).
• #47: - DEPENDENCIES: JAR FILES DEPENDS ON THE JAVA VM
• #51: Examples… Based on these models, that allow us to describe execution environments of scientific workflows….
• #52: These system is composed by 3 main stages, which process the available experimental materials, for obtaining the corresponding enactment files These enactment files can be executed for deploying a reproduced execution environment. These overview can be decomposed into a set of modules and intermediate results generated during the process of reproducing an experiment _________________________________________ There are several input files and registries that can be used to extract information about the execution environment of the workflows Wf spec (DAG, make, etc.) SW comp registry (TC) WMS annotations (manual) SVA catalog (manual)
• #53: We evaluated a total of 6 different workflows All of them expose different computational characteristics, From small ones, such as internal extinction, to really large ones such as SoyKb Or those requiring small amount of time for execution, such as xcorr, or montage, to the ones requiring 20 to 24 hours, such as BLAST All these workflows have been developed by different institutions, and published in different conferences and journals Some of them date from a decade ago, whereas others have been published recently We have selected them, based on their domain and the availability of their materials and support by the communities.
• #54: Executed the 6 workflows in their original context Documented their execution environment Executed the ISA, obtaining enactment scripts Enacted the reproduced environments and executed the workflows. Workflow results compared to the corresponding baseline executions Montage: pHash similarity, factor 1.0, 0.85 factor Epigenomics and SoyKB: non-deterministic, out files equal in terms of number of lines and content, with no errors. Internal Extinction and xcorr: exact same results, even when in the case of internal extinction they may vary BLAST: equal results With this we consider the reproduction of the execution environments to be successful
• #55: Executed the 6 workflows in their original context Documented their execution environment Executed the ISA, obtaining enactment scripts Enacted the reproduced environments and executed the workflows. Workflow results compared to the corresponding baseline executions Montage: which generates an image of the sky, pHash similarity, factor 1.0, 0.85 factor Epigenomics and SoyKB: non-deterministic, out files equal in terms of number of lines and content, with no errors. Internal Extinction and xcorr: exact same results, even when in the case of internal extinction they may vary BLAST: equal results With this we consider the reproduction of the execution environments to be successful
• #56: Executed the 6 workflows in their original context Documented their execution environment Executed the ISA, obtaining enactment scripts Enacted the reproduced environments and executed the workflows. Workflow results compared to the corresponding baseline executions Montage: which generates an image of the sky, pHash similarity, factor 1.0, 0.85 factor Epigenomics and SoyKB: non-deterministic, out files equal in terms of number of lines and content, with no errors. Internal Extinction and xcorr: exact same results, even when in the case of internal extinction they may vary BLAST: equal results With this we consider the reproduction of the execution environments to be successful
• #57: Executed the 6 workflows in their original context Documented their execution environment Executed the ISA, obtaining enactment scripts Enacted the reproduced environments and executed the workflows. Workflow results compared to the corresponding baseline executions Montage: which generates an image of the sky, pHash similarity, factor 1.0, 0.85 factor Epigenomics and SoyKB: non-deterministic, out files equal in terms of number of lines and content, with no errors. Internal Extinction and xcorr: exact same results, even when in the case of internal extinction they may vary BLAST: equal results With this we consider the reproduction of the execution environments to be successful
• #58: Executed the 6 workflows in their original context Documented their execution environment Executed the ISA, obtaining enactment scripts Enacted the reproduced environments and executed the workflows. Workflow results compared to the corresponding baseline executions Montage: which generates an image of the sky, pHash similarity, factor 1.0, 0.85 factor Epigenomics and SoyKB: non-deterministic, out files equal in terms of number of lines and content, with no errors. Internal Extinction and xcorr: exact same results, even when in the case of internal extinction they may vary BLAST: equal results With this we consider the reproduction of the execution environments to be successful
• #59: Executed the 6 workflows in their original context Documented their execution environment Executed the ISA, obtaining enactment scripts Enacted the reproduced environments and executed the workflows. Workflow results compared to the corresponding baseline executions Montage: which generates an image of the sky, pHash similarity, factor 1.0, 0.85 factor Epigenomics and SoyKB: non-deterministic, out files equal in terms of number of lines and content, with no errors. Internal Extinction and xcorr: exact same results, even when in the case of internal extinction they may vary BLAST: equal results With this we consider the reproduction of the execution environments to be successful
• #62: Cambiar la licencia por la que aplique.