Reproducibility, Research Objects and Reality, Leiden 2016
Download as PPTX, PDF1 like1,095 views
The document discusses the importance of reproducibility in computational science, emphasizing guidelines, standards, and infrastructure necessary for reliable research practices. It highlights the challenges faced in achieving reproducibility, such as data sharing, methodological rigor, and software configurations. The document proposes frameworks and tools for better research object management, ensuring that data, methods, and models can be archived, reused, and verified effectively.
![Motivation: Knowledge Turning
research infrastructures
• Computational tools
• Sharing platforms
• Knowledge
Exchange
• Reproducible
research
• Software and data
practices
• Policies
[Josh Sommer, for the picture]](https://image.slidesharecdn.com/leiden-goble-presented-161205202412/85/Reproducibility-Research-Objects-and-Reality-Leiden-2016-4-320.jpg)
![… in situ reproducible models
metadata annotation against standards
model validation, comparison and simulation
SBML Model simulation
Model comparison
Model versioning
Reproducing simulations
[Jacky Snoep, Dagmar Waltemate, Martin Peters, Martin Scharm]](https://image.slidesharecdn.com/leiden-goble-presented-161205202412/85/Reproducibility-Research-Objects-and-Reality-Leiden-2016-57-320.jpg)
![Preparation pain. Goldilocks paradox.
[Norman Morrison]
replication hostility no funding, time, recognition, place to publish
resource intensive access to the complete environment](https://image.slidesharecdn.com/leiden-goble-presented-161205202412/85/Reproducibility-Research-Objects-and-Reality-Leiden-2016-64-320.jpg)
![Sharing by side effect …. libertarian paternalism
[Kristian Garza]](https://image.slidesharecdn.com/leiden-goble-presented-161205202412/85/Reproducibility-Research-Objects-and-Reality-Leiden-2016-71-320.jpg)
Reproducibility, Research Objects and Reality, Leiden 2016
- 1. Reproducibility, Research Objects and Reality Professor Carole Goble The University of Manchester, UK Software Sustainability Institute, UK ELIXIR UK, FAIRDOMAssociation e.V. carole.goble@manchester.ac.uk University of Leiden,The Netherlands, 24 November 2016
- 2. Acknowledgements • Dagstuhl Seminar 16041 , January 2016 – http://www.dagstuhl.de/en/program/calendar/semhp/?semnr=16041 • ATI Symposium Reproducibility, Sustainability and Preservation , April 2016 – https://turing.ac.uk/events/reproducibility-sustainability-and-preservation/ – https://osf.io/bcef5/files/ • CTitus Brown • Juliana Freire • David De Roure • Stian Soiland-Reyes • Barend Mons • Tim Clark • Daniel Garijo • Norman Morrison • Katy Wolstencroft Phil Bourne Natalie Stanford Jacky Snoep Stuart Owen Marco Roos Kristina Hettne AlanWilliams Sean Bechhofer Ian Fore Rafael Jimenez …. And many more Michael Crusoe Paul Groth Niall Beard
- 3. Context: Computational Science http://tpeterka.github.io/maui-project/ From:The Future of ScientificWorkflows, Report of DOEWorkshop 2015, http://science.energy.gov/~/media/ascr/pdf/programdocuments/docs/workflows_final_report.pd 1. Observational, experimental 2. Theoretical 3. Simulation 4. Data intensive
- 4. Motivation: Knowledge Turning research infrastructures • Computational tools • Sharing platforms • Knowledge Exchange • Reproducible research • Software and data practices • Policies [Josh Sommer, for the picture]
- 7. NIH Rigor and Reproducibility https://www.nih.gov/research- training/rigor-reproducibility Plenty of guidelines cos.io/top
- 9. https://wellcomeopenresearch.org/ Nature Scientific Data Data as a first class citizen + Data Citation Scholarly Communications Providers
- 10. Software as a first class citizen + Software Citation
- 12. republic of science* regulation of science institution cores / libraries / public services *Merton’s four norms of scientific behaviour (1942)
- 14. Research Infrastructure for FAIR Management and Sharing of Data, Operating Procedures, Model For Systems and Synthetic Biology Projects Research Infrastructure for FAIR Data for Life Sciences in Europe Data-Driven Science
- 16. design cherry picking data, random seed reporting, non-independent bias, poor positive and negative controls, dodgy normalisation, arbitrary cut-offs, premature data triage, un-validated materials, improper statistical analysis, poor statistical power, stop when “get to the right answer”, software misconfigurations misapplied black box software reporting incomplete reporting of software configurations, parameters & resource versions, missed steps, missing data, vague methods, missing software Empirical Statistical Computational V. Stodden, IMS Bulletin (2013) Reproducibility and reliability of biomedical research: improving research practice
- 17. “When I use a word," Humpty Dumpty said in rather a scornful tone, "it means just what I choose it to mean - neither more nor less.” Carroll, Through the Looking Glass re-compute replicate rerun repeat re-examine repurpose recreate reuse restore reconstruct review regenerate revise recycle redo robustness tolerance verificationcompliancevalidation assurance remix
- 18. Scientific publications goals: (i) announce a result (ii) convince readers its correct. Papers in experimental science should describe the results and provide a clear enough protocol to allow successful repetition and extension. Papers in computational science should describe the results and provide the complete software development environment, data and set of instructions which generated the figures. VirtualWitnessing* *Leviathan and theAir-Pump: Hobbes, Boyle, and the Experimental Life (1985) Shapin and Schaffer. Jill Mesirov David Donoho
- 21. Repeatability: “Sameness” Same result 1 Lab 1 experiment Reproducibility: “Similarity” Similar result > 1 Lab > 1 experiment why the differences? https://2016-oslo- repeatability.readthedocs.org/en/latest/repeatability-discussion.htm Validate Verify
- 22. Method Reproducibility the provision of enough detail about study procedures and data so the same procedures could, in theory or in actuality, be exactly repeated. Result Reproducibility (aka replicability) obtaining the same results from the conduct of an independent study whose procedures are as closely matched to the original experiment as possible Goodman, et al ScienceTranslational Medicine 8 (341) 2016 Validate Verify
- 24. reviewers want additional work statistician wants more runs analysis needs to be repeated post-doc leaves, student arrives new/revised datasets updated/new versions of algorithms/codes sample was contaminated better kit - longer simulations new partners, new projects Personal & Lab Productivity Public Good Reproducibility
- 25. Computational “Datascopes” Methods techniques, algorithms, spec. of the steps, models Materials datasets, parameters, algorithm seeds Instruments codes, services, scripts, underlying libraries, workflows, ref datasets Laboratory sw and hw infrastructure, systems software, integrative platforms computational environment
- 26. “Datascope” Practicalities Methods Materials Instruments Laboratory Change Dependencies science, methods, datasets questions stay, answers change breakage, labs decay, services, techniques and instruments change, updated datasets, services, codes, hardware software entropy one offs, streams, stochastics, sensitivities, scale, non-portable data supercomputer access non-portable software licensing restrictions unreliable resources and third party codes complexity Blackboxes blackbox software hidden manual steps blackbox software hidden manual steps
- 28. Active Instrument Byte level preservation Reproduce by RunningReproduce by Reading Archived Record Prepare to repair ELNs Markup Languages Reporting Guidelines Common Formats Community vocabularies
- 29. Record All Automate All Contain All Expose All Findable Accessible Interoperable Reusable
- 30. provenance portability preservation robustness versioning access description standards common APIs licensing standards, common metadata change variation sensitivity discrepancy handling packaging, containers FAIR RACE shades of reproducibility dependencies stepsids
- 31. A robust infrastructure for biological information. bio.tools
- 32. https://usegalaxy.org/ Workflow Description Workflows Preservation Workflow Portability Workflow Interoperability
- 33. Workflow Preservation and Exchange Experiments Workflows &Workflow Runs Workflow Commons Third Party Services Scattered resources
- 34. Workflow Preservation and Exchange Experiments Workflows &Workflow Runs Workflow Commons Third Party Services Scattered resources Rich descriptions Prepare to Repair
- 35. Standards-based metadata framework for bundling resources with context Citable Reproducible Packaging Metadata for bundling resources scattered and stored somewhere else
- 36. Container Research Object in a nutshell Packaging content & links: Zip files, BagIt, Docker images Catalogues & Commons Platforms: FAIRDOM, myExperiment
- 37. Manifest Construction Aggregates link things together Annotations about things & their relationships Container Research Object in a nutshell Manifest Description Dependencies what else is needed Versioning its evolution Checklists what should be there Provenance where it came from Identification locate things regardless where id Packaging content & links: Zip files, BagIt, Docker images Catalogues & Commons Platforms: FAIRDOM, myExperiment
- 38. Manifest Construction Aggregates link things together Annotations about things & their relationships Container Research Object Profile forWorkflows… Manifest Description Identification locate things regardless where Minimum information for one content type Common properties among content types
- 39. Research Object Profile forWorkflows… Manifest Description Minimum information for one content type Common properties among content types
- 40. Belhajjame et al (2015) Using a suite of ontologies for preserving workflow-centric research objects, JWeb Semantics doi:10.1016/j.websem.2015.01.003 Hettne KM, et al (2014), Structuring research methods and data with the research object model: genomics workflows as a case study. J. Biomedical Semantics 5: 41 Workflow Research Object Bundles exchange, portability and maintenance BagIt workflows packaged into various containers for sharing Checksum
- 41. Workflow and Workflow Management System Zoo https://github.com/common-workflow-language/common-workflow-language/wiki/Existing-Workflow-systems
- 42. bio.tools A community led standard way of expressing and running workflows and command line tools using containers Ontologies for describing tools and their inputs and outputs Metadata framework for the manifest versioning, file integrity, more metadata about the workflow Workflow fragment containers
- 43. Findable Accessible Interoperable Reusable Data Operations Models Systems and Synthetic Biology Projects
- 45. Project Support Community Actions Platforms,Tools Web-based Portal Public Commons 50+ projects 5 programmes 400+ people 22 independent installations
- 46. Systems Approach… Multiple, interrelated assets, Multiple, dispersed repositories Literature SOPS STANDARDS versioning, tracking: provenance, parameters, citation Operations
- 47. FAIR Data and Metadata Standards that help to improve understanding and exchange…. Nicolas Le Novère, Babraham Institute, UK. …researchers do not always use them....
- 48. … model reuse and reproducibility tricky… Stanford et alThe evolution of standards and data management practices in systems biology, Molecular Systems Biology (2015) 11: 851 DOI 10.15252/msb.20156053
- 49. Systems Approach… teams, processes, multi-partner, multi-discipline, legacy Funders Researchers Publishers
- 50. What methods are been used to determine enzyme activity? What SOP was used for this sample? Where is the validation data for this model? Is there any group generating kinetic data? Is this data available? Track versions of my model Whats the relationship between the data and model? Which data belong to which publications? FAIR
- 53. Investigation Study Analysis Data Model SOP(Assay) ….organised in Investigation, Study, Assay/Analysis format ….registered using Just Enough Results description
- 54. ….organised in Investigation, Study, Assay/Analysis format ….registered using Just Enough Results description. Just Enough Results Model Common elements
- 55. ….organised in Investigation, Study, Assay/Analysis format ….registered using Just Enough Results description. Uploaded into the FAIRDOM Store Linked to entry in Public Archive Linked to entry in Project store
- 56. ... aggregating catalogue metadata across repositories, retain context-> reproduce, reuse Local Stores External Databases Publishing services Secure Stores Model Resources
- 57. … in situ reproducible models metadata annotation against standards model validation, comparison and simulation SBML Model simulation Model comparison Model versioning Reproducing simulations [Jacky Snoep, Dagmar Waltemate, Martin Peters, Martin Scharm]
- 58. …. Nested Packages context and credit
- 59. Research Objects • Link • Nest • Span • Bundle • Snapshot Systematic, Standards- based metadata framework for logically and physically bundling resources with context • Exchange • Reproduce • Release packages
- 60. Reproducible Exchange and Publishing and better credit Author List: Joe Bloggs; Jane Doe Title: My Investigation Date: September 2016 DOI: https://doi.org/10.15490/seek## information travels with the data and models
- 61. How do we do? Pretty well. Reproducibility window. But that’s ok! • Can’t contain everything – Pesky Internet in a Box • Can’t automate everything – Pesky people • Can’t fix everything – Pesky science
- 62. Asthma Research e-Laboratory Release builds of pharmacological knowledge warehouse Exchanging large datasets
- 63. Samiul Hasan, GSK Biocuration need in Pharma: Drivers from aTranslational Bioinformatics Perspective, Poster S16 1st EASYMConference, Berlin 2016 Reality
- 64. Preparation pain. Goldilocks paradox. [Norman Morrison] replication hostility no funding, time, recognition, place to publish resource intensive access to the complete environment
- 65. “Data Parasites” “Data Flirters” “Share Drift” Family Friends Potential Friends Acquaintances Strangers Rivals Reciprocity
- 66. Using FAIRDOM my own lab colleagues saw what I was doing and called to collaborate! Jurgen Hannstra Vrije Universiteit Amsterdam, Netherlands Trust …
- 69. Half of researchers make research data available so they can be used by another. Most not experienced any direct benefits nor experienced many bad effects. Caveat: shared but usable? fake sharing funder requirements fear data will be misused or misinterpreted journal requirements good research practice facilitate collaborations enable validation and replication higher citation rates time and effort new collaborations extra funding for cost of data prep enhance their academic reputation feedback on how other researchers were using their data taken into account in funding taken into account in career jeopardise future publications its not ready to share scrutiny scruples answering questions I won’t get credited
- 70. Metadata in by side effect Tooling for annotations and checklist templates for different types of assay data. Embed ontologies into Excel templates Excel spreadsheets enriched with ontology annotations Upload, extract metadata and register http://www.rightfield.org.uk Spreadsheet Ramps!!
- 71. Sharing by side effect …. libertarian paternalism [Kristian Garza]
- 72. Finding and Citing by side effect • Schema.org • Structured markup in web pages • Supported by Content Management Systems • Harvested by search engines • Builds snippets and sidebars Bioschemas.org
- 73. Data repository Data repository Training Resource Bioschemas Bioschemas Bioschemas Search engine Bio Registries Biosharing OLS, TeSS bio.tools UKCRC Tissue Directory bioCADDIE DATAMED PDBe UniProt Interpro Molgenis Pfam Gene3DBiosamples Biobank websites BRENDA HPA TransPlantEGA Beacons EBI-Search Google Finding and Citing by side effect Bioschemas.org
- 74. Big co-operative data-driven science makes reproducibility desirable but also means dependency and change are to be expected Words matter. 50 Shades of Reproducibility. form vs function Reproducibility is not a end. Beware zealots. Amplify Side effects Think Research Objects!