Computational Reproducibility vs. Transparency: Is It FAIR Enough?
0 likes109 views
The document discusses the concepts of computational reproducibility and transparency in research, emphasizing the importance of fair data principles and proper reporting to address the reproducibility crisis in science. It highlights the distinction between reproducibility, replicability, and exact repeatability, while also advocating for the use of transparent research objects to facilitate reproducibility and improve access to research data. The paper proposes a multidimensional approach to understanding reproducibility and the necessity of capturing and querying provenance in computational research.
Computational Reproducibility vs. Transparency: Is It FAIR Enough?
- 1. Computational Reproducibility vs Transparency: Is it FAIR enough? Bertram Ludäscher Director, Center for Informatics Research in Science & Scholarship (CIRSS) School of Information Sciences (iSchool@Illinois) & National Center for Supercomputing Applications (NCSA) & Department of Computer Science (CS@Illinois) with special thanks (and apologies) to Timothy McPhillips & Workshop on Research Objects (RO), eScience, San Diego, 2019 & Reproducibility of Data-Oriented Experiments in e-Science, Dagstuhl Seminar, 2016 Whole Tale
- 2. Overview • FAIR data, code, and reproducibility • The Reproducibility Crisis ... • ... and an R-Words (terminology) crisis? • Reproducibility and Information Gain (PRIMAD) • => shift from R-Words to T-Words: Transparency … • Capturing and querying Provenance • Reproducibility & Transparency in Whole Tale Reproducibility & Transparency 2
- 3. FAIR data, code, … Reproducibility • FAIR data principles: data should be findable, accessible, interoperable, reusable • Metadata (duh!) is key! • .. the principles are now being adapted (mutatis mutandis) for code, scientific workflows, … • Can we do something about “the reproducibility crisis”? – e.g. by focusing on computational reproducibility …!? Reproducibility & Transparency 3
- 4. Is Reproducibility really so complicated? § Reproducibility crisis? § Terminology crisis? § Or gullibility crisis? § What is reproducibility anyway? § And who is responsible for it? Reproducibility & Transparency 4
- 5. Pop QUIZ: What is the single most effective way to make your research more reproducible? a) Employ the interoperability standards for scientific data, metadata, software, and Research Objects b) Carefully record and report your work c) Use open source software and make any new or modified code freely available. d) Apply FAIR (findable, accessible, interoperable, reusable) principles e) Do all of your work in software containers f) Focus your research on intrinsically reproducible phenomena Reproducibility & Transparency 5
- 6. Basic Assumptions made by researchers in the Natural Sciences … § We are discovering things that are the way they are whether we go and look for them or not. § We are discovering things that conceivably could be different than they happen to be. To find out how things actually are we must go look. § It does not matter who does the looking. Everyone with the same opportunity to look will find the same things to be true. … nature as the ultimate reproducibility arbiter … Reproducibility & Transparency 6
- 7. Is there a hierarchy of intrinsic reproducibility? Logic Math Physics Chemistry Biology Human Sciences : : Emergence of natural laws Greater reproducibility of phenomena It’s not so simple… Reproducibility & Transparency 7 • … but things tend to get “messier” further up …
- 8. Limits on reproducibility in the natural sciences • Nature is not a digital computer. It’s more of an entropy generator built on chaos and (true) randomness with natural laws, math, and logic serving as constraints. • Good experiments are hard to design and to perform even once. • Instruments can be costly and limited in supply. • Many phenomena cannot be studied via experiment at all. • Past events are crucial to many theories. • Some things happen only once. • … so let’s hold back the horses (for now) on extensive and expensive computational reproducibility studies?? … But what is always possible? Transparency! Reproducibility & Transparency 8
- 9. FASEB* definition of transparency * The Federation of American Societies for Experimental Biology comprises 30 scientific societies and over 130,000 researchers. Transparency: The reporting of experimental materials and methods in a manner that provides enough information for others to independently assess and/or reproduce experimental findings. • Transparency is what allows an experiment to be reviewed and assessed independently by others. • Transparency facilitates reproduction of results but does not require reproduction to support review and assessment. • It is considered a problem if exact repetition of the steps in reported research is required either to evaluate the work or to reproduce results. 9 Reproducibility & Transparency
- 10. Quantifying Repeatability • Experiments on natural phenomena generally are not exactly repeatable. • Materials, conditions, equipment, and instruments all vary. • Uncertainty is intrinsic to most measurements. • Experimental biologists perform replicate experiments to assess end-to-end repeatability. A mystery?? Why are these “replicates”, not “reproductions”? Technical replicates: Measurements and data analyses performed on the same sample using the same equipment multiple times. Biological replicates: Measurements and data analyses performed on different but biologically equivalent samples on the same equipment. 10 Reproducibility & Transparency
- 11. Replication and Reproduction are natural processes that biologists study (.. a lot!) • Amazing aspect of life is the incredible fidelity with which genetic material— DNA—is replicated within cells. • DNA replication is carried out by the replisome—which even detects and corrects errors on the fly! • Organisms reproduce and have reproductive systems. • Biological reproduction is much lower fidelity than DNA replication. In fact, the process of reproduction often encourages variation in the children. Experimental replicates assess the highest possible fidelity at which an experiment can be repeated—by the same researcher, using the same equipment, on the same or equivalent samples, immediately one after the other in time. 11 Reproducibility & Transparency
- 12. Theorists talk about replication • Dawkins’ selfish genes are replicators. • Debate in origins of life research: Did replication or metabolism come first? • Could life have started before high-fidelity replication of genetic material was achieved? • For these theorists and philosophers high-fidelity is the defining characteristic of replication. 12 Reproducibility & Transparency
- 13. FASEB* definitions of reproducibility and replicability Replicability: The ability to duplicate (i.e., repeat) a prior result using the same source materials and methodologies. This term should only be used when referring to repeating the results of a specific experiment rather than an entire study. Maximal fidelity to original experiment, greater fidelity to original result. Reproducibility: The ability to achieve similar or nearly identical results using comparable materials and methodologies. This term may be used when specific findings from a study are obtained by an independent group of researchers. Less fidelity to original study, lower fidelity result expected. 13 * The Federation of American Societies for Experimental Biology comprises 30 scientific societies and over 130,000 researchers. Reproducibility & Transparency
- 14. Beyond Reproduction and Replication: Exact Repeatability • Digital computers use logic gates to achieve replication of information at such a low error rate we can call it exact. • Computers pull the exactness of logic and discrete mathematics up to the level of macroscale phenomena– quite a feat. • Exactness is (effectively) achievable for computer hardware, compiled software, program executions, and computing environments. • Researchers employing digital computers have access to a new kind of reproducibility never before seen in science: exact repeatability. Reproducibility & Transparency 14
- 15. ACM Initiative … Reproducibility & Transparency 15
- 16. ACM Initiative … reloaded? Reproducibility & Transparency 16
- 17. ACM Initiative … reloaded? Reproducibility & Transparency 17 This was “same” before! This was “different” before! The big switcheroo …
- 18. ACM caves to new terminology policey? Reproducibility & Transparency 18
- 19. Reproducibility badges and verification workflows … choices & options galore ... • ACM SIGMOD defines a defines a procedure for assessing database research reproducibility. • ACM awards (currently) four different reproducibility badges distinct from the SIGMOD reproducibility assessment. • ACM has defined eight versions of the guidelines for awarding its badges since 2015. • The workflow used by the American Journal of Political Science (AJPS) to verify computational artifacts also is versioned. • Does the meaning of reproducibility badges may change from year to year even within a single organization? Is there light at the end of the terminology tunnel? db-reproducibility.seas.harvard.edu, www.acm.org/publications/policies/artifact-review-badging , ajps.org/wp-content/uploads/2019/01/ajps-quant-data-checklist-ver-1-2.pdf If we want these badges to have any meaning at all they should be mapped to something that isn’t constantly changing. 19 Reproducibility & Transparency
- 20. Reproducibility & Transparency 20 ACM was aligned - just not “in harmony” with NAS committee … Now it’s a more aligned with NAS, but no longer with FASEB, … (some crossed wires are now aligned; some previously aligned wires are now crossed … ) Yes, we need to Mind our Vocabulary! with namespaces: NAS:reproducibility ~ FASB:replicability NAS:replicability ~ FASB:reproducibility
- 21. Chaos is a ladder. Is reproducibility a staircase? Data published and accessible to all Code shared and freely licensed Computing environment repeatable Code produces expected artifacts Computed artifacts support paper Greater reproducibility? Code reusable ! It is tempting to think about reproducibility one-dimensionally … Study fully reproducible ! Reproducibility & Transparency 21
- 22. But isn’t scientific reproducibility multidimensional? • Do the R-words have an obvious order, where achieving one must precede achieving the next?? • Or might they represent base vectors of a multidimensional space? experiment replicability code re-executability findings reproducibility cf. PRIMAD 22 Reproducibility & Transparency
- 23. Modeling reproducibility as multidimensional may offer way out of the terminology quagmire • Recognize that different terminologies refer to different sets of dimensions; communities focus on different subspaces, or different choices of basis vectors. • Map conflicting definitions onto shared dimensions; use mappings to convert claims made using one terminology to claims using a different terminology. • Allow each community to focus on dimensions of interest to them using the most intuitive terminology; use namespaces to eliminate ambiguity. • Use Research Objects to attach claims about reproducibility to research artifacts, to disambiguate these claims, and to support queries using terminology of the user’s choosing. Reproducibility & Transparency 23
- 24. Transparent Research Objects • Transparency in the natural sciences enables research to be evaluated—and reported results used with confidence—without actually repeating others’ work. • How can ROs extend the advantages of transparency to computational research and the computational components of experimental studies? • Researchers need to be able to query the reproducibility characteristics of artifacts in ROs. • These queries need to be poseable using terminology familiar to the researcher—terminology likely different from that used by the author of the RO (minimizing headaches no matter which terminology you grew up with..) • Queries about computational reproducibility need to take the longevity of technological approaches to reproducibility into account. 24 Reproducibility & Transparency
- 25. Food for Thought: Research Objects & Information Gain • An object of research is the primary target of scholarly investigation. In contrast, we may think of a research object as an artifact that (a) performs a specific function, (b) is guided by and underlying theory (c) whose objective might be to allow information gains towards falsifying a particular hypothesis, and (d) Which admits representation through a metalanguage that captures its role in a science-driven discourse. Reproducibility & Transparency 25
- 26. PRIMAD (what have you “primed”?) Reproducibility & Transparency 26 Dagstuhl Seminar #16041 Report Outputs = Exec(M,I,P,D) | RO, A - M = parsimony/bootstrap/.. - I = package XYZ - P = MacOS .. - D = (Params, Files)
- 27. PRIMAD & Information Gain • Original study: Y = FP(X) Reproduction: Y’ = F’P’(X’) – Y’ ≈ Y => Reproduction Success else Reproduction Failure 27 no wiggle biggest wiggle no wiggle biggest wiggle Information Gain (Failure) Reproducibility & Transparency NOTE: This does NOT mean that a small delta in a parameter results couldn’t have a large change in the output …
- 28. PRIMAD (what have you “primed”?) Reproducibility & Transparency 28 Dagstuhl Seminar #16041 Report
- 29. Back to computational reproducibility: Journal verification workflows in Whole Tale ● Important new use case for Whole Tale ● Study of journal reproducibility initiatives (Willis, 2020a) -- FINDINGS: ○ Initiatives have common, basic requirements for transparency and computational reproducibility ○ Initiatives rely on established research repositories for artifact preservation and long-term access (so does WT) ○ Editorial infrastructure is lacking (tools to support packaging, access to computational infrastructure) -- WT provides this, but they need more ○ Need for standards for the description and packaging of reproducible and transparent computational Research Objects (our Tale format) Willis, C. (2020a). Trust, but verify: An investigation of methods of verification and dissemination of computational research artifacts for transparency and reproducibility (Ph.D. thesis). University of Illinois at Urbana-Champaign 29
- 30. Whole Tale & the Elements of a … Reproducible Computational Research Platform 30 Easy-to-access cloud-based computational environments Transparent access to research data Collaborate and share with others Export or publish executable research objects Re-execute Review Verify Re-use Develop Analyze Share Reproduce Package Support users (researchers, scientists) & the tools they already use! Reproducibility & Transparency
- 31. What’s in a tale? 31 Reproducibility & Transparency
- 32. 32 Whole Tale Platform Overview Research & Quantitative Computational Environments External Data Sources Code + Narrative ●Authenticate using your institutional identity ●Access commonly-used computational environments ●Easily customize your environment (via repo2docker) ●Reference and access externally registered data ●Create or upload your data and code ●Add metadata (including provenance information) ●Submit code, data, & environment to archival repository ●Get a persistent identifier ●Share for verification and re-use Publish Tale Create tale Analyze data <your biodiversity repos here> Upcoming Whole Tale releases & new features: • WT-v1.1: Git integration; Tale Sharing & Versioning; Support for licensed software (MATLAB and STATA) • WT-v1.2: Recorded Runs; Publishing Images Reproducibility & Transparency
- 33. Tale Creation Workflow Register telemetry dataset by digital object identifier: doi:10.24431/rw1k118 Create a Tale, entering a name and selecting the RStudio (Rocker) environment A container is launched based on selected environment with an empty workspace and external data mounted read-only Upload/create R Markdown notebook and install.R script Execute code/scripts to generate results/ outputs Export the Tale in compressed BagIt-RO format to run locally for verification. Publish the Tale to a DataONE member node generating a persistent identifier. Enter descriptive metadata including authors, title, description, and illustration image schema:author schema:name schema:categor y pav:createdBy schema:license Re-execute in Whole Tale 33 Reproducibility & Transparency
- 34. Some new, related features: Recorded Run* to support Transparency ● Automated workflow execution with provenance capture ● User specified execution entrypoint ● System provenance captured using ReproZip ● Converted to comprehensive provenance record (CPR) => query and reason about provenance => provenance reports ● Each recorded run is a version ● User can access past runs ● Standards-based Provenance information included in published tale 34
- 35. Recorded Run: Provenance Capture* sqlite3.db config.yml reprozip trace <my_cmd> rpz2cpr RDF <SPARQL> Queries ● Detailed computational provenance captured using reprozip trace ● ReproZip output converted to CPR as RDF triples ● Imported to Blazegraph for queries and reports Blazegraph 35
- 36. Comprehensive Provenance Record* (CPR) ● General provenance model that supports querying & reasoning across multiple “worldviews” => hybrid provenance model ● Retrospective provenance (system/runtime provenance) (… ptrace/strace via ReproZip …) ● Prospective provenance (e.g., YesWorkflow, CWL, … ) ● Language-level provenance (e.g., SDTL, … ) 36
- 37. Recorded Run: Example Queries* ● Q1: Show me all inputs and outputs of a given run ● Q2: Show me what software was installed at the time of the run ● Q3: Show me what software packages were actually used by the run ● Q4: Show me the packages/versions used by a particular script ● Q5: Show me scripts that use a particular package/version ● Q6: Show me which inputs where used or outputs created by a particular script ● … è Through queries and inference rules: additional information can be derived for reports (e.g. Deltas: what was installed by not used, ...) 37
- 38. § Prospective provenance declared using YesWorkflow annotations e.g. in Python. § Retrospective provenance captured at run time using noWorkflow (or: Reprozip, recordR, …) § Script run can produce hundreds of output files. § Each output has a distinct provenance. § Jointly querying YesWorkflow and noWorkflow yields answers to provenance questions that are meaningful to scientists. … for energy, frame_number, intensity, raw_image_path in collect_next_image( cassette_id, sample_id, num_images, energies, 'run/raw/{cassette_id}/{sample_id}/e{energy}/image_{frame_number:03d}.raw’): # @end collect_data_set # @begin transform_images @desc Correct raw image using the detector calibration image. # @param sample_id energy frame_number # @in raw_image_path @as raw_image # @in calibration_image @uri file:calibration.img # @out corrected_image @uri file:run/data/{sample_id}/{sample_id}_{energy}eV_{frame_number}.img # @out corrected_image_path total_intensity pixel_count corrected_image_path = 'run/data/{0}/{0}_{1}eV_{2:03d}.img'.format(sample_id, energy, frame_number) (total_intensity, pixel_count) = transform_image(raw_image_path, corrected_image_path, 'calibration.img') # @end transform_images # @begin log_average_image_intensity @desc Record statistics about each diffraction image. … average_intensity = total_intensity / pixel_count … Prospective and retrospective provenance: better together 38 Reproducibility & Transparency
- 39. § Prospective provenance declared using YesWorkflow annotations e.g. in Python. § Retrospective provenance captured at run time using noWorkflow (or: Reprozip, recordR, …) § Script run can produce hundreds of output files. § Each output has a distinct provenance. § Jointly querying YesWorkflow and noWorkflow yields answers to provenance questions that are meaningful to scientists. Prospective and retrospective provenance: better together 39 Reproducibility & Transparency
- 40. § Prospective provenance declared using YesWorkflow annotations e.g. in Python. § Retrospective provenance captured at run time using noWorkflow (or: Reprozip, recordR, …) § Script run can produce hundreds of output files. § Each output has a distinct provenance. § Jointly querying YesWorkflow and noWorkflow yields answers to provenance questions that are meaningful to scientists. Prospective and retrospective provenance: better together 40 Reproducibility & Transparency
- 41. § Prospective provenance declared using YesWorkflow annotations e.g. in Python. § Retrospective provenance captured at run time using noWorkflow (or: Reprozip, recordR, …) § Script run can produce hundreds of output files. § Each output has a distinct provenance. § Jointly querying YesWorkflow and noWorkflow yields answers to provenance questions that are meaningful to scientists. Prospective and retrospective provenance: better together 41 Reproducibility & Transparency
- 42. Takeaway Points • Computational reproducibility doesn’t mean what you might think it means (≈ re-executability) • Computational reproducibility is not required for reproducible science • Transparency on the other hand, is required for science. • Both have a place in (data- and compute-intensive) scientific publishing – You still need to read & understand the paper! (and maybe the code!?) – Special use cases, e.g. Craig Willis’ thesis: Trust but verify => support for “validation workflows” (cf. “badging” ) – In economics, social sciences => cf. Lars Vilhuber’s work • Opportunity cost by getting stuck with R-words => Shifting attention from R-words to T-words 42 Reproducibility & Transparency
- 43. T7 Workshop on Provenance for Transparent Research … write a page & participate!! 43 Organizers: Shawn Bowers (Gonzaga) Carole Goble (U Manchester) Bertram Ludäscher (UIUC) *Timothy McPhillips (UIUC) Craig Willis (UIUC) *Contact: tmcphill@illinois.edu Reproducibility & Transparency Trustworthy Transparent True Traceable Trials Tests … https://iitdbgroup.github.io/ProvenanceWeek2021/t7.html Part of ProvenanceWeek: July 19-22 2021.
- 44. Opportunities for future work … • There are many opportunities, e.g., … • 1) Sorting out terminological issues (NAS vs FASEB vs ACM … ) • 2) … Information Gain / PRIMAD+ (PRIMAD 2.0) !? • 3) Provenance Tools R&D : Provenance => Transparency => Science (… for a suitable definition of “=>” … ) • 4) Join T7 Workshop on Provenance for Transparent Research! 44 Reproducibility & Transparency
- 45. References • McPhillips, Timothy, Craig Willis, Michael R. Gryk, Santiago Nunez-Corrales, and Bertram Ludäscher. Reproducibility by other means: Transparent research objects. In 2019 15th International Conference on EScience (EScience), pp. 502-509. IEEE, 2019 • Rauber, A; Braganholo, V; Dittrich, J; Ferro, N; Freire, J; Fuhr, N; Garijo, D; Goble, C; Järvelin, K; Ludäscher B; Stein B; Stotzka R: PRIMAD: Information gained by different types of reproducibility. In: Reproducibility of Data-Oriented Experiments in e-Science (Seminar 16041). Vol. 6, Leibniz-Zentrum für Informatik, Schloss Dagstuhl. • Brinckman, A., Chard, K., Gaffney, N., Hategan, M., Jones, M.B., Kowalik, K., Kulasekaran, S., Ludäscher, B., Mecum, B.D., Nabrzyski, J. and Stodden, V., 2019. Computing environments for reproducibility: Capturing the “Whole Tale”. Future Generation Computer Systems, 94, pp.854-867. • McPhillips, Song, Kolisnik, Aulenbach, Belhajjame, Bocinsky, Cao, Cheney, Chirigati, Dey, Freire, Jones, Hanken, Kintigh, Kohler, Koop, Macklin, Missier, Schildhauer, Schwalm, Wei, Bieda, Ludäscher (2015). YesWorkflow: A User-Oriented, Language-Independent Tool for Recovering Workflow Information from Scripts. International Journal of Digital Curation (IJDC) 10, 298-313. • João Pimentel, Saumen Dey, Timothy McPhillips, Khalid Belhajjame, David Koop, Leonardo Murta, Vanessa Braganholo, Bertram Ludäscher. Yin & Yang: Demonstrating Complementary Provenance from noWorkflow & YesWorkflow. Intl. Workshop on Provenance and Annotation of Data and Processes (IPAW) LNCS 9672, 2016. • Craig Willis. Trust, but verify: An investigation of methods of verification and dissemination of computational research artifacts for transparency and reproducibility. PhD Thesis, University of Illinois, Urbana-Champaign, 2020. Reproducibility & Transparency 45