Results Vary: The Pragmatics of Reproducibility and Research Object Frameworks

ResultsVary:The Pragmatics of
Reproducibility and Research Object
Frameworks
Professor Carole Goble CBE FREng FBCS
The University of Manchester, UK
The Software Sustainability Institute
carole.goble@manchester.ac.uk
iConference, 26 March 2015, Newport Beach, Los Angeles, USA

What do I do? CyberInfrastructure EcoSystems.
e-Lab Collabs. &
Shared Asset
Repositories
Knowledge,
Metadata, Linked
Data, Ontologies
Software Engineering
for Scientists
Computational
Workflow Systems
Scholarly
Comms
Reproducibility
Micro
Publications
Open Science
Research
Objects
Linked Data for
Science

Scientific EgoSystems
Biodiversity
Systems Biology
Synthetic Biology
Astronomy
HelioPhysics
Genomics
Health
Epidemiology
Digital
Preservation
Social
Science
Pharmacology

KnowledgeTurning, Flow
Barriers to Cure
» Access to scientific
resources
» Coordination and
Collaboration
» Flow of Information
http://fora.tv/2010/04/23/Sage_Commons_Josh_Sommer_Chordoma_Foundation

[Pettifer, Attwood]
http://getutopia.com

Results Vary: The Pragmatics of Reproducibility and Research Object Frameworks

VirtualWitnessing*
Scientific publications:
» announce a result
» convince readers the result is correct
“papers in experimental [and computational
science] should describe the results and
provide a clear enough protocol [algorithm]
to allow successful repetition and extension”
Jill Mesirov, Broad Institute, 2010**
**Accessible Reproducible Research, Science 22January 2010,Vol. 327 no. 5964 pp. 415-416, DOI: 10.1126/science.1179653
*Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life (1985) Shapin and Schaffer.

Bramhall et al QUALITY OF METHODS REPORTING IN ANIMAL MODELS OF
COLITIS Inflammatory Bowel Diseases, , 2015,
“Only one of the 58 papers reported all essential
criteria on our checklist. Animal age, gender, housing
conditions and mortality/morbidity were all poorly
reported…..”
http://www.nature.com/news/male-researchers-stress-out-rodents-1.15106

“An article about computational science in a scientific
publication is not the scholarship itself, it is merely
advertising of the scholarship.The actual scholarship
is the complete software development
environment, [the complete data] and the complete
set of instructions which generated the figures.”
David Donoho, “Wavelab and Reproducible Research,” 1995
Datasets, Data collections
Standard operating procedures
Software, algorithms
Configurations,
Tools and apps, services
Codes, code libraries
Workflows, scripts
System software
Infrastructure
Compilers, hardware
Morin et al Shining Light into Black Boxes Science 2012: 336(6078) 159-160 , Ince et alThe case for open computer programs, Nature 482, 2012
50papers randomly chosen from 378
manuscripts in 2011 that use BurrowsWheeler
Aligner for mapping Illumina reads
31no s/w version, parameters, exact
version of genomic reference sequence
26no access to primary data sets
Nekrutenko &Taylor, Next-generation sequencing data interpretation: enhancing, reproducibility and accessibility, Nature Genetics 13 (2012)

Broken software Broken science
» GeoffreyChang, Scripps Institute
» Homemade data-analysis program
inherited from another lab
» Flipped two columns of data,
inverting the electron-density map
used to derive protein structure
» Retract 3 Science papers and 2
papers in other journals
» One paper cited by 364
The structures of MsbA (purple) and
Sav1866 (green) overlap little (left)
until MsbA is inverted (right).
Miller A Scientist's Nightmare: Software Problem Leads to Five Retractions Science 22 December 2006: vol. 314 no. 5807 1856-1857
http://www.software.ac.uk/blog/2014-12-04-its-impossible-conduct-research-without-software-say-7-out-10-uk-researchers

Software making practices
“As a general rule,
researchers do not
test or document their
programs rigorously,
and they rarely
release their codes,
making it almost
impossible to
reproduce and verify
published results
generated by
scientific software”
2000 scientists. J.E. Hannay et al., “How Do Scientists Develop and Use Scientific Software?” Proc. ICSEWorkshop Software Eng. for
Computational Science and Eng., 2009, pp. 1–8.

republic of science*
regulation of science
institution cores libraries
*Merton’s four norms of scientific behaviour (1942)
public services

Tools, Standards
Machine actionable,
Formats, Reporting,
Policies, Practices

Record and
Automate
Everything.
PotentialTrace
Heaven Folks!
recomputation.org
sciencecodemanifesto.org

Honest Error Science is messy
Inherent
Reinhart/Rogoff Austerity economics
Thomas Herndon
Nature Oct ’12
Zoë Corbyn
Fraud

“I can’t immediately reproduce the research in my own laboratory.
It took an estimated 280 hours for an average user to approximately
reproduce the paper.”
Prof Phil Bourne
Associate Director, NIH Big Data 2 Knowledge Program

When research goes “wrong”
»Tainted resources
»Black boxes
»Poor Reporting
»Unavailable resources /
results: data, software
»Bad maths
»Sins of omission
»Poor training, sloppiness
https://www.sciencenews.org/article/12-reasons-research-goes-wrong (adapted)
Ioannidis, Why Most Published Research Findings Are False, August 2005
Joppa, et al,TroublingTrends inScientificSoftwareUseSCIENCE 340 May 2013
Scientific method

Social environment
» Impact factor mania
» Pressure to publish
» Broken peer review
» Research never reported
» Disorganisation
» Time pressures
» Prep & curate costs
When research goes “wrong”
https://www.sciencenews.org/article/12-reasons-research-goes-wrong (adapted)
Morrison
Do a Replication Study?
No thanks! Not FAIR.
Hard. Resource intensive.
Unrecognised. Trolled.
Just gathering the bits together .

Cross-Institutional e-Laboratory Fragmentation
Scattered parts, Subject specific / General resources
101 Innovations in Scholarly Communication - the Changing ResearchWorkflow, Boseman and Kramer, 2015,

Process at Scale
More on Models

https://doi.org/10.15490/seek.1.investigation.56
[Snoep, 2015]

https://doi.org/10.15490/seek.1.investigation.56

Personal Data
Local Stores
External
Databases
Articles
Models
Standards
SOPs

Aggregated Commons Infrastructure
Consistent,Comparative Reporting
Design, protocols, samples, software,
models….
http://www.seek4science.org http://www.fair-dom.org http://isatools.org

Pop-Up Start Ups
Little Science within Big Science

How do Scientists Collaborate &
Cooperatively Exchange?
Cautiously.
Its all aboutTheTrust.
Extrinsic
Driver

How do you get Scientists and
Developers to work together?
Socially. Its all aboutTheTrust.
Jam today, Jam tomorrow, Jam for all, Just enough Jam Just inTime not Just in Case.

Research Objects
Compound Interconnected Investigations, Research Products
Multi-various
Products,
Platforms/Resources
Units of exchange, commons, contextual metadata
http://www.researchobject.org

First class citizens - data, software, methods
- id, manage, credit, track, profile, focus
A Framework to Bundle and Link (scattered) resources, related
experiments. Metadata Objects that carry Research Context
Research Objects

Bigger on the inside than the outside
Content
• closed <-> open
• local <-> alien
• embed <-> refer
• fixed <-> fluid
• nested
• cite? resolve? steward?
Contributions
• multi –typed, stewarded,
sited, authored
• span research, researchers,
platforms, time
• cite? resolve? steward?

Identity + Minimal Provenance
RO Resolution and Citation:
› Defend it (snapshot)
› Locate it (most recent)
› Reuse it (a version, a component)
› Credit it (contributory authorship)
› Cross link it (connections)
Biological Study Records (e.g. PRIDE): stable
Biological Knowledge (e.g. UNIPROT): evolving

Goble, De Roure, Bechhofer, Accelerating KnowledgeTurns, I3CK, 2013
means
ends
driver

Research Object packages codes, study,
and metadata to exchange descriptions
of clinical study cohorts, statistical
scripts, data. Farr ResearchObject
Commons
STELARAsthma e-Lab: StudyTeam for Early
Life Asthma Research
Platform exchange: ClinicalCodes.org coded
patient cohorts exchange with NHS FARSITE
system
STELAR e-Lab
Platform 1
Platform 2
Platform 3
A multi-site collaboration to
support safe use of patient and
research data for medical research
Research Object Currency
Cohort Studies

Focus on methods, models, workflows, scripts, software, data, figures….
Research Object Pivots and Profiles

Focus on the figure: F1000Research Living Figures,
versioned articles, in-article data manipulation
R Lawrence Force2015, Vision Award Runner Up http://f1000.com/posters/browse/summary/1097482
Simply data + code
Can change the definition of
a figure, and ultimately the
journal article
Colomb J and Brembs B.
Sub-strains of Drosophila Canton-S differ
markedly in their locomotor behavior [v1;
ref status: indexed, http://f1000r.es/3is]
F1000Research 2014, 3:176
Other labs can replicate the study, or
contribute their data to a meta-
analysis or disease model - figure
automatically updates.
Data updates time-stamped.
New conclusions added via versions.

Jennifer Schopf,Treating Data Like Software: A Case for Production Quality Data,JCDL 2012
Software-like Release paradigm
Not a static document paradigm
Reproduce looks backwards -> Release looks forwards
» Science, methods, data
change -> agile
evolution
» Comparisons , versions,
forks & merges,
dependencies
» Id & Citations
» Interlinked ROs

[McEntyre]
Retrospective Release Research Object

recompute
replicate
rerun
repeat
re-examine
repurpose
recreate
reuse
restore
reconstruct review
regenerate
revise
recycle
redo
What IS reproducibility?
Re: “do again”, “return to original state”
“show A is true by doing B”
verify but not falsify
[Yong, Nature 485, 2012]
robustness tolerance
verificationcompliance
validation assurance

RO as Instrument, Materials, Method
Input Data
Software
Output Data
Config
Parameters
Drummond, Replicability is not Reproducibility: Nor is it Good Science, online
Peng, Reproducible Research in Computational Science Science 2 Dec 2011: 1226-1227.

1. Science Changes. So does the Lab.
“The questions don’t
change but the
answers do”
Dan Reed
The lab is not fixed
Updated resources
UncertaintyBioSTIF

Zhao, et al .Why workflows break - Understanding and combating decay in
Taverna workflows, 8th Intl Conf e-Science 2012
2. Instruments Break, Labs Decay
materials become unavailable, technicians leave
Reproducibility Window
» Bit rot, Black boxes
» Proprietary Licenses
» Clown services*
» Partial replication
» Prepare to Repair
› form or function?
› preserve or sustain?
*Jason Scott

RO as Instrument, Materials, Method
Input Data
Software
Output Data
Config
Parameters
Methods
(techniques, algorithms,
spec. of the steps)
Materials
(datasets, parameters,
algorithm seeds)
Experiment
Instruments
(codes, services, scripts,
underlying libraries)
Laboratory
(sw and hw infrastructure,
systems software,
integrative platforms)
Setup
Drummond, Replicability is not Reproducibility: Nor is it Good Science, online
Peng, Reproducible Research in Computational Science Science 2 Dec 2011: 1226-1227.

Research Environment
submit article
and move on…
publish article
Publication
Environment

Research Environment
publish article
Publication
Environment
submit article
and move on…

[Adapted Freire, 2013]
transparency
dependencies
steps, features
provenance trace
portability
robustness
preservation
access
available
description
intelligible
standards
common APIs
licensing
standards
common
metadata
change management
versioning
packaging
Machine
actionable
Machine
actionable
Reproducibility Framework

submit article
and move on…
Reporting
Documentation
Provenance –
ThickTrace Data
to Distilled Reporting
Distillation
and
Summarisation
Alper P , et al LabelFlow: Exploiting Workflow Provenance to
Surface Scientific Data Provenance. IPAW 2014: 84-96;

Reproduce by Reading
Archived Record, Retain the Process/Code

The IT Crowd, Series 3, Episode 4
The eLabVirtual Machine* (or Docker Image**)
* a black box though
**docker.com
Reproduce by Running:
Active Instrument
Retain the bits

service
Science as a Service
Integrative frameworks
Open Source
Workflows/Scripts
Virtual Machines
Portable Packaging
Portability
Transparency

ReproZip
Workflows,makefiles
service
Science as a Service
Integrative frameworks
Open Source
Workflows/Scripts
Virtual Machines
Portable Packaging

Fifty Shades of Research Object
Workflow Instrument
Example data and
config.
Components.
Plug-ins,Versions
Workflow System Instrument
Software package
Workflow Runs
Data and
configs
Provenance
logs
Study
Shared Repository
Personal Notebook
Community Registry
Publishing Resource

Fifty Shades of Research Object
Workflow Instrument
Example data and
config.
Components.
Plug-ins,Versions
Workflow System Instrument
Software package
Workflow Runs
Data and
configs
Provenance
logs
Study

standards
Adobe
UCF
ORE PROVODF
formats
api
Instrument

http://www.cnri.reston.va.us/papers/OverviewDigitalObjectArchitecture.pdf
Instrument

NISO-JATS
Instrument
J Zhao,G Klyne, M Gamble,CA Goble -A Checklist-Based Approach
for QualityAssessment of Scientific Information
Proceedings of theThird Linked Science Workshop 2013

Platform profiles
NISO-JATS
Instrument

Container
Manifest
OMEX archive
https://researchobject.github.io/specifications/bundle/
Bergman et al COMBINE archive and OMEX format: one file to share all information to
reproduce a modeling project, BMC Bioinformatics 2014, 15:369
Retro-Fitted ROs
using off the shelf
platforms

Method Matters
Reproducibility Smarts
Commons not Repository
ResearchTardis
Retro-fit ROs
Do As Little As Possible
Make -> Born
Native RO platforms
RARE & FAIR KnowledgeTurns Means Research Objects

http://doctorwhosite1.weebly.com/sonic-screwdrivers.html
Researchers.
Silver bullet tools.
Psychic paper.
http://bowjamesbow.ca/2008/06/08/shhhhhhh-silenc.shtml
PI Team
RARE Research Reality Check!

Tribal Behaviour
» Gangs share, but not with the public
» Tribal behaviours
› Modellers share more than Experimentalists
› Experimentalists reuse models more than
Modellers
» Trading behaviours
› Collaboration – complementarity
correlations
» Structured consortia less likely to
publicly share than individuals
» Post-hoc rationalised Data/Model
Cycles
[Garza, 2014]

» Fluid, transient collaborations > “my
gang” management
» Shameless exploitation of head
teacher (PI) competitiveness & vanity
» Class captains (prefects)
» Get the cool kids on board.
» Head teacher leadership
[Garza, 2014]
Playground Rules

me
ME
my team
close
colleagues
peers
The Research Release Creep Spiral
» Data Hugging & Flirting.
» Reciprocity norms.
» HansW request.
» Dowry phenomenon.
» Private installations.
» Private spaces on shared
installations.
» Safe havens.

Too ugly to show anyone else.
Readers who have access will want user support.
No-one else would be interested/find it useful/be able to use it.
The code is too sophisticated for most readers/referees.
I didn't work out all the details.
I didn't actually write the code -- my student did.
My competitors would be unfair to me.
Its valuable intellectual property.
It would make papers much longer.
Referees would never agree to check the code.
My code invokes other code with unpublished (proprietary) code.
Randall J. LeVeque ,TopTen ReasonsTo Not ShareYour Code (and why you should anyway) April 2013 SIAM News
Victoria Stodden,AMP 2011 http://www.stodden.net/AMP2011/,

Drivers
love money
fame duty
fear time/
effort
shame duty
[Apologies to Resnick and Malone]

Stealthy not Sneaky
reduce the friction
instrumentation
span RARE and FAIR
OptimisingThe Neylon Equation

Interface Framing
» Limited scheduled sharing choices
› Never say never
» “Citable” not “Shared”
» Feedback
› Guilt tripping
› Outlier finger pointing
[Garzia]

Auto-magical end-to-end Instrumentation
https://www.youtube.com/watch?v=QVQwSOX5S08?
ELNs and
Authoring Platforms
Sweave

Credit ≠ Authorship
Research Currencies
“ResearchBitCoin”
Citation Semantics

Training
56%
Of UK researchers develop their own
research software or scripts
73% Of UK researchers have had no formal
software engineering training
Survey of researchers from 15 RussellGroup universities conducted by SSI between August - October 2014.
406 respondents covering representative range of funders, discipline and seniority.

http://www.rse.ac.uk
Instrument Artisans
[Shapin 84]

Make SoftwareVisible
[1960s Boeing 747-100 Software Configuration]
* Howison and Bullard 2014The visibility of software in the scientific literature: how do scientists mention software and how effective are those
mentions? J Assoc fo Info Science andTechnology In review
87% software findable
78% credit
37% formal citation 5% actual version
90 Bio articles
24% journals had citation policy

BUT……
two years time when the paper is written
reviewers want additional work
statistician wants more runs
analysis may need to be repeated
post-doc leaves, student arrives
new data, revised data
updated versions of algorithms/codes
sample was contaminated

Inspired by Bob Harrison
• Incremental shift for
infrastructure providers.
• Moderate shift for policy
makers and stewards.
• Paradigm shift for researchers
and their institutions.
The RO & Reproducibility Challenge

All the members of the Wf4Ever team
Colleagues in Manchester’s Information
Management Group
http://www.wf4ever-project.org
http://www.fair-dom.org
http://seek4science.org
http://rightfield.org.uk
http://www.software.ac.uk
http://www.datafairport.orgAlanWilliams
Jo McEntyre
Norman Morrison
Stian Soiland-Reyes
Paul Groth
Tim Clark
Juliana Freire
Alejandra Gonzalez-Beltran
Philippe Rocca-Serra
Ian Cottam
Susanna Sansone
Kristian Garza
Barend Mons
Sean Bechhofer
Philip Bourne
Matthew Gamble
Raul Palma
Jun Zhao
Neil Chue Hong
Josh Sommer
Matthias Obst
Jacky Snoep
David Gavaghan
Rebecca Lawrence

Contact…
Professor Carole Goble
The University of Manchester, UK
carole.goble@manchester.ac.uk
https://sites.google.com/site/carolegoble
@CaroleAnneGoble

Results Vary: The Pragmatics of Reproducibility and Research Object Frameworks

More Related Content

What's hot (20)

Similar to Results Vary: The Pragmatics of Reproducibility and Research Object Frameworks (20)

More from Carole Goble (20)

Recently uploaded (20)

Results Vary: The Pragmatics of Reproducibility and Research Object Frameworks