Data collection methods to improve reproducibility

Data collection methods to improve
reproducibility
Louis Culot, CEO, BioData

Published 1977
Main thesis:
Running = Good !
Fix dies at age 52 (running)
In 1984.
Two camps:
Running = Good !
Running = maybe good…
moderation
too much of a good thing?

Conclusions The findings suggest a U-shaped association between all-cause mortality
and dose of jogging as calibrated by pace, quantity, and frequency of jogging. Light
and moderate joggers have lower mortality than sedentary nonjoggers, whereas
strenuous joggers have a mortality rate not statistically different from that of the
sedentary group.
UHow much running
Bad Bad
Great !
Schnohr, Peter, et al. "Dose of jogging and long-term mortality: the
Copenhagen City Heart Study." Journal of the American College of Cardiology
65.5 (2015): 411-419.

Why Jogging May Be Better For Your Health Than
Running
So the connection was this: Joggers of mild and moderate intensity had a lower risk of
death than the strenuous joggers. In fact, the lowest mortality risk was that of the
mild intensity joggers. The fast-paced joggers had about the same rate of mortality as
sedentary people. This suggests that there may be an upper limit to in vigorous
exercise, after which the benefits fall off.

A recent study reported
that joggers who exercise
strenuously have the same
life expectancy as people
who do barely any exercise
at all. But the author has
now admitted he hasn't
actually proved this.
But, although the overall
number of people studied
was large, the number of
strenuous joggers was not.
Only 36 people fitted the
strenuous jogging category
- two of them had died.

State of Reproducibility
-Amgen study: Only 6 of the 53 studies were
reproduced (about 11%).1
-Bayer study: Only 14 out of the 67 projects
(about 21%).2
-From 28% to 18% for clinical trials
2006-2007 – 2008-2010.3
”…poor training, an emphasis on provocative
conclusions in papers, a dearth of experimental
details, and an overemphasis on publications in
high-impact journals” – Francis Collins, NIH
Director
1. Begley, C. Glenn, and Ellis, Lee M., "Drug development: Raise standards for preclinical cancer research." Nature 483.7391
(2012): 531-533
2. Prinz, Florian, Thomas Schlange, and Khusru Asadullah. "Believe it or not: how much can we rely on published data on
potential drug targets?." Nature reviews Drug discovery 10.9 (2011): 712-712.
3. Nature Reviews Drug Discovery 10, 328-329 2011

Examples of Recommendations:
“We recommend the following steps to change the culture of oncology research and
improve the relevance of translational studies” (Begley and Ellis):
There must be more opportunities to present negative data. It should be the expectation that negative preclinical data will be
presented at conferences and in publications. Preclinical investigators should be required to report all findings, regardless of the
outcome. To facilitate this, funding agencies, reviewers and journal editors must agree that negative data can be just as
informative as positive data.
Journal editors must play an active part in initiating a cultural change. There must be mechanisms to report negative data that
are accessible through PubMed or other search engines. There should be links to journal articles in which investigators have
reported alternative findings to those in an initial (sometimes considered landmark) publication. One suggestion is to include
'tags' that report whether the key findings of a seminal paper were confirmed.
Other: Reporting unethical behavior, dialog between clinicians and patients and scientists; universities prioritizing teaching in
tenure decisions (research is too “high stakes”). Better access to technologies – e.g., new cell lines, capabilities for genetic
characterization of new tumor cell lines and xenografts.

“5 Practical Points to Consider”
Poland, C. A., Miller, M. R., Duffin, R., & Cassee, F. (2014). The elephant in the room: reproducibility in toxicology.
Particle and fibre toxicology, 11(1), 42.
1. Be specific.
2. Characterize, Characterize, Characterize
3. Use statistics to question your results rather than simply confirming a theory
4. Write your hypothesis in advance of running the experiment.
5. Take a ‘belts and suspenders’ approach to confirming findings.

1. Be specific.
For example, what is your sample? If it’s graphene, then is it actually graphene i.e. a monolayer, or is it few
layer graphene, or graphite platelets? If they are ambient particles, where and how were they collected (date,
time, weather conditions etc.)? State this, it is important - later studies may not replicate your results simply
due to inadvertently testing the non-similar materials i.e. comparing apples and oranges.
=?

1. Be specific.
Only with independent characterisation do you truly know what you are testing and allow others to reproduce
your findings. …. characterise in relation to your hypothesis or research question - do not characterise because
there is a default list of parameters that you need to check.
=? +
1. Sorge, Robert E., et al. "Olfactory exposure to males, including men, causes stress and related analgesia
in rodents." Nature methods 11.6 (2014): 629-632.
JAX CD34 JAX CD34

Ask yourself, is barely statistically significant really biologically relevant? Can you support/validate an entire
hypothesis or more controversially, a provocative statement using P<0.05?
1. Be specific.
Revisit the power of negative results

The Economist
‘Trouble at the
Lab’ 2013 Oct 19;
available at
http://go.nature.
com/dstij3

The Economist
‘Trouble at the
Lab’ 2013 Oct 19;
available at
http://go.nature.
com/dstij3
“The root of all superstition is that men observe
when a thing hits, but not when it misses.”
…Francis Bacon

1. Be specific.
It should be clear from the experimental design how you have tried to disprove this. If your theory is robust it
should survive your best efforts to challenge it.

1. Be specific.
5. Take a ‘belts and suspenders’ approach to confirming findings.
Replicate time points, dosages, protocols, etc.
IC50 = 3.25 μl/mg
vs
vs
Researcher 1
Researcher 2
Researcher 3

How labs use Labguru
- Biospecimen management / sample tracking.
- Project and experiment planing.
- Sharing of data (internally and externally).
- Managing collaborations.
- Communicating with funders.
- In-house repository for raw data and analysis.

What does it enable ?
Science is struggling with reproducibility
Scientific traceability
0
1
Ability to build your own and others’ work
Communication as part of the experiment
0
2
0
3

It starts with how scientists manage their
research.
What affects reproducibility?
It starts with scientists
1. Protocol Management:
Lost protocols, inconsistent protocol updates, lab members
using different protocols, and a lack of protocol –
experimental and materials linkage.
2. Reagent and Specimen Details:
Reagent – experiment coupling. Reagent conditions.
Location of reagents. Inconsistent procedures for reagent
synthesis. Genealogy.
3. Access to negative results:
Often more reliable, and containing valuable information to
develop new hypotheses.
4. Experimental Design:
Disconnected data, lack of scientific context, and an
incomplete picture make it hard for other scientists to
reproduce work.
“a problem well put is
half solved.”… John Dewey

1. Protocol Management
• Lost protocols
• Inconsistent protocol updates
• Lab members using different
protocols
• Lack of protocol - experimental
linkage
The problem:
• Keep track of all protocols used
• Designate & protocols between lab
members
• Keep a record of what versions were
used for which experiments
• Annotate protocols as they are executed.
The solution:
• Protocol Versioning
• Built in protocol versioning and tracking
system
• Link to experiments for full scientific context
The Labguru Approach:

2. Reagent and Specimen Details
• Reagent – experiment coupling
• Reagent conditions
• Specimen meta-data and provenance.
• Location, origin, date, and handling.
The problem:
• Record & track all successful reagents
and specimens along with their metadata
• Link specific specimens & reagents
back to individual experiments
• Identify problems from experimental
records
The solution:
• Specimen and Reagent Tracking
• Purchased materials are linked to experiments which use them
• Smart modules for plasmids, cell lines bacteria, tube storage, and
more

3. Access to Negative Results
• Negative results often ‘lost’ over
time in laboratories. Never
published.
The problem:
• Ready access to all project and
experiments regardless of outcome.
• Ability to back-test and build new
hypotheses.
• Increase ability to rescue failed
experiments.
The solution:
• All data available within a lab (under PI control).
• Data can be selectively shared and pushed to public repositories.
• Simple ability to save “all data” in one place (including linking to big data, such as
genomics and images).

4. Experimental Design
• Disconnected data
• Lack of scientific context
• Incomplete picture
• Loss of data when someone
leaves the lab
The problem:
• Begin with hypothesis testing.
• Create detailed mini-reports that encompass
all relevant data for an experiment
• Connect related experiments and
assets/materials.
• Selectively reproduce “difficult” parts of
experiments.
The solution:
• Platform for Searchability and Collaboration
• Access data based on researcher, materials used, project,
keywords
• Integrated commenting and activity feeds increase accessibility

Versioning
Tracking
Sharing
history
Provenance
Genealogy
Vendors
Experiments &
Protocols
Calibration
Experiments
Vendor
Results (e.g., drift)
Change history
Duplicates
Results
Contributors

The Digital Laboratory Ecosystem
Labguru
Lab Logistics/Notebook

Labguru
Physical Materials/Specimens
Equipment
Big data servers
Genomics
Imaging

Labguru
Physical Materials/Specimens
Equipment
Big data servers
General Repositories
Domain Repositories
Genomics
Imaging
Manuscripts

Please contact us with any questions
Visit labguru.com for a 30 day free trial of Labguru
THANK YOU FOR COMING!

Data collection methods to improve reproducibility

More Related Content

Similar to Data collection methods to improve reproducibility (20)

Recently uploaded (20)

Data collection methods to improve reproducibility

Editor's Notes