SlideShare a Scribd company logo
Reproducibility workshop 26 Nov
http://bizlab.unsw.edu.au/website/conference-2015.php
A new era of
large-scale replication and collaboration?
Alex.Holcombe@sydney.edu.au
@ceptional
http://www.slideshare.net/holcombea/
Reproducibility workshop 26 Nov
http://bizlab.unsw.edu.au/website/conference-2015.php
False positives zone
Look for areas wherein victory declared whenever
null hypothesis is rejected, where size of effect,
plausibility considered unimportant.
Where science is not self-correcting
Reproducibility workshop 26 Nov
http://bizlab.unsw.edu.au/website/conference-2015.php
"This one's Psych Science. 1998. Put
your money on ‘will replicate’?”
"What, you think I was born
yesterday?”
"Fine," muttered the bookie.
#AcademicNoir
Reproducibility workshop 26 Nov
http://bizlab.unsw.edu.au/website/conference-2015.php
Well, Mr. P. Hacker, your luck is about to run out…
down at the journals, they’re running a new game.
They call it ‘preregistration’.
Reproducibility workshop 26 Nov
http://bizlab.unsw.edu.au/website/conference-2015.php
5
https://osf.io/tvyxz/wiki/home/
Reproducibility workshop 26 Nov
http://bizlab.unsw.edu.au/website/conference-2015.php
6
https://osf.io/tvyxz/wiki/home/
Reproducibility workshop 26 Nov
http://bizlab.unsw.edu.au/website/conference-2015.php
7
https://osf.io/tvyxz/wiki/home/
100% P-hacking Free
Here, check our
numbers.
Here’s how you
can replicate our
result.
Reproducibility workshop 26 Nov
http://bizlab.unsw.edu.au/website/conference-2015.php
"This one's Psych Science. 1998. Put
your money on ‘will replicate’?”
"What, you think I was born
yesterday?”
"Fine," muttered the bookie.
Reproducibility workshop 26 Nov
http://bizlab.unsw.edu.au/website/conference-2015.php 9
Reproducibility workshop 26 Nov
http://bizlab.unsw.edu.au/website/conference-2015.php 10
Failed to replicate my result?
Have you
considered
the unknown
moderators?
Reproducibility workshop 26 Nov
http://bizlab.unsw.edu.au/website/conference-2015.php 11
Any one of us alone, he’ll say we messed up.
But if we all run the replication together, we
can show the big guy we’re right.
Reproducibility workshop 26 Nov
http://bizlab.unsw.edu.au/website/conference-2015.php
Meta-analysis forest plot
Reproducibility workshop 26 Nov
http://bizlab.unsw.edu.au/website/conference-2015.php
Meta-analysis forest plot
Failed to replicate my
result?
Have you considered
the unknown
moderators?
You messed
up!
Reproducibility workshop 26 Nov
http://bizlab.unsw.edu.au/website/conference-2015.php
Reproducibility workshop 26 Nov
http://bizlab.unsw.edu.au/website/conference-2015.php
Reproducibility workshop 26 Nov
http://bizlab.unsw.edu.au/website/conference-2015.php
Reproducibility workshop 26 Nov
http://bizlab.unsw.edu.au/website/conference-2015.php
Preregistration
Large-scale replication
Open data
Open protocol and code
A new era of
large-scale replication
and collaboration?
Can we do this bottom-up?
Reproducibility workshop 26 Nov
http://bizlab.unsw.edu.au/website/conference-2015.php
Reproducibility workshop 26 Nov
http://bizlab.unsw.edu.au/website/conference-2015.php

More Related Content

PDF
Reproducibility, preregistration, etc.: Making good science even better
PPTX
VI Estudio Anual Mobile Marketing 2014
DOCX
Tugas 3
DOC
PROTOCOLO PRIMERA ACTIVIDAD
PDF
volume of my thesis in English
PPTX
NFC tech
PPTX
USAGE OF DATABASE IN SOFTWARE DEVELOPMENT
PDF
Datapowercommonusecases 130509114200-phpapp02
Reproducibility, preregistration, etc.: Making good science even better
VI Estudio Anual Mobile Marketing 2014
Tugas 3
PROTOCOLO PRIMERA ACTIVIDAD
volume of my thesis in English
NFC tech
USAGE OF DATABASE IN SOFTWARE DEVELOPMENT
Datapowercommonusecases 130509114200-phpapp02

Similar to Unsw reproducibility workshop (20)

PDF
open access, p-hacking, registered reports, registered replication reports
PPTX
10 Recommendations from the Reproducibility Crisis in Psychological Science
PPTX
Open Data and the Social Sciences - OpenCon Community Webcast
PPTX
Excursions into the garden of the forking paths
PDF
Fixing Science: The Replicability Crisis
PDF
Bosman and Kramer Open Research: A 2024 NISO Training Series, Session Three: ...
PPTX
How do we make science better?
PPTX
The Reproducibility Crisis in Psychological Science: One Year Later
PPT
Replication Crisis in Psycholhhhyogy.ppt
PDF
Burger_SSIB_Open_Sci_NutriXiv_7_2019_draft
PPTX
Ruminations on replication
PDF
Aslin.discussion
PPTX
Improve your study with pre-registration
PDF
Five selfish reasons to work reproducibly
PDF
What is the reproducibility crisis in science and what can we do about it?
PPTX
Sharing, Reproducibility, Replication – AN NIH View
PPTX
Bias and the Data Lifecycle
PDF
Open Science in Psychology
PPTX
Open experiments and open-source
PDF
Crises of confidence and publishing reforms: What every social psychologist n...
open access, p-hacking, registered reports, registered replication reports
10 Recommendations from the Reproducibility Crisis in Psychological Science
Open Data and the Social Sciences - OpenCon Community Webcast
Excursions into the garden of the forking paths
Fixing Science: The Replicability Crisis
Bosman and Kramer Open Research: A 2024 NISO Training Series, Session Three: ...
How do we make science better?
The Reproducibility Crisis in Psychological Science: One Year Later
Replication Crisis in Psycholhhhyogy.ppt
Burger_SSIB_Open_Sci_NutriXiv_7_2019_draft
Ruminations on replication
Aslin.discussion
Improve your study with pre-registration
Five selfish reasons to work reproducibly
What is the reproducibility crisis in science and what can we do about it?
Sharing, Reproducibility, Replication – AN NIH View
Bias and the Data Lifecycle
Open Science in Psychology
Open experiments and open-source
Crises of confidence and publishing reforms: What every social psychologist n...
Ad

Recently uploaded (20)

PPTX
Introduction to Fisheries Biotechnology_Lesson 1.pptx
PPTX
EPIDURAL ANESTHESIA ANATOMY AND PHYSIOLOGY.pptx
PPTX
2. Earth - The Living Planet Module 2ELS
PDF
Biophysics 2.pdffffffffffffffffffffffffff
PDF
HPLC-PPT.docx high performance liquid chromatography
PPTX
Cell Membrane: Structure, Composition & Functions
PPTX
G5Q1W8 PPT SCIENCE.pptx 2025-2026 GRADE 5
PPTX
ECG_Course_Presentation د.محمد صقران ppt
PPTX
famous lake in india and its disturibution and importance
PDF
Formation of Supersonic Turbulence in the Primordial Star-forming Cloud
PPTX
TOTAL hIP ARTHROPLASTY Presentation.pptx
PDF
VARICELLA VACCINATION: A POTENTIAL STRATEGY FOR PREVENTING MULTIPLE SCLEROSIS
PPTX
Classification Systems_TAXONOMY_SCIENCE8.pptx
PDF
The scientific heritage No 166 (166) (2025)
PDF
Placing the Near-Earth Object Impact Probability in Context
PDF
AlphaEarth Foundations and the Satellite Embedding dataset
DOCX
Viruses (History, structure and composition, classification, Bacteriophage Re...
PPTX
ognitive-behavioral therapy, mindfulness-based approaches, coping skills trai...
PPTX
GEN. BIO 1 - CELL TYPES & CELL MODIFICATIONS
PPTX
Introduction to Cardiovascular system_structure and functions-1
Introduction to Fisheries Biotechnology_Lesson 1.pptx
EPIDURAL ANESTHESIA ANATOMY AND PHYSIOLOGY.pptx
2. Earth - The Living Planet Module 2ELS
Biophysics 2.pdffffffffffffffffffffffffff
HPLC-PPT.docx high performance liquid chromatography
Cell Membrane: Structure, Composition & Functions
G5Q1W8 PPT SCIENCE.pptx 2025-2026 GRADE 5
ECG_Course_Presentation د.محمد صقران ppt
famous lake in india and its disturibution and importance
Formation of Supersonic Turbulence in the Primordial Star-forming Cloud
TOTAL hIP ARTHROPLASTY Presentation.pptx
VARICELLA VACCINATION: A POTENTIAL STRATEGY FOR PREVENTING MULTIPLE SCLEROSIS
Classification Systems_TAXONOMY_SCIENCE8.pptx
The scientific heritage No 166 (166) (2025)
Placing the Near-Earth Object Impact Probability in Context
AlphaEarth Foundations and the Satellite Embedding dataset
Viruses (History, structure and composition, classification, Bacteriophage Re...
ognitive-behavioral therapy, mindfulness-based approaches, coping skills trai...
GEN. BIO 1 - CELL TYPES & CELL MODIFICATIONS
Introduction to Cardiovascular system_structure and functions-1
Ad

Unsw reproducibility workshop

  • 1. Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php A new era of large-scale replication and collaboration? Alex.Holcombe@sydney.edu.au @ceptional http://www.slideshare.net/holcombea/
  • 2. Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php False positives zone Look for areas wherein victory declared whenever null hypothesis is rejected, where size of effect, plausibility considered unimportant. Where science is not self-correcting
  • 3. Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php "This one's Psych Science. 1998. Put your money on ‘will replicate’?” "What, you think I was born yesterday?” "Fine," muttered the bookie. #AcademicNoir
  • 4. Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php Well, Mr. P. Hacker, your luck is about to run out… down at the journals, they’re running a new game. They call it ‘preregistration’.
  • 5. Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php 5 https://osf.io/tvyxz/wiki/home/
  • 6. Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php 6 https://osf.io/tvyxz/wiki/home/
  • 7. Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php 7 https://osf.io/tvyxz/wiki/home/ 100% P-hacking Free Here, check our numbers. Here’s how you can replicate our result.
  • 8. Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php "This one's Psych Science. 1998. Put your money on ‘will replicate’?” "What, you think I was born yesterday?” "Fine," muttered the bookie.
  • 9. Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php 9
  • 10. Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php 10 Failed to replicate my result? Have you considered the unknown moderators?
  • 11. Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php 11 Any one of us alone, he’ll say we messed up. But if we all run the replication together, we can show the big guy we’re right.
  • 12. Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php Meta-analysis forest plot
  • 13. Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php Meta-analysis forest plot Failed to replicate my result? Have you considered the unknown moderators? You messed up!
  • 14. Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php
  • 15. Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php
  • 16. Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php
  • 17. Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php Preregistration Large-scale replication Open data Open protocol and code A new era of large-scale replication and collaboration? Can we do this bottom-up?
  • 18. Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php
  • 19. Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php