Challenges and Guidelines for Reproducible Research with Jupyter Notebook
1 like330 views
The document discusses challenges in making research reproducible using Jupyter Notebooks and proposes guidelines to address these challenges. The guidelines include telling a story by showing the workflow from data to results, capturing the entire workflow by structuring notebooks with a beginning, middle, and end, avoiding copy and pasting by refactoring common code into importable functions, removing clutter by organizing notebooks with markdown sections and splitting long notebooks, making notebooks reproducible by specifying dependencies and versions, using version control like GitHub, and sharing notebooks by using services like Binder and Nbviewer.
![Challenges and Guidelines for Reproducible Research
and Interactive Education with Jupyter Notebook
Shih-Cheng Huang, Niema Moshiri, Michael Reich, Peter W. Rose
In [2]:
In [3]:
Introduction
Jupyter Notebooks have the potential to make research more
reproducible. However, in practice, many notebooks fall short of this
promise. Here we identify challenges and propose guidelines to
organize, document, and deploy notebooks to increase reproducibility
and reusability. These guidelines also apply to instructional materials.
Jupyter Notebooks are extensively used in research and education.
We recently organized a workshop at UC San Diego and invited
students, postdocs, research scientists, and faculty to share
experiences and identify challenges of using Jupyter notebooks. The
workshop covered the use of Jupyter in many disciplines, ranging from
Astrophysics, Bioinformatics, Datascience, Genomics, Medicine, to
Structural Bioinformatics, as well as classroom use with hundreds of
students and publication of reproducible science. Here we present a
summary of our findings in the form of guidelines.
Tell a
Story
•Show how you got from
initial data to final results
•Avoid manual steps, use a
notebook for data
preparation
•Split workflow into steps,
e.g., data preparation,
data analysis, data
visualization
•Show intermediate results
so users can follow the
dataflow
•Challenges
How to combine
multiple notebooks into
a complex workflow
Capture
Entire
Workflow
•Like any good story, a
notebook should have a
• Beginning
• Introduce the topic and the
aims of the notebook
• Middle
• Explain the steps of the
workflow
• Use markdown to split
notebook into sections
• End
• Interpret the results
• What trends do tables, plots,
or figures show?
•Write notebook for an
audience
• New users may need
instructions how to run a
notebook
• Educational materials need
background and step by step
explanations
In [1]:
Avoid
Copy &
Paste
•Split common functions
into separate files and
import them into
notebooks
•Challenges
•Need tools to refactor
notebooks
•Find and extract
common code among
related notebooks
Remove
Clutter
In [4]:
•Use markdown to organize a
notebook into sections
•Split long notebooks into a
series of notebooks
• Keep a top level notebook with
links to the individual notebooks
•Avoid long cells
• Split text and code into cells
• One cell -> one paragraph or
one task (e.g., create a plot)
•Modularize code by defining
functions or classes
•Reuse code by importing
functions or classes
•Put low-level documentation in
code comments
•Challenges
• Need to be able to collapse
sections of text or code to hide
low-level details, e.g., setup for
a plot
In [5]:
Make it
Repro-
ducible
•Specify version numbers of
dependencies
•Always specify the
source/location of the data
•Keep a copy of the raw data if
possible
•Make copies of data in a
notebook to avoid corrupting
datasets when running cells out
of order
•Specify random number seeds
•Before saving a notebook, rerun
notebook to ensure linear
execution order
•Challenges
• Need functionality to freeze
current state, execution order
In [7]:
Use
Version
Control
•Keep your notebooks under
version control, e.g., GitHub
•Describe content of repository in
README files
•Specify a license to encourage
and enable use by others
•Structure your repository
• See for example
http://drivendata.github.io/cookie
cutter-data-science/
•Challenges
• Each time notebook is run, the
ipynb file is modified,
• Need a ”diff” tool for notebooks
In [6]:
Share
It!
•Use open source projects as your
dependencies
•Add a liberal open source license
(e.g., MIT, Apache 2) to your
repository
•Use Nbviewer to provide static
views of your executed notebook
https://nbviewer.jupyter.org/
•Use Binder to provide a zero-
install environment to run your
notebooks in the cloud
https://mybinder.org/
•Create a Docker image of your
environment
https://docs.docker.com/
•Challenges
• Data intensive applications
• Compute intensive applications
• Special hardware requirements,
e.g., GPU
• Multi-step workflows
References
- Jupyter Notebooks – a publishing format for reproducible computational workflows
(2016) Jupyter Dev. Team, IOS Press, doi: 10.3233/978-1-61499-649-1-87
- Exploration and Explanation in Computational Notebooks, A. Rule, et al. (2018) Proc.
of the 2018 CHI Conference on Human Factors in Computing Systems, ACM.
- Binder 2.0 - Reproducible, interactive, sharable environments for science at scale,
Project Jupyter, et al. (2018) Proc. of the 17th Python in Science Conf. (SCIPY 2018).
- The GenePattern Notebook Environment, M. Reich, et al. (2017) Cell Systems 5.2,
149-151.
Acknowledgements
Amanda Birmingham, Ilkay Altintas, Rob Knight, Tiago Leao, Nathan Mih, Mai Nguyen,
Shweta Purawat, Brin Rosenthal, Adam Rule, Britton Smith, Shuai Tang, Guorong Xu
Image credit: 1. http://higher-ed.us/wp-content/uploads/2017/12/copy-and-paste-pictures-10-unbelievable-why-a-blogger-should-never-webmasters-nigeria.jpg, 2. https://blog.prototypr.io/meet-overflow-9b2d926b6093, 3. http://romeo.landinez.co/workflow/rapid-workflow-protoype.html, 4. https://productivitysteps.files.wordpress.com/2016/09/clutter.jpg
In [*]:
We are crowdsourcing a Jupyter Guide for
reproducible research. Please help us at:
https://github.com/sbl-sdsc/jupyter-guide](https://image.slidesharecdn.com/jupyterconposter-180819202730/85/Challenges-and-Guidelines-for-Reproducible-Research-with-Jupyter-Notebook-1-320.jpg)
Challenges and Guidelines for Reproducible Research with Jupyter Notebook
- 1. Challenges and Guidelines for Reproducible Research and Interactive Education with Jupyter Notebook Shih-Cheng Huang, Niema Moshiri, Michael Reich, Peter W. Rose In [2]: In [3]: Introduction Jupyter Notebooks have the potential to make research more reproducible. However, in practice, many notebooks fall short of this promise. Here we identify challenges and propose guidelines to organize, document, and deploy notebooks to increase reproducibility and reusability. These guidelines also apply to instructional materials. Jupyter Notebooks are extensively used in research and education. We recently organized a workshop at UC San Diego and invited students, postdocs, research scientists, and faculty to share experiences and identify challenges of using Jupyter notebooks. The workshop covered the use of Jupyter in many disciplines, ranging from Astrophysics, Bioinformatics, Datascience, Genomics, Medicine, to Structural Bioinformatics, as well as classroom use with hundreds of students and publication of reproducible science. Here we present a summary of our findings in the form of guidelines. Tell a Story •Show how you got from initial data to final results •Avoid manual steps, use a notebook for data preparation •Split workflow into steps, e.g., data preparation, data analysis, data visualization •Show intermediate results so users can follow the dataflow •Challenges How to combine multiple notebooks into a complex workflow Capture Entire Workflow •Like any good story, a notebook should have a • Beginning • Introduce the topic and the aims of the notebook • Middle • Explain the steps of the workflow • Use markdown to split notebook into sections • End • Interpret the results • What trends do tables, plots, or figures show? •Write notebook for an audience • New users may need instructions how to run a notebook • Educational materials need background and step by step explanations In [1]: Avoid Copy & Paste •Split common functions into separate files and import them into notebooks •Challenges •Need tools to refactor notebooks •Find and extract common code among related notebooks Remove Clutter In [4]: •Use markdown to organize a notebook into sections •Split long notebooks into a series of notebooks • Keep a top level notebook with links to the individual notebooks •Avoid long cells • Split text and code into cells • One cell -> one paragraph or one task (e.g., create a plot) •Modularize code by defining functions or classes •Reuse code by importing functions or classes •Put low-level documentation in code comments •Challenges • Need to be able to collapse sections of text or code to hide low-level details, e.g., setup for a plot In [5]: Make it Repro- ducible •Specify version numbers of dependencies •Always specify the source/location of the data •Keep a copy of the raw data if possible •Make copies of data in a notebook to avoid corrupting datasets when running cells out of order •Specify random number seeds •Before saving a notebook, rerun notebook to ensure linear execution order •Challenges • Need functionality to freeze current state, execution order In [7]: Use Version Control •Keep your notebooks under version control, e.g., GitHub •Describe content of repository in README files •Specify a license to encourage and enable use by others •Structure your repository • See for example http://drivendata.github.io/cookie cutter-data-science/ •Challenges • Each time notebook is run, the ipynb file is modified, • Need a ”diff” tool for notebooks In [6]: Share It! •Use open source projects as your dependencies •Add a liberal open source license (e.g., MIT, Apache 2) to your repository •Use Nbviewer to provide static views of your executed notebook https://nbviewer.jupyter.org/ •Use Binder to provide a zero- install environment to run your notebooks in the cloud https://mybinder.org/ •Create a Docker image of your environment https://docs.docker.com/ •Challenges • Data intensive applications • Compute intensive applications • Special hardware requirements, e.g., GPU • Multi-step workflows References - Jupyter Notebooks – a publishing format for reproducible computational workflows (2016) Jupyter Dev. Team, IOS Press, doi: 10.3233/978-1-61499-649-1-87 - Exploration and Explanation in Computational Notebooks, A. Rule, et al. (2018) Proc. of the 2018 CHI Conference on Human Factors in Computing Systems, ACM. - Binder 2.0 - Reproducible, interactive, sharable environments for science at scale, Project Jupyter, et al. (2018) Proc. of the 17th Python in Science Conf. (SCIPY 2018). - The GenePattern Notebook Environment, M. Reich, et al. (2017) Cell Systems 5.2, 149-151. Acknowledgements Amanda Birmingham, Ilkay Altintas, Rob Knight, Tiago Leao, Nathan Mih, Mai Nguyen, Shweta Purawat, Brin Rosenthal, Adam Rule, Britton Smith, Shuai Tang, Guorong Xu Image credit: 1. http://higher-ed.us/wp-content/uploads/2017/12/copy-and-paste-pictures-10-unbelievable-why-a-blogger-should-never-webmasters-nigeria.jpg, 2. https://blog.prototypr.io/meet-overflow-9b2d926b6093, 3. http://romeo.landinez.co/workflow/rapid-workflow-protoype.html, 4. https://productivitysteps.files.wordpress.com/2016/09/clutter.jpg In [*]: We are crowdsourcing a Jupyter Guide for reproducible research. Please help us at: https://github.com/sbl-sdsc/jupyter-guide