The Scientific Filesystem
0 likes66 views
The document discusses the Scientific Filesystem (SCIF) specification for organizing scientific applications and software in a container-agnostic way. It provides an overview of SCIF goals and components, describes the SCIF recipe format for packaging applications, and demonstrates running a Snakemake workflow across multiple container technologies using SCIF.
![%apprun hello-world
/bin/bash hello-world.sh
%appinstall hello-world
echo "echo 'Hello World!'" >> $SCIF_APPBIN/hello-world.sh
chmod u+x $SCIF_APPBIN/hello-world.sh
%appenv hello-world
THEBESTAPP=$SCIF_APPNAME
export THEBESTAPP
%applabels hello-world
MAINTAINER Vanessasaur
%apphelp hello-world
This is an example "Hello World" application.
You can install it to a Scientific Filesystem
(scif) with the command:
scif install hello-world.scif
hello-world.scif
https://sci-f.github.io/tutorial-quick-start
%app[action] [name]](https://image.slidesharecdn.com/scientificfilesystemsochat1-191212165613/85/The-Scientific-Filesystem-46-320.jpg)
![%appenv hello-world
OMG=TACOS
export OMG
./scif exec hello-world
docker run vanessa/scif:hw exec hello-world echo [e]OMG
./scif-cli exec hello-world echo [e]OMG](https://image.slidesharecdn.com/scientificfilesystemsochat1-191212165613/85/The-Scientific-Filesystem-54-320.jpg)
![%appenv hello-world
OMG=TACOS
export OMG
./scif exec hello-world
docker run vanessa/scif:hw exec hello-world echo [e]OMG
./scif-cli exec hello-world echo [e]OMG](https://image.slidesharecdn.com/scientificfilesystemsochat1-191212165613/85/The-Scientific-Filesystem-55-320.jpg)
![/foo
/bin
/lib
/scif
runscript
runscript.help
labels.json
environment.sh
/scif
/apps
./container exec bar [command]
Interaction](https://image.slidesharecdn.com/scientificfilesystemsochat1-191212165613/85/The-Scientific-Filesystem-67-320.jpg)
![./container run bar [args]
/scif
/apps
/foo
/bin
/lib
/scif
runscript
runscript.help
labels.json
environment.sh
Interaction](https://image.slidesharecdn.com/scientificfilesystemsochat1-191212165613/85/The-Scientific-Filesystem-68-320.jpg)
The Scientific Filesystem
- 2. “ Welcome! 2
- 3. “ Welcome! 3
- 5. “ Let’s all be friends! 5
- 6. Once upon a time… There was a scientist
- 10. Foo Bar foO baR
- 12. 12 science +
- 17. Give it a try?
- 18. foo bar
- 20. “ What about the human component of reproducibility? 20
- 24. “ Where are foo and bar? 24
- 26. foo bar content metadata control ENTRYPOINT exec ./foobar
- 27. foo bar content metadata control ENTRYPOINT exec ./foobar ./foo./bar
- 28. foo bar content metadata control ENTRYPOINT exec ./foobar ./foo./bar bar foo
- 29. “ I just want to run bar. 29
- 30. foo bar Solution 1: More containers
- 31. foo bar Solution 1: More containers
- 32. If foo: exec foo else If bar: exec bar else: exec foobar Solution 2: Custom entrypoint
- 33. If foo: exec foo else If bar: exec bar else: exec foobar Solution 2: Custom entrypoint BESTAPP=FOO BESTAPP=BAR EXPORT BESTAPP
- 34. bar foo
- 35. “ The Linux Filesystem Hierarchy was not optimized for scientific applications. 35
- 36. foo control
- 38. global content foo metadata control foo bar bar metadata
- 39. “ Let’s make a Scientific Filesystem! 39
- 40. What is SCIF? ▪ The specification defines a filesystem structure and environment variable namespace that interact with functions to produce discoverable software applications. 40
- 41. Client A controller for a SCIF, either for a developer or a user 41 A third party software or tool that understands the SCIF structure and interacts with all or some portion of it Integration
- 42. Goals ● Enable users to easily generate predictable, discoverable applications ● A recipe format that maps to a SCIF, and vice versa ● Software to generate and manage SCIF 42
- 43. The Scientific Filesystem (SCIF) explicitly does not implement, define, or provide in v0.2: ● Package Management ● Workflow Management ● A mechanism for authentication and authorization or management of file permissions. ● Support or integration with non-linux operating systems. 43 Non-Goals
- 44. “ The SCIF Recipe 44 Metadata BESTAPP=foo VERSION=1.0 Entry Points ./run-foo.sh ./run-bar.sh Content Files Install Routine
- 45. The SCIF Recipe 45 Metadata BESTAPP=foo VERSION=1.0 Entry Points ./run-foo.sh ./run-bar.sh Content Files Install Routine
- 46. %apprun hello-world /bin/bash hello-world.sh %appinstall hello-world echo "echo 'Hello World!'" >> $SCIF_APPBIN/hello-world.sh chmod u+x $SCIF_APPBIN/hello-world.sh %appenv hello-world THEBESTAPP=$SCIF_APPNAME export THEBESTAPP %applabels hello-world MAINTAINER Vanessasaur %apphelp hello-world This is an example "Hello World" application. You can install it to a Scientific Filesystem (scif) with the command: scif install hello-world.scif hello-world.scif https://sci-f.github.io/tutorial-quick-start %app[action] [name]
- 47. scif run fooscif install recipe.scifrecipe.scif write recipe install interact Developer Interaction
- 48. scif run fooscif install recipe.scifrecipe.scif write recipe install interact User Interaction
- 49. scif run foo interact User Interaction ./container … apps run exec help inspect shell pyshell
- 50. ./scif apps docker run vanessa/scif:hw apps ./scif-cli apps
- 51. ./scif apps docker run vanessa/scif:hw apps ./scif-cli apps
- 52. %apprun hello-world-echo exec echo “Hello World!” ./scif run hello-world docker run vanessa/scif:hw run hello-world-echo ./scif-cli run hello-world-echo
- 53. ./scif run hello-world docker run vanessa/scif:hw run hello-world-echo ./scif-cli run hello-world-echo %apprun hello-world-echo exec echo “Hello World!”
- 54. %appenv hello-world OMG=TACOS export OMG ./scif exec hello-world docker run vanessa/scif:hw exec hello-world echo [e]OMG ./scif-cli exec hello-world echo [e]OMG
- 55. %appenv hello-world OMG=TACOS export OMG ./scif exec hello-world docker run vanessa/scif:hw exec hello-world echo [e]OMG ./scif-cli exec hello-world echo [e]OMG
- 58. %apphelp hello-world This is an example "Hello World" application. You can install it to a Scientific Filesystem (scif) with the command: scif install hello-world.scif ./scif help hello-world docker run vanessa/scif:hw help hello-world ./scif-cli help hello-world
- 59. %appenv hello-world OMG=TACOS export OMG ./scif shell hello-world
- 61. ./scif pyshell
- 63. The Scientific Filesystem has a default root on the host to intentionally be separate from standard linux folders, and to not interfere with likely existing folders (e.g., just /data or /apps). /scif /apps /data Architecture
- 65. foo runtime scif entrypoint foo /foo /bin /lib /scif runscript runscript.help labels.json environment.sh /scif /apps ← $PATH ← $LD_LIBRARY_PATH Organization
- 67. /foo /bin /lib /scif runscript runscript.help labels.json environment.sh /scif /apps ./container exec bar [command] Interaction
- 68. ./container run bar [args] /scif /apps /foo /bin /lib /scif runscript runscript.help labels.json environment.sh Interaction
- 69. ./container shell bar $SCIF_SHELL ./container pyshell bar /scif /apps /foo /bin /lib /scif runscript runscript.help labels.json environment.sh Interaction
- 72. foo active bar sleeping ./container run foo
- 73. foo active bar sleeping ./container run foo BESTAPP=FOO!
- 74. foo active bar sleeping ./container run foo BESTAPP=FOO!
- 75. foo active
- 77. “ What can I do with a SCIF? 77
- 78. - expose multiple container entrypoints - package metrics/evaluation alongside the analysis - SCIF apps as different working environments, runtime - SCIF apps for different runtime contexts
- 79. “ What are we going to do today? 79
- 80. What are we going to do today? Build Containers Across different container technologies, all with the same scientific filesystem (SCIF) Run a Workflow The same snakemake workflow made modular by SCIF applications Evaluate How do the different container technologies compare? 80
- 82. Scientific Pipeline - Felix Bartusch - variant calling - Snakemake 82
- 83. “ What level of modularity? 83
- 84. - Level 1: Container Modularity - One entrypoint - Run the entire thing, or don’t ./container
- 85. - Level 2: Pipeline Modularity - Multiple entrypoints - Correspond to pipeline steps - Step → collection of commands ./container apps mapping sorting ./container run mapping sorting mapping
- 86. - Level 3: Developer Modularity - Expose lowest level executables - Ideal for building pipelines - Step → core tools ./container apps bwa samtools ./container run samtools samtools bwa
- 87. A Scientific Filesystem Design for Variant Calling What level of modularity should be exposed to interact with the container? ./container apps bwa samtools graphviz_create_dag snakemake bwa samtools graphviz_create_dag snakemake
- 88. What did I do with the container? Build Containers Run a Workflow Evaluate https://github.com/sci-f/snakemake.scif 88
- 89. Container Comparisons What steps did we take across container technologies? 1. build the container 3. run the entire workflow 0. install the software 2. run components of the workflow
- 90. What did we learn? What did I learn about user interaction? Good software without documentation and tutorial for several use cases is akin to a treasure chest on a remote planet.
- 91. Container Comparisons What steps did we take across container technologies? 0. install the software
- 92. Container Comparisons What steps did we take across container technologies? 0. install the software 1. Documentation is the first interaction a. Finding it b. Is it up to date? c. Is there too much detail? d. Is there a clear avenue to get more help?
- 93. Container Comparisons What steps did we take across container technologies? 1. build the container 0. install the software
- 94. Container Comparisons What steps did we take across container technologies? 1. build the container 0. install the software 1. Examples are needed for learning a. Does it show me all the things I might need? i. binds ii. options iii. permissions
- 95. Container Comparisons What steps did we take across container technologies? 1. build the container 2. run components of the workflow 0. install the software
- 96. Container Comparisons What steps did we take across container technologies? 1. build the container 2. run components of the workflow 0. install the software 1. Establish boundaries a. show inside vs. outside
- 97. Container Comparisons What steps did we take across container technologies? 1. build the container 3. run the entire workflow 0. install the software 2. run components of the workflow
- 98. “ 98 Run the entire workflow !
- 99. runc --root /tmp/runc run --bundle /tmp/runc snakemake.runc docker run -v $PWD/data:/scif/data:z -v $PWD:/scif/data/snakemake:z -it vanessa/snakemake.scif run snakemake all singularity run --bind data/:/scif/data snakemake.simg run snakemake all ch-run --cd $PWD -b data:/scif/data /var/tmp/snakemake.ch -- /opt/conda/bin/scif run snakemake all shifter --image=vanessa/snakemake.scif --volume `pwd`/data:/scif/data --workdir /scif/data /opt/conda/bin/scif run snakemake all
- 100. runc --root /tmp/runc run --bundle /tmp/runc snakmake.runc docker run -v $PWD/data:/scif/data:z -v $PWD:/scif/data/snakemake:z -it vanessa/snakemake.scif run snakemake all singularity run --bind data/:/scif/data snakemake.simg run snakemake all ch-run --cd $PWD -b data:/scif/data /var/tmp/snakemake.ch -- /opt/conda/bin/scif run snakemake all shifter --image=vanessa/snakemake.scif --volume `pwd`/data:/scif/data --workdir /scif/data /opt/conda/bin/scif run snakemake all command
- 101. runc --root /tmp/runc run --bundle /tmp/runc snakemake.runc docker run -v $PWD/data:/scif/data -v $PWD:/scif/data/snakemake -it vanessa/snakemake.scif run snakemake all singularity run --bind data/:/scif/data snakemake.simg run snakemake all ch-run --cd $PWD -b data:/scif/data /var/tmp/snakemake.ch -- /opt/conda/bin/scif run snakemake all shifter --image=vanessa/snakemake.scif --volume `pwd`/data:/scif/data --workdir /scif/data /opt/conda/bin/scif run snakemake all executable
- 102. runc --root /tmp/runc run --bundle /tmp/runc snakmake.runc docker run -v $PWD/data:/scif/data -v $PWD:/scif/data/snakemake -it vanessa/snakemake.scif run snakemake all singularity run --bind data/:/scif/data snakemake.simg run snakemake all ch-run --cd $PWD -b data:/scif/data /var/tmp/snakemake.ch -- /opt/conda/bin/scif run snakemake all shifter --image=vanessa/snakemake.scif --volume `pwd`/data:/scif/data --workdir /scif/data /opt/conda/bin/scif run snakemake all executable + action
- 103. runc --root /tmp/runc run --bundle /tmp/runc snakemake.runc docker run -v $PWD/data:/scif/data -v $PWD:/scif/data/snakemake -it vanessa/snakemake.scif run snakemake all singularity run --bind data/:/scif/data snakemake.simg run snakemake all ch-run --cd $PWD -b data:/scif/data /var/tmp/snakemake.ch -- /opt/conda/bin/scif run snakemake all shifter --image=vanessa/snakemake.scif --volume `pwd`/data:/scif/data --workdir /scif/data /opt/conda/bin/scif run snakemake all container
- 104. runc --root /tmp/runc run --bundle /tmp/runc snakmake.runc docker run -v $PWD/data:/scif/data -v $PWD:/scif/data/snakemake -it vanessa/snakemake.scif run snakemake all singularity run --bind data/:/scif/data snakemake.simg run snakemake all ch-run --cd $PWD -b data:/scif/data /var/tmp/snakemake.ch -- /opt/conda/bin/scif run snakemake all shifter --image=vanessa/snakemake.scif --volume `pwd`/data:/scif/data --workdir /scif/data /opt/conda/bin/scif run snakemake all volumes / mounts
- 105. executable action options container options command
- 107. “ Are the outputs comparable? 107
- 108. $ diff results/docker/calls.vcf results/singularity/calls.vcf 28c28 < ##bcftools_callCommand=call -mv -; Date=Sat Mar 10 01:07:35 2018 --- > ##bcftools_callCommand=call -mv -; Date=Fri Mar 9 20:07:47 2018 $ diff results/docker/calls.vcf results/runc/calls.vcf 28c28 < ##bcftools_callCommand=call -mv -; Date=Sat Mar 10 01:07:35 2018 --- > ##bcftools_callCommand=call -mv -; Date=Sat Mar 10 01:08:05 2018 $ diff results/docker/calls.vcf results/shifter/calls.vcf 28c28 < ##bcftools_callCommand=call -mv -; Date=Sat Mar 10 01:07:35 2018 --- > ##bcftools_callCommand=call -mv -; Date=Mon Mar 12 20:45:21 2018 $ diff results/docker/calls.vcf results/charliecloud/calls.vcf 28c28 < ##bcftools_callCommand=call -mv -; Date=Sat Mar 10 01:07:35 2018 --- > ##bcftools_callCommand=call -mv -; Date=Sat Mar 10 01:07:57 2018 Results are the same vs
- 109. What did we learn? Do the technologies produce the same result? From the perspective of the scientist, we have comparable resource usage and produce the same output.
- 110. “ What about resource usage? 110
- 111. singularity run --bind data/:/scif/data snakemake.simg run valgrind singularity run --bind data/:/scif/data snakemake.simg run timeit
- 117. ...but how do my choices of the content, and host, influence my result?
- 118. Does Heap Consumption Vary Across Machine Type?
- 120. sregistry build https://www.github.com/vsoch/hello-world Local machine build template cloud builder cloud storage sregistry pull sregistry pull vsoch/hello-world:latest...
- 124. Does Heap Consumption Vary Across Machine Type? Variant calling memory usage is comparable from 1 to 64 cores
- 125. What is the optimal design?
- 126. tensorflow 17.0 tensorflow 17.1 theano 1.0.1 theano 1.0.0
- 127. tensorflow 17.0 tensorflow 17.1 theano 1.0.1 theano 1.0.0
- 128. “ TLDR 128
- 129. 129 It does the same thing!
- 130. Containerization seems to produce a reproducible workflow.
- 131. We need to make tools that are usable across technologies.
- 132. Transparency, ease of use, and ability to contribute are important.
- 133. It’s our job to encourage introspection and evaluation of variation.
- 134. Once upon a time… There was a scientist
- 135. There was a reproducibility crisis
- 136. ...but friends were there to help :)
- 138. PRESENTER: VANESSASAURUS THIS IS THE END vsochat@stanford.edu