flowr streamlining computing workflows
0 likes190 views
The document discusses optimizing data processing workflows by using computing clusters to execute multi-step pipelines efficiently and transparently. It highlights the ability to run workflows both locally and in the cloud while maintaining reproducibility, tracking, and re-running of flows. The content also includes examples of job submission types and dependencies using a genomic analysis pipeline as a case study.
flowr streamlining computing workflows
- 3. ➡when one needs to wrangle a lot of data whybother?
- 4. ➡when one needs to wrangle a lot of data ➡and there are multiple steps involved whybother?
- 5. ➡when one needs to wrangle a lot of data ➡and there are multiple steps involved ➡esp. when some of the steps can be further broken down and processed in parallel whybother?
- 6. ➡when one needs to wrangle a lot of data ➡and there are multiple steps involved ➡esp. when some of the steps can be further broken down and processed in parallel ➡use a computing cluster, submit a web of jobs whybother?
- 7. ➡when one needs to wrangle a lot of data ➡and there are multiple steps involved ➡esp. when some of the steps can be further broken down and processed in parallel ➡use a computing cluster, submit a web of jobs whybother? ✓ Effectively process a multi-step pipeline, spawning it across the computing cluster
- 8. ➡when one needs to wrangle a lot of data ➡and there are multiple steps involved ➡esp. when some of the steps can be further broken down and processed in parallel ➡use a computing cluster, submit a web of jobs whybother? ✓ Effectively process a multi-step pipeline, spawning it across the computing cluster ✓ Reproducible and transparent, with cleanly structured execution logs
- 9. ➡when one needs to wrangle a lot of data ➡and there are multiple steps involved ➡esp. when some of the steps can be further broken down and processed in parallel ➡use a computing cluster, submit a web of jobs whybother? ✓ Effectively process a multi-step pipeline, spawning it across the computing cluster ✓ Reproducible and transparent, with cleanly structured execution logs ✓ Track and re-run flows
- 10. ➡when one needs to wrangle a lot of data ➡and there are multiple steps involved ➡esp. when some of the steps can be further broken down and processed in parallel ➡use a computing cluster, submit a web of jobs whybother? ✓ Effectively process a multi-step pipeline, spawning it across the computing cluster ✓ Reproducible and transparent, with cleanly structured execution logs ✓ Track and re-run flows ✓ Lean and Portable, with easy installation
- 11. ➡when one needs to wrangle a lot of data ➡and there are multiple steps involved ➡esp. when some of the steps can be further broken down and processed in parallel ➡use a computing cluster, submit a web of jobs whybother? ✓ Effectively process a multi-step pipeline, spawning it across the computing cluster ✓ Reproducible and transparent, with cleanly structured execution logs ✓ Track and re-run flows ✓ Lean and Portable, with easy installation ✓ Run the same pipeline in the cloud (using star cluster) OR a local machine
- 12. ➡when one needs to wrangle a lot of data ➡and there are multiple steps involved ➡esp. when some of the steps can be further broken down and processed in parallel ➡use a computing cluster, submit a web of jobs whybother? ✓ Effectively process a multi-step pipeline, spawning it across the computing cluster ✓ Reproducible and transparent, with cleanly structured execution logs ✓ Track and re-run flows ✓ Lean and Portable, with easy installation ✓ Run the same pipeline in the cloud (using star cluster) OR a local machine ✓ Supports multiple cluster computing platforms (torque, lsf, sge, slurm …)
- 15. submission types scatter serial fivesimpleterms,definingallrelationships decide how pieces of a single step are processed
- 16. submission types scatter serial fivesimpleterms,definingallrelationships decide how pieces of a single step are processed all in parallel
- 17. submission types scatter serial fivesimpleterms,definingallrelationships decide how pieces of a single step are processed all in parallel sequentially
- 18. serial dependency types burstgather submission types scatter serial fivesimpleterms,definingallrelationships decide how pieces of a single step are processed all in parallel sequentially
- 19. serial dependency types burstgather submission types scatter serial fivesimpleterms,definingallrelationships decide how pieces of a single step are processed decide the relationship b/w steps all in parallel sequentially
- 20. serial dependency types burstgather submission types scatter serial fivesimpleterms,definingallrelationships decide how pieces of a single step are processed decide the relationship b/w steps many-to-many all in parallel sequentially
- 21. serial dependency types burstgather submission types scatter serial fivesimpleterms,definingallrelationships decide how pieces of a single step are processed decide the relationship b/w steps many-to-many many-to-one all in parallel sequentially
- 22. serial dependency types burstgather submission types scatter serial fivesimpleterms,definingallrelationships decide how pieces of a single step are processed decide the relationship b/w steps many-to-many many-to-one one-to-manyall in parallel sequentially
- 23. one to many submission type dependency type relationship Usingagenomicsexampleflow,withflowrconcepts
- 26. fastq sam bam fastq sam bam fastq sam bam fastq sam bam fastq sam bam scatter serial many to many merged bam serial gather many to one alignment stats sort & index Mutations Copy Number variation Indel Calling downstream analysis scatter burst one to many submission type dependency type relationship Usingagenomicsexampleflow,withflowrconcepts
- 30. ★ wewouldsleepforafewseconds ★ createafewsmallfiles ★ mergethosefiles asimplepipeline,where
- 31. ★ wewouldsleepforafewseconds ★ createafewsmallfiles ★ mergethosefiles ★ getthesizeoftheresultingmergedfile asimplepipeline,where
- 32. echo 'Hello World !' sleep 5 sleep 5 cat $RANDOM > tmp1 cat $RANDOM > tmp2 cat tmp1 tmp2 > tmp du -sh tmp simplepipelineinbash
- 33. echo 'Hello World !' sleep 5 sleep 5 cat $RANDOM > tmp1 cat $RANDOM > tmp2 cat tmp1 tmp2 > tmp du -sh tmp simplepipelineinbash say Hello to the world
- 34. echo 'Hello World !' sleep 5 sleep 5 cat $RANDOM > tmp1 cat $RANDOM > tmp2 cat tmp1 tmp2 > tmp du -sh tmp simplepipelineinbash say Hello to the world wait for a few seconds…
- 35. echo 'Hello World !' sleep 5 sleep 5 cat $RANDOM > tmp1 cat $RANDOM > tmp2 cat tmp1 tmp2 > tmp du -sh tmp simplepipelineinbash say Hello to the world wait for a few seconds… create two small files
- 36. echo 'Hello World !' sleep 5 sleep 5 cat $RANDOM > tmp1 cat $RANDOM > tmp2 cat tmp1 tmp2 > tmp du -sh tmp simplepipelineinbash say Hello to the world wait for a few seconds… create two small files merge the two files
- 37. echo 'Hello World !' sleep 5 sleep 5 cat $RANDOM > tmp1 cat $RANDOM > tmp2 cat tmp1 tmp2 > tmp du -sh tmp simplepipelineinbash say Hello to the world wait for a few seconds… create two small files merge the two files check the size of the resulting file
- 38. wrapbashcommandsintoR hello='echo Hello World !' sleep=c('sleep 5', 'sleep 5') tmp=c('cat $RANDOM > tmp1', 'cat $RANDOM > tmp2') merge='cat tmp1 tmp2 > tmp' size='du -sh tmp'
- 39. wrapbashcommandsintoR hello='echo Hello World !' sleep=c('sleep 5', 'sleep 5') tmp=c('cat $RANDOM > tmp1', 'cat $RANDOM > tmp2') merge='cat tmp1 tmp2 > tmp' size='du -sh tmp' say Hello to the world
- 40. wrapbashcommandsintoR hello='echo Hello World !' sleep=c('sleep 5', 'sleep 5') tmp=c('cat $RANDOM > tmp1', 'cat $RANDOM > tmp2') merge='cat tmp1 tmp2 > tmp' size='du -sh tmp' say Hello to the world wait for a few seconds…
- 41. wrapbashcommandsintoR hello='echo Hello World !' sleep=c('sleep 5', 'sleep 5') tmp=c('cat $RANDOM > tmp1', 'cat $RANDOM > tmp2') merge='cat tmp1 tmp2 > tmp' size='du -sh tmp' say Hello to the world wait for a few seconds… create two small files
- 42. wrapbashcommandsintoR hello='echo Hello World !' sleep=c('sleep 5', 'sleep 5') tmp=c('cat $RANDOM > tmp1', 'cat $RANDOM > tmp2') merge='cat tmp1 tmp2 > tmp' size='du -sh tmp' say Hello to the world wait for a few seconds… create two small files merge the two files
- 43. wrapbashcommandsintoR hello='echo Hello World !' sleep=c('sleep 5', 'sleep 5') tmp=c('cat $RANDOM > tmp1', 'cat $RANDOM > tmp2') merge='cat tmp1 tmp2 > tmp' size='du -sh tmp' say Hello to the world wait for a few seconds… create two small files merge the two files check the size of the resulting file
- 44. createatableofallcommands library(flowr) lst = list(hello=hello, sleep=sleep, tmp=tmp, merge=merge, size=size) flowmat = to_flowmat(lst, "samp1") hello='echo Hello World !' sleep=c('sleep 5', 'sleep 5') tmp=c('cat $RANDOM > tmp1', 'cat $RANDOM > tmp2') merge='cat tmp1 tmp2 > tmp' size='du -sh tmp'
- 45. createatableofallcommands library(flowr) lst = list(hello=hello, sleep=sleep, tmp=tmp, merge=merge, size=size) flowmat = to_flowmat(lst, "samp1") hello='echo Hello World !' sleep=c('sleep 5', 'sleep 5') tmp=c('cat $RANDOM > tmp1', 'cat $RANDOM > tmp2') merge='cat tmp1 tmp2 > tmp' size='du -sh tmp' create a named list
- 46. createatableofallcommands library(flowr) lst = list(hello=hello, sleep=sleep, tmp=tmp, merge=merge, size=size) flowmat = to_flowmat(lst, "samp1") hello='echo Hello World !' sleep=c('sleep 5', 'sleep 5') tmp=c('cat $RANDOM > tmp1', 'cat $RANDOM > tmp2') merge='cat tmp1 tmp2 > tmp' size='du -sh tmp' create a named list create a table
- 47. createatableofallcommands library(flowr) lst = list(hello=hello, sleep=sleep, tmp=tmp, merge=merge, size=size) flowmat = to_flowmat(lst, "samp1") hello='echo Hello World !' sleep=c('sleep 5', 'sleep 5') tmp=c('cat $RANDOM > tmp1', 'cat $RANDOM > tmp2') merge='cat tmp1 tmp2 > tmp' size='du -sh tmp' create a named list create a table |samplename |jobname |cmd | |:----------|:-------|:-------------------| |samp1 |hello |echo Hello World ! | |samp1 |sleep |sleep 5 | |samp1 |sleep |sleep 5 | |samp1 |tmp |cat $RANDOM > tmp1 | |samp1 |tmp |cat $RANDOM > tmp2 | |samp1 |merge |cat tmp1 tmp2 > tmp | |samp1 |size |du -sh tmp |
- 48. createatableofallcommands library(flowr) lst = list(hello=hello, sleep=sleep, tmp=tmp, merge=merge, size=size) flowmat = to_flowmat(lst, "samp1") hello='echo Hello World !' sleep=c('sleep 5', 'sleep 5') tmp=c('cat $RANDOM > tmp1', 'cat $RANDOM > tmp2') merge='cat tmp1 tmp2 > tmp' size='du -sh tmp' create a named list create a table |samplename |jobname |cmd | |:----------|:-------|:-------------------| |samp1 |hello |echo Hello World ! | |samp1 |sleep |sleep 5 | |samp1 |sleep |sleep 5 | |samp1 |tmp |cat $RANDOM > tmp1 | |samp1 |tmp |cat $RANDOM > tmp2 | |samp1 |merge |cat tmp1 tmp2 > tmp | |samp1 |size |du -sh tmp | asimpletab-delimtable
- 49. connectthedots…
- 52. flowdef = to_flowdef(flowmat, sub_type = c("serial", "scatter", "scatter", "serial", "serial"), dep_type = c("none", "burst", "serial", "gather", "serial"), platform = "local") createaflowdefinition
- 53. flowdef = to_flowdef(flowmat, sub_type = c("serial", "scatter", "scatter", "serial", "serial"), dep_type = c("none", "burst", "serial", "gather", "serial"), platform = "local") |jobname |sub_type |prev_jobs |dep_type | cpu| |:-------|:--------|:---------|:--------|---:| |hello |serial |none |none | 1| |sleep |scatter |hello |burst | 1| |tmp |scatter |sleep |serial | 1| |merge |serial |tmp |gather | 1| |size |serial |merge |serial | 1| createaflowdefinition asimpletab-delimtable
- 54. flowdef = to_flowdef(flowmat, sub_type = c("serial", "scatter", "scatter", "serial", "serial"), dep_type = c("none", "burst", "serial", "gather", "serial"), platform = "local") hello dep: none sub: serial sleepsleepsleepsleep dep: burst sub: scatter tmptmptmptmp dep: serial sub: scatter merge dep: gather sub: serial size dep: serial sub: serial |jobname |sub_type |prev_jobs |dep_type | cpu| |:-------|:--------|:---------|:--------|---:| |hello |serial |none |none | 1| |sleep |scatter |hello |burst | 1| |tmp |scatter |sleep |serial | 1| |merge |serial |tmp |gather | 1| |size |serial |merge |serial | 1| plot_flow(flowdef) createaflowdefinition asimpletab-delimtable
- 57. stitch&submittothecluster (cloudorserver) |jobname |sub_type |prev_jobs |dep_type | cpu| |:-------|:--------|:---------|:--------|---:| |hello |serial |none |none | 1| |sleep |scatter |hello |burst | 1| |tmp |scatter |sleep |serial | 1| |merge |serial |tmp |gather | 1| |size |serial |merge |serial | 1| |samplename |jobname |cmd | |:----------|:-------|:-------------------| |samp1 |hello |echo Hello World ! | |samp1 |sleep |sleep 5 | |samp1 |sleep |sleep 5 | |samp1 |tmp |cat $RANDOM > tmp1 | |samp1 |tmp |cat $RANDOM > tmp2 | |samp1 |merge |cat tmp1 tmp2 > tmp | |samp1 |size |du -sh tmp | flowmat flowdef +
- 58. stitch&submittothecluster (cloudorserver) |jobname |sub_type |prev_jobs |dep_type | cpu| |:-------|:--------|:---------|:--------|---:| |hello |serial |none |none | 1| |sleep |scatter |hello |burst | 1| |tmp |scatter |sleep |serial | 1| |merge |serial |tmp |gather | 1| |size |serial |merge |serial | 1| |samplename |jobname |cmd | |:----------|:-------|:-------------------| |samp1 |hello |echo Hello World ! | |samp1 |sleep |sleep 5 | |samp1 |sleep |sleep 5 | |samp1 |tmp |cat $RANDOM > tmp1 | |samp1 |tmp |cat $RANDOM > tmp2 | |samp1 |merge |cat tmp1 tmp2 > tmp | |samp1 |size |du -sh tmp | flowmat flowdef + fobj = to_flow(flowmat, flowdef, execute = TRUE)
- 59. Working on: hello |===== | 25% Working on: sleep |================ | 50% Working on: merge |==================================| 100% Working on: size Flow is being processed. Track it from R/Terminal using: flowr status x=~/flowr/runs/flowname-samp1-20151005-16-01-38-M8WniKJo OR from R using: status(x='~/flowr/runs/flowname-samp1-20151005-16-01-38-M8WniKJo') stitch&submittothecluster (cloudorserver) |jobname |sub_type |prev_jobs |dep_type | cpu| |:-------|:--------|:---------|:--------|---:| |hello |serial |none |none | 1| |sleep |scatter |hello |burst | 1| |tmp |scatter |sleep |serial | 1| |merge |serial |tmp |gather | 1| |size |serial |merge |serial | 1| |samplename |jobname |cmd | |:----------|:-------|:-------------------| |samp1 |hello |echo Hello World ! | |samp1 |sleep |sleep 5 | |samp1 |sleep |sleep 5 | |samp1 |tmp |cat $RANDOM > tmp1 | |samp1 |tmp |cat $RANDOM > tmp2 | |samp1 |merge |cat tmp1 tmp2 > tmp | |samp1 |size |du -sh tmp | flowmat flowdef + fobj = to_flow(flowmat, flowdef, execute = TRUE)
- 61. submitaflow,then… status() monitor the status of a single flow OR multiple flows
- 62. submitaflow,then… status() monitor the status of a single flow OR multiple flows kill() kill all the associated jobs, of one or many flows
- 63. submitaflow,then… status() monitor the status of a single flow OR multiple flows kill() kill all the associated jobs, of one or many flows rerun() One can rerun the flow from a intermediate step
- 64. github.com/sahilseth/flowr complete documentation: docs.flowr.space email: sahil.seth@me.com
- 65. Extradetails
- 66. - creativily define relationships using submission and dependency types - each row describes resources for one step, providing full flexibility Flow mat samplename jobname cmd sample1 A sleep 2 && sleep 5;echo hello sample1 A sleep 13 && sleep 7;echo hello sample1 B head -c 100000 /dev/urandom > tmp1 sample1 B head -c 100000 /dev/urandom > tmp1 sample1 C cat tmp1 tmp2 tmp3 > merged sample1 D du -sh merged sample1 D ls merged Flow Definition Define Relationships Resource Requirements jobname submission type previous job(s) dependency type queue memory time cpu platform A scatter none none medium 163185 23:00 1 lsf B scatter A serial medium 163185 23:00 1 lsf C serial B gather medium 163185 23:00 1 lsf D scatter C burst medium 163185 23:00 1 lsf
- 67. - creativily define relationships using submission and dependency types - each row describes resources for one step, providing full flexibility Flow mat samplename jobname cmd sample1 A sleep 2 && sleep 5;echo hello sample1 A sleep 13 && sleep 7;echo hello sample1 B head -c 100000 /dev/urandom > tmp1 sample1 B head -c 100000 /dev/urandom > tmp1 sample1 C cat tmp1 tmp2 tmp3 > merged sample1 D du -sh merged sample1 D ls merged Flow Definition Define Relationships Resource Requirements jobname submission type previous job(s) dependency type queue memory time cpu platform A scatter none none medium 163185 23:00 1 lsf B scatter A serial medium 163185 23:00 1 lsf C serial B gather medium 163185 23:00 1 lsf D scatter C burst medium 163185 23:00 1 lsf
- 68. - creativily define relationships using submission and dependency types - each row describes resources for one step, providing full flexibility Flow mat samplename jobname cmd sample1 A sleep 2 && sleep 5;echo hello sample1 A sleep 13 && sleep 7;echo hello sample1 B head -c 100000 /dev/urandom > tmp1 sample1 B head -c 100000 /dev/urandom > tmp1 sample1 C cat tmp1 tmp2 tmp3 > merged sample1 D du -sh merged sample1 D ls merged Flow Definition Define Relationships Resource Requirements jobname submission type previous job(s) dependency type queue memory time cpu platform A scatter none none medium 163185 23:00 1 lsf B scatter A serial medium 163185 23:00 1 lsf C serial B gather medium 163185 23:00 1 lsf D scatter C burst medium 163185 23:00 1 lsf
- 69. - creativily define relationships using submission and dependency types - each row describes resources for one step, providing full flexibility Flow mat samplename jobname cmd sample1 A sleep 2 && sleep 5;echo hello sample1 A sleep 13 && sleep 7;echo hello sample1 B head -c 100000 /dev/urandom > tmp1 sample1 B head -c 100000 /dev/urandom > tmp1 sample1 C cat tmp1 tmp2 tmp3 > merged sample1 D du -sh merged sample1 D ls merged Flow Definition Define Relationships Resource Requirements jobname submission type previous job(s) dependency type queue memory time cpu platform A scatter none none medium 163185 23:00 1 lsf B scatter A serial medium 163185 23:00 1 lsf C serial B gather medium 163185 23:00 1 lsf D scatter C burst medium 163185 23:00 1 lsf
- 70. - creativily define relationships using submission and dependency types - each row describes resources for one step, providing full flexibility Flow mat samplename jobname cmd sample1 A sleep 2 && sleep 5;echo hello sample1 A sleep 13 && sleep 7;echo hello sample1 B head -c 100000 /dev/urandom > tmp1 sample1 B head -c 100000 /dev/urandom > tmp1 sample1 C cat tmp1 tmp2 tmp3 > merged sample1 D du -sh merged sample1 D ls merged Flow Definition Define Relationships Resource Requirements jobname submission type previous job(s) dependency type queue memory time cpu platform A scatter none none medium 163185 23:00 1 lsf B scatter A serial medium 163185 23:00 1 lsf C serial B gather medium 163185 23:00 1 lsf D scatter C burst medium 163185 23:00 1 lsf - use any language to create a flow mat (a tsv file) - cmd column defines commands to run
- 71. - creativily define relationships using submission and dependency types - each row describes resources for one step, providing full flexibility Flow mat samplename jobname cmd sample1 A sleep 2 && sleep 5;echo hello sample1 A sleep 13 && sleep 7;echo hello sample1 B head -c 100000 /dev/urandom > tmp1 sample1 B head -c 100000 /dev/urandom > tmp1 sample1 C cat tmp1 tmp2 tmp3 > merged sample1 D du -sh merged sample1 D ls merged Flow Definition Define Relationships Resource Requirements jobname submission type previous job(s) dependency type queue memory time cpu platform A scatter none none medium 163185 23:00 1 lsf B scatter A serial medium 163185 23:00 1 lsf C serial B gather medium 163185 23:00 1 lsf D scatter C burst medium 163185 23:00 1 lsf - use any language to create a flow mat (a tsv file) - cmd column defines commands to run
- 72. - creativily define relationships using submission and dependency types - each row describes resources for one step, providing full flexibility Flow mat samplename jobname cmd sample1 A sleep 2 && sleep 5;echo hello sample1 A sleep 13 && sleep 7;echo hello sample1 B head -c 100000 /dev/urandom > tmp1 sample1 B head -c 100000 /dev/urandom > tmp1 sample1 C cat tmp1 tmp2 tmp3 > merged sample1 D du -sh merged sample1 D ls merged Flow Definition Define Relationships Resource Requirements jobname submission type previous job(s) dependency type queue memory time cpu platform A scatter none none medium 163185 23:00 1 lsf B scatter A serial medium 163185 23:00 1 lsf C serial B gather medium 163185 23:00 1 lsf D scatter C burst medium 163185 23:00 1 lsf - use any language to create a flow mat (a tsv file) - cmd column defines commands to run
- 73. - creativily define relationships using submission and dependency types - each row describes resources for one step, providing full flexibility Flow mat samplename jobname cmd sample1 A sleep 2 && sleep 5;echo hello sample1 A sleep 13 && sleep 7;echo hello sample1 B head -c 100000 /dev/urandom > tmp1 sample1 B head -c 100000 /dev/urandom > tmp1 sample1 C cat tmp1 tmp2 tmp3 > merged sample1 D du -sh merged sample1 D ls merged Flow Definition Define Relationships Resource Requirements jobname submission type previous job(s) dependency type queue memory time cpu platform A scatter none none medium 163185 23:00 1 lsf B scatter A serial medium 163185 23:00 1 lsf C serial B gather medium 163185 23:00 1 lsf D scatter C burst medium 163185 23:00 1 lsf - use any language to create a flow mat (a tsv file) - cmd column defines commands to run