Standardising Swedish genomics analyses using nextflow
3 likes1,162 views
This document outlines the standardization of Swedish genomics analyses using Nextflow, highlighting key methods, tools, and workflows for handling genomic data efficiently. It discusses the implementation of bioinformatics processes, quality control measures, and collaborative practices like versioning and using containers for reproducibility. Additionally, it presents challenges faced and plans for future improvements in genomic sequencing pipelines.
![reference genomes
params {
genomes {
'GRCh37' {
fasta = '/refs/human/genome.fasta'
gtf = '/refs/human/genes.gtf'
}
'GRCm38' {
fasta = '/refs/mouse/genome.fasta'
gtf = '/refs/mouse/genes.gtf'
}
}
}
conf/references.conf
params.fasta = params.genome ? params.genomes[ params.genome ].fasta ?: false : false
params.gtf = params.genome ? params.genomes[ params.genome ].gtf ?: false : false
main.nf
$ nextflow run main.nf --genome GRCh37
$ nextflow run main.nf --fasta /path/to/my/genome.fa](https://image.slidesharecdn.com/philewels-nextflowbarcelona2017-170915073502/85/Standardising-Swedish-genomics-analyses-using-nextflow-20-320.jpg)
![email notifications
workflow.onComplete {
def subject = 'My pipeline execution'
def recipient = 'me@gmail.com'
['mail', '-s', subject, recipient].execute() << """
Pipeline execution summary
---------------------------
Completed at: ${workflow.complete}
Duration : ${workflow.duration}
Success : ${workflow.success}
workDir : ${workflow.workDir}
exit status : ${workflow.exitStatus}
Error report: ${workflow.errorReport ?: '-'}
"""
}
Nextflow documentation](https://image.slidesharecdn.com/philewels-nextflowbarcelona2017-170915073502/85/Standardising-Swedish-genomics-analyses-using-nextflow-32-320.jpg)
![email notifications
workflow.onComplete {
// Render the HTML template
def hf = new File("$baseDir/assets/email_template.html")
def html_template = engine.createTemplate(hf).make(email_fields)
def email_html = html_template.toString()
// Send the HTML e-mail
if (params.email) {
[ 'sendmail', '-t' ].execute() << sendmail_html
log.info "[NGI-MethylSeq] Sent summary e-mail to $params.email (sendmail)"
}
// Write summary e-mail HTML to a file
def output_d = new File( "${params.outdir}/pipeline_info/" )
if( !output_d.exists() ) {
output_d.mkdirs()
}
def output_hf = new File( output_d, "pipeline_report.html" )
output_hf.withWriter { w -> w << email_html }
}
main.nf](https://image.slidesharecdn.com/philewels-nextflowbarcelona2017-170915073502/85/Standardising-Swedish-genomics-analyses-using-nextflow-33-320.jpg)
![[NGI-RNAseq] Successful: Test RNA Run
email notifications](https://image.slidesharecdn.com/philewels-nextflowbarcelona2017-170915073502/85/Standardising-Swedish-genomics-analyses-using-nextflow-35-320.jpg)
![email notifications
[NGI-RNAseq] FAILED: Test RNA Run](https://image.slidesharecdn.com/philewels-nextflowbarcelona2017-170915073502/85/Standardising-Swedish-genomics-analyses-using-nextflow-37-320.jpg)
Standardising Swedish genomics analyses using nextflow
- 1. Standardising Swedish genomics analyses using nextflow Phil Ewels @ewels @tallphil Nextflow Meeting 2017-09-14 CRG, Barcelona
- 2. 2 x MiSeq 5 x HiSeq 2500 5 x HiSeq X10 NovaSeq RNA-Seq WG Re-Seq Targeted Re-Seq Metagenomics Others 0 2000 4000 6000 8000 10000 12000 14000 1,265 2,580 3,214 8,934 12,017 Number of Samples in 2016 1141 Gbp/day 1X Human Genome every 4 minutes NGI stockholmstockholm SciLifeLab NGI
- 3. CumulativeOutput(MBp) 0 250,000,000 500,000,000 750,000,000 1,000,000,000 Jan 2012 Sep 2012 M ay 2013 Jan 2014 Sep 2014 M ay 2015 Jan 2016 Sep 2016 M ay 2017 NGI stockholmstockholm - sequencing output SciLifeLab NGI
- 4. NGI bioinformatics • Initial data analysis for major protocols • Internal QC and standardised starting point for users • Team of 10 bioinformaticians • Accredited facility
- 5. analysis requirements Automated Reliable Easy for others to run Reproducible results icons: the noun project
- 7. what have we learnt?
- 9. sharing is caring • Open-source on GitHub from day one • Easier help and feedback from others • Other people may help to develop your code • https://github.com/nextflow-io/awesome-nextflow
- 10. use containers
- 11. use containers • Create a docker image, even if you don’t think you need to • Makes local and automated testing possible • Future proof for cloud / singularity / other people
- 12. test, test and test again
- 13. test, test and test again • Find a small test dataset • Make a bash script to fetch data and launch pipeline • Different flags to launch with different parameters • Use Travis build matrix to launch parallel test runs
- 16. minimal configs
- 17. minimal configs • Build config files around blocks of function • Hardware / software deps / genome references • Use nextflow profiles • Even if only using ‘standard’ default • Don’t be afraid to use multiple configs per profile • Build on a base profile and be clever with limits
- 21. reference genomes • illumina iGenomes is a great resource for this • Standard organisation allows easy use of multiple genomes • Use AWS iGenomes for free on AWS S3 • See https://ewels.github.io/AWS-iGenomes/
- 24. dodgy file patterns Channel .fromFilePairs( params.reads, size: -1 ) Channel .fromFilePairs( params.reads, size: params.singleEnd ? 1 : 2 ) If glob pattern doesn’t use {1,2} then all PE files are run in SE mode If glob pattern doesn’t use {1,2} then pipeline exits with no matching files
- 27. quick-fire round
- 35. [NGI-RNAseq] Successful: Test RNA Run email notifications
- 37. email notifications [NGI-RNAseq] FAILED: Test RNA Run
- 38. Groovy syntax highlighting run-STAR = params.runstar run-STAR = params.runstar #!/usr/bin/env nextflow /* vim: syntax=groovy -*- mode: groovy;-*- */ main.nf without highlighting: with highlighting:
- 40. future plans • Use singularity for everything • Benchmark AWS run pricing for future planning • Refine pipelines • Improve resource requests • Automate launch and run management
- 41. Phil Ewels phil.ewels@scilifelab.se ewels tallphil Acknowledgements http://github.com/SciLifeLab http://opensource.scilifelab.se NGI stockholm Max Käller Rickard Hammarén Denis Moreno Francesco Vezzi NGI Stockholm Genomics Applications Development Group Paolo Di Tommaso The nextflow community