NBIS ChIP-seq course
1 like338 views
The document provides a comprehensive overview of the NGI Stockholm's chip-seq processing services, detailing the methodology, guidelines, and computational pipeline management using Nextflow. It covers various steps from sample preparation to data analysis, emphasizing automation, reproducibility, and computational efficiency. Additionally, it describes the available tools, costs, and support, as well as implementation details for running analyses on different computational environments.
![NGI stockholm
Default: Run locally, assume
software is installed
Nextflow
#!/usr/bin/env nextflow
input = Channel.fromFilePairs( params.reads )
process fastqc {
input:
file reads from input
output:
file "*_fastqc.{zip,html}" into results
script:
"""
fastqc -q $reads
"""
} process {
executor = 'slurm'
clusterOptions = { "-A b2017123" }
cpus = 1
memory = 8.GB
time = 2.h
$fastqc {
module = ['bioinfo-tools', ‘FastQC']
}
}
Submit jobs to SLURM queue
Use environment modules](https://image.slidesharecdn.com/philewels-nbischip-seqcourse-171124160626/85/NBIS-ChIP-seq-course-16-320.jpg)
![NGI stockholm
Nextflow
#!/usr/bin/env nextflow
input = Channel.fromFilePairs( params.reads )
process fastqc {
input:
file reads from input
output:
file "*_fastqc.{zip,html}" into results
script:
"""
fastqc -q $reads
"""
} process {
executor = 'slurm'
clusterOptions = { "-A b2017123" }
cpus = 1
memory = 8.GB
time = 2.h
$fastqc {
module = ['bioinfo-tools', ‘FastQC']
}
}
docker {
enabled = true
}
process {
container = 'biocontainers/fastqc'
cpus = 1
memory = 8.GB
time = 2.h
}
Run locally, use docker container
for all software dependencies](https://image.slidesharecdn.com/philewels-nbischip-seqcourse-171124160626/85/NBIS-ChIP-seq-course-17-320.jpg)
![NGI stockholm
Read trimming
• Pipeline runs TrimGalore! to remove adapter
contamination and low quality bases automatically
• Use --notrim to disable this
• Some library preps also include additional adapters
--clip_r1 [int]
--clip_r2 [int]
--three_prime_clip_r1 [int]
--three_prime_clip_r2 [int]](https://image.slidesharecdn.com/philewels-nbischip-seqcourse-171124160626/85/NBIS-ChIP-seq-course-28-320.jpg)
![NGI stockholm
Extending Read Length
• When using single-end data, sequenced read length is
shorter than the sequence fragment length
• For DeepTools, need to "extend" the read length
• Set to 100bp by default. Use this parameter to
customise this value.
• Expected fragment length - sequence read length
--extendReadsLen [int]](https://image.slidesharecdn.com/philewels-nbischip-seqcourse-171124160626/85/NBIS-ChIP-seq-course-31-320.jpg)
![NGI stockholm
Nextflow config files
• Can save a config file with defaults
• Anything with two hyphens is a params
params {
email = 'phil.ewels@scilifelab.se'
project = "b2017123"
}
process.$multiqc.module = []
./nextflow.config
~/.nextflow/config
-c /path/to/my.config](https://image.slidesharecdn.com/philewels-nbischip-seqcourse-171124160626/85/NBIS-ChIP-seq-course-35-320.jpg)
![NGI stockholm
NGI-ChIPseq config
N E X T F L O W ~ version 0.26.1
Launching `SciLifeLab/NGI-ChIPseq` [deadly_bose] - revision: 28e24c2a2a
=========================================
NGI-ChIPseq: ChIP-Seq Best Practice v1.4
=========================================
Run Name : deadly_bose
Reads : data/*fastq.gz
Data Type : Single-End
Genome : GRCh37
BWA Index : /sw/data/uppnex/igenomes//Homo_sapiens/Ensembl/GRCh37/Sequence/BWAIndex/
MACS Config : data/macsconfig.txt
Saturation analysis : false
MACS broad peaks : false
Blacklist filtering : false
Extend Reads : 100 bp
Current home : /home/phil
Current user : phil
Current path : /home/phil/demo_data/ChIP/Human/test
Working dir : /home/phil/demo_data/ChIP/Human/test/work
Output dir : ./results
R libraries : /home/phil/R/nxtflow_libs/
Script dir : /home/phil/GitHub/NGI-ChIPseq
Save Reference : false
Save Trimmed : false
Save Intermeds : false
Trim R1 : 0
Trim R2 : 0
Trim 3' R1 : 0
Trim 3' R2 : 0
Config Profile : UPPMAX
UPPMAX Project : b2017001
E-mail Address : phil.ewels@scilifelab.se
====================================](https://image.slidesharecdn.com/philewels-nbischip-seqcourse-171124160626/85/NBIS-ChIP-seq-course-36-320.jpg)
![NGI stockholm
Version control
$fastqc.module = ['FastQC/0.7.2']
$trim_galore.module = ['TrimGalore/0.4.1', 'FastQC/0.7.2']
$bw.module = ['bwa/0.7.8', 'samtools/1.5']
$samtools.module = ['samtools/1.5', 'BEDTools/2.26.0']
$picard.module = ['picard/2.10.3', 'samtools/1.5']
$phantompeakqualtools.module = ['phantompeakqualtools/1.1']
$deepTools.module = ['deepTools/2.5.1']
$ngsplot.module = ['samtools/1.5', 'R/3.2.3', 'ngsplot/2.61']
$macs.module = ['MACS/2.1.0']
$saturation.module = ['MACS/2.1.0', 'samtools/1.5']
$saturation_r.module = ['R/3.2.3']](https://image.slidesharecdn.com/philewels-nbischip-seqcourse-171124160626/85/NBIS-ChIP-seq-course-37-320.jpg)
NBIS ChIP-seq course
- 1. NGI stockholm NGI-ChIPseq Processing ChIP-seq data at the National Genomics Infrastructure Phil Ewels phil.ewels@scilifelab.se NBIS ChIP-seq tutorial 2017-11-29
- 2. NGI stockholm SciLifeLab NGI Our mission is to offer a state-of-the-art infrastructure for massively parallel DNA sequencing and SNP genotyping, available to researchers all over Sweden
- 3. NGI stockholm SciLifeLab NGI National resource State-of-the-art infrastructure Guidelines and support We provide guidelines and support for sample collection, study design, protocol selection and bioinformatics analysis
- 4. NGI stockholm NGI Organisation NGI Stockholm NGI Uppsala
- 5. NGI stockholm NGI Organisation Funding Staff salaries Premises and service contracts Capital equipment Host universities SciLifeLab VR KAW User fees Reagent costs NGI Stockholm NGI Uppsala
- 6. NGI stockholm Project timeline Sample QC Library preparation, Sequencing, Genotyping Data processing and primary analysis Scientific support and project consultation Data delivery
- 7. NGI stockholm Methods offered at NGI Exome sequencing Nanoporesequencing ATAC-seq Metagenomics ChIP-seqBisulphite sequencing RAD-seq RNA-seq de novo Whole Genome seq Data analysis included for FREE Just Sequencing Accredited methods
- 8. NGI stockholm ChIP-seq: NGI Stockholm • You do the ChIP, we do the seq • Rubicon ThruPlex DNA (NGI Production) • Min 1 ng input • Min 10 μl • 0.2-10 ng/μl • Ins. size 200-800 bp • 963 kr / prep
- 9. NGI stockholm ChIP-seq: NGI Stockholm • You do the ChIP, we do the seq • Rubicon ThruPlex DNA (NGI Production) • Typically run SE 50bp • Illumina HiSeq High Output mode v4, SR 1x50bp • 1226 kr / sample (40M reads)
- 10. NGI stockholm ChIP-seq: NGI Stockholm • You do the ChIP, we do the seq • Rubicon ThruPlex DNA (NGI Production) • Typically run SE 50bp • Start by organising a planning meeting https://ngisweden.scilifelab.se
- 11. NGI stockholm ChIP-seq Pipeline • Takes raw FastQ sequencing data as input • Provides range of results • Alignments (BAM) • Peaks (optionally filtered) • Quality Control • Pipeline in use since early 2017 (on request) NGI-ChIPseq
- 12. ChIP-seq Pipeline FastQC TrimGalore! BWA Samtools, Picard Phantompeakqualtools deepTools NGSPlot MACS2 Bedtools MultiQC Sequence QC Read trimming Alignment Sort, index, mark duplicates Strand cross-correlation QC Fingerprint, sample correlation TSS / Gene profile plots Peak calling Filtering blacklisted regions Reporting NGI-ChIPseq FastQ BAM HTML BED
- 13. NGI stockholm Nextflow • Tool to manage computational pipelines • Handles interaction with compute infrastructure • Easy to learn how to run, minimal oversight required
- 15. NGI stockholm Nextflow #!/usr/bin/env nextflow input = Channel.fromFilePairs( params.reads ) process fastqc { input: file reads from input output: file "*_fastqc.{zip,html}" into results script: """ fastqc -q $reads """ } https://www.nextflow.io/
- 16. NGI stockholm Default: Run locally, assume software is installed Nextflow #!/usr/bin/env nextflow input = Channel.fromFilePairs( params.reads ) process fastqc { input: file reads from input output: file "*_fastqc.{zip,html}" into results script: """ fastqc -q $reads """ } process { executor = 'slurm' clusterOptions = { "-A b2017123" } cpus = 1 memory = 8.GB time = 2.h $fastqc { module = ['bioinfo-tools', ‘FastQC'] } } Submit jobs to SLURM queue Use environment modules
- 17. NGI stockholm Nextflow #!/usr/bin/env nextflow input = Channel.fromFilePairs( params.reads ) process fastqc { input: file reads from input output: file "*_fastqc.{zip,html}" into results script: """ fastqc -q $reads """ } process { executor = 'slurm' clusterOptions = { "-A b2017123" } cpus = 1 memory = 8.GB time = 2.h $fastqc { module = ['bioinfo-tools', ‘FastQC'] } } docker { enabled = true } process { container = 'biocontainers/fastqc' cpus = 1 memory = 8.GB time = 2.h } Run locally, use docker container for all software dependencies
- 20. NGI stockholm Running NGI-ChIPseq Step 1: Install Nextflow • Uppmax - load the Nextflow module module load nextflow • Anywhere (including Uppmax) - install Nextflow curl -s https://get.nextflow.io | bash Step 2: Try running NGI-ChIPseq pipeline nextflow run SciLifeLab/NGI-ChIPseq --help
- 21. NGI stockholm Running NGI-ChIPseq Step 3: Choose your reference • Common organism - use iGenomes --genome GRCh37 • MACS peak calling config file --macsconfig config.csv Step 4: Organise your data • One (if single-end) or two (if paired-end) FastQ per sample • Everything in one directory, simple filenames help!
- 22. NGI stockholm Running NGI-ChIPseq Step 5: Run the pipeline on your data • Remember to run detached from your terminal screen / tmux / nohup Step 6: Check your results • Read the Nextflow log and check the MultiQC report Step 7: Delete temporary files • Delete the ./work directory, which holds all intermediates
- 23. NGI stockholm Using UPPMAX nextflow run SciLifeLab/NGI-ChIPseq --project b2017123 --genome GRCh37 --macsconfig p.txt --reads "data/*_R{1,2}.fastq.gz" • Default config is for UPPMAX • Knows about central iGenomes references • Uses centrally installed software
- 24. NGI stockholm Using other clusters nextflow run SciLifeLab/NGI-ChIPseq -profile hebbe --bwaindex ./ref --macsconfig p.txt --reads "data/*_R{1,2}.fastq.gz" • Can run just about anywhere • Supports local, SGE, LSF, SLURM, PBS/Torque, HTCondor, DRMAA, DNAnexus, Ignite, Kubernetes
- 25. NGI stockholm Using Docker nextflow run SciLifeLab/NGI-ChIPseq -profile docker --fasta genome.fa --macsconfig p.txt --reads "data/*_R{1,2}.fastq.gz" • Can run anywhere with Docker • Downloads required software and runs in a container • Portable and reproducible.
- 26. NGI stockholm Using AWS nextflow run SciLifeLab/NGI-ChIPseq -profile aws --genome GRCh37 --macsconfig p.txt --reads "s3://my-bucket/*_{1,2}.fq.gz" --outdir "s3://my-bucket/results/" • Runs on the AWS cloud with Docker • Pay-as-you go, flexible computing • Can launch from anywhere with minimal configuration
- 27. NGI stockholm Input data ERROR ~ Cannot find any reads matching: XXXX NB: Path needs to be enclosed in quotes! NB: Path requires at least one * wildcard! If this is single-end data, please specify --singleEnd on the command line. --reads '*_R{1,2}.fastq.gz' --reads '*.fastq.gz' --singleEnd --reads *_R{1,2}.fastq.gz --reads '*.fastq.gz' --reads sample.fastq.gz
- 28. NGI stockholm Read trimming • Pipeline runs TrimGalore! to remove adapter contamination and low quality bases automatically • Use --notrim to disable this • Some library preps also include additional adapters --clip_r1 [int] --clip_r2 [int] --three_prime_clip_r1 [int] --three_prime_clip_r2 [int]
- 29. NGI stockholm Blacklist filtering • Some parts of the reference genome collect incorrectly mapped reads • Good practice to remove these peaks • Pipeline has ENCODE regions for Human & Mouse • Can pass own BED file of custom regions --blacklist_filtering --blacklist regions.bed
- 30. NGI stockholm Broad Peaks • Some chromatin profiles don't have narrow, sharp peaks • For example, H3K9me3 & H3K27me3 • MACS2 can call peaks in "broad peak" mode • Pipeline uses default qvalue cutoff of 0.1 --broad
- 31. NGI stockholm Extending Read Length • When using single-end data, sequenced read length is shorter than the sequence fragment length • For DeepTools, need to "extend" the read length • Set to 100bp by default. Use this parameter to customise this value. • Expected fragment length - sequence read length --extendReadsLen [int]
- 32. NGI stockholm Saving intermediates • By default, the pipeline doesn't save some intermediate files to your final results directory • Reference genome indices that have been built • FastQ files from TrimGalore! • BAM files from STAR (we have BAMs from Picard) --saveReference --saveTrimmed --saveAlignedIntermediates
- 33. NGI stockholm Resuming pipelines • If something goes wrong, you can resume a stopped pipeline • Will use cached versions of completed processes • NB: Only one hyphen! -resume • Can resume specific past runs • Use nextflow log to find job names -resume job_name
- 34. NGI stockholm Customising output -name Give a name to your run. Used in logs and reports --outdir Specify the directory for saved results --saturation Run saturation analysis, subsampling reads from 10% - 100% --email Get e-mailed a summary report when the pipeline finishes
- 35. NGI stockholm Nextflow config files • Can save a config file with defaults • Anything with two hyphens is a params params { email = 'phil.ewels@scilifelab.se' project = "b2017123" } process.$multiqc.module = [] ./nextflow.config ~/.nextflow/config -c /path/to/my.config
- 36. NGI stockholm NGI-ChIPseq config N E X T F L O W ~ version 0.26.1 Launching `SciLifeLab/NGI-ChIPseq` [deadly_bose] - revision: 28e24c2a2a ========================================= NGI-ChIPseq: ChIP-Seq Best Practice v1.4 ========================================= Run Name : deadly_bose Reads : data/*fastq.gz Data Type : Single-End Genome : GRCh37 BWA Index : /sw/data/uppnex/igenomes//Homo_sapiens/Ensembl/GRCh37/Sequence/BWAIndex/ MACS Config : data/macsconfig.txt Saturation analysis : false MACS broad peaks : false Blacklist filtering : false Extend Reads : 100 bp Current home : /home/phil Current user : phil Current path : /home/phil/demo_data/ChIP/Human/test Working dir : /home/phil/demo_data/ChIP/Human/test/work Output dir : ./results R libraries : /home/phil/R/nxtflow_libs/ Script dir : /home/phil/GitHub/NGI-ChIPseq Save Reference : false Save Trimmed : false Save Intermeds : false Trim R1 : 0 Trim R2 : 0 Trim 3' R1 : 0 Trim 3' R2 : 0 Config Profile : UPPMAX UPPMAX Project : b2017001 E-mail Address : phil.ewels@scilifelab.se ====================================
- 37. NGI stockholm Version control $fastqc.module = ['FastQC/0.7.2'] $trim_galore.module = ['TrimGalore/0.4.1', 'FastQC/0.7.2'] $bw.module = ['bwa/0.7.8', 'samtools/1.5'] $samtools.module = ['samtools/1.5', 'BEDTools/2.26.0'] $picard.module = ['picard/2.10.3', 'samtools/1.5'] $phantompeakqualtools.module = ['phantompeakqualtools/1.1'] $deepTools.module = ['deepTools/2.5.1'] $ngsplot.module = ['samtools/1.5', 'R/3.2.3', 'ngsplot/2.61'] $macs.module = ['MACS/2.1.0'] $saturation.module = ['MACS/2.1.0', 'samtools/1.5'] $saturation_r.module = ['R/3.2.3']
- 38. NGI stockholm Version control • Pipeline is always released under a stable version tag • Software versions and code reproducible • For full reproducibility, specify version revision when running the pipeline nextflow run SciLifeLab/NGI-ChIPseq -r v1.3
- 39. Conclusion • Use NGI-ChIPseq to prepare your data if you want: • To not have to remember every parameter for every tool • Extreme reproducibility • Ability to run on virtually any environment • Now running for all ChIPseq projects at NGI-Stockholm
- 41. Conclusion NGI stockholm SciLifeLab/NGI-RNAseq https://github.com/ SciLifeLab/NGI-MethylSeq SciLifeLab/NGI-smRNAseq SciLifeLab/NGI-ChIPseq Acknowledgements Phil Ewels Chuan Wang Jakub Westholm Rickard Hammarén Max Käller Denis Moreno NGI Stockholm Genomics Applications Development Group support@ngisweden.se http://opensource.scilifelab.se