Big data solution for ngs data analysis
3 likes609 views
This document outlines a presentation on big data solutions for NGS data analysis using software containerization and distributed analytics. It discusses Docker containerization and its use at Atgenomix to simplify cluster environments. It also covers NGS genome analysis techniques like read mapping, variant calling, and using Spark and Hadoop for data parallelization. Elasticsearch is introduced for distributed, RESTful search and analytics of variant data.
![Lucene Engine
70
The quick brown fox jumped over the lazy dogs
[The] [quick] [brown] [fox] [jumped] [over] [the] [lazy] [dogs]
[the] [quick] [brown] [fox] [jumped] [over] [the] [lazy] [dogs]
Whitespace analysis
Simple analysis
[quick] [brown] [fox] [jumped] [over] [lazy] [dogs]
Stop analysis
[quick] [brown] [fox] [jump] [over] [lazy] [dog]
Standard analysis
https://www.slideshare.net/otisg/lucene-introduction](https://image.slidesharecdn.com/bigdatasolutionforngsdataanalysis-180718054011/85/Big-data-solution-for-ngs-data-analysis-70-320.jpg)
![Query DSL
83
• curl with shell:
curl -XPOST --header "content-type: application/JSON"
es-docker:9200/demo_table/variant/_search?
-d '{ "query": { "bool": { "filter": { "bool": { "must": [], "must_not": [] } } } } }'](https://image.slidesharecdn.com/bigdatasolutionforngsdataanalysis-180718054011/85/Big-data-solution-for-ngs-data-analysis-83-320.jpg)
Big data solution for ngs data analysis
- 1. Taipei | www.atgenomix.com | info@atgenomix.com BigData Solution for NGS Data Anlaysis Steven Li, Sr. Data Scientist yunlung_li@atgenomix.com
- 2. Outline •Software Containerization •NGS genome analysis •Distributed, RESTful search and analytics 2
- 3. Software Containerization •Docker Introduction •Docker In Action •Docker in Atgenomix SeqsLab 3
- 5. Docker Container Image 5 • container image - a lightweight, stand-alone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, settings. https://www.docker.com/what-container
- 6. Docker Container 6 https://www.docker.com/what-container • A VM as light as a process
- 9. Run Your System Everywhere 9 https://www.wired.com/2016/07/windows-10-free-upgrade-ends / WindowsWindows
- 11. Container Warehouse 11 https://hub.docker.com/ • https://hub.docker.com/ • https://store.docker.com/ • https://aws.amazon.com/ecr/ • private docker registry • …
- 14. 14 https://github.com/Haufe-Lexware/docker-style-guide/blob/master/DockerImage.md
- 15. Workshop Environment 15 • docker cloud account registration • https://cloud.docker.com/ • docker training course and browser-based terminal labs • https://training.play-with-docker.com/ops-s1-hello/
- 17. Workshop 1-1 Hello World 17 https://training.play-with-docker.com/ops-s1-hello/
- 18. Workshop 1-2 Image created with Container 18 • generate image from a fresh ubuntu container https://training.play-with-docker.com/ops-s1-images/
- 20. Workshop 1-4 Filesystem Mount 20 /mnt • mounting host OS file path to container file path host OS /root/mount_test
- 21. Workshop 1-5 Network - Container to Host 21 • mapping host OS port to container ports host OS :8080 :80
- 22. Workshop 1-6 Network - Inter-container es-docker test my-net • https://docs.docker.com/network/bridge/#connect-a-container-to-a-user-defined-bridge
- 23. Workshop 1-7 Network - Run Container as Host 23 • run container using host network stack
- 24. Atgenomix SeqsLab portal master slave slave
- 26. Docker in Atgenomix SeqsLab portal master slave slave • We use docker to simplify cluster environment management Atgenomix SedsLab Atgenomix SedsLab Atgenomix SedsLab Atgenomix SedsLab
- 27. Docker with IT Automation Window Atgenomix SeqsLab Window Atgenomix SeqsLab Windows Atgenomix SeqsLab
- 28. Demo 28 •SeqsLab ansible deployment •SeqsLab docker environment
- 29. Outline •Software Containerization •NGS genome analysis •Distributed, RESTful search and analytics 29
- 30. NGS Genome Analysis •NGS Bioinformatics introduction •Workshop: contaiernized DNA-Seq Pipeline •DNA-Seq Pipeline in Atgenomix SeqsLab 30
- 31. Bioinformatics 31 • Science of biological sequences started from 1970s • 1980s to 1990s, protein era • 2000s, human genome project • 2007, Next Generation Sequencing Margrett Dayhoff http://www.whatisbiotechnology.org/index.php/people/summary/Dayhoff
- 32. Next Generation Sequencing 32 NGS Debuted Hiseq X10 Debuted https://www.genome.gov/images/content/costpergenome2015_4.jpg 2017 Illumina NovaSeq 6000 $100 per genome • Hiseq X10 cost: $ 1K throughput: 50 WGS per day ~5TB per day • data deluge • precision medicine
- 33. Sequencing by Synthesis Technology 33 https://www.youtube.com/watch?v=77r5p8IBwJk
- 35. Sequencing Type •Whole Genome Sequencing, 3 Billion bp, ~100GB per sample (30x) •Whole Exome Seqeuencing, exon regions (1.1% of genome), ~10GB per sample (200x) •Target Panel Sequencing, full gene regions of certain genes 35
- 36. Coverage 36 •How many times the genome is covered by reads C = N*L / G •C: coverage •N: number of reads •L: read length •G: genome length
- 39. Fastq 39 Fastq format @SEQ_ID Sequenced DNA base + Sequencing quality size: 100GB gz (WGS, 30x) counts: billions
- 41. Resequencing Approach reference genome sequnced reads reference genome sequnced genome
- 42. Resequencing Approach reference genome sequnced reads reference genome sequnced genome fastq bam vcf read mapping variant calling
- 43. Reads to Genome Mapping suffix array / FM-index data structure
- 44. SAM/BAM
- 45. SAM/BAM
- 46. Variant Type •Single NucleotideVariant (SNV) •short insertion or deleteion (INDEL): <50bp •structure variant (SV): >50bp
- 50. Structure Variant •short read challange •repeat challange 50 http://www.nature.com/nmeth/journal/v9/n2/full/nmeth.1858.html
- 51. Variant Call Format • header (##) • chromsome, POS, rsid • ref allele / alt allele • quality / filter • info • sample info
- 52. Workshop2 FASTQ to VCF Pipeline 52 1. SNP-indel pipeline: bwa-samtools 2. structure variant pipeline: bwa-delly2 https://training.play-with-docker.com/ops-s1-hello/
- 53. Workshop2-1 bwa-samtools Pipeline read mapping BWA mem BWA index ref.fa ref.fa.amb ref.fa.ann ref.fa.bwt ref.fa.pac ref.fa.sa in.fq t.sam samtools sort t.bam samtools index t.bai t.bam samtools mpileup r.bcf bcftools call snp.vcf.gz variant calling 1. https://github.com/obigbando/demo_case 2. https://github.com/lh3/bwa-docker 3. https://hub.docker.com/r/obigbando/samtools/ 4. https://hub.docker.com/r/obigbando/bcftools/ 5. OS filesystem mount bam process
- 54. Workshop2-2 bwa-delly Pipeline ref.fa samtools faidx t.bai t.bam delly2 call r.bcf bcftools view sv.vcf.gz variant calling Read Mapping bam process ref.fa.fai 1. https://github.com/dellytools/delly/tree/master/docker 2. https://hub.docker.com/r/obigbando/samtools/ 3. https://hub.docker.com/r/obigbando/bcftools/
- 55. Atgenomix SeqsLab 55 • Bioinformatics on Spark / Hadoop • Data parallelization
- 57. Hadoop / Spark https://www.safaribooksonline.com/library/view/data-analytics-with/9781491913734/ch04.html http://vijaytraining.com/product/big-data/ • Hadoop distributed filesystem (HDFS) • Yet Another Resource Negotiator(Yarn) • Spark
- 58. Atgenomix SeqsLab 58 Genome reference HG19 HG38 (primary assembly) (primary assembly+alt+decoy+HLA) Filtering with population databases: 1000G indels, dbSNP, COSMIC Short-Read mapping BWA-MEM alt-aware Germline SNP/INDEL GATK3/GATK4 HaplotypeCaller Somatic SNP/INDEL MuTect2 VarScan2 Germline / Somatic SV Delly2 - DEL, DUP, INS, INV, TRA Somatic CNV GATK4 CNV Joint Genotyping GATK4 GenotypeGVCFs Haplotype phasing WhatsHap
- 59. Data Parallelization FastQ to Bam 59
- 60. Data Parallelization Bam to VCF • Human genome is naturally separated by chromosome, so we can partition DNA data by chromosome without any information loss. • There are several partitioning strategies to get more partitions for better parallelization. - Centromere - Long ambiguous regions - Any customized regions 60
- 61. Data Parallelization Bam to VCF 61
- 62. 62 SeqsLab Demo
- 63. Outline •Software Containerization •NGS genome analysis •Distributed, RESTful search and analytics 63
- 64. Containerized DNA-Seq Pipeline •Elasticsearch Introduction •Workshop: from variant data to search •Elasticsearch in Atgenomix SeqsLab 64
- 66. Elasticsearch 66 • NoSQL with full-text search • Lucene Search Engine • Distributed Architecture • RESTFUL API
- 67. NoSQL • NoSQL category: • key-value • document-oriented • terms comparison to RDB • example 67 RDB ElasticSearch database index table type row document column field schema mapping SQL sql DSL
- 68. Lucene Engine 68 https://www.slideshare.net/otisg/lucene-introduction • all fields indexed
- 69. Lucene Engine 69
- 70. Lucene Engine 70 The quick brown fox jumped over the lazy dogs [The] [quick] [brown] [fox] [jumped] [over] [the] [lazy] [dogs] [the] [quick] [brown] [fox] [jumped] [over] [the] [lazy] [dogs] Whitespace analysis Simple analysis [quick] [brown] [fox] [jumped] [over] [lazy] [dogs] Stop analysis [quick] [brown] [fox] [jump] [over] [lazy] [dog] Standard analysis https://www.slideshare.net/otisg/lucene-introduction
- 72. Elasticsearch Glossary 72 • node: an Elasticsearch instance (server node) • cluster: some nodes • index: RDB database in Elasticsearch world • shard: a component of index for horizontal scaling and operation parallelization • primary / secondary shard https://www.slideshare.net/dadoonet/elasticsearch-devoxx-france-2012-english-version?qid=d5dd27a4-2c66-43b9-9ae4-6e9668ca8978&v=&b=&from_search=22
- 73. Distributed Architecture 73 node0 shard0 { "_index": “tour_sites", "_type": “Taipei”, "_id": “0000000000", "_source": { "title": “National Palace Museum” } } { "_index": “tour_sites", "_type": “Taipei”, "_id": “0000000001", "_source": { "title": “Taipei 101” } } https://www.slideshare.net/ssuser6bb62e/elasticsearch-97018162?qid=09f88107-4bf9-4caa-be51-e2d9286819e7&v=&b=&from_search=11 tour_sites
- 74. Distributed Architecture 74 node2node1 P1 R1 P0 node3 R0 P2P2 https://www.slideshare.net/ssuser6bb62e/elasticsearch-97018162?qid=09f88107-4bf9-4caa-be51-e2d9286819e7&v=&b=&from_search=11 { "_index": “tour_sites", "_type": “Taipei”, "_id": "0000000000", "_source": { "title": “National Palace Museum” } } { "_index": “tour_sites", "_type": “Taipei”, "_id": “0000000001", "_source": { "title": “Taipei 101” } } tour_sites
- 75. RESTful API 75 • Extensive REST API for everthing about Elasticsearch • cluster API • cat API • document API • query DSL • kibana UI interface
- 76. 76 • get cluster wise information Cluster API • sample command
- 77. Document API 77 DML ES endpoint description create _bulk elasticsearch document import read _search elasticsearch document retrival update _update_by_query elasticsearch data update delete _delete_by_query elasticsearch data deletion
- 78. CREATE 78https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html • index API • bulk API
- 79. READ 79https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html • index API • search API
- 80. UPDATE 80https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html • _update API: partial document merge and update • index API: overwrite existing document
- 81. DELETE 81https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html • index API • delete_by_query API
- 82. Query DSL 82 • Elasticsearch supports vast amount of different queries • compound query clauses • term query for structured data
- 83. Query DSL 83 • curl with shell: curl -XPOST --header "content-type: application/JSON" es-docker:9200/demo_table/variant/_search? -d '{ "query": { "bool": { "filter": { "bool": { "must": [], "must_not": [] } } } } }'
- 84. Compound Query 84 • term query
- 85. Compound Query 85 • range query
- 86. Compound Query 86 • terms query => or condition
- 87. Compound Query 87 • term query and range query
- 88. Aggregation 88 • Elasticsearch supports vast amount of different aggregations
- 89. 89 • Elasticsearch supports vast amount of different aggregations Aggregation
- 90. Query x Aggregation 90 • aggregate on entry returned by query
- 91. 91
- 92. Workshop3 VCF2ES es-kibana vcf2es my-net • https://hub.docker.com/r/obigbando/es-kibana • https://hub.docker.com/r/obigbando/vcf2es/ • https://docs.docker.com/network/bridge/#connect-a-container-to-a-user-defined-bridge
- 93. Workshop3 VCF2ES • Answer the following questions: • how many variant in the demoVCF • how many transitions / trasversion variants • how many deletion variants in chrX • how many variant in chromosome 22 p-arm • quality distribution (qual) of all variants • how many INS, DEL, INV, DUP,TRA in chromosome 1
- 95. Discussion 95