Hadoop for Bioinformatics
46 likes19,668 views
This document discusses how Hadoop can be used for bioinformatics applications. It provides examples of how Hadoop has been used to efficiently process large genomic datasets, such as read mapping and genome assembly, in a distributed, parallel manner. Hadoop allows bioinformatics workflows and algorithms to be rethought and scaled to handle the growing size of genomic data. Key applications discussed include read mapping, variant discovery, and de novo assembly.
Hadoop for Bioinformatics
- 1. Hadoop for Bioinformatics Deepak Singh Amazon Web Services Hadoop World, NYC
- 3. Via Reavel under a CC-BY-NC-ND license
- 4. By ~Prescott under a CC-BY-NC license
- 5. data sets
- 7. PFAM PDB GENBANK ENSEMBL Many Others
- 8. manageable
- 10. Human genom e Image: Matt Wood
- 13. Image: Matt Wood
- 14. ~100 TB/Week Image: Matt Wood
- 15. ~100 TB/Week >2 PB/Year Image: Matt Wood
- 17. years
- 18. days
- 19. hours
- 20. gigabytes
- 21. terabytes
- 22. petabytes
- 23. really fast
- 30. Via Christolakis under a CC-BY-NC-ND license
- 31. Via Argonne National Labs under a CC-BY-SA license
- 32. killer app Via Argonne National Labs under a CC-BY-SA license
- 33. Via asklar under a CC-BY license
- 43. scalability
- 44. we are data geeks not data center geeks
- 45. two key trends
- 51. use applications
- 52. some work
- 53. filters some work ^
- 54. High Throughput Sequence Analysis Mike Schatz, University of Maryland
- 55. • Read Mapping • Mapping & SNP Discovery • De novo Genome Assembly
- 57. Asian Individual Genome: 3.3 Billion 35bp, 104 GB (Wang et al., 2008) African Individual Genome: 4.0 Billion 35bp, 144 GB (Bentley et al., 2008)
- 58. Alignment > 10000 CPU hrs
- 59. Seed & Extend Good alignments must have significant exact alignment Minimal exact alignment length = l/(k+1)
- 60. Seed & Extend Good alignments must have significant exact alignment Minimal exact alignment length = l/(k+1) Expensive to scale
- 61. Seed & Extend Good alignments must have significant exact alignment Minimal exact alignment length = l/(k+1) Expensive to scale
- 62. Seed & Extend Good alignments must have significant exact alignment Minimal exact alignment length = l/(k+1) Expensive to scale Need parallelization framework
- 63. CloudBurst Catalog k-mers Collect seeds End-to-end alignment
- 66. CloudBurst efficiently reports every k-difference alignment of every read
- 67. many applications only need the best alignment
- 68. Bowtie: Ultrafast short read aligner Langmead B, Trapnell C, Pop M, Salzberg SL. Ultrafast and memory-efficient alignment of short DNA sequences to the human genome. Genome Biol 10 (3): R25.
- 69. SOAPSnp: Consensus alignment and SNP calling Langmead B, Trapnell C, Pop M, Salzberg SL. Ultrafast and memory-efficient alignment of short DNA sequences to the human genome. Genome Biol 10 (3): R25.
- 70. Crossbow: Rapid whole genome SNP analysis Ben Langmead Langmead B, Trapnell C, Pop M, Salzberg SL. Ultrafast and memory-efficient alignment of short DNA sequences to the human genome. Genome Biol 10 (3): R25.
- 73. Preprocessed reads Map: Bowtie
- 74. Preprocessed reads Map: Bowtie Sort: Bin and partition
- 75. Preprocessed reads Map: Bowtie Sort: Bin and partition Reduce: SoapSNP
- 76. Crossbow condenses over 1,000 hours of resequencing computation into a few hours without requiring the user to own or operate a computer cluster
- 78. Estimating relative evolutionary rates from sequence comparisons: Identification of probable orthologs Admissible comparisons: A or B vs. D C vs. E Inadmissible comparisons: A or B vs. E C vs. D A B C D E species tree gene tree S. cerevisiae C. elegans
- 79. Estimating relative evolutionary rates from sequence comparisons: 1. Orthologs found using the Reciprocal smallest distance algorithm 2. Build alignment between two orthologs >Sequence C MSGRTILASTIAKPFQEEVTKAVKQLNFT-----PKLVGLLSNEDPAAKMYANWTGKTCESLGFKYEL-… >Sequence E MSGRTILASKVAETFNTEIINNVEEYKKTHNGQGPLLVGFLANNDPAAKMYATWTQKTSESMGFRYDL… 3. Estimate distance given a substitution matrix Phe Ala Pro Leu Thr Phe Ala µπ Pro µπ µπ µπ Leu µπ µπ µπ µπ A B C D E species tree gene tree S. cerevisiae C. elegans
- 80. RSD algorithm summary Genome I Genome J Ib Jc Align sequences & Calculate distances L Orthologs: Align sequences & Calculate distances H ib - jc D = 0.1 c vs. D=1.2 vs. D=0.2 a b a vs. D=0.1 vs. D=0.3 c b b b vs. D=0.9 vs. D=0.1 c c b c
- 81. Prof. Dennis Wall Harvard Medical School
- 82. Roundup is a database of orthologs and their evolutionary distances. To get started, click browse. Alternatively, you can read our documentation here. Good luck, researchers!
- 84. 1000 genomes = 5,994,000 processes = 23,976,000 hours
- 85. 2737 years
- 86. periodic task
- 87. must scale up
- 89. hadoop streaming
- 92. what’s next?
- 93. de novo assembly
- 94. machine learning and statistics
- 96. docking
- 99. the ecosystem
- 100. Pig
- 101. Cascading
- 102. Hive
- 103. RHIPE
- 104. domain specific libraries and tools
- 109. Thank you! deesingh@amazon.com; Twitter:@mndoci Presentation ideas from @mza, @simon and @lessig