New methods deep variant evaluation of draft v4alpha
Download as PPTX, PDF0 likes462 views
DeepVariant is using a convolutional neural network with 26 layers to call variants from genomic data encoded as tensors representing probability distributions. The document evaluates DeepVariant's performance on different datasets, finding that while indel call accuracy improved in the v4 truth set, SNP call accuracy was mixed due to errors in the v4 labels. Error analysis found DeepVariant was usually correct when it differed from v4, and v4 errors dominated remaining discrepancies, suggesting label errors limit training. With label errors addressed, DeepVariant may achieve very high accuracy.
New methods deep variant evaluation of draft v4alpha
- 1. Assessing the v4𝛂 truth set with DeepVariant Andrew Carroll - awcarroll@google.com Genomics team in Google Brain
- 2. P 2 s v or a Genomics Conv Network - 26 layers Data encoded as "tensors" { 0.001, 0.994, 0.005 } { 0.001, 0.990, 0.009 } { 0.000, 0.001, 0.999 } { 0.600, 0.399, 0.001 } { HOM_REF, HET, HOM_VAR } Probability distribution DeepVariant: Deep Learning for Variant Calling
- 4. P 4 s v or a Genomics DeepVariant: Deep Learning for Variant Calling HET VAR HOM VAR REF Training: Show example + label to update weights Label Example Updated Weights Training Strategy: Use chr1-19 for training, chr21-22 for tune. Never train on chr20 in any sample, Never train on HG002 (except only now on PacBio CCS)
- 5. P 5Genomics Training and Applying DeepVariant on CCS Input Data: Model Trained On: Results: 8M chip, 32x coverage 11kb insert, In GIAB FTP Mix of 1M+8M, inserts 9x-30x coverage Release Model 8M chip 100x coverage Mix of inserts 8M chip 100x coverage Mix of inserts Type Metric v3.3.2 v4 SNP Recall 99.95% 99.70% Precision 99.85% 99.71% F1 99.90% 99.71% Indel Recall 98.22% 97.83% Precision 98.35% 98.22% F1 98.29% 98.03% Type Metric v3.3.2 v4 SNP Recall 99.96% 99.40% Precision 99.84% 99.92% F1 99.90% 99.66% Indel Recall 99.35% 99.26% Precision 99.30% 99.48% F1 99.33% 99.37%
- 6. P 6Genomics Observations SNP accuracy not improved with more coverage (more in a bit) Indel accuracy in v4 is higher. Also much improved by more coverage Experiment: Train with v4 truth set. Then evaluate on v3.3.2 Result: Indel F1 improves ; SNP F1 decreases All evidence here points to Indel changes in v4 being quite positive, but SNP mixed
- 7. P 7Genomics Error Analysis on 100x model:
- 8. P 8Genomics Summary of Error Analysis: INDELS: Indel errors are beyond my ability to assess (homopolymer junctions, diverse evidence in reads) SNPs: When DeepVariant makes a SNP call not in v4𝛂, DeepVariant is almost always correct 128 putative FP on chr20 || I assess 2 are real FP, 4 unclear, 122 v4𝛂 errors When DeepVariant does not call a SNP present in v4𝛂, v4𝛂 is always correct (that I looked) And there is a characteristic signature of nearby variants in these errors
- 9. P 9Genomics DeepVariant FNs/Unlikely Errors in v4 100x PacBio CCS (8M) 28x PacBio CCS (1M) 84x 10X HP-trio
- 10. P 10Genomics DeepVariant FPs/Likely Errors in v4 100x PacBio CCS (8M) 28x PacBio CCS (1M) 84x 10X HP-trio
- 11. P 11Genomics Conclusions ● Truth labels for indels look improved across every metric investigated ● Truth label errors dominate the real remaining errors for DeepVariant in SNPs on v4 ● DeepVariant is able to call true signal in this data very well, is left trying to figure out mislabel signature ● This seems to be be associated with nearby variants in regions hard to map with Illumina ● This label error currently limits our ability to use v4 in training (v4 still needs some work for SNPs) ● If the label error is corrected, this may solve the remaining FP+FN (since DeepVariant will not learn the nearby variant signature). Potential accuracy might be very high based on FP # ‘
- 12. P 12 1 605031 . A ATACATGGAGGGGAACAACACACACCAGGGCCTCTCAGGGGGACAGGGGGTAGGAGACCATCAGGAC AAACACGTGGATACATGGAGGGGAACAACACACACCAGGGCCTCTCAGGGGGACAGGGGGTAGGAGA CCATCAGGACAAACACGTGGATACATGGAGGGGAACAACACACACCAGGGCCTCTCAGGGGGACAGG GGGTAGGAGACCATCAGGACAAACACGTGGG 22.3 PASS . GT:GQ:DP:AD:VAF:PL 0/1:22:63:39,19:0.301587:22,0,50 Unexpected Finding - SV Calls in CCS SV Calls in Data: (e.g. 232 bp HET insertion) Type Recall Precision F1 ALL 39.5 % 94.3 % 55.7 % INS 54.3 % 93.2 % 68.6 % DEL 20.3 % 98.6 % 33.7 % Evaluated by Aaron Wenger (Pacbio) on Truvari v0.6 SV setGenomics