Impact.Tech "Statistical Literacy for Deep Tech"
1 like248 views
The document discusses statistical literacy in deep tech, emphasizing the importance of understanding different types of distributions and their implications for data interpretation in fields like healthcare. It highlights how misleading statistics, sensitivity, and specificity can influence decision-making, stressing the need to deconstruct data and analyze its components. Additionally, it addresses the significance of validation methods and the interpretation of p-values in assessing data relevance and quality.
![Truly good and too good to be true
Deconstructing the data into components
is necessary, but not sufficient
Banana factor + RNA
n = 80
AUC 0.97
Breast Cancer
Additional questions
• What’s the current and
assumed future standard
[of care]?
• What’s already known [in
the literature]?
• Etc.
Then…
Dig into the sample
n = 80
Healthy vs diseased? Disease
stage? Age? Prior treatments?
Method of diagnosis? Region?](https://image.slidesharecdn.com/20210504-statisticalliteracy-session123-210504204648/85/Impact-Tech-Statistical-Literacy-for-Deep-Tech-22-320.jpg)
Impact.Tech "Statistical Literacy for Deep Tech"
- 2. Think in distributions Part 1 Noel Jee, PhD https://medium.com/mytake/understanding-different-types-of-distributions-you-will-encounter-as-a-data-scientist-27ea4c375eec
- 3. Noel Jee, PhD Disclaimers • All examples are made up or blinded • None of this is investment advice • All views are my own • Everything is illustrative • This is not a statistics class Credentials • PhD in something science-y • Took that one STAT400 class in college • 4 years of evaluating startups • Until last week, worked at
- 4. Research from top institutions Combined team experience of over 67 years! Changing the world through new biology
- 5. Research from top institutions Combined team experience of over 67 years! Changing the world through new biology
- 6. For illustrative purposes only Healthy Disease Research from top institutions Combined team experience of over 67 years! Changing the world through new biology
- 7. Banana factor + DNA n = 51 AUC 1.00 Breast Cancer Banana factor + RNA n = 80 AUC 0.97 Breast Cancer Banana factor + Protein n = 369 AUC 0.94 Breast Cancer Banana factor + DNA n = 87 AUC 0.98 Colon Cancer Banana factor + Protein n = 46 AUC 0.97 Colon Cancer Banana factor n = 36 AUC 0.89 Systemic Lupus Erythematosus Amazing numbers can be deceiving AUC => ROC => Sensitivity/Specificity
- 8. Most data can be viewed as distributions
- 9. Most data can be viewed as distributions uniform exponential log
- 10. Most data can be viewed as distributions
- 11. Most data can be viewed as distributions Thinking in distributions is a framework – not always applicable but very useful when it is uniform exponential log
- 12. Distributions explain sens/spec tradeoffs Healthy Diseased
- 13. Distributions explain sens/spec tradeoffs True positives Healthy Diseased True negatives FN FP
- 14. Distributions explain sens/spec tradeoffs True positives Healthy Diseased True negatives FN FP
- 15. Distributions explain sens/spec tradeoffs True positives Healthy Diseased True negatives FN FP Technologies change distributions to decrease tradeoffs
- 16. The ROC and the AUC – the AUROC
- 17. The ROC and the AUC – the AUROC Banana factor + Protein n = 369 AUC 0.94 Breast Cancer Banana factor + Protein n = 46 AUC 0.97 Colon Cancer
- 18. The ROC and the AUC – the AUROC AUC can be deceptive – look at the shape of the ROC curve
- 19. Truly good and too good to be true Banana factor + RNA n = 80 AUC 0.97 Breast Cancer
- 20. Truly good and too good to be true Banana factor + RNA n = 80 AUC 0.97 Breast Cancer
- 21. Truly good and too good to be true Banana factor + RNA n = 80 AUC 0.97 Breast Cancer Deconstructing the data into components is necessary, but not sufficient
- 22. Truly good and too good to be true Deconstructing the data into components is necessary, but not sufficient Banana factor + RNA n = 80 AUC 0.97 Breast Cancer Additional questions • What’s the current and assumed future standard [of care]? • What’s already known [in the literature]? • Etc. Then… Dig into the sample n = 80 Healthy vs diseased? Disease stage? Age? Prior treatments? Method of diagnosis? Region?
- 23. Beyond sensitivity and specificity https://step1.medbullets.com/stats/101006/testing-and-screening https://www.medmastery.com/guide/covid-19-clinical-guide/covid-19-test-validity-how-accurate-are-available-tests
- 24. Beyond sensitivity and specificity https://step1.medbullets.com/stats/101006/testing-and-screening https://www.medmastery.com/guide/covid-19-clinical-guide/covid-19-test-validity-how-accurate-are-available-tests
- 25. Beyond sensitivity and specificity https://step1.medbullets.com/stats/101006/testing-and-screening https://www.medmastery.com/guide/covid-19-clinical-guide/covid-19-test-validity-how-accurate-are-available-tests 93.8% sensitivity 95.6% specificity
- 26. Beyond sensitivity and specificity https://step1.medbullets.com/stats/101006/testing-and-screening https://www.medmastery.com/guide/covid-19-clinical-guide/covid-19-test-validity-how-accurate-are-available-tests 93.8% sensitivity 95.6% specificity 53% PPV 99.7% NPV
- 27. Beyond sensitivity and specificity https://step1.medbullets.com/stats/101006/testing-and-screening Great numbers can still lead to a useless test – it’s important to understand the whole picture https://www.medmastery.com/guide/covid-19-clinical-guide/covid-19-test-validity-how-accurate-are-available-tests 93.8% sensitivity 95.6% specificity 53% PPV 99.7% NPV
- 28. A COVID story DOI: 10.3389/fmicb.2020.01818 PCR Antigen Antibody
- 29. A COVID story Antigen test attributes • 80% sensitivity • 15-30 min TAT • < 50% of PCR cost PCR Antigen Antibody
- 30. A COVID story DOI: 10.3389/fmicb.2020.01818 Antigen test attributes • 80% sensitivity • 15-30 min TAT • < 50% of PCR cost PCR Antigen Antibody Sensitivity ~80% of patients who tested positive with PCR “True positive” is a relative phrase; always note the baseline comparison
- 31. A COVID story Antigen test attributes • 80% sensitivity • 15-30 min TAT • < 50% of PCR cost PCR Antigen Antibody Sensitivity ~80% of patients who tested positive with PCR Sample Symptomatic patients with COVID-19 “True positive” is a relative phrase; always note the baseline comparison Including asymptomatic patients reduced the sensitivity below 40%
- 32. Sometimes numbers just mean nothing Accuracy = True positives + True negatives All tests 90% accuracy could mean nothing
- 33. Sometimes numbers just mean nothing Combined team experience of over 67 years! Accuracy = True positives + True negatives All tests 90% accuracy could mean nothing
- 34. Sometimes numbers just mean nothing Combined team experience of over 67 years! Accuracy = True positives + True negatives All tests 90% accuracy could mean nothing https://investor.aclaristx.com/static-files/a47d4b59-da7b-417a-b62f-02ac4210dea7
- 35. “There are three kinds of lies: lies, damned lies, and statistics” Mark Twain* * who was wrongly quoting someone else – sometimes words are deceptive too
- 38. Deconstruct everything Part 2 Noel Jee, PhD
- 39. Defeating aging through big data
- 40. Using AI … big data … unique signature of bok bok 99.9% accuracy … novel drugs bok bok capture $100B total addressable market Not a real company Defeating aging through big data
- 41. “AI” – What do you want it to mean? Statistical regression R2 = explained variation total variation Confidence interval: the range of values where the true mean likely lies Linear Non-linear
- 42. “AI” – What do you want it to mean? Healthy Disease Machine learning sample feature extraction classification output Binary – healthy, disease Ordinal – mild, moderate, severe Nominal – lupus, alzheimer’s, cancer Logistic regression
- 43. “AI” – What do you want it to mean? Healthy Disease Machine learning sample feature extraction classification output
- 44. “AI” – What do you want it to mean? Healthy Disease Machine learning sample feature extraction classification output Start with the output and input before digging into the “AI”
- 45. What’s goes in? The output: 99.9% sensitivity / specificity Output looks great, let’s move on to the input
- 46. What’s goes in? The output: 99.9% sensitivity / specificity “Over 2,000 micro RNAs and 20,000 genes were fed into our proprietary algorithms” “We trained our algorithms on 200 patient samples” Output looks great, let’s move on to the input “Our algorithm was validated on a public dataset of 2,000,000 individuals”
- 47. When is bigger not better? DOI: 10.1093/ofid/ofv093 Heatmap of gene expression “Over 2,000 micro RNAs and 20,000 genes were fed into our proprietary algorithms”
- 48. When is bigger not better? DOI: 10.1093/ofid/ofv093 Overfitting More variables may lead to Heatmap of gene expression
- 49. When is bigger not better? DOI: 10.1093/ofid/ofv093 https://brainstation.io/blog/3-ux-tips-from-the-worlds-first-ux-designer Overfitting Impractical More variables may lead to may be Heatmap of gene expression
- 50. When is bigger not better? DOI: 10.1093/ofid/ofv093 https://brainstation.io/blog/3-ux-tips-from-the-worlds-first-ux-designer Overfitting Impractical More variables may lead to may be and is almost certainly biased Heatmap of gene expression
- 51. Where might bias be hiding? 6 billion base pairs 99% accurate sequencer 60 million errors Whole genome
- 52. Where might bias be hiding? 6 billion base pairs 99% accurate sequencer 60 million errors Whole genome 1% error is biased, not random Bias is intrinsic to each technique
- 53. Where might bias be hiding? DOI: 10.1093/ofid/ofv093
- 54. Where might bias be hiding? 20,000 genes 30% captured 70% not captured DOI: 10.1093/ofid/ofv093
- 55. Where might bias be hiding? 20,000 genes 30% captured 70% not captured May only be 2,000 genes but contain most relevant genes May not be relevant DOI: 10.1093/ofid/ofv093
- 56. Where might bias be hiding? 20,000 genes 30% captured Even bias might not be a dealbreaker, look for consistency 70% not captured DOI: 10.1093/ofid/ofv093 May only be 2,000 genes but contain most relevant genes May not be relevant
- 57. When are great samples not great? 200 doctor-annotated samples were used to train our algorithms
- 58. Patients are not representative of the targeted population When are great samples not great? 200 doctor-annotated samples were used to train our algorithms Googled some celebrity images One doctor diagnosed the entire sample population, introducing bias for the “true positive”
- 59. Patients are not representative of the targeted population When are great samples not great? 200 doctor-annotated samples were used to train our algorithms Googled some celebrity images Great data can still be garbage – context is key One doctor diagnosed the entire sample population, introducing bias for the “true positive”
- 60. When are great samples not great? “But it’s not why the polls were wrong. It just isn’t. People tell the truth when you ask them who they’re voting for. They really do, on average. The reason why the polls are wrong is because the people who were answering these surveys were the wrong people.” David Shor (Vox interview)
- 61. How do I know it’s real? Validation methods have hard tradeoffs for startups Retrospective Prospective Aging Ageless
- 62. How do I know it’s real? Retrospective • Quality of data • Sample size • Sample breakdown • Blinding • Training vs validation Validation methods have hard tradeoffs for startups
- 63. How do I know it’s real? Prospective Retrospective • Quality of data • Sample size • Sample breakdown • Blinding • Training vs validation Validation methods have hard tradeoffs for startups • Recruitment centers • Trial arms • Statistical power • Endpoints • Duration
- 64. How do I know it’s real? Prospective Retrospective • Quality of data • Sample size • Sample breakdown • Blinding • Training vs validation Validation methods have hard tradeoffs for startups • Recruitment centers • Trial arms • Statistical power • Endpoints • Duration
- 68. Find Relevance Part 3 Noel Jee, PhD
- 69. Significance and the pesky p-value
- 70. Significance and the pesky p-value Possible outcomes large p small p
- 71. Significance and the pesky p-value Possible outcomes large p small p Presented p values will almost always be small – when they’re not, they’re usually hidden in supplemental / backup data
- 72. Significance doesn’t imply relevance Significance (decreasing p-value) Effect size
- 73. Significance doesn’t imply relevance Significance (decreasing p-value) Effect size Meh Meh
- 74. Significance doesn’t imply relevance Significance (decreasing p-value) Effect size Meh Meh
- 75. When significance is assumed relevant $- $5 $10 $15 $20 $25 $30 7/1/20 8/20/20 10/9/20 11/28/20 1/17/21 3/8/21 ATI-450 P2a - ~$19 Today - ~$25 Left 4 Dead Data Readout - ~$6 More than 3x jump in share price overnight. Why?
- 76. Significant jump on Significant data https://investor.aclaristx.com/static-files/a47d4b59-da7b-417a-b62f-02ac4210dea7
- 77. Significant jump on significant data https://investor.aclaristx.com/static-files/a47d4b59-da7b-417a-b62f-02ac4210dea7
- 78. Relevance isn’t found in significance • Randomization led to 2 control subjects • Control group was ~10 years younger Significance can add little benefit to assessing risks https://investor.aclaristx.com/static-files/a47d4b59-da7b-417a-b62f-02ac4210dea7
- 79. Getting started with technical diligence Dissect the key data from the pitch
- 80. Getting started with technical diligence Dissect the key data from the pitch Ask yourself if it makes sense
- 81. Getting started with technical diligence Catalogue data into a ‘Rumsfeld Matrix’ Dissect the key data from the pitch Ask yourself if it makes sense
- 82. Getting started with technical diligence Catalogue data into a ‘Rumsfeld Matrix’ Dissect the key data from the pitch Understand experimental methods Ask yourself if it makes sense
- 83. Getting started with technical diligence Catalogue data into a ‘Rumsfeld Matrix’ Dissect the key data from the pitch Understand experimental methods Prioritize and “read” related papers Ask yourself if it makes sense
- 84. Getting started with technical diligence Catalogue data into a ‘Rumsfeld Matrix’ Dissect the key data from the pitch Understand experimental methods Prioritize and “read” related papers Talk to people who know more than you Ask yourself if it makes sense
- 85. How to read journal articles 1. Read the title and abstract 2. Dive into the figures (data) 3. Check out the methods 4. Look at the supplemental data If everything checks out, read the paper (or don’t)
- 86. Reading list