Applying NLP to Personalized Healthcare - 2021
- 1. Dr. David Talby, CTO, John Snow Labs Real-World Lessons from Applying Natural Language Processing to Personalized Healthcare
- 2. 2 Agenda Personalized Medicine has a big NLP challenge State-of-the-art accuracy has recently improved Real-world solutions require hyper-specialization 1. 2. 3.
- 4. 4 John Snow Labs: The Team Behind Spark NLP Most popular O’Reilly Media 54% share of healthcare AI teams use Spark NLP Gradient Flow 9x growth In downloads of the library during 2020 PyPI Download Stats NLP library in the enterprise
- 5. 5 Today’s Medicine Is Imprecise 80% non-responders for the top 20 drugs So, how shall we fix it?
- 6. 6 Case Study: Recommending The Next Best Action • A clinical guideline is a document that guides decisions and criteria regarding diagnosis, management, and treatment. • modern medical guidelines are based on an examination of current evidence within the paradigm of evidence-based medicine. • They usually summarize consensus statements on best practice. • A doctor is obliged to know the medical guidelines of their profession and must decide whether to follow them for an individual patient.
- 7. 7 Where Is The Clinically Relevant Oncology Data? A detailed study found that only 40% of about 300 data points required for clinical decision support were available in structured data “
- 8. 8 ONCOLOGY: Information Extraction Examples Token (Word) Entity Category (Labels) Upper outer quadrant Tumor Site (Localization) pT1 Primary Tumor (pT1) pN1 Regional Lymph Nodes (pN1) pM1 Distant Metastasis (pM1) Left Specimen Laterality (Laterality) 4.5 x 2.0 x 2.0 cm Size of Invasive Carcinoma (Size) Medullary Carcinoma Histologic Type (Type) Poorly Differentiated Histologic Grade (Grade)
- 9. 9 Case Study #2: Matching Patients To Clinical Trials 5,446 cutting-edge treatments are behind the gates of clinical trials 1,870 Cancer drugs 139 Alzheimer’s drugs 94 Asthma drugs 71 COPD drugs 162 138 91 68 Pain drugs Rheumatoid arthritis drugs Parkinson’s drugs HIV/AIDS drugs 154 Type 2 diabetes drugs 105 2,554 Other Psoriasis drugs SOURCE: Estimated based on Pharmaprojects, “Pharma R&D Annual Review 2017”
- 10. 10 Not Enough People Participate In Clinical Trials The consequences, for one blockbuster drug, due to the average delay in clinical trial recruiting: 993 human lives lost $2.9B revenue lost 5x drug price increase SOURCE: Cutting Edge Information, FiercePharma, Life Extension Magazine https://medrio.com/blog/overcoming-patient-recruitment-and-retention-hurdles/
- 11. 11 The NLP Challenge: Complex Enrollment Criteria Increasingly complex enrollment criteria made off-the-shelf models that extract findings, treatments, drugs, labs, etc. largely useless. “
- 12. 12 Case Study #3: Populating Patient Registries • A patient registry is a database of facts about patients who are affected by a particular condition. • In rare diseases, they play an important role in the therapy development pathway. • They can identify candidates for clinical trials, help develop standards of care, and accelerate medical research. • The average U.S. healthcare facility participates in 5 to 10 registries, sometimes due to regulatory requirements. • Registries are usually filled in manually, implying significant cost, time, and effort. https://https://treat-nmd.org/patient-registries/
- 13. 13 This Is Largely An NLP Challenge, Too PLAN: Follow up with dietitian to manage hypertension & borderline LDL and increase dosage of lisinopril since BP was 152/85 today “ (for Patient K) Source: https://www.ahrq.gov/ncepcr/tools/pf-handbook/mod20-appendix-b.html
- 14. 14 • It takes weeks to manually update a registry (how fast can we get real-world data on COVID-19 vaccines and long-term impact?) • It delays clinical trials, by adding major projects to each trial • Trained nurses who do this work aren’t treating patients • It’s expensive, making exploratory abstraction Why this NLP Challenge is Important
- 15. 15 Agenda Personalized Medicine has a big NLP challenge State-of-the-art accuracy has recently improved Real-world solutions require hyper-specialization 1. 2. 3.
- 16. 16 “State of the Art Natural Language Processing” What does that mean? Academic SOTA: • Best accuracy on public benchmarks • Published as peer-reviewed papers Applied SOTA adds: • In production in multiple organizations • Open source or open core
- 17. 17 Clinical Named Entity Recognition
- 18. 18 1. State of the Art Biomedical NER Spark NLP for Healthcare holds the top spot on 8 of 11 benchmarks in the ‘Medical Named Entity Recognition’ category on ‘Papers With Code’ Disease Chemicals Species Cancer Genetics Anatomy Species Cellular Chemical, Disease
- 19. 19 2. Real-World Use Interpreting millions of patient stories with deep learned OCR and NLP Stacy Ashworth Executive Vice President of Clinical Innovation Spark NLP in action: Improving patient flow forecasting at Kaiser Permanente Santosh Kulkarni Director of Product Mgmt., Kaiser Permanente A Unified CV, OCR, and NLP for Scalable Document Understanding Text Analytics and its Applications in the Pharma Industry Harsha Gurulingappa, Ph.D. Text Analytics Product Owner at Merck NLP in Oncology Real World Data: Opportunities to develop a true learning healthcare system Patrick Beukema, Ph.D. Senior ML Engineer, DocuSign Automated & Explainable Deep Learning for Clinical Language Understanding at Roche Vishakha Sharma, Ph.D. Principal Data Scientist, Roche George A. Komatsoulis, Ph.D. Chief of Bioinformatics at CancerLinQ
- 20. 20 3. Optimized for Modern Platforms • Optimized builds of Spark NLP for both Intel and Nvidia were tested on AWS • Task: Training a Named Entity Recognizer in French • Achieving F1-score of 89% requires at least 80 Epochs with batch size of 512 • CPU can train even faster by using batch size of 1024, since they can leverage all system memory beyond just 12GB Intel-Optimized Spark NLP on Cascade Lake is 19% faster and 46% cheaper than GPU-optimized Spark NLP on Tesla v100
- 21. 21 Clinical Assertion Status Detection • Recognized entities are useless, in most real cases, without also knowing their assertion status. • This is a separate NLP task, with its own algorithms and deep learning models to address. • Assertion status models are domain-specific and must sometimes be trained for your use case: o Orthopedics: Is a condition chronic? o Surgery: Is this a past event, a plan, or proposed treatment? o Insurance: Is this a statement of evidence, question about a fact, or an answer? Entities For each recognized entity: Is this assertion positive, negative, suspected, conditional, or about someone else?
- 22. 22 Current SOTA Metrics Source: https://arxiv.org/abs/2012.04005 Improving Clinical Document Understanding on COVID-19 Research with Spark NLP, Kocaman and Talby, SDU workshop at AAAI 2021 Peer-Reviewed Accuracy: Assertion detection model test metrics. Our implementation exceeds the benchmarks in the latest best model in 4 out of 6 assertion labels – and in overall accuracy. Assertion Label Spark NLP Latest Best Absent 0.944 0.937 Someone-else 0.904 0.869 Conditional 0.441 0.422 Hypothetical 0.862 0.890 Possible 0.680 0.630 Present 0.953 0.957 micro F1 0.939 0.934
- 23. 23 Relation Extraction Identify and name relations between named entities. Source: https://demo.johnsnowlabs.com/healthcare/RE_CLINICAL_EVENTS/
- 24. 24 Current SOTA Metrics Sourcehttps://nlp.johnsnowlabs.com/docs/en/licensed_release_notes – Spark NLP for Healthcare 2.7.3 Release Notes Model Spark NLP Current Best Temporal relations between clinical events 71.0 80.2 1 Relations between symptoms, procedures, and treatments 69.2 68.2 2 Genes and human phenotypes 87.9 67.2 3 Drug-drug interactions 72.1 83.8 4 Chemicals and proteins 94.1 83.64 5
- 25. 25 Text De-Identification Patient AIQING, 25 month years-old , born in Beijing, was transferred to the Johns Hopkins Hospital. Original Text: Patient BLONDIE, 35 years-old , born in Talihina, was transferred to the Mid-Columbia Medical Center. Obfuscated: Patient <NAME>, <AGE> month years-old , born in <LOCATION>, was transferred to the <LOCATION>. Tagged:
- 26. 26 Real-World De-Identification Far more complex that obfuscating text: • Compliance requirements differ by country & usage • Technical, data, and personnel controls must be placed • Structured data, unstructured text, image, audio, video, and sensor data should be jointly de-identified • Text de-identification is also a computer vision challenge when PDF, DICOM, and image files are involved SOTA metrics: • Similar accuracy to what human domain experts achieve Source: https://demo.johnsnowlabs.com/ocr/DEID_PDF_HIPAA/ Benchmark Spark NLP Current Best 2014 n2c2 0.961 0.955
- 27. 27 Agenda Personalized Medicine has a big NLP challenge State-of-the-art accuracy has recently improved Real-world solutions require hyper-specialization 1. 2. 3.
- 28. 28 Healthcare has hundreds of languages
- 29. 29 Clinical NER: 100+ Entities and Going Posology NER Anatomy NER PHI NER
- 30. 30 30+ Medical Relations & Counting • Overusing relation extraction will create many relations, not all relevant to your use case • 90% accuracy means that 2-4 relation extraction models will show 1-4 errors in every paragraph. • Your UI must assume some incorrect results and help the user choose between them • Was a drug given in order to treat a symptom? • Or was the symptom a result of taking the drug? • Or was the drug avoided because of the symptom? • Maybe the drug caused the symptom to worsen? Source: https://nlp.johnsnowlabs.com/demo
- 31. 31 Entity Normalization Normalization can be about mapping to an ontology, spell checking, changing units, … adalimumab 54.5 + 43.2 gm Agnogenic one half cup interferon alfa-2b 10 million unit ( 1 ml ) injec adalimumab 97700 mg Agnogenic 0.5 oral solution interferon alfa - 2b 10000000 unt ( 1 ml ) injection Source: https://nlp.johnsnowlabs.com/docs/en/licensed_release_notes/
- 32. 32 De-identification: Use document & specialty-specific models Source Output Comments Reid A. Smith, M.D. <PERSON> A. <PERSON>, M.D. Doctor names don’t have to be de-identified per HIPAA. KIDNEY STONE SURGERY KIDNEY <PERSON> SURGERY Polysemic terms (sometimes a person’s name, but not always) Patient with Parkinson. Patient with <PERSON>. Deseases that are surnames Fergus Falls. <PERSON> Falls. City “Fergus Falls” in Minnesota, versus someone named Fergus who is falling.
- 33. 33 Role in Organization Technical Leaders All Respondents Source: gradientflow.com Considerations for a healthcare NLP solution Trainable Clinically Validated Private
- 34. 34 Summary Personalized Medicine has a big NLP challenge State-of-the-art accuracy has recently improved Real-world solutions require hyper-specialization 1. 2. 3.
- 35. 35 Thank you! © 2015-2021 John Snow Labs Inc. All rights reserved. The John Snow Labs logo is a trademarks of John Snow Labs Inc. The included information is for informational purposes only and represents the current view of John Snow Labs as of the date of this presentation. Since John Snow Labs must respond to changing market conditions, it should not be interpreted to be a commitment on its part, and John Snow Labs cannot guarantee the accuracy of any information provided after the date of this presentation. John Snow Labs makes no warranties, express or statutory, as to the information in this presentation. www.johnsnowlabs.com/spark-nlp-in-action/ www.johnsnowlabs.com/spark-nlp-try-free/ Live demos: Free trial: