AMIA Joint Summits 2017 - Electronic phenotyping with APHRODITE and the Observational Health Sciences and Informatics (OHDSI) data network
Download as PPTX, PDF1 like648 views
The document presents the framework called Aphrodite for electronic phenotyping in the OHDSI data network, integrating statistical models with less development effort compared to traditional approaches. It combines two existing phenotyping methods to create a robust model for phenotype identification, demonstrating competitive performance using noisy labels. Aphrodite facilitates the sharing of phenotype models and configurations, promoting collaborative research across multiple sites.
![Anchors sugested for MI and T2DM
Myocardial Infarction (MI) Type 2 diabetes mellitus (T2DM)
Source
importance /
rank
concept_name Source
Importance /
rank
concept_name
obs 1.9036 1 Renal function lab 0.5637 1
Serum HDL/non-HDL cholesterol
ratio measurement
obs 1.8554 2 Every eight hours lab 0.5466 2 Glucose measurement
obs 1.7831 3 Chest CT obs 0.5328 3 Lipid panel
obs 1.7108 4 Cataract obs 0.5156 4 Asthma
obs 1.6386 5 Hypercholesterolemia obs 0.4984 5 Diabetes mellitus
obs 1.5663 6 Osteopenia lab 0.4778 6 Hyaline casts
obs 1.4699 7 Palpitations obs 0.4675 7 Pulmonary edema
obs 1.3735 8 Commode obs 0.4366 8 Obesity
obs 1.3012 9
Tightness sensation
quality
obs 0.4228 9 Hypoglycemia
obs 1.253 10 Afternoon obs 0.3987 10 Metformin
obs 1.012 11 Deficiency obs 0.3816 11 Duplex
drugEx 0.8916 12 ferrous sulfate obs 0.3712 12 Palpitations
drugEx 0.7711 13 Dobutamine visit 0.3541 13 Obesity
obs 0.7229 14 Fibrovascular lab 0.3334 14 Glucose
lab 0.5542 15
Sodium [Moles/volume] in
Blood
lab 0.3059 15 Hemoglobin A1c
drugEx 0.4819 16 clopidogrel obs 0.2784 16 Subclavicular approach
lab 0.3855 17 Lactic acid measurement obs 0.2303 17 Colonoscopy
obs 0.3133 18 Pressure ulcer obs 0.2131 18 Cholesterol
drugEx 0.1928 19 zolpidem drugEx 0.1925 19 Insulin, Regular, Human
obs 0.0964 20
Aspirin 81 MG Enteric
Coated Tablet
drugEx 0.0206 20 Oxycodone
22](https://image.slidesharecdn.com/amia-tbi-2017-170329015128/85/AMIA-Joint-Summits-2017-Electronic-phenotyping-with-APHRODITE-and-the-Observational-Health-Sciences-and-Informatics-OHDSI-data-network-23-320.jpg)
AMIA Joint Summits 2017 - Electronic phenotyping with APHRODITE and the Observational Health Sciences and Informatics (OHDSI) data network
- 1. Electronic phenotyping with APHRODITE and the Observational Health Sciences and Informatics (OHDSI) data network S16: Papers - EHR-based Phenotyping 2017 Joint Summits on Translational Science, March 28th, 2017 J U A N M . B A N D A , P H D 1 , Y O N I H A L P E R N , P H D 2 , D AV I D S O N T A G , P H D 3 , N I G A M H . S H A H , P H D 1 1 S T A N F O R D C E N T E R F O R B I O M E D I C A L I N F O R M A T I C S R E S E A R C H , S T A N F O R D , C A ; 2 N E W Y O R K U N I V . , N E W Y O R K , N Y ; 3 M I T , C A M B R I D G E , M A
- 2. DISCLOSURE: SPEAKER DISCLOSES THAT HE HAS NO RELATIONSHIPS WITH COMMERCIAL INTERESTS. 1 Electronic phenotyping with APHRODITE and the Observational Health Sciences and Informatics (OHDSI) data network
- 3. Motivation - Phenotyping is the beginning step to almost every biomedical study, yet in many cases is very complicated - Popular modern approaches are rule-based which take very long to develop - Can we build a statistical model which provides performance comparable to a consensus definition and requires less development effort? …. Turns out we can! (Agarwal V, et al. and Halpern Y, et al.) - Can we port these two approaches to the Observational Health Sciences and Informatics (OHDSI) tech stack? 2 Agarwal V, et al. Learning statistical models of phenotypes using noisy labeled training data. JAMIA. 2016. 23 (6), 1166-1173 Halpern Y, et al. Electronic medical record phenotyping using the anchor and learn framework. JAMIA 2016. 23 (4): 731-740
- 4. Automated PHenotype Routine for Observational Definition, Identification, Training and Evaluation (APHRODITE) - Framework is designed to enable phenotyping via supervised (or semi- supervised) learning of phenotype models - Reads patient data from the OHDSI/OMOP CDM version 5 - Combines two recently published phenotyping approaches: - XPRESS (Agarwal V, et al.) - Achor learning (Halpern Y, et al.) - In addition, to enable sharing and reproducibility of the underlying phenotype 'recipes', APHRODITE allows sharing of either the trained model or sharing of the configuration settings and anchor selections across multiple sites Agarwal V, et al. Learning statistical models of phenotypes using noisy labeled training data. JAMIA. 2016. 23 (6), 1166-1173 Halpern Y, et al. Electronic medical record phenotyping using the anchor and learn framework. JAMIA 2016. 23 (4): 731-740 3
- 5. 4 +140 collaborators 16 countries Common CDM and vocabulary The patient network available includes 84 databases, both clinical and claims, totaling over 650 million patients
- 6. OHDSI /OMOP CDMv5 APHRODITE Overview 5
- 7. Step 1 – Build Keyword List Function call: wordLists <- buildKeywordList(conn, aphrodite_concept_name, cdmSchema, dbms) Generated keywords for “Myocardial Infarction” Vocabulary v5 6
- 8. Step 2 – Find Cases and Controls CDMv5 Function call: casesANDcontrolspatient_ids_df<- getdPatientCohort(conn, dbms, as.character(keywordList_FF$V3),as.character(ignoreList_FF$V3), cdmSchema,nCases,nControls) 7
- 9. Step 3 – Extract Patient Data CDMv5 Function calls: dataFcases <- getPatientData(conn, dbms, cases, as.character(ignoreList_FF$V3), flag, cdmSchema) dataFcontrols <- getPatientData(conn, dbms, controls, as.character(ignoreList_FF$V3), flag, cdmSchema) 8
- 10. Step 4 – Create Feature Vector Function calls: fv_all<-buildFeatureVector(flag, dataFcases,dataFcontrols) fv_full_data <- combineFeatureVectors(flag, data.frame(cases), controls, fv_all, outcomeName) 9
- 11. Step 5 – Build Model Function calls: model_predictors <- buildModel(flag, fv_full_data, outcomeName, folder) model<-model_predictors$model predictorsNames<-model_predictors$predictorsNames 10
- 12. Step 6 – Get Evaluation Results 11
- 13. How do we know ‘noisy labeling’ works? Using the evaluation of Agarwal V., et al.: - Selected one phenotype each from defined by the Electronic Medical Records and Genomics (eMERGE) and the Observational Medical Outcomes Partnership (OMOP) initiatives: - Myocardial infarction - Type 2 diabetes mellitus - Created manually reviewed gold standard of cases and controls Agarwal V., et al. Learning statistical models of phenotypes using noisy labeled training data. JAMIA. 2016. 23 (6), 1166-1173 Cases Controls Accuracy Recall PPV Cases Controls Accuracy Recall PPV Source Myocardial Infarction (MI) Type 2 Diabetes Mellitus (T2DM) OMOP / PheKB Definition 94 94 0.87 0.91 0.84 152 152 0.92 0.88 0.96 XPRESS Noisy labels 94 94 0.85 0.93 0.8 152 152 0.89 0.99 0.81 APHRODITE Noisy labels 94 94 0.94 0.87 1.00 152 152 0.91 0.98 0.87 12
- 14. Building phenotype models with APHRODITE Using the following keywords Performance of noisly labeled trained models Performance of gold-standard test set Myocardial Infarction (MI) Type 2 diabetes mellitus (T2DM) Old myocardial infarction Type 2 diabetes mellitus with hyperosmolar coma True posterior myocardial infarction Type 2 diabetes mellitus Myocardial infarction with complication Pre-existing type 2 diabetes mellitus Myocardial infarction in recovery phase Type 2 diabetes mellitus with multiple complications Microinfarct of heart Type 2 diabetes mellitus in non-obese Cases Cont. Acc. Recall PPV Acc. Recall PPV Myocardial Infarction (MI) Type 2 Diabetes Mellitus (T2DM) XPRESS 750 750 0.86 0.89 0.84 0.88 0.89 0.87 APHRODITE 750 750 0.9 0.92 0.89 0.89 0.92 0.87 APHRODITE 1,500 1,500 0.9 0.93 0.9 0.91 0.93 0.88 APHRODITE 10,000 10,000 0.91 0.93 0.91 0.92 0.94 0.89 Cases Cont. Acc. Recall PPV Cases Cont. Acc. Recall PPV Source Myocardial Infarction (MI) Type 2 Diabetes Mellitus (T2DM) OMOP/PheKB 94 94 0.87 0.91 0.84 152 152 0.92 0.88 0.96 XPRESS 94 94 0.89 0.93 0.86 152 152 0.89 0.88 0.9 APHRODITE (750) 94 94 0.91 0.93 0.90 152 152 0.91 0.95 0.88 APHRODITE (1,500) 94 94 0.92 0.93 0.91 152 152 0.92 0.95 0.89 APHRODITE (10,000) 94 94 0.92 0.94 0.91 152 152 0.93 0.96 0.89 13
- 15. 15 Top 15 features for T2DM and MI Type 2 diabetes mellitus (T2DM) Myocardial Infarction (MI) Rank Feature Concept Name Concept Vocabulary Rank Feature Concept Name Concept Vocabulary 1 drugEx:1503297 Metformin RxNorm 1 obs:40398391 Hypertensive disease SNOMED 2 obs: 433736 Obesity SNOMED 2 obs:4243768 Auscultation SNOMED 3 obs:16890009 Insulin measurement SNOMED 3 obs:4342792 Infarction NDFRT 4 obs:434005 Morbid Obesity SNOMED 4 obs:4191705 Coronary heart disease review SNOMED 5 lab:4149519:NA Glucose measurement SNOMED 5 obs:4167217 Family history of clinical finding SNOMED 6 drugEx:253182 Regular Insulin, Human RxNorm 6 obs:4282140 Chewable tablet SNOMED 7 drugEx:1112807 Aspirin RxNorm 7 drugEx:1112807 Aspirin RxNorm 8 lab: 4144235:NA Glucose measurement, blood SNOMED 8 obs:4325480 Aspirin NDFRT 9 obs:45889912 Hemoglobin SNOMED 9 lab:3002030 Lymphocytes % LOINC 10 obs:45889669 Insulin CPT4 10 lab:3014576 Chloride serum/plasma LOINC 11 lab:3004410 Hemoglobin A1c (Glycated) LOINC 11 obs:4329041 Pain SNOMED 12 obs: 40398391 Hypertensive disease SNOMED 12 lab:40789893 Potassium | Bld-Ser-Plas LOINC 13 obs:1503297 Metformin RxNorm 13 obs:433595 Edema SNOMED 14 obs:4325480 Aspirin NDFRT 14 visit:77670 Chest Pain SNOMED 15 obs:40301556 Past medical history SNOMED 15 obs:1126658 Hydromorphone RxNorm 14
- 16. Using Anchors with APHRODITE: Step 1 – Build Keyword List - Anchors Function call: wordLists <- buildKeywordList(conn, aphrodite_concept_name, cdmSchema, dbms) Generated keywords for “Myocardial Infarction” 15
- 17. Step 2 – Find Anchors CDM v5 Function call: anchor_list <- getAnchors(conn, dbms, cdmSchema, cases, controls, as.character(ignoreList_FF$V3), studyName, outcomeName, flag, numAnchors=50) 16
- 18. Step 3 – Curate Anchors Manual Step 17
- 19. Step 4 – Find Cases and Controls - Anchors CDMv5 Function call: Anchored_casesANDcontrolspatient_ids_df<- getPatientCohort_w_Anchors(conn, dbms, as.character(keywordList_FF$V3),as.character(ignoreList_FF$V3), cdmSchema,nCases,nControls, flag) 18
- 20. Step 5 – Extract Patient Data - Anchors CDMv5 Function calls: dataFcases <- getPatientData(conn, dbms, cases, as.character(ignoreList_FF$V3), flag, cdmSchema) dataFcontrols <- getPatientData(conn, dbms, controls, as.character(ignoreList_FF$V3), flag, cdmSchema) 19
- 21. Step 6 – Create Feature Vector - Anchors Function calls: fv_all<-buildFeatureVector(flag, dataFcases,dataFcontrols) fv_full_data <- combineFeatureVectors(flag, data.frame(cases), controls, fv_all, outcomeName) 20
- 22. Step 7 – Build Model - Anchors Function calls: model_predictors <- buildModel(flag, fv_full_data, outcomeName, folder) model<-model_predictors$model predictorsNames<-model_predictors$predictorsNames 21
- 23. Anchors sugested for MI and T2DM Myocardial Infarction (MI) Type 2 diabetes mellitus (T2DM) Source importance / rank concept_name Source Importance / rank concept_name obs 1.9036 1 Renal function lab 0.5637 1 Serum HDL/non-HDL cholesterol ratio measurement obs 1.8554 2 Every eight hours lab 0.5466 2 Glucose measurement obs 1.7831 3 Chest CT obs 0.5328 3 Lipid panel obs 1.7108 4 Cataract obs 0.5156 4 Asthma obs 1.6386 5 Hypercholesterolemia obs 0.4984 5 Diabetes mellitus obs 1.5663 6 Osteopenia lab 0.4778 6 Hyaline casts obs 1.4699 7 Palpitations obs 0.4675 7 Pulmonary edema obs 1.3735 8 Commode obs 0.4366 8 Obesity obs 1.3012 9 Tightness sensation quality obs 0.4228 9 Hypoglycemia obs 1.253 10 Afternoon obs 0.3987 10 Metformin obs 1.012 11 Deficiency obs 0.3816 11 Duplex drugEx 0.8916 12 ferrous sulfate obs 0.3712 12 Palpitations drugEx 0.7711 13 Dobutamine visit 0.3541 13 Obesity obs 0.7229 14 Fibrovascular lab 0.3334 14 Glucose lab 0.5542 15 Sodium [Moles/volume] in Blood lab 0.3059 15 Hemoglobin A1c drugEx 0.4819 16 clopidogrel obs 0.2784 16 Subclavicular approach lab 0.3855 17 Lactic acid measurement obs 0.2303 17 Colonoscopy obs 0.3133 18 Pressure ulcer obs 0.2131 18 Cholesterol drugEx 0.1928 19 zolpidem drugEx 0.1925 19 Insulin, Regular, Human obs 0.0964 20 Aspirin 81 MG Enteric Coated Tablet drugEx 0.0206 20 Oxycodone 22
- 24. APHRODITE anchored phenotype performance Cases Cont. Acc. Recall PPV Cases Cont. Acc. Recall PPV Myocardial Infarction (MI) Type 2 Diabetes Mellitus (T2DM) OMOP/PheKB 94 94 0.87 0.91 0.84 152 152 0.92 0.88 0.96 XPRESS 94 94 0.89 0.93 0.86 152 152 0.89 0.99 0.9 APHRODITE 94 94 0.91 0.93 0.9 152 152 0.91 0.98 0.88 APHRODITE (Anchors) 94 94 0.92 0.97 0.89 152 152 0.92 0.95 0.9 23
- 25. Conclusions We have successfully implemented the framework proposed by Agarwal et al. and the Anchor learning framework by Halpern et al. to build and refine phenotype models We have demonstrated that it is possible identify anchors during the model building process to generate a better labeled training set which leads to a better performing model than just using keyword for noisy labeling Our main contribution is the APHRODITE package, which allows for the potential redistribution of locally validated phenotype models as well as the sharing of the workflows for learning phenotype models at multiple sites of the OHDSI data network Agarwal V, et al. Learning statistical models of phenotypes using noisy labeled training data. JAMIA. 2016. 23 (6), 1166-1173 Halpern Y, et al. Electronic medical record phenotyping using the anchor and learn framework. JAMIA 2016. 23 (4): 731-740 24
- 26. Acknowledgements Stanford - Nigam Shah - Vibhu Agarwal - Shah lab New York University - Yoni Halpern MIT - David Sontag OHDSI Community 25
- 28. 28 Why do noisy labels work Learning theory - Generalization error of a model trained using data with inaccurate labels can be made small Model trained using m observations with accurate labels has the same gen error as a model trained using m/(1 – 2τ) 2 observations with noisy labels. Model inaccuracies in automatic labeling as random classification noise given by error rate τ Unless τ is 0.5, having more observations will help The cost of getting additional observations is negligible
- 29. 29 Performance results for data removal experiments Cases Cont. Acc. Recall PPV Cases Cont. Acc. Recall PPV Source Myocardial Infarction (MI) Type 2 Diabetes Mellitus (T2DM) APHRODITE 94 94 0.91 0.93 0.9 152 152 0.91 0.98 0.88 Observations removed 94 94 0.75 0.84 0.78 152 152 0.67 0.76 0.71 Labs removed 94 94 0.87 0.85 0.82 152 152 0.69 0.78 0.72 Drugs removed 94 94 0.85 0.85 0.82 152 152 0.83 0.9 0.84 Visits removed 94 94 0.89 0.88 0.86 152 152 0.86 0.91 0.86 APHRODITE w Anchors 94 94 0.92 0.97 0.89 152 152 0.92 0.95 0.9 Observations removed 94 94 0.77 0.89 0.79 152 152 0.7 0.77 0.73 Labs removed 94 94 0.89 0.88 0.81 152 152 0.71 0.79 0.75 Drugs removed 94 94 0.86 0.87 0.84 152 152 0.86 0.91 0.85 Visits removed 94 94 0.91 0.9 0.87 152 152 0.88 0.9 0.84
Editor's Notes
- #5: Less cumbersome: (not requiring long lists of rules) Nearly unsupervised (minimial user input).
- #6: - Multi-stakeholder, interdisciplinary collaborative to bring out the value of health data through large-scale analytics - All our solutions are open-source
- #14: Manual review: Each record was voted a case (or control) if two clinicians agreed, and a third clinician approved We ensure that the potential cases and controls found in the noisy-labeled sets are completely disjoint from this manually reviewed evaluation set.
- #15: Potential question: You will get a question here about why is the case:control 50:50 and not what you would have in real life. (e.g. ~14% cases for diabetes)
- #27: 140 collaborators in 16 countries
- #29: Not surprising. Learning theory - Generalization error of a model trained using data with inaccurate labels can be made small Model trained using m observations with accurate labels has the same gen error as a model trained using m/(1 – 2τ) 2 observations with noisy labels. Model inaccuracies in automatic labeling as random classification noise given by error rate τ Unless τ is 0.5, having more observations will help The cost of getting additional observations is negligible
- #30: With over 20,000 features (on average) the regular APHRODITE models with and without anchors perform the best, but with very close performance to the models that exclude the visits data. For the phenotypes we present, this indicates that most of the coded data is not particularly useful in making the phenotype assignments. It is also quite evident that removing the text features (observations) results in nearly a 15% drop in accuracy demonstrating that access to the unstructured portions of the medical record is crucial for the success of training phenotype models with imperfectly labeled training data.