Digital Pathology: Precision Medicine, Deep Learning and Computer Aided Interpretation
5 likes2,462 views
The document discusses advancements in digital pathology, emphasizing its role in precision medicine through deep learning and computer-aided interpretation. It details methods for analyzing and classifying pathological images to improve reproducibility and accuracy in traditional assessments, including applications in cancer diagnosis and treatment. Additionally, it outlines collaborative efforts in research and the importance of integrating image and molecular data to enhance cancer treatment outcomes.
Digital Pathology: Precision Medicine, Deep Learning and Computer Aided Interpretation
- 1. Digital Pathology: Precision Medicine, Deep Learning and Computer Aided Interpretation Joel Saltz MD, PhD Chair and Professor Department of Biomedical Informatics Professor Department of Pathology Cherith Endowed Chair Stony Brook University UPMC: Biomedical Informatics Lecture Series and Pittsburgh Computational Pathology Lecture Series
- 2. NO CONFLICTS TO DISCLOSE
- 5. The Virtual Microscope: Johns Hopkins 1997
- 6. Johns Hopkins School of Medicine Virtual Microscope
- 7. From 1998 Johns Hopkins Grand Rounds
- 8. The Virtual Microscope arose from University of Maryland College Park Active Data Repository Project
- 9. and National Science Foundation Grand Challenge Grant
- 10. Seven Years before Google EarthS
- 11. 1999 DARPA – HUBS Project Joel Saltz, Mike Becich and David Foran • Exploration of full slide digitized datasets • Classification of hematologic malignancies • Content Based Image Retrieval • Remotely steered microscope
- 12. Classification of Hematologic Malignancies – David Foran -- UMDNJ
- 13. Pathology Image Driven Decision Support • Improve reproducibility in traditional Pathology assessments (e.g. Gleason grade, NSCLC subtypes) • Precise scoring of well known criteria ( tumor infiltrating lymphocytes, mitoses and IHC staining) • Development of novel computational methods to employ Pathology image information to predict response to cancer treatment and outcomes.
- 14. Early Steps to Pathology Computer Aided Classification 2005-2010 Gurcan, Shamada, Kong, Saltz Hiro Shimada, Metin Gurcan, Jun Kong, Lee Cooper Joel Saltz BISTI/NIBIB Center for Grid Enabled Image Analysis - P20 EB000591, PI Saltz
- 15. Neuroblastoma Classification FH: favorable histology UH: unfavorable histology CANCER 2003; 98:2274-81 <5 yr Schwannian Development ≥50% Grossly visible Nodule(s) absent present Microscopic Neuroblastic foci absent present Ganglioneuroma (Schwannian stroma-dominant) Maturing subtype Mature subtype Ganglioneuroblastoma, Intermixed (Schwannian stroma-rich) FH FH Ganglioneuroblastoma, Nodular (composite, Schwannian stroma-rich/ stroma-dominant and stroma-poor) UH/FH* Variant forms* None to <50% Neuroblastoma (Schwannian stroma-poor) Poorly differentiated subtype Undifferentiated subtype Differentiating subtype Any age UH ≥200/5,000 cells Mitotic & karyorrhectic cells 100-200/5,000 cells <100/5,000 cells Any age ≥1.5 yr <1.5 yr UH UH FH ≥200/5,000 cells 100-200/5,000 cells <100/5,000 cells Any age UH ≥1.5 yr <1.5 yr ≥5 yr UH FH UH FH
- 16. Multi-Scale Machine Learning Based Shimada Classification System • Background Identification • Image Decomposition (Multi- resolution levels) • Image Segmentation (EMLDA) • Feature Construction (2nd order statistics, Tonal Features) • Feature Extraction (LDA) + Classification (Bayesian) • Multi-resolution Layer Controller (Confidence Region) No Yes Image Tile Initialization I = L Background? Label Create Image I(L) Segmentation Feature Construction Feature Extraction Classification Segmentation Feature Construction Feature Extraction Classifier Training Down-sampling Training Tiles Within Confidence Region ? I = I -1 I > 1? Yes Yes No No TRAINING TESTING
- 21. Digital Pathology as Precision Medicine • Statistical analyses and machine learning to link Radiology/Pathology features to “omics” and outcome biological phenomena • Image analysis and deep learning methods to extract features from images • Support queries against ensembles of features extracted from multiple datasets • Identify and segment trillions of objects – nuclei, glands, ducts, nodules, tumor niches • Analysis of integrated spatially mapped structural/”omic” information to gain insight into cancer mechanism and to choose best intervention
- 22. Quantitative Feature Analysis in Pathology: Emory In Silico Center for Brain Tumor Research (PI = Dan Brat, PD= Joel Saltz) 2009 - 2013
- 23. Using TCGA Data to Study Glioblastoma Diagnostic Improvement Molecular Classification Predictors of Progression
- 25. Oligodendroglioma Astrocytoma Nuclear Qualities Can we use image analysis of TCGA GBMs TO INFORM diagnostic criteria based on molecular or clinical endpoints? Application: Oligodendroglioma Component in GBM
- 27. Integrative Morphology/”omics” Quantitative Feature Analysis in Pathology: Emory In Silico Center for Brain Tumor Research (PI = Dan Brat, PD= Joel Saltz) NLM/NCI: Integrative Analysis/Digital Pathology R01LM011119, R01LM009239 (Dual PIs Joel Saltz, David Foran) Marcus Foundation Grant – Ari Kaufman, Joel Saltz
- 28. Associations
- 29. • Stony Brook, Institute for Systems Biology, MD Anderson, Emory group • TCGA Pan Cancer Immune Group – led by ISB researchers • Deep dive into linked molecular and image based characterization of cancer related immune response http://www.cell.com/cell-reports/pdf/S2211-1247(18)30447-9.pdf
- 31. ● Deep learning based computational stain for staining tumor infiltrating lymphocytes (TILs) ●TIL patterns generated from 4,759 TCGA subjects (5,202 H&E slides), 13 cancer types ●Computationally stained TILs correlate with pathologist eye and molecular estimates ●TIL patterns linked to tumor and immune molecular features, cancer type, and outcome
- 32. Le Hou – Graduate Student Computer Science Vu Nguyen– Graduate Student Computer Science Anne Zhao – Pathology Informatics Biomedical Informatics, Pathology (now Surg Path Fellow SBM) Raj Gupta – Pathology Informatics Biomedical Informatics, Pathology Deep Learning and Lymphocytes: Stony Brook Digital Pathology Trainee Team The future of Digital Pathology
- 33. Importance of Immune System in Cancer Treatment and Prognosis • Tumor spatial context and cellular heterogeneity are important in cancer prognosis • Spatial TIL densities in different tumor regions have been shown to have high prognostic value – they may be superior to the standard TNM classification • Immune related assays used to determine Checkpoint Inhibitor immune therapy in several cancer types • Strong relationships with molecular measures of tumor immune response – results to soon appear in TCGA Pan Cancer Immune group publications • TIL maps being computed for SEER Pathology studies and will be routinely computed for data contributed to TCIA archive • Ongoing study to relate TIL patterns with immune gene expression groups and patient response
- 34. Training, Model Creation • Algorithm first trained on image patches • Several cooperating deep learning algorithms generate heat maps • Heat maps used to generate new predictions • Companion molecular statistical data analysis pipelines
- 35. Training, threshold adjustment, quality control
- 36. Tools: Quantitative Imaging Pathology - QuIP Tool Set
- 37. Interactive Deep Learning Training Tool
- 40. Validation – Stratified sampling from 5K whole slide images Arvind Rao, expert in spatial biostatistics (U Michigan)
- 41. Quantitative Assessment of TIL Fractions
- 42. Characterization of TIL Pattern and Relationship to Molecular Immune Subtype • Pattern of immune infiltrate • Division of immune infiltrate between different compartments • Surround tumor region? Present in tumor, invasive margin? • What type of cells should be included? • Different criteria for different tumors with efforts to standardize • A Practical Review for Pathologists and Proposal for a Standardized Method From the International Immunooncology Biomarkers Working Group – part 1 and 2 - Adv Anat Pathol Volume 24, Number 5, September 2017 (figure to right from that reference)
- 47. TIL Pattern Descriptions Qualitative (Alex Lazar, Raj Gupta) • ‘‘Brisk, diffuse’’ diffusely infiltrative TILs scattered throughout at least 30% of the area of the tumor (1,856 cases); • ‘‘Brisk, band-like’’ - band-like boundaries bordering the tumor at its periphery (1,185); • ‘‘Nonbrisk, multi-focal’’ loosely scattered TILs present in less • than 30% but more than 5% of the area of the tumor (1,083); • ‘‘Non-brisk, focal’’ for TILs scattered throughout less than 5% but greater than 1% of the area of the tumor (874); • ‘‘None’’ < 1% TILS - in 143 cases Quantitative – Arvind Rao • Agglomerative clustering • Cluster indices representing cluster number, density, cluster size, distance between clusters • Traditional spatial statistics measures • R package clusterCrit by Bernard Desgraupes - Ball- Hall, Banfield-Raftery, C Index, and Determinant Ratio indices
- 49. TCGA Pan Cancer Atlas – Immune Landscape of Cancer • Six identified immune subtypes span cancer tissue types and molecular subtypes • Immune subtypes differ by somatic aberrations, microenvironment, and survival • Multiple control modalities of molecular networks affect tumor- immune interactions • These analyses serve as a resource for exploring immunogenicity across cancer types http://www.cell.com/immunity/fullt ext/S1074-7613(18)30121-3
- 50. Spatial Patterns vs TCGA Tumor, Molecular Subtypes
- 51. Tools to Analyze Morphology and Spatially Mapped Molecular Data - U24 CA180924 • Specific Aim 1 Analysis pipelines for multi- scale, integrative image analysis. • Specific Aim 2: Database infrastructure to manage and query Pathomics features. • Specific Aim 3: HPC software that targets clusters, cloud computing, and leadership scale systems. • Specific Aim 4: Develop visualization middleware to relate Pathomics feature and image data and to integrate Pathomics image and “omic” data.
- 52. Methods and tools for integrating pathomics data into cancer registries Saltz, Sharma, Foran and Durban • Enhance SEER registry data with machine learning based classifications and quantitative pathomics feature sets. • The New Jersey State Cancer Registry, Georgia and Kentucky State Cancer Registries • Prostate Cancer, Lymphoma and NSCLC • Repository of high-quality digitized pathology images for subjects whose data is being collected by the registries. • Extract computational features and establish deep linkages with registry data, thus enabling the creation of information-rich, population cohorts containing objective imaging and clinical attributes
- 53. QuIP Segmentation Curation Web Application 1. User interactively marks up regions and selects results from best analysis run for each region. 2. Selections are refined through review processes supervised by an expert Pathologist Curation of Segmentation Results Pathology Features at Scale
- 54. TCIA encourages and supports the cancer imaging open science community by hosting and managing Findable Accessible, Interoperable, and Reusable (FAIR) images and related data. http://www.cancerimagingarchive.net/ Clark, et al. J Digital Imag 26.6 (2013): 1045-1057. Cancer Imaging Archive – Integration of Pathology and Radiology for Community Clinical Studies
- 55. TCIA sustainment and scalability Platforms for quantitative imaging informatics in precision medicine Prior, Saltz, Sharma -- U24CA215109-01 • Identify quantitative imaging phenotypes across scale through the use of Radiomic/Pathomic analyses • Well-curated data for algorithm testing and validation. • Integrative Radiology/Pathology Image-Omics studies • Extend TCIA to support its rapidly growing user community and continue to promote research reproducibility and data reuse in cancer precision medical research.
- 56. Thanks!
- 57. ITCR Team Stony Brook University Joel Saltz Tahsin Kurc Yi Gao Allen Tannenbaum Erich Bremer Jonas Almeida Alina Jasniewski Fusheng Wang Tammy DiPrima Andrew White Le Hou Furqan Baig Mary Saltz Raj Gupta Emory University Ashish Sharma Adam Marcus Oak Ridge National Laboratory Scott Klasky Dave Pugmire Jeremy Logan Yale University Michael Krauthammer Harvard University Rick Cummings
- 58. Funding – Thanks! • This work was supported in part by U24CA180924, U24CA215109, NCIP/Leidos 14X138 and HHSN261200800001E, UG3CA225021-01 from the NCI; R01LM011119-01 and R01LM009239 from the NLM • This research used resources provided by the National Science Foundation XSEDE Science Gateways program under grant TG-ASC130023 and the Keeneland Computing Facility at the Georgia Institute of Technology, which is supported by the NSF under Contract OCI-0910735.