Learning, Training, Classification, Common Sense and Exascale Computing
Download as PPTX, PDF1 like280 views
The document discusses the integration of deep learning and sensor data analysis in scientific computing and its implications for various fields, particularly in classification and object detection. It highlights advancements in training methodologies for deep learning models using large datasets and specific examples in pathology and cancer research. Furthermore, it emphasizes future efforts in optimizing systems software performance and expanding whole slide imaging analytics for enhanced data analysis.
Learning, Training, Classification, Common Sense and Exascale Computing
- 1. Learning, Training, Classification, Common Sense and Exascale Computing Joel Saltz MD, PhD Chair and Professor Department of Biomedical Informatics Professor Department of Computer Science Cherith Endowed Chair Stony Brook University Oak Ridge National Laboratory, December 3, 2018
- 2. Overview • Sensor data analysis is becoming an integral component of scientific computing workflows • Results from numerical simulations are often also spatio-temporal data • In virtually all scientific fields, deep learning methods are being adopted to make sense of sensor and scientific data • Extremely broad implications – making sense of data via deep learning generally involves a type of semantically based data compression • Making sense of data also allows intelligent methods for steering data acquisition and computation • These issues were debated and discussed in great detail at the recent DOE SSIO workshop • Hot off the press – Nov 29-30th BDEC workshop – same basic conclusions
- 4. Manish Parashar, Director of Advanced Cyberinfrastru cture, NSF BDEC 2 Kickoff – Bloomington Indiana
- 5. Dan Reed, Provost University of Utah BDEC 2 Kickoff – Bloomington Indiana
- 6. Scene Understanding ●Scene understanding from coarse-grained to fine-grained inference ●Classify scene as a whole ●Identify objects ●Specify where objects are located ●Identify each object category’s pixels or voxels ●Identify pixels and voxels associated with each instance of each object Image from Garcia et al arXiv 1704.06857
- 7. Scenes from Science Applications • Size: Gigabyte to Terabyte per image; Petabyte to Exabyte per collection • Number of objects: can be billions or more, can be very few – needle in hay stack • Categories of object: Many, many ways of classifying natural objects so category definition depends on what question is being asked • Object boundaries: Depends on exactly how an object is defined. What is the boundary of a hurricane, tumor or city? • Sources: Sensors and cameras, scientific simulations
- 10. Detection of oil leaks and spills (Serge Petiton from U. de Lille and TOTAL)
- 11. Wind Farm Simulation – Dimitri Mavriplis
- 12. Neural Networks and Deep Learning
- 13. Training ●Classify scene as a whole ●Identify objects ●Specify where objects are located ●Identify each object category’s pixels or voxels ●Identify pixels and voxels associated with each instance of each object Image from Garcia et al arXiv 1704.06857 Increasing Training Effort
- 14. Deep Learning, Sensor/Scientific Data, Architectures and Training • Extremely large datasets • Humans, animals and human built objects (tables, chairs, houses etc) are different from objects from the natural world and can be harder to identify, classify and segment • Coupled deep learning architectures and training strategies are crucial for success • In this talk, I will present several examples of work we’ve done and results obtained
- 15. Prehistory – first decade of 2000’s • Reverse engineer classification – break scene (whole slide image) into patches • Create machine learning model and generate training data by examining characteristics of each patch, understanding classification system and classifying each training patch • Close collaboration with applications person (Hiro Shimada) • Patch level predictions aggregated to generate predictions for each scene https://www.sciencedirect.com/science/article/abs/pii/S0031320 308003439
- 16. Early Steps to Pathology Computer Aided Classification 2005-2010 Gurcan, Shamada, Kong, Saltz Hiro Shimada, Metin Gurcan, Jun Kong, Lee Cooper Joel Saltz BISTI/NIBIB Center for Grid Enabled Image Analysis - P20 EB000591, PI Saltz
- 17. Neuroblastoma Classification FH: favorable histology UH: unfavorable histology CANCER 2003; 98:2274-81 <5 yr Schwannian Development ≥50% Grossly visible Nodule(s) absent present Microscopic Neuroblastic foci absent present Ganglioneuroma (Schwannian stroma-dominant) Maturing subtype Mature subtype Ganglioneuroblastoma, Intermixed (Schwannian stroma-rich) FH FH Ganglioneuroblastoma, Nodular (composite, Schwannian stroma-rich/ stroma-dominant and stroma-poor) UH/FH* Variant forms* None to <50% Neuroblastoma (Schwannian stroma-poor) Poorly differentiated subtype Undifferentiated subtype Differentiating subtype Any age UH ≥200/5,000 cells Mitotic & karyorrhectic cells 100-200/5,000 cells <100/5,000 cells Any age ≥1.5 yr <1.5 yr UH UH FH ≥200/5,000 cells 100-200/5,000 cells <100/5,000 cells Any age UH ≥1.5 yr <1.5 yr ≥5 yr UH FH UH FH
- 18. Multi-Scale Machine Learning Based Shimada Classification System • Background Identification • Image Decomposition (Multi- resolution levels) • Image Segmentation (EMLDA) • Feature Construction (2nd order statistics, Tonal Features) • Feature Extraction (LDA) + Classification (Bayesian) • Multi-resolution Layer Controller (Confidence Region) No Yes Image Tile Initialization I = L Background? Label Create Image I(L) Segmentation Feature Construction Feature Extraction Classification Segmentation Feature Construction Feature Extraction Classifier Training Down-sampling Training Tiles Within Confidence Region ? I = I -1 I > 1? Yes Yes No No TRAINING TESTING
- 20. Training using only scene level categories • Gigapixel image scenes – only scene level labels are provided • In Pathology application – the Pathology classification of a whole slide image is given • No fine level training information • Method must infer which patches are crucial (discriminative) to the scene level classification • Hidden variable representing likelihood that the label of the patch is the same as the true label for the entire image • Neural network architecture carries out computations to determine which patches are discriminative • Patch level classifiers are used to determine scene level prediction using multi-class logistic regression or SVM
- 21. Deep Learning - Brain Tumor Classification – CVPR 2016 https://www.cv-foundation.org/openaccess/content_cvpr_2016/html/Hou_Patch- Based_Convolutional_Neural_CVPR_2016_paper.html
- 22. Heterogeneity: category shared by two patches in each column
- 23. EM Method iteratively eliminates non discriminative patches
- 24. Image level decision fusion model
- 25. Brain Le Hou, Dimitris Samaras, Tahsin Kurc, Yi Gao, Liz Vanner, James Davis, Joel Saltz
- 26. Specify where objects are located -> Semantic Segmentation Input: Percent pixels in each patch belonging to a category Output: High resolution semantic segmentation Hou, Jojic, Malkin, Robinson, Samaras, Saltz (Microsoft Research and Stony Brook University)
- 27. Input image with low resolution labels – generates a high resolution image – “Super resolution”
- 28. Labels from the Multi-Resolution Land Characteristics Consortium - https://www.mrlc.gov/tools
- 29. Example Accuracy and Jaccard Performance Data : see upcoming paper
- 30. Ground truth vs Super-resolution and High Resolution Models
- 31. Application to Pathology Lymphocyte Segmentation
- 32. Instance Segmentation using synthetic scenes Uses Generative Associative Networks (GANS) Two players: a generator and a discriminator Generator generates new instances of an object while the discriminator determines whether the new instance belongs to the actual dataset Simple version of this – generate scene that looks realistic. Because the scene is artificial, we can semantically segment all object instances We then use the artificial scenes as training data
- 33. The challenge • Conceptually identical objects can look different in different kinds of scenes • Generate training data to find objects in new types of scenes • In Pathology, nuclei look different in different tissue types – e.g. brain, kidney, liver, prostate etc, etc • Clearly not limited to Pathology – consider segmenting trees, vortices, tornados or microfossils Microfossils - TOTAL
- 34. Requirements and Approach • GAN scene generation need a good starting point – we need to generate somewhat realistic artificial tissue samples • Use excellent and OK artificial scenes as training data by using GAN provided artificial tissue quality estimates • Evaluate impact on segmentation loss function and give preference to “hard” examples
- 35. Generation of starting point synthetic tissue
- 36. GAN based refinement of synthetic tissue
- 37. Examples of generated tissue with quality estimates
- 38. Deep Learning Pipeline for NCI consortium research Deep learning problem - specify where objects are located • National Cancer Institute Sponsored Cancer Genome Atlas Pan Cancer Atlas Collaboration • High resolution (250K patches per whole slide image) mapping of immune cells • 13 Cancer Types • Correlation of spatial immune cell distribution with very detailed molecular datasets • Datasets available in Cancer Imaging Archive; publications in Cell journal series • Linked learning/molecular can can be interactively explored in CRI iATLAS - https://www.cri-iatlas.org/
- 39. TCGA Pan Cancer Atlas – Immune Landscape of Cancer • Six identified immune subtypes span cancer tissue types and molecular subtypes • Immune subtypes differ by somatic aberrations, microenvironment, and survival • Multiple control modalities of molecular networks affect tumor- immune interactions • These analyses serve as a resource for exploring immunogenicity across cancer types http://www.cell.com/immunity/fullt ext/S1074-7613(18)30121-3
- 40. • Stony Brook, Institute for Systems Biology, MD Anderson, Emory group • TCGA Pan Cancer Immune Group – led by ISB researchers • Deep dive into linked molecular and image based characterization of cancer related immune response http://www.cell.com/cell-reports/pdf/S2211-1247(18)30447-9.pdf
- 41. ● Deep learning based computational stain for staining tumor infiltrating lymphocytes (TILs) ●TIL patterns generated from 4,759 TCGA subjects (5,202 H&E slides), 13 cancer types ●Computationally stained TILs correlate with pathologist eye and molecular estimates ●TIL patterns linked to tumor and immune molecular features, cancer type, and outcome
- 42. Le Hou – Graduate Student Computer Science Vu Nguyen– Graduate Student Computer Science Anne Zhao – Pathology Informatics Biomedical Informatics, Pathology (now Surg Path Fellow SBM) Raj Gupta – Pathology Informatics Biomedical Informatics, Pathology Deep Learning and Lymphocytes
- 43. Importance of Immune System in Cancer Treatment and Prognosis • Tumor spatial context and cellular heterogeneity are important in cancer prognosis • Spatial TIL densities in different tumor regions have been shown to have high prognostic value • Immune related assays used to steer cancer immune therapy • TIL maps being computed for SEER Pathology studies and will be routinely computed for data contributed to TCIA archive • Ongoing study to relate TIL patterns with immune gene expression groups and patient response
- 44. Training, Model Creation • Algorithm first trained on image patches • Several cooperating deep learning algorithms generate heat maps • Heat maps used to generate new predictions • Companion molecular statistical data analysis pipelines
- 45. Training, threshold adjustment, quality control
- 46. We pre-train using sparse autoencoder that ”likes” immune cells
- 47. Visual tools used to QC and iteratively refine training data: Quantitative Imaging Pathology - QuIP Tool Set
- 48. Interactive Deep Learning Training Tool
- 50. Validation – Stratified sampling from 5K whole slide images Arvind Rao, expert in spatial biostatistics (U Michigan)
- 55. TIL Pattern Descriptions Qualitative (Alex Lazar, Raj Gupta) • ‘‘Brisk, diffuse’’ diffusely infiltrative TILs scattered throughout at least 30% of the area of the tumor (1,856 cases); • ‘‘Brisk, band-like’’ - band-like boundaries bordering the tumor at its periphery (1,185); • ‘‘Nonbrisk, multi-focal’’ loosely scattered TILs present in less • than 30% but more than 5% of the area of the tumor (1,083); • ‘‘Non-brisk, focal’’ for TILs scattered throughout less than 5% but greater than 1% of the area of the tumor (874); • ‘‘None’’ < 1% TILS - in 143 cases Quantitative – Arvind Rao • Agglomerative clustering • Cluster indices representing cluster number, density, cluster size, distance between clusters • Traditional spatial statistics measures • R package clusterCrit by Bernard Desgraupes - Ball- Hall, Banfield-Raftery, C Index, and Determinant Ratio indices
- 57. Use of Pathology Deep Learning Methods in Multi-institutional Consortia • NCI Quantitative Imaging for Pathology (QuIP): Stony Brook, Emory, MD Anderson, Institute for Systems Biology, Oak Ridge • NCI SEER Pathology: Stony Brook, Emory, Rutgers, University of Kentucky (three Cancer registries) • Cancer Imaging Archive: Arkansas, Stony Brook, Emory (Stony Brook leads Pathology component) • Virtual Tissue Repository: Led by NCI SEER; Stony Brook, Emory • TIES Research Network - Integrated Pathology text and imaging: Pittsburgh, Stony Brook main sites, 6+ other sites (Stony Brook leads digital Pathology)
- 58. SEER coverage includes 31.9 percent of Whites, 30.0 percent of African Americans, 44.0 percent of Hispanics, 49.3 percent of American Indians and Alaska Natives, 57.5 percent of Asians, and 68.5 percent of Hawaiian/Pacific Islanders.
- 59. Methods and tools for integrating pathomics data into cancer registries Saltz, Sharma, Foran and Durban • Enhance SEER registry data with machine learning based classifications and quantitative pathomics feature sets. • The New Jersey State Cancer Registry, Georgia and Kentucky State Cancer Registries • Prostate Cancer, Lymphoma and NSCLC • Repository of high‐quality digitized pathology images for subjects whose data is being collected by the registries. • Extract computational features and establish deep linkages with registry data, thus enabling the creation of information‐rich, population cohorts containing objective imaging and clinical attributes
- 60. Future Exascale Efforts I - Interactive Incorporation of high throughput training data • Iterative paradigm of human in the loop training set correction • For example, current training in TIL application involves roughly 100K image patches • Our 5000K WSI dataset, iteratively corrected by humans, now has 500M image patches that can be used for training • Super-resolution semantic segmentation can make use of multiple types of sensor and map data to provide ground truth – sensor and human markup integration challenge • Crucial issue is support for interactive exploration of impact on training data • Rapid iteration on impact of variations in CNN architecture also crucial
- 61. Future Exascale Efforts II – Leveraging Deep Learning to Optimize Systems Software Performance • Methods for optimizing movement of computation and data through complex storage hierarchies • My past work in this has assumed the need for simple, effective heuristics • Was developed before the acute need for this kind of thing (but does have 487 citations)
- 62. Active Disk/Data Cutter Re-Do with Deep Learning • Complex storage and I/O Systems require intelligent management of work and data • Extreme scale computing systems can support complex optimizations • Deep learning systems capable of carrying out policies with previously infeasible levels of complexity
- 63. Future Exascale Efforts III • Scaling up whole slide imaging analytics • Current studies involve order 10K whole slide images • TIES consortium aims to obtain 500M slides over next three years • SEER will ultimately require roughly 6M whole slide images • Human tumor and human tissue atlas are adding 3rd dimension and deep molecular annotation to analysis
- 64. THANKS!
- 65. ITCR Team Stony Brook University Joel Saltz Tahsin Kurc Yi Gao Allen Tannenbaum Erich Bremer Jonas Almeida Alina Jasniewski Fusheng Wang Tammy DiPrima Andrew White Le Hou Furqan Baig Mary Saltz Raj Gupta Emory University Ashish Sharma Adam Marcus Oak Ridge National Laboratory Scott Klasky Dave Pugmire Jeremy Logan Yale University Michael Krauthammer Harvard University Rick Cummings
- 66. Funding – Thanks! • This work was supported in part by U24CA180924, U24CA215109, NCIP/Leidos 14X138 and HHSN261200800001E, UG3CA225021-01 from the NCI; R01LM011119-01 and R01LM009239 from the NLM • This research used resources provided by the National Science Foundation XSEDE Science Gateways program under grant TG-ASC130023 and the Keeneland Computing Facility at the Georgia Institute of Technology, which is supported by the NSF under Contract OCI-0910735.
Editor's Notes
- #18: This is Dr. Shimada’s prognosis classfication