SlideShare a Scribd company logo

Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

Institute of Information Systems (HES-SO), profile picture
Institute of Information Systems (HES-SO)

The document discusses the application of texture-based computational models in analyzing biomedical images and digital histopathology to improve personalized medicine. It highlights the challenges associated with current methods, such as biopsies being invasive and slow, while emphasizing the potential of radiomics and histopatholomics in non-invasive image analysis to predict diagnosis and treatment responses. Key objectives include developing highly adaptive models that can characterize tissue properties and their relationships to organ anatomy, ultimately leading to enhanced patient outcomes.

Health & Medicine
Related topics:
Image Analysis Techniques
1 of 50
Downloaded 54 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
TEXTURE-BASED COMPUTATIONAL MODELS OF TISSUE IN BIOMEDICAL IMAGES: 
 INITIAL EXPERIENCE WITH DIGITAL HISTOPATHOLOGY Adrien Depeursinge, PhD Journées GdR ISIS “Analyse de Tissu Biologique et Histopathologie Numérique” Paris, June 23rd 2014
BACKGROUND – RADIOMICS - HISTOPATHOLOMICS • Personalized medicine aims at enhancing the patient’s 
 quality of life and prognosis • Tailored treatment and medical decisions based 
 on the molecular composition of diseased tissue • Current limitations [1] • Molecular analysis of tissue composition 
 is invasive (biopsy), slow and costly • Cannot capture molecular heterogeneity 2 [1] Intratumor heterogeneity and branched evolution revealed by multiregion sequencing, Gerlinger et al., 
 N Engl J Med, 366(10):883-92, 2012. Intratumor Heterogeneity Reveale B Regional Distribution of Mutations A Biopsy Sites SOX9 CENPN PSMD7 RIMBP2 GALNT11 ABHD11 UGT2A1 MTOR PPP6R2 ZNF780A WSCD2 CDKN1B PPFIA1 TH SSNA1 CASP2 PLRG1 SETD2 CCBL2 SESN2 MAGEB16 NLRP7 IGLON5 KLK4 WDR62 KIAA0355 CYP4F3 AKAP8 ZNF519 DDX52 ZC3H18 TCF12 NUSAP1 X4 KDM2B MRPL51 C11orf68 ANO5 EIF4G2 MSRB2 RALGDS EXT1 ZC3HC1 PTPRZ1 INTS1 CCR6 DOPEY1 ATXN1 WHSC1 CLCN2 SSR3 KLHL18 SGOL1 VHL C2orf21 ALS2CR12 PLB1 FCAMR IFI16 BCAS2 IL12RB2 Ubiquitous Shared prima 10 cm R7 (G4) R5 (G4) R9 R3 (G4) R1 (G3) R2 (G3) R4 (G1) R6 (G1) Hilum R8 (G4) ution of Mutations ationships of Tumor Regions D Ploidy Profiling PreP PreM R1 R2 R3 R5 R8 R9 R4 M1 M2a M2b C2orf85 WDR7 SUPT6H CDH19 LAMA3 DIXDC1 HPS5 NRAP KIAA1524 SETD2 PLCL1 BCL11A IFNAR1 DAMTS10 C3 KIAA1267 RT4 CD44 ANKRD26 TM7SF4 SLC2A1 DACH2 MMAB ZNF521 HMG20A DNMT3A RLF MAMLD1 MAP3K6 HDAC6 PHF21B FAM129B RPS8 CIB2 RAB27A SLC2A12 DUSP12 ADAMTSL4 NAP1L3 USP51 KDM5C SBF1 TOM1 MYH8 WDR24 ITIH5 AKAP9 FBXO1 LIAS TNIK SETD2 C3orf20 MR1 PIAS3 DIO1 ERCC5 KL ALKBH8 DAPK1 DDX58 SPATA21 ZNF493 NGEF DIRAS3 LATS2 ITGB3 FLNA SATL1 KDM5C KDM5C RBFOX2 NPHS1 SOX9 CENPN PSMD7 RIMBP2 GALNT11 ABHD11 UGT2A1 MTOR PPP6R2 ZNF780A WSCD2 CDKN1B PPFIA1 TH SSNA1 CASP2 PLRG1 SETD2 CCBL2 SESN2 MAGEB16 NLRP7 IGLON5 KLK4 WDR62 KIAA0355 CYP4F3 AKAP8 ZNF519 DDX52 ZC3H18 TCF12 NUSAP1 X4 KDM2B MRPL51 C11orf68 ANO5 EIF4G2 MSRB2 RALGDS EXT1 ZC3HC1 PTPRZ1 Privatebiquitous Shared primary Shared metastasis Lung metastases Chest-wall metastasis Perinephric metastasis M1 10 cm R2 (G3) R4 (G1) R6 (G1) HilumR8 (G4) Primary tumor M2b M2a ution of Mutations PreP PreM R1 R2 R3 R5 R8 R9 R4 M1 M2a M2b C2orf85 WDR7 SUPT6H CDH19 LAMA3 DIXDC1 HPS5 NRAP KIAA1524 SETD2 PLCL1 BCL11A IFNAR1 DAMTS10 C3 KIAA1267 RT4 CD44 ANKRD26 TM7SF4 SLC2A1 DACH2 MMAB ZNF521 HMG20A DNMT3A RLF MAMLD1 MAP3K6 HDAC6 PHF21B FAM129B RPS8 CIB2 RAB27A SLC2A12 DUSP12 ADAMTSL4 NAP1L3 USP51 KDM5C SBF1 TOM1 MYH8 WDR24 ITIH5 AKAP9 FBXO1 LIAS TNIK SETD2 C3orf20 MR1 PIAS3 DIO1 ERCC5 KL ALKBH8 DAPK1 DDX58 SPATA21 ZNF493 NGEF DIRAS3 LATS2 ITGB3 FLNA SATL1 KDM5C KDM5C RBFOX2 NPHS1 SOX9 CENPN PSMD7 RIMBP2 GALNT11 ABHD11 UGT2A1 MTOR PPP6R2 ZNF780A WSCD2 CDKN1B PPFIA1 TH SSNA1 CASP2 PLRG1 SETD2 CCBL2 SESN2 MAGEB16 NLRP7 IGLON5 KLK4 WDR62 KIAA0355 CYP4F3 AKAP8 ZNF519 DDX52 ZC3H18 TCF12 NUSAP1 X4 KDM2B MRPL51 C11orf68 ANO5 EIF4G2 MSRB2 RALGDS EXT1 ZC3HC1 PTPRZ1 Privatebiquitous Shared primary Shared metastasis Lung metastases Chest-wall metastasis Perinephric metastasis M1 10 cm R2 (G3) R4 (G1) R6 (G1) Hilum R8 (G4) Primary tumor M2b M2a
BACKGROUND – RADIOMICS - HISTOPATHOLOMICS • Huge potential for computerized medical image analysis • Create imaging biomarkers to predict diagnosis, prognosis, 
 treatment response [2] 3 [2] Imaging and genomics: is there a synergy?, Jaffe et al., Radiology, 264(2):329-31, 2012
 [3] Radiomics: the process and the challenges, Kumar et al., Magn Reson Imaging, 30(9):1234-48, 2012
 [4] Histopathological image analysis: a review, Gurcan et al., IEEE Reviews in Biomed Eng, 2:147-71, 2009 Radiomics [3] “Histopatholomics” [4] Reuse existing 
 diagnostic images ✓ radiology data ✓ digital pathology Capture tissue 
 heterogeneity ✓ 3D neighborhoods
 (e.g., 512x512x512) ✓ large 2D regions
 (e.g., 15,000x15,000) Analytic power beyond 
 naked eyes ✓ complex 3D tissue morphology ✓exhaustive characterization of 2D tissue structures Non-invasive ✓ x
BACKGROUND – RADIOMICS - HISTOPATHOLOMICS • Huge potential for computerized medical image analysis • Create imaging biomarkers to predict diagnosis, prognosis, 
 treatment response • Local quantitative image feature extraction • Supervised machine learning 4 malignant, nonresponder malignant, responder benign pre-malignant undefined quant. feat. #1 quant.feat.#2 Supervised learning, 
 big data This could include an additional step of studying the spatial relationships between local image properties (e.g., using image graphs)
• Shape, intensity, margin, texture, … • Shape and margin features often 
 require prior image segmentation • 2D and 3D texture analysis can quantify micro- and macro- structures in biomedical images [4,6] IMAGE FEATURES 5 GURCANetal.:HISTOPATHOLOGICALIMAGEANALYSIS:AREVIEW Fig.5.Resultsoftheautomaticsegmentationalgorithm(bluecontours:lumen boundary,blackcontours:innerboundaryofthenucleioftheepithelialcells surroundingthegland).Shownfromlefttorightareexampleimagesofbenign epithelium,intermediate-,andhigh-gradecancer. andare2-DCartesiancoordinatesof.Theevolutionof isthendescribedbyalevel-setformulationadoptedfrom[78] (4.1) [4] Histopathological image analysis: a review, Gurcan et al., IEEE Reviews in Biomed Eng, 2:147-71, 2009
 [5] Quantifying the margin sharpness of lesions on radiological images for content-based image retrieval, Xu et al., Med Phys, 39(9):5405-18, 2012
 [6] Three-dimensional solid texture analysis in biomedical imaging: review and opportunities, Depeursinge et al., Med Image Anal, 18(1):176-96, 2014
 [7] Prediction of prognosis for prostatic adenocarcinoma by combined histological grading and clinical staging, Gleason et al., J Urol, 111(1):58-64, 1974
 [8] HyMaP: A hybrid magnitude-phase approach to unsupervised segmentation of tumor areas in breast cancer histology images, Khan et al., 
 J Pathol Inform, 4, 2013 [4] entsdrawnontheborderof formationtocharacterizethe hescaleandwindowparam- entsfromthisportionofthe (d) )byradiologist.(b)Automatically Ioutline(inwhite).(c)Finallung ewithlinesegments(inwhite)that [5] J Pathol Inform 2013, 1:1 http://www.jpath the phase spectrum to repres HyperCS regions in a breast the recently established efficac exhibiting randomness. Let vi (x,y) denote the ith Gab normalized and smoothened v I(x,y), where i = ,1,2,...,Ng , Ng the number of orientations. We v x y v x y j x yi i i( , ) ( , ) exp( ( , ))= φ where |·| denotes the magnit denotes the local phase. The g and its magnitude can then be φi i i i i x y v x y v x y v x y v x y ′ ′ ′ = − ⎡ ⎢ ⎢ ⎤ ⎥ ⎥ ( , ) ( , ) ( , ) ( , ) ( , ) Figure 1:A sample H & E–stained breast cancer histology image:(a) Original image, and (b) Overlaid image, with four types of contents shown in different colors.The tumor areas are shown in Red,HypoCS in Purple,and HyperCS in Green.Areas containing background or fat tissue are shown in white with black outline. Note the difference in morphology of the Hypo- and Hypercellular stromal regions ba [Downloaded free from http://www.jpathinformatics.org on Tuesday, June 16, 2015, IP: 128.179.146.236] [7] [8]
COMPUTERIZED TEXTURE ANALYSIS directionsscale 6 • Image scales and directions are important for visual texture discrimination • Most approaches are leveraging these two properties • Explicitly: Gray-level co-occurrence matrices (GLCMs), run-length matrices (RLE), directional filterbanks and wavelets, Fourier, histograms of gradients (HOG), local binary patterns (LBP) • Implicitly: Convolutional neural networks (CNN), scattering transform, topographic independant component analysis (TICA)
COMPUTERIZED TEXTURE ANALYSIS 7 • Texture invariances: computer vision VS biomedical imaging Computer vision Biomedical image analysis scale scale-invariant multi-scale rotation rotation-invariant rotation-invariant [4] Histopathological image analysis: a review, Gurcan et al., IEEE Reviews in Biomed Eng, 2:147-71, 2009
 [9] A sparse texture representation using local affine regions, Lazebnik et al., IEEE Trans on Pattern Anal and 
 Mach Intel, 27(8):1265-78, 2005 160 Fig. 10. (a) A digitized histopathology image of Grade 4 CaP and different graph-based r Diagram, and Minimum Spanning tree. Fig. 11. Digitized histological image at successively higher scales (magnifica- tions) yields incrementally more discriminatory information in order to detect suspicious regions. or resolution. For instance at low or coarse scales color or tex- ture cues are commonly used and at medium scales architec- tural arrangement of individual histological structures (glands and nuclei) start to become resolvable. It is only at higher res- olutions that morphology of specific histological structures can be discerned. In [93], [94], a multiresolution approach has been used for the classification of high-resolution whole-slide histopathology im- ages. The proposed multiresolution approach mimics the eval- uation of a pathologist such that image analysis starts from the lowest resolution, which corresponds to the lower magnification levels in a microscope and uses the higher resolution represen- Fig. 12 image 1, (c) r as susp show three scale (scal the n dition highe tumo At is com COMPUTERIZED TEXTURE ANALYSIS 7 • Invariances: computer vision versus biomedical imaging Computer vision Biomedical image analysis scale scale-invariant multi-scale rotation rotation-invariant rotation-invariant [4] Histopathological image analysis: a review, Gurcan et al., IEEE Reviews in Biomed Eng, 2:147-71, 2009
 160 IE Fig. 10. (a) A digitized histopathology image of Grade 4 CaP and different graph-based representation Diagram, and Minimum Spanning tree. Fig. 11. Digitized histological image at successively higher scales (magnifica- tions) yields incrementally more discriminatory information in order to detect suspicious regions. or resolution. For instance at low or coarse scales color or tex- ture cues are commonly used and at medium scales architec- tural arrangement of individual histological structures (glands and nuclei) start to become resolvable. It is only at higher res- olutions that morphology of specific histological structures can be discerned. In [93], [94], a multiresolution approach has been used for the classification of high-resolution whole-slide histopathology im- ages. The proposed multiresolution approach mimics the eval- uation of a pathologist such that image analysis starts from the lowest resolution, which corresponds to the lower magnification levels in a microscope and uses the higher resolution represen- tations for the regions requiring more detailed information for a classification decision. To achieve this, images were decom- posed into multiresolution representations using the Gaussian pyramid approach [95]. This is followed by color space con- version and feature construction followed by feature extraction and feature selection at each resolution level. Once the classifier is confident enough at a particular resolution level, the system assigns a classification label (e.g., stroma-rich, stroma-poor or Fig. 12. Results fr image with the tum 1, (c) results at scale as suspicious at low shows the origin three columns s scales. Pixels cl (scale) are disc the number of p ditionally, the p higher scales al tumor and nont At lower reso is commonly us pattern of gland tized histologic scenes can be to every pixel i dient, and Gab the scale, orien gion of interest features within [4][9]
• Averaging texture properties over the entire lesions or images discards tissue heterogeneity [10] • Exhaustive and non-specific features • Limited characterization of directions [6] BIOMEDICAL TEXTURE ANALYSIS: LIMITATIONS (1/4) 8 [6] Three-dimensional solid texture analysis in biomedical imaging: review and opportunities, Depeursinge et al., Med Image Anal, 18(1):176-96, 2014
 [10] Quantitative imaging in cancer evolution and ecology, Gatenby et al., Radiology, 269(1):8-15, 2013
 feartures are not specific when not learned from data
• A global characterization of directions is not enough [11] • local organization of image directions: • independently from their local orientation: • Rotation-covariance • (local) grouped steering 
 of the operators: BIOMEDICAL TEXTURE ANALYSIS: LIMITATIONS (2/4) image operators: grouped steering: 9 [11] Rotation–covariant texture learning using steerable Riesz wavelets, Depeursinge et al., IEEE Trans Imag Proc, 23(2):898-908, 2014

BIOMEDICAL TEXTURE ANALYSIS: LIMITATIONS (3/4) GLCM contrast [1 0] GLCMcontrast[01] GLCMs (unaligned) Riesz (unaligned) Riesz (aligned) 10
BIOMEDICAL TEXTURE ANALYSIS: LIMITATIONS (4/4) GLCM contrast [1 0] GLCMcontrast[01] GLCMs (unaligned) Riesz (unaligned) Riesz (aligned) 11
OBJECTIVES 12 • Highly adaptive texture-based computational models of biomedical tissue: ✓ Complete coverage of image scales and 
 directions in 2-D and 3-D ✓ Rotation-covariance ✓ Specificity: the models can be trained to 
 characterize specific tissue types ✓ Local characterization of tissue properties ✓ Locate tissue properties in organ anatomy 
 to create digital phenotypes • Goal: predict survival, function, treatment response 
 and reveal subtypes be averaged over the folds of the CV and used to build texture models for each location along dROI (see Fig 9). The sum of the weights for all channels from each location will reveal the subregions that are specific to each tumor subtypes, and most related to patient survival. A K–means clustering of the vectors w will be carried out for each locations to evaluate the stability of the regional models over the folds of the CV and define homogeneous groups among patients. A selection of the models based on stability and location importance will be carried out. Fig. 9 Prototype tissue archi- tecture of a GBM tumor [128]. In a second step, the selected models will be locally steered to maximize their magnitude. The energies of the maximal magnitudes will be used to construct a final feature space for (1) predicting the tumor subtypes and (2) performing Kaplan–Meier survival analysis. The performance of the proposed approach for predicting tumor subtypes and patient survival will be compared to (1) unaligned wavelet energies and (2) average wavelet energies over the entire tumor. While starting with a LOPO CV on the TCGA–TCIA dataset, the generalizability of the approach will be further assessed by training with the TCGA–TCIA and testing with the SU dataset. Deliverable 2.1: Predicting tumor subtype and survival from localized of tissue properties in GBM tumors. Task 2.2: Digital lung tissue atlases of ILD diagnoses (8 months of the PI) In this task, we will use a simple atlas of the lungs to locate texture properties and create prototype diagnosis phenotypes of ILDs. In previous work, we developed a simple 3–D atlas of the lungs with 36 subregions that will be used in this task [42]. In a first step, diagnosis–wise digital tissue atlases will be created by learning 3–D texture models (i.e., the average of w over the folds of the LOPO CV) for each 36 regions of the lungs (see Fig. 10). The regions for which the models are most distant11 from all other diagnoses will be highlighted and compared to previously built models of tissue patterns [25] to create 3–D prototype tissue atlases for each diagnosis. The obtained results will be validated using medical knowledge (e.g., Table 1 of [42]). Fig. 10 Tissue atlas of the lungs. For each diagnosis, K–means clustering of the vectors w will be carried out for each locations to evaluate the stability of the regional models over the folds of the CV and reveal homogeneous groups among patients (e.g., subtypes of UIP). The most stable models will be kept for the further characterization of lung tissue types. A hierarchical clustering of the models from all diagnoses will be carried out to define a radiomics–based hierarchy of all diagnoses, which will be compared to medical knowledge [140] (e.g., Fig. 1 of [40]). A large feature space including the energies of the steered models from each of the 36 localizations will be used to predict the diagnoses with uncertainty assessment (using e.g., pairwise coupling [141]). When a minimum amount of trust is not achieved when predicting a given diagnosis, the parent group of ILD diagnoses in the previously built hierarchy will be predicted instead [19]. Deliverable 2.2: Digital tissue atlases of ILD diagnoses and their subtypes. Task 2.3: Digital tissue atlases of ILDs: correlation with PFTs and survival (6 months ) Digital tissue atlases will be constructed for poor versus normal/high (1) PFTs and (2) survival. Regions for which the models significantly di↵er between poor versus normal/high will be revealed as being of primary importance to evaluate pulmonary function. The links between these models and previously built models of tissue patterns will be investigated to define the combination of regions and patterns that are most responsible for lung function impairment. The feature space spanned by the energies of the steered models will be used to predict (1) PFT values or (2) survival with a LOPO CV, which can be evaluated using ground truth. Deliverable 2.3: Estimating pulmonary function from digital tissue atlases of ILD diagnoses in CT images. 5.3 WP3. Imaging genomics 8 (a) Original Image (c) Voronoi Diagram 8 (a) Original Image (b) Simple Cell Graph (c) Voronoi Diagram (d) Delaunay Triangulation (e) ECM-Aware Cell-Graph Fig. 2 A fractured bone tissue example is shown in 2(a). Note the fracture cells in the middle of the original image. The simple-cell-graph representation, the Voronoi diagram and the Delaunay triangulation for this sample tissue are depicted in 2(b), 2(c) and 2(d). The corresponding ECM-aware cell-graph is drawn in 2(e). The interactions between fracture cells are drawn with blue and the red cells with red color. Delaunay triangulation represents the tissue as a single connected component and does not allow crossing of edges. Simple-cell- graphs relaxes these restrictions and allows the tissue to non-planar graph and disconnected. Likewise, ECM-aware cell-graphs do not put such restrictions on the tissue and moreover can capture the structural organization of different cells in a tissue. Furthermore, Delaunay triangulations are fixed representations whereas ECM-aware cell-graphs can be adjusted with different linking thresholds.[12] Automated classification of usual interstitial pneumonia using regional volumetric texture analysis in high- resolution CT, Depeursinge et al., Invest Radiol, 50(4):261-67, 2015
 [13] ECM-Aware Cell-Graph Mining for Bone Tissue Modeling and Classification, Bilgin et al., Data Min Knowl Discov, 20(3):416-38, 2009 [12] [13]
Steerable texture models   Σ associated texture model EXAMPLE: PATTERN = WIGGLED CHECKERBOARD pattern to learn: 30 Steerable texture models: our workhorse …. 13
• Multi-directional, multi-scale and rotation-covariant image analysis is achieved using Riesz wavelets • The components of the th-order Riesz transform of a 2-D signal are defined in the Fourier domain as [14]: • Yields allpass filters: only phase (i.e., direction) is kept and is defined by th-order partial derivatives THE RIESZ TRANSFORM 14 [14] Wavelet Steerability and the Higher-Order Riesz Transform, Unser et al., IEEE Trans on Imag Proc, 
 19(3):636-52, 2010
• Riesz components THE RIESZ TRANSFORM input image 15
• Riesz components THE RIESZ TRANSFORM 16
• Riesz components THE RIESZ TRANSFORM 17
• Multi-scale filterbanks are obtained by combining the Riesz transform with isotropic wavelets [15]
 • E.g., dyadic ( ) • Systematic coverage 
 of image scales THE RIESZ TRANSFORM 18 UNSER et al.: STEERABLE PYRAMIDS AND TIGHT WAVELET FRAMES IN 2711 of unity) guarantees that is dense in ; it is essential for the -completeness of the wavelet decomposition. We now proceed with the construction of orienta- tion-free wavelets by projecting the multiresolution Riesz basis onto some appropriate wavelet subspace . B. Isotropic Bandlimited Wavelet Frames There are a number of constructions in the literature that fall into this category [13], [27]–[29]. Before reviewing them, we apply the aforementioned projection strategy to obtain a straightforward design that is in direct correspondence with Shannon’s sampling theorem, and that is the starting point for the specification of Meyer-type wavelets. 1) Construction of Isotropic, Shannon-Type Wavelets: By selecting in (19), we specify the so-called Shannon multiresolution analysis of , which consists of a sequence of embedded subspaces that are bandlimited to We then define some corresponding wavelet subspaces of radi- ally bandpass functions (20) Since is a closed subspace of , we can apply Proposition 3 to its orthogonal sinc basis to obtain the tight wavelet frame of with (21) where is the impulse response of the ideal radial bandpass filter, whose frequency response is Fig. 1. Tiling of the 2-D frequency domain using radial-bandpass filters. The shaded area corresponds to the spectral support of the wavelet subspace ; it is included in the spectral support of (enclosing square). satisfy (22). This leads to the following extended definition of the wavelet subspaces which is equivalent to (20) if is the impulse response of the ideal radial bandpass filter. Since can be written as , there exists a sequence such that where the wavelet functions are still given by (21). This indicates that is a frame of , albeit not neces- sarily a tight one. Yet, if Condition (22) is satisfied, then one recovers the tight frame property over which is the union of the wavelet subspaces , . The condition for the wavelet frame to be isotropic is that the restriction of the fil- tering function over be isotropic, i.e., . [15] Steerable pyramids and tight wavelet frames in , Unser et al., IEEE Trans on Imag Proc, 
 20(10):2705-21, 2011
THE RIESZ TRANSFORM 19 • Steerability [16]: • Example : • Higher-order steering can be done fully analytical ✓Complete coverage of the image directions ✓Steerability enables rotation-covariance [16] The design and use of steerable filters, Adelson et al., IEEE Trans on Pattern Anal and Mach Intel, 
 13(9):891-906, 1991
• A Riesz filterbank constitutes a dictionary of basic textures: • Higher-level texture models are built from linear combinations of Riesz components LEARNING TEXTURE MODELS 20
  Σ EXAMPLE: PATTERN = WIGGLED CHECKERBOARD pattern to learn:
  Σ EXAMPLE: PATTERN = WIGGLED CHECKERBOARD pattern to learn:
  Σ EXAMPLE: PATTERN = WIGGLED CHECKERBOARD pattern to learn:
  Σ EXAMPLE: PATTERN = WIGGLED CHECKERBOARD pattern to learn:
  Σ EXAMPLE: PATTERN = WIGGLED CHECKERBOARD pattern to learn:
  Σ EXAMPLE: PATTERN = WIGGLED CHECKERBOARD pattern to learn:
  Σ EXAMPLE: PATTERN = WIGGLED CHECKERBOARD pattern to learn:
  Σ associated texture model EXAMPLE: PATTERN = WIGGLED CHECKERBOARD pattern to learn: 28
• Support vector machines (SVM) were used to learn (multi-scale ) versus   ONE-VERSUS-ALL SUPERVISED MODEL LEARNING [11]: texture all others [11] Rotation-covariant texture learning using steerable Riesz wavelets, Depeursinge et al., IEEE Trans Imag Proc, 23(2):898-908, 2014
 29
• 2-D synthetic textures with noise ( ) ✓ Captures specific local properties in terms of image 
 scales and directions LEARNING TEXTURE MODELS 30
• 2-D tissue from interstitial lung diseases (ILD) in CT [17] • ( , 32x32 patches) LEARNING TEXTURE MODELS healthy emphysema ground glass fibrosis micronodules 3011 patches, 7 patients. 407 patches, 
 6 patients. 2226 patches, 32 patients. 2962 patches, 37 patients. 5988 patches, 16 patients. 31 [17] Multiscale lung texture signature learning using the Riesz transform, Depeursinge et al., Med Image Comput Comput Assist Interv (MICCAI), 15(3):517-24, 2012

• 3-D synthetic textures ( , based on [18]) • Vertical planes • 3-D checkerboard • 3-D wiggled 
 checkerboard LEARNING TEXTURE MODELS 32 [18] 3D Steerable Wavelets and Monogenic Analysis for Bioimaging, Chenouard et al., IEEE 8th Int Symp on Biomed Imag (ISBI), 2132-5, 2011
• Steering texture models: STEERABLE TEXTURE MODELS [11] The expression of the rotated texture model remains a linear combination of the initial Riesz components [11] Rotation-covariant texture learning using steerable Riesz wavelets, Depeursinge et al., IEEE Trans Imag Proc, 23(2):898-908, 2014

• Steering texture models: • Rotation-covariant models: local steering of • Keep the maximum magnitude of the model • Local quantitative features: energies/abs values of the magnitudes in the patch STEERABLE TEXTURE MODELS [11] 34 The expression of the rotated texture model remains a linear combination of the initial Riesz components [11] Rotation-covariant texture learning using steerable Riesz wavelets, Depeursinge et al., IEEE Trans Imag Proc, 23(2):898-908, 2014

Classification of 2-D natural textures IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2013 5 1) canvas001 2) canvas002 3) canvas003 4) canvas005 5) canvas006 6) canvas009 7) canvas011 8) canvas021 9) canvas022 10) canvas023 11) canvas025 12) canvas026 13) canvas031 14) canvas032 15) canvas033 16) canvas035 17) canvas038 18) canvas039 19) tile005 20) tile006 21) carpet002 22) carpet004 23) carpet005 24) carpet009 Fig. 5. 128 ⇥ 128 blocks from the 24 texture classes of the Outex database. 1) canvas 2) cloth 3) cotton 4) grass 5) leather 6) matting 7) paper 8) pigskin APPLICATIONS (1/2) 35
• 24 classes, 180 images/class, 9 rotation angles in • A SVM classifier is trained with unrotated images only • 98.4% best acc. • aligned models 
 ( ) • Literature: 90-99% • E.g., scattering transform: 
 98.75% [18] 2-D TEXTURE CLASSIFICATION: OUTEX DATABASE [17] Errors occur with most stochastic textures confusion matrix: IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2013 TABLE AVERAGE Ac OBTAINED WITH THE LOCAL ORIENTATION MAXIMIZATION O Outex TC 00010 Outex TC 0001 ¯Ac for N = 2, 4, 6, 8, 10. 98 ± 0.7 97.2 ± 0 ¯Ac for N = 1, 3, 5, 7, 9. 94.4 ± 0.7 93.6 ± 0 ¯Ac for N = 1, . . . , 10. 96.2 ± 2 95.4 ± 0.4 0.5 0.6 0.7 0.8 0.9 1 1 2 3 4 5 6 7 8 9 10 classificationaccuracy N aligned aligned initial Riesz N c,1 R(0,N) Fig. 7. Classification performance with the Outex TC 00010 test suite. The two rotation covariant approaches are performing much better than using the initial Riesz coefficients. The local orientation maximization of N c,1 outperforms the local orientation of the first Riesz template R(0,N) as proposed in [8]. A maximum Ac of 98.4% is reached with N = 8. classes are balanced, the classification accuracy Ac is used as a performance measure of the methods. All performances are summarized in Table I. 1) Outex TC 00010: The classification performance for orders N = 1, . . . , 10 is shown in Fig. 7. The performance using the energy of the coefficients that are maximizing the response of the first Riesz template (i.e., R(0,N) ) at the smallest scale was also evaluated as a first rotation–covariant approach [8]. An average Ac of 96.2 ± 2% is obtained with N = 1 . . . 10 and the local orientation maximization of N . F f m c a i IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2013 5 1) canvas001 2) canvas002 3) canvas003 4) canvas005 5) canvas006 6) canvas009 7) canvas011 8) canvas021 9) canvas022 10) canvas023 11) canvas025 12) canvas026 13) canvas031 14) canvas032 15) canvas033 16) canvas035 17) canvas038 18) canvas039 19) tile005 20) tile006 21) carpet002 22) carpet004 23) carpet005 24) carpet009 Fig. 5. 128 ⇥ 128 blocks from the 24 texture classes of the Outex database. 1) canvas 2) cloth 3) cotton 4) grass 5) leather 6) matting 7) paper 8) pigskin 9) raffia 10) rattan 11) reptile 12) sand 13) straw 14) weave 15) wood 16) wool Fig. 6. 16 Brodatz texture classes of the Contrib TC 00000 test suite. 180 ⇥ 180 images from rotation angles 20 , 70 , 90 , 120 , 135 and 150 of the other seven Brodatz images for each class. The total number of images in the test set is 672. G. Experimental setup OVA SVM models using Gaussian kernels as K(xi, xj) = exp( ||xi xj ||2 2 2 k ) are used both to learn texture signatures and to classify the texture instances in the final feature space obtained after k iterations. A number of scales J = 6 Inc., 2012. The computational complexity is dominated by the local orientation of N c in Eq. 11, which consists of finding the roots of the polynomials defined by the steering matrix A✓ . It is therefore NP–hard (Non–deterministic Polynomial–time hard), where the order of the polynomials is controlled by the order of the Riesz transform N. III. RESULTS The performance of our approach is demonstrated with [17] Multiresolution gray-scale and rotation invariant texture classification with local binary patterns, Ojala et al., IEEE Trans on Pattern Anal and Mach Intel, 24:7(971-87), 2002
 [18] Combined scattering for rotation invariant texture analysis, Sifre et al., European Symposium on Artificial Neural Networks, 2012 Importance of rotation-covariance and learned models
Medulloblastoma tumor classification APPLICATIONS (2/2) ompute the UFL features learned by TICA and the supervised features learned with Riesz wavelets for each image as described in Sections 2.1 and 2.2. Once TICA and Riesz wavelets are computed a final step of supervised classification is made using the combination of the computed features in a concatenated vector as input for a standard softmax classifier as described in Section 2.3. Parameter uning is presented in Section 3.2. [2.4] Medulloblastoma Image Cases Fig. 2. Flowchart for MB feature extraction and classification for both learned repre- entations: Riesz and TICA, the details of each stage are described in subsections. steerable Riesz texture models topographic ICA 37
• Medulloblastoma (MB) • Most common (i.e., 25%) pediatric brain tumor • Major cause of death in pediatric oncology • Histological subtypes of MB have 
 different prognosis and treatments • Anaplastic subtype is the worst! • “marked nuclear pleomorphism, 
 cell wrapping, 
 high mitotic count, 
 and abundant apoptotic bodies” [19] MEDULLOBLASTOMA TUMORS [19] 38 [19] Childhood medulloblastoma: novel approaches to the classification of a heterogeneous disease, Ellison et al., Acta Neuropathol, 20(3):305-16, 2010
 Pleomorphism (cytology): variability in the size and shape of cells and/or their nuclei
• Proposed approach [20] • Anaplastic versus non-anaplastic • Compare/combine steerable Riesz texture models with 
 unsupervised topographic independent component analysis (TICA) MEDULLOBLASTOMA TUMOR CLASSIFICATION 39 [20] Combining Unsupervised Feature Learning and Riesz Wavelets for Histopathology Image Representation: Application to Identifying Anaplastic Medulloblastoma, Otálora et al., Med Image Comput Comput Assist Interv (MICCAI), 2015
 The workflow of the proposed approach is summarized in Fig. 2. As first step, we ompute the UFL features learned by TICA and the supervised features learned with Riesz wavelets for each image as described in Sections 2.1 and 2.2. Once TICA and Riesz wavelets are computed a final step of supervised classification is made using the combination of the computed features in a concatenated vector as input for a standard softmax classifier as described in Section 2.3. Parameter uning is presented in Section 3.2. [2.4] Medulloblastoma Image Cases Fig. 2. Flowchart for MB feature extraction and classification for both learned repre- entations: Riesz and TICA, the details of each stage are described in subsections. steerable Riesz texture models topographic ICA
• Unsupervised topographic independent component analysis (TICA) [21] • Learn while minimizing cost function : MEDULLOBLASTOMA TUMOR CLASSIFICATION 40 [21] Energy correlations and topographic organization, Hyvärinen et al., Natural Image Statistics, 39:249-272, 2009 TICA is an unsupervised feature learning model, inspired by findings of the visual cortex behaviour. It groups activations of units in order to discover features that are rotation and translation invariant [1]. These are appropriate features for histopathology image characterization since shapes and cell organizations can be present regardless of the position or orientation of cells. Particularly, TICA organizes feature detectors in a square matrix for l groups such that adjacent feature detectors activate in a similar proportion to the same stimulus. To learn such groups, we need to optimize the cost function: JTICA(W) = 2 TX i=1 WT Wx(i) x(i) 2 2 + mX i=1 lX k=1 » Hk(Wx(i))2 + ✏ (1) where x(i) 2 Rm is the i-th sample, T is the number of samples, W 2 Rn⇥m is the matrix that encodes the features in each row, and H 2 {0, 1}l⇥n is the binary topographic organization where H (j) k = 1, if the j-th feature detector, j-th row of W , belongs to the k-th group, and 0 otherwise. This model sets H fixed while learning W. In addition, TICA has two main computational advantages. First, the only parameters to be tuned are the regularization hyperparameter and the sparsity controller ✏. Second, it is an unconstrained optimization problem, which can be solved e ciently by optimization techniques such as Limited memory- Broyden-Fletcher-Goldfarb-Shanno (L-BFGS). iriiiiéiiiiiiiiiiiiosiiióiiiiïiiiiiii1=11.e1111MMIIIM 111111111111111M 11=1 MMMMMMMMMMMMMMMMMMM iiiMUMMMiMMMiiiiiii iíiíi iiiíiii ii UWE:lMiiM ME= M MEE is i iiiiíiiiiMóóóii iMMMMMlMiMMMMMMMiMMMiMMMMMMMMMiiMMiMMiMMM iMMMiMMMiiMMMMiiiiiiiMiMMiMiMiiMMiiMiiiiiiiiMM JTICA(W ) reconstruction penaltyL2 topographic constraint: 
 if the patch belongs 
 to the local group of patches Hk = 1 x(i) k ~scale inv. ~rotational inv. ~translational inv. W
• Dataset: • 5 anaplastic, 5 non-anaplastic patients • 750 patches (200x200) / patient • Leave-2-patients-out cross-validation • Softmax classifier • Comparison with state-of-the-art [22] • Convolutional neural networks (CNN), sparse autoencoders (sAE),
 bag of features (BOF), Haar, MR8, … • Predictions over entire-slides: MEDULLOBLASTOMA TUMOR CLASSIFICATION 41 [22] A comparative evaluation of supervised and unsupervised representation learning approaches for anaplastic medulloblastoma differentiation, Cruz-Roa et al., Proc. SPIE 9287, Int Symp Med Inf Proc Anal, 92870G, 2015
 Medulloblastoma Di↵erentiat Table 1. MB classification performance ( averaged over the 20 test runs with standa Method Accuracy Se TICA + Riesz[N1 3 , N2 2 , N2 1 ] 0.997 ± 0.002 0.99 TICA [1] 0.972 ± 0.018 0.97 Riesz [N1 3 , N2 2 , N2 1 ] [3] 0.964 ± 0.038 0.99 Riesz [N1 3 ] [3] 0.958 ± 0.062 0.9 Riesz [N2 2 ] [3] 0.94 ± 0.02 0.9 2-Layer CNN [1] 0.90 ± 0.1 0.8 sAE [1] 0.90 BOF + A2NMF (Haar) [2] 0.87 Riesz [N2 1 ] [3] 0.85 ± 0.23 0. BOF + K - NN (Haar) [7] 0.80 BOF + K - NN (MR8) [7] 0.62 4 Concluding Remarks We present a feature fusion between u vised Riesz wavelet representation that [22] [22] [22] [22] [22] TICA [1] 0.972 ± 0.018 0.977 ± 0.021 0.967 ± 0.031 Riesz [N1 3 , N2 2 , N2 1 ] [3] 0.964 ± 0.038 0.999 ± 0.001 0.932 ± 0.07 Riesz [N1 3 ] [3] 0.958 ± 0.062 0.963 ± 0.05 0.916 ± 0.125 Riesz [N2 2 ] [3] 0.94 ± 0.02 0.94 ± 0.02 0.3 ± 0.04 2-Layer CNN [1] 0.90 ± 0.1 0.89 ± 0.18 0.9 ± 0.0.3 sAE [1] 0.90 0.87 0.93 BOF + A2NMF (Haar) [2] 0.87 0.86 0.87 Riesz [N2 1 ] [3] 0.85 ± 0.23 0.9 ± 0.15 0.7 ± 0.47 BOF + K - NN (Haar) [7] 0.80 - - BOF + K - NN (MR8) [7] 0.62 - - 4 Concluding Remarks We present a feature fusion between unsupervised feature learning and super- vised Riesz wavelet representation that captures subtle pattern of textures as well as high level features, allowing to create a more separable feature space where the di↵erentiation of medulloblastoma into anaplastic and non-anaplastic can be made with high classification accuracy outperforming any other result previously described in the literature. To our knowledge this is the first time that a feature fusion method is presented between UFL and the Riesz wavelets in the context of histopathology image analysis showing the complementarity between these learned features for the challenging task of tumour di↵erentia- tion, we are currently working on extending the method to other patch-based histopathology image analysis problems with larger cohorts of patients. Fig. 3. Predictions over two WSIs, non-anaplastic MB (left) and anaplastic (right). non-anaplastic anaplastic TICA: unsupervised
 Riesz: rotation-covariant information Small dataset
• Highly adaptive texture-based computational models of biomedical tissue: ✓ Complete/systematic coverage of image 
 scales and directions in 2-D and 3-D ✓ Rotation-covariance ✓ Specificity: the models can be trained to 
 characterize specific tissue types ✓ Local characterization of tissue properties • Locate tissue properties in organ anatomy 
 to create digital phenotypes • localization systems, 
 quant. graph analysis CONCLUSIONS model learning Multi-resolution, steerability patch-based analysis 42 be averaged over the folds of the CV and used to build texture models for each location along dROI (see Fig 9). The sum of the weights for all channels from each location will reveal the subregions that are specific to each tumor subtypes, and most related to patient survival. A K–means clustering of the vectors w will be carried out for each locations to evaluate the stability of the regional models over the folds of the CV and define homogeneous groups among patients. A selection of the models based on stability and location importance will be carried out. Fig. 9 Prototype tissue archi- tecture of a GBM tumor [128]. In a second step, the selected models will be locally steered to maximize their magnitude. The energies of the maximal magnitudes will be used to construct a final feature space for (1) predicting the tumor subtypes and (2) performing Kaplan–Meier survival analysis. The performance of the proposed approach for predicting tumor subtypes and patient survival will be compared to (1) unaligned wavelet energies and (2) average wavelet energies over the entire tumor. While starting with a LOPO CV on the TCGA–TCIA dataset, the generalizability of the approach will be further assessed by training with the TCGA–TCIA and testing with the SU dataset. Deliverable 2.1: Predicting tumor subtype and survival from localized of tissue properties in GBM tumors. Task 2.2: Digital lung tissue atlases of ILD diagnoses (8 months of the PI) In this task, we will use a simple atlas of the lungs to locate texture properties and create prototype diagnosis phenotypes of ILDs. In previous work, we developed a simple 3–D atlas of the lungs with 36 subregions that will be used in this task [42]. In a first step, diagnosis–wise digital tissue atlases will be created by learning 3–D texture models (i.e., the average of w over the folds of the LOPO CV) for each 36 regions of the lungs (see Fig. 10). The regions for which the models are most distant11 from all other diagnoses will be highlighted and compared to previously built models of tissue patterns [25] to create 3–D prototype tissue atlases for each diagnosis. The obtained results will be validated using medical knowledge (e.g., Table 1 of [42]). Fig. 10 Tissue atlas of the lungs. For each diagnosis, K–means clustering of the vectors w will be carried out for each locations to evaluate the stability of the regional models over the folds of the CV and reveal homogeneous groups among patients (e.g., subtypes of UIP). The most stable models will be kept for the further characterization of lung tissue types. A hierarchical clustering of the models from all diagnoses will be carried out to define a radiomics–based hierarchy of all diagnoses, which will be compared to medical knowledge [140] (e.g., Fig. 1 of [40]). A large feature space including the energies of the steered models from each of the 36 localizations will be used to predict the diagnoses with uncertainty assessment (using e.g., pairwise coupling [141]). When a minimum amount of trust is not achieved when predicting a given diagnosis, the parent group of ILD diagnoses in the previously built hierarchy will be predicted instead [19]. Deliverable 2.2: Digital tissue atlases of ILD diagnoses and their subtypes. Task 2.3: Digital tissue atlases of ILDs: correlation with PFTs and survival (6 months )
CONCLUSIONS • Importance of rotation-covariant information to model biomedical textures • i.e., the local organization of directions • Rotational invariance is not enough • Called “roto-translation invariance” in [23] TextureQbased'biomarkers:'current'limitaGons' x  Assume'homogeneous'texture'properGes'over'the' enGre'lesion'[5]' ' x  NonQspecific'features' x  Global'vs'local'characterizaGon'of'image'direcGons'[6]' REVIEW: Quantitative Imaging in Cancer Evolution and Ecology Gatenby et al with the mean signal value. By using just two sequences, a contrast-enhanced T1 sequence and a fluid-attenuated inver- sion-recovery sequence, we can define four habitats: high or low postgadolini- um T1 divided into high or low fluid-at- tenuated inversion recovery. When these voxel habitats are projected into the tu- mor volume, we find they cluster into spatially distinct regions. These habitats can be evaluated both in terms of their relative contributions to the total tumor volume and in terms of their interactions with each other, based on the imaging characteristics at the interfaces between regions. Similar spatially explicit analysis can be performed with CT scans (Fig 5). Analysis of spatial patterns in cross-sectional images will ultimately re- quire methods that bridge spatial scales from microns to millimeters. One possi- ble method is a general class of numeric tools that is already widely used in ter- restrial and marine ecology research to link species occurrence or abundance with environmental parameters. Species distribution models (48–51) are used to gain ecologic and evolutionary insights and to predict distributions of species or morphs across landscapes, sometimes extrapolating in space and time. They can easily be used to link the environ- mental selection forces in MR imaging- defined habitats to the evolutionary dy- namics of cancer cells. Summary Imaging can have an enormous role in the development and implementation of patient-specific therapies in cancer. The achievement of this goal will require new methods that expand and ultimately re- place the current subjective qualitative assessments of tumor characteristics. rise to local-regional phenotypic adap- tations. Phenotypic alterations can re- sult from epigenetic, genetic, or chro- mosomal rearrangements, and these in turn will affect prognosis and response to therapy. Changes in habitats or the relative abundance of specific ecologic communities over time and in response to therapy may be a valuable metric with which to measure treatment efficacy and emergence of resistant populations. Emerging Strategies for Tumor Habitat Characterization A method for converting images to spa- tially explicit tumor habitats is shown in Figure 4. Here, three-dimensional MR microenvironment can be rewarded by increased proliferation. This evolution- ary dynamic may contribute to distinct differences between the tumor edges and the tumor cores, which frequently can be seen at analysis of cross-sec- tional images (Fig 5). Interpretation of the subsegmenta- tion of tumors will require computa- tional models to understand and predict the complex nonlinear dynamics that lead to heterogeneous combinations of radiographic features. We have ex- ploited ecologic methods and models to investigate regional variations in cancer environmental and cellular properties that lead to specific imaging character- istics. Conceptually, this approach as- Figure 4 Figure 4: Left: Contrast-enhanced T1 image from subject TCGA-02-0034 in The Cancer Genome Atlas–Glioblastoma Multiforme repository of MR volumes of glioblastoma multiforme cases. Right: Spatial distribution of MR imaging–defined habitats within the tumor. The blue region (low T1 postgadolinium, low fluid-attenuated inversion recovery) is particularly notable because it presumably represents a habitat with low blood flow but high cell density, indicating a population presumably adapted to hypoxic acidic conditions. rise to local-regional phenotypic adap- tations. Phenotypic alterations can re- sult from epigenetic, genetic, or chro- mosomal rearrangements, and these in turn will affect prognosis and response to therapy. Changes in habitats or the relative abundance of specific ecologic communities over time and in response to therapy may be a valuable metric with which to measure treatment efficacy and emergence of resistant populations. Emerging Strategies for Tumor Habitat Characterization A method for converting images to spa- tially explicit tumor habitats is shown in microenvironment can be rewarded by increased proliferation. This evolution- ary dynamic may contribute to distinct differences between the tumor edges and the tumor cores, which frequently can be seen at analysis of cross-sec- tional images (Fig 5). Interpretation of the subsegmenta- tion of tumors will require computa- tional models to understand and predict the complex nonlinear dynamics that lead to heterogeneous combinations of radiographic features. We have ex- ploited ecologic methods and models to investigate regional variations in cancer environmental and cellular properties that lead to specific imaging character- Figure 4: Left: Contrast-enhanced T1 image from subject TCGA-02-0034 in The Cancer Genome Atlas–Glioblastoma Multiforme repository of MR volumes of glioblastoma multiforme cases. Right: Spatial distribution of MR imaging–defined habitats within the tumor. The blue region (low T1 postgadolinium, low fluid-attenuated inversion recovery) is particularly notable because it presumably represents a habitat with low blood flow but high cell density, indicating a population presumably adapted to hypoxic acidic conditions. [5]'QuanGtaGve'imaging'in'cancer'evoluGon'and'ecology,'Gatenby'et'al.,'Radiology,'269(1):8Q15,'2013' 5' global'direcGonal'operators:' local'grouped'steering:' [6]'RotaGonQcovariant'texture'learning'using'steerable'Riesz'wavelets,'Depeursinge'et'al.,'IEEE'Trans'Imag'Proc.,'23(2):898Q908,'2014.' global directional operators local grouped steering 43 [11] Rotation-covariant texture learning using steerable Riesz wavelets, Depeursinge et al., IEEE Trans Imag Proc, 23(2):898-908, 2014
 [23] Rotation, Scaling and Deformation Invariant Scattering for Texture Discrimination, Sifre et al., IEEE Conf Comp Vis and Pat Rec (CVPR), 1233-40, 2013 with a second operator R2 which is invariant to the action of G2. Indeed for all g1.g2 ∈ G1 G2 and all images x(u): R2(R1(g1.g2.x)) = R2(g2.R1(x)) = R2(R1(x)). However, such separable invariants do not capture the joint property of the action of G2 relatively to G1, and may lose important information. This is why two-dimensional trans- lation invariant representations are not computed by cascad- ing invariants to horizontal and vertical translations. It is also important for rotations and translations. Let us consider for example the two texture patches of Figure 1. A separa- ble product of translation and rotation invariant operators can represent the relative positions of the vertical patterns, and the relative positions of the horizontal patterns, up to global translations. However, it can not represent the po- sitions of horizontal patterns relatively to vertical patterns, because it is not sensitive to a relative shift between these two sets of oriented structures. It loses the relative positions of different orientations, which is needed to be sensitive to curvature, crossings and corners. Such a separable invariant thus can not discriminate the two textures of Figure 1. Figure 1: The left and right textures are not discriminated by a separable invariant along rotations and translations, but can be discriminated by a joint roto-translation invariant. Several authors [6, 7, 8] have proposed to take into ac- count the joint structure of roto-translation operators in im- age processing, particularly to implement diffusion oper- ators. Computing a joint invariant between rotations and translations also means taking into account the joint rela- tive positions and orientations of image structures, so that the textures of Figure 1 can be discriminated. Section 3 introduces a roto-translation scattering operator, which is computed by cascading wavelet transforms on the roto- translation group. Calculating joint invariants on large non-commutative groups may however become very complex. Keeping a sep- information. 2.2. Hierarchical Architecture We now explain how to build an affine invariant repre- sentation, with a hierarchical architecture. We separate vari- abilities of potentially large amplitudes such as translations, rotations and scaling, from smaller amplitude variabilities, but which may belong to much higher dimensional groups such as shearing and general diffeomorphisms. These small amplitude deformations are linearized to remove them with linear projectors. Image variabilities typically differ over domains of dif- ferent sizes. Most image representations build localized in- variants over small image patches, for example with SIFT descriptors [15]. These invariant coefficients are then ag- gregated into more invariant global image descriptors, for example with bag of words [10] or multiple layers of deep neural network [4, 5]. We follow a similar strategy by first computing invariants over image patches and then aggregat- ing them at the global image scale. This is illustrated by the computational architecture of Figure 2. x roto-trans. patch scattering log global space-scale averaging deformat. invariant linear proj. Figure 2: An affine invariant scattering is computed by ap- plying a roto-translation scattering on image patches, a log- arithmic non-linearity and a global space-scale averaging. Invariants to small shearing and deformations are computed with linear projectors optimized by a supervised classifier. Within image patches, as previously explained, one must keep the joint information between positions and orienta- tions. This is done by calculating a scattering invariant on the joint roto-translation group. Scaling invariance is then implemented with a global scale-space averaging between patches, described in Section 4. A logarithmic non-linearity is first applied to invariant scattering coefficients to linearize their power law behavior across scales. This is similar to the normalization strategies used by bag of words [10] and deep neural networks [5]. Because of three dimensional surface curvature in the vi- sual scene, the image patches are also deformed. A scat- tering transform was proved to be stable to deformations [23] [11]
• Rotation-covariance • Local orientation of the models is computationally intensive • 2D: • 3D: • Use graphics processor units (GPUs) [24]: 60x speedup • Explore other steerable wavelet representations [25,26] LIMITATIONS AND FUTURE WORK 44 G ⇤ R(2,0,0) G ⇤ R(0,2,0) G ⇤ R(0,0,2) G ⇤ R(1,1,0) G ⇤ R(1,0,1) G ⇤ R(0,1,1) Fig. 1. Second–order Riesz kernels R(n1,n2,n3) convolved with isotropic Gaussian kernels G(x). Support vector machines (SVM) are then used to classify between 9,347 normal and embolic cubic instances of lung parenchyma from 19 patients with APE and 8 control cases. 2. MATERIAL AND METHODS 2.1. Rotation–covariant texture analysis 3D multiscale Riesz filterbanks are used to characterize the texture of the lung parenchyma in 3D at a given CT energy level. The N–th order Riesz transform R(N) of a three– dimensional signal f(x) is defined in the Fourier domain as: ¤R(n1,n2,n3)f(!) = … n1 + n2 + n3 n1!n2!n3! ( j!1)n1 ( j!2)n2 ( j!3)n3 ||!||n1+n2+n3 ˆf(!), (1) Hospitals of Geneva with an inter–slice distance of 1mm, a slice thickness of 1.25mm, and a sub–millimetric resolution in the axial plane. 11 energy levels are used from 40keV to 140keV with a step of 10keV. All five lobes of each patient have been manually segmented using the OsiriX software2 . The perfusion levels of each lobe were quantified using the Qanadli index (QI) on a lobe basis [14]. The QI is defined as the sum of the scores of all arteries as: 0 if no occlusion is visible, 1 if partially occluded, and 2 if totally obstructed. DECT data of all energy levels are preprocessed to have an isotropic voxel resolution, which is obtained by dividing samples along the z axis. All lobes are divided into 323 overlapping blocks to constitute a local instance of the lung parenchyma. A block is considered as valid when at least 95% of its voxels belong to it. To obtain a sufficient number of blocks for the middle right lobe, this rule was changed to IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2013 10 RN f✓ (x) = 0 B B B B B B @ R(0,N) f✓ (x) ... R(n,N n) f✓ (x) ... R(N,0) f✓ (x) 1 C C C C C C A | {z } RN {f✓}(x) = 0 B B B B B B @ A✓ 0,0 . . . A✓ 0,l . . . A✓ 0,N ... ... ... A✓ n,1 . . . A✓ n,l . . . A✓ n,N ... ... ... A✓ N,1 . . . A✓ N,l . . . A✓ N,N 1 C C C C C C A | {z } A✓ 0 B B B B B B @ R(0,N) {f} (x) ... R(l,N l) {f} (x) ... R(N,0) {f} (x) 1 C C C C C C A | {z } RN {f}(x) . (12) ˆR(n,N n) =   N n!(N n)! ( j(cos(✓)!1 + sin(✓)!2))n ( j( sin(✓)!1 + cos(✓)!2))N n ||!||N = 1 ||!||N   N n!(N n)! nX k1=0 Ç n k1 å (cos(✓))k1 ( j!1)k1 (sin(✓))n k1 ( j!2)n k1 N nX k2=0 Ç N n k2 å (cos(✓))k2 ( j!1)k2 (sin(✓))N n k2 ( j!2)N n k2 = 1 ||!||N   N n!(N n)! nX k1=0 N nX k2=0 ( j!1)k1+k2 ( j!2)N k1 k2 ( 1)k2 Ç n k1 åÇ N n k2 å (cos(✓))N n k2+k1 (sin(✓))n k1+k2 = nX k1=0 N nX k2=0   N (k1 + k2)!(N k1 k2)! ( j!1)k1+k2 ( j!2)N k1 k2 ||!||N | {z } ˆR(k1+k2,N k1 k2)   (k1 + k2)!(N k1 k2)! n!(N n)! ( 1)k2 n! k1!(n k1)! (N n)! k2!(N n k2)! (cos(✓))N n k2+k1 (sin(✓))n k1+k2 . (13) ˆR(n,N n) (!) = NX l=0   N l!(N l)! ( j!1)l ( j!2)N l ||!||N | {z } ˆR(l,N l) min(l,n) X l1=max(0,l N+n) ( 1)l l1   l!(N l)! l1!(n l1)!(l l1)! (cos(✓))N n+2l1 l (sin(✓))n 2l1+l | {z } A✓ n,l . (14) [32] T. Leung and J. Malik. Representing and recognizing the visual appearance of materials using three–dimensional textons. International Journal of Computer Vision, 43(1):29–44, 2001. invariant texture classification with local binary patterns. In Computer Vision — ECCV 2000, volume 1842 of Lecture Notes in Computer Science, pages 404–420. Springer Berlin Heidelberg, 2000. [24] GPU-accelerated texture analysis using steerable Riesz wavelets, Vizitiu et al., 11th Int Conf Par Proc and App Math (PPAM), 2015 (submitted)
 [25] A unifying parametric framework for 2D steerable wavelet transforms, Unser et al., SIAM Jour Imag Sci, 6(1):102-35, 2013
 [26] Harmonic Singular Integrals and Steerable Wavelets in , Ward et al., App and Comp Harm Anal, 36(2):183-197, 2014
• Multi-scale • Influence of surrounding objects: bandlimitedness VS compact support [27] • Continuous band-limited scale characterization [28] • Dyadic is not enough! LIMITATIONS AND FUTURE WORK 45 Contact and more information: adrien.depeursinge@epfl.ch, http://bigww References [1] An Official ATS/ERS/JRS/ALAT Statement: Idiopathic Pulmonary Fibrosis: Evidence-based Guidelines for Diagnosis and Management, G. Raghu et al., Am J Respir Crit Care Med 2011; 183(6):788-824 [2] VOW: Variance Optimal Wavelets for the Steerable Pyramid, P. Pad et al., IEEE ICIP 2014; 2973-2977 [3] 3D Steerable Wavelets and Monogenic Analysis for Bioimaging, N. Chenouard et al., IEEE ISBI 2011; 2132-2135 [4] A Parametric Texture Model Based on Joint Statistics of Complex Wavelet Coefficients, J. Portilla et al., Int Jour Comput Vision 2000; 1: 49-70 [5] Nonseparable radial frame multiresolution analysis in multidimensions and isotropic fast wavelet algorithms, M. Papadakis et al., SPIE Wavelets 2003; 5207: 631-642 [6] Ten Lectures on Wavelets, I. Daubechies, SIAM 1992; 61 •  bandwidth limited to •  generates tight frames •  Analytical approximation in Fourier: Results •  The proposed texture AUC and ACC for isotropic wavelet pyra Conclusions and •  New family of 3D isot bandwidth to balance operators and the infl •  Importance of rotation wavelet bandwidth de •  Future work includes higher orders of the R . . . . . . . . . . . . [27] Optimized steerable wavelets for texture analysis of lung tissue in 3-D CT: classification of usual interstitial pneumonia, Depeursinge et al., IEEE Int Symp on Biomed Imag (ISBI), 403-6, 2015
 [28] Fast detection and refined scale estimation using complex isotropic wavelets, Püspöki et al., IEEE Int Symp on Biomed Imag (ISBI), 512-5, 2015 spatial domain Fourier
(a) Synthetic image containing 3 visual concepts: 1) vertical lines (quadrants I and III), 2) checkerboard (quadrant II), 3) wiggled checkerboard (quadrant IV). PCA 1 PCA2 10 1 10 2 10 3 (b) PCA visualization of 32⇥32 overlapping blocks and clus- ters from the left image (N = 10, J = 4, K = 3). The tem- plates 10 k corresponding to the respective visual concepts are dislayed for scale j = 3. 10 • Model learning • Limited performance for stochastic textures with no clear multi-scale signature • Reveal visual diversity with unsupervised learning [29,30] LIMITATIONS AND FUTURE WORK 46 [29] Rotation-covariant visual concept detection using steerable Riesz wavelets and bags of visual words, Depeursinge et al., SPIE Wavelets and Sparsity XV, 8858:885816-885816-11, 2013
 [30] Unsupervised texture segmentation using monogenic curvelets and the Potts model, Storath et al., IEEE Int Conf Imag Proc, 4348-52, 2014 (a) Synthetic image containing 3 visual concepts: 1) vertical lines (quadrants I and III), 2) checkerboard (quadrant II), 3) wiggled checkerboard (quadrant IV). PCA 1 PCA2 10 1 10 2 10 3 (b) PCA visualization of 32⇥32 overlapping blocks and clus- ters from the left image (N = 10, J = 4, K = 3). The tem- plates 10 k corresponding to the respective visual concepts are dislayed for scale j = 3. Figure 5: Qualitative evaluation of the visual concepts 10 k found using K–means in the feature space spanned 20135 nvas0055)canvas0066)canvas0097)canvas0118)canvas021 anvas02613)canvas03114)canvas03215)canvas03316)canvas035 ile00621)carpet00222)carpet00423)carpet00524)carpet009 atabase.
• THANKS ! 47 Matlab code available! adrien.depeursinge@epfl.ch MICCAI tutorial on Biomedical 
 Texture Analysis: Oct 5th in Münich
 https://sites.google.com/site/btamiccai2015/
BIOMEDICAL TISSUE MODELING IN 2D AND 3D • Interstitial lung diseases in CT • Lung texture classification using locally-oriented Riesz components, Depeursinge A, Foncubierta- Rodriguez A, Van de Ville D, Müller H, Med Image Comput Comput Assist Interv. (MICCAI) 2011;14(3):231-8. • Multiscale lung texture signature learning using the Riesz transform, Depeursinge A, Foncubierta- Rodriguez A, Van de Ville D, Müller H, Med Image Comput Comput Assist Interv. (MICCAI) 2012;15(3):517-24. • Automated classification of usual interstitial pneumonia using regional volumetric texture analysis in high-resolution CT, Depeursinge A, Chin A, Leung A, Terrone D, Bristow M, Rosen G, Rubin D, Invest Radiol.,
 in press. • Pulmonary embolism in dual-energy CT • Rotation-covariant texture analysis of 4D dual-energy CT as an indicator of local pulmonary perfusion, Depeursinge A, Foncubierta-Rodriguez A, Vargas A, Van de Ville D, Platon A, Poletti PA, 48
BIOMEDICAL TISSUE MODELING IN 2D AND 3D • Liver lesions in CT • Predicting visual semantic descriptive terms from radiological image data: preliminary results with liver lesions in CT, Depeursinge A, Kurtz C, Beaulieu C, Napel S, Rubin D, IEEE Trans Med Imag. 2014;33(8):1669-76. • Brain epileptogenic lesions in MRI • Epileptogenic lesion quantification in MRI using contralateral 3D texture comparisons, Jiménez del Toro OA, Foncubierta-Rodríguez A, Vargas Gómez MI, Müller H, Depeursinge A, Med Image Comput Comput Assist Interv. (MICCAI) 2013;16(2):353-60. 49
EVEN VS ODD ORDERS: 1-D signal: Heaviside filter: 1st (dashed) and 2nd order Gaussian derivatives convolution: 1st (dashed) 
 versus 2nd 50

More Related Content

PDF
3D Riesz-wavelet Based Covariance Descriptors for Texture Classi cation of Lu...
Institute of Information Systems (HES-SO)
 
PDF
FUNDAMENTALS OF TEXTURE PROCESSING FOR BIOMEDICAL IMAGE ANALYSIS
Institute of Information Systems (HES-SO)
 
PDF
Wavelet based histogram method for classification of textu
IAEME Publication
 
PDF
Hybrid Technique Based on N-GRAM and Neural Networks for Classification of Ma...
csandit
 
PDF
Survey on Brain MRI Segmentation Techniques
Editor IJMTER
 
PDF
International Journal of Engineering Research and Development
IJERD Editor
 
PDF
A Novel Multiple-kernel based Fuzzy c-means Algorithm with Spatial Informatio...
CSCJournals
 
PDF
A New Approach for Segmentation of Fused Images using Cluster based Thresholding
IDES Editor
 
3D Riesz-wavelet Based Covariance Descriptors for Texture Classi cation of Lu...
Institute of Information Systems (HES-SO)
 
FUNDAMENTALS OF TEXTURE PROCESSING FOR BIOMEDICAL IMAGE ANALYSIS
Institute of Information Systems (HES-SO)
 
Wavelet based histogram method for classification of textu
IAEME Publication
 
Hybrid Technique Based on N-GRAM and Neural Networks for Classification of Ma...
csandit
 
Survey on Brain MRI Segmentation Techniques
Editor IJMTER
 
International Journal of Engineering Research and Development
IJERD Editor
 
A Novel Multiple-kernel based Fuzzy c-means Algorithm with Spatial Informatio...
CSCJournals
 
A New Approach for Segmentation of Fused Images using Cluster based Thresholding
IDES Editor
 

What's hot (20)

PDF
ROLE OF HYBRID LEVEL SET IN FETAL CONTOUR EXTRACTION
sipij
 
PDF
Dual Tree Complex Wavelet Transform, Probabilistic Neural Network and Fuzzy C...
IJAEMSJORNAL
 
PDF
A NOVEL PROBABILISTIC BASED IMAGE SEGMENTATION MODEL FOR REALTIME HUMAN ACTIV...
sipij
 
PDF
Image Compression based on DCT and BPSO for MRI and Standard Images
IJERA Editor
 
PDF
National Flags Recognition Based on Principal Component Analysis
ijtsrd
 
PDF
Brain Tumor Extraction from T1- Weighted MRI using Co-clustering and Level Se...
CSCJournals
 
PDF
Corner Detection Using Mutual Information
CSCJournals
 
PDF
Massive Regional Texture Extraction for Aerial and Natural Images
IOSR Journals
 
PDF
L045066671
IJERA Editor
 
PPTX
Transfer Learning of Tissue Photon Interaction in Optical Coherence Tomograph...
Debdoot Sheet
 
PDF
A Survey OF Image Registration
CSCJournals
 
PDF
Early Detection of Cancerous Lung Nodules from Computed Tomography Images
CSCJournals
 
PDF
Image Registration for Recovering Affine Transformation Using Nelder Mead Sim...
CSCJournals
 
PDF
Microarray spot partitioning by autonomously organising maps through contour ...
IJECEIAES
 
PDF
A Novel Method for Detection of Architectural Distortion in Mammogram
IDES Editor
 
PDF
MAGNETIC RESONANCE BRAIN IMAGE SEGMENTATION
VLSICS Design
 
PDF
Ea4301770773
IJERA Editor
 
PDF
Multi modal medical image fusion using weighted
eSAT Publishing House
 
PDF
Lec8: Medical Image Segmentation (II) (Region Growing/Merging)
Ulaş Bağcı
 
PDF
FUZZY SEGMENTATION OF MRI CEREBRAL TISSUE USING LEVEL SET ALGORITHM
AM Publications
 
ROLE OF HYBRID LEVEL SET IN FETAL CONTOUR EXTRACTION
sipij
 
Dual Tree Complex Wavelet Transform, Probabilistic Neural Network and Fuzzy C...
IJAEMSJORNAL
 
A NOVEL PROBABILISTIC BASED IMAGE SEGMENTATION MODEL FOR REALTIME HUMAN ACTIV...
sipij
 
Image Compression based on DCT and BPSO for MRI and Standard Images
IJERA Editor
 
National Flags Recognition Based on Principal Component Analysis
ijtsrd
 
Brain Tumor Extraction from T1- Weighted MRI using Co-clustering and Level Se...
CSCJournals
 
Corner Detection Using Mutual Information
CSCJournals
 
Massive Regional Texture Extraction for Aerial and Natural Images
IOSR Journals
 
L045066671
IJERA Editor
 
Transfer Learning of Tissue Photon Interaction in Optical Coherence Tomograph...
Debdoot Sheet
 
A Survey OF Image Registration
CSCJournals
 
Early Detection of Cancerous Lung Nodules from Computed Tomography Images
CSCJournals
 
Image Registration for Recovering Affine Transformation Using Nelder Mead Sim...
CSCJournals
 
Microarray spot partitioning by autonomously organising maps through contour ...
IJECEIAES
 
A Novel Method for Detection of Architectural Distortion in Mammogram
IDES Editor
 
MAGNETIC RESONANCE BRAIN IMAGE SEGMENTATION
VLSICS Design
 
Ea4301770773
IJERA Editor
 
Multi modal medical image fusion using weighted
eSAT Publishing House
 
Lec8: Medical Image Segmentation (II) (Region Growing/Merging)
Ulaş Bağcı
 
FUZZY SEGMENTATION OF MRI CEREBRAL TISSUE USING LEVEL SET ALGORITHM
AM Publications
 
Ad

Similar to Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology (20)

PPTX
Introduction to Machine Learning and Texture Analysis for Lesion Characteriza...
Kevin Mader
 
PPTX
How is machine learning significant to computational pathology in the pharmac...
Pubrica
 
PDF
How is machine learning significant to computational pathology in the pharmac...
Pubrica
 
PDF
Tools to Analyze Morphology and Spatially Mapped Molecular Data - Informatio...
Joel Saltz
 
PPTX
Intelligent computer aided diagnosis system for liver fibrosis
Aboul Ella Hassanien
 
PDF
Medical 3D data retrieval
Institute of Information Systems (HES-SO)
 
PDF
PERFORMANCE ANALYSIS OF TEXTURE IMAGE RETRIEVAL FOR CURVELET, CONTOURLET TRAN...
ijfcstjournal
 
PDF
PERFORMANCE ANALYSIS OF TEXTURE IMAGE RETRIEVAL FOR CURVELET, CONTOURLET TR...
ijfcstjournal
 
PDF
PERFORMANCE ANALYSIS OF TEXTURE IMAGE RETRIEVAL FOR CURVELET, CONTOURLET TRAN...
ijfcstjournal
 
PDF
PERFORMANCE ANALYSIS OF TEXTURE IMAGE RETRIEVAL FOR CURVELET, CONTOURLET TRAN...
ijfcstjournal
 
PDF
PERFORMANCE ANALYSIS OF TEXTURE IMAGE RETRIEVAL FOR CURVELET, CONTOURLET TRAN...
ijfcstjournal
 
PDF
IRJET- A Survey on Categorization of Breast Cancer in Histopathological Images
IRJET Journal
 
PPTX
Why Big Image Analytics is essential for Imaging Biobanks: An Osteoporosis Ra...
Kevin Mader
 
PPTX
Machine Learning in Pathology Diagnostics with Simagis Live
khvatkov
 
PDF
Images as Occlusions of Textures: A Framework for Segmentation
john236zaq
 
PDF
Definiens In Digital Pathology Hr
Daniel Nicolson
 
PPTX
Peter Hamilton on Next generation Imaging and Computer Vision in Pathology: p...
Cirdan
 
PPTX
High Dimensional Fused-Informatics
Joel Saltz
 
PDF
Assessment
Bathshebaparimala
 
PDF
A Wavelet Based Automatic Segmentation of Brain Tumor in CT Images Using Opti...
CSCJournals
 
Introduction to Machine Learning and Texture Analysis for Lesion Characteriza...
Kevin Mader
 
How is machine learning significant to computational pathology in the pharmac...
Pubrica
 
How is machine learning significant to computational pathology in the pharmac...
Pubrica
 
Tools to Analyze Morphology and Spatially Mapped Molecular Data - Informatio...
Joel Saltz
 
Intelligent computer aided diagnosis system for liver fibrosis
Aboul Ella Hassanien
 
Medical 3D data retrieval
Institute of Information Systems (HES-SO)
 
PERFORMANCE ANALYSIS OF TEXTURE IMAGE RETRIEVAL FOR CURVELET, CONTOURLET TRAN...
ijfcstjournal
 
PERFORMANCE ANALYSIS OF TEXTURE IMAGE RETRIEVAL FOR CURVELET, CONTOURLET TR...
ijfcstjournal
 
PERFORMANCE ANALYSIS OF TEXTURE IMAGE RETRIEVAL FOR CURVELET, CONTOURLET TRAN...
ijfcstjournal
 
PERFORMANCE ANALYSIS OF TEXTURE IMAGE RETRIEVAL FOR CURVELET, CONTOURLET TRAN...
ijfcstjournal
 
PERFORMANCE ANALYSIS OF TEXTURE IMAGE RETRIEVAL FOR CURVELET, CONTOURLET TRAN...
ijfcstjournal
 
IRJET- A Survey on Categorization of Breast Cancer in Histopathological Images
IRJET Journal
 
Why Big Image Analytics is essential for Imaging Biobanks: An Osteoporosis Ra...
Kevin Mader
 
Machine Learning in Pathology Diagnostics with Simagis Live
khvatkov
 
Images as Occlusions of Textures: A Framework for Segmentation
john236zaq
 
Definiens In Digital Pathology Hr
Daniel Nicolson
 
Peter Hamilton on Next generation Imaging and Computer Vision in Pathology: p...
Cirdan
 
High Dimensional Fused-Informatics
Joel Saltz
 
Assessment
Bathshebaparimala
 
A Wavelet Based Automatic Segmentation of Brain Tumor in CT Images Using Opti...
CSCJournals
 
Ad

More from Institute of Information Systems (HES-SO) (20)

PPTX
MIE20232.pptx
Institute of Information Systems (HES-SO)
 
PPTX
Classification of noisy free-text prostate cancer pathology reports using nat...
Institute of Information Systems (HES-SO)
 
PPTX
Machine learning assisted citation screening for Systematic Reviews - Anjani ...
Institute of Information Systems (HES-SO)
 
PPTX
Exploiting biomedical literature to mine out a large multimodal dataset of ra...
Institute of Information Systems (HES-SO)
 
PDF
L'IoT dans les usines. Quels avantages ?
Institute of Information Systems (HES-SO)
 
PDF
Studying Public Medical Images from Open Access Literature and Social Network...
Institute of Information Systems (HES-SO)
 
PPTX
Risques opérationnels et le système de contrôle interne : les limites d’un te...
Institute of Information Systems (HES-SO)
 
PDF
Le contrôle interne dans les administrations publiques tient-il toutes ses pr...
Institute of Information Systems (HES-SO)
 
PPTX
Le système de contrôle interne : Présentation générale, enjeux et méthodes
Institute of Information Systems (HES-SO)
 
PPTX
Crowdsourcing-based Mobile Application for Wheelchair Accessibility
Institute of Information Systems (HES-SO)
 
PDF
Quelle(s) valeur(s) pour le leadership stratégique ?
Institute of Information Systems (HES-SO)
 
PDF
A 3-D Riesz-Covariance Texture Model for the Prediction of Nodule Recurrence ...
Institute of Information Systems (HES-SO)
 
PDF
Challenges in medical imaging and the VISCERAL model
Institute of Information Systems (HES-SO)
 
PDF
NOSE: une approche Smart-City pour les zones périphériques et extra-urbaines
Institute of Information Systems (HES-SO)
 
PDF
Medical image analysis and big data evaluation infrastructures
Institute of Information Systems (HES-SO)
 
PDF
Medical image analysis, retrieval and evaluation infrastructures
Institute of Information Systems (HES-SO)
 
PDF
How to detect soft falls on devices
Institute of Information Systems (HES-SO)
 
PDF
MOBILE COLLECTION AND DISSEMINATION OF SENIORS’ SKILLS
Institute of Information Systems (HES-SO)
 
PDF
Enhanced Students Laboratory The GET project
Institute of Information Systems (HES-SO)
 
PDF
Solar production prediction based on non linear meteo source adaptation
Institute of Information Systems (HES-SO)
 
MIE20232.pptx
Institute of Information Systems (HES-SO)
 
Classification of noisy free-text prostate cancer pathology reports using nat...
Institute of Information Systems (HES-SO)
 
Machine learning assisted citation screening for Systematic Reviews - Anjani ...
Institute of Information Systems (HES-SO)
 
Exploiting biomedical literature to mine out a large multimodal dataset of ra...
Institute of Information Systems (HES-SO)
 
L'IoT dans les usines. Quels avantages ?
Institute of Information Systems (HES-SO)
 
Studying Public Medical Images from Open Access Literature and Social Network...
Institute of Information Systems (HES-SO)
 
Risques opérationnels et le système de contrôle interne : les limites d’un te...
Institute of Information Systems (HES-SO)
 
Le contrôle interne dans les administrations publiques tient-il toutes ses pr...
Institute of Information Systems (HES-SO)
 
Le système de contrôle interne : Présentation générale, enjeux et méthodes
Institute of Information Systems (HES-SO)
 
Crowdsourcing-based Mobile Application for Wheelchair Accessibility
Institute of Information Systems (HES-SO)
 
Quelle(s) valeur(s) pour le leadership stratégique ?
Institute of Information Systems (HES-SO)
 
A 3-D Riesz-Covariance Texture Model for the Prediction of Nodule Recurrence ...
Institute of Information Systems (HES-SO)
 
Challenges in medical imaging and the VISCERAL model
Institute of Information Systems (HES-SO)
 
NOSE: une approche Smart-City pour les zones périphériques et extra-urbaines
Institute of Information Systems (HES-SO)
 
Medical image analysis and big data evaluation infrastructures
Institute of Information Systems (HES-SO)
 
Medical image analysis, retrieval and evaluation infrastructures
Institute of Information Systems (HES-SO)
 
How to detect soft falls on devices
Institute of Information Systems (HES-SO)
 
MOBILE COLLECTION AND DISSEMINATION OF SENIORS’ SKILLS
Institute of Information Systems (HES-SO)
 
Enhanced Students Laboratory The GET project
Institute of Information Systems (HES-SO)
 
Solar production prediction based on non linear meteo source adaptation
Institute of Information Systems (HES-SO)
 

Recently uploaded (20)

PPTX
NEET PG 2025 Pharmacology Recall | Real Exam Questions from 3rd August with D...
Shivankan Kakkar
 
PDF
NEET PG 2025 | 200 High-Yield Recall Topics Across All Subjects
Shivankan Kakkar
 
PPTX
neonatal infection(7392992y282939y5.pptx
kanuni1
 
PPTX
SKIN Anatomy and physiology and associated diseases
ReshmaMurali14
 
PPTX
Imaging of parasitic D. Case Discussions.pptx
IbrahimAboAlasaad
 
PPTX
CME 2 Acute Chest Pain preentation for education
KhausalyaAndy
 
PPT
ASRH Presentation for students and teachers 2770633.ppt
alpha june
 
PPT
Obstructive sleep apnea in orthodontics treatment
IOPARGER
 
PPTX
Neuropathic pain.ppt treatment managment
BaharBazooyar
 
DOC
Adobe Premiere Pro CC Crack With Serial Key Full Free Download 2025
muhammadsharaz350
 
PPTX
Respiratory drugs, drugs acting on the respi system
scsbtzbq9b
 
PPT
1b - INTRODUCTION TO EPIDEMIOLOGY (comm med).ppt
YusuffDamilareAdewol
 
PPT
genitourinary-cancers_1.ppt Nursing care of clients with GU cancer
MohammedGazo
 
PPTX
Electromyography (EMG) in Physiotherapy: Principles, Procedure & Clinical App...
BALAJI SOMA
 
PPTX
Pathophysiology And Clinical Features Of Peripheral Nervous System .pptx
neuroptnagarajssmaka
 
PPTX
History and examination of abdomen, & pelvis .pptx
drmariaalrafi
 
PDF
Handout_ NURS 220 Topic 10-Abnormal Pregnancy.pdf
zuluuke
 
PDF
Oral Aspect of Metabolic Disease_20250717_192438_0000.pdf
pandeynitin400605
 
PPT
OPIOID ANALGESICS AND THEIR IMPLICATIONS
prasamspam
 
PPTX
ACID BASE management, base deficit correction
bhoomikagowda71
 
NEET PG 2025 Pharmacology Recall | Real Exam Questions from 3rd August with D...
Shivankan Kakkar
 
NEET PG 2025 | 200 High-Yield Recall Topics Across All Subjects
Shivankan Kakkar
 
neonatal infection(7392992y282939y5.pptx
kanuni1
 
SKIN Anatomy and physiology and associated diseases
ReshmaMurali14
 
Imaging of parasitic D. Case Discussions.pptx
IbrahimAboAlasaad
 
CME 2 Acute Chest Pain preentation for education
KhausalyaAndy
 
ASRH Presentation for students and teachers 2770633.ppt
alpha june
 
Obstructive sleep apnea in orthodontics treatment
IOPARGER
 
Neuropathic pain.ppt treatment managment
BaharBazooyar
 
Adobe Premiere Pro CC Crack With Serial Key Full Free Download 2025
muhammadsharaz350
 
Respiratory drugs, drugs acting on the respi system
scsbtzbq9b
 
1b - INTRODUCTION TO EPIDEMIOLOGY (comm med).ppt
YusuffDamilareAdewol
 
genitourinary-cancers_1.ppt Nursing care of clients with GU cancer
MohammedGazo
 
Electromyography (EMG) in Physiotherapy: Principles, Procedure & Clinical App...
BALAJI SOMA
 
Pathophysiology And Clinical Features Of Peripheral Nervous System .pptx
neuroptnagarajssmaka
 
History and examination of abdomen, & pelvis .pptx
drmariaalrafi
 
Handout_ NURS 220 Topic 10-Abnormal Pregnancy.pdf
zuluuke
 
Oral Aspect of Metabolic Disease_20250717_192438_0000.pdf
pandeynitin400605
 
OPIOID ANALGESICS AND THEIR IMPLICATIONS
prasamspam
 
ACID BASE management, base deficit correction
bhoomikagowda71
 

Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

  • 1. TEXTURE-BASED COMPUTATIONAL MODELS OF TISSUE IN BIOMEDICAL IMAGES: 
 INITIAL EXPERIENCE WITH DIGITAL HISTOPATHOLOGY Adrien Depeursinge, PhD Journées GdR ISIS “Analyse de Tissu Biologique et Histopathologie Numérique” Paris, June 23rd 2014
  • 2. BACKGROUND – RADIOMICS - HISTOPATHOLOMICS • Personalized medicine aims at enhancing the patient’s 
 quality of life and prognosis • Tailored treatment and medical decisions based 
 on the molecular composition of diseased tissue • Current limitations [1] • Molecular analysis of tissue composition 
 is invasive (biopsy), slow and costly • Cannot capture molecular heterogeneity 2 [1] Intratumor heterogeneity and branched evolution revealed by multiregion sequencing, Gerlinger et al., 
 N Engl J Med, 366(10):883-92, 2012. Intratumor Heterogeneity Reveale B Regional Distribution of Mutations A Biopsy Sites SOX9 CENPN PSMD7 RIMBP2 GALNT11 ABHD11 UGT2A1 MTOR PPP6R2 ZNF780A WSCD2 CDKN1B PPFIA1 TH SSNA1 CASP2 PLRG1 SETD2 CCBL2 SESN2 MAGEB16 NLRP7 IGLON5 KLK4 WDR62 KIAA0355 CYP4F3 AKAP8 ZNF519 DDX52 ZC3H18 TCF12 NUSAP1 X4 KDM2B MRPL51 C11orf68 ANO5 EIF4G2 MSRB2 RALGDS EXT1 ZC3HC1 PTPRZ1 INTS1 CCR6 DOPEY1 ATXN1 WHSC1 CLCN2 SSR3 KLHL18 SGOL1 VHL C2orf21 ALS2CR12 PLB1 FCAMR IFI16 BCAS2 IL12RB2 Ubiquitous Shared prima 10 cm R7 (G4) R5 (G4) R9 R3 (G4) R1 (G3) R2 (G3) R4 (G1) R6 (G1) Hilum R8 (G4) ution of Mutations ationships of Tumor Regions D Ploidy Profiling PreP PreM R1 R2 R3 R5 R8 R9 R4 M1 M2a M2b C2orf85 WDR7 SUPT6H CDH19 LAMA3 DIXDC1 HPS5 NRAP KIAA1524 SETD2 PLCL1 BCL11A IFNAR1 DAMTS10 C3 KIAA1267 RT4 CD44 ANKRD26 TM7SF4 SLC2A1 DACH2 MMAB ZNF521 HMG20A DNMT3A RLF MAMLD1 MAP3K6 HDAC6 PHF21B FAM129B RPS8 CIB2 RAB27A SLC2A12 DUSP12 ADAMTSL4 NAP1L3 USP51 KDM5C SBF1 TOM1 MYH8 WDR24 ITIH5 AKAP9 FBXO1 LIAS TNIK SETD2 C3orf20 MR1 PIAS3 DIO1 ERCC5 KL ALKBH8 DAPK1 DDX58 SPATA21 ZNF493 NGEF DIRAS3 LATS2 ITGB3 FLNA SATL1 KDM5C KDM5C RBFOX2 NPHS1 SOX9 CENPN PSMD7 RIMBP2 GALNT11 ABHD11 UGT2A1 MTOR PPP6R2 ZNF780A WSCD2 CDKN1B PPFIA1 TH SSNA1 CASP2 PLRG1 SETD2 CCBL2 SESN2 MAGEB16 NLRP7 IGLON5 KLK4 WDR62 KIAA0355 CYP4F3 AKAP8 ZNF519 DDX52 ZC3H18 TCF12 NUSAP1 X4 KDM2B MRPL51 C11orf68 ANO5 EIF4G2 MSRB2 RALGDS EXT1 ZC3HC1 PTPRZ1 Privatebiquitous Shared primary Shared metastasis Lung metastases Chest-wall metastasis Perinephric metastasis M1 10 cm R2 (G3) R4 (G1) R6 (G1) HilumR8 (G4) Primary tumor M2b M2a ution of Mutations PreP PreM R1 R2 R3 R5 R8 R9 R4 M1 M2a M2b C2orf85 WDR7 SUPT6H CDH19 LAMA3 DIXDC1 HPS5 NRAP KIAA1524 SETD2 PLCL1 BCL11A IFNAR1 DAMTS10 C3 KIAA1267 RT4 CD44 ANKRD26 TM7SF4 SLC2A1 DACH2 MMAB ZNF521 HMG20A DNMT3A RLF MAMLD1 MAP3K6 HDAC6 PHF21B FAM129B RPS8 CIB2 RAB27A SLC2A12 DUSP12 ADAMTSL4 NAP1L3 USP51 KDM5C SBF1 TOM1 MYH8 WDR24 ITIH5 AKAP9 FBXO1 LIAS TNIK SETD2 C3orf20 MR1 PIAS3 DIO1 ERCC5 KL ALKBH8 DAPK1 DDX58 SPATA21 ZNF493 NGEF DIRAS3 LATS2 ITGB3 FLNA SATL1 KDM5C KDM5C RBFOX2 NPHS1 SOX9 CENPN PSMD7 RIMBP2 GALNT11 ABHD11 UGT2A1 MTOR PPP6R2 ZNF780A WSCD2 CDKN1B PPFIA1 TH SSNA1 CASP2 PLRG1 SETD2 CCBL2 SESN2 MAGEB16 NLRP7 IGLON5 KLK4 WDR62 KIAA0355 CYP4F3 AKAP8 ZNF519 DDX52 ZC3H18 TCF12 NUSAP1 X4 KDM2B MRPL51 C11orf68 ANO5 EIF4G2 MSRB2 RALGDS EXT1 ZC3HC1 PTPRZ1 Privatebiquitous Shared primary Shared metastasis Lung metastases Chest-wall metastasis Perinephric metastasis M1 10 cm R2 (G3) R4 (G1) R6 (G1) Hilum R8 (G4) Primary tumor M2b M2a
  • 3. BACKGROUND – RADIOMICS - HISTOPATHOLOMICS • Huge potential for computerized medical image analysis • Create imaging biomarkers to predict diagnosis, prognosis, 
 treatment response [2] 3 [2] Imaging and genomics: is there a synergy?, Jaffe et al., Radiology, 264(2):329-31, 2012
 [3] Radiomics: the process and the challenges, Kumar et al., Magn Reson Imaging, 30(9):1234-48, 2012
 [4] Histopathological image analysis: a review, Gurcan et al., IEEE Reviews in Biomed Eng, 2:147-71, 2009 Radiomics [3] “Histopatholomics” [4] Reuse existing 
 diagnostic images ✓ radiology data ✓ digital pathology Capture tissue 
 heterogeneity ✓ 3D neighborhoods
 (e.g., 512x512x512) ✓ large 2D regions
 (e.g., 15,000x15,000) Analytic power beyond 
 naked eyes ✓ complex 3D tissue morphology ✓exhaustive characterization of 2D tissue structures Non-invasive ✓ x
  • 4. BACKGROUND – RADIOMICS - HISTOPATHOLOMICS • Huge potential for computerized medical image analysis • Create imaging biomarkers to predict diagnosis, prognosis, 
 treatment response • Local quantitative image feature extraction • Supervised machine learning 4 malignant, nonresponder malignant, responder benign pre-malignant undefined quant. feat. #1 quant.feat.#2 Supervised learning, 
 big data This could include an additional step of studying the spatial relationships between local image properties (e.g., using image graphs)
  • 5. • Shape, intensity, margin, texture, … • Shape and margin features often 
 require prior image segmentation • 2D and 3D texture analysis can quantify micro- and macro- structures in biomedical images [4,6] IMAGE FEATURES 5 GURCANetal.:HISTOPATHOLOGICALIMAGEANALYSIS:AREVIEW Fig.5.Resultsoftheautomaticsegmentationalgorithm(bluecontours:lumen boundary,blackcontours:innerboundaryofthenucleioftheepithelialcells surroundingthegland).Shownfromlefttorightareexampleimagesofbenign epithelium,intermediate-,andhigh-gradecancer. andare2-DCartesiancoordinatesof.Theevolutionof isthendescribedbyalevel-setformulationadoptedfrom[78] (4.1) [4] Histopathological image analysis: a review, Gurcan et al., IEEE Reviews in Biomed Eng, 2:147-71, 2009
 [5] Quantifying the margin sharpness of lesions on radiological images for content-based image retrieval, Xu et al., Med Phys, 39(9):5405-18, 2012
 [6] Three-dimensional solid texture analysis in biomedical imaging: review and opportunities, Depeursinge et al., Med Image Anal, 18(1):176-96, 2014
 [7] Prediction of prognosis for prostatic adenocarcinoma by combined histological grading and clinical staging, Gleason et al., J Urol, 111(1):58-64, 1974
 [8] HyMaP: A hybrid magnitude-phase approach to unsupervised segmentation of tumor areas in breast cancer histology images, Khan et al., 
 J Pathol Inform, 4, 2013 [4] entsdrawnontheborderof formationtocharacterizethe hescaleandwindowparam- entsfromthisportionofthe (d) )byradiologist.(b)Automatically Ioutline(inwhite).(c)Finallung ewithlinesegments(inwhite)that [5] J Pathol Inform 2013, 1:1 http://www.jpath the phase spectrum to repres HyperCS regions in a breast the recently established efficac exhibiting randomness. Let vi (x,y) denote the ith Gab normalized and smoothened v I(x,y), where i = ,1,2,...,Ng , Ng the number of orientations. We v x y v x y j x yi i i( , ) ( , ) exp( ( , ))= φ where |·| denotes the magnit denotes the local phase. The g and its magnitude can then be φi i i i i x y v x y v x y v x y v x y ′ ′ ′ = − ⎡ ⎢ ⎢ ⎤ ⎥ ⎥ ( , ) ( , ) ( , ) ( , ) ( , ) Figure 1:A sample H & E–stained breast cancer histology image:(a) Original image, and (b) Overlaid image, with four types of contents shown in different colors.The tumor areas are shown in Red,HypoCS in Purple,and HyperCS in Green.Areas containing background or fat tissue are shown in white with black outline. Note the difference in morphology of the Hypo- and Hypercellular stromal regions ba [Downloaded free from http://www.jpathinformatics.org on Tuesday, June 16, 2015, IP: 128.179.146.236] [7] [8]
  • 6. COMPUTERIZED TEXTURE ANALYSIS directionsscale 6 • Image scales and directions are important for visual texture discrimination • Most approaches are leveraging these two properties • Explicitly: Gray-level co-occurrence matrices (GLCMs), run-length matrices (RLE), directional filterbanks and wavelets, Fourier, histograms of gradients (HOG), local binary patterns (LBP) • Implicitly: Convolutional neural networks (CNN), scattering transform, topographic independant component analysis (TICA)
  • 7. COMPUTERIZED TEXTURE ANALYSIS 7 • Texture invariances: computer vision VS biomedical imaging Computer vision Biomedical image analysis scale scale-invariant multi-scale rotation rotation-invariant rotation-invariant [4] Histopathological image analysis: a review, Gurcan et al., IEEE Reviews in Biomed Eng, 2:147-71, 2009
 [9] A sparse texture representation using local affine regions, Lazebnik et al., IEEE Trans on Pattern Anal and 
 Mach Intel, 27(8):1265-78, 2005 160 Fig. 10. (a) A digitized histopathology image of Grade 4 CaP and different graph-based r Diagram, and Minimum Spanning tree. Fig. 11. Digitized histological image at successively higher scales (magnifica- tions) yields incrementally more discriminatory information in order to detect suspicious regions. or resolution. For instance at low or coarse scales color or tex- ture cues are commonly used and at medium scales architec- tural arrangement of individual histological structures (glands and nuclei) start to become resolvable. It is only at higher res- olutions that morphology of specific histological structures can be discerned. In [93], [94], a multiresolution approach has been used for the classification of high-resolution whole-slide histopathology im- ages. The proposed multiresolution approach mimics the eval- uation of a pathologist such that image analysis starts from the lowest resolution, which corresponds to the lower magnification levels in a microscope and uses the higher resolution represen- Fig. 12 image 1, (c) r as susp show three scale (scal the n dition highe tumo At is com COMPUTERIZED TEXTURE ANALYSIS 7 • Invariances: computer vision versus biomedical imaging Computer vision Biomedical image analysis scale scale-invariant multi-scale rotation rotation-invariant rotation-invariant [4] Histopathological image analysis: a review, Gurcan et al., IEEE Reviews in Biomed Eng, 2:147-71, 2009
 160 IE Fig. 10. (a) A digitized histopathology image of Grade 4 CaP and different graph-based representation Diagram, and Minimum Spanning tree. Fig. 11. Digitized histological image at successively higher scales (magnifica- tions) yields incrementally more discriminatory information in order to detect suspicious regions. or resolution. For instance at low or coarse scales color or tex- ture cues are commonly used and at medium scales architec- tural arrangement of individual histological structures (glands and nuclei) start to become resolvable. It is only at higher res- olutions that morphology of specific histological structures can be discerned. In [93], [94], a multiresolution approach has been used for the classification of high-resolution whole-slide histopathology im- ages. The proposed multiresolution approach mimics the eval- uation of a pathologist such that image analysis starts from the lowest resolution, which corresponds to the lower magnification levels in a microscope and uses the higher resolution represen- tations for the regions requiring more detailed information for a classification decision. To achieve this, images were decom- posed into multiresolution representations using the Gaussian pyramid approach [95]. This is followed by color space con- version and feature construction followed by feature extraction and feature selection at each resolution level. Once the classifier is confident enough at a particular resolution level, the system assigns a classification label (e.g., stroma-rich, stroma-poor or Fig. 12. Results fr image with the tum 1, (c) results at scale as suspicious at low shows the origin three columns s scales. Pixels cl (scale) are disc the number of p ditionally, the p higher scales al tumor and nont At lower reso is commonly us pattern of gland tized histologic scenes can be to every pixel i dient, and Gab the scale, orien gion of interest features within [4][9]
  • 8. • Averaging texture properties over the entire lesions or images discards tissue heterogeneity [10] • Exhaustive and non-specific features • Limited characterization of directions [6] BIOMEDICAL TEXTURE ANALYSIS: LIMITATIONS (1/4) 8 [6] Three-dimensional solid texture analysis in biomedical imaging: review and opportunities, Depeursinge et al., Med Image Anal, 18(1):176-96, 2014
 [10] Quantitative imaging in cancer evolution and ecology, Gatenby et al., Radiology, 269(1):8-15, 2013
 feartures are not specific when not learned from data
  • 9. • A global characterization of directions is not enough [11] • local organization of image directions: • independently from their local orientation: • Rotation-covariance • (local) grouped steering 
 of the operators: BIOMEDICAL TEXTURE ANALYSIS: LIMITATIONS (2/4) image operators: grouped steering: 9 [11] Rotation–covariant texture learning using steerable Riesz wavelets, Depeursinge et al., IEEE Trans Imag Proc, 23(2):898-908, 2014

  • 10. BIOMEDICAL TEXTURE ANALYSIS: LIMITATIONS (3/4) GLCM contrast [1 0] GLCMcontrast[01] GLCMs (unaligned) Riesz (unaligned) Riesz (aligned) 10
  • 11. BIOMEDICAL TEXTURE ANALYSIS: LIMITATIONS (4/4) GLCM contrast [1 0] GLCMcontrast[01] GLCMs (unaligned) Riesz (unaligned) Riesz (aligned) 11
  • 12. OBJECTIVES 12 • Highly adaptive texture-based computational models of biomedical tissue: ✓ Complete coverage of image scales and 
 directions in 2-D and 3-D ✓ Rotation-covariance ✓ Specificity: the models can be trained to 
 characterize specific tissue types ✓ Local characterization of tissue properties ✓ Locate tissue properties in organ anatomy 
 to create digital phenotypes • Goal: predict survival, function, treatment response 
 and reveal subtypes be averaged over the folds of the CV and used to build texture models for each location along dROI (see Fig 9). The sum of the weights for all channels from each location will reveal the subregions that are specific to each tumor subtypes, and most related to patient survival. A K–means clustering of the vectors w will be carried out for each locations to evaluate the stability of the regional models over the folds of the CV and define homogeneous groups among patients. A selection of the models based on stability and location importance will be carried out. Fig. 9 Prototype tissue archi- tecture of a GBM tumor [128]. In a second step, the selected models will be locally steered to maximize their magnitude. The energies of the maximal magnitudes will be used to construct a final feature space for (1) predicting the tumor subtypes and (2) performing Kaplan–Meier survival analysis. The performance of the proposed approach for predicting tumor subtypes and patient survival will be compared to (1) unaligned wavelet energies and (2) average wavelet energies over the entire tumor. While starting with a LOPO CV on the TCGA–TCIA dataset, the generalizability of the approach will be further assessed by training with the TCGA–TCIA and testing with the SU dataset. Deliverable 2.1: Predicting tumor subtype and survival from localized of tissue properties in GBM tumors. Task 2.2: Digital lung tissue atlases of ILD diagnoses (8 months of the PI) In this task, we will use a simple atlas of the lungs to locate texture properties and create prototype diagnosis phenotypes of ILDs. In previous work, we developed a simple 3–D atlas of the lungs with 36 subregions that will be used in this task [42]. In a first step, diagnosis–wise digital tissue atlases will be created by learning 3–D texture models (i.e., the average of w over the folds of the LOPO CV) for each 36 regions of the lungs (see Fig. 10). The regions for which the models are most distant11 from all other diagnoses will be highlighted and compared to previously built models of tissue patterns [25] to create 3–D prototype tissue atlases for each diagnosis. The obtained results will be validated using medical knowledge (e.g., Table 1 of [42]). Fig. 10 Tissue atlas of the lungs. For each diagnosis, K–means clustering of the vectors w will be carried out for each locations to evaluate the stability of the regional models over the folds of the CV and reveal homogeneous groups among patients (e.g., subtypes of UIP). The most stable models will be kept for the further characterization of lung tissue types. A hierarchical clustering of the models from all diagnoses will be carried out to define a radiomics–based hierarchy of all diagnoses, which will be compared to medical knowledge [140] (e.g., Fig. 1 of [40]). A large feature space including the energies of the steered models from each of the 36 localizations will be used to predict the diagnoses with uncertainty assessment (using e.g., pairwise coupling [141]). When a minimum amount of trust is not achieved when predicting a given diagnosis, the parent group of ILD diagnoses in the previously built hierarchy will be predicted instead [19]. Deliverable 2.2: Digital tissue atlases of ILD diagnoses and their subtypes. Task 2.3: Digital tissue atlases of ILDs: correlation with PFTs and survival (6 months ) Digital tissue atlases will be constructed for poor versus normal/high (1) PFTs and (2) survival. Regions for which the models significantly di↵er between poor versus normal/high will be revealed as being of primary importance to evaluate pulmonary function. The links between these models and previously built models of tissue patterns will be investigated to define the combination of regions and patterns that are most responsible for lung function impairment. The feature space spanned by the energies of the steered models will be used to predict (1) PFT values or (2) survival with a LOPO CV, which can be evaluated using ground truth. Deliverable 2.3: Estimating pulmonary function from digital tissue atlases of ILD diagnoses in CT images. 5.3 WP3. Imaging genomics 8 (a) Original Image (c) Voronoi Diagram 8 (a) Original Image (b) Simple Cell Graph (c) Voronoi Diagram (d) Delaunay Triangulation (e) ECM-Aware Cell-Graph Fig. 2 A fractured bone tissue example is shown in 2(a). Note the fracture cells in the middle of the original image. The simple-cell-graph representation, the Voronoi diagram and the Delaunay triangulation for this sample tissue are depicted in 2(b), 2(c) and 2(d). The corresponding ECM-aware cell-graph is drawn in 2(e). The interactions between fracture cells are drawn with blue and the red cells with red color. Delaunay triangulation represents the tissue as a single connected component and does not allow crossing of edges. Simple-cell- graphs relaxes these restrictions and allows the tissue to non-planar graph and disconnected. Likewise, ECM-aware cell-graphs do not put such restrictions on the tissue and moreover can capture the structural organization of different cells in a tissue. Furthermore, Delaunay triangulations are fixed representations whereas ECM-aware cell-graphs can be adjusted with different linking thresholds.[12] Automated classification of usual interstitial pneumonia using regional volumetric texture analysis in high- resolution CT, Depeursinge et al., Invest Radiol, 50(4):261-67, 2015
 [13] ECM-Aware Cell-Graph Mining for Bone Tissue Modeling and Classification, Bilgin et al., Data Min Knowl Discov, 20(3):416-38, 2009 [12] [13]
  • 13. Steerable texture models   Σ associated texture model EXAMPLE: PATTERN = WIGGLED CHECKERBOARD pattern to learn: 30 Steerable texture models: our workhorse …. 13
  • 14. • Multi-directional, multi-scale and rotation-covariant image analysis is achieved using Riesz wavelets • The components of the th-order Riesz transform of a 2-D signal are defined in the Fourier domain as [14]: • Yields allpass filters: only phase (i.e., direction) is kept and is defined by th-order partial derivatives THE RIESZ TRANSFORM 14 [14] Wavelet Steerability and the Higher-Order Riesz Transform, Unser et al., IEEE Trans on Imag Proc, 
 19(3):636-52, 2010
  • 15. • Riesz components THE RIESZ TRANSFORM input image 15
  • 16. • Riesz components THE RIESZ TRANSFORM 16
  • 17. • Riesz components THE RIESZ TRANSFORM 17
  • 18. • Multi-scale filterbanks are obtained by combining the Riesz transform with isotropic wavelets [15]
 • E.g., dyadic ( ) • Systematic coverage 
 of image scales THE RIESZ TRANSFORM 18 UNSER et al.: STEERABLE PYRAMIDS AND TIGHT WAVELET FRAMES IN 2711 of unity) guarantees that is dense in ; it is essential for the -completeness of the wavelet decomposition. We now proceed with the construction of orienta- tion-free wavelets by projecting the multiresolution Riesz basis onto some appropriate wavelet subspace . B. Isotropic Bandlimited Wavelet Frames There are a number of constructions in the literature that fall into this category [13], [27]–[29]. Before reviewing them, we apply the aforementioned projection strategy to obtain a straightforward design that is in direct correspondence with Shannon’s sampling theorem, and that is the starting point for the specification of Meyer-type wavelets. 1) Construction of Isotropic, Shannon-Type Wavelets: By selecting in (19), we specify the so-called Shannon multiresolution analysis of , which consists of a sequence of embedded subspaces that are bandlimited to We then define some corresponding wavelet subspaces of radi- ally bandpass functions (20) Since is a closed subspace of , we can apply Proposition 3 to its orthogonal sinc basis to obtain the tight wavelet frame of with (21) where is the impulse response of the ideal radial bandpass filter, whose frequency response is Fig. 1. Tiling of the 2-D frequency domain using radial-bandpass filters. The shaded area corresponds to the spectral support of the wavelet subspace ; it is included in the spectral support of (enclosing square). satisfy (22). This leads to the following extended definition of the wavelet subspaces which is equivalent to (20) if is the impulse response of the ideal radial bandpass filter. Since can be written as , there exists a sequence such that where the wavelet functions are still given by (21). This indicates that is a frame of , albeit not neces- sarily a tight one. Yet, if Condition (22) is satisfied, then one recovers the tight frame property over which is the union of the wavelet subspaces , . The condition for the wavelet frame to be isotropic is that the restriction of the fil- tering function over be isotropic, i.e., . [15] Steerable pyramids and tight wavelet frames in , Unser et al., IEEE Trans on Imag Proc, 
 20(10):2705-21, 2011
  • 19. THE RIESZ TRANSFORM 19 • Steerability [16]: • Example : • Higher-order steering can be done fully analytical ✓Complete coverage of the image directions ✓Steerability enables rotation-covariance [16] The design and use of steerable filters, Adelson et al., IEEE Trans on Pattern Anal and Mach Intel, 
 13(9):891-906, 1991
  • 20. • A Riesz filterbank constitutes a dictionary of basic textures: • Higher-level texture models are built from linear combinations of Riesz components LEARNING TEXTURE MODELS 20
  • 21.   Σ EXAMPLE: PATTERN = WIGGLED CHECKERBOARD pattern to learn:
  • 22.   Σ EXAMPLE: PATTERN = WIGGLED CHECKERBOARD pattern to learn:
  • 23.   Σ EXAMPLE: PATTERN = WIGGLED CHECKERBOARD pattern to learn:
  • 24.   Σ EXAMPLE: PATTERN = WIGGLED CHECKERBOARD pattern to learn:
  • 25.   Σ EXAMPLE: PATTERN = WIGGLED CHECKERBOARD pattern to learn:
  • 26.   Σ EXAMPLE: PATTERN = WIGGLED CHECKERBOARD pattern to learn:
  • 27.   Σ EXAMPLE: PATTERN = WIGGLED CHECKERBOARD pattern to learn:
  • 28.   Σ associated texture model EXAMPLE: PATTERN = WIGGLED CHECKERBOARD pattern to learn: 28
  • 29. • Support vector machines (SVM) were used to learn (multi-scale ) versus   ONE-VERSUS-ALL SUPERVISED MODEL LEARNING [11]: texture all others [11] Rotation-covariant texture learning using steerable Riesz wavelets, Depeursinge et al., IEEE Trans Imag Proc, 23(2):898-908, 2014
 29
  • 30. • 2-D synthetic textures with noise ( ) ✓ Captures specific local properties in terms of image 
 scales and directions LEARNING TEXTURE MODELS 30
  • 31. • 2-D tissue from interstitial lung diseases (ILD) in CT [17] • ( , 32x32 patches) LEARNING TEXTURE MODELS healthy emphysema ground glass fibrosis micronodules 3011 patches, 7 patients. 407 patches, 
 6 patients. 2226 patches, 32 patients. 2962 patches, 37 patients. 5988 patches, 16 patients. 31 [17] Multiscale lung texture signature learning using the Riesz transform, Depeursinge et al., Med Image Comput Comput Assist Interv (MICCAI), 15(3):517-24, 2012

  • 32. • 3-D synthetic textures ( , based on [18]) • Vertical planes • 3-D checkerboard • 3-D wiggled 
 checkerboard LEARNING TEXTURE MODELS 32 [18] 3D Steerable Wavelets and Monogenic Analysis for Bioimaging, Chenouard et al., IEEE 8th Int Symp on Biomed Imag (ISBI), 2132-5, 2011
  • 33. • Steering texture models: STEERABLE TEXTURE MODELS [11] The expression of the rotated texture model remains a linear combination of the initial Riesz components [11] Rotation-covariant texture learning using steerable Riesz wavelets, Depeursinge et al., IEEE Trans Imag Proc, 23(2):898-908, 2014

  • 34. • Steering texture models: • Rotation-covariant models: local steering of • Keep the maximum magnitude of the model • Local quantitative features: energies/abs values of the magnitudes in the patch STEERABLE TEXTURE MODELS [11] 34 The expression of the rotated texture model remains a linear combination of the initial Riesz components [11] Rotation-covariant texture learning using steerable Riesz wavelets, Depeursinge et al., IEEE Trans Imag Proc, 23(2):898-908, 2014

  • 35. Classification of 2-D natural textures IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2013 5 1) canvas001 2) canvas002 3) canvas003 4) canvas005 5) canvas006 6) canvas009 7) canvas011 8) canvas021 9) canvas022 10) canvas023 11) canvas025 12) canvas026 13) canvas031 14) canvas032 15) canvas033 16) canvas035 17) canvas038 18) canvas039 19) tile005 20) tile006 21) carpet002 22) carpet004 23) carpet005 24) carpet009 Fig. 5. 128 ⇥ 128 blocks from the 24 texture classes of the Outex database. 1) canvas 2) cloth 3) cotton 4) grass 5) leather 6) matting 7) paper 8) pigskin APPLICATIONS (1/2) 35
  • 36. • 24 classes, 180 images/class, 9 rotation angles in • A SVM classifier is trained with unrotated images only • 98.4% best acc. • aligned models 
 ( ) • Literature: 90-99% • E.g., scattering transform: 
 98.75% [18] 2-D TEXTURE CLASSIFICATION: OUTEX DATABASE [17] Errors occur with most stochastic textures confusion matrix: IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2013 TABLE AVERAGE Ac OBTAINED WITH THE LOCAL ORIENTATION MAXIMIZATION O Outex TC 00010 Outex TC 0001 ¯Ac for N = 2, 4, 6, 8, 10. 98 ± 0.7 97.2 ± 0 ¯Ac for N = 1, 3, 5, 7, 9. 94.4 ± 0.7 93.6 ± 0 ¯Ac for N = 1, . . . , 10. 96.2 ± 2 95.4 ± 0.4 0.5 0.6 0.7 0.8 0.9 1 1 2 3 4 5 6 7 8 9 10 classificationaccuracy N aligned aligned initial Riesz N c,1 R(0,N) Fig. 7. Classification performance with the Outex TC 00010 test suite. The two rotation covariant approaches are performing much better than using the initial Riesz coefficients. The local orientation maximization of N c,1 outperforms the local orientation of the first Riesz template R(0,N) as proposed in [8]. A maximum Ac of 98.4% is reached with N = 8. classes are balanced, the classification accuracy Ac is used as a performance measure of the methods. All performances are summarized in Table I. 1) Outex TC 00010: The classification performance for orders N = 1, . . . , 10 is shown in Fig. 7. The performance using the energy of the coefficients that are maximizing the response of the first Riesz template (i.e., R(0,N) ) at the smallest scale was also evaluated as a first rotation–covariant approach [8]. An average Ac of 96.2 ± 2% is obtained with N = 1 . . . 10 and the local orientation maximization of N . F f m c a i IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2013 5 1) canvas001 2) canvas002 3) canvas003 4) canvas005 5) canvas006 6) canvas009 7) canvas011 8) canvas021 9) canvas022 10) canvas023 11) canvas025 12) canvas026 13) canvas031 14) canvas032 15) canvas033 16) canvas035 17) canvas038 18) canvas039 19) tile005 20) tile006 21) carpet002 22) carpet004 23) carpet005 24) carpet009 Fig. 5. 128 ⇥ 128 blocks from the 24 texture classes of the Outex database. 1) canvas 2) cloth 3) cotton 4) grass 5) leather 6) matting 7) paper 8) pigskin 9) raffia 10) rattan 11) reptile 12) sand 13) straw 14) weave 15) wood 16) wool Fig. 6. 16 Brodatz texture classes of the Contrib TC 00000 test suite. 180 ⇥ 180 images from rotation angles 20 , 70 , 90 , 120 , 135 and 150 of the other seven Brodatz images for each class. The total number of images in the test set is 672. G. Experimental setup OVA SVM models using Gaussian kernels as K(xi, xj) = exp( ||xi xj ||2 2 2 k ) are used both to learn texture signatures and to classify the texture instances in the final feature space obtained after k iterations. A number of scales J = 6 Inc., 2012. The computational complexity is dominated by the local orientation of N c in Eq. 11, which consists of finding the roots of the polynomials defined by the steering matrix A✓ . It is therefore NP–hard (Non–deterministic Polynomial–time hard), where the order of the polynomials is controlled by the order of the Riesz transform N. III. RESULTS The performance of our approach is demonstrated with [17] Multiresolution gray-scale and rotation invariant texture classification with local binary patterns, Ojala et al., IEEE Trans on Pattern Anal and Mach Intel, 24:7(971-87), 2002
 [18] Combined scattering for rotation invariant texture analysis, Sifre et al., European Symposium on Artificial Neural Networks, 2012 Importance of rotation-covariance and learned models
  • 37. Medulloblastoma tumor classification APPLICATIONS (2/2) ompute the UFL features learned by TICA and the supervised features learned with Riesz wavelets for each image as described in Sections 2.1 and 2.2. Once TICA and Riesz wavelets are computed a final step of supervised classification is made using the combination of the computed features in a concatenated vector as input for a standard softmax classifier as described in Section 2.3. Parameter uning is presented in Section 3.2. [2.4] Medulloblastoma Image Cases Fig. 2. Flowchart for MB feature extraction and classification for both learned repre- entations: Riesz and TICA, the details of each stage are described in subsections. steerable Riesz texture models topographic ICA 37
  • 38. • Medulloblastoma (MB) • Most common (i.e., 25%) pediatric brain tumor • Major cause of death in pediatric oncology • Histological subtypes of MB have 
 different prognosis and treatments • Anaplastic subtype is the worst! • “marked nuclear pleomorphism, 
 cell wrapping, 
 high mitotic count, 
 and abundant apoptotic bodies” [19] MEDULLOBLASTOMA TUMORS [19] 38 [19] Childhood medulloblastoma: novel approaches to the classification of a heterogeneous disease, Ellison et al., Acta Neuropathol, 20(3):305-16, 2010
 Pleomorphism (cytology): variability in the size and shape of cells and/or their nuclei
  • 39. • Proposed approach [20] • Anaplastic versus non-anaplastic • Compare/combine steerable Riesz texture models with 
 unsupervised topographic independent component analysis (TICA) MEDULLOBLASTOMA TUMOR CLASSIFICATION 39 [20] Combining Unsupervised Feature Learning and Riesz Wavelets for Histopathology Image Representation: Application to Identifying Anaplastic Medulloblastoma, Otálora et al., Med Image Comput Comput Assist Interv (MICCAI), 2015
 The workflow of the proposed approach is summarized in Fig. 2. As first step, we ompute the UFL features learned by TICA and the supervised features learned with Riesz wavelets for each image as described in Sections 2.1 and 2.2. Once TICA and Riesz wavelets are computed a final step of supervised classification is made using the combination of the computed features in a concatenated vector as input for a standard softmax classifier as described in Section 2.3. Parameter uning is presented in Section 3.2. [2.4] Medulloblastoma Image Cases Fig. 2. Flowchart for MB feature extraction and classification for both learned repre- entations: Riesz and TICA, the details of each stage are described in subsections. steerable Riesz texture models topographic ICA
  • 40. • Unsupervised topographic independent component analysis (TICA) [21] • Learn while minimizing cost function : MEDULLOBLASTOMA TUMOR CLASSIFICATION 40 [21] Energy correlations and topographic organization, Hyvärinen et al., Natural Image Statistics, 39:249-272, 2009 TICA is an unsupervised feature learning model, inspired by findings of the visual cortex behaviour. It groups activations of units in order to discover features that are rotation and translation invariant [1]. These are appropriate features for histopathology image characterization since shapes and cell organizations can be present regardless of the position or orientation of cells. Particularly, TICA organizes feature detectors in a square matrix for l groups such that adjacent feature detectors activate in a similar proportion to the same stimulus. To learn such groups, we need to optimize the cost function: JTICA(W) = 2 TX i=1 WT Wx(i) x(i) 2 2 + mX i=1 lX k=1 » Hk(Wx(i))2 + ✏ (1) where x(i) 2 Rm is the i-th sample, T is the number of samples, W 2 Rn⇥m is the matrix that encodes the features in each row, and H 2 {0, 1}l⇥n is the binary topographic organization where H (j) k = 1, if the j-th feature detector, j-th row of W , belongs to the k-th group, and 0 otherwise. This model sets H fixed while learning W. In addition, TICA has two main computational advantages. First, the only parameters to be tuned are the regularization hyperparameter and the sparsity controller ✏. Second, it is an unconstrained optimization problem, which can be solved e ciently by optimization techniques such as Limited memory- Broyden-Fletcher-Goldfarb-Shanno (L-BFGS). iriiiiéiiiiiiiiiiiiosiiióiiiiïiiiiiii1=11.e1111MMIIIM 111111111111111M 11=1 MMMMMMMMMMMMMMMMMMM iiiMUMMMiMMMiiiiiii iíiíi iiiíiii ii UWE:lMiiM ME= M MEE is i iiiiíiiiiMóóóii iMMMMMlMiMMMMMMMiMMMiMMMMMMMMMiiMMiMMiMMM iMMMiMMMiiMMMMiiiiiiiMiMMiMiMiiMMiiMiiiiiiiiMM JTICA(W ) reconstruction penaltyL2 topographic constraint: 
 if the patch belongs 
 to the local group of patches Hk = 1 x(i) k ~scale inv. ~rotational inv. ~translational inv. W
  • 41. • Dataset: • 5 anaplastic, 5 non-anaplastic patients • 750 patches (200x200) / patient • Leave-2-patients-out cross-validation • Softmax classifier • Comparison with state-of-the-art [22] • Convolutional neural networks (CNN), sparse autoencoders (sAE),
 bag of features (BOF), Haar, MR8, … • Predictions over entire-slides: MEDULLOBLASTOMA TUMOR CLASSIFICATION 41 [22] A comparative evaluation of supervised and unsupervised representation learning approaches for anaplastic medulloblastoma differentiation, Cruz-Roa et al., Proc. SPIE 9287, Int Symp Med Inf Proc Anal, 92870G, 2015
 Medulloblastoma Di↵erentiat Table 1. MB classification performance ( averaged over the 20 test runs with standa Method Accuracy Se TICA + Riesz[N1 3 , N2 2 , N2 1 ] 0.997 ± 0.002 0.99 TICA [1] 0.972 ± 0.018 0.97 Riesz [N1 3 , N2 2 , N2 1 ] [3] 0.964 ± 0.038 0.99 Riesz [N1 3 ] [3] 0.958 ± 0.062 0.9 Riesz [N2 2 ] [3] 0.94 ± 0.02 0.9 2-Layer CNN [1] 0.90 ± 0.1 0.8 sAE [1] 0.90 BOF + A2NMF (Haar) [2] 0.87 Riesz [N2 1 ] [3] 0.85 ± 0.23 0. BOF + K - NN (Haar) [7] 0.80 BOF + K - NN (MR8) [7] 0.62 4 Concluding Remarks We present a feature fusion between u vised Riesz wavelet representation that [22] [22] [22] [22] [22] TICA [1] 0.972 ± 0.018 0.977 ± 0.021 0.967 ± 0.031 Riesz [N1 3 , N2 2 , N2 1 ] [3] 0.964 ± 0.038 0.999 ± 0.001 0.932 ± 0.07 Riesz [N1 3 ] [3] 0.958 ± 0.062 0.963 ± 0.05 0.916 ± 0.125 Riesz [N2 2 ] [3] 0.94 ± 0.02 0.94 ± 0.02 0.3 ± 0.04 2-Layer CNN [1] 0.90 ± 0.1 0.89 ± 0.18 0.9 ± 0.0.3 sAE [1] 0.90 0.87 0.93 BOF + A2NMF (Haar) [2] 0.87 0.86 0.87 Riesz [N2 1 ] [3] 0.85 ± 0.23 0.9 ± 0.15 0.7 ± 0.47 BOF + K - NN (Haar) [7] 0.80 - - BOF + K - NN (MR8) [7] 0.62 - - 4 Concluding Remarks We present a feature fusion between unsupervised feature learning and super- vised Riesz wavelet representation that captures subtle pattern of textures as well as high level features, allowing to create a more separable feature space where the di↵erentiation of medulloblastoma into anaplastic and non-anaplastic can be made with high classification accuracy outperforming any other result previously described in the literature. To our knowledge this is the first time that a feature fusion method is presented between UFL and the Riesz wavelets in the context of histopathology image analysis showing the complementarity between these learned features for the challenging task of tumour di↵erentia- tion, we are currently working on extending the method to other patch-based histopathology image analysis problems with larger cohorts of patients. Fig. 3. Predictions over two WSIs, non-anaplastic MB (left) and anaplastic (right). non-anaplastic anaplastic TICA: unsupervised
 Riesz: rotation-covariant information Small dataset
  • 42. • Highly adaptive texture-based computational models of biomedical tissue: ✓ Complete/systematic coverage of image 
 scales and directions in 2-D and 3-D ✓ Rotation-covariance ✓ Specificity: the models can be trained to 
 characterize specific tissue types ✓ Local characterization of tissue properties • Locate tissue properties in organ anatomy 
 to create digital phenotypes • localization systems, 
 quant. graph analysis CONCLUSIONS model learning Multi-resolution, steerability patch-based analysis 42 be averaged over the folds of the CV and used to build texture models for each location along dROI (see Fig 9). The sum of the weights for all channels from each location will reveal the subregions that are specific to each tumor subtypes, and most related to patient survival. A K–means clustering of the vectors w will be carried out for each locations to evaluate the stability of the regional models over the folds of the CV and define homogeneous groups among patients. A selection of the models based on stability and location importance will be carried out. Fig. 9 Prototype tissue archi- tecture of a GBM tumor [128]. In a second step, the selected models will be locally steered to maximize their magnitude. The energies of the maximal magnitudes will be used to construct a final feature space for (1) predicting the tumor subtypes and (2) performing Kaplan–Meier survival analysis. The performance of the proposed approach for predicting tumor subtypes and patient survival will be compared to (1) unaligned wavelet energies and (2) average wavelet energies over the entire tumor. While starting with a LOPO CV on the TCGA–TCIA dataset, the generalizability of the approach will be further assessed by training with the TCGA–TCIA and testing with the SU dataset. Deliverable 2.1: Predicting tumor subtype and survival from localized of tissue properties in GBM tumors. Task 2.2: Digital lung tissue atlases of ILD diagnoses (8 months of the PI) In this task, we will use a simple atlas of the lungs to locate texture properties and create prototype diagnosis phenotypes of ILDs. In previous work, we developed a simple 3–D atlas of the lungs with 36 subregions that will be used in this task [42]. In a first step, diagnosis–wise digital tissue atlases will be created by learning 3–D texture models (i.e., the average of w over the folds of the LOPO CV) for each 36 regions of the lungs (see Fig. 10). The regions for which the models are most distant11 from all other diagnoses will be highlighted and compared to previously built models of tissue patterns [25] to create 3–D prototype tissue atlases for each diagnosis. The obtained results will be validated using medical knowledge (e.g., Table 1 of [42]). Fig. 10 Tissue atlas of the lungs. For each diagnosis, K–means clustering of the vectors w will be carried out for each locations to evaluate the stability of the regional models over the folds of the CV and reveal homogeneous groups among patients (e.g., subtypes of UIP). The most stable models will be kept for the further characterization of lung tissue types. A hierarchical clustering of the models from all diagnoses will be carried out to define a radiomics–based hierarchy of all diagnoses, which will be compared to medical knowledge [140] (e.g., Fig. 1 of [40]). A large feature space including the energies of the steered models from each of the 36 localizations will be used to predict the diagnoses with uncertainty assessment (using e.g., pairwise coupling [141]). When a minimum amount of trust is not achieved when predicting a given diagnosis, the parent group of ILD diagnoses in the previously built hierarchy will be predicted instead [19]. Deliverable 2.2: Digital tissue atlases of ILD diagnoses and their subtypes. Task 2.3: Digital tissue atlases of ILDs: correlation with PFTs and survival (6 months )
  • 43. CONCLUSIONS • Importance of rotation-covariant information to model biomedical textures • i.e., the local organization of directions • Rotational invariance is not enough • Called “roto-translation invariance” in [23] TextureQbased'biomarkers:'current'limitaGons' x  Assume'homogeneous'texture'properGes'over'the' enGre'lesion'[5]' ' x  NonQspecific'features' x  Global'vs'local'characterizaGon'of'image'direcGons'[6]' REVIEW: Quantitative Imaging in Cancer Evolution and Ecology Gatenby et al with the mean signal value. By using just two sequences, a contrast-enhanced T1 sequence and a fluid-attenuated inver- sion-recovery sequence, we can define four habitats: high or low postgadolini- um T1 divided into high or low fluid-at- tenuated inversion recovery. When these voxel habitats are projected into the tu- mor volume, we find they cluster into spatially distinct regions. These habitats can be evaluated both in terms of their relative contributions to the total tumor volume and in terms of their interactions with each other, based on the imaging characteristics at the interfaces between regions. Similar spatially explicit analysis can be performed with CT scans (Fig 5). Analysis of spatial patterns in cross-sectional images will ultimately re- quire methods that bridge spatial scales from microns to millimeters. One possi- ble method is a general class of numeric tools that is already widely used in ter- restrial and marine ecology research to link species occurrence or abundance with environmental parameters. Species distribution models (48–51) are used to gain ecologic and evolutionary insights and to predict distributions of species or morphs across landscapes, sometimes extrapolating in space and time. They can easily be used to link the environ- mental selection forces in MR imaging- defined habitats to the evolutionary dy- namics of cancer cells. Summary Imaging can have an enormous role in the development and implementation of patient-specific therapies in cancer. The achievement of this goal will require new methods that expand and ultimately re- place the current subjective qualitative assessments of tumor characteristics. rise to local-regional phenotypic adap- tations. Phenotypic alterations can re- sult from epigenetic, genetic, or chro- mosomal rearrangements, and these in turn will affect prognosis and response to therapy. Changes in habitats or the relative abundance of specific ecologic communities over time and in response to therapy may be a valuable metric with which to measure treatment efficacy and emergence of resistant populations. Emerging Strategies for Tumor Habitat Characterization A method for converting images to spa- tially explicit tumor habitats is shown in Figure 4. Here, three-dimensional MR microenvironment can be rewarded by increased proliferation. This evolution- ary dynamic may contribute to distinct differences between the tumor edges and the tumor cores, which frequently can be seen at analysis of cross-sec- tional images (Fig 5). Interpretation of the subsegmenta- tion of tumors will require computa- tional models to understand and predict the complex nonlinear dynamics that lead to heterogeneous combinations of radiographic features. We have ex- ploited ecologic methods and models to investigate regional variations in cancer environmental and cellular properties that lead to specific imaging character- istics. Conceptually, this approach as- Figure 4 Figure 4: Left: Contrast-enhanced T1 image from subject TCGA-02-0034 in The Cancer Genome Atlas–Glioblastoma Multiforme repository of MR volumes of glioblastoma multiforme cases. Right: Spatial distribution of MR imaging–defined habitats within the tumor. The blue region (low T1 postgadolinium, low fluid-attenuated inversion recovery) is particularly notable because it presumably represents a habitat with low blood flow but high cell density, indicating a population presumably adapted to hypoxic acidic conditions. rise to local-regional phenotypic adap- tations. Phenotypic alterations can re- sult from epigenetic, genetic, or chro- mosomal rearrangements, and these in turn will affect prognosis and response to therapy. Changes in habitats or the relative abundance of specific ecologic communities over time and in response to therapy may be a valuable metric with which to measure treatment efficacy and emergence of resistant populations. Emerging Strategies for Tumor Habitat Characterization A method for converting images to spa- tially explicit tumor habitats is shown in microenvironment can be rewarded by increased proliferation. This evolution- ary dynamic may contribute to distinct differences between the tumor edges and the tumor cores, which frequently can be seen at analysis of cross-sec- tional images (Fig 5). Interpretation of the subsegmenta- tion of tumors will require computa- tional models to understand and predict the complex nonlinear dynamics that lead to heterogeneous combinations of radiographic features. We have ex- ploited ecologic methods and models to investigate regional variations in cancer environmental and cellular properties that lead to specific imaging character- Figure 4: Left: Contrast-enhanced T1 image from subject TCGA-02-0034 in The Cancer Genome Atlas–Glioblastoma Multiforme repository of MR volumes of glioblastoma multiforme cases. Right: Spatial distribution of MR imaging–defined habitats within the tumor. The blue region (low T1 postgadolinium, low fluid-attenuated inversion recovery) is particularly notable because it presumably represents a habitat with low blood flow but high cell density, indicating a population presumably adapted to hypoxic acidic conditions. [5]'QuanGtaGve'imaging'in'cancer'evoluGon'and'ecology,'Gatenby'et'al.,'Radiology,'269(1):8Q15,'2013' 5' global'direcGonal'operators:' local'grouped'steering:' [6]'RotaGonQcovariant'texture'learning'using'steerable'Riesz'wavelets,'Depeursinge'et'al.,'IEEE'Trans'Imag'Proc.,'23(2):898Q908,'2014.' global directional operators local grouped steering 43 [11] Rotation-covariant texture learning using steerable Riesz wavelets, Depeursinge et al., IEEE Trans Imag Proc, 23(2):898-908, 2014
 [23] Rotation, Scaling and Deformation Invariant Scattering for Texture Discrimination, Sifre et al., IEEE Conf Comp Vis and Pat Rec (CVPR), 1233-40, 2013 with a second operator R2 which is invariant to the action of G2. Indeed for all g1.g2 ∈ G1 G2 and all images x(u): R2(R1(g1.g2.x)) = R2(g2.R1(x)) = R2(R1(x)). However, such separable invariants do not capture the joint property of the action of G2 relatively to G1, and may lose important information. This is why two-dimensional trans- lation invariant representations are not computed by cascad- ing invariants to horizontal and vertical translations. It is also important for rotations and translations. Let us consider for example the two texture patches of Figure 1. A separa- ble product of translation and rotation invariant operators can represent the relative positions of the vertical patterns, and the relative positions of the horizontal patterns, up to global translations. However, it can not represent the po- sitions of horizontal patterns relatively to vertical patterns, because it is not sensitive to a relative shift between these two sets of oriented structures. It loses the relative positions of different orientations, which is needed to be sensitive to curvature, crossings and corners. Such a separable invariant thus can not discriminate the two textures of Figure 1. Figure 1: The left and right textures are not discriminated by a separable invariant along rotations and translations, but can be discriminated by a joint roto-translation invariant. Several authors [6, 7, 8] have proposed to take into ac- count the joint structure of roto-translation operators in im- age processing, particularly to implement diffusion oper- ators. Computing a joint invariant between rotations and translations also means taking into account the joint rela- tive positions and orientations of image structures, so that the textures of Figure 1 can be discriminated. Section 3 introduces a roto-translation scattering operator, which is computed by cascading wavelet transforms on the roto- translation group. Calculating joint invariants on large non-commutative groups may however become very complex. Keeping a sep- information. 2.2. Hierarchical Architecture We now explain how to build an affine invariant repre- sentation, with a hierarchical architecture. We separate vari- abilities of potentially large amplitudes such as translations, rotations and scaling, from smaller amplitude variabilities, but which may belong to much higher dimensional groups such as shearing and general diffeomorphisms. These small amplitude deformations are linearized to remove them with linear projectors. Image variabilities typically differ over domains of dif- ferent sizes. Most image representations build localized in- variants over small image patches, for example with SIFT descriptors [15]. These invariant coefficients are then ag- gregated into more invariant global image descriptors, for example with bag of words [10] or multiple layers of deep neural network [4, 5]. We follow a similar strategy by first computing invariants over image patches and then aggregat- ing them at the global image scale. This is illustrated by the computational architecture of Figure 2. x roto-trans. patch scattering log global space-scale averaging deformat. invariant linear proj. Figure 2: An affine invariant scattering is computed by ap- plying a roto-translation scattering on image patches, a log- arithmic non-linearity and a global space-scale averaging. Invariants to small shearing and deformations are computed with linear projectors optimized by a supervised classifier. Within image patches, as previously explained, one must keep the joint information between positions and orienta- tions. This is done by calculating a scattering invariant on the joint roto-translation group. Scaling invariance is then implemented with a global scale-space averaging between patches, described in Section 4. A logarithmic non-linearity is first applied to invariant scattering coefficients to linearize their power law behavior across scales. This is similar to the normalization strategies used by bag of words [10] and deep neural networks [5]. Because of three dimensional surface curvature in the vi- sual scene, the image patches are also deformed. A scat- tering transform was proved to be stable to deformations [23] [11]
  • 44. • Rotation-covariance • Local orientation of the models is computationally intensive • 2D: • 3D: • Use graphics processor units (GPUs) [24]: 60x speedup • Explore other steerable wavelet representations [25,26] LIMITATIONS AND FUTURE WORK 44 G ⇤ R(2,0,0) G ⇤ R(0,2,0) G ⇤ R(0,0,2) G ⇤ R(1,1,0) G ⇤ R(1,0,1) G ⇤ R(0,1,1) Fig. 1. Second–order Riesz kernels R(n1,n2,n3) convolved with isotropic Gaussian kernels G(x). Support vector machines (SVM) are then used to classify between 9,347 normal and embolic cubic instances of lung parenchyma from 19 patients with APE and 8 control cases. 2. MATERIAL AND METHODS 2.1. Rotation–covariant texture analysis 3D multiscale Riesz filterbanks are used to characterize the texture of the lung parenchyma in 3D at a given CT energy level. The N–th order Riesz transform R(N) of a three– dimensional signal f(x) is defined in the Fourier domain as: ¤R(n1,n2,n3)f(!) = … n1 + n2 + n3 n1!n2!n3! ( j!1)n1 ( j!2)n2 ( j!3)n3 ||!||n1+n2+n3 ˆf(!), (1) Hospitals of Geneva with an inter–slice distance of 1mm, a slice thickness of 1.25mm, and a sub–millimetric resolution in the axial plane. 11 energy levels are used from 40keV to 140keV with a step of 10keV. All five lobes of each patient have been manually segmented using the OsiriX software2 . The perfusion levels of each lobe were quantified using the Qanadli index (QI) on a lobe basis [14]. The QI is defined as the sum of the scores of all arteries as: 0 if no occlusion is visible, 1 if partially occluded, and 2 if totally obstructed. DECT data of all energy levels are preprocessed to have an isotropic voxel resolution, which is obtained by dividing samples along the z axis. All lobes are divided into 323 overlapping blocks to constitute a local instance of the lung parenchyma. A block is considered as valid when at least 95% of its voxels belong to it. To obtain a sufficient number of blocks for the middle right lobe, this rule was changed to IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2013 10 RN f✓ (x) = 0 B B B B B B @ R(0,N) f✓ (x) ... R(n,N n) f✓ (x) ... R(N,0) f✓ (x) 1 C C C C C C A | {z } RN {f✓}(x) = 0 B B B B B B @ A✓ 0,0 . . . A✓ 0,l . . . A✓ 0,N ... ... ... A✓ n,1 . . . A✓ n,l . . . A✓ n,N ... ... ... A✓ N,1 . . . A✓ N,l . . . A✓ N,N 1 C C C C C C A | {z } A✓ 0 B B B B B B @ R(0,N) {f} (x) ... R(l,N l) {f} (x) ... R(N,0) {f} (x) 1 C C C C C C A | {z } RN {f}(x) . (12) ˆR(n,N n) =   N n!(N n)! ( j(cos(✓)!1 + sin(✓)!2))n ( j( sin(✓)!1 + cos(✓)!2))N n ||!||N = 1 ||!||N   N n!(N n)! nX k1=0 Ç n k1 å (cos(✓))k1 ( j!1)k1 (sin(✓))n k1 ( j!2)n k1 N nX k2=0 Ç N n k2 å (cos(✓))k2 ( j!1)k2 (sin(✓))N n k2 ( j!2)N n k2 = 1 ||!||N   N n!(N n)! nX k1=0 N nX k2=0 ( j!1)k1+k2 ( j!2)N k1 k2 ( 1)k2 Ç n k1 åÇ N n k2 å (cos(✓))N n k2+k1 (sin(✓))n k1+k2 = nX k1=0 N nX k2=0   N (k1 + k2)!(N k1 k2)! ( j!1)k1+k2 ( j!2)N k1 k2 ||!||N | {z } ˆR(k1+k2,N k1 k2)   (k1 + k2)!(N k1 k2)! n!(N n)! ( 1)k2 n! k1!(n k1)! (N n)! k2!(N n k2)! (cos(✓))N n k2+k1 (sin(✓))n k1+k2 . (13) ˆR(n,N n) (!) = NX l=0   N l!(N l)! ( j!1)l ( j!2)N l ||!||N | {z } ˆR(l,N l) min(l,n) X l1=max(0,l N+n) ( 1)l l1   l!(N l)! l1!(n l1)!(l l1)! (cos(✓))N n+2l1 l (sin(✓))n 2l1+l | {z } A✓ n,l . (14) [32] T. Leung and J. Malik. Representing and recognizing the visual appearance of materials using three–dimensional textons. International Journal of Computer Vision, 43(1):29–44, 2001. invariant texture classification with local binary patterns. In Computer Vision — ECCV 2000, volume 1842 of Lecture Notes in Computer Science, pages 404–420. Springer Berlin Heidelberg, 2000. [24] GPU-accelerated texture analysis using steerable Riesz wavelets, Vizitiu et al., 11th Int Conf Par Proc and App Math (PPAM), 2015 (submitted)
 [25] A unifying parametric framework for 2D steerable wavelet transforms, Unser et al., SIAM Jour Imag Sci, 6(1):102-35, 2013
 [26] Harmonic Singular Integrals and Steerable Wavelets in , Ward et al., App and Comp Harm Anal, 36(2):183-197, 2014
  • 45. • Multi-scale • Influence of surrounding objects: bandlimitedness VS compact support [27] • Continuous band-limited scale characterization [28] • Dyadic is not enough! LIMITATIONS AND FUTURE WORK 45 Contact and more information: adrien.depeursinge@epfl.ch, http://bigww References [1] An Official ATS/ERS/JRS/ALAT Statement: Idiopathic Pulmonary Fibrosis: Evidence-based Guidelines for Diagnosis and Management, G. Raghu et al., Am J Respir Crit Care Med 2011; 183(6):788-824 [2] VOW: Variance Optimal Wavelets for the Steerable Pyramid, P. Pad et al., IEEE ICIP 2014; 2973-2977 [3] 3D Steerable Wavelets and Monogenic Analysis for Bioimaging, N. Chenouard et al., IEEE ISBI 2011; 2132-2135 [4] A Parametric Texture Model Based on Joint Statistics of Complex Wavelet Coefficients, J. Portilla et al., Int Jour Comput Vision 2000; 1: 49-70 [5] Nonseparable radial frame multiresolution analysis in multidimensions and isotropic fast wavelet algorithms, M. Papadakis et al., SPIE Wavelets 2003; 5207: 631-642 [6] Ten Lectures on Wavelets, I. Daubechies, SIAM 1992; 61 •  bandwidth limited to •  generates tight frames •  Analytical approximation in Fourier: Results •  The proposed texture AUC and ACC for isotropic wavelet pyra Conclusions and •  New family of 3D isot bandwidth to balance operators and the infl •  Importance of rotation wavelet bandwidth de •  Future work includes higher orders of the R . . . . . . . . . . . . [27] Optimized steerable wavelets for texture analysis of lung tissue in 3-D CT: classification of usual interstitial pneumonia, Depeursinge et al., IEEE Int Symp on Biomed Imag (ISBI), 403-6, 2015
 [28] Fast detection and refined scale estimation using complex isotropic wavelets, Püspöki et al., IEEE Int Symp on Biomed Imag (ISBI), 512-5, 2015 spatial domain Fourier
  • 46. (a) Synthetic image containing 3 visual concepts: 1) vertical lines (quadrants I and III), 2) checkerboard (quadrant II), 3) wiggled checkerboard (quadrant IV). PCA 1 PCA2 10 1 10 2 10 3 (b) PCA visualization of 32⇥32 overlapping blocks and clus- ters from the left image (N = 10, J = 4, K = 3). The tem- plates 10 k corresponding to the respective visual concepts are dislayed for scale j = 3. 10 • Model learning • Limited performance for stochastic textures with no clear multi-scale signature • Reveal visual diversity with unsupervised learning [29,30] LIMITATIONS AND FUTURE WORK 46 [29] Rotation-covariant visual concept detection using steerable Riesz wavelets and bags of visual words, Depeursinge et al., SPIE Wavelets and Sparsity XV, 8858:885816-885816-11, 2013
 [30] Unsupervised texture segmentation using monogenic curvelets and the Potts model, Storath et al., IEEE Int Conf Imag Proc, 4348-52, 2014 (a) Synthetic image containing 3 visual concepts: 1) vertical lines (quadrants I and III), 2) checkerboard (quadrant II), 3) wiggled checkerboard (quadrant IV). PCA 1 PCA2 10 1 10 2 10 3 (b) PCA visualization of 32⇥32 overlapping blocks and clus- ters from the left image (N = 10, J = 4, K = 3). The tem- plates 10 k corresponding to the respective visual concepts are dislayed for scale j = 3. Figure 5: Qualitative evaluation of the visual concepts 10 k found using K–means in the feature space spanned 20135 nvas0055)canvas0066)canvas0097)canvas0118)canvas021 anvas02613)canvas03114)canvas03215)canvas03316)canvas035 ile00621)carpet00222)carpet00423)carpet00524)carpet009 atabase.
  • 47. • THANKS ! 47 Matlab code available! adrien.depeursinge@epfl.ch MICCAI tutorial on Biomedical 
 Texture Analysis: Oct 5th in Münich
 https://sites.google.com/site/btamiccai2015/
  • 48. BIOMEDICAL TISSUE MODELING IN 2D AND 3D • Interstitial lung diseases in CT • Lung texture classification using locally-oriented Riesz components, Depeursinge A, Foncubierta- Rodriguez A, Van de Ville D, Müller H, Med Image Comput Comput Assist Interv. (MICCAI) 2011;14(3):231-8. • Multiscale lung texture signature learning using the Riesz transform, Depeursinge A, Foncubierta- Rodriguez A, Van de Ville D, Müller H, Med Image Comput Comput Assist Interv. (MICCAI) 2012;15(3):517-24. • Automated classification of usual interstitial pneumonia using regional volumetric texture analysis in high-resolution CT, Depeursinge A, Chin A, Leung A, Terrone D, Bristow M, Rosen G, Rubin D, Invest Radiol.,
 in press. • Pulmonary embolism in dual-energy CT • Rotation-covariant texture analysis of 4D dual-energy CT as an indicator of local pulmonary perfusion, Depeursinge A, Foncubierta-Rodriguez A, Vargas A, Van de Ville D, Platon A, Poletti PA, 48
  • 49. BIOMEDICAL TISSUE MODELING IN 2D AND 3D • Liver lesions in CT • Predicting visual semantic descriptive terms from radiological image data: preliminary results with liver lesions in CT, Depeursinge A, Kurtz C, Beaulieu C, Napel S, Rubin D, IEEE Trans Med Imag. 2014;33(8):1669-76. • Brain epileptogenic lesions in MRI • Epileptogenic lesion quantification in MRI using contralateral 3D texture comparisons, Jiménez del Toro OA, Foncubierta-Rodríguez A, Vargas Gómez MI, Müller H, Depeursinge A, Med Image Comput Comput Assist Interv. (MICCAI) 2013;16(2):353-60. 49
  • 50. EVEN VS ODD ORDERS: 1-D signal: Heaviside filter: 1st (dashed) and 2nd order Gaussian derivatives convolution: 1st (dashed) 
 versus 2nd 50