![Previous Work: Congealing
(Miller et al. 2000)
• General framework, widely applicable
• Utilizes entropy-based objective function
• Iterative conditional maximization optimization
• Expectation regularization step: E[⇢] = 0](https://image.slidesharecdn.com/62c80372-3103-4a15-96c9-f4ba2d9a5d72-150916072638-lva1-app6891/85/Mattar_PhD_Thesis-10-320.jpg)
Mattar_PhD_Thesis
- 1. Unsupervised Joint Alignment, Clustering, and Feature Learning Marwan A. Mattar ! PhD Thesis Defense December 12, 2013
- 2. Unsupervised Joint Alignment, Clustering, and Feature Learning
- 3. Research Goal Develop an unsupervised data set-agnostic processing module that includes alignment, clustering, and feature learning
- 4. Joint Alignment Transform instances to make data set more coherent
- 5. Joint Alignment: Domains • Computer vision: remove spatial variability in images • Speech: remove temporal variability in speech signals • Psychology: recover stimulus onset in ERP signals • Radiology: remove bias in MRI scans • ...
- 6. Joint Alignment != Pairwise Alignment • Pairwise alignment involves two instances • Joint alignment can be achieved through (N - 1) pairwise alignments • Can be ineffective (local view & optimization)
- 7. Joint Alignment: Unsupervised • Supervision: fiducial points or examples of transforms • Can be expensive to obtain • We focus on unsupervised joint alignment
- 8. Joint Alignment • Input: • Data set, • Transformation function, • Output: • Transformed data set, s.t. is maximized • Transformation parameters, {x1, x2, . . . , xN } {y1, y2, . . . , yN } yi = ⌧(xi, ⇢i) {⇢1, ⇢2, . . . , ⇢N } S(y)
- 9. Joint Alignment: Previous Work • Procrustes analysis (1939 & 1970’s) • Congealing framework (2000) • Domain-specific algorithms (e.g. curves or images) • Data-set specific algorithms (e.g. ERP)
- 10. Previous Work: Congealing (Miller et al. 2000) • General framework, widely applicable • Utilizes entropy-based objective function • Iterative conditional maximization optimization • Expectation regularization step: E[⇢] = 0
- 11. Research Goals • Develop a framework for joint alignment • Handle complexities due to multi-modalities and representation ! • Release open-source implementations of all models clustering feature learning
- 12. Research Goals • Develop a • Handle complexities due to ! • Release clustering deep learning Develop an unsupervised data set-agnostic processing module that includes alignment, clustering, and feature learning
- 13. Today .... alignment clustering feature learning
- 14. Today .... alignment clustering feature learning
- 15. Today .... alignment clustering feature learning
- 16. Today .... alignment clustering feature learning
- 17. Future .... alignment clustering feature learning
- 18. Curve Alignment
- 19. Curve Alignment: Objectives • Adapt congealing framework to 1-dimensional curves • Nonparametric: entropy-based objective function • Efficient: global transformation parametrization • Experiment with effective parameterizations • Evaluate effect of alignment on classification
- 20. Curve Alignment: Objective Function • Input: curves of length • Sum of location-wise entropy (or variance) ! ! • Marginal entropy computed usingVasicek estimator {yn(t)}N n=1 ⇠ Y tS(y1, y2, . . . , yN ) = TX t=0 H(Y t ) H(Y) ˆH(x1, x2, . . . , xN ) / X log(x(mi+m+1) x(mi+1) ) N T
- 21. Curve Alignment: Optimization • Conditional maximization with expectation regularization step • Step sizes are sampled: • Dampen parameters: p ⇠ U(0, p)
- 22. Curve Alignment:Transformation y(t) = ↵g(t0 )x(t0 ) + t0 = h(t) linear amplitude scaling non-linear time warping non-linear amplitude scaling amplitude translation
- 23. Curve Alignment:Transformation • is a smooth monotone function • Utilize monotonicity operator of Ramsey (1998) y(t) = ↵g(t0 )x(t0 ) + t0 = h(t) linear amplitude scaling non-linear time warping non-linear amplitude scaling amplitude translation h(t) unconstrained function monotonicity operator
- 24. Curve Alignment:Transformation • Unconstrained and estimated using Fourier basis • 2 and 7 frequencies, respectively, were sufficient • Total # of parameters: 20 y(t) = ↵g(t0 )x(t0 ) + t0 = h(t) linear amplitude scaling non-linear time warping non-linear amplitude scaling amplitude translation w(t) g(t)
- 25. Curve Alignment: Experiments • Alignment of synthetic data sets • Alignment of real data sets • Improving classification with unsupervised joint alignment
- 26. Curve Alignment: Synthetic • 5 curves x 5 transformations x 3 levels of difficulty = 75 data sets • ~90% of the data sets converged to correct alignments
- 27. Curve Alignment: Synthetic Non-linear time scaling
- 28. Curve Alignment: Synthetic All transformations
- 29. Curve Alignment: Synthetic Failure case
- 30. Curve Alignment: Real 2-class data set, color-coded
- 31. Curve Alignment: Comparison to CPM TIC data set Original w/ variance
- 32. Curve Alignment: Comparison to CPM TIC data set Original w/ entropy
- 33. Curve Alignment: Classification Aligned Data Set Transformation Parameters Original Data
- 34. Curve Alignment: Classification • Perform unsupervised alignment • Represent each instance by its original + aligned + transformation • Improve the performance of an SVM
- 35. Curve Alignment: Classification Method Accuracy Original Curves (no alignment) 78.83% Dynamic Time Warping (pairwise alignment) 82.35% 13 Data Sets from UCRTime Series Repository
- 36. Curve Alignment: Classification Method Accuracy Original Curves (no alignment) 78.83% Dynamic Time Warping (pairwise alignment) 82.35% Joint Alignment (entropy + non-linear time) 82.97% 13 Data Sets from UCRTime Series Repository
- 37. Curve Alignment: Classification Method Accuracy Original Curves (no alignment) 78.83% Dynamic Time Warping (pairwise alignment) 82.35% Joint Alignment (entropy + non-linear time) 82.97% Joint Alignment (entropy + all transformations) 84.12% 13 Data Sets from UCRTime Series Repository
- 38. Curve Alignment: Classification Method Accuracy Original Curves (no alignment) 78.83% Dynamic Time Warping (pairwise alignment) 82.35% Joint Alignment (entropy + non-linear time) 82.97% Joint Alignment (entropy + all transformations) 84.12% Joint Alignment (variance + all transformations) 85.46% 13 Data Sets from UCRTime Series Repository
- 39. Curve Alignment: Classification Method Accuracy Original Curves (no alignment) 78.83% Dynamic Time Warping (pairwise alignment) 82.35% Joint Alignment (entropy + non-linear time) 82.97% Joint Alignment (entropy + all transformations) 84.12% Joint Alignment (variance + all transformations) 85.46% Joint Alignment (variance + all trans. + MKL) 85.79% 13 Data Sets from UCRTime Series Repository
- 40. Curve Alignment: Classification 13 Data Sets from UCRTime Series Repository
- 41. Curve Alignment: Conclusions • Effective on 1-dimensional curves • Small # of parameters needed • Unsupervised alignment improves classification
- 42. Curve Alignment: Drawbacks • Lack of a feature representation • Independence assumption in entropy computation • Results in co-alignment and difficulty aligning complex data sets • Transformation regularization is ad hoc
- 44. CRBM Alignment: Objectives • Alignment of complex data sets requires an appropriate representation • Automate feature selection • Use statistics of data set to learn an appropriate feature representation • Experimented with convolutional restricted Boltzmann machines
- 45. ! • Weights between visible units and hidden unit utilized as feature detector • Hidden units are binary and activate if feature is detected • Higher layers represent more complex features CRBM Alignment: Feature Learning v1 v2 v3 v4 v5 v6 v7 h1 h2 h3 h4 v1 v2 v3 v4 v5 v6 v7 h1 1 h1 2 h1 3 h1 4 p1 2p1 1 RBM CRBM (1 group)
- 46. CRBM Alignment: Overview • Train a CRBM using unaligned data • Use pooling unit activation probability as feature representation • Use (stochastic) congealing algorithm • Applied successfully to both curves and face images
- 47. CRBM Alignment: Curve Features • Trained a CRBM on training data from 13 UCR data sets • Learned 32 filters of length 10, with pooling area of 5 • 576 pooling unit activations
- 48. CRBM Alignment: Curve Features
- 49. CRBM Alignment: Curve Features
- 50. CRBM Alignment: Curve Features
- 51. CRBM Alignment: Classification Method Accuracy Alignment Classification None Raw 76.89% 13 Data Sets from UCRTime Series Repository
- 52. CRBM Alignment: Classification Method Accuracy Alignment Classification None Raw 76.89% Raw Raw 84.63% 13 Data Sets from UCRTime Series Repository
- 53. CRBM Alignment: Classification Method Accuracy Alignment Classification None Raw 76.89% Raw Raw 84.63% None CRBM 84.96% 13 Data Sets from UCRTime Series Repository
- 54. CRBM Alignment: Classification Method Accuracy Alignment Classification None Raw 76.89% Raw Raw 84.63% None CRBM 84.96% CRBM CRBM 87.40% 13 Data Sets from UCRTime Series Repository
- 55. CRBM Alignment: Classification 13 Data Sets from UCRTime Series Repository
- 56. CRBM Alignment: FaceVerification • Use alignment as pre-processing for face verification task • Use alternative / consistent feature representation for verification step
- 57. CRBM Alignment: Filters CRBM 32 filters, 10x10 size pooling: 5x5 KMean clusters
- 58. CRBM Alignment: Classification Method Accuracy Original (no alignment) 74.2% Alignment with SIFT features 75.8% Alignment with CRBM features 82.0% Supervised alignment 80.5% View 1 from LFW
- 59. CRBM Alignment: Summary • Effective for alignment of both curves & faces • Effective for curve classification
- 60. Joint Alignment & Clustering
- 61. JAC: Objectives • Develop nonparametric/unsupervised alignment & clustering framework • Applicable to any domain • Allows use of continuous transformations • Discovers # clusters automatically • Regularize transformations principally • Evaluate model on curve and image data sets
- 62. JAC: Objectives • Simultaneous > sequentially ! ! ! ! • KMeans clustering accuracy = 54% (almost random guessing)
- 63. JAC: Overview Bayesian alignment (assumes single mode) Infinite Bayesian alignment & clustering
- 64. JAC: Bayesian Alignment xi = ⌧(yi, ⇢i)
- 65. JAC: Bayesian Alignment • Explicit regularization of transformations • Direct maximization that treats transformation function as black box • Low memory footprint with sufficient stats
- 66. JAC: Bayesian Alignment Average pairwise distance: 8.7 (before), 5.2 (with congealing), 5.1 (with BA)
- 67. JAC: Bayesian Alignment 75 synthetic curve data sets
- 68. JAC: Bayesian Alignment Sample result
- 69. JAC: Model Bayesian alignment Single set of parameters Alignment & Clustering Potentially infinite parameters
- 70. JAC: Model Blocked Gibbs sampler: Direct generalization of BA transformation termdata termcluster term (CRP) (zi, ⇢i) ⇠ p(zi | z i; )p(yi | yzi ; )p(⇢i | ⇢zi ; ↵)
- 71. JAC: Model Second sampler integrates out transformations zi ⇠ p(zi | z i; )p(xi | x i, z; , ↵) Transformation parameter updated with EM
- 72. JAC: Digits ‘4’ & ‘9’
- 73. JAC: Digits ‘4’ & ‘9’ Method Clustering Alignment Original - 4.54 KMeans 54.0% -Affinity Propagation 82.0% (3) Infinite mixture model 86.0% (4)
- 74. JAC: Digits ‘4’ & ‘9’ Method Clustering Alignment Original - 4.54 KMeans 54.0% -Affinity Propagation 82.0% (3) Infinite mixture model 86.0% (4) Congealing (+KMeans) 90.0% 1.80
- 75. JAC: Digits ‘4’ & ‘9’ Method Clustering Alignment Original - 4.54 KMeans 54.0% -Affinity Propagation 82.0% (3) Infinite mixture model 86.0% (4) Congealing (+KMeans) 90.0% 1.80 Unsupervised JAC 94.0% (2) 1.20
- 76. JAC: Digits ‘4’ & ‘9’
- 77. JAC:All 10 Digit Classes
- 78. JAC:All 10 Digit Classes Method Clustering Alignment Original - 5.22 KMeans 62.5% -Affinity Propagation 67.5% (14) Infinite mixture model 69.0% (13)
- 79. JAC:All 10 Digit Classes Method Clustering Alignment Original - 5.22 KMeans 62.5% -Affinity Propagation 67.5% (14) Infinite mixture model 69.0% (13) Congealing (+KMeans) 64.0% 3.43 Liu et al. 56.5% / 73.7% 3.80
- 80. JAC:All 10 Digit Classes Method Clustering Alignment Original - 5.22 KMeans 62.5% -Affinity Propagation 67.5% (14) Infinite mixture model 69.0% (13) Congealing (+KMeans) 64.0% 3.43 Liu et al. 56.5% / 73.7% 3.80 Unsupervised JAC 87.0% (11) 1.58
- 81. JAC:All 10 Digit Classes
- 82. JAC: ECG Heart Data
- 83. JAC: ECG Heart Data
- 84. JAC: ECG Heart Data Method Clustering Alignment Original - 93.6 KMeans (2 clusters) 58.70% - KMeans (5 clusters) 82.61% Affinity Propagation 65.22% (4) Infinite mixture model 78.26% (2)
- 85. JAC: ECG Heart Data Method Clustering Alignment Original - 93.6 KMeans (2 clusters) 58.70% - KMeans (5 clusters) 82.61% Affinity Propagation 65.22% (4) Infinite mixture model 78.26% (2) Congealing (+KMeans, 5 clusters) 76.09% 53.85
- 86. JAC: ECG Heart Data Method Clustering Alignment Original - 93.6 KMeans (2 clusters) 58.70% - KMeans (5 clusters) 82.61% Affinity Propagation 65.22% (4) Infinite mixture model 78.26% (2) Congealing (+KMeans, 5 clusters) 76.09% 53.85 Unsupervised JAC 86.96% (5) 5.93
- 87. JAC: Summary • Effective, large improvements over prior work • Allows finite, semi-supervised, online & distributed variations • Modular implementation
- 88. Thesis Summary Curve Alignment Alignment & Feature Learning Alignment & Clustering
- 90. Future: Closing the Loop • Simultaneous unsupervised alignment, clustering, and feature learning • Update learned features through alignment • Different filters / weights for each cluster • Multi-resolution framework using different network depths
- 91. Research Goal Develop an unsupervised data set-agnostic processing module that includes alignment, clustering, and feature learning
- 92. Thank you. Photo credit: Gary B. Huang
- 93. Previous Work: Procrustes Analysis (Mosier 1939, Hartley & Cattel 1962, Kristof & Wingersky 1971) • Framework for computing optimal matching under transformations • Residual sum-of-squares distance called procrustes statistic • Applied to joint alignment and several other applications ! !
- 94. Previous Work: Procrustes Analysis (Mosier 1939, Hartley & Cattel 1962, Kristof & Wingersky 1971) • Framework for computing optimal matching under transformations • Residual sum-of-squares distance called procrustes statistic • Applied to joint alignment and several other applications • Drawbacks: • Mean may not be appropriate reference • Assumes unimodal data set
- 95. Previous Work: Procrustes Analysis (Mosier 1939, Hartley & Cattel 1962, Kristof & Wingersky 1971)
- 96. Previous Work: Congealing (Binary images of handwritten digits: Miller et al. 2000) before after
- 97. Previous Work: Congealing (Bias removal in MRI scans: Learned-Miller et al. 2004 & 2005) before after
- 98. Previous Work: Congealing • Complex images of cars and faces • 3D MRI scans • Images of Drosophila discs • Facial contour labeling
- 99. Previous Work: Congealing (Grayscale images of faces: Huang et al. 2007 & 2012)
- 100. Previous Work: Congealing (3D MRI scans: Zollei et al. 2005)
- 101. Previous Work: Curves • HMM-based models: Continuous Profile Model & Segmental HMM • Many parameters • Assumes unimodal data set • Regression-based models:Transformed mixture of regression • Requires # of clusters • Only linear transformations
- 102. Previous Work: Images • Congealing variants: • Specific to image domains • Improved performance on digit and simple face data sets • Clustering-based models
- 103. h(t) = 1 Z Z t 0 exp ✓Z r=t 0 w(s)ds ◆ dr = (w(t)) Curve Alignment:Transformation • is a smooth monotone function • Utilize monotonicity operator of Ramsey (1998) y(t) = ↵g(t0 )x(t0 ) + t0 = h(t) linear amplitude scaling non-linear time warping non-linear amplitude scaling amplitude translation h(t) d2 h dt2 = w dh dt
- 104. h(t) = 1 Z Z t 0 exp ✓Z r=t 0 w(s)ds ◆ dr = (w(t)) Curve Alignment:Transformation • is a smooth monotone function • Utilize monotonicity operator of Ramsey (1998) y(t) = ↵g(t0 )x(t0 ) + t0 = h(t) linear amplitude scaling non-linear time warping non-linear amplitude scaling amplitude translation h(t) d2 h dt2 = w dh dt unconstrained function monotonicity operator
- 105. Curve Congealing: Drawbacks Co-alignment original mean supervised unsupervised
- 106. Curve Congealing: Drawbacks Complex multi-modal 50 100 150 200 250 −3 −2 −1 0 1 2 3 4 Original Data Set 50 100 150 200 250 −3 −2 −1 0 1 2 3 4 with Congealing
- 107. CRBM Alignment: Overview • Train a convolutional deep belief network using unaligned images ! ! ! • Use pooling activations as feature representation • Use (stochastic) congealing algorithm CRBM (1-layer) CDBN (2-layer)
- 108. JAC:Toy Example (2) Congealing (3) Clustering (4) Alignment and Clustering (1) Original
- 109. JAC: Bayesian Alignment 8i=1:N ⇢i ⇠ p(yi | y i; )p(⇢i | ⇢ i; ↵) Collapsed (Rao-Blackwellized) Gibbs Sampler
- 110. JAC: Bayesian Alignment 8i=1:N ⇢i ⇠ p(yi | y i; )p(⇢i | ⇢ i; ↵) ⇢i = arg max ⇢i p(yi | y i; )p(⇢i | ⇢ i; ↵) Collapsed (Rao-Blackwellized) Gibbs Sampler
- 111. JAC: Bayesian Alignment 8i=1:N ⇢i ⇠ p(yi | y i; )p(⇢i | ⇢ i; ↵) ⇢i = arg max ⇢i p(yi | y i; )p(⇢i | ⇢ i; ↵) data term transformation term Collapsed (Rao-Blackwellized) Gibbs Sampler
- 112. JAC:Alignment & Clustering Second sampler integrates out transformations zi ⇠ p(zi | z i; )p(xi | x i, z; , ↵) p(xi | x i, z; , ↵) ⇡ P l wlp(xi | ⇢l, ✓zi ; ) P l wl s.t. 8l=1:L ⇢l ⇠ q(⇢) & wl = p(⇢l | 'zi ; ↵) q(⇢l)
- 113. JAC: Implementation • visDataSet: represents data set (e.g. images, curves) • visProcessor: handles feature representation (e.g. HOG) • visTransform: represents transformation function (e.g. affine trans) • visSufficientStats: represents exponential family members and priors • visAlignment: base class for alignment algorithms • visMixtureModelBase: base class for DP-based clustering/alignment
- 119. JAC: Implementation visDataSet visSufficientStatsvisTransforms visProcessor visImageTransforms visCurveTransforms visRotatePointTransforms visVectorDataSet visImageDataSet visDiscretize visHOG visCDBN visMultinomial visGaussian
- 120. Thank you. Photo credit: Gary B. Huang
- 121. Thank you. Photo credit: Gary B. Huang