Talk icml
1 like473 views
This document presents an online dictionary learning algorithm for sparse coding. It proposes solving the dictionary learning problem using an online approach that can handle large datasets and adapt to dynamic training sets much faster than batch algorithms. The online dictionary learning algorithm alternates between sparse coding and dictionary learning steps on minibatches of data. Experimental results show it converges much faster than batch approaches and other online methods like SGD. The algorithm is also extended to problems like NMF and sparse PCA.
![The Dictionary Learning Problem
[Elad & Aharon (’06)]
Solving the denoising problem
Extract all overlapping 8 × 8 patches xi .
Solve a matrix factorization problem:
n 1
min ||x − Dαi ||2 + λ||αi ||1 ,
αi ,D∈C i=1 2 i 2
sparsity
reconstruction
with n > 100, 000
Average the reconstruction of each patch.](https://image.slidesharecdn.com/talkicml-121203135648-phpapp01/85/Talk-icml-6-320.jpg)
![The Dictionary Learning Problem
[Mairal, Bach, Ponce, Sapiro & Zisserman (’09)]
Denoising result](https://image.slidesharecdn.com/talkicml-121203135648-phpapp01/85/Talk-icml-7-320.jpg)
![The Dictionary Learning Problem
[Mairal, Sapiro & Elad (’08)]
Image completion example](https://image.slidesharecdn.com/talkicml-121203135648-phpapp01/85/Talk-icml-8-320.jpg)
![Online Dictionary Learning
Which formulation are we interested in?
1 n
min f (D) = Ex [l(x, D)] ≈ lim l(xi , D)
D∈C n→+∞ n
i=1](https://image.slidesharecdn.com/talkicml-121203135648-phpapp01/85/Talk-icml-13-320.jpg)
![Online Dictionary Learning
Online learning can
handle potentially infinite datasets,
adapt to dynamic training sets,
be dramatically faster than batch
algorithms [Bottou & Bousquet (’08)].](https://image.slidesharecdn.com/talkicml-121203135648-phpapp01/85/Talk-icml-14-320.jpg)
Talk icml
- 1. Online Dictionary Learning for Sparse Coding Julien Mairal1 Francis Bach1 Jean Ponce2 Guillermo Sapiro3 1 INRIA–Willow project 2 Ecole Normale Supérieure 3 University of Minnesota ICML, Montréal, June 2008
- 2. What this talk is about Learning efficiently dictionaries (basis set) for sparse coding. Solving a large-scale matrix factorization problem. Making some large-scale image processing problems tractable. Proposing an algorithm which extends to NMF, sparse PCA,. . .
- 3. 1 The Dictionary Learning Problem 2 Online Dictionary Learning 3 Extensions
- 4. 1 The Dictionary Learning Problem 2 Online Dictionary Learning 3 Extensions
- 5. The Dictionary Learning Problem y = xorig + w measurements original image noise
- 6. The Dictionary Learning Problem [Elad & Aharon (’06)] Solving the denoising problem Extract all overlapping 8 × 8 patches xi . Solve a matrix factorization problem: n 1 min ||x − Dαi ||2 + λ||αi ||1 , αi ,D∈C i=1 2 i 2 sparsity reconstruction with n > 100, 000 Average the reconstruction of each patch.
- 7. The Dictionary Learning Problem [Mairal, Bach, Ponce, Sapiro & Zisserman (’09)] Denoising result
- 8. The Dictionary Learning Problem [Mairal, Sapiro & Elad (’08)] Image completion example
- 9. The Dictionary Learning Problem What does D look like?
- 10. The Dictionary Learning Problem n 1 min ||xi − Dαi ||2 + λ||αi ||1 2 α∈R i=1 2 k×n D∈C C = {D ∈ Rm×k s.t. ∀j = 1, . . . , k, ||dj ||2 ≤ 1}. Classical optimization alternates between D and α. Good results, but very slow!
- 11. 1 The Dictionary Learning Problem 2 Online Dictionary Learning 3 Extensions
- 12. Online Dictionary Learning Classical formulation of dictionary learning 1 n min fn (D) = min l(xi , D), D∈C D∈C n i=1 where 1 l(x, D) = mink ||x − Dα||2 + λ||α||1 . 2 α∈R 2
- 13. Online Dictionary Learning Which formulation are we interested in? 1 n min f (D) = Ex [l(x, D)] ≈ lim l(xi , D) D∈C n→+∞ n i=1
- 14. Online Dictionary Learning Online learning can handle potentially infinite datasets, adapt to dynamic training sets, be dramatically faster than batch algorithms [Bottou & Bousquet (’08)].
- 15. Online Dictionary Learning Proposed approach 1: for t=1,. . . ,T do 2: Draw xt 3: Sparse Coding 1 αt ← arg min ||xt − Dt−1 α||2 + λ||α||1 , 2 α∈Rk 2 4: Dictionary Learning 1 t 1 Dt ← arg min ||xi −Dαi ||2 +λ||αi ||1 , 2 D∈C t i=1 2 5: end for
- 16. Online Dictionary Learning Proposed approach Implementation details Use LARS for the sparse coding step, Use a block-coordinate approach for the dictionary update, with warm restart, Use a mini-batch.
- 17. Online Dictionary Learning Proposed approach Which guarantees do we have? Under a few reasonable assumptions, ˆ we build a surrogate function ft of the expected cost f verifying ˆ lim ft (Dt ) − f (Dt ) = 0, t→+∞ Dt is asymptotically close to a stationary point.
- 18. Online Dictionary Learning Experimental results, batch vs online m = 8 × 8, k = 256
- 19. Online Dictionary Learning Experimental results, batch vs online m = 12 × 12 × 3, k = 512
- 20. Online Dictionary Learning Experimental results, batch vs online m = 16 × 16, k = 1024
- 21. Online Dictionary Learning Experimental results, ODL vs SGD m = 8 × 8, k = 256
- 22. Online Dictionary Learning Experimental results, ODL vs SGD m = 12 × 12 × 3, k = 512
- 23. Online Dictionary Learning Experimental results, ODL vs SGD m = 16 × 16, k = 1024
- 24. Online Dictionary Learning Inpainting a 12-Mpixel photograph
- 25. Online Dictionary Learning Inpainting a 12-Mpixel photograph
- 26. Online Dictionary Learning Inpainting a 12-Mpixel photograph
- 27. Online Dictionary Learning Inpainting a 12-Mpixel photograph
- 28. 1 The Dictionary Learning Problem 2 Online Dictionary Learning 3 Extensions
- 29. Extension to NMF and sparse PCA NMF extension 1n min ||xi − Dαi ||2 s.t. αi ≥ 0, 2 D ≥ 0. α∈R i=1 2 k×n D∈C SPCA extension n 1 min ||xi − Dαi ||2 + λ||α1 ||1 2 α∈Rk×n i=1 2 D∈C C = {D ∈ Rm×k s.t. ∀j ||dj ||2 + γ||dj ||1 ≤ 1}. 2
- 30. Extension to NMF and sparse PCA Faces: Extended Yale Database B (a) PCA (b) NNMF (c) DL
- 31. Extension to NMF and sparse PCA Faces: Extended Yale Database B (d) SPCA, τ = 70% (e) SPCA, τ = 30% (f) SPCA, τ = 10%
- 32. Extension to NMF and sparse PCA Natural Patches (a) PCA (b) NNMF (c) DL
- 33. Extension to NMF and sparse PCA Natural Patches (d) SPCA, τ = 70% (e) SPCA, τ = 30% (f) SPCA, τ = 10%
- 34. Conclusion Take-home message Online techniques are adapted to the dictionary learning problem. Our method makes some large-scale image processing tasks tractable—. . . . . . — and extends to various matrix factorization problems.