![n
• Sparse Coding
Learned ISTA (LISTA)
n Sparse Coding
• Lasso
n LISTA ISTA
• ISTA: Iterative Soft-Threshold Algorithm
proximal gradient descent
5
[Gregor and LeCun, ICML’10]](https://image.slidesharecdn.com/haraneurips18yomikai190112-190108032205/85/Theoretical-Linear-Convergence-of-Unfolded-ISTA-and-its-Practical-Weights-and-Thresholds-5-320.jpg)
![Sparse Coding Learned ISTA
n Learned ISTA (LISTA) [Gregor and LeCun, ICML’10]
ISTA
n LISTA
•
•
•
8
MNIST LISTA](https://image.slidesharecdn.com/haraneurips18yomikai190112-190108032205/85/Theoretical-Linear-Convergence-of-Unfolded-ISTA-and-its-Practical-Weights-and-Thresholds-8-320.jpg)
![Learned ISTA (LISTA) [Gregor and LeCun, ICML’10]
n ISTA
• !" = ℎ%&
'()
→ !+ = ℎ%,
-"!" + '")
→ !/ = ℎ%0
-+!+ + '+)
→ …
n ISTA
12
-2 = 34 − 6787
'2 = 678
92 = :;
!"
)
!+ !/-" ℎ%, -+ ℎ%0
'(
'" '+
←
↑
ℎ%&
ISTA 3
4 DNN](https://image.slidesharecdn.com/haraneurips18yomikai190112-190108032205/85/Theoretical-Linear-Convergence-of-Unfolded-ISTA-and-its-Practical-Weights-and-Thresholds-12-320.jpg)
![Learned ISTA (LISTA) [Gregor and LeCun, ICML’10]
n LISTA
• !" = $"(&; (, *, +) K+1 DNN
• ISTA -! DNN !"
(, *, + DNN
13
-!
(, *, +
DNN
!.
/
!0 !1(. ℎ34 (0 ℎ35
*6
*. *0
←
↑
ℎ37](https://image.slidesharecdn.com/haraneurips18yomikai190112-190108032205/85/Theoretical-Linear-Convergence-of-Unfolded-ISTA-and-its-Practical-Weights-and-Thresholds-13-320.jpg)
![Learned ISTA (LISTA) [Gregor and LeCun, ICML’10]
n LISTA -
• ! "#
{ !%, "#% }%()
*
• K+1 DNN #+ = -+(!; 0, 1, 2)
•
min
7,8,9
1
2
<=, "> "# − -+(!; 0, 1, 2) @
n LISTA -
• #+ = -+(!; 0, 1, 2)
DNN
14
SGD
MNIST LISTA](https://image.slidesharecdn.com/haraneurips18yomikai190112-190108032205/85/Theoretical-Linear-Convergence-of-Unfolded-ISTA-and-its-Practical-Weights-and-Thresholds-14-320.jpg)
Theoretical Linear Convergence of Unfolded ISTA and its Practical Weights and Thresholds
- 1. 1
- 2. n • ~2013.3, PhD@ , • 2013.4~2016.3, @IBM • 2016.4~2017.8, @ ERATO, NII • 2017.9~, @ , n • ECML’11 AISTATS’15,17 • AISTATS’18 AAAI’17,18 2
- 3. Cont. n • ℓ" n • Lasso min & 1 2 ) − +, - + / , " 3
- 4. Cont. n • ℓ" n • Lasso min & 1 2 ) − +, - + / , " n • Lasso 4
- 5. n • Sparse Coding Learned ISTA (LISTA) n Sparse Coding • Lasso n LISTA ISTA • ISTA: Iterative Soft-Threshold Algorithm proximal gradient descent 5 [Gregor and LeCun, ICML’10]
- 6. Sparse Coding ISTA n / n ISTA Lasso 6 ISTA
- 7. Sparse Coding ISTA n / n ISTA Lasso 7 ISTA
- 8. Sparse Coding Learned ISTA n Learned ISTA (LISTA) [Gregor and LeCun, ICML’10] ISTA n LISTA • • • 8 MNIST LISTA
- 9. n n • ISTA • Learned ISTA (LISTA) n n LISTA n n 9
- 10. Iterative Soft-Thresholding Algorithm (ISTA) n • !∗ ∈ ℝ% ← n • & ∈ ℝ' ← • / ( ∈ ℝ'×% ← !∗ & = (!∗ + , • !∗ n Lasso -! = argmin 4 1 2 & − (! 8 + 9 ! : 10
- 11. Iterative Soft-Thresholding Algorithm (ISTA) n !" = argmin * 1 2 - − /" 0 + 2 " 3 n ISTA Lasso • " ← 06 • " 3 0 - − /" 0 " • " ← " − 7 /8/" − /8- = 96 − 7/8/ " + 7/8- " • " ← ℎ;< " = sign " ⊙ max {0, abs " − 72} 11→
- 12. Learned ISTA (LISTA) [Gregor and LeCun, ICML’10] n ISTA • !" = ℎ%& '() → !+ = ℎ%, -"!" + '") → !/ = ℎ%0 -+!+ + '+) → … n ISTA 12 -2 = 34 − 6787 '2 = 678 92 = :; !" ) !+ !/-" ℎ%, -+ ℎ%0 '( '" '+ ← ↑ ℎ%& ISTA 3 4 DNN
- 13. Learned ISTA (LISTA) [Gregor and LeCun, ICML’10] n LISTA • !" = $"(&; (, *, +) K+1 DNN • ISTA -! DNN !" (, *, + DNN 13 -! (, *, + DNN !. / !0 !1(. ℎ34 (0 ℎ35 *6 *. *0 ← ↑ ℎ37
- 14. Learned ISTA (LISTA) [Gregor and LeCun, ICML’10] n LISTA - • ! "# { !%, "#% }%() * • K+1 DNN #+ = -+(!; 0, 1, 2) • min 7,8,9 1 2 <=, "> "# − -+(!; 0, 1, 2) @ n LISTA - • #+ = -+(!; 0, 1, 2) DNN 14 SGD MNIST LISTA
- 15. n n • ISTA • Learned ISTA (LISTA) n n LISTA n n 15
- 16. Research Question LISTA n 10~20 n • ! = #$∗ + ' (($∗, ') • (($∗ , ') !′ ISTA -$′ 16 ISTA 100
- 17. n • !∗ # !∗ $ ≤ &, !∗ ( ≤ ), # * ≤ + n • ,- = / − 1-2 i.e. !-3* = ℎ56 !- + 1- 8 − 2!- • s 9 !- :, ; > 0 1-, >- !- − !∗ ? ≤ )&exp −:9 + ;+ 17 s 8 = 2!∗ + # !-3* = ℎ56 ,!- + 1-8
- 18. n • !∗ # !∗ $ ≤ &, !∗ ( ≤ ), # * ≤ + n • ,- = / − 12- i.e. !34* = ℎ67 !3 + 2- 9 − 1!3 • : !3 ;, < > 0 2-, ?- !3 − !∗ @ ≤ )&exp −;: + <+ s 9 = 1!∗ + # 18 LISTA DNN
- 19. n • !" = $ − &"' i.e. (")* = ℎ,- (" + &" / − '(" • s 0 (" 1, 3 > 0 &", 6" (" − (∗ 8 ≤ :;exp −10 + 3? n @ = supp (∗ • @ = supp (∗ : = D: (E ∗ ≠ 0 n 2 • &E, 6E 1. (" E = 0, ∀D ∉ @, ∀0 2. (" − (∗ * 0 19
- 20. n Notation • ! ! " # ! $ % • & &"' (#, *) &",: # &:,' * &:,$ % • -. = 01 sup 5∗,7 8. − 8∗ : + <=> 01: = max "B' C. ",:D:,' <=: = max ",' C. "' 20
- 21. 1. n DNN • !"#$ = ℎ'( !" + *" + − -!" n !" . = 0, ∀2 ∉ 4 • + = -!∗ + 6 ∀2 ∉ 4 !"#$ . = ℎ'( !" . + *" .,:- !∗ − !" + *" .,:6 = ℎ'( *" .,:-:,8 !∗ − !" 8 + *" .,:6 n 9" = :; sup ?∗,@ !" − !∗ $ + ABC • :;: = max .GH *" .,:-:,H AB: = max .,H *" .H *" .,:-:,8 !∗ − !" 8 + *" .,:6 ≤ 9" ⇒ !"#$ . = 0 21≤ :; sup ?∗,@ !" − !∗ $ ≤ ABC
- 22. 2. !" − !∗ % & n ' ∈ ) !"*% + = ℎ./ !" + + 1" +,:4:,5 !∗ − !" 5 + 1" +,:6 n 1" +,:4:,+ = 1 1" !" + + 1" +,:4:,5 !∗ − !" 5 = !" + + 1" +,:4:,5∖+ !∗ − !" 5∖+ + !∗ − !" + = !∗ + + 1" +,:4:,5∖+ !∗ − !" 5∖+ !"*% + − !∗ + − 1" +,:4:,5∖+ !∗ − !" 5∖+ − 1" +,:6 ≤ :" !"*% + − !∗ + ≤ 1" +,:4:,5∖+ !∗ − !" 5∖+ + 1" +,:6 + :" ≤ ;< !∗ − !" 5∖+ % + =>? + :" 22 ℎ./ :" shrink
- 23. 2. !" − !∗ % & n !"'% − !∗ % !"'% − !∗ % ≤ ∑*∈, !"'% * − !∗ * ≤ ∑*∈, -. !∗ − !" ,∖* % + 123 + 4" ≤ -. 5 − 1 !∗ − !" % + 5 4" + 5 123 n sup sup :∗,< !"'% − !∗ % ≤ 2-.> − -. sup :∗,< !∗ − !" % + 2>123 ≤ 2-.> − -. "'%sup :∗,< !∗ − !? % + 2>123 ∑@A? "'% 2-.> − -. @ ≤ 2-.> − -. ">B + 13 ≤ >Bexp −E& + 13 23 E = −log 2-.> − -. 1 = JKLM %'NOPJNOK2 -.> − -. < 1 = !* R = 0, ∀U ∉ 5 4" = -. sup :∗,< !" − !∗ % + 123
- 24. n LISTA-CP (LISTA with weights coupling) • LISTA !"#$ = ℎ'( )"!" + +", • LISTA-CP !"#$ = ℎ'( !" + +" , − .!" n • LISTA-CP +", 0" !" − !∗ 2 ≤ 45exp −9: + ;< n LISTA-CP LISTA LISTA )", +", 0" 24
- 25. n !∗ !# − !∗ % n LISTA ISTA &! !∗ !# − !∗ % n !# − &! % • !# − &! % % ≤ !∗ − &! % % + !# − !∗ % % !# − &! % tight 25
- 26. n n • ISTA • Learned ISTA (LISTA) n n LISTA n n 26
- 27. LISTA n ℎ" • ℎ" # $ = sign #$ ⊙ max #$ − / n Support Selection ℎ" 0 • #1 #1 → shrink • ℎ" 0 # $ = 2 #$ ℎ" # $ 27 # shrink #1 3% shrink shrink
- 28. LISTA n LISTA-CPSS • s ! "# $%% ≥ $, (%% ≤ ( *#, +# "# − "∗ . ≤ /0exp −$%%! + (%%5 28 Weights Coupling Weights Coupling Support Selection LISTA "#67 = ℎ:; <#"# + *#= LISTA-CP "#67 = ℎ:; "# + *#(= − ?"#) Support Selection LISTA-SS "#67 = ℎ:; A <#"# + *#= LISTA-CPSS "#67 = ℎ:; A "# + *#(= − ?"#) LIST-CP
- 29. n n • ISTA • Learned ISTA (LISTA) n n LISTA n n 29
- 30. n • !∗ 500 • # 250 • $ n LISTA-CP 30 NMSE / ISTA, FISTA AMP LISTA LISTA-CP NMSE / ISTA, FISTA AMP LISTA LISTA-CP LIST-CP https://github.com/xchen-tamu/linear-lista-cpss
- 31. n • !∗ 500 • # 250 • $ n LISTA-CPSS 31 LIST-CPSS NMSE / LISTA-CPSS NMSE LISTA-CPSS / https://github.com/xchen-tamu/linear-lista-cpss
- 32. n Research Question • Learned ISTA (LISTA) n • LISTA • LISTA weights coupling (CP), support selection (SS) n LISTA-CPSS 32