QMC: Operator Splitting Workshop, Forward-Backward Splitting Algorithm without Cocoercivity - Saverio Salzo, Mar 22, 2018
0 likes197 views
The document discusses the forward-backward splitting algorithm for finding the zeros of the sum of maximal monotone operators. It notes that the standard convergence analysis relies on one of the operators being cocoercive. It proposes investigating strategies for the algorithm that do not require cocoercivity, such as using backtracking line searches to determine the step size or involving the duality gap function in the backtracking condition. Removing the cocoercivity assumption would expand the applicability of the forward-backward splitting method.
![is a real Hilbert space with scalar product
is maximal monotone if is
proper, convex, and lower semicontinuous.
is -cocoercive iff it is -Lipschitz
continuous.
MINIMIZATION OF SUM OF FUNCTIONS
The problem
h· , ·i
find ¯x 2 H s.t. 0 2 @f(¯x) + rg(¯x)
@f : H ! 2H f : H ! ] 1, +1]
rg: H ! H (1/ )
H](https://image.slidesharecdn.com/samsi2018salzo-180322204148/85/QMC-Operator-Splitting-Workshop-Forward-Backward-Splitting-Algorithm-without-Cocoercivity-Saverio-Salzo-Mar-22-2018-3-320.jpg)
![TSENG’S SPLITTING
(Tseng ’00)
is maximal monotone
is uniformly continuous on bounded sets
is chosen by the backtracking procedure: let
B : H ! H
k
kB(yk) B(xk)k
k
kyk xkk
2 ]0, 1[
The algorithm converges if
A: H ! 2H
yk = J kA(xk kB(xk))
xk+1 = yk kB(yk) + kB(xk)
Strategies without cocoercivity](https://image.slidesharecdn.com/samsi2018salzo-180322204148/85/QMC-Operator-Splitting-Workshop-Forward-Backward-Splitting-Algorithm-without-Cocoercivity-Saverio-Salzo-Mar-22-2018-8-320.jpg)
![, is proper, convex, and lsc
is uniformly continuous on bounded sets.
is determined by backtracking line search procedures
(a priori choices are not possible anymore).
THE FORWARD-BACKWARD SPLITTING
FOR MINIMIZATION PROBLEMS
xk+1 = J kA(xk kB(xk))
B = rg
A = @f f : H ! ] 1, +1]
k
The algorithm converges if
S. Salzo, The variable metric forward-backward splitting algorithm under mild
differentiability assumptions. SIOPT 2017.
Strategies without cocoercivity](https://image.slidesharecdn.com/samsi2018salzo-180322204148/85/QMC-Operator-Splitting-Workshop-Forward-Backward-Splitting-Algorithm-without-Cocoercivity-Saverio-Salzo-Mar-22-2018-9-320.jpg)
![BACKTRACKING LINE SEARCHES
L1)
L2)
L3)
We considered three possible inequalities: let
krg(xk+1) rg(xk)k
k
kxk+1 xkk
g(xk+1) g(xk) hxk+1 xk, rg(xk)i
k
kxk+1 xkk2
(f + g)(xk+1) (f + g)(xk)
(1 ) f(xk+1) f(xk) + hxk+1 xk, rg(xk)i
2 ]0, 1[
L1) L2) L3)
L1) is not quite appropriate for FB algorithm since it leads
to halve the step-sizes w.r.t. L2) and L3)
=) =)
Strategies without cocoercivity](https://image.slidesharecdn.com/samsi2018salzo-180322204148/85/QMC-Operator-Splitting-Workshop-Forward-Backward-Splitting-Algorithm-without-Cocoercivity-Saverio-Salzo-Mar-22-2018-10-320.jpg)
QMC: Operator Splitting Workshop, Forward-Backward Splitting Algorithm without Cocoercivity - Saverio Salzo, Mar 22, 2018
- 1. THE FORWARD-BACKWARD SPLITTING ALGORITHM WITHOUT COCOERCIVITY? Saverio Salzo Istituto Italiano di Tecnologia Genova, Italy Workshop on Operator Splitting Methods in Data Analysis Samsi, Durham, NC, March, 21-23, 2018
- 2. is a real Hilbert space with scalar product is maximal monotone is -cocoercive, that is ZEROS OF SUM OF MONOTONE OPERATORS The problem H h· , ·i A: H ! 2H B : H ! H find ¯x 2 H s.t. 0 2 A(¯x) + B(¯x) kB(x) B(y)k2 hx y, B(x) B(y)i
- 3. is a real Hilbert space with scalar product is maximal monotone if is proper, convex, and lower semicontinuous. is -cocoercive iff it is -Lipschitz continuous. MINIMIZATION OF SUM OF FUNCTIONS The problem h· , ·i find ¯x 2 H s.t. 0 2 @f(¯x) + rg(¯x) @f : H ! 2H f : H ! ] 1, +1] rg: H ! H (1/ ) H
- 4. is a real Hilbert space with scalar product is maximal monotone if is nonempty, closed, and convex. is -cocoercive. VARIATIONAL INEQUALITIES The problem find ¯x 2 C s.t. 8 x 2 C h¯x x, B¯xi 0 A = @iC = NC C ⇢ H B : H ! H h· , ·iH
- 5. THE FORWARD-BACKWARD SPLITTING (Mercier ’79) The algorithm if and , then xk+1 = J kA(xk kB(xk)) Convergence: if k < 2 xk * ¯x 2 zer(A + B) A = @f B = rg (f + g)(xk) inf(f + g) = o(1/k) Cocoercivity of is a fundamental assumption!B
- 6. minimize x2Rd + DKL(b, Ax) + ⌧kxk1 minimize x2Rd 1 p kAx bkp p + ⌧kxk1 (p > 1) minimize w2Rd nX i=1 L(yi, hw, xii) + ⌧ p kwkp p (1 < p < 2) When cocoercivity is an issue
- 7. minimize x2Rd 1 p kAx bkp p + ⌧kxk1 (p > 1) minimize ↵2Rn 1 q kX⇤ ↵kq q + 1 ⌧ nX i=1 L⇤ (yi, ⌧↵i) (q > 2) When cocoercivity is an issue minimize x2Rd + DKL(b, Ax) + ⌧kxk1
- 8. TSENG’S SPLITTING (Tseng ’00) is maximal monotone is uniformly continuous on bounded sets is chosen by the backtracking procedure: let B : H ! H k kB(yk) B(xk)k k kyk xkk 2 ]0, 1[ The algorithm converges if A: H ! 2H yk = J kA(xk kB(xk)) xk+1 = yk kB(yk) + kB(xk) Strategies without cocoercivity
- 9. , is proper, convex, and lsc is uniformly continuous on bounded sets. is determined by backtracking line search procedures (a priori choices are not possible anymore). THE FORWARD-BACKWARD SPLITTING FOR MINIMIZATION PROBLEMS xk+1 = J kA(xk kB(xk)) B = rg A = @f f : H ! ] 1, +1] k The algorithm converges if S. Salzo, The variable metric forward-backward splitting algorithm under mild differentiability assumptions. SIOPT 2017. Strategies without cocoercivity
- 10. BACKTRACKING LINE SEARCHES L1) L2) L3) We considered three possible inequalities: let krg(xk+1) rg(xk)k k kxk+1 xkk g(xk+1) g(xk) hxk+1 xk, rg(xk)i k kxk+1 xkk2 (f + g)(xk+1) (f + g)(xk) (1 ) f(xk+1) f(xk) + hxk+1 xk, rg(xk)i 2 ]0, 1[ L1) L2) L3) L1) is not quite appropriate for FB algorithm since it leads to halve the step-sizes w.r.t. L2) and L3) =) =) Strategies without cocoercivity
- 11. FBA FBFA They work under exactly the same assumptions! MINIMIZATION PROBLEMS Strategies without cocoercivity
- 12. THE PROBLEM: DEVISE AN APPROPRIATE BACKTRACKING LINESEARCH THAT REPLACES THE COCOERCIVITY What about the FB algorithm for monotone operators? Removing cocoercivity
- 13. Where does cocoercivity enter? kxk+1 ¯xk2 kxk ¯xk2 kxk+1 xkk2 2 khxk ¯x, B(xk) B(¯x)i 2 khxk+1 xk, B(xk) B(¯x)i = kxk ¯xk2 k(2 k)kB(xk) B(¯x)k2 k(xk+1 xk) k(B(xk) B(¯x))k2 FB ALGORITHM FOR MONOTONE OPERATORS
- 14. Where does cocoercivity enter? FB ALGORITHM FOR MINIMIZATION PROBLEMS kxk+1 xk2 kxk xk2 kxk+1 xkk2 + 2 k (f + g)(x) (f + g)(xk) 2 k g(xk+1) g(xk) + hxk+1 xk, rg(xk)i kxk xk2 + (2 1)kxk+1 xkk2 + 2 k (f + g)(x) (f + g)(xk) + 2 k (f + g)(xk) (f + g)(xk+1)
- 15. Possible strategies only value of B? would not work! the values of function (to be determined), that hopefully should decrease along the iterations? k kB(xk+1) B(xk)k2 hB(xk+1) B(xk), xk+1 xki The inequality should involve:
- 16. , Possible strategies PRIMAL DUAL ALGORITHMS min x2H g(x) + h(Lx) ✓ 0 0 ◆ 2 A ✓ ¯x ¯v ◆ + B ✓ ¯x ¯v ◆ A ✓ x v ◆ = ✓ L⇤ v @h⇤ (v) Lx ◆ B ✓ x v ◆ = ✓ rg(x) 0 ◆ is convex and continuously differentiable.g: H ! R (Combettes-Pesquet ’12, Condat ’13, Vu ’13) Should backtracking involve the duality gap function?
- 17. Thank you!