Tutorial APS 2023: Phase transition for statistical estimation: algorithms and fundamental limits.
1 like289 views
The document discusses phase transition in statistical estimation, highlighting the Approximate Message Passing (AMP) algorithm and its applications, particularly in high-dimensional statistics and random matrix theory. It covers key concepts such as the performance of Principal Component Analysis (PCA) under various conditions, Wigner's semicircle law, and optimal denoising techniques. The conclusion emphasizes the adaptability of AMP in various fields beyond electrical engineering, including ecology.
![AMP and its state evolution
Given a matrix W ∈ Rn×n and scalar functions ft : R → R, let
x0 ∈ Rn and
xt+1
= Wft(xt
) − btft−1(xt−1
) ∈ Rn
where
bt =
1
n
n
X
i=1
f 0
t (xt
i ) ∈ R.
If W ∼ GOE(n), ft are Lipschitz and the components of x0 are i.i.d
∼ X0 with E
X2
0
= 1, then for any nice test function Ψ : Rt → R,
1
n
n
X
i=1
Ψ
x1
i , . . . , xt
i
→ E [Ψ(Z1, . . . , Zt)] ,
where (Z1, . . . , Zt)
d
= (σ1G1, . . . , σtGt), where Gs ∼ N(0, 1) i.i.d.
(Bayati Montanari ’11)](https://image.slidesharecdn.com/tutorial-230701184223-71efd420/85/Tutorial-APS-2023-Phase-transition-for-statistical-estimation-algorithms-and-fundamental-limits-15-320.jpg)
![AMP proof of Wigner’s semicircle law
xt+1
= Wxt
− xt−1
In this case, AMP state evolution gives
1
n
hxs
, xt
i → E [ZsZt] = 1(s = t)](https://image.slidesharecdn.com/tutorial-230701184223-71efd420/85/Tutorial-APS-2023-Phase-transition-for-statistical-estimation-algorithms-and-fundamental-limits-20-320.jpg)
![AMP proof of Wigner’s semicircle law
xt+1
= Wxt
− xt−1
In this case, AMP state evolution gives
1
n
hxs
, xt
i → E [ZsZt] = 1(s = t)
Since 1
n hxs, xti ≈ trPs(W)Pt(Wt), the polynomials Pt are
orthonormal w.r.t the limit empirical spectral distribution of W
which must be µSC.](https://image.slidesharecdn.com/tutorial-230701184223-71efd420/85/Tutorial-APS-2023-Phase-transition-for-statistical-estimation-algorithms-and-fundamental-limits-21-320.jpg)
![AMP proof of Wigner’s semicircle law
xt+1
= Wxt
− xt−1
In this case, AMP state evolution gives
1
n
hxs
, xt
i → E [ZsZt] = 1(s = t)
Since 1
n hxs, xti ≈ trPs(W)Pt(Wt), the polynomials Pt are
orthonormal w.r.t the limit empirical spectral distribution of W
which must be µSC.
Credit: Zhou Fan.](https://image.slidesharecdn.com/tutorial-230701184223-71efd420/85/Tutorial-APS-2023-Phase-transition-for-statistical-estimation-algorithms-and-fundamental-limits-22-320.jpg)
![Bayes optimal AMP
We define mmse(γ) = E
h
X0 − E[X0|
√
γX0 + Z]
2
i
and the
recursion:
q0 = 1 − λ−1
qt+1 = 1 − mmse(λqt).
With the optimal denoiser gP0 (y, γ) = E[X0|
√
γX0 + Z = y], AMP
is defined by:
xt+1
= Y
s
λ
n
ft(xt
) − λbtft−1(xt−1
),
where ft(y) = gP0 (y/
√
λqt, λqt).](https://image.slidesharecdn.com/tutorial-230701184223-71efd420/85/Tutorial-APS-2023-Phase-transition-for-statistical-estimation-algorithms-and-fundamental-limits-41-320.jpg)
Tutorial APS 2023: Phase transition for statistical estimation: algorithms and fundamental limits.
- 1. Phase transition for statistical estimation: algorithms and fundamental limits Marc Lelarge INRIA-ENS APS - INFORMS 2023
- 2. A bit of history: 70’s
- 3. A bit of history: 70’s
- 4. A bit of history: 2010’s
- 5. A bit of history: 2010’s
- 6. A bit of history: 2010’s
- 7. Applications to high-dimensional statistics
- 10. This tutorial: demystifying statistical physics! - A simple version of AMP algoritm
- 11. This tutorial: demystifying statistical physics! - A simple version of AMP algoritm - Gap between information-theoretically optimal and computationally feasible estimators
- 12. This tutorial: demystifying statistical physics! - A simple version of AMP algoritm - Gap between information-theoretically optimal and computationally feasible estimators - Running example: matrix model I connection to random matrix theory I sparse PCA, community detection, Z2 synchronization, submatrix localization, hidden clique...
- 14. AMP and its state evolution Given a matrix W ∈ Rn×n and scalar functions ft : R → R, let x0 ∈ Rn and xt+1 = Wft(xt ) − btft−1(xt−1 ) ∈ Rn where bt = 1 n n X i=1 f 0 t (xt i ) ∈ R.
- 15. AMP and its state evolution Given a matrix W ∈ Rn×n and scalar functions ft : R → R, let x0 ∈ Rn and xt+1 = Wft(xt ) − btft−1(xt−1 ) ∈ Rn where bt = 1 n n X i=1 f 0 t (xt i ) ∈ R. If W ∼ GOE(n), ft are Lipschitz and the components of x0 are i.i.d ∼ X0 with E X2 0 = 1, then for any nice test function Ψ : Rt → R, 1 n n X i=1 Ψ x1 i , . . . , xt i → E [Ψ(Z1, . . . , Zt)] , where (Z1, . . . , Zt) d = (σ1G1, . . . , σtGt), where Gs ∼ N(0, 1) i.i.d. (Bayati Montanari ’11)
- 16. Sanity check We have x1 = Wf0(x0) so that x1 i = X j Wijf0(x0 j ), where Wij ∼ N(0, 1/n) i.i.d. (ignore diagonal terms). Hence x1 is a centred Gaussian vector with entries having variance 1 N X j f0(x0 j )2 ≈ E h f0(X0)2 i = σ1.
- 17. AMP proof of Wigner’s semicircle law Consider AMP with linear functions ft(x) = x, so that x1 = Wx0 x2 = Wx1 − x0 = (W2 − Id)x0 x3 = Wx2 − x1 = (W3 − 2W)x0 ,
- 18. AMP proof of Wigner’s semicircle law Consider AMP with linear functions ft(x) = x, so that x1 = Wx0 x2 = Wx1 − x0 = (W2 − Id)x0 x3 = Wx2 − x1 = (W3 − 2W)x0 , so xt = Pt(W)x0 with P0(x) = 1, P1(x) = x Pt+1(x) = xPt(x) − Pt−1(x). {Pt} are Chebyshev polynomials orthonormal wr.t. the semicircle density µSC(x) = 1 2π q (4 − x2)+.
- 19. AMP proof of Wigner’s semicircle law Consider AMP with linear functions ft(x) = x, so that x1 = Wx0 x2 = Wx1 − x0 = (W2 − Id)x0 x3 = Wx2 − x1 = (W3 − 2W)x0 , so xt = Pt(W)x0 with P0(x) = 1, P1(x) = x Pt+1(x) = xPt(x) − Pt−1(x). {Pt} are Chebyshev polynomials orthonormal wr.t. the semicircle density µSC(x) = 1 2π q (4 − x2)+. When 1 n kx0k = 1, we have 1 n hxs, xti ≈ trPs(W)Pt(Wt).
- 20. AMP proof of Wigner’s semicircle law xt+1 = Wxt − xt−1 In this case, AMP state evolution gives 1 n hxs , xt i → E [ZsZt] = 1(s = t)
- 21. AMP proof of Wigner’s semicircle law xt+1 = Wxt − xt−1 In this case, AMP state evolution gives 1 n hxs , xt i → E [ZsZt] = 1(s = t) Since 1 n hxs, xti ≈ trPs(W)Pt(Wt), the polynomials Pt are orthonormal w.r.t the limit empirical spectral distribution of W which must be µSC.
- 22. AMP proof of Wigner’s semicircle law xt+1 = Wxt − xt−1 In this case, AMP state evolution gives 1 n hxs , xt i → E [ZsZt] = 1(s = t) Since 1 n hxs, xti ≈ trPs(W)Pt(Wt), the polynomials Pt are orthonormal w.r.t the limit empirical spectral distribution of W which must be µSC. Credit: Zhou Fan.
- 23. Wigner’s semicircle law: experiments
- 24. Wigner’s semicircle law: experiments
- 25. Wigner’s semicircle law: experiments
- 26. Wigner’s semicircle law: experiments
- 27. Explaining the Onsager term xt+1 = Wxt − xt−1 The first iteration with an Onsager term appears for t = 2.
- 28. Explaining the Onsager term xt+1 = Wxt − xt−1 The first iteration with an Onsager term appears for t = 2. Then we have x2 = Wx1 − x0 = W2x0 − x0 so that x2 1 = X i W 2 1i x0 1 + X i,j6=1 W1i Wijx0 j − x0 1 = X i W 2 1i x0 1 + X i,j6=1 W1i Wijx0 j | {z } N(0,1) − x0 1
- 29. Explaining the Onsager term xt+1 = Wxt − xt−1 The first iteration with an Onsager term appears for t = 2. Then we have x2 = Wx1 − x0 = W2x0 − x0 so that x2 1 = X i W 2 1i x0 1 + X i,j6=1 W1i Wijx0 j − x0 1 = X i W 2 1i x0 1 + X i,j6=1 W1i Wijx0 j | {z } N(0,1) − x0 1 The Onsager term is very similar to the Itô-correction in stochastic calculus.
- 30. This tutorial: demystifying statistical physics! - A simple version of AMP algoritm - Gap between information-theoretically optimal and computationally feasible estimators - Running example: matrix model I connection to random matrix theory I sparse PCA, community detection, Z2 synchronization, submatrix localization, hidden clique...
- 31. Low-rank matrix estimation “Spiked Wigner” model Y |{z} observations = v u u u t λ n XX| | {z } signal + Z |{z} noise I X: vector of dimension n with entries Xi i.i.d. ∼ P0. EX1 = 0, EX2 1 = 1. I Zi,j = Zj,i i.i.d. ∼ N(0, 1). I λ: signal-to-noise ratio. I λ and P0 are known by the statistician. Goal: recover the low-rank matrix XX| from Y.
- 32. Principal component analysis (PCA) Spectral estimator: Estimate X using the eigenvector x̂n associated with the largest eigenvalue µn of Y/ √ n.
- 33. Principal component analysis (PCA) Spectral estimator: Estimate X using the eigenvector x̂n associated with the largest eigenvalue µn of Y/ √ n. B.B.P. phase transition I if λ 6 1 µn a.s. − − − → n→∞ 2 X · x̂n a.s. − − − → n→∞ 0 I if λ 1 µn a.s. − − − → n→∞ √ λ + 1 √ λ 2 |X · x̂n| a.s. − − − → n→∞ p 1 − 1/λ 0 (Baik, Ben Arous, Péché ’05)
- 34. Questions I PCA fails when λ 6 1, but is it still possible to recover the signal?
- 35. Questions I PCA fails when λ 6 1, but is it still possible to recover the signal? I When λ 1, is PCA optimal?
- 36. Questions I PCA fails when λ 6 1, but is it still possible to recover the signal? I When λ 1, is PCA optimal? I More generally, what is the best achievable estimation performance in both regimes?
- 37. Plot of MMSE Figure: Spiked Wigner model, centred binary prior (unit variance).
- 38. We can certainly improve spectral algorithm!
- 39. A scalar denoising problem For Y = √ γX0 + Z where X0 ∼ P0 and Z ∼ N(0, 1)
- 40. A scalar denoising problem For Y = √ γX0 + Z where X0 ∼ P0 and Z ∼ N(0, 1)
- 41. Bayes optimal AMP We define mmse(γ) = E h X0 − E[X0| √ γX0 + Z] 2 i and the recursion: q0 = 1 − λ−1 qt+1 = 1 − mmse(λqt). With the optimal denoiser gP0 (y, γ) = E[X0| √ γX0 + Z = y], AMP is defined by: xt+1 = Y s λ n ft(xt ) − λbtft−1(xt−1 ), where ft(y) = gP0 (y/ √ λqt, λqt).
- 42. Bayes optimal AMP: experiment
- 43. Plot of MMSE Figure: Spiked Wigner model, centred binary prior (unit variance).
- 44. Plot of MMSE Figure: Spiked Wigner model, centred binary prior (unit variance).
- 45. Limiting formula for the MMSE Theorem (L, Miolane ’19) MMSEn − − − → n→∞ 1 |{z} Dummy MSE − q∗ (λ)2 where q∗ (λ) is the minimizer of q 0 7→ −EX0∼P0 Z0∼N log Z x0 dP0(x0)e √ λqZ0x0+λqX0x0+λq 2 x2 0 + λ 4 q2 A simplified “free energy landscape”: 0.0 0.2 0.4 0.6 0.8 1.0 q −0.06 −0.05 −0.04 −0.03 −0.02 −0.01 0.00 −F(λ, q) (a) “Easy” phase (λ = 1.01) 0.0 0.2 0.4 0.6 0.8 q −0.002 −0.001 0.000 0.001 0.002 0.003 −F(λ, q) (b) “Hard” phase (λ = 0.625) 0.0 0.2 0.4 0.6 0.8 q 0.0000 0.0025 0.0050 0.0075 0.0100 0.0125 0.0150 0.0175 −F(λ, q) (c) “Impossible” phase (λ = 0.5)
- 46. Phase diagram Figure: Spiked Wigner model, centred binary prior (unit variance).
- 47. Proof ideas: a planted spin system P(X = x | Y) = 1 Zn P0(x)eHn(x) where Hn(x) = X ij s λ n Yi,jxi xj − λ 2n x2 i x2 j .
- 48. Proof ideas: a planted spin system P(X = x | Y) = 1 Zn P0(x)eHn(x) where Hn(x) = X ij s λ n Yi,jxi xj − λ 2n x2 i x2 j . Two step proof: I Lower bound: Guerra’s interpolation technique. Adapted in (Korada, Macris ’09) (Krzakala, Xu, Zdeborová ’16) ( Y = √ t p λ/n XX| + Z Y0 = √ 1 − t √ λ X + Z0 I Upper bound: Cavity computations (Mézard, Parisi, Virasoro ’87). Aizenman-Sims-Starr scheme:(Aizenman, Sims,Starr ’03) (Talagrand ’10)
- 49. Conclusion AMP is an iterative denoising algorithm which is optimal when the energy landscape is simple. Main references for this tutorial: (Montanari Venkataramanan ’21) (L. Miolane ’19) Many recent research directions: universality, structured matrices, composition... and new applications outside electrical engineering like in ecology.
- 50. Conclusion AMP is an iterative denoising algorithm which is optimal when the energy landscape is simple. Main references for this tutorial: (Montanari Venkataramanan ’21) (L. Miolane ’19) Many recent research directions: universality, structured matrices, composition... and new applications outside electrical engineering like in ecology. Deep learning, the new kid on the block:
- 51. Thank you for your attention !