Matrix Completion Presentation
- 1. Sparse Matrix Reconstruction Michael Hankin University of Southern California mhankin@usc.edu December 5, 2013 Michael Hankin (USC) Matrix Completion December 5, 2013 1 / 28
- 2. Overview 1 Initial Problem 2 Algorithm Algorithm Explanation 3 Convergence 4 Extensions 5 Demos 6 Other topics 7 References Michael Hankin (USC) Matrix Completion December 5, 2013 2 / 28
- 3. Overview of Matrix Completion Problem Motivation: Say that Netflix has NMovies movies and NUsers users. Given universal knowledge they could construct an NMovies xNUsers matrix of ratings and thus predict which movies their users would enjoy, and how much so. However, all they have are the few ratings their users have taken the time to input, and the data on which accounts have watched which movies. Can the full matrix be reconstructed from this VERY sparse, noisy sample? Michael Hankin (USC) Matrix Completion December 5, 2013 3 / 28
- 4. Overview of Matrix Completion Problem Idea: Without some constraint the values of the missing points could be any real (or even complex!) number. Obviously we have to impose some restrictions beginning with real numbers only! Less obvious is the condition that the matrix be of low rank. In the Netflix problem this is obvious: there really aren’t that many types of people (as far a taste profile goes) or movies (as far as genre/appeal profile goes). However this condition is relevant in many other scenarios as well. Michael Hankin (USC) Matrix Completion December 5, 2013 4 / 28
- 5. Notational Interlude For a matrix X define the nuclear norm to be r X ∗ |σi | = i=1 where the σi ’s are the singular values of the matrix and r is its rank (and therefore the number of nonzero singular values). Grievously abusing notation, we might say X ∗ = σ 1 If the true matrix is M and we observe only Mi,j ∀(i, j) ∈ Ω for some Ω then let Xi,j : (i, j) ∈ Ω PΩ (X ) = 0 : (i, j) ∈ Ω / Michael Hankin (USC) Matrix Completion December 5, 2013 5 / 28
- 6. Problem statement Given PΩ (M) and the knowledge that M is of low rank, recover M. To do so we work with an approximation X . We want to minimize the rank of X , however a direct approach would be NP-hard, in the same way that σ 0 would be, so we relax our conditions in the same vein as LASSO, and set up the problem: min X ∗ ≈ min σ 1 (1) s.t. PΩ (M) = PΩ (X ) we end up with something slightly resembling the Dantzig selector, which we know gives sparse results, and sparsity in σ is equivalent to low rank for X. Michael Hankin (USC) Matrix Completion December 5, 2013 6 / 28
- 7. To expand on this notion, consider the Dantzig case. Level sets of 1 can be visually represented as the diamond in the image to the right. 1 In the case of LASSO regression 1 ≤ 1 can be considered to be the convex hull of the single-parameter euclidean basis vectors of unit length. In the nuclear norm case, ∗ ≤ 1 is just the convex hull of the set of rank 1 matrices whose spectral norm X 2 ≤ 1 (keeping in mind that X 2 = σ ∞) The solution to the previous minimization problem is the point at which the smallest level set of the nuclear norm to intersect the subspace {X : PΩ (M) = PΩ (X )} does so. Using the spatial intuition gleaned from our study of LASSO we recognize that this will give a sparse set of singular values, and therefore a low rank matrix, that agrees with M on all of Ω. 1 Credit to Nicolai Meinshausen: http://www.stats.ox.ac.uk/~meinshau/ Michael Hankin (USC) Matrix Completion December 5, 2013 7 / 28
- 8. Algorithm Background Candes, Cai, and Shen introduced an algorithm that comes close to solving our problem. Let X be of low rank r and UΣV ∗ be its SVD, where Σ = diag ({σi }1≤i≤r ) (because it has only r nonzero singular values. N ext they define the soft-thresholding operator: Dτ (X ) = UDτ (Σ)V ∗ Dτ (Σ) = diag ({(σi − τ )+ }1≤i≤r ) for τ > 0, so that it shrinks all of the singular values of X , setting any that were originally ≤ τ to 0, thereby reducing its rank. Note: Dτ (X ) = arg minY 1 Y − X 2 + τ Y F 2 This will affect the output of the algorithm. Michael Hankin (USC) Matrix Completion ∗ . December 5, 2013 8 / 28
- 9. Algorithm Start with some Y 0 that vanishes outside of Ω (an efficient choice for Y 0 will be discussed later, but for now just use 0 or even M.) Choose values for τ > 0 and a sequence δk corresponding to step sizes At step k set X k = Dτ (Y k−1 ) Then set Y k = Y k−1 + δk PΩ (M − X k ) Michael Hankin (USC) Matrix Completion December 5, 2013 9 / 28
- 10. Algorithm Discussion Notes on the algorithm: Low Rank The X k ’s will tend to have low rank, unless to many of the singular values end up growing beyond τ so that further iterations do not lower the rank. Both the authors and I found (empirically) that the rank of the X k ’s tend to start low, and grow to a stable point after a few dozen iterations. As long as the original matrix was of low rank, this stable point also tends to be of low rank. Unfortunately the authors have been unable to prove this. When the dimensions of X k are high, this low rank property allows us to economize on memory by maintaining only the portion of its SVD corresponding to non-0’d singular values instead of the entire, dense matrix itself. Michael Hankin (USC) Matrix Completion December 5, 2013 10 / 28
- 11. Algorithm Discussion Notes on the algorithm: Sparsity The Y k ’s will always be sparse, and vanish outside of Ω. This is obvious because we require that Y 0 be either equal to 0 or at least vanish outside of Ω. PΩ (M − X k ) vanishes outside of Ω by definition, and if we assume Y k−1 does to, then Y k = Y k−1 + PΩ (M − X k ) must have the same property, and is therefore sparse. This lessens our storage requirements (though we must still maintain the dense matrices X k ) but more importantly it makes computing the SVD of Y k much faster as long as clever computational approaches and a sparse solver are used. Michael Hankin (USC) Matrix Completion December 5, 2013 11 / 28
- 12. Proof that algorithm gives a solution to 1 X 2 F 2 s.t. PΩ (M) = PΩ (X ) min τ X Michael Hankin (USC) ∗ + Matrix Completion (2) December 5, 2013 12 / 28
- 13. Convergence Significance Figure : Convergence towards true value for different tau and delta values Michael Hankin (USC) Matrix Completion December 5, 2013 13 / 28
- 14. Convergence Significance Figure : Convergence towards true value for different tau and delta values Michael Hankin (USC) Matrix Completion December 5, 2013 14 / 28
- 15. Convergence Significance As seen in the proof, the algorithm converges to the solution of: 1 X 2 F 2 s.t. PΩ (M) = PΩ (X ) min τ X Michael Hankin (USC) ∗ + Matrix Completion (3) December 5, 2013 15 / 28
- 16. Convergence Significance Why is a solution to 1 X 2 F 2 s.t. PΩ (M) = PΩ (X ) min τ X ∗ + (4) satisfactory when we’re looking for a solution to min X ∗ (5) s.t. PΩ (M) = PΩ (X ) Michael Hankin (USC) Matrix Completion December 5, 2013 16 / 28
- 17. Proof of adequacy in a more general case. Michael Hankin (USC) Matrix Completion December 5, 2013 17 / 28
- 18. Convergence Significance Figure : Convergence towards true value for different tau and delta values Michael Hankin (USC) Matrix Completion December 5, 2013 18 / 28
- 19. General Convex Constraints Cai Candes and Shen extend their algorithm to the more general case, addressed in the previous proof: min fτ (X ) (6) s.t. fi (X ) ≤ 0 ∀i Where the fi (X )’s are convex, left semi-continuous functionals. Michael Hankin (USC) Matrix Completion December 5, 2013 19 / 28
- 20. Generalized Algorithm In that case, the algorithm is as follows: Denote F(X ) = (f1 (X )...fn (X )) and initialize y 0 X k = arg minX fτ (X ) + y k−1 , F(X ) y k = (y k−1 + δk F(X k ))+ In the special case where the constraints are linear, ie A(X ) ≤ b for some linear functional A, the iterations are as follows: X k = Dτ (A∗ (y k−1 )) y k = (y k−1 + δk (b − A(X k ))+ Consider b = {Mi,j }(i,j)∈Ω , A(X ) = {Mi,j }(i,j)∈Ω , and its adjoint A∗ (y ) mapping y to a sparse matrix X with entries only on indices in Ω and values equal to those in y . Michael Hankin (USC) Matrix Completion December 5, 2013 20 / 28
- 21. Use Case Noise! If our data is noisy we can use |Xi,j − Mi,j | < ∀(i, j) ∈ Ω This is Example Triangulation If the matrix in question is distances between points we can fill in the relative locations with just a few entries. Michael Hankin (USC) Matrix Completion December 5, 2013 21 / 28
- 22. Noise Free Figure : Rank 10 matrix Michael Hankin (USC) Matrix Completion December 5, 2013 22 / 28
- 23. Noisy Figure : Rank 10 matrix with a little noise, using exact matrix reconstruction Michael Hankin (USC) Matrix Completion December 5, 2013 23 / 28
- 24. Images Michael Hankin (USC) Matrix Completion December 5, 2013 24 / 28
- 25. Images Michael Hankin (USC) Matrix Completion December 5, 2013 25 / 28
- 26. Stopping Criteria Because we expect PΩ (M − X ) to converge to zero the authors suggest using PΩ (M−X ) F ≤ as a stopping criteria. Because I generated my own PΩ (M) F data I can actually plot M−X M F F Figure : Rank 10 matrix with no noise Michael Hankin (USC) Matrix Completion December 5, 2013 26 / 28
- 27. WORK IN PROGRESS When can a matrix be reconstructed, and how much data is required? The most obvious issues arise when either a row or a column of PΩ (M) is all 0. In that case nothing can be done as that row (or column) could be totally independent of the others. Along those lines, if any row or column in the unshredded M is all 0, we are out of luck, as PΩ (M) must also have a 0 row (or column). Even when there are no such rows or columns in M, if any of its singular vectors are too heavily skewed in a euclidean basis direction, the likelihood of one of the rows (or columns) of PΩ (M) being 0 is high. Also, note that an n1 xn2 matrix of rank r has (n1 − r )r + r 2 + (n2 − r )r degrees of freedom. Michael Hankin (USC) Matrix Completion December 5, 2013 27 / 28
- 28. References Cai, J.-F., Cands, E. J. and Shen, Z. (2010) A singular value thresholding algorithm for matrix completion. SIAM J. Optim. 20, 1956-1982. Cands, E. J. and Plan, Y. (2010) Matrix completion with noise Proceedings of the IEEE 98, 925-936 Michael Hankin (USC) Matrix Completion December 5, 2013 28 / 28
- 29. The End Michael Hankin (USC) Matrix Completion December 5, 2013 29 / 28