![Simulation results
• i.i.d. Gaussian design matrix, k = 4, n = 64, p ∈ [2, 256], SNR = 1
• Note stability to non-sparsity and variability
0.9 L1 0.9 L1
L2 L2
0.8 greedy 0.8 greedy
oracle
0.7 0.7
mean square error
mean square error
0.6 0.6
0.5 0.5
0.4 0.4
0.3 0.3
0.2 0.2
0.1 0.1
0 0
2 4 6 8 2 4 6 8
log2(p) log2(p)
Sparse Rotated (non sparse)](https://image.slidesharecdn.com/tutocvprpart1-110515231937-phpapp02/85/CVPR2010-Sparse-Coding-and-Dictionary-Learning-for-Image-Analysis-Part-1-Sparse-Models-in-Machine-Learning-17-320.jpg)
CVPR2010: Sparse Coding and Dictionary Learning for Image Analysis: Part 1: Sparse Models in Machine Learning
- 1. Sparse methods for machine learning Francis Bach Willow project, INRIA - Ecole Normale Sup´rieure e CVPR Tutorial - June 2010 Special thanks to R. Jenatton, G. Obozinski
- 2. Sparse methods for machine learning Outline • Sparse linear estimation with the ℓ1-norm – Lasso – Important theoretical results • Structured sparse methods on vectors – Groups of features / Multiple kernel learning • Sparse methods on matrices – Multi-task learning – Matrix factorization (low-rank, sparse PCA, dictionary learning)
- 3. Supervised learning and regularization • Data: xi ∈ X , yi ∈ Y, i = 1, . . . , n • Minimize with respect to function f : X → Y: n λ ℓ(yi, f (xi)) + f 2 i=1 2 Error on data + Regularization Loss & function space ? Norm ? • Two theoretical/algorithmic issues: 1. Loss 2. Function space / norm
- 4. Regularizations • Main goal: avoid overfitting • Two main lines of work: 1. Euclidean and Hilbertian norms (i.e., ℓ2-norms) – Possibility of non linear predictors – Non parametric supervised learning and kernel methods – Well developped theory and algorithms (see, e.g., Wahba, 1990; Sch¨lkopf and Smola, 2001; Shawe-Taylor and Cristianini, 2004) o
- 5. Regularizations • Main goal: avoid overfitting • Two main lines of work: 1. Euclidean and Hilbertian norms (i.e., ℓ2-norms) – Possibility of non linear predictors – Non parametric supervised learning and kernel methods – Well developped theory and algorithms (see, e.g., Wahba, 1990; Sch¨lkopf and Smola, 2001; Shawe-Taylor and Cristianini, 2004) o 2. Sparsity-inducing norms – Usually restricted to linear predictors on vectors f (x) = w⊤x p – Main example: ℓ1-norm w 1 = i=1 |wi| – Perform model selection as well as regularization – Theory and algorithms “in the making”
- 6. ℓ2-norm vs. ℓ1-norm • ℓ1-norms lead to interpretable models • ℓ2-norms can be run implicitly with very large feature spaces (e.g., kernel trick) • Algorithms: – Smooth convex optimization vs. nonsmooth convex optimization • Theory: – better predictive performance?
- 7. ℓ2 vs. ℓ1 - Gaussian hare vs. Laplacian tortoise • First-order methods (Fu, 1998; Beck and Teboulle, 2009) • Homotopy methods (Markowitz, 1956; Efron et al., 2004)
- 8. Why ℓ1-norm constraints leads to sparsity? • Example: minimize quadratic function Q(w) subject to w 1 T. – coupled soft thresholding • Geometric interpretation – NB : penalizing is “equivalent” to constraining w2 w2 w1 w1
- 9. ℓ1-norm regularization (linear setting) • Data: covariates xi ∈ Rp, responses yi ∈ Y, i = 1, . . . , n • Minimize with respect to loadings/weights w ∈ Rp: n J(w) = ℓ(yi, w⊤xi) + λ w 1 i=1 Error on data + Regularization • Including a constant term b? Penalizing or constraining? • square loss ⇒ basis pursuit in signal processing (Chen et al., 2001), Lasso in statistics/machine learning (Tibshirani, 1996)
- 10. Lasso - Two main recent theoretical results 1. Support recovery condition (Zhao and Yu, 2006; Wainwright, 2009; Zou, 2006; Yuan and Lin, 2007): the Lasso is sign-consistent if and only if QJcJQ−1sign(wJ) ∞ 1, JJ n where Q = 1 limn→+∞ n i=1 xix⊤ i ∈ Rp×p and J = Supp(w)
- 11. Lasso - Two main recent theoretical results 1. Support recovery condition (Zhao and Yu, 2006; Wainwright, 2009; Zou, 2006; Yuan and Lin, 2007): the Lasso is sign-consistent if and only if QJcJQ−1sign(wJ) ∞ 1, JJ n where Q = 1 limn→+∞ n i=1 xix⊤ i ∈ Rp×p and J = Supp(w) • The Lasso is usually not model-consistent – Selects more variables than necessary (see, e.g., Lv and Fan, 2009) – Fixing the Lasso: adaptive Lasso (Zou, 2006), relaxed Lasso (Meinshausen, 2008), thresholding (Lounici, 2008), Bolasso (Bach, 2008a), stability selection (Meinshausen and B¨hlmann, 2008) u
- 12. Adaptive Lasso and concave penalization • Adaptive Lasso (Zou, 2006; Huang et al., 2008) p |wj | – Weighted ℓ1-norm: minp L(w) + λ w∈R j=1 |wj |α ˆ – w estimator obtained from ℓ2 or ℓ1 regularization ˆ • Reformulation in terms of concave penalization p minp L(w) + g(|wj |) w∈R j=1 – Example: g(|wj |) = |wj |1/2 or log |wj |. Closer to the ℓ0 penalty – Concave-convex procedure: replace g(|wj |) by affine upper bound – Better sparsity-inducing properties (Fan and Li, 2001; Zou and Li, 2008; Zhang, 2008b)
- 13. Lasso - Two main recent theoretical results 1. Support recovery condition (Zhao and Yu, 2006; Wainwright, 2009; Zou, 2006; Yuan and Lin, 2007): the Lasso is sign-consistent if and only if QJcJQ−1sign(wJ) ∞ 1, JJ n where Q = 1 limn→+∞ n i=1 xix⊤ i ∈ Rp×p and J = Supp(w)
- 14. Lasso - Two main recent theoretical results 1. Support recovery condition (Zhao and Yu, 2006; Wainwright, 2009; Zou, 2006; Yuan and Lin, 2007): the Lasso is sign-consistent if and only if QJcJQ−1sign(wJ) ∞ 1, JJ n where Q = 1 limn→+∞ n i=1 xix⊤ i ∈ Rp×p and J = Supp(w) 2. Exponentially many irrelevant variables (Zhao and Yu, 2006; Wainwright, 2009; Bickel et al., 2009; Lounici, 2008; Meinshausen and Yu, 2008): under appropriate assumptions, consistency is possible as long as log p = O(n)
- 15. Alternative sparse methods Greedy methods • Forward selection • Forward-backward selection • Non-convex method – Harder to analyze – Simpler to implement – Problems of stability • Positive theoretical results (Zhang, 2009, 2008a) – Similar sufficient conditions than for the Lasso • Bayesian methods : see Seeger (2008)
- 16. Comparing Lasso and other strategies for linear regression • Compared methods to reach the least-square solution 1 λ – Ridge regression: minp y − Xw 2 + w 2 2 2 w∈R 2 2 1 – Lasso: minp y − Xw 2 + λ w 1 2 w∈R 2 – Forward greedy: ∗ Initialization with empty set ∗ Sequentially add the variable that best reduces the square loss • Each method builds a path of solutions from 0 to ordinary least- squares solution
- 17. Simulation results • i.i.d. Gaussian design matrix, k = 4, n = 64, p ∈ [2, 256], SNR = 1 • Note stability to non-sparsity and variability 0.9 L1 0.9 L1 L2 L2 0.8 greedy 0.8 greedy oracle 0.7 0.7 mean square error mean square error 0.6 0.6 0.5 0.5 0.4 0.4 0.3 0.3 0.2 0.2 0.1 0.1 0 0 2 4 6 8 2 4 6 8 log2(p) log2(p) Sparse Rotated (non sparse)
- 18. Extensions - Going beyond the Lasso • ℓ1-norm for linear feature selection in high dimensions – Lasso usually not applicable directly
- 19. Extensions - Going beyond the Lasso • ℓ1-norm for linear feature selection in high dimensions – Lasso usually not applicable directly • Sparse methods are not limited to the square loss – logistic loss: algorithms (Beck and Teboulle, 2009) and theory (Van De Geer, 2008; Bach, 2009) • Sparse methods are not limited to supervised learning – Learning the structure of Gaussian graphical models (Meinshausen and B¨hlmann, 2006; Banerjee et al., 2008) u – Sparsity on matrices (last part of this session) • Sparse methods are not limited to linear variable selection – Multiple kernel learning (next part of this session)
- 20. Sparse methods for machine learning Outline • Sparse linear estimation with the ℓ1-norm – Lasso – Important theoretical results • Structured sparse methods on vectors – Groups of features / Multiple kernel learning • Sparse methods on matrices – Multi-task learning – Matrix factorization (low-rank, sparse PCA, dictionary learning)
- 21. Penalization with grouped variables (Yuan and Lin, 2006) • Assume that {1, . . . , p} is partitioned into m groups G1, . . . , Gm m • Penalization by i=1 wGi 2, often called ℓ1-ℓ2 norm • Induces group sparsity – Some groups entirely set to zero – no zeros within groups • In this tutorial: – Groups may have infinite size ⇒ MKL – Groups may overlap ⇒ structured sparsity
- 22. Linear vs. non-linear methods • All methods in this tutorial are linear in the parameters • By replacing x by features Φ(x), they can be made non linear in the data • Implicit vs. explicit features – ℓ1-norm: explicit features – ℓ2-norm: representer theorem allows to consider implicit features if their dot products can be computed easily (kernel methods)
- 23. Kernel methods: regularization by ℓ2-norm • Data: xi ∈ X , yi ∈ Y, i = 1, . . . , n, with features Φ(x) ∈ F = Rp – Predictor f (x) = w⊤Φ(x) linear in the features n ⊤ λ 2 • Optimization problem: minp ℓ(yi, w Φ(xi)) + w 2 w∈R i=1 2
- 24. Kernel methods: regularization by ℓ2-norm • Data: xi ∈ X , yi ∈ Y, i = 1, . . . , n, with features Φ(x) ∈ F = Rp – Predictor f (x) = w⊤Φ(x) linear in the features n ⊤ λ 2 • Optimization problem: minp ℓ(yi, w Φ(xi)) + w 2 w∈R i=1 2 • Representer theorem (Kimeldorf and Wahba, 1971): solution must n be of the form w = i=1 αiΦ(xi) n λ ⊤ – Equivalent to solving: min ℓ(yi, (Kα)i) + α Kα α∈R n i=1 2 – Kernel matrix Kij = k(xi, xj ) = Φ(xi)⊤Φ(xj )
- 25. Multiple kernel learning (MKL) (Lanckriet et al., 2004b; Bach et al., 2004a) • Sparsity with non-linearities p – replace f (x) = j=1 ⊤ wj xj with x ∈ Rp and wj ∈ R p ⊤ – by f (x) = j=1 wj Φj (x) with x ∈ X , Φj (x) ∈ Fj an wj ∈ Fj p p • Replace the ℓ1-norm j=1 |wj | by “block” ℓ1-norm j=1 wj 2 • Multiple feature maps / kernels on x ∈ X : – p “feature maps” Φj : X → Fj , j = 1, . . . , p. – Predictor: f (x) = w1⊤Φ1(x) + · · · + wp⊤Φp(x) – Generalized additive models (Hastie and Tibshirani, 1990)
- 26. Regularization for multiple features Φ1(x)⊤ w1 ր . . . . ց ⊤ ⊤ x −→ Φj (x)⊤ wj −→ w1 Φ1(x) + · · · + wp Φp(x) ց . . . . ր Φp(x)⊤ wp p 2 p • Regularization by j=1 wj 2 is equivalent to using K = j=1 Kj – Summing kernels is equivalent to concatenating feature spaces
- 27. Regularization for multiple features Φ1(x)⊤ w1 ր . . . . ց ⊤ ⊤ x −→ Φj (x)⊤ wj −→ w1 Φ1(x) + · · · + wp Φp(x) ց . . . . ր Φp(x)⊤ wp p 2 p • Regularization by j=1 wj 2 is equivalent to using K = j=1 Kj p • Regularization by j=1 wj 2 imposes sparsity at the group level • Main questions when regularizing by block ℓ1-norm: 1. Algorithms (Bach et al., 2004a; Rakotomamonjy et al., 2008) 2. Analysis of sparsity inducing properties (Bach, 2008b) p 3. Equivalent to learning a sparse combination j=1 ηj Kj
- 28. Applications of multiple kernel learning • Selection of hyperparameters for kernel methods • Fusion from heterogeneous data sources (Lanckriet et al., 2004a) • Two regularizations on the same function space: – Uniform combination ⇔ ℓ2-norm – Sparse combination ⇔ ℓ1-norm – MKL always leads to more interpretable models – MKL does not always lead to better predictive performance ∗ In particular, with few well-designed kernels ∗ Be careful with normalization of kernels (Bach et al., 2004b)
- 29. Applications of multiple kernel learning • Selection of hyperparameters for kernel methods • Fusion from heterogeneous data sources (Lanckriet et al., 2004a) • Two regularizations on the same function space: – Uniform combination ⇔ ℓ2-norm – Sparse combination ⇔ ℓ1-norm – MKL always leads to more interpretable models – MKL does not always lead to better predictive performance ∗ In particular, with few well-designed kernels ∗ Be careful with normalization of kernels (Bach et al., 2004b) • Sparse methods: new possibilities and new features
- 30. Sparse methods for machine learning Outline • Sparse linear estimation with the ℓ1-norm – Lasso – Important theoretical results • Structured sparse methods on vectors – Groups of features / Multiple kernel learning • Sparse methods on matrices – Multi-task learning – Matrix factorization (low-rank, sparse PCA, dictionary learning)
- 31. Learning on matrices - Image denoising • Simultaneously denoise all patches of a given image • Example from Mairal, Bach, Ponce, Sapiro, and Zisserman (2009b)
- 32. Learning on matrices - Collaborative filtering • Given nX “movies” x ∈ X and nY “customers” y ∈ Y, • predict the “rating” z(x, y) ∈ Z of customer y for movie x • Training data: large nX × nY incomplete matrix Z that describes the known ratings of some customers for some movies • Goal: complete the matrix.
- 33. Learning on matrices - Multi-task learning • k linear prediction tasks on same covariates x ∈ Rp – k weight vectors wj ∈ Rp – Joint matrix of predictors W = (w1, . . . , wk ) ∈ Rp×k • Classical application – Multi-category classification (one task per class) (Amit et al., 2007) • Share parameters between tasks • Joint variable selection (Obozinski et al., 2009) – Select variables which are predictive for all tasks • Joint feature selection (Pontil et al., 2007) – Construct linear features common to all tasks
- 34. Matrix factorization - Dimension reduction • Given data matrix X = (x1, . . . , xn) ∈ Rp×n – Principal component analysis: xi ≈ Dαi ⇒ X = DA – K-means: xi ≈ dk ⇒ X = DA
- 35. Two types of sparsity for matrices M ∈ Rn×p I - Directly on the elements of M • Many zero elements: Mij = 0 M • Many zero rows (or columns): (Mi1, . . . , Mip) = 0 M
- 36. Two types of sparsity for matrices M ∈ Rn×p II - Through a factorization of M = UV⊤ • Matrix M = UV⊤, U ∈ Rn×k and V ∈ Rp×k • Low rank: m small T V M = U • Sparse decomposition: U sparse T M = U V
- 37. Structured sparse matrix factorizations • Matrix M = UV⊤, U ∈ Rn×k and V ∈ Rp×k • Structure on U and/or V – Low-rank: U and V have few columns – Dictionary learning / sparse PCA: U has many zeros – Clustering (k-means): U ∈ {0, 1}n×m, U1 = 1 – Pointwise positivity: non negative matrix factorization (NMF) – Specific patterns of zeros (Jenatton et al., 2010) – Low-rank + sparse (Cand`s et al., 2009) e – etc. • Many applications • Many open questions (Algorithms, identifiability, etc.)
- 38. Low-rank matrix factorizations Trace norm • Given a matrix M ∈ Rn×p – Rank of M is the minimum size m of all factorizations of M into M = UV⊤, U ∈ Rn×m and V ∈ Rp×m – Singular value decomposition: M = U Diag(s)V⊤ where U and V have orthonormal columns and s ∈ Rm are singular values + • Rank of M equal to the number of non-zero singular values
- 39. Low-rank matrix factorizations Trace norm • Given a matrix M ∈ Rn×p – Rank of M is the minimum size m of all factorizations of M into M = UV⊤, U ∈ Rn×m and V ∈ Rp×m – Singular value decomposition: M = U Diag(s)V⊤ where U and V have orthonormal columns and s ∈ Rm are singular values + • Rank of M equal to the number of non-zero singular values • Trace-norm (a.k.a. nuclear norm) = sum of singular values • Convex function, leads to a semi-definite program (Fazel et al., 2001) • First used for collaborative filtering (Srebro et al., 2005)
- 40. Sparse principal component analysis • Given data X = (x⊤, . . . , x⊤) ∈ Rp×n, two views of PCA: 1 n – Analysis view: find the projection d ∈ Rp of maximum variance (with deflation to obtain more components) – Synthesis view: find the basis d1, . . . , dk such that all xi have low reconstruction error when decomposed on this basis • For regular PCA, the two views are equivalent
- 41. Sparse principal component analysis • Given data X = (x⊤, . . . , x⊤) ∈ Rp×n, two views of PCA: 1 n – Analysis view: find the projection d ∈ Rp of maximum variance (with deflation to obtain more components) – Synthesis view: find the basis d1, . . . , dk such that all xi have low reconstruction error when decomposed on this basis • For regular PCA, the two views are equivalent • Sparse extensions – Interpretability – High-dimensional inference – Two views are differents ∗ For analysis view, see d’Aspremont, Bach, and El Ghaoui (2008)
- 42. Sparse principal component analysis Synthesis view • Find d1, . . . , dk ∈ Rp sparse so that n k 2 n 2 minm xi − (αi)j dj = minm xi − Dαi 2 is small αi∈R αi∈R i=1 j=1 2 i=1 – Look for A = (α1, . . . , αn) ∈ Rk×n and D = (d1, . . . , dk ) ∈ Rp×k such that D is sparse and X − DA 2 is small F
- 43. Sparse principal component analysis Synthesis view • Find d1, . . . , dk ∈ Rp sparse so that n k 2 n 2 minm xi − (αi)j dj = minm xi − Dαi 2 is small αi∈R αi∈R i=1 j=1 2 i=1 – Look for A = (α1, . . . , αn) ∈ Rk×n and D = (d1, . . . , dk ) ∈ Rp×k such that D is sparse and X − DA 2 is small F • Sparse formulation (Witten et al., 2009; Bach et al., 2008) – Penalize/constrain dj by the ℓ1-norm for sparsity – Penalize/constrain αi by the ℓ2-norm to avoid trivial solutions n k 2 min xi − Dαi 2 +λ dj 1 s.t. ∀i, αi 2 1 D ,A i=1 j=1
- 44. Sparse PCA vs. dictionary learning • Sparse PCA: xi ≈ Dαi, D sparse
- 45. Sparse PCA vs. dictionary learning • Sparse PCA: xi ≈ Dαi, D sparse • Dictionary learning: xi ≈ Dαi, αi sparse
- 46. Structured matrix factorizations (Bach et al., 2008) n k 2 min xi − Dαi 2 +λ dj ⋆ s.t. ∀i, αi • 1 D ,A i=1 j=1 n n 2 min xi − Dαi 2 +λ αi • s.t. ∀j, dj ⋆ 1 D ,A i=1 i=1 • Optimization by alternating minimization (non-convex) • αi decomposition coefficients (or “code”), dj dictionary elements • Two related/equivalent problems: – Sparse PCA = sparse dictionary (ℓ1-norm on dj ) – Dictionary learning = sparse decompositions (ℓ1-norm on αi) (Olshausen and Field, 1997; Elad and Aharon, 2006; Lee et al., 2007)
- 47. Probabilistic topic models and matrix factorization • Latent Dirichlet allocation (Blei et al., 2003) – For a document, sample θ ∈ Rk from a Dirichlet(α) – For the n-th word of the same document, ∗ sample a topic zn from a multinomial with parameter θ ∗ sample a word wn from a multinomial with parameter β(zn, :)
- 48. Probabilistic topic models and matrix factorization • Latent Dirichlet allocation (Blei et al., 2003) – For a document, sample θ ∈ Rk from a Dirichlet(α) – For the n-th word of the same document, ∗ sample a topic zn from a multinomial with parameter θ ∗ sample a word wn from a multinomial with parameter β(zn, :) • Interpretation as multinomial PCA (Buntine and Perttu, 2003) – Marginalizing over topic zn, given θ, each word wn is selected from a multinomial with parameter k θk β(z, :) = β ⊤θ z=1 – Row of β = dictionary elements, θ code for a document
- 49. Probabilistic topic models and matrix factorization • Two different views on the same problem – Interesting parallels to be made – Common problems to be solved • Structure on dictionary/decomposition coefficients with adapted priors (Blei et al., 2004; Jenatton et al., 2010) • Identifiability and interpretation/evaluation of results • Discriminative tasks (Blei and McAuliffe, 2008; Lacoste-Julien et al., 2008; Mairal et al., 2009a) • Optimization and local minima – Online learning (Mairal et al., 2009c)
- 50. Sparse methods for machine learning Why use sparse methods? • Sparsity as a proxy to interpretability – Structured sparsity • Sparsity for high-dimensional inference – Influence on feature design • Sparse methods are not limited to least-squares regression • Faster training/testing • Better predictive performance? – Problems are sparse if you look at them the right way
- 51. Conclusion - Interesting questions/issues • Exponentially many features – Can we algorithmically achieve log p = O(n)? – Use structure among features (Bach, 2008c) • Norm design – What type of behavior may be obtained with sparsity-inducing norms? • Overfitting convexity – Do we actually need convexity for matrix factorization problems? – Convexity used in inner loops – Joint convexity requires reformulation (Bach et al., 2008)
- 52. References Y. Amit, M. Fink, N. Srebro, and S. Ullman. Uncovering shared structures in multiclass classification. In Proceedings of the 24th international conference on Machine Learning (ICML), 2007. F. Bach. Bolasso: model consistent lasso estimation through the bootstrap. In Proceedings of the Twenty-fifth International Conference on Machine Learning (ICML), 2008a. F. Bach. Consistency of the group Lasso and multiple kernel learning. Journal of Machine Learning Research, 9:1179–1225, 2008b. F. Bach. Exploring large feature spaces with hierarchical multiple kernel learning. In Advances in Neural Information Processing Systems, 2008c. F. Bach. Self-concordant analysis for logistic regression. Technical Report 0910.4627, ArXiv, 2009. F. Bach, G. R. G. Lanckriet, and M. I. Jordan. Multiple kernel learning, conic duality, and the SMO algorithm. In Proceedings of the International Conference on Machine Learning (ICML), 2004a. F. Bach, R. Thibaux, and M. I. Jordan. Computing regularization paths for learning multiple kernels. In Advances in Neural Information Processing Systems 17, 2004b. F. Bach, J. Mairal, and J. Ponce. Convex sparse matrix factorizations. Technical Report 0812.1869, ArXiv, 2008. O. Banerjee, L. El Ghaoui, and A. d’Aspremont. Model selection through sparse maximum likelihood estimation for multivariate Gaussian or binary data. The Journal of Machine Learning Research, 9: 485–516, 2008.
- 53. A. Beck and M. Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM Journal on Imaging Sciences, 2(1):183–202, 2009. P. Bickel, Y. Ritov, and A. Tsybakov. Simultaneous analysis of Lasso and Dantzig selector. Annals of Statistics, 37(4):1705–1732, 2009. D. Blei, T.L. Griffiths, M.I. Jordan, and J.B. Tenenbaum. Hierarchical topic models and the nested Chinese restaurant process. Advances in neural information processing systems, 16:106, 2004. D.M. Blei and J. McAuliffe. Supervised topic models. In Advances in Neural Information Processing Systems (NIPS), volume 20, 2008. D.M. Blei, A.Y. Ng, and M.I. Jordan. Latent dirichlet allocation. The Journal of Machine Learning Research, 3:993–1022, 2003. W. Buntine and S. Perttu. Is multinomial PCA multi-faceted clustering or dimensionality reduction. In International Workshop on Artificial Intelligence and Statistics (AISTATS), 2003. E.J. Cand`s, X. Li, Y. Ma, and J. Wright. Robust principal component analysis? e Arxiv preprint arXiv:0912.3599, 2009. S. S. Chen, D. L. Donoho, and M. A. Saunders. Atomic decomposition by basis pursuit. SIAM Review, 43(1):129–159, 2001. A. d’Aspremont, F. Bach, and L. El Ghaoui. Optimal solutions for sparse principal component analysis. Journal of Machine Learning Research, 9:1269–1294, 2008. B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani. Least angle regression. Annals of statistics, 32 (2):407–451, 2004. M. Elad and M. Aharon. Image denoising via sparse and redundant representations over learned
- 54. dictionaries. IEEE Transactions on Image Processing, 15(12):3736–3745, 2006. J. Fan and R. Li. Variable Selection Via Nonconcave Penalized Likelihood and Its Oracle Properties. Journal of the American Statistical Association, 96(456):1348–1361, 2001. M. Fazel, H. Hindi, and S.P. Boyd. A rank minimization heuristic with application to minimum order system approximation. In Proceedings of the American Control Conference, volume 6, pages 4734–4739, 2001. W. Fu. Penalized regressions: the bridge vs. the Lasso. Journal of Computational and Graphical Statistics, 7(3):397–416, 1998). T. J. Hastie and R. J. Tibshirani. Generalized Additive Models. Chapman & Hall, 1990. J. Huang, S. Ma, and C.H. Zhang. Adaptive Lasso for sparse high-dimensional regression models. Statistica Sinica, 18:1603–1618, 2008. R. Jenatton, J. Mairal, G. Obozinski, and F. Bach. Proximal methods for sparse hierarchical dictionary learning. In Proc. ICML, 2010. G. S. Kimeldorf and G. Wahba. Some results on Tchebycheffian spline functions. J. Math. Anal. Applicat., 33:82–95, 1971. S. Lacoste-Julien, F. Sha, and M.I. Jordan. DiscLDA: Discriminative learning for dimensionality reduction and classification. Advances in Neural Information Processing Systems (NIPS) 21, 2008. G. R. G. Lanckriet, T. De Bie, N. Cristianini, M. I. Jordan, and W. S. Noble. A statistical framework for genomic data fusion. Bioinformatics, 20:2626–2635, 2004a. G. R. G. Lanckriet, N. Cristianini, L. El Ghaoui, P. Bartlett, and M. I. Jordan. Learning the kernel matrix with semidefinite programming. Journal of Machine Learning Research, 5:27–72, 2004b.
- 55. H. Lee, A. Battle, R. Raina, and A. Ng. Efficient sparse coding algorithms. In Advances in Neural Information Processing Systems (NIPS), 2007. K. Lounici. Sup-norm convergence rate and sign concentration property of Lasso and Dantzig estimators. Electronic Journal of Statistics, 2:90–102, 2008. J. Lv and Y. Fan. A unified approach to model selection and sparse recovery using regularized least squares. Annals of Statistics, 37(6A):3498–3528, 2009. J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman. Supervised dictionary learning. Advances in Neural Information Processing Systems (NIPS), 21, 2009a. J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman. Non-local sparse models for image restoration. In International Conference on Computer Vision (ICCV), 2009b. Julien Mairal, F. Bach, J. Ponce, and Guillermo Sapiro. Online learning for matrix factorization and sparse coding. 2009c. H. M. Markowitz. The optimization of a quadratic function subject to linear constraints. Naval Research Logistics Quarterly, 3:111–133, 1956. N. Meinshausen. Relaxed Lasso. Computational Statistics and Data Analysis, 52(1):374–393, 2008. N. Meinshausen and P. B¨hlmann. High-dimensional graphs and variable selection with the lasso. u Annals of statistics, 34(3):1436, 2006. N. Meinshausen and P. B¨hlmann. Stability selection. Technical report, arXiv: 0809.2932, 2008. u N. Meinshausen and B. Yu. Lasso-type recovery of sparse representations for high-dimensional data. Annals of Statistics, 37(1):246–270, 2008. G. Obozinski, B. Taskar, and M.I. Jordan. Joint covariate selection and joint subspace selection for
- 56. multiple classification problems. Statistics and Computing, pages 1–22, 2009. B. A. Olshausen and D. J. Field. Sparse coding with an overcomplete basis set: A strategy employed by V1? Vision Research, 37:3311–3325, 1997. M. Pontil, A. Argyriou, and T. Evgeniou. Multi-task feature learning. In Advances in Neural Information Processing Systems, 2007. A. Rakotomamonjy, F. Bach, S. Canu, and Y. Grandvalet. SimpleMKL. Journal of Machine Learning Research, 9:2491–2521, 2008. B. Sch¨lkopf and A. J. Smola. Learning with Kernels. MIT Press, 2001. o M.W. Seeger. Bayesian inference and optimal design for the sparse linear model. The Journal of Machine Learning Research, 9:759–813, 2008. J. Shawe-Taylor and N. Cristianini. Kernel Methods for Pattern Analysis. Cambridge University Press, 2004. N. Srebro, J. D. M. Rennie, and T. S. Jaakkola. Maximum-margin matrix factorization. In Advances in Neural Information Processing Systems 17, 2005. R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of The Royal Statistical Society Series B, 58(1):267–288, 1996. S. A. Van De Geer. High-dimensional generalized linear models and the Lasso. Annals of Statistics, 36 (2):614, 2008. G. Wahba. Spline Models for Observational Data. SIAM, 1990. M. J. Wainwright. Sharp thresholds for noisy and high-dimensional recovery of sparsity using ℓ1- constrained quadratic programming. IEEE transactions on information theory, 55(5):2183, 2009.
- 57. D.M. Witten, R. Tibshirani, and T. Hastie. A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis. Biostatistics, 10(3):515–534, 2009. M. Yuan and Y. Lin. Model selection and estimation in regression with grouped variables. Journal of The Royal Statistical Society Series B, 68(1):49–67, 2006. M. Yuan and Y. Lin. On the non-negative garrotte estimator. Journal of The Royal Statistical Society Series B, 69(2):143–161, 2007. T. Zhang. Adaptive forward-backward greedy algorithm for sparse learning with linear models. Advances in Neural Information Processing Systems, 22, 2008a. T. Zhang. Multi-stage convex relaxation for learning with sparse regularization. Advances in Neural Information Processing Systems, 22, 2008b. T. Zhang. On the consistency of feature selection using greedy least squares regression. The Journal of Machine Learning Research, 10:555–568, 2009. P. Zhao and B. Yu. On model selection consistency of Lasso. Journal of Machine Learning Research, 7:2541–2563, 2006. H. Zou. The adaptive Lasso and its oracle properties. Journal of the American Statistical Association, 101(476):1418–1429, 2006. H. Zou and R. Li. One-step sparse estimates in nonconcave penalized likelihood models. Annals of Statistics, 36(4):1509–1533, 2008.