Sparse inverse covariance estimation using skggm
- 1. Manjari Narayan, Postdoctoral Scholar, Stanford University (School of Medicine) (PI: Amit Etkin, M.D., Ph.D.) Tutorial presented at Junior Scientist Workshop at HHMI, Janelia Farms Sparse Inverse Covariance Estimation using skggm skggm: Collaboration with Dr. Jason Laska, ML R&D at Clara Labs.
- 2. Explosion of Functional Imaging Tools fMRI, fNIRS EEG, MEG Intracranial EEG, micro-ECoG Molecular fMRI Credit: Marie Suver, Ph.D. and Ainul Huda, University of Washington and Michael H. Dickinson, Ph.D., California Institute of Technology http://newsroom.cumc.columbia.edu/blog/2014/11/11/researchers- receive-nih-brain-initiative-funding/ Calcium imaging Credit: Misha Ahrens, Ph.D., Janelia Farms https://www.simonsfoundation.org/features/ foundation-news/how-do-different-brain-regions- interact-to-enhance-function/ Light sheet microscopy Photo Credit: Tang, 2015, Scientific Reports. Voltage-sensitive Dye Imaging Light field microscopy Credit: Raju Tomer, Ph.D. & Deisseroth Lab, Stanford University http://techfinder.stanford.edu/technology_detail.php?ID=36402
- 3. Application: Functional Connectomics Network as a unit of interest Unobserved stochastic dependence/interaction between neurons, circuits, regions, … Ahrens, et. al. Nature (2012) A shared goal across modalities & resolutions Macroscale Mesoscale T or n p
- 4. Probabilistic Graphical Models Many probabilistic models available, both directed and undirected Graph G = (V, E) Vertices V = (1, . . . , p), Edges E ⇢ V ⌦ V X = (X1, . . . , Xp) ⇠ PX Probabilistic graphical model relates PX to G (j, k) 62 E () independence or conditional independence between Xj and Xk Graph G = (V, E) Vertices V = (1, . . . , p), Edges E ⇢ V ⌦ V X = (X1, . . . , Xp) ⇠ PX Probabilistic graphical model relates PX to G E () independence or conditional independence between Xj and Xk Observed: Unobserved: Examples Directed Acyclic Graphs (DAGs/Bayes-nets) State-Space Models including linear/nonlinear VAR, Undirected Graphical Models or Markov networks Bivariate associations (Correlation, Granger-Causality, Transfer Entropy)
- 5. More informative than correlations: A measure of “direct” interactions that elements “indirect” interactions due to observed common causes. Benefits: Studying cognitive mechanisms Designing interventional targets Science-wide efficient use of data Models for Connectivity: Conditional Dependence & Markov Networks conditional dependence (“partial correlations”) marginal dependence (“marginal correlations”)
- 6. Introduction to Markov Networks
- 7. Markov Properties • Graph G = (V, E) • Vertices V = {1, 2, . . . p} and Edges E ⇢ V ⇥ V • Multivariate Normal x1, . . . , xT i.i.d ⇠ Np(0, ⌃) • Inverse Covariance ⌃ 1 = ⇥ 1 5 2 4 3 X5 ? X1|XV {1,5} Pairwise Markov Property (P): Two variables conditionally independent, given all other nodes Lauritzen (1996)
- 8. Markov Properties • Graph G = (V, E) • Vertices V = {1, 2, . . . p} and Edges E ⇢ V ⇥ V • Multivariate Normal x1, . . . , xT i.i.d ⇠ Np(0, ⌃) • Inverse Covariance ⌃ 1 = ⇥ 1 5 2 4 3 X5 ? XV ne(5)|Xne(5), ne(5) = {2, 5} Local Markov Property (L): A variable is conditionally independent of all others given its neighbors Lauritzen (1996)
- 9. Global Markov Property (G): Given 3 disjoint sets A, B and C such that all paths between A to B go through C, then A conditionally independent of B given C Markov Properties • Graph G = (V, E) • Vertices V = {1, 2, . . . p} and Edges E ⇢ V ⇥ V • Multivariate Normal x1, . . . , xT i.i.d ⇠ Np(0, ⌃) • Inverse Covariance ⌃ 1 = ⇥ 1 5 2 4 3 A B C XA ? XB|XC, where XA = {Xa}a2A Lauritzen (1996)
- 10. Intersection property: Holds for positive densities, e.g. Gaussian Factorizes probability distribution Computational tractability & Statistical power to identify all conditional independences Extended to some non-positive densities! Benefits of Global Markov Properties P(X) = P(XA|XC)P(XB|XC)P(XC) If A ? B|(C, D) and A ? C|(B, D) then A ? (B [ C)|D Lauritzen (1996) 1 5 2 4 3 A B C
- 11. Generality of Markov Networks For many types of pairwise associations Markov Networks that satisfy Global Markov Property Pairwise Association Markov Networks Correlation Zero partial correlation = Conditional independence Coherence or Coherency Zero partial coherence = Conditional independence Directed Information (including transfer entropy, Sims/Granger prediction, … ) Dynamic extensions to standard Markov properties, local independence (Didelez 2008) Pairwise ordering between variables DAGs, CPDAGs, MAGs, PAGs, …. This is not an exhaustive list!
- 12. Generality of Markov Networks For many probability distributions Markov Networks that satisfy at least Local if not Global Markov Property Distributional Assumptions Markov Networks Exponential Families (Binary, Poisson, Circular, …) Exponential MRFs including Binary Ising Models, Poisson Graphical Models, Nonparametric Distributions Nonparanormal (copulas) Graphical Models, Kernel Graphical Models Separable Covariance Structure (Spatio-Temporal) Separable Markov Networks (P. Ravikumar, G.I. Allen, and others) (H. Liu, E. Xing, B. Scholkopf, and others) (G.I. Allen, S. Zhou, A. Hero, P. Hoff, and many others)
- 13. From now on: Gaussian Graphical Model Xk ? Xl|XV {k,l} () ⌃ 1 kl = 0 () (k, l) 62 E • Graph G = (V, E) • Vertices V = {1, 2, . . . p} and Edges E ⇢ V ⇥ V • Multivariate Normal x1, . . . , xT i.i.d ⇠ Np(0, ⌃) • Inverse Covariance ⌃ 1 = ⇥ Zero in Inverse Covariance = Conditional Independence 3 1 5 2 4 Lauritzen (1996) 1 2 3 4 5 12345
- 14. From now on: Gaussian Graphical Model Xk ? Xl|XV {k,l} () ⌃ 1 kl = 0 () (k, l) 62 E • Graph G = (V, E) • Vertices V = {1, 2, . . . p} and Edges E ⇢ V ⇥ V • Multivariate Normal x1, . . . , xT i.i.d ⇠ Np(0, ⌃) • Inverse Covariance ⌃ 1 = ⇥ Zero in Inverse Covariance = Conditional Independence 3 1 5 2 4 Lauritzen (1996) Important for nonparametric distributions + exponential family 1 2 3 4 5 12345
- 15. Estimation in High Dimensions
- 16. Gaussian Log-Likelihood Input to log-likelihood is effectively sample covariance Likelihood for Inverse Covariance “Covariance Selection”, Dempster, 1972; Banerjee et. al. (2006); Yuan (2006) ˆ⌃ = 1 T X> X, Data matrix XT ⇥p is centered L(ˆ⌃; ⇥) ⌘ log det ⇥ D ˆ⌃, ⇥ E
- 17. Gaussian Log-Likelihood Input to log-likelihood is effectively sample covariance Likelihood for Inverse Covariance “Covariance Selection”, Dempster, 1972; Banerjee et. al. (2006); Yuan (2006) ˆ⌃ = 1 T X> X, Data matrix XT ⇥p is centered L(ˆ⌃; ⇥) ⌘ log det ⇥ D ˆ⌃, ⇥ E
- 18. Likelihood for Inverse Covariance Put all variables on the same scale “Covariance Selection”, Dempster, 1972; Banerjee et. al. (2006); Yuan (2006) ˆ⌃ = 1 T X> X, R(ˆ⌃) = D 1 2 ˆ⌃D 1 2 , D = diag(ˆ⌃) L(ˆ⌃; ⇥) ⌘ log det ⇥ D R(ˆ⌃), ⇥ E Gaussian Log-Likelihood Variance-Correlation Decomposition
- 19. Degeneracy of Likelihood in High Dimensions Credit: Negaban, Ravikumar, Wainwright & Yu, Statistical Science, 2012; “A Unified Framework for High Dimensional Analysis of M-estimators with Decomposable Regularizers” High curvature (Easy) Low curvature (Hard) Given XT ⇥p, T ⇡ p
- 20. Encourage sparsity with Lasso penalty Convex problem: Many optimization solutions available Popular alternative if (L)=>(G): neighborhood selection Sparse Inverse Covariance Sparse penalized Maximum Likelihood ˆ⇥( ) = maximize ⇥ 0 L(ˆ⌃; ⇥) k⇥k1,o↵, k⇥k1,o↵ = X j6=k |✓j,k| “Covariance Selection”, Dempster, 1972; Banerjee et. al. (2006); Yuan (2006); Friedman et. al (2008);“QUIC”, Hsieh et. al. (2011 & 2013); Buhlmann & Van De Geer (2011); Meinshausen & Buhlmann (2006)
- 21. Fisher Information (F) of the inverse covariance needs to be well conditioned, not incoherent. Signal strength of edges needs to be sufficiently larger than noise Caveat: Might always hold at infinite sample size but only probabilistically in finite samples Model Identifiability of Sparse MLE When is perfect edge recovery possible? Meinshausen et. al. 2006; Ravikumar et. al. (2010, 2011); Van De Geer & Buhlmann (2013); and others
- 22. Model Identifiability: Network Structure Matters Theoretical assumptions often violated for many networks at finite samples Narayan et. al (2015a) Do two unconnected nodes, share “mutual friends” ? Increases with degree Dependent on structure Meinshausen et. al. 2006; Ravikumar et. al. (2010, 2011); Cai & Zhou (2015) More correlated nodes, more errors in distinguishing edges from non-edges
- 23. We will only look at Lasso and its improved variants Different estimators have slightly different limitations Pseudolikelihood; Least squares; Dantzig type …. Other regularizers behave differently as well Model Identifiability of Sparse MLE When is perfect edge recovery possible? See review on graphical models from Drton & Maathius (2016)
- 24. Features: scikit-learn interface Comprehensive range of estimators, model selection procedures, metrics, monte-carlo benchmarks of statistical error control, … For researcher: Benchmark new estimator/algorithm against others For data analyst: Best practices for estimation & structure learning Github repo: http://github.com/jasonlaska/skggm Tutorial notebooks: http://neurostats.org/jf2016-skggm/ skggm: Inverse covariance estimation By @jasonlaska and @mnarayan
- 25. Binder Instructions: http://mybinder.org/repo/neuroquant/jf2016-skggm Alternative/Backup: Install in local anaconda environment Install skggm: pip install skggm Download notebooks: git clone git@github.com:neuroquant/jf2016-skggm.git skggm: Tutorial Setup Tutorial: http://neurostats.org/jf2016-skggm/
- 26. Ground Truth Toy Example: Simple Banded or Chain Network Structure
- 27. Saturated Precision Matrices Saturation: Estimate all entries of inverse covariance (precision) Recall: High curvature of likelihood, easy to distinguish different graphs
- 28. Saturated Precision Matrices Degeneracy at low sample sizes. (Using pseudo-inverse for degenerate sample covariance) Recall: Low curvature of likelihood, hard to distinguish different graphs
- 29. Standard Graphical Lasso Model Selection: How do we choose regularization/sparsity/non-zero support? ˆ⇥( ) = arg min ⇥ 0 L(ˆ⌃; ⇥) + Pen(⇥) Friedman et al. 2007; Meinshausen and Buhlmann 2006; Banerjee et al. 2006; Rothman 2008; Hsieh et al; Cai et al. 2011; and many more. Sparse Penalized Maximum Likelihood
- 30. Cross Validation: Minimizes Type II Yuan and Lin (2007); Bickel and Levina (2008) X ! (X⇤,train , X⇤,test ) { ˆ⇥⇤ ( )}train Training Hold-out Loss({ ˆ⇥⇤ ( )}train ; { ˆ⌃⇤ }test ) E.g. Kullback-Leibler; Log-Likelihood
- 31. Extended BIC: Minimizes Type I Foygel & Drton (2010) Alternatives (StARS, Liu et. al.) Coming soon Privileges sparser models than BIC min BIC(ˆE( )) = min Ln(ˆ⌃; ⇥) + |ˆE| log(n) + 4 |ˆE| log(p) ˆE d = no. of non-zeros in ˆ⇥( )
- 32. One-Stage vs. Two-Stage Estimators Use initial estimates to reduce bias in estimation Standard Graphical Lasso Weighted Graphical Lasso Stage I: Stage 2: ˆ⇥( ) = maximize ⇥ 0 L(ˆ⌃; ⇥) kW ⇥k1,o↵, kW ⇥k1,o↵ = X j6=k |wj,k✓j,k| ˆ⇥( ) = maximize ⇥ 0 L(ˆ⌃; ⇥) k⇥k1,o↵, k⇥k1,o↵ = X j6=k |✓j,k| wjk = 1 |ˆ✓|in jk E.g. Adaptive weights Zou 2006; Zhou et. al. 2011 Buhlmann & Van De Geer 2011; Cai & Zhou 2015
- 33. Strong edges shrink less Shrinkage of Edges: Lasso vs. Adaptive Performance very dependent on weights i.e. Need good separation between strong vs. weak edges Coefficient (entryofinversecovariance) Regularization parameter (lambda) All edges shrink by same value Zou (2006)
- 34. Weights can be data dependent/adaptive Stage I: Any estimator not just MLE Stage II: Adaptive MLE Use to create randomized model averaging Locally linear approximations to non-convex penalties (coming soon to skggm) Variety of Two-Stage Estimators Weights can be specified in many ways Adaptive Estimation: Zhou, Van De Geer, Buhlmann (2009); Breheny and Huang (2011); Cai et. al. (2011) and others
- 35. High Sparsity Case True Parameters n p = 75, degree = .15p
- 36. High Sample Size, High Sparsity Adaptive Estimator improves on Initial Estimator n p = 75, degree = .15p Difference in sparsity: 69,77 Support Error: 4.0, False Pos: 4.0, False Neg: 0.0 Difference in sparsity: 69,141 Support Error: 36.0, False Pos: 36.0, False Neg: 0.0
- 37. Low Sample Size, High Sparsity Adaptivity less useful without good initial estimate n p = 15, degree = .15p Difference in sparsity: 69,85 Support Error: 8.0, False Pos: 8.0, False Neg: 0.0 Difference in sparsity: 69,149 Support Error: 40.0, False Pos: 40.0, False Neg: 0.0
- 38. Moderate Sparsity Case n p = 75, degree = .4p True Parameters
- 39. High Sample Size, Moderate Sparsity Nodes more correlated with each other, but adaptivity still does well n p = 75, degree = .4p Difference in sparsity: 115,129 Support Error: 7.0, False Pos: 7.0, False Neg: 0.0 Difference in sparsity: 115,169 Support Error: 27.0, False Pos: 27.0, False Neg: 0.0
- 40. Low Sample Size, Moderate Sparsity Nodes more correlated with each other, more false negatives n p = 15, degree = .4p Difference in sparsity: 115,135 Support Error: 22.0, False Pos: 16.0, False Neg: 6.0 Difference in sparsity: 115,111 Support Error: 18.0, False Pos: 8.0, False Neg: 10.0
- 41. Model Averaging & Stability Selection For any initial estimator build an ensemble of estimators and aggregate n p = 15, degree Threshold stability scores => Familywise error control over edges Meinshausen & Buhlmann (2010) ˆ⇥⇤b ( ) = maximize ⇥ 0 L(ˆ⌃⇤b ; ⇥) Pen(W⇤b ( ) ⇥), w⇤b jk = w⇤b kj 2 { /a, a }, with Ber(⇢), for j 6= k Aggregate I ⇣ ˆ⇥⇤b ( ) 6= 0 ⌘
- 42. Future plans include Computational scalability (big-quic, support for Apache spark) Monte-Carlo “unit-testing” of statistical error control Novel case studies and more examples Other estimator classes (pseudo-likelihood, non-convex, …) Regularizers beyond sparsity: mixture of regularizers, … Other Markov network models for time-series Directed graphical models skggm: Inverse covariance estimation Version 0.1