Redundancy and synergy in dynamical systems
- 1. Synergy and redundancy in dynamical systems: towards a practical and operative definition Daniele Marinazzo1 Luca Faes 2 Sebastiano Stramaglia 3 1Ghent University, Belgium 2Fondazione Bruno Kessler, Italy 3University of Bari and INFN, Italy December 16, 2016 @dan marinazzo http://users.ugent.be/~dmarinaz/ Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 2. Granger causality to recover dynamical networks Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 3. Granger causality to recover dynamical networks Context Two time series X and Y x, the future values of X Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 4. Granger causality to recover dynamical networks Context Two time series X and Y x, the future values of X Operative definition, Wiener 1956, Granger 1969 Y is cause of X if the knowledge of Y allows to make more precise predictions about x Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 5. GC in multivariate datasets: a well-known issue Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 6. GC in multivariate datasets: a well-known issue Condition GC estimation to the effect of other variables, to avoid false positives Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 7. GC in multivariate datasets: a well-known issue Condition GC estimation to the effect of other variables, to avoid false positives Several proposed approaches, starting from Geweke et al 1984 Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 8. Granger causality: definition Predictive model of a multivariate system n time series {xα(t)}α=1,...,n, state vectors Xα(t) = (xα(t − m), . . . , xα(t − 1)) , m order of the model Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 9. Granger causality: definition Predictive model of a multivariate system n time series {xα(t)}α=1,...,n, state vectors Xα(t) = (xα(t − m), . . . , xα(t − 1)) , m order of the model Conditioned Granger Causality δmv (β → α) = log (xα|X Xβ) (xα|X) Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 10. Granger causality: definition Predictive model of a multivariate system n time series {xα(t)}α=1,...,n, state vectors Xα(t) = (xα(t − m), . . . , xα(t − 1)) , m order of the model Conditioned Granger Causality δmv (β → α) = log (xα|X Xβ) (xα|X) Pairwise Granger Causality δbv (β → α) = log (xα|Xα) (xα|Xα, Xβ) Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 11. Granger causality and Transfer entropy GC and TE are equivalent for Gaussian variables and other quasi-Gaussian distributions (Barnett et al 2009, Hlavackova-Schindler 2011, Barnett and Bossomaier 2012) In this case they both measure information transfer. Unified approach (model based and model free) Mathematically more treatable Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 12. False positives in pairwise GC Ten unidirectionally coupled noisy logistic maps, with x1(t) = f (x1(t − 1)) + 0.01η1(t), and xi (t) = (1 − ρ)f (xi (t − 1)) + ρf (xi−1(t − 1)) + 0.01ηi (t), with i = 2, . . . , 10, η Gaussian noise terms, coupling ρ, f (x) = 1 − 1.8x2 Stramaglia, Cortes and Marinazzo, New Journal of Physics 2014 Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 13. False negatives in pairwise GC due to synergy Three unit variance iid Gaussian noise terms x1, x2 and x3. Let x4(t) = 0.1(x1(t − 1) + x2(t − 1)) + ρx2(t − 1)x3(t − 1) + 0.1η(t) . x2 is a suppressor variable for x3 w.r.t. the influence on x4 Stramaglia, Cortes and Marinazzo, New Journal of Physics 2014 Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 14. Redundancy due to a hidden source h(t) hidden Gaussian variable, influencing n variables xi (t) = h(t − 1) + sηi (t), and w(t) = h(t − 2) + sη0(t) influenced by h but with a larger delay, s is the noise level. Stramaglia, Cortes and Marinazzo, New Journal of Physics 2014 Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 15. Redundancy due to synchronization Multiplet of logistic maps {xi }, i = 1, . . . , 4,: xi (t) = (1 − ρ)f (xi (t − 1)) + ρ 4 j=1,j=i f (xj (t − 1)) + 0.01ηi (t), and x5(t) = 4 i=1 xi (t−1) 8 + η5(t), where η are unit variance Gaussian noise terms, coupling ρ. multiplettox5 multiplettomultiplet Stramaglia, Cortes and Marinazzo, New Journal of Physics 2014 Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 16. Partial conditioning Conditioned Granger Causality (CGC) δmv (β → α) = log (xα|X Xβ) (xα|X) Pairwise Granger Causality (PWGC) δbv (β → α) = log (xα|Xα) (xα|Xα, Xβ) Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 17. Partial conditioning Conditioned Granger Causality (CGC) δmv (β → α) = log (xα|X Xβ) (xα|X) Pairwise Granger Causality (PWGC) δbv (β → α) = log (xα|Xα) (xα|Xα, Xβ) Partially conditioned Granger causality (PCGC) δY c (β → α) = log (xα|Xα, Y) (xα|Xα, Xβ, Y) Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 18. Partial conditioning Fix a subset Y of the variables in X, excluding Xα and Xβ Partially conditioned Granger causality δY c (β → α) = log (xα|Xα, Y) (xα|Xα, Xβ, Y) Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 19. Partial conditioning Fix a subset Y of the variables in X, excluding Xα and Xβ Partially conditioned Granger causality δY c (β → α) = log (xα|Xα, Y) (xα|Xα, Xβ, Y) Strategy 1, Information-Based (IB) Y maximizes the mutual information I{Xβ; Y} among all the subsets of nd variables Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 20. Partial conditioning Fix a subset Y of the variables in X, excluding Xα and Xβ Partially conditioned Granger causality δY c (β → α) = log (xα|Xα, Y) (xα|Xα, Xβ, Y) Strategy 1, Information-Based (IB) Y maximizes the mutual information I{Xβ; Y} among all the subsets of nd variables Strategy 2, Pairwise-Based (PB) Select Y = {Xγ}nd γ=1 as the nd variables with the maximal pairwise GC δbv (γ → α) w.r.t. that target node, excluding Xβ Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 21. Information-based partial conditioning Given the previous Yk−1 , the set Yk is obtained adding the variable with greatest information gain Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 22. Information-based partial conditioning Given the previous Yk−1 , the set Yk is obtained adding the variable with greatest information gain This is repeated until nd variables are selected Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 23. Information-based partial conditioning Given the previous Yk−1 , the set Yk is obtained adding the variable with greatest information gain This is repeated until nd variables are selected Marinazzo et al. Comput. Mat. Methods Med. 2012, Wu et al. Brain Connectivity 2013 Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 24. Pairwise-based conditioned Granger causality Facts CGC performs poorly in presence of redundancy Partial conditioning does not solve redundancy Information about redundancy can be extracted from PWGC Proposed approach Some links inferred from PWGC are retained and added to those obtained by CGC The PWGC links that are discarded are those that can be derived as indirect links from the CGC pattern Stramaglia, Cortes and Marinazzo, New Journal of Physics 2014 Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 25. Interim summary on partial conditioning Synergy The search for synergetic contributions in information flow is equivalent to the search for suppressors PWGC bad, CGC ok, PCGC even better if the selection strategy succeeds in picking the suppressors Information-based PCGC better with redundancy Pruning-based PCGC better in tree-like structures Redundancy Bad for CGC, and not solvable Indirect connections of CGC from PWGC links Links not explained as indirect connections (redundant) are merged into CGC Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 26. Synergy and redundancy Pairwise information measures are commonly agreed upon (e.g. mutual information) Shannon’s information theory does not fit multivariate information measures dealing with the notions of synergy and redundancy (Williams, Beer, Lizier, Wibral, Faes, Barrett) All the proposed partial information decompositions, in the Gaussian case, lead to the following (undesirable) results: (i) redundancy is the minimum of MI between the target and each source (ii) synergy is the extra information provided by the weaker source when the stronger source is known (Barrett, PRE 2015) Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 27. Joint information Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 28. Joint information Let’s go for an operative and practical definition Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 29. Joint information Let’s go for an operative and practical definition Relation (B and C) → A synergy: (B and C) contributes to A with more information than the sum of its variables redundancy: (B and C) contributes to A with less information than the sum of its variables Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 30. Generalization of GC for sets of driving variables Conditioned Granger Causality in a multivariate system δX(B → α) = log (xα|X B) (xα|X) Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 31. Generalization of GC for sets of driving variables Conditioned Granger Causality in a multivariate system δX(B → α) = log (xα|X B) (xα|X) Unnormalized version δu X(B → α) = (xα|X B) − (xα|X) Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 32. Generalization of GC for sets of driving variables Conditioned Granger Causality in a multivariate system δX(B → α) = log (xα|X B) (xα|X) Unnormalized version δu X(B → α) = (xα|X B) − (xα|X) An interesting property If {Xβ}β∈B are statistically independent and their contributions in the model for xα are additive, then δu X(B → α) = β∈B δu X(β → α). We remark that this property does not hold for the standard definition of Granger causality neither for entropy-rooted quantities, due to the presence of the logarithm. Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 33. Question from the audience: What does it ever mean to have an unnormalized measure of Granger causality? Don’t you lose any link with information? Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 34. Question from the audience: What does it ever mean to have an unnormalized measure of Granger causality? Don’t you lose any link with information? Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 35. Define synergy and redundancy in this framework Synergy δu X(B → α) > β∈B δu XB,β(β → α) Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 36. Define synergy and redundancy in this framework Synergy δu X(B → α) > β∈B δu XB,β(β → α) Redundancy δu X(B → α) < β∈B δu XB,β(β → α) Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 37. Define synergy and redundancy in this framework Synergy δu X(B → α) > β∈B δu XB,β(β → α) Redundancy δu X(B → α) < β∈B δu XB,β(β → α) Pairwise syn/red index ψα(i, j) = δu Xj(i → α) − δu X(i → α) = δu X({i, j} → α) − δu X(i → α) − δu X(j → α) Stramaglia et al. IEEE Trans Biomed. Eng. 2016 Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 38. ψ as cumulant expansion of the prediction error (xα|Xα) − (xα|X) = B⊂X S(B). The Moebius inversion formula allows to reconstruct S(B). Calling |nB | and |nΓ| the number of variables in the subsets B and Γ respectively, and exploiting also the relation: Γ⊂B (−1) |nΓ| = 0, leads to the cumulant expansion: S(B) = Γ⊂B (−1) |nB |+|nΓ| δ u B (Γ → α). The first order cumulant is then S(i) = δ u i (i → α), the second cumulant is S(i, j) = δ u ij ({ij} → α) − δ u ij (i → α) − δ u ij (j → α) , the third cumulant is S(i, j, k) = δ u ijk ({ijk} → α) − δ u ijk ({ij} → α) −δ u ijk ({jk} → α) − δ u ijk ({ik} → α) +δ u ijk (i → α) + δ u ijk (j → α) + δ u ijk (k → α) , (1) and so on. The index ψ may then be seen as the order two cumulant of the expansion of the prediction error of the target variable; Stramaglia et al. IEEE Trans Biomed. Eng. 2016 Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 39. Predictive multivariate models Faes et al. Phil. Trans. A 2016 Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 40. Variance decomposition Faes et al. Phil. Trans. A 2016 Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 41. Entropy decomposition Faes et al. Phil. Trans. A 2016 Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 42. Overview Faes et al. Phil. Trans. A 2016 Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 43. fMRI data, N=90, Human Connectome Project Regions forming redundant and synergetic multiplets with a representative region (black) A B C D A B DC RED SYN RED SYN Hierarchical structure of synergy and redundancy networks Stramaglia et al. IEEE Trans Biomed. Eng. 2016 Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 44. Take-home message With a wider applicability in sight, we advocate an intuitive rather than axiomatic view of partial information decomposition We aim to detect the presence of redundant and synergetic multiplets rather than precisely measure synergy and redundancy Variance decomposition is a viable alternative to entropy decomposition Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems
- 45. Thanks @dan marinazzo http://users.ugent.be/~dmarinaz/ Marinazzo, Faes, Stramaglia Synergy and redundancy in dynamical systems