Understanding High-dimensional Networks for Continuous Variables Using ECL
1 like402 views
The document discusses high-dimensional networks for continuous variables, focusing on the challenges of estimating covariance matrices when the number of variables exceeds the sample size. It introduces a new method called Convex Sparse Cholesky Selection (CSCS) to effectively estimate sparse covariance matrices, demonstrating its superiority through a case study on call center data. The method achieves improved forecasting accuracy compared to traditional sample covariance techniques.
![CSCS in ECL
CSCS2(DATASET(Elem) YY, REAL lambda3, REAL tol2=0.00001,
UNSIGNED maxIter2=100) := FUNCTION
nobs := MAX(YY,x);
pvars := MAX(YY,y);
S:= Mat.Scale(Mat.Mul(Mat.Trans(YY),YY),(1/nobs));
S1 := DISTRIBUTE(NORMALIZE(S, (pvars DIV 2),
TRANSFORM(DistElem, SELF.nid:=COUNTER, SELF:=LEFT)), nid);
L:= Identity(pvars);
LL:= DISTRIBUTE(L,nid);
L11 := PROJECT(CHOOSEN(S1(x=1 AND y=1 AND nid=1),1),
TRANSFORM(DistElem, SELF.x := 1, SELF.y := 1,
SELF.value:=1/LEFT.value, SELF.nid := 1, SELF:=[]));
newL := LL(x <> 1) + L11;
newLL := LOOP(newL,COUNTER<pvars,OuterOuterBody(ROWS(LEFT), S1,
COUNTER, lambda3, maxIter2, tol2));
RETURN newLL;
END;
22 / 22](https://image.slidesharecdn.com/008rahmankhareuf-161020203416/85/Understanding-High-dimensional-Networks-for-Continuous-Variables-Using-ECL-23-320.jpg)
Understanding High-dimensional Networks for Continuous Variables Using ECL
- 1. Understanding high-dimensional networks for continuous variables using ECL Kshitij Khare and Syed Rahman Department of Statistics University of Florida
- 2. Motivation • Availability of high-dimensional data from various applications • Number of variables (p) much larger than (or sometimes comparable to) the sample size (n) • Examples: Biology: gene expression data Environmental science: climate data on spatial grid Finance: returns on thousands of stocks 1 / 22
- 3. Goal: Understanding relationships between variables • Common goal in many applications: Understand complex network of relationships between variables • Covariance matrix: a fundamental quantity to help understand multivariate relationships • Even if estimating the covariance matrix is not the end goal, it is a crucial first step before further analysis 2 / 22
- 4. Quick recap: What is a covariance matrix? • The covariance of two variables/features (say two stock prices) is a measure of linear dependence between these variables • Positive covariance indicates similar behavior, Negative covariance indicates opposite behavior, zero covariance indicates lack of linear dependence 3 / 22
- 5. Lets say we have five stock prices S1, S2, S3, S4, S5. The covariance matrix of these five stocks looks like S2 S2 S3 S4 S5 S1 S1 S3 S4 S5 4 / 22
- 6. Challenges in high-dimensional estimation • If p = 1000, we need to estimate roughly 1 million covariance parameters • If sample size n is much smaller (or even same order) than p, this is not viable • The sample covariance matrix (classical estimator) can perform very poorly in high-dimensional situations (not even invertible when n < p) 5 / 22
- 7. Is there a way out? • Reliably estimate small number of parameters in the covariance matrix or an appropriate function of the covariance matrix • Set insignificant parameters to zero • Sparsity pattern (pattern of 0s) can be represented by graphs/networks 6 / 22
- 8. Directed acyclic graph models: Sparsity in the Cholesky Parameter • Set entries of L to be zero: corresponds to assuming certain conditional independences • Sparsity pattern in L can be represented by an directed graph • Build a graph from sparse L 7 / 22
- 9. Directed acyclic graph models: Sparsity in the Cholesky Parameter • Consider Cholesky decompostion of Σ−1 = LtL • 4.29 0.65 0.76 0.80 0.65 4.25 0.76 0.80 0.76 0.76 4.16 0.8 0.80 0.80 0.80 4.0 Σ−1 = 2.0 0.2 0.3 0.4 0 2.0 0.3 0.4 0 0 2.0 0.4 0 0 0 2.0 Lt 2.0 0 0 0 0.2 2.0 0 0 0.3 0.3 2.0 0 0.4 0.4 0.4 2 L • Entires of L have a concrete and direct interpretation in terms of appropriate conditional covariances 8 / 22
- 10. Directed acyclic graph models: Sparsity in the Cholesky Parameter • Consider Cholesky decompostion of Σ−1 = LtL • Set entries of L to be zero • Sparsity pattern in L can be represented by a directed graph L = A B C D 2 0 0 0 A 0.2 2 0 0 B 0.3 0 2 0 C 0 0.4 0.4 2 D A B C D 9 / 22
- 11. STATISTICAL CHALLENGE How do we estimate a covariance matrix with a sparse Cholesky factor based on data? 10 / 22
- 12. Convex Sparse Cholesky Selection(CSCS) • Obtain a sparse estimate for L by minimizing the objective function: QCSCS (L) = tr(Lt LS) − 2log|L| log-likelihood + λ 1 j<i p |Lij |. penalty term to induce sparsity/zeros • λ (chosen by the user) controls the level of sparsity in the estimator • Larger the λ, sparser the estimator 11 / 22
- 13. CSCS method: Comparison with other methods METHOD Property SparseCholesky SparseDAG CSCS No constraints on sparsity pattern + + + No constraints on D + + Convergence guarantee to acceptable global minimum + + Asymptotic consistency (n, p → ∞) + + CSCS outperfroms and improves on existing methods! 12 / 22
- 14. Breaking up the objective function row-wise • QCSCS (L) breaks up as a sum of independent functions of the rows of L • If F(x, y, z) = F1(x) + F2(y) + F3(z), then to minimize F(x, y, z), we can minimize F1(x) with respect to x, F2(y) with respect to y and F3(z) with respect to z • QCSCS (L) = Q1(L1.) Function of entries of 1st row of L + . . . + Qp(Lp.) Function of entries of pth row of L 13 / 22
- 15. Call center data • The data come from one call centre in a major U.S. northeastern financial organisation. • For each day, a 17-hour period was divided into p = 102 10-minute intervals, and the number of calls arriving at the service queue during each interval was counted • n = 239 days using the singular value decomposition to screen out outliers that include holidays and days when the recording equipment was faulty • Hence 102 2 = 10506 parameters need to be estimated 14 / 22
- 16. GOAL Build a predictor to forecast the calls coming in during the second half of the day based on the number of calls during the first half of the day 15 / 22
- 17. Call Center data • The best mean squared error forecast of y (2) i , calls coming in during the second half of day t, using y (1) i , calls coming in during the first half of the day is ˆy (2) it = µ2 + Σ21Σ−1 11 (y (1) it − µ1) where µ2, µ1, Σ21 and Σ11 must be estimated. • To evaluate the performance of the sample covariance versus CSCS, we use Average Absolute Forecast Error, defined as, AEt = 1 34 239 i=206 ˆy (2) it − y (2) it • For CSCS, λ is chosen using 5- fold cross-validation 16 / 22
- 18. Call center data 0.5 1.0 1.5 50 60 70 80 90 100 Time AE Method S CSCS Figure : Average Absolute Forecast error using cross-validation CSCS outperforms the sample covariance matrix 46 out of 51 times! 17 / 22
- 19. Call center data 0 250 500 750 1000 Non−parallel Parallel System Timing Figure : Timing Comparison between Parallel and Non-parallel Versions of CSCS 18 / 22
- 20. Algorithm1 Algorithm 1 Cyclic coordinatewise algorithm for hk,A,λ Input: Fix k, A, λ Input: Fix maximum number of iterations: rmax Input: Fix initial estimate: ˆx(0) Input: Fix convergence threshold: Set r ← 1 converged = FALSE Set ˆxcurrent ← ˆx(0) repeat ˆxold ← ˆxcurrent ) for j ← 1, 2, ..., k − 1 do ˆxcurrent j ← Tj (j, λ, A, ˆxold ) end for ˆxcurrent k ← Tk (λ, A, ˆxold ) ˆxr ← ˆxcurrent Convergence Checking if ˆxcurrent − ˆxold < then converged = TRUE else r ← r + 1 end if until converged = TRUE or r > rmax Return final estimate: ˆxr 19 / 22
- 21. Algorithm1 in ECL CSCS_h1(DATASET(xElement) xx0, DATASET(DistElem) A, UNSIGNED k, REAL lambda, UNSIGNED maxIter = 100, REAL tol = 0.00001):= FUNCTION out := LOOP(xx0,(COUNTER<=maxIter AND MaxAbsDff(ROWS(LEFT))>tol), OuterBody(ROWS(LEFT), A, COUNTER, k, lambda)); RETURN SORT(PROJECT(out(typ=xType.x), DistElem),x,y); END; 20 / 22
- 22. CSCS algorithm Algorithm 2 CSCS Algorithm Input: Data Y1, Y2, ..., Yn and λ Input: Fix maximum number of iterations: rmax Input: Fix initial estimate: ˆL(0) Input: Fix convergence threshold: Can be done in parallel for i ← 1, 2, . . . , p do (ˆηi )(0) ← ith row of ˆL(0) Set ˆηi to be minimizer of objective function QCSCS,i obtained by using Algorithm 1 with k = 1, A = Si , λ, rmax , ˆx(0) = (ˆηi )(0) , end for Construct ˆL by setting its ith row (up to the diagonal) as ˆηi Return final estimate: ˆL 21 / 22
- 23. CSCS in ECL CSCS2(DATASET(Elem) YY, REAL lambda3, REAL tol2=0.00001, UNSIGNED maxIter2=100) := FUNCTION nobs := MAX(YY,x); pvars := MAX(YY,y); S:= Mat.Scale(Mat.Mul(Mat.Trans(YY),YY),(1/nobs)); S1 := DISTRIBUTE(NORMALIZE(S, (pvars DIV 2), TRANSFORM(DistElem, SELF.nid:=COUNTER, SELF:=LEFT)), nid); L:= Identity(pvars); LL:= DISTRIBUTE(L,nid); L11 := PROJECT(CHOOSEN(S1(x=1 AND y=1 AND nid=1),1), TRANSFORM(DistElem, SELF.x := 1, SELF.y := 1, SELF.value:=1/LEFT.value, SELF.nid := 1, SELF:=[])); newL := LL(x <> 1) + L11; newLL := LOOP(newL,COUNTER<pvars,OuterOuterBody(ROWS(LEFT), S1, COUNTER, lambda3, maxIter2, tol2)); RETURN newLL; END; 22 / 22