Computing Inner Eigenvalues of Matrices in Tensor Train Matrix Format

GAMM Workshop Applied and
Numerical Linear Algebra 2011
September 22, 2011, Bremen

Computing Inner Eigenvalues of Matrices in
Tensor Train Matrix Format
Thomas Mach
joint work with Peter Benner

Max Planck Institute for Dynamics of Complex Technical Systems
Computational Methods in Systems and Control Theory
Magdeburg
MAX PLANCK INSTITUTE
FOR DYNAMICS OF COMPLEX
TECHNICAL SYSTEMS
MAGDEBURG

Max Planck Institute Magdeburg Thomas Mach, Computing Eigenvalues of Matrices in TTM Format 1/21

Tensor Trains PINVIT and Folded Spectrum Method Numerical Results

Outline

1 Tensor Trains

2 PINVIT and Folded Spectrum Method

3 Numerical Results



Tensor Trains
[Oseledets, Tyrtyshnikov ’09]

d
T ∈ Rm



Tensor Trains

d
T ∈ Rm

r
T = α=1 U1 (i1 , α)U2 (i2 , α) · · · Ud (id , α), with Uj (·, α) ∈ Rm

U1 (i1 , α)

U4 (i4 , α) α U2 (i2 , α)

U3 (i3 , α)



Tensor Trains

d
T ∈ Rm

r
r r d
T (i1 , i2 , . . . , id ) = α1 =1 · · · αd =1 Gα1 ,...,αd
1 d
j=1 Uj (ij , αj ), with
G ∈ Rr1 ×···×rd and Uj (·, αj ) ∈ Rm

U1 (i1 , α1 ) U3 (i3 , α3 )
α1 α3

U4 (i4 , α4 ) α4 G (α1 , . . . , α4 ) α2 U2 (i2 , α2 )



Tensor Trains

d
T ∈ Rm

r
r r d
T (i1 , i2 , . . . , id ) = α1 =1 · · · αd =1 Gα1 ,...,αd
1 d
j=1 Uj (ij , αj ), with
G ∈ Rr1 ×···×rd and Uj (·, αj ) ∈ Rm

T (i1 , i2 , . . . , id ) = G1 (i1 , α1 )G2 (α1 , i2 , α2 ) · · ·
α1 ,...,αd−1

Gd−1 (αd−2 , id−1 , αd−1 )Gd (αd−1 , id )



Tensor Trains

T (i1 , i2 , . . . , id ) = G1 (i1 , α1 )G2 (α1 , i2 , α2 ) · · ·
α1 ,...,αd−1

Gd−1 (αd−2 , id−1 , αd−1 )Gd (αd−1 , id )

G1 (i1 , α1 ) α1 G2 (α1 , i2 , α2 ) α2 G3 (α2 , i3 , α3 ) α3

··· αd−1 Gd (αd−1 , id )



Tensor Train Matrix Format (TTM)
[Oseledets ’10]

G1 (i1 , j1 , α1 ) α1 G2 (α1 , i2 , j2 , α2 ) α2

··· αd−1 Gd (αd−1 , id , jd )

d ×md
M ∈ Rm



[Oseledets ’10]

G1 (i1 , j1 , α1 ) α1 G2 (α1 , i2 , j2 , α2 ) α2

··· αd−1 Gd (αd−1 , id , jd )

d ×md
M ∈ Rm

M (i1 , i2 , . . . , id ; j1 , j2 , . . . , jd ) =
α1 ,...,αd−1 G1 (i1 , j1 , α1 )G2 (α1 , i2 , j2 , α2 ) · · · · · · Gd (αd−1 , id , jd )



[Oseledets ’10]

G1 (i1 , j1 , α1 ) α1 G2 (α1 , i2 , j2 , α2 ) α2

··· αd−1 Gd (αd−1 , id , jd )

d ×md
M ∈ Rm

M (i1 , i2 , . . . , id ; j1 , j2 , . . . , jd ) =

TTM is a data-sparse matrix format.



[Oseledets ’10]

G1 (i1 , j1 , α1 ) α1 G2 (α1 , i2 , j2 , α2 ) α2

··· αd−1 Gd (αd−1 , id , jd )

d ×md
M ∈ Rm

M (i1 , i2 , . . . , id ; j1 , j2 , . . . , jd ) =

TTM is a data-sparse matrix format.
m = 2 ⇒ QTT matrix format


Matrix Vector Product in TT and TTM Format
[Oseledets ’10]

T ∈ (Q)TT , M ∈ (Q)TTM ⇒ MT = W ∈ (Q)TT



[Oseledets ’10]

W (i1 , i2 , . . . , id ) = M(i1 , j1 , i2 , j2 , . . . , id , jd )T (j1 , j2 , . . . , jd )
j1 ,j2 ,...,jd



[Oseledets ’10]

j1 ,j2 ,...,jd

G1 (i1 , j1 , α1 ) j1 H1 (j1 , β1 )

α1 β1

G2 (α1 , i2 , j2 , α2 ) j2 H2 (β1 , j2 , β2 )

α2 β2
.
. .
.
. .


[Oseledets ’10]

j1 ,j2 ,...,jd

j1
G1 (i1 , j1 , αK1 (i1 , (α1 , β1 )) H1 (j1 , β1 )
1)

α1 β1

G2 (α1 , i2 , j2 , α2 ) j2 H2 (β1 , j2 , β2 )

α2 β2
.
. .
.
. .


[Oseledets ’10]

j1 ,j2 ,...,jd

j1
G1 (i1 , j1 , αK1 (i1 , (α1 , β1 )) H1 (j1 , β1 )
1)

α1 β1

G2 (α1 , iK2j((α1 ,)β1 ),j22 , (α2H22 ))1 , j2 , β2 )
2 , 2 , α2 i , β (β

α2 β2
.
. .
.
. .


[Oseledets ’10]

j1 ,j2 ,...,jd

j1
G1 (i1 , j1 , αK1 (i1 , (α1 , β1 )) H1 (j1 , β1 )
1)

(α1 , β1 )

G2 (α1 , iK2j((α1 ,)β1 ),j22 , (α2H22 ))1 , j2 , β2 )
2 , 2 , α2 i , β (β

α2 β2
.
. .
.
. .


[Oseledets ’10]

j1 ,j2 ,...,jd

j1
G1 (i1 , j1 , αK1 (i1 , (α1 , β1 )) H1 (j1 , β1 )
1)

o n
ati
(α1 , β1 )
c
t run
G2 (α1 , iK2j((α1 ,)β1 ),j22 , (α2H22 ))1 , j2 , β2 )
2 , 2 , α2 i , β (β

+
α2 β2
.
. .
.
. .


Inversion of a Matrix in TTM Format
[Schulz 1933, Oseledets ’10]

Newton-Schulz Iteration
Xk+1 = 2Xk − Xk MXk

X0 initial approximation to M −1 with,

ρ(MX0 − I ) < 1.
2
If M is symmetric, positive deﬁnite, then X0 = M I is an
2
admissible initial approximation.

Hk+1 = I − Yk
Yk+1 = Yk Hk+1
Xk+1 = Hk+1 Xk



Inversion of a Matrix in TTM Format
[Schulz 1933, Oseledets ’10]

Newton-Schulz Iteration
Xk+1 = 2Xk − Xk MXk

X0 initial approximation to M −1 with,

ρ(MX0 − I ) < 1.
2
If M is symmetric, positive deﬁnite, then X0 = M I is an
2
admissible initial approximation.

Hk+1 = T (I − Yk , )
Yk+1 = T (Yk Hk+1 , )
Xk+1 = T (Hk+1 Xk , )



Eigenvalue Problem

Problem Setting
d d
Assume M ∈ R2 ×2 is given in TTM. M is sym. pos. deﬁnite.
d
Compute eigenvalue λ and eigenvector v ∈ R2 of M.
d
R2 Mv = λv

quantum molecular dynamics
[Lebedeva ’11]



Preconditioned Inverse Iteration
[Knyazev, Neymeyr, et al.]

Deﬁnition
The function
x T Mx
µ(x) = µ(x, M) =
xT x
is called the Rayleigh quotient.




Deﬁnition
The function
x T Mx
µ(x) = µ(x, M) =
xT x

Minimize the Rayleigh quotient by a gradient method:
2
xi+1 := xi − α µ(xi ), µ(x) = T (Mx − xµ(x)) ,
x x




Residual r (x) = Mx − xµ(x).

2
x x




Deﬁnition
The function
x T Mx
µ(x) = µ(x, M) =
xT x

2
x x
+ preconditioning ⇒ update equation:
xi+1 := xi − B −1 (Mxi − xi µ(xi )) .




Preconditioned residual B −1 r (x) = B −1 (Mx − xµ(x)).
⇒ inexact Newton-method

+ preconditioning ⇒ update equation:
xi+1 := xi − B −1 (Mxi − xi µ(xi )) .



[Knyazev, Neymeyr 2009]

xi+1 := xi − B −1 (Mxi − xi µ(xi ))

If
M ∈ Rn×n symmetric positive deﬁnite and
B −1 approximates the inverse of M, so that

I − B −1 M M
≤ c < 1,

then Preconditioned INVerse ITeration (PINVIT) converges and
the number of iterations is independent of n.




The residual

ri = Mxi − xi µ(xi )

converges to 0, so that

ri 2 <

is a useful termination criterion.



Algorithm

The number of iterations is independent of matrix size n = 2d .

PINVIT(1,s)
Input: M ∈ Rn×n , X0 ∈ Rn×s (X0 X0 = I , e.g. randomly chosen)
T

Output: Xp ∈ R n×s , µ ∈ Rs×s , with MX − X µ ≤
p p

Approximative inversion B −1 ≈ (M)−1
R := MX0 − X0 µ, µ = X0 MX0T

for (i := 1; R F > ; i + +) do
Xi := Orthogonalize Xi−1 − B −1 R
R := MXi − Xi µ, µ = XiT MXi
end



Algorithm


PINVIT(1,s)
T

p p

R := MX0 − X0 µ, µ = X0 MX0T

for (i := 1; R F > ; i + +) do
end Newton-Schulz iteration:
Bk+1 = 2Bk − Bk MBk
[Oseledets ’10] for TTM



Algorithm


PINVIT(1,s)
T

p p

R := MX0 − X0 µ, µ = X0 MX0T

for (i := 1; R F > ; i + +) do
end

TTM-TT products



Folded Spectrum Method

How to ﬁnd λi ?
If i = n − s with s < O(log n),
use subspace version PINVIT(·,s).

...
0λn λn−1 λn−2 λ1




How to ﬁnd λi ?
If i = n − s with s log n?

... ...
0λn λi+1 λi λi−1 λ1




How to ﬁnd λi ?
If i = n − s with s log n,
shift with σ near λi .

... ...
σ




How to ﬁnd λi ?
If i = n − s with s log n,
shift with σ near λi .

... ...
σ

But (M − σI) is not positive deﬁnite.



Folded Spectrum Method [Wang, Zunger 1994, Morgan 1991]

Mσ = (M − σI)2

Mσ is s.p.d., if M is s.p.d. and σ = λi .
Assume all eigenvalues of Mσ are simple.

Mv = λv ⇔ Mσ v = (M − σI)2 v
= M 2 v − 2σMv + σ 2 v
= λ2 v − 2σλv + σ 2 v
= (λ − σ)2 v





Mσ = (M − σI)2


(2, v2 ), (3, v3 ), σ = 2.5 ⇒ Mσ has eigenvalue 0.25 of
multiplicity 2.





Mσ = (M − σI)2


multiplicity 2.
PINVIT computes v ∈ span(v2 , v3 ). ⇒ v T Mv /v T v ∈ [2, 3]





Mσ = (M − σI)2


multiplicity 2.
PINVIT computes v ∈ span(v2 , v3 ). ⇒ v T Mv /v T v ∈ [2, 3]
Use PINVIT to compute V ∈ Rn×2
⇒ Λ(V T MV /V T V ) = {2, 3}



1 Choose σ.
2 Compute
2
a) Mσ := (M − σI) and
b) B −1 :≈ Mσ .
−1

3 Use PINVIT to ﬁnd the smallest eigenpair (µσ , v ) of Mσ .
4 Compute µ := v T Mv /v T v .
(µ, v ) is the nearest eigenpair to σ.



1 Choose σ.
2 Compute
2
b) B −1 :≈ Mσ .
−1


In TTM M can be shifted, squared and inverted with
reasonable costs.



1 Choose σ.
2 Compute
2
b) B −1 :≈ Mσ .
−1


In TTM M can be shifted, squared and inverted with
reasonable costs.

If M is sparse, then squaring and inverting is prohibitive.



Drawbacks

The condition number of (M − σI )2 is larger.
−1
⇒ The computation of Mσ is more expensive.
−1
⇒ Mσ has larger local ranks.
−1
⇒ Mσ v is more expensive.



Drawbacks

The condition number of (M − σI )2 is larger.
−1
⇒ The computation of Mσ is more expensive.
−1
⇒ Mσ has larger local ranks.
−1
⇒ Mσ v is more expensive.

Multiple eigenvalues of Mσ may lead to
incomplete subspace information.
⇒ v T Mv /v T v does not approximate λ.



Numerical Results

Numerical Results



TT-Toolbox 2.1
[Oseledets et al. ’09–’11]

We use TT-Toolbox 2.1 for MATLAB from I.V. Oseledets at al.



2D Laplace
d )2 ×(2d )2
M = α∆2 = α (∆1 ⊗ I + I ⊗ ∆1 ) ∈ R(2 , with
∆1 = tridiag([−1, 2, −1]).
without shift, 3 smallest eigenvalues
n d tinv in s tPINVIT in s # it. error
64 3 0.449 1.577 17 1.2230 e−07
256 4 0.345 1.376 18 1.1450 e−07
1 024 5 0.997 3.236 27 2.7432 e−07
4 096 6 2.347 8.172 19 1.8911 e−07
16 384 7 4.789 19.885 16 6.4020 e−08
65 536 8 11.895 43.717 20 1.0688 e−07
262 144 9 19.076 63.727 20 4.1304 e−09
1 048 576 10 29.865 99.808 20 3.7722 e−09
4 194 304 11 110.712* 331.059 27 5.8789 e−10
16 777 216 12 165.560* 439.341 23 4.5062 e−09
67 108 864 13 240.226* 587.796 26 8.9795 e−09


2D Laplace
d )2 ×(2d )2
M = α∆2 = α (∆1 ⊗ I + I ⊗ ∆1 ) ∈ R(2 , with
∆1 = tridiag([−1, 2, −1]).

with shift σ = 203.3139, folded spectrum method, 4 eigenvalues

64 3 0.796 0.843 8 5.5601 e−10
256 4 3.360 3.053 45 2.9268 e−09
1 024 5 22.659 4.581 20 1.1411 e−10
4 096 6 60.755 32.264 31 2.5295 e−12
16 384 7 411.165* 80.827 28 6.3665 e−12
65 536 8 1 808.302* 344.696 27 2.5835 e−11



3D Laplace
M = α∆3 = α (∆1 ⊗ I ⊗ I + I ⊗ ∆1 ⊗ I + I ⊗ I ⊗ ∆1 ) ,
d 3 d 3
M ∈ R(2 ) ×(2 ) .


64 2 0.219 4.518 30 2.4762 e−07
512 3 0.561 6.742 27 2.9333 e−07
4 096 4 1.109 23.715 24 1.5441 e−07
32 768 5 7.197 99.543 28 9.1209 e−09
262 144 6 11.052 249.084 24 1.9956 e−08
2 097 152 7 20.893 923.874 26 1.6577 e−07
16 777 216 8 34.937 5 131.664 34 1.1977 e−08



3D Laplace
M = α∆3 = α (∆1 ⊗ I ⊗ I + I ⊗ ∆1 ⊗ I + I ⊗ I ⊗ ∆1 ) ,
d 3 d 3
M ∈ R(2 ) ×(2 ) .

with shift σ = 230.6195, folded spectrum method, 6 eigenvalues

512 3 14.300 45.979 29 2.8479 e−07
4 096 4 137.502 293.154 54 1.8060 e−08
32 768 5 1 952.980* 1 223.395 19 1.0971 e−11
262 144 6 37 149.998* out of mem1

1
canceled after 60 hours while using > 300 GB RAM


4D Laplace
M = α∆4 = α (∆2 ⊗ I + I ⊗ ∆2 ) ,
d 4 d 4
M ∈ R(2 ) ×(2 )


256 2 0.240 9.265 36 2.5997 e−07
4 096 3 0.873 23.993 28 1.7104 e−07
65 536 4 1.436 119.348 28 5.9876 e−08
1 048 576 5 5.975 497.812 32 2.2100 e−08
16 777 216 6 12.655 1 710.326 31 7.9299 e−09
268 435 456 7 23.628 7 898.374 41 5.1963 e−10



Conclusions
Finding the eigenvalues by PINVIT is cheap and storage
eﬃcient.



Conclusions
eﬃcient.
The folded spectrum method enables us to compute inner
eigenvalues, too.



Conclusions
eﬃcient.
eigenvalues, too.
The use of the folded spectrum method leads to not well
conditioned problems.



Conclusions
eﬃcient.
eigenvalues, too.
Choose the shift and the subspace dimension carefully.



Conclusions
eﬃcient.
eigenvalues, too.
Choose the shift and the subspace dimension carefully.

Thank you for your attention.


Computing Inner Eigenvalues of Matrices in Tensor Train Matrix Format

More Related Content

What's hot (18)

Similar to Computing Inner Eigenvalues of Matrices in Tensor Train Matrix Format (20)

More from Thomas Mach (9)

Recently uploaded (20)

Computing Inner Eigenvalues of Matrices in Tensor Train Matrix Format