power method.pdf
- 1. Lecture 4 Eigenvalue problems Weinan E1,2 and Tiejun Li2 1 Department of Mathematics, Princeton University, weinan@princeton.edu 2 School of Mathematical Sciences, Peking University, tieli@pku.edu.cn No.1 Science Building, 1575
- 2. Review Power method QR method Outline Review Power method QR method
- 3. Review Power method QR method Eigenvalue problem I Eigenvalue problem Find λ and x such that Ax = λx, x 6= 0. λ is called the eigenvalues of A which satisfies the eigenpolynomial det(λI − A) = 0, x is called the eigenvector corresponds to λ. I The are n complex eigenvalues according to Fundamental Theorem of Algebra.
- 4. Review Power method QR method Eigenvalue problem for symmetric matrix Theorem (For symmetric matrix) The eigenvalue problem for real symmetric matrix has the properties 1. The eigenvalues are real, i.e. λi ∈ R, i = 1, . . . , n. 2. The multiplicity of a eigenvalue to the eigenpolynomial = the number of linearly independent eigenvectors corresponding to this eigenvalue. 3. The linearly independent eigenvectors are orthogonal each other. 4. A has the following spectral decomposition A = QΛQT where Q = (xT 1 , · · · , xT n ), Λ = diag(λ1, · · · , λn).
- 5. Review Power method QR method Variational form for symmetric matrix Theorem (Courant-Fisher Theorem) Suppose A is symmetric, and the eigenvalues λ1 ≥ · · · ≥ λn, if we define the Rayleigh quotient as RA(u) = uT Au uT u then we have, λ1 = max RA(u), λn = min RA(u).
- 6. Review Power method QR method Jordan form for non-symmetric matrix Theorem (Jordan form) Suppose A ∈ Cn×n , if A has r different eigenvalues λ1, . . . , λr with multiplicity n1, . . . , nr, then there exists nonsingular P such that A has the following decomposition A = P JP −1 where J = diag(J1, . . . , Jr), and Jk = λk 1 λk ... ... 1 λk , k = 1, . . . , r
- 7. Review Power method QR method Gershgorin’s disks theorem Definition Suppose that n ≥ 2 and A ∈ Cn×n . The Gershgorin discs Di, i = 1, 2, . . . , n, of the matrix A are defined as the closed circular regions Di = {z ∈ C : |z − aii| ≤ Ri} in the complex plane, where Ri = n X j=1, j6=i |aij| is the radius of Di. Theorem (Gershgorin theorem) All eigenvalues of the matrix A lie in the region D = ∪n i=1Di, where Di are the Gershgorin discs of A.
- 8. Review Power method QR method Gershgorin’s disks theorem Geometrical interpretation of Gershgorin’s disks theorem for A = 30 1 2 4 15 −4 −1 0 3 15 3 30
- 9. Review Power method QR method Outline Review Power method QR method
- 10. Review Power method QR method Basic idea of power method I First suppose A is diagonizable, i.e. A = P ΛP −1 and Λ = diag(λ1, . . . , λn). We will assume |λ1| |λ2| ≥ · · · ≥ |λn| in the follows and assume xi are the eigenvectors corresponding to λi. I For any initial u0 = α1x1 + · · · + αnxn, where αk ∈ C. We have Ak u0 = n X j=1 αjAk xj = n X j=1 αjλk j xj = λk 1 α1x1 + n X j=2 αj λj λ1 k xj
- 11. Review Power method QR method Power method I We have lim k→∞ Ak u0 λk 1 = α1x1. I Though λ1 and α1 is not known, the direction of x1 is enough! I Power method 1. Set up initial u0, k = 1; 2. Perform a power step yk = Auk−1; 3. Find the maximal component for the absolute value of µk = kykk∞; 4. Normalize uk = 1 µk yk and repeat. I We will have uk → x1, µk → λ1.
- 12. Review Power method QR method Power method: example I Example 1: compute the eigenvalue with largest modulus for A = 30 2 3 13 5 11 10 8 9 7 6 12 4 14 15 1 I Example 2: compute the eigenvalue with largest modulus for the second order ODE example (n=30)
- 13. Review Power method QR method Power method Theorem (Convergence of power method) If the eigenvalues of A has the order |λ1| |λ2| ≥ · · · ≥ |λp| (counting multiplicity), and the algebraic multiplicity of λ1 is equal to the geometric multiplicity. Suppose the projection of u0 to the eigenspace of λ1 is not 0, then the iterating sequence is convergent uk → x1, µk → λ1, and the convergence rate is decided by |λ2| |λ1| .
- 14. Review Power method QR method Shifted power method I Shifted power method: Since the convergence rate is decided by |λ2| |λ1| , if |λ2| |λ1| / 1, the convergence will be slow. An idea to overcome this issue is to “shift” the eigenvalues, i.e. to apply power method to B = A − µI (µ is suitably chosen) such that |λ2(B)| |λ1(B)| = |λ2 − µ| |λ1 − µ| 1 the eigenvalue with largest modulus keeps invariant. I Shifted Power method 1. Set up initial u0, k = 1; 2. Perform a power step yk = (A − µI)uk−1; 3. Find the maximal component for the absolute value of ak = kykk∞; 4. Normalize uk = 1 ak yk and repeat. 5. λmax(A) = λmax(A − µI) + µ (under suitable shift). I Example 1: Shifted power method µ =?
- 15. Review Power method QR method Inverse power method I How to obtain the smallest eigenvalue of A? This is closely related to computing the ground state energy E0 for Schrödinger operator in quantum mechanics: − ~2 2µ ∇2 + U(r) ψ = E0ψ where ψ is the wave function. I Inverse power method: applying power method to A−1 . The inverse of the largest eigenvalue (modulus) of A−1 corresponds to the smallest eigenvalue of A. I Just change the step yk = Auk−1 in power method into Ayk = uk−1 I Compute the smallest eigenvalue of Example 2 (n=30).
- 16. Review Power method QR method Inverse power method I Sometimes inverse power method is cooperated with shifting to obtain the eigenvalue and eigenvector corresponding to some λ∗ if we already have an approximate λ̃ ≈ λ∗ , then the power step (A − λ̃I)yk = uk−1 Notice since λ̃ ≈ λ∗ , we have λmax(A − λ̃I) = 1 |λ̃ − λ∗| 1 The convergence will be very fast. I Compute the eigenvalue closest to 0.000 for Example 2 (n=30).
- 17. Review Power method QR method Rayleigh quotient accelerating I When do we need Rayleigh quotient accelerating? If A is symmetric and we already have an approximate eigenvector u0, we want to refine this eigenvector and corresponding eigenvalue λ. I Rayleigh quotient iteration: (Inverse power method + shift) 1. Choose initial u0, k = 1; 2. Compute Rayleigh quotient µk = RA(uk−1); 3. Solve equation for uk, (A − µkI)yk = uk−1; 4. Normalize uk = 1 kykk∞ yk and repeat. I Remark on Rayleigh quotient iteration and inverse power method.
- 18. Review Power method QR method Outline Review Power method QR method
- 19. Review Power method QR method QR method I Suppose A ∈ Rn×n , then QR method is to apply iterations as follows Am−1 = QmRm Am = RmQm where Qm is a orthogonal matrix, Rm is an upper triangular matrix. I Finally Rm will tend to λ1 ∗ · · · ∗ λ2 ... ∗ ... ∗ λn . We find all of the eigenvalues of A! I How to find matrix Q and R efficiently to perform QR factorization?
- 20. Review Power method QR method Simplest example I Vector x = 3 4 ! Try to eliminate the second component of x to 0. I Define y = Qx, Q = 0.6 −0.8 0.8 0.6 ! , y = 5 0 ! ,
- 21. Review Power method QR method Givens transformation I Suppose x = a b ! I Define rotation matrix G = c s −s c ! where c = a √ a2+b2 = cos θ, s = b √ a2+b2 = sin θ. It’s quite clear that G is a orthogonal matrix. I We have Gx = y = √ a2 + b2 0 ! I This rotation is called Givens transformation.
- 22. Review Power method QR method Givens transformation I Geometrical interpretation of Givens transformation x x’ y’ y x θ
- 23. Review Power method QR method General Givens transformation I Define Givens matrix G(i, k; θ) = 1 ... c · · · s . . . . . . −s · · · c ... 1 ← i-th row ← k-th row where c = cos θ, s = sin θ. I Geometrical interpretation: Rotation with θ angle in i − k plane.
- 24. Review Power method QR method Properties of Givens transformation I Suppose the vector x = (x1, . . . , xn) and we want to eliminate xk to 0 with xi. I Define c = xi p x2 i + x2 k , s = xk p x2 i + x2 k and y = G(i, k; θ)x, then we have yi = q x2 i + x2 k, yk = 0
- 25. Review Power method QR method Householder transformation I Definition. Suppose w ∈ Rn and kwk2 = 1, define H ∈ Rn×n as H = I − 2wwT . H is called a Householder transformation. I Properties of Householder transformation 1. Symmetric HT = H; 2. Orthogonal HT H = I; 3. Reflection (Go on to the next page! :-) )
- 26. Review Power method QR method Householder transformation I For any x ∈ Rn , Hx = x − 2(wT x)w which is the mirror image of x w.r.t. the plane perpendicular to w. I Geometrical interpretation ω ω
- 27. Review Power method QR method Application of Householder transformation I For arbitrary x ∈ Rn , there exists w such that Hx = αe1 where α = ±kxk2. Taking w = x − αe1 kx − αe1k2 is OK. I Proof: Define β = kx − αe1k2, then Hx = x − 2(wT x)w = x − 2 β2 (α2 − αeT 1 · x)(x − αe1) = x − 2 2α2 − 2αeT 1 · x (α2 − αeT 1 · x)(x − αe1) = x − (x − αe1) = αe1
- 28. Review Power method QR method Application of Householder transformation I If define x0 = (x2, . . . , xn)T , there exists H0 ∈ R(n−1)×(n−1) such that H0 x0 = αe0 1 Define H = 1 0 0 H0 ! Then we have the last n − 2 entries of Hx will be 0. i.e. Hx = (x1, q x2 2 + . . . + x2 n, 0, . . . , 0)
- 29. Review Power method QR method Upper Hessenberg form and QR method I Upper Hessenberg form Upper Hessenberg matrix A with entry aij = 0, j ≤ i − 2, i.e. with the following form a11 a12 · · · a1n a21 a22 · · · a2n ... ... . . . an−1,n ann
- 30. Review Power method QR method Why upper Hessenberg form I Why take upper Hessenberg form? It can be proved that if Am−1 is in upper Hessenberg form then Am−1 = QmRm, Am = RmQm. Am will be in upper Hessenberg form, too. I The computational effort for QR factorization of upper Hessenberg form will be small. I Example 3: A QR-factorization step for matrix 3 1 4 2 4 3 0 3 5
- 31. Review Power method QR method QR method for upper Hessenberg form I How to transform upper Hessenberg form into QR form? The approach is to apply Givens transformation to A column by column to eliminate the sub-diagonal entries. I Suppose A = d1 ∗ · · · ∗ b1 d2 · · · ∗ ... ... . . . bn−1 dn Apply Givens transformation G(1, 2; θ1), where cos θ1 = d1 √ d2 1+b2 1 , sin θ1 = b1 √ d2 1+b2 1 , then we have A = d1 ∗ · · · ∗ 0 d0 2 · · · ∗ ... ... . . . bn−1 dn
- 32. Review Power method QR method QR method for upper Hessenberg form I Now A = d1 ∗ · · · ∗ 0 d0 2 · · · ∗ ... ... . . . bn−1 dn Apply Givens transformation G(2, 3; θ2), where cos θ2 = d0 2 √ d0 2 2+b2 2 , sin θ2 = b2 √ d0 2 2+b2 2 . We would zero out the entry a32. I Applying this procedure successively, we obtain A = R = d1 ∗ · · · ∗ 0 d0 2 · · · ∗ ... ... . . . 0 d0 n
- 33. Review Power method QR method Transformation to upper Hessenberg form I How to transform a matrix into upper Hessenberg form? The approach is to apply Householder transformation to A column by column. A = a11 a12 · · · a1n a21 a22 · · · a2n · · · · · · · · · · · · an1 an2 · · · ann I Suitably choose Householder matrix H1 such that H1 · a11 a21 a31 . . . an1 = a0 11 a0 21 0 . . . 0
- 34. Review Power method QR method Transformation to upper Hessenberg form I Now we have A1 = H1AH1 = a0 11 a0 12 · · · a0 1n a0 21 a0 22 · · · a0 2n · · · · · · · · · · · · 0 a0 n2 · · · a0 nn I Suitably choose Householder matrix H2 such that H2 · a0 12 a0 22 a0 32 a0 42 . . . a0 n2 = a0 12 a0 22 a0 32 0 . . . 0
- 35. Review Power method QR method Transformation to upper Hessenberg form I Apply the Householder transformation A2 = H2A1H2, . . . successively, we will have the upper Hessenberg form B = a11 a12 · · · a1n a21 a22 · · · a2n ... ... . . . an−1,n ann I B has the same eigenvalues as A because of similarity transformation.
- 36. Review Power method QR method Transformation to upper Hessenberg form Compute all the eigenvalues of Example 2 (second order ODE, n=5) with QR method.
- 37. Review Power method QR method Homework assignment 4 1. Compute all the eigenvalues of Example 2 (second order ODE, n=20) with QR method.