Quadratic form and functional optimization
4 likes2,303 views
The document covers the optimization of multivariate quadratic functions, including techniques such as Taylor's expansion and Newton's method for solving quadratic approximations. It details concepts like Jacobians, Hessians, and symmetric matrices, highlighting how to achieve optimal solutions through gradient analysis and linear regression. Additionally, the document explores residual sums of squares and methods for analyzing residuals in linear regression models.
Quadratic form and functional optimization
- 1. Quadratic Form and Functional Optimization 9th June, 2011 Junpei Tsuji
- 2. Optimization of multivariate quadratic function 𝑥1 1 3 1 𝑥1 𝐽 𝑥1 , 𝑥2 = 1.2 + 0.2, 0.3 𝑥2 + 𝑥1 , 𝑥2 𝑥2 2 1 4 3 2 = 1.2 + 0.2𝑥1 + 0.3𝑥2 + 𝑥1 + 𝑥1 𝑥2 + 2𝑥2 2 2 𝑥1 , 𝑥2 , 𝐽 = 0.045, 0.064, 1.1881
- 3. Quadratic approximation 1 𝑓 𝒙 ≈ 𝑓 ̅ + 𝑱̅ ∙ 𝒙 − � + 𝒙 𝒙−� 𝒙 𝑇� 𝑯 𝒙−� 𝒙 By Taylor's expansion 2 constant linear form quadratic form 𝒙 ∶= 𝑥1 , 𝑥2 , ⋯ 𝑥 𝑝 𝑇 where 𝑓 ̅ ∶= 𝑓 � 𝒙 • 𝑱̅: = , ,⋯, 𝜕𝑓 𝜕𝑓 𝜕𝑓 • 𝜕𝑥1 𝜕𝑥2 𝜕𝑥 𝑝 � 𝒙=𝒙 • Jacobian (gradient) ⋯ 𝜕2 𝑓 𝜕2 𝑓 𝜕𝑥1 𝜕𝑥1 𝜕𝑥1 𝜕𝑥 𝑝 • � ∶= 𝑯 ⋮ ⋱ ⋮ ⋯ 𝜕2 𝑓 𝜕2 𝑓 Hessian (constant) 𝜕𝑥 𝑝 𝜕𝑥1 𝜕𝑥 𝑝 𝜕𝑥 𝑝 � 𝒙=𝒙
- 4. Completing the square 1 𝑓 𝒙 = 𝑓 ̅ + 𝑱̅ ∙ 𝒙 − � + 𝒙 𝒙−� 𝒙 𝑇� 𝑯 𝒙−� 𝒙 2 • Let � = 𝒙∗ where 𝑱 𝒙∗ 𝑇 = 𝟎 then 𝒙 1 𝑓 𝒙 = 𝑓∗ + 𝒙 − 𝒙∗ 𝑇 𝑯∗ 𝒙 − 𝒙∗ 2 constant quadratic form
- 5. Completing the square 1 𝑇 𝑓 𝒙 = 𝑐 + 𝒃 𝒙 + 𝒙 𝑨𝑨 𝑇 2 1 𝑓 𝒙 = 𝑑+ 𝒙 − 𝒙0 𝑇 𝑨 𝒙 − 𝒙0 2 1 1 1 = 𝑑 + 𝒙0 𝑇 𝑨𝒙0 − 𝒙0 𝑇 𝑨 + 𝑨 𝑇 𝒙 + 𝒙 𝑇 𝑨𝑨 2 2 2 𝒃𝑇 =− 𝒙0 𝑇 𝑨 + 𝑨 𝑇 1 2 𝒙0 𝑇 = −2𝒃 𝑇 𝑨 + 𝑨 𝑇 −1 • 𝒙0 = −2 𝑨 + 𝑨 𝑇 −1 𝒃 𝑐= 𝑑+ 𝒙0 𝑇 𝑨𝒙0 1 1 𝑇 2 𝑑= 𝑐− 𝒙0 𝑨𝒙0 = 𝑐 − 2𝒃 𝑇 𝑨 + 𝑨 𝑇 𝑨 𝑨+ 𝑨𝑇 𝒃 • −1 −1 2 𝑓 𝒙 = 𝑐 − 2𝒃 𝑇 𝑨 + 𝑨 𝑇 −1 𝑨 𝑨+ 𝑨𝑇 −1 𝒃 Therefore, 1 + 𝒙 + 2 𝑨 + 𝑨 𝑇 −1 𝒃 𝑇 𝑨 𝒙 + 2 𝑨 + 𝑨 𝑇 −1 𝒃 2 If 𝑨 was symmetric matrix, 1 𝑇 −1 1 𝑓 𝒙 = 𝑐− 𝒃 𝑨 𝒃+ 𝒙 + 𝑨−1 𝒃 𝑇 𝑨 𝒙 + 𝑨−1 𝒃 • 2 2
- 6. Quadratic form 𝑓 𝒙𝒙 = 𝒙𝒙 𝑇 𝑺𝑺𝑺 • 𝑺 is symmetric matrix. where
- 7. Symmetric matrix • Symmetric matrix 𝑺 is defined as a matrix that satisfies the 𝑺𝑇 = 𝑺 following formula: • Symmetric matrix 𝑺 has real eigenvalues 𝜆 𝑖 and eigenvectors 𝒖 𝑖 that consist of normal orthogonal base. 𝑺𝒖 𝑖 = 𝜆 𝑖 𝒖 𝑖 where 𝜆1 ≥ 𝜆2 ≥ ⋯ ≥ 𝜆 𝑝 𝒖 𝑖 , 𝒖 𝑗 = 𝛿 𝑖𝑖 𝛿 𝑖𝑖 is Kronecker's delta
- 8. Diagonalization of symmetric matrix • We define an orthogonal matrix 𝑼 as follows: 𝑼 = 𝒖1 , 𝒖2 , ⋯ , 𝒖 𝑝 • Then, 𝑼 satisfies the following formulas: 𝑼𝑇 𝑼= 𝑰 ∴ 𝑼−1 = 𝑼 𝑇 • where 𝑰 is an identity matrix. 𝑺𝑺 = 𝑺 𝒖1 , 𝒖2 , ⋯ , 𝒖 𝑝 = 𝑺𝒖1 , 𝑺𝒖2 , ⋯ , 𝑺𝒖 𝑝 𝜆1 = 𝜆1 𝒖1 , 𝜆2 𝒖2 , ⋯ , 𝜆 𝑝 𝒖 𝑝 = 𝒖1 , ⋯ , 𝒖 𝑝 ⋱ 𝜆𝑝 = 𝑼 𝐝𝐝𝐝𝐝 𝜆1 , 𝜆2 , ⋯ , 𝜆 𝑝 ∴ 𝑺 = 𝑼 𝐝𝐝𝐝𝐝 𝜆1 , 𝜆2 , ⋯ , 𝜆 𝑝 𝑼𝑇
- 9. Transformation to principal axis 𝑓 𝒙′ = 𝒙′ 𝑇 𝑺𝑺′ • Then, we assume 𝒙𝒙 = 𝑼 𝑇 𝒛, where 𝒛 = 𝑧1 , 𝑧1 , ⋯ , 𝑧 𝑝 . 𝑓 𝑼 𝑇 𝒛 = 𝑼 𝑇 𝒛 𝑇 𝑺 𝑼 𝑇 𝒛 = 𝒛 𝑇 𝑼𝑺𝑼 𝑇 𝒛 = 𝒛 𝑇 𝐝𝐝𝐝𝐝 𝜆1 , 𝜆2 , ⋯ , 𝜆 𝑝 𝒛 𝑝 ∴ 𝑓 𝒛 = � 𝜆 𝑖 𝑧 𝑖2 𝑖=1
- 10. Contour surface • If we assume 𝑓 𝒛 equals constant 𝑐, 𝑝 𝑓 𝒛 = � 𝜆 𝑖 𝑧 𝑖2 = 𝑐 𝑖=1 • When 𝑝 = 2, – a locus of 𝒛 illustrates an ellipse if 𝜆1 𝜆2 > 0. – a locus of 𝒛 illustrates a hyperbola if 𝜆1 𝜆2 < 0.
- 11. Contour surface 𝑧2 2 𝑓 𝒛 = � 𝜆 𝑖 𝑧 𝑖 2 = 𝑐𝑐𝑐𝑐𝑐. 𝑖=1 𝜆1 𝜆2 > 0 𝑧1 maximal or minimal point 𝑓 𝑥1 , 𝑥2 = −𝑥1 2 − 2𝑥2 2 + 20.0
- 12. Transformation to principal axis 𝑥𝑥2 𝑓 𝒙𝒙 = 𝑐𝑐𝑐𝑐𝑐. 𝑥𝑥1 𝒙𝒙 = 𝑼 𝑇 𝒛 ∴ 𝒛 = 𝑼𝒙′ Transformation to principal axis
- 13. Parallel translation 𝑥𝑥2 𝑥2 � 𝒙 𝑥𝑥1 𝑓 𝒙 = 𝑐𝑐𝑐𝑐𝑐. 𝑥1 𝒙𝒙 = 𝒙 − �𝒙
- 14. 1 Contour surface of quadratic function 𝑓 𝒙 = 𝑓 + ∗ 𝒙 − 𝒙∗ 𝑇 𝑯∗ 𝒙 − 𝒙∗ 2 𝑥2 � 𝒙 𝑓 𝒙 = 𝑐𝑐𝑐𝑐𝑐. 𝑥1
- 15. Contour surface 𝑧2 2 𝑓 𝒛 = � 𝜆 𝑖 𝑧 𝑖 2 = 𝑐𝑐𝑐𝑐𝑐. 𝑖=1 𝜆1 𝜆2 < 0 𝑧1 saddle point 𝑓 𝑥1 , 𝑥2 = 𝑥1 2 − 𝑥2 2
- 16. Stationary points 𝑓 𝑥1 , 𝑥2 = 𝑥1 3 + 𝑥2 3 + 3𝑥1 𝑥2 + 2 maximal point saddle point
- 17. Stationary points 1 3 𝑓 𝑥1 , 𝑥2 = exp − 𝑥1 + 𝑥1 − 𝑥2 2 3 saddle point maximal point
- 19. Newton-Raphson method 𝑓𝑓 𝒙 = 𝟎 where 𝑓 𝒙 is 𝑁-th polynomial by • Newton’s method is an approximate solver of using a quadratic approximation. 𝑓 𝒙 quadratic approximation of 𝑓 𝒙 in 𝒙 1 𝑓 𝒙 + Δ𝒙 ≈ 𝑓 𝒙 + 𝑱 𝒙 ∙ Δ𝒙 + Δ𝒙 𝑇 𝑯 𝒙 Δ𝒙 2 𝜕𝑓 𝒙 + Δ𝒙 𝑓𝑓 𝒙∗ = 𝟎 𝜕 Δ𝒙 = 𝑱 𝒙 𝑇 + 𝑯 𝒙 Δ𝒙 𝒙∗ 𝒙 + 𝚫𝒙 𝒙 𝒙
- 20. Algorithm of Newton’s method Procedure Newton (𝑱 𝒙 , 𝑯 𝒙 ) 1. Initialize 𝒙. 2. Calculate 𝑱 𝒙 and 𝑯 𝒙 . equation and giving ∆𝒙 : 𝑱 𝒙 𝑇 + 𝑯 𝒙 ∆𝒙 = 𝟎 3. Solve the following simultaneous 4. Update 𝒙 as follows: 𝒙 ← 𝒙 + ∆𝒙 5. If ∆𝒙 < 𝛿 then return 𝒙 else go back to 2.
- 21. Linear regression 𝑝 𝑦 𝑦 = 𝑓 𝒙 = 𝛽0 + � 𝛽 𝑗 𝑥 𝑗 𝑁 samples 𝒙 𝑖, 𝑦 𝑖 𝑗=1 𝒙 𝑝-th dimensional space We would like to find 𝜷∗ that minimizes the residual sum of square (RSS).
- 22. Linear regression min RSS 𝜷 𝜷 2 𝑁 𝑁 𝑝 • where RSS 𝜷 = � 𝑦 𝑖 − 𝑓 𝒙 𝑖 2 = � 𝑦𝑖 − 𝛽0 + � 𝛽 𝑗 𝑥 𝑖𝑖 𝑖=1 𝑖=1 𝑗=1 • Given 𝑿, 𝒚, 𝜷 as follows: 𝑥11 ⋯ 𝑥1𝑝 1 𝑦1 𝛽1 𝑿= ⋮ ⋱ ⋮ ⋮ , 𝒚= ⋮ , 𝜷= ⋮ 𝑥 𝑁𝑁 ⋯ 𝑥 𝑁𝑁 1 𝑦𝑁 𝛽𝑝 ∴ RSS 𝜷 = 𝒚 − 𝑿𝜷 2
- 23. Linear regression RSS 𝜷 = 𝐽 𝜷 = 𝒚 − 𝑿𝜷 2 = 𝒚 − 𝑿𝜷 𝑇 𝒚 − 𝑿𝜷 = 𝒚 𝑇 𝒚 − 𝜷 𝑇 𝑿 𝑇 𝒚 − 𝒚 𝑇 𝑿𝜷 + 𝜷 𝑇 𝑿 𝑇 𝑿𝜷 𝒂𝑇 𝜷 = 𝒂 𝜕 𝜕𝜷 𝜷𝑇 𝒂 = 𝒂 • 𝜕 𝜕𝜷 𝜷 𝑇 𝑨𝜷 = 𝑨 • 𝜕 𝜕𝜷 𝜕𝐽 𝐽′ 𝜷 = = −2𝑿 𝑇 𝒚 + 2𝑿 𝑇 𝑿𝜷 • 𝜕𝜷
- 24. Linear regression Given 𝜷∗ that satisfies 𝐽′ 𝜷∗ = 𝟎, 𝑿 𝑇 𝒚 = 𝑿 𝑇 𝑿𝜷∗ 𝒚 𝑇 𝑿 = 𝜷∗ 𝑇 𝑿 𝑇 𝑿 ∴ 𝜷∗ = 𝑿𝑇 𝑿 −1 𝑿𝑇 𝒚 ∴ 𝐽 𝜷 = 𝒚 𝒚 − 𝜷 𝑿 𝑿𝜷 − 𝜷 𝑿 𝑇 𝑿𝜷 + 𝜷 𝑇 𝑿 𝑇 𝑿𝜷 𝑇 𝑇 𝑇 ∗ ∗𝑇 ∴ 𝐽 𝜷 = 𝒚 𝑇 𝒚 − 𝜷∗ 𝑇 𝑿 𝑇 𝑿𝜷∗ + 𝜷∗ 𝑇 𝑿 𝑇 𝑿𝜷∗ − 𝜷 𝑇 𝑿 𝑇 𝑿𝜷∗ − 𝜷∗ 𝑇 𝑿 𝑇 𝑿𝜷 + 𝜷 𝑇 𝑿 𝑇 𝑿𝜷 ∴ 𝐽 𝜷 = 𝒚 𝑇 𝒚 − 𝜷∗ 𝑇 𝑿 𝑇 𝑿𝜷∗ + 𝜷 − 𝜷∗ 𝑇 𝑿 𝑇 𝑿 𝜷 − 𝜷∗ completing the square
- 25. Linear regression 𝐽 𝜷 = 𝒚 𝑇 𝒚 − 𝜷∗ 𝑇 𝑿 𝑇 𝑿𝜷∗ + 𝜷 − 𝜷∗ 𝑇 𝑿 𝑇 𝑿 𝜷 − 𝜷∗ = 𝒚 − 𝑿𝜷∗ 2 + 𝜷 − 𝜷∗ 𝑇 𝑿 𝑇 𝑿 𝜷 − 𝜷∗ 1 = 𝐽 𝜷 + ∗ 𝜷 − 𝜷 ∗ 𝑇 𝑯 𝜷 − 𝜷∗ 2 Residual sum of squares (RSS) quadratic form 𝛽2 𝐽 𝜷 = 𝑐𝑐𝑐𝑐𝑐. by Linear Regression 𝜷∗ 𝜷∗ = 𝑿 𝑇 𝑿 −1 𝑿 𝑇 𝒚 𝑯 = 2𝑿 𝑇 𝑿 𝛽1
- 26. Hessian • 𝑯≔ = 2𝑿 𝑇 𝑿 𝜕2 𝐽 𝜕𝛽 𝑖 𝜕𝛽 𝑗 • 𝑯 has the following two features: 𝑯𝑇 = 𝑯 ∀ 𝒙 ≠ 𝟎, 𝒙 𝑇 𝑯𝑯 > 0 – symmetric matrix: – positive-definite matrix: Therefore, 𝜷∗ = 𝑿𝑇 𝑿 −1 𝑿 𝑇 𝒚 is the minimum of 𝐽 𝜷 .
- 27. Analysis of residuals 𝒚∗ = 𝑿𝜷∗ • Then, we substitute 𝜷∗ = 𝑿 𝑇 𝑿 −1 𝑿 𝑇 𝒚 in the above, 𝒚∗ = 𝑿𝜷∗ = 𝑿 𝑿 𝑇 𝑿 −1 𝑿𝑇 𝒚 ∴ 𝒚∗ = ℋ𝒚 (Hat matrix) • the vector of residuals 𝒓 can be expressed by follows: 𝒓 = 𝒚 − 𝒚∗ = 𝒚 − ℋ𝒚 = 𝑰 − ℋ 𝒚 𝑉𝑉𝑉 𝒓 = 𝑉𝑉𝑉 𝑰 − ℋ 𝒚 = 𝑰 − ℋ 𝑉𝑉𝑉 𝒚 𝑰 − ℋ 𝑇
- 28. Analysis of residuals ℋ = 𝑿 𝑿 𝑇 𝑿 −1 𝑿 𝑇 The hat matrix ℋ is a projection matrix, which 1. Projection: ℋ 2 = ℋ satisfies the following equations: ℋ 2 = ℋ ∙ ℋ = 𝑿 𝑿 𝑇 𝑿 −1 𝑿 𝑇 ∙ 𝑿 𝑿 𝑇 𝑿 −1 𝑿𝑇 = 𝑿 𝑿 𝑇 𝑿 −1 𝑿 𝑇 𝑿 𝑿 𝑇 𝑿 −1 𝑿 𝑇 = 𝑿 𝑿 𝑇 𝑿 −1 𝑿 𝑇 = ℋ 2. Orthogonal: ℋ 𝑇 = ℋ
- 29. Analysis of residuals 𝑥11 ⋯ 𝑥1𝑝 1 𝛽1 ∗ 𝑦1 ∗ ⋮ ⋮ = ⋮ ⋱ ⋮ ⋮ 𝛽𝑝 ∗ 𝑦 𝑁∗ 𝑥 𝑁1 ⋯ 𝑥 𝑁𝑁 1 𝛽0 ∗ 𝑥11 𝑥1𝑝 1 = 𝛽1 ∗ ⋮ + ⋯ + 𝛽 𝑝∗ ⋮ + 𝛽0 ⋮ ∗ 𝑥 𝑁1 𝑥 𝑁𝑁 1 𝒙1 𝒙𝑝 𝒙 𝑝+1 = 𝟏 linear combination in 𝑝 + 1 -th vector space
- 30. Analysis of residuals 𝒚 𝒚∗ = ℋ𝒚 (Projection) 𝒙𝑝 𝒚∗ 𝒙𝑗 𝑝 + 1 -th dimensional super surface 𝑁-th dimensional space
- 31. Analysis of residuals 𝒚 = 𝑿𝜷 • 𝜷 = 𝑿−1 𝒚, where 𝑿−1 is M-P generalized inverse. 𝑝= 𝑁 𝑝> 𝑁 1. Unique solution: 𝑝< 𝑁 2. Many solutions: 𝑿−1 3. No solution: 𝑿 −1 =� 𝑿𝑿 𝑿𝑿𝑿 −1 𝜷 = 𝑿−1 𝒚 is min in 𝜷 𝑿𝑿𝑿 −1 𝑿𝑿 𝒚 − 𝑿𝜷 2 is min •