![9/ 16
Energy and response properties
E.g. a force component is the first-order energy response wrt. x:
Fx = −
∂
∂x
U
Electric dipole moment:
⃗µ = −
∂
∂ ⃗E
U
Energy in kernel-based regression models:
U = K α
Any response property:
ω ≡ O [U] = O [K] α
Analytical solution; minimize the following Lagrangian:
J(α) = ∥ωtraining
− O[K]α∥2
2
Normal-equation solution:
α =
[ ∑
γ
Oγ[K]T
Oγ[K]
]−1[ ∑
γ
(
ωtraining
γ
)T
Oγ[K]
]](https://image.slidesharecdn.com/main-181003092651/85/Operators-in-Machine-Learning-Response-Properties-in-Chemical-Space-16-320.jpg)
![9/ 16
Energy and response properties
E.g. a force component is the first-order energy response wrt. x:
Fx = −
∂
∂x
U
Electric dipole moment:
⃗µ = −
∂
∂ ⃗E
U
Energy in kernel-based regression models:
U = K α
Any response property:
ω ≡ O [U] = O [K] α
Analytical solution; minimize the following Lagrangian:
J(α) = ∥ωtraining
− O[K]α∥2
2
Normal-equation solution:
α =
[ ∑
γ
Oγ[K]T
Oγ[K]
]−1[ ∑
γ
(
ωtraining
γ
)T
Oγ[K]
]](https://image.slidesharecdn.com/main-181003092651/85/Operators-in-Machine-Learning-Response-Properties-in-Chemical-Space-17-320.jpg)
![9/ 16
Energy and response properties
E.g. a force component is the first-order energy response wrt. x:
Fx = −
∂
∂x
U
Electric dipole moment:
⃗µ = −
∂
∂ ⃗E
U
Energy in kernel-based regression models:
U = K α
Any response property:
ω ≡ O [U] = O [K] α
Analytical solution; minimize the following Lagrangian:
J(α) = ∥ωtraining
− O[K]α∥2
2
Normal-equation solution:
α =
[ ∑
γ
Oγ[K]T
Oγ[K]
]−1[ ∑
γ
(
ωtraining
γ
)T
Oγ[K]
]](https://image.slidesharecdn.com/main-181003092651/85/Operators-in-Machine-Learning-Response-Properties-in-Chemical-Space-18-320.jpg)
![9/ 16
Energy and response properties
E.g. a force component is the first-order energy response wrt. x:
Fx = −
∂
∂x
U
Electric dipole moment:
⃗µ = −
∂
∂ ⃗E
U
Energy in kernel-based regression models:
U = K α
Any response property:
ω ≡ O [U] = O [K] α
Analytical solution; minimize the following Lagrangian:
J(α) = ∥ωtraining
− O[K]α∥2
2
Normal-equation solution:
α =
[ ∑
γ
Oγ[K]T
Oγ[K]
]−1[ ∑
γ
(
ωtraining
γ
)T
Oγ[K]
]](https://image.slidesharecdn.com/main-181003092651/85/Operators-in-Machine-Learning-Response-Properties-in-Chemical-Space-19-320.jpg)
![9/ 16
Energy and response properties
E.g. a force component is the first-order energy response wrt. x:
Fx = −
∂
∂x
U
Electric dipole moment:
⃗µ = −
∂
∂ ⃗E
U
Energy in kernel-based regression models:
U = K α
Any response property:
ω ≡ O [U] = O [K] α
Analytical solution; minimize the following Lagrangian:
J(α) = ∥ωtraining
− O[K]α∥2
2
Normal-equation solution:
α =
[ ∑
γ
Oγ[K]T
Oγ[K]
]−1[ ∑
γ
(
ωtraining
γ
)T
Oγ[K]
]](https://image.slidesharecdn.com/main-181003092651/85/Operators-in-Machine-Learning-Response-Properties-in-Chemical-Space-20-320.jpg)
![9/ 16
Energy and response properties
E.g. a force component is the first-order energy response wrt. x:
Fx = −
∂
∂x
U
Electric dipole moment:
⃗µ = −
∂
∂ ⃗E
U
Energy in kernel-based regression models:
U = K α
Any response property:
ω ≡ O [U] = O [K] α
Analytical solution; minimize the following Lagrangian:
J(α) = ∥ωtraining
− O[K]α∥2
2
Normal-equation solution:
α =
[ ∑
γ
Oγ[K]T
Oγ[K]
]−1[ ∑
γ
(
ωtraining
γ
)T
Oγ[K]
]](https://image.slidesharecdn.com/main-181003092651/85/Operators-in-Machine-Learning-Response-Properties-in-Chemical-Space-21-320.jpg)
![10/ 16
Conventional kernel ridge regression:
E = K αE
ω = K αω
Same kernel matrix
Different regression coefficients
New model for every property
Response operator kernel-based regression:
E = K α
ω = O [K] α
Same kernel matrix (but different operator)
Same regression coefficients
Same model for multiple response properties
AS Christensen, FA Faber, OA von Lilienfeld (2018) “Operators in Machine Learning: Response Properties in Chemical
Space” arXiv:1807.08811](https://image.slidesharecdn.com/main-181003092651/85/Operators-in-Machine-Learning-Response-Properties-in-Chemical-Space-22-320.jpg)
![10/ 16
Conventional kernel ridge regression:
E = K αE
ω = K αω
Same kernel matrix
Different regression coefficients
New model for every property
Response operator kernel-based regression:
E = K α
ω = O [K] α
Same kernel matrix (but different operator)
Same regression coefficients
Same model for multiple response properties
AS Christensen, FA Faber, OA von Lilienfeld (2018) “Operators in Machine Learning: Response Properties in Chemical
Space” arXiv:1807.08811](https://image.slidesharecdn.com/main-181003092651/85/Operators-in-Machine-Learning-Response-Properties-in-Chemical-Space-23-320.jpg)
![11/ 16
Electric field first-order response (dipole moment)
Representation:
AM (I) = {A1(I), A2(I), A3(I), . . . , AM (I)}
Place fictitious partial charges
on atom centers.
Weight terms by resulting
dipole.
Weight for two-body terms:
ξ∗IJ
2 = ξIJ
2 − ϵ(⃗µIJ · ⃗E)
Weight for three-body terms:
ξ∗IJK
3 = ξIJK
3 − ϵ(⃗µIJK · ⃗E)
180 120 60 0 60 120 180
Angle [degrees]
0.5
0.0
0.5
1.0
1.5
2.0
E[mHa]
ML Test (w/ dipole)
ML Test (w/o dipole)
MP2
ML Training
H-F molecule in electric field of 0.001 a.u.
AS Christensen et al. (2018) arXiv:1807.08811](https://image.slidesharecdn.com/main-181003092651/85/Operators-in-Machine-Learning-Response-Properties-in-Chemical-Space-26-320.jpg)
![12/ 16
Learning dipole moments
The QM9 dataset (again)
Red: Learn the dipole norm
Conventional KRR
Blue: Learn the energy
derivative
Using response operators
100 1000 2500 500010000
N
1.5
1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
MAE||[D]
20x less data
Norm
Response
Learning curve for QM9
AS Christensen et al. (2018) arXiv:1807.08811](https://image.slidesharecdn.com/main-181003092651/85/Operators-in-Machine-Learning-Response-Properties-in-Chemical-Space-27-320.jpg)
![13/ 16
Learning forces
0.05
0.1
0.2
0.4
0.6
MAEE
[kcal/mol]
GDML
FCHL*
SchNet
200 500 1000
0.2
0.3
0.4
0.6
0.8
1.0
2.0
MAEFx
[kcal/mol/Å]
500 1000 500 1000 500 1000 500 1000 500 1000 500 1000 500 1000
N
First derivative wrt. nuclear coordinates
“MD17” benchmark set
QM-MD trajectory, DFT forces and energies
S Chmiela et al. (2017) Sci. Advances, e1603015
AS Christensen et al. (2018) arXiv:1807.08811](https://image.slidesharecdn.com/main-181003092651/85/Operators-in-Machine-Learning-Response-Properties-in-Chemical-Space-28-320.jpg)
![14/ 16
Vibrational normal modes
Normal modes for C6N3H7
Training set: non-equilibirum
geometries of fragments
N = number of samples in the
training set
“Rank” = total number of
energies and force
components in training set
0.0
0.5
Mode:1
N = 1; Rank: 214
0.0
0.5
Mode:5
0.1 0.0 0.1
0.0
0.5
Mode:15
N = 4; Rank: 856
0.0 0.1
N = 32; Rank: 6848
0.0 0.1
E[kcal/mol]
RMSD displacement along normal mode [Å]](https://image.slidesharecdn.com/main-181003092651/85/Operators-in-Machine-Learning-Response-Properties-in-Chemical-Space-29-320.jpg)
![15/ 16
Mixed derivatives and IR spectra
IR intensity:
Mixed derivative wrt. nuclear
coordinates and external electric field.
Vibrational analysis using interface to
Gaussian09
Model trained on distorted geometries of
dichloromethane. Training includes:
Energy
Dipole moments
Forces
Many other operators to try out!
0 1000 2000 3000 4000 5000
[cm
1
]
MP2
N = 100
FCHL* MAE: 2.4 cm 1
N = 50
FCHL* MAE: 5.7 cm 1
N = 25
FCHL* MAE: 25.6 cm 1
N = 10
FCHL* MAE: 126.1 cm 1
N = 5
FCHL* MAE: 340.4 cm 1
AS Christensen et al. (2018) arXiv:1807.08811](https://image.slidesharecdn.com/main-181003092651/85/Operators-in-Machine-Learning-Response-Properties-in-Chemical-Space-30-320.jpg)
![15/ 16
Mixed derivatives and IR spectra
IR intensity:
Mixed derivative wrt. nuclear
coordinates and external electric field.
Vibrational analysis using interface to
Gaussian09
Model trained on distorted geometries of
dichloromethane. Training includes:
Energy
Dipole moments
Forces
Many other operators to try out!
0 1000 2000 3000 4000 5000
[cm
1
]
MP2
N = 100
FCHL* MAE: 2.4 cm 1
N = 50
FCHL* MAE: 5.7 cm 1
N = 25
FCHL* MAE: 25.6 cm 1
N = 10
FCHL* MAE: 126.1 cm 1
N = 5
FCHL* MAE: 340.4 cm 1
AS Christensen et al. (2018) arXiv:1807.08811](https://image.slidesharecdn.com/main-181003092651/85/Operators-in-Machine-Learning-Response-Properties-in-Chemical-Space-31-320.jpg)
Operators in Machine Learning: Response Properties in Chemical Space
- 1. Operators in Machine Learning: Response Properties in Chemical Space Anders S. Christensen <anders.christensen@unibas.ch> University of Basel October 3, 2018 0/ 16
- 2. 1/ 16 Introduction Machine learning: Training set of structures and properties Train model, identify similarities New predictions: Interpolation between training instances Figure by Felix Faber Better predictions? More training data Better models
- 3. 1/ 16 Introduction Machine learning: Training set of structures and properties Train model, identify similarities New predictions: Interpolation between training instances Figure by Felix Faber Better predictions? More training data Better models
- 4. 2/ 16 Learning curves Learning curves follow a power law: Error ≈ a Nb log (Error) ≈ log (a) − b log (N) Vapnik, The Nature of Statistical Learning Theory, Springer (1995)
- 5. 3/ 16 Kernel ridge regression and representations Prediction of a property, y for a molecule ˜x: y (˜x) = ∑ i k (˜x, xi) αi Kernel: The similarity between two representations. Non-linear transformation that makes the problem linear E.g. Gaussian kernel: k (xi, xj) = exp ( − ∥xi − xj∥2 2 2σ2 ) Simply a number between 1 and 0. Analytical solution: α = K−1 y Better representations ⇒ better kernel-based ML models!
- 6. 3/ 16 Kernel ridge regression and representations Prediction of a property, y for a molecule ˜x: y (˜x) = ∑ i k (˜x, xi) αi Kernel: The similarity between two representations. Non-linear transformation that makes the problem linear E.g. Gaussian kernel: k (xi, xj) = exp ( − ∥xi − xj∥2 2 2σ2 ) Simply a number between 1 and 0. Analytical solution: α = K−1 y Better representations ⇒ better kernel-based ML models!
- 7. 3/ 16 Kernel ridge regression and representations Prediction of a property, y for a molecule ˜x: y (˜x) = ∑ i k (˜x, xi) αi Kernel: The similarity between two representations. Non-linear transformation that makes the problem linear E.g. Gaussian kernel: k (xi, xj) = exp ( − ∥xi − xj∥2 2 2σ2 ) Simply a number between 1 and 0. Analytical solution: α = K−1 y Better representations ⇒ better kernel-based ML models!
- 8. 3/ 16 Kernel ridge regression and representations Prediction of a property, y for a molecule ˜x: y (˜x) = ∑ i k (˜x, xi) αi Kernel: The similarity between two representations. Non-linear transformation that makes the problem linear E.g. Gaussian kernel: k (xi, xj) = exp ( − ∥xi − xj∥2 2 2σ2 ) Simply a number between 1 and 0. Analytical solution: α = K−1 y Better representations ⇒ better kernel-based ML models!
- 9. 4/ 16 Kernel principal component analysis Kernel matrix for 1000 small organic molecules, elements: HCNO
- 10. 4/ 16 Kernel principal component analysis Kernel matrix for 1000 small organic molecules, elements: HCNO
- 11. 5/ 16 Our latest representation (FCHL) Atom I represented by a many-body expansion distribution of its environment, AM (I) = {A1(I), A2(I), A3(I), . . . , AM (I)} L2 distance: ∥AM (I) − AM (J)∥2 2 ≡ ∑M m=0 ∫ (Am(I) − Am(J))2 d⃗Zd⃗r
- 12. 6/ 16 Kernel principal component analysis Kernel matrix for 1000 small organic molecules, elements: HCNO
- 13. 7/ 16 The “QM9” benchmark set 134K small organic molecules DFT atomization energy
- 14. 8/ 16 Learning properties Figure: FA Faber, et al. (2017) JCTC, collaboration with Google Brain
- 15. 8/ 16 Learning properties Figure: FA Faber, et al. (2017) JCTC, collaboration with Google Brain
- 16. 9/ 16 Energy and response properties E.g. a force component is the first-order energy response wrt. x: Fx = − ∂ ∂x U Electric dipole moment: ⃗µ = − ∂ ∂ ⃗E U Energy in kernel-based regression models: U = K α Any response property: ω ≡ O [U] = O [K] α Analytical solution; minimize the following Lagrangian: J(α) = ∥ωtraining − O[K]α∥2 2 Normal-equation solution: α = [ ∑ γ Oγ[K]T Oγ[K] ]−1[ ∑ γ ( ωtraining γ )T Oγ[K] ]
- 17. 9/ 16 Energy and response properties E.g. a force component is the first-order energy response wrt. x: Fx = − ∂ ∂x U Electric dipole moment: ⃗µ = − ∂ ∂ ⃗E U Energy in kernel-based regression models: U = K α Any response property: ω ≡ O [U] = O [K] α Analytical solution; minimize the following Lagrangian: J(α) = ∥ωtraining − O[K]α∥2 2 Normal-equation solution: α = [ ∑ γ Oγ[K]T Oγ[K] ]−1[ ∑ γ ( ωtraining γ )T Oγ[K] ]
- 18. 9/ 16 Energy and response properties E.g. a force component is the first-order energy response wrt. x: Fx = − ∂ ∂x U Electric dipole moment: ⃗µ = − ∂ ∂ ⃗E U Energy in kernel-based regression models: U = K α Any response property: ω ≡ O [U] = O [K] α Analytical solution; minimize the following Lagrangian: J(α) = ∥ωtraining − O[K]α∥2 2 Normal-equation solution: α = [ ∑ γ Oγ[K]T Oγ[K] ]−1[ ∑ γ ( ωtraining γ )T Oγ[K] ]
- 19. 9/ 16 Energy and response properties E.g. a force component is the first-order energy response wrt. x: Fx = − ∂ ∂x U Electric dipole moment: ⃗µ = − ∂ ∂ ⃗E U Energy in kernel-based regression models: U = K α Any response property: ω ≡ O [U] = O [K] α Analytical solution; minimize the following Lagrangian: J(α) = ∥ωtraining − O[K]α∥2 2 Normal-equation solution: α = [ ∑ γ Oγ[K]T Oγ[K] ]−1[ ∑ γ ( ωtraining γ )T Oγ[K] ]
- 20. 9/ 16 Energy and response properties E.g. a force component is the first-order energy response wrt. x: Fx = − ∂ ∂x U Electric dipole moment: ⃗µ = − ∂ ∂ ⃗E U Energy in kernel-based regression models: U = K α Any response property: ω ≡ O [U] = O [K] α Analytical solution; minimize the following Lagrangian: J(α) = ∥ωtraining − O[K]α∥2 2 Normal-equation solution: α = [ ∑ γ Oγ[K]T Oγ[K] ]−1[ ∑ γ ( ωtraining γ )T Oγ[K] ]
- 21. 9/ 16 Energy and response properties E.g. a force component is the first-order energy response wrt. x: Fx = − ∂ ∂x U Electric dipole moment: ⃗µ = − ∂ ∂ ⃗E U Energy in kernel-based regression models: U = K α Any response property: ω ≡ O [U] = O [K] α Analytical solution; minimize the following Lagrangian: J(α) = ∥ωtraining − O[K]α∥2 2 Normal-equation solution: α = [ ∑ γ Oγ[K]T Oγ[K] ]−1[ ∑ γ ( ωtraining γ )T Oγ[K] ]
- 22. 10/ 16 Conventional kernel ridge regression: E = K αE ω = K αω Same kernel matrix Different regression coefficients New model for every property Response operator kernel-based regression: E = K α ω = O [K] α Same kernel matrix (but different operator) Same regression coefficients Same model for multiple response properties AS Christensen, FA Faber, OA von Lilienfeld (2018) “Operators in Machine Learning: Response Properties in Chemical Space” arXiv:1807.08811
- 23. 10/ 16 Conventional kernel ridge regression: E = K αE ω = K αω Same kernel matrix Different regression coefficients New model for every property Response operator kernel-based regression: E = K α ω = O [K] α Same kernel matrix (but different operator) Same regression coefficients Same model for multiple response properties AS Christensen, FA Faber, OA von Lilienfeld (2018) “Operators in Machine Learning: Response Properties in Chemical Space” arXiv:1807.08811
- 24. 11/ 16 Electric field first-order response (dipole moment) Representation: AM (I) = {A1(I), A2(I), A3(I), . . . , AM (I)} Place fictitious partial charges on atom centers. Weight terms by resulting dipole. Weight for two-body terms: ξ∗IJ 2 = ξIJ 2 − ϵ(⃗µIJ · ⃗E) Weight for three-body terms: ξ∗IJK 3 = ξIJK 3 − ϵ(⃗µIJK · ⃗E)
- 25. 11/ 16 Electric field first-order response (dipole moment) Representation: AM (I) = {A1(I), A2(I), A3(I), . . . , AM (I)} Place fictitious partial charges on atom centers. Weight terms by resulting dipole. Weight for two-body terms: ξ∗IJ 2 = ξIJ 2 − ϵ(⃗µIJ · ⃗E) Weight for three-body terms: ξ∗IJK 3 = ξIJK 3 − ϵ(⃗µIJK · ⃗E)
- 26. 11/ 16 Electric field first-order response (dipole moment) Representation: AM (I) = {A1(I), A2(I), A3(I), . . . , AM (I)} Place fictitious partial charges on atom centers. Weight terms by resulting dipole. Weight for two-body terms: ξ∗IJ 2 = ξIJ 2 − ϵ(⃗µIJ · ⃗E) Weight for three-body terms: ξ∗IJK 3 = ξIJK 3 − ϵ(⃗µIJK · ⃗E) 180 120 60 0 60 120 180 Angle [degrees] 0.5 0.0 0.5 1.0 1.5 2.0 E[mHa] ML Test (w/ dipole) ML Test (w/o dipole) MP2 ML Training H-F molecule in electric field of 0.001 a.u. AS Christensen et al. (2018) arXiv:1807.08811
- 27. 12/ 16 Learning dipole moments The QM9 dataset (again) Red: Learn the dipole norm Conventional KRR Blue: Learn the energy derivative Using response operators 100 1000 2500 500010000 N 1.5 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 MAE||[D] 20x less data Norm Response Learning curve for QM9 AS Christensen et al. (2018) arXiv:1807.08811
- 28. 13/ 16 Learning forces 0.05 0.1 0.2 0.4 0.6 MAEE [kcal/mol] GDML FCHL* SchNet 200 500 1000 0.2 0.3 0.4 0.6 0.8 1.0 2.0 MAEFx [kcal/mol/Å] 500 1000 500 1000 500 1000 500 1000 500 1000 500 1000 500 1000 N First derivative wrt. nuclear coordinates “MD17” benchmark set QM-MD trajectory, DFT forces and energies S Chmiela et al. (2017) Sci. Advances, e1603015 AS Christensen et al. (2018) arXiv:1807.08811
- 29. 14/ 16 Vibrational normal modes Normal modes for C6N3H7 Training set: non-equilibirum geometries of fragments N = number of samples in the training set “Rank” = total number of energies and force components in training set 0.0 0.5 Mode:1 N = 1; Rank: 214 0.0 0.5 Mode:5 0.1 0.0 0.1 0.0 0.5 Mode:15 N = 4; Rank: 856 0.0 0.1 N = 32; Rank: 6848 0.0 0.1 E[kcal/mol] RMSD displacement along normal mode [Å]
- 30. 15/ 16 Mixed derivatives and IR spectra IR intensity: Mixed derivative wrt. nuclear coordinates and external electric field. Vibrational analysis using interface to Gaussian09 Model trained on distorted geometries of dichloromethane. Training includes: Energy Dipole moments Forces Many other operators to try out! 0 1000 2000 3000 4000 5000 [cm 1 ] MP2 N = 100 FCHL* MAE: 2.4 cm 1 N = 50 FCHL* MAE: 5.7 cm 1 N = 25 FCHL* MAE: 25.6 cm 1 N = 10 FCHL* MAE: 126.1 cm 1 N = 5 FCHL* MAE: 340.4 cm 1 AS Christensen et al. (2018) arXiv:1807.08811
- 31. 15/ 16 Mixed derivatives and IR spectra IR intensity: Mixed derivative wrt. nuclear coordinates and external electric field. Vibrational analysis using interface to Gaussian09 Model trained on distorted geometries of dichloromethane. Training includes: Energy Dipole moments Forces Many other operators to try out! 0 1000 2000 3000 4000 5000 [cm 1 ] MP2 N = 100 FCHL* MAE: 2.4 cm 1 N = 50 FCHL* MAE: 5.7 cm 1 N = 25 FCHL* MAE: 25.6 cm 1 N = 10 FCHL* MAE: 126.1 cm 1 N = 5 FCHL* MAE: 340.4 cm 1 AS Christensen et al. (2018) arXiv:1807.08811
- 32. 16/ 16 QML Code Python2/3 compatible toolkit Open Source (MIT License) Manual with examples pip install qml Website: qmlcode.org GitHub: github.com/qmlcode/qml Follow me on Twitter: @AndersSChristen