A machine-learning view on heterogeneous catalyst design and discovery

/ 35
1
A machine-learning view on
heterogeneous catalyst design and discovery
Ichigaku Takigawa
ichigaku.takigawa@riken.jp
1 July 2021 @ Telluride, Colorado
Telluride Workshop on Computational Materials Chemistry
Advanced Intelligence Project

/ 35
2
RIKEN Center for AI Project Inst. Chemical Reaction Design & Discovery
Hokkaido Univ
Two Interrelated Research Interests:
Hi, I am a ML researcher working for
ML for Stem Cell Biology ML for Chemistry
ML with discrete (combinatorial) structures ML for natural sciences
Edit-aware graph autocompletion (Hu+) Low-electron dose TEM image improvement (Katsuno+)

/ 35
3
Today’s talk
Gas-phase reactions on sold-phase catalyst surface (Heterogeneous catalysis)
https://en.wikipedia.org/wiki/Heterogeneous_catalysis
Industrial Synthesis (e.g. Haber-Bosch), Automobile Exhaust Gas Puriﬁcation, Methane Conversion, etc.
Reactants
(Gas)
Catalysts
(Solid)
Nano-particle
surface
Adsorption
Diffusion
Dissociation
Recombination
Desorption
Our struggles for better ML practices with underspeciﬁed, sparse, biased
observational data (i.e. a collection of experimental facts from literature)

/ 35
3
Today’s talk
https://en.wikipedia.org/wiki/Heterogeneous_catalysis
Reactants
(Gas)
Catalysts
(Solid)
Nano-particle
surface
High Temperature, High Pressure
Adsorption
Diffusion
Dissociation
Recombination
Desorption

/ 35
4
Today’s talk
Reactants
(Gas)
Catalysts
(Solid)
Nano-particle
surface
Adsorption
Diffusion
Dissociation
Recombination
Desorption
God made the bulk;
the surface was invented by the devil
Devilishly complex too-many-factor process!!
—— Wolfgang Pauli

/ 35
5
simulation
input output
ML: A new way for (lazy) programming
computer program
input output
computer program
ML
full specification in
every detail required
give up explicit model
instead, grab a tunable
model, and show it many
input-output instances
inductive (empiricism)
airhead with a god-like
learning capability
Random Forest Neural Networks SVR Kernel Ridge
All about fitting a very-flexible function to finite points in high-dimensional space.
deductive (rationalism)

/ 35
6
ResNet50: 26 million params
EﬃcientNet-B7: 66 million params
VGG19: 144 million params
12-layer, 12-heads BERT: 110 million params
GPT-2 XL: 1558 million params
GPT-3: 175 billion params
simulation
input output
computer program
input output
computer program
ML
learning capability

/ 35
6
EﬃcientNet-B7: 66 million params
VGG19: 144 million params
GPT-2 XL: 1558 million params
GPT-3: 175 billion params
Modern ML: Can we imagine what would happen if we try to ﬁt a function having 175
billion parameters to 100 million data points in 10 thousand dimension??
simulation
input output
computer program
input output
computer program
ML
learning capability

/ 35
7
Rashomon Effect: multiplicity of good models
ML models are too flexible to overrepresent given finite instances, and many different
shapes of functions exist for representing the same finite data. (even if it’s huge)
The Rashomon Effect
In many practical cases, we have many equally-accurate but different ML models.
(the choice of ML methods or the design of NN architectures doesn’t really matter)
Note: The Rashomon Effect in ML is attributed to Leo Breiman’s very influential “The Two Cultures” paper published in
2001, but obviously “Rashomon” itself originates from a classic Japanese movie in 1950 by Kurosawa, where four
witnesses to a murder describe entirely different contradictory perspectives, but all of them sound true.
We often see this in ML competitions. Top ranking solutions are very competitive in
performance (equally accurate in practice) but can be very different approaches.
5-CV RMSE: 0.12016 5-CV RMSE: 0.13209 5-CV RMSE: 0.11976 5-CV RMSE: 0.12432 5-CV RMSE: 0.20899 5-CV RMSE: 0.17446
Neural Network Random Forest ExtraTrees GBDT Gaussian Process Kernel Ridge

/ 35
8
We see differences in underspecified cases
right data scarce/underspeciﬁed + outliers
Neural Networks (ReLU)
Random Forest
Extra Trees (bootstrap)
Random Forest
Neural Networks (Tanh)
Linear Regression
Kernel Ridge (RBF) Kernel Ridge (Laplacian)
Extra Trees (no bootstrap)
Gradient Boosting

/ 35
8
Random Forest
Random Forest
Linear Regression
Gradient Boosting
some apparently underﬁtted?

/ 35
8
Random Forest
Random Forest
Linear Regression
Gradient Boosting
some apparently underﬁtted?
But we often still see the Rashomon
(i.e. similar CV performances)
and these can predict very differently for
further test cases. (in particular, out-of-
distribution cases)

/ 35
9
Designing relevant “inductive biases”
Use heuristic assumptions, domain knowledge to constrain/regularize the model space.
Prediction
Input
variables
Classifier or
Regressor
<latexit sha1_base64="Ill3Als4zZd947f5Xm9sW99d0QA=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTomJcEd245CGPBAlp64gNpW3aQkTiD5i4lYUrTVwYP8APcOMPuOATjEtM3LjwUpoYJeJtpnPmzD13zsyVTU21Hca6PmFsfGJyyj8dmJmdmw+GFhbzttGwFJ5TDM2wirJkc03Vec5RHY0XTYtLdVnjBbm2198vNLllq4Z+4LRMXq5LVV09VhXJISp7WhEroQiLMTfCw0D0QARepIzQIw5xBAMKGqiDQ4dDWIMEm74SRDCYxJXRJs4ipLr7HOcIkLZBWZwyJGJr9K/SquSxOq37NW1XrdApGg2LlGFE2Qu7Zz32zB7YK/v8s1bbrdH30qJZHmi5WQleLGc//lXVaXZw8q0a6dnBMbZdryp5N12mfwtloG+edXrZnUy0vcZu2Rv5v2Fd9kQ30Jvvyl2aZ65H+JHJC70YNUj83Y5hkI/HxK1YPL0RSe56rfJjBatYp34kkMQ+UshR/SoucYWO4BdiwqaQGKQKPk+zhB8hJL8AVA6Qmg==</latexit>
x1
<latexit sha1_base64="QFtMwnKe2I12XGZu0bNJbdnDaaE=">AAAChnichVG7TgJBFD2sL8QHqI2JDZFgrMhAVIwV0caShzwSJGR3HXHDvrK7EJH4Aya2UlhpYmH8AD/Axh+w4BOMJSY2Fl6WTYwS8W5m58yZe+6cmSuZqmI7jHV9wtj4xOSUfzowMzs3HwwtLBZso2HJPC8bqmGVJNHmqqLzvKM4Ki+ZFhc1SeVFqb7X3y82uWUrhn7gtExe0cSarhwrsugQlTutJqqhCIsxN8LDIO6BCLxIG6FHHOIIBmQ0oIFDh0NYhQibvjLiYDCJq6BNnEVIcfc5zhEgbYOyOGWIxNbpX6NV2WN1Wvdr2q5aplNUGhYpw4iyF3bPeuyZPbBX9vlnrbZbo++lRbM00HKzGrxYzn38q9JodnDyrRrp2cExtl2vCnk3XaZ/C3mgb551ermdbLS9xm7ZG/m/YV32RDfQm+/yXYZnr0f4kcgLvRg1KP67HcOgkIjFt2KJzEYkteu1yo8VrGKd+pFECvtII0/1a7jEFTqCX4gJm0JykCr4PM0SfoSQ+gJWLpCb</latexit>
x2
<latexit sha1_base64="lFhRrRrVTrFR31ebbMgRp5myJpc=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDPjCuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmNdjzA2PjE55Z32zczOzfsDC4s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFTb7+/nm9y0FF07tFsGL9XFqqZUFFm0icqcljfKgRCLMCeCwyDqghDcSOqBRxzhGDpkNFAHhwabsAoRFn1FRMFgEFdCmziTkOLsc5zDR9oGZXHKEImt0b9Kq6LLarTu17QctUynqDRMUgYRZi/snvXYM3tgr+zzz1ptp0bfS4tmaaDlRtl/sZz5+FdVp9nGybdqpGcbFew4XhXybjhM/xbyQN886/Qyu+lwe43dsjfyf8O67IluoDXf5bsUT1+P8CORF3oxalD0dzuGQS4WiW5HYqnNUGLPbZUXK1jFOvUjjgQOkESW6ldxiSt0BK8QEbaE+CBV8LiaJfwIIfEFWE6QnA==</latexit>
x3
<latexit sha1_base64="0IPXcU0UIDvzZlYURjV2A/THv9U=">AAACiXichVG7SgNBFD2ur/hM1EawEYNiFWZFNKQKprGMj0TBBNndTHR0X+xOFmLwB6zsRK0ULMQP8ANs/AELP0EsFWwsvNksiAbjXWbnzJl77pyZq7um8CVjz11Kd09vX39sYHBoeGQ0nhgbL/pOzTN4wXBMx9vWNZ+bwuYFKaTJt12Pa5Zu8i39MNfc3wq45wvH3pR1l5ctbc8WVWFokqhiKag40t9NJFmKhTHdDtQIJBFF3knco4QKHBiowQKHDUnYhAafvh2oYHCJK6NBnEdIhPscxxgkbY2yOGVoxB7Sf49WOxFr07pZ0w/VBp1i0vBIOY1Z9sRu2Rt7ZHfshX3+WasR1mh6qdOst7Tc3Y2fTG58/KuyaJbY/1Z19CxRRTr0Ksi7GzLNWxgtfXB09raRWZ9tzLFr9kr+r9gze6Ab2MG7cbPG1y87+NHJC70YNUj93Y52UFxIqUuphbXFZHYlalUMU5jBPPVjGVmsIo8C1T/AKc5xoQwpqpJWMq1UpSvSTOBHKLkvAi+SPA==</latexit>
.
.
.
Input representation design,
selection, and its engineering
(descriptors + feature engineering)
Function
model
vs “Kitchen-sink” models
(feeding every possible descriptor)
Simple model is enough whenever we can have determining input variables necessary
and sufficient to fully define the desired output.
Every requirement should be
explicitly encoded into model,
otherwise we need to provide it
through examples.
(symmetry, invariance, etc)

/ 35
10
Prediction
Input
variables
Function
model
x2
x3
.
.
.
Latent
variables
Learnable variable
transformation
Representation learning
Classiﬁer or
Regressor
x1
Architecture design when we go for representation learning
“Kitchen-sink”
(raw) inputs

/ 35
10
Prediction
Input
variables
Function
model
x2
x3
.
.
.
Latent
variables
Learnable variable
transformation
Representation learning
Classiﬁer or
Regressor
Linear
x1
Architecture design when we go for representation learning
“Kitchen-sink”
(raw) inputs
Again, simple model is enough
when we have good features.

/ 35
11
A toy example: approximate adders
Try to teach ML “arithmetic addition” only by examples.
We can also add both 5+6 and 6+5 to tell ML
by examples that addition is commutative.
adder
<latexit sha1_base64="tiacEhQgmTkomNeV5LHmrY6hmbk=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTXxhXRDcuecgjQULaOmJDaZu2EJH4AyZuZeFKExfGD/AD3PgDLvgE4xITNy68lCZGiXib6Zw5c8+dM3NlU1Nth7GOTxgZHRuf8E8GpqZnZoOhufmcbdQthWcVQzOsgizZXFN1nnVUR+MF0+JSTdZ4Xq7u9fbzDW7ZqqEfOE2Tl2pSRVePVUVyiMqclsVyKMJizI3wIBA9EIEXSSP0iEMcwYCCOmrg0OEQ1iDBpq8IEQwmcSW0iLMIqe4+xzkCpK1TFqcMidgq/Su0KnqsTuteTdtVK3SKRsMiZRhR9sLuWZc9swf2yj7/rNVya/S8NGmW+1puloMXi5mPf1U1mh2cfKuGenZwjG3Xq0reTZfp3ULp6xtn7W5mJx1trbBb9kb+b1iHPdEN9Ma7cpfi6eshfmTyQi9GDRJ/t2MQ5NZi4lZsPbURSex6rfJjCctYpX7EkcA+kshS/QoucYW24BdiwqYQ76cKPk+zgB8hJL4AVGCQmw==</latexit>
x1
<latexit sha1_base64="4Sn0JXQ8Nli9zYaRMiYfd4a9JHg=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDPjCuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmNdjzA2PjE55Z32zczOzfsDC4s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFTb7+/nm9y0FF07tFsGL9XFqqZUFFm0icqclmPlQIhFmBPBYRB1QQhuJPXAI45wDB0yGqiDQ4NNWIUIi74iomAwiCuhTZxJSHH2Oc7hI22DsjhliMTW6F+lVdFlNVr3a1qOWqZTVBomKYMIsxd2z3rsmT2wV/b5Z622U6PvpUWzNNByo+y/WM58/Kuq02zj5Fs10rONCnYcrwp5Nxymfwt5oG+edXqZ3XS4vcZu2Rv5v2Fd9kQ30Jrv8l2Kp69H+JHIC70YNSj6ux3DIBeLRLcjG6nNUGLPbZUXK1jFOvUjjgQOkESW6ldxiSt0BK8QEbaE+CBV8LiaJfwIIfEFVoCQnA==</latexit>
x2
<latexit sha1_base64="SKy45aVsIvNNIc9wr1Q6tiMY1uI=">AAACinichVHLSsNAFL2Nr1qrrboR3BRLRRDKpIrPTVEXLvuwD6glJHFah+ZFkpbW4g+4cyXYlYIL8QP8ADf+gIt+gris4MaFN2lAtFhvmMyZM/fcOTNXMhRm2YR0fdzI6Nj4hH8yMBWcngmFZ+fyll43ZZqTdUU3i5JoUYVpNGczW6FFw6SiKim0INX2nf1Cg5oW07Uju2XQsipWNVZhsmgjVWgK/GpTSAjhKIkTNyKDgPdAFLxI6eFHOIYT0EGGOqhAQQMbsQIiWPiVgAcCBnJlaCNnImLuPoVzCKC2jlkUM0Rka/iv4qrksRqunZqWq5bxFAWHicoIxMgLuSc98kweyCv5/LNW263heGnhLPW11BBCFwvZj39VKs42nH6rhnq2oQJbrleG3g2XcW4h9/WNs6tedicTay+TW/KG/m9IlzzhDbTGu3yXppnOED8SesEXwwbxv9sxCPKJOL8RX0uvR5N7Xqv8sAhLsIL92IQkHEIKcm79S7iGDhfkEtw2t9tP5XyeZh5+BHfwBX8Zkfc=</latexit>
x1 + x2
<latexit sha1_base64="1coS+Mt9eL/r4jEoYDkZ5NYfoeE=">AAACiHichVHLSsNAFD3Gd3206kZwUywVQSg3KlYFQerGZVutCiqSxGkdTJOQpIVa/AE3LlVcKbgQP8APcOMPuPATxKWCGxfepgFRUW+YzJkz99w5M1d3TOn5RI8tSmtbe0dnV3ekp7evPxobGFzz7IpriIJhm7a7oWueMKUlCr70TbHhuEIr66ZY1/eXGvvrVeF60rZW/ZojtstayZJFaWg+UwV1YmpheieWoBQFEf8J1BAkEEbWjt1iC7uwYaCCMgQs+IxNaPD424QKgsPcNurMuYxksC9wiAhrK5wlOENjdp//JV5thqzF60ZNL1AbfIrJw2VlHEl6oGt6oXu6oSd6/7VWPajR8FLjWW9qhbMTPRpeeftXVebZx96n6k/PPoqYDbxK9u4ETOMWRlNfPTh5WZnPJ+tjdEnP7P+CHumOb2BVX42rnMif/+FHZy/8Ytwg9Xs7foK1yZQ6k5rKTScWM2GrujCCUYxzP9JYxDKyKHB9iWOc4kyJKKSklblmqtISaobwJZTMB48DkKc=</latexit>
1 + 3 = 4
<latexit sha1_base64="Jl1LVcl+mtrFJMn2AKjNmRINEM4=">AAACiHichVHLSsNAFD3GV62vqhvBjVgUQSi39VEVhFI3LtVaW1CRJE7rYJqEJC3U4g+4caniSsGF+AF+gBt/wIWfIC4ruHHhbRoQFfWGyZw5c8+dM3M125CuR/TUorS2tXd0hrrC3T29ff2RgcFN1yo7usjqlmE5eU11hSFNkfWkZ4i87Qi1pBkipx0sN/ZzFeG40jI3vKotdkpq0ZQFqaseU9nE1OxScjcSpRj5MfoTxAMQRRCrVuQO29iDBR1llCBgwmNsQIXL3xbiINjM7aDGnMNI+vsCRwiztsxZgjNUZg/4X+TVVsCavG7UdH21zqcYPBxWjmKcHumG6vRAt/RM77/Wqvk1Gl6qPGtNrbB3+4+HM2//qko8e9j/VP3p2UMB875Xyd5tn2ncQm/qK4en9czi+nhtgq7ohf1f0hPd8w3Myqt+vSbWL/7wo7EXfjFuUPx7O36CzUQsPhebXpuJptJBq0IYwRgmuR9JpLCCVWS5vsQJznCuhBVSkspCM1VpCTRD+BJK+gOby5Ct</latexit>
2 + 5 = 7
<latexit sha1_base64="NRtmyWkgZ5nlm1ETxVkZxPMO8Rc=">AAACiXichVHNLgNRGD0df6V+io3ERjREImnuUJREIrqxpNWSIM3MuGV0/jJz24TGC1jZCVYkFuIBPICNF7DwCGJZiY2Fb6aTCKK+yZ177rnf+e6591MdQ/cEY88RqaW1rb0j2tkV6+7p7Yv3DxQ8u+JqPK/Zhu1uqorHDd3ieaELg286LldM1eAbajnj729UuevptrUuDh2+Yyp7ll7SNUUQVZiZnF+UU8V4giVZECO/gRyCBMJYteP32MYubGiowASHBUHYgAKPvi3IYHCI20GNOJeQHuxzHKOLtBXK4pShEFum/x6ttkLWorVf0wvUGp1i0HBJOYIx9sRuWZ09sjv2wj7+rFULavheDmlWG1ruFPtOhnLv/6pMmgX2v1RNPQuUkA686uTdCRj/FlpDXz06q+cWsmO1cXbNXsn/FXtmD3QDq/qm3azx7GUTPyp5oRejBsk/2/EbFKaS8mxyei2VWFoOWxXFMEYxQf2YwxJWsIo81T/AKc5xIcUkWUpLC41UKRJqBvEtpMwnMx2Q7A==</latexit>
5 + 9 = 14
<latexit sha1_base64="wFMJb33bMt0hW8th8BMIqvUdNU8=">AAACinichVFLSwJRFD5OLzNLq03QRhIjCOSMRk8CqRYtfeQDTGRmutrgvJgZBZP+QLtWQa4KWkQ/oB/Qpj/Qwp8QLQ3atOg4DkRJdoY797vfPd+5371HNBTZshE7Hm5kdGx8wjvpm/JPzwSCs3M5S6+bEstKuqKbBVGwmCJrLGvLtsIKhskEVVRYXqzt9/bzDWZasq4d2U2DlVShqskVWRJsovLxVR53+Xg5GMYoOhEaBLwLwuBGUg8+wjGcgA4S1EEFBhrYhBUQwKKvCDwgGMSVoEWcSUh29hmcg4+0dcpilCEQW6N/lVZFl9Vo3atpOWqJTlFomKQMQQRf8B67+IwP+Iqff9ZqOTV6Xpo0i30tM8qBi4XMx78qlWYbTr9VQz3bUIFNx6tM3g2H6d1C6usbZ1fdzHY60lrGW3wj/zfYwSe6gdZ4l+5SLN0e4kckL/Ri1CD+dzsGQS4W5dej8dRaOLHntsoLi7AEK9SPDUjAISQh69S/hGtoc34uxm1xO/1UzuNq5uFHcAdfqESRGw==</latexit>
3 + 10 = 13
<latexit sha1_base64="FoI7hqDrtQtFjlgLPgriDwtw1iU=">AAACiXichVHLSsNAFD2Nr1pfVTeCm2JRBKFMfNQiCKIbl33YVlApSRx1NE1CMi1o8QdcuRN1peBC/AA/wI0/4MJPEJcV3LjwNg2IFvWGyZw5c8+dM3N1xxSeZOw5pLS1d3R2hbsjPb19/QPRwaGCZ1dcg+cN27TddV3zuCksnpdCmnzdcblW1k1e1A9WGvvFKnc9YVtr8tDhW2Vt1xI7wtAkUYW5qeSiqpaicZZgfsRagRqAOIJI29F7bGIbNgxUUAaHBUnYhAaPvg2oYHCI20KNOJeQ8Pc5jhEhbYWyOGVoxB7Qf5dWGwFr0bpR0/PVBp1i0nBJGcM4e2K3rM4e2R17YR+/1qr5NRpeDmnWm1rulAZORnLv/6rKNEvsfan+9Cyxg5TvVZB3x2catzCa+urRWT23kB2vTbBr9kr+r9gze6AbWNU34ybDs5d/+NHJC70YNUj92Y5WUJhOqMnETGY2vrQctCqMUYxhkvoxjyWsIo081d/HKc5xofQoqpJSFpqpSijQDONbKCufJlSQ5g==</latexit>
5 + 6 = 11
<latexit sha1_base64="n3Yy/9dJfdSB5yYDzUvImVPJv/g=">AAACiHichVHLSsNAFD3Gd3206kZwIxZFEMqNii8QRTcu1VotVClJnNahaRKStKDFH3DjUsWVggvxA/wAN/6Ai36CuFRw48LbNCAq1hsmc+bMPXfOzNUdU3o+UbVJaW5pbWvv6Ix0dff0RmN9/dueXXINkTJs03bTuuYJU1oi5UvfFGnHFVpRN8WOXlit7e+UhetJ29ryDx2xV9TylsxJQ/OZSqkT6uJSNhanBAUx/BuoIYgjjHU7do9d7MOGgRKKELDgMzahweMvAxUEh7k9VJhzGclgX+AYEdaWOEtwhsZsgf95XmVC1uJ1raYXqA0+xeThsnIYo/REt/RKj3RHz/TxZ61KUKPm5ZBnva4VTjZ6Mph8/1dV5NnHwZeqoWcfOcwFXiV7dwKmdgujri8fnb0mFzZHK2N0TS/s/4qq9MA3sMpvxs2G2Lxs4EdnL/xi3CD1Zzt+g+3JhDqTmNqYji+vhK3qwBBGMM79mMUy1rCOFNeXOMU5LpSIQsqsMl9PVZpCzQC+hbLyCaIfkLA=</latexit>
1 + 1 =?
<latexit sha1_base64="hRdVP+e2y4bjP1zOby3ZyPytH4w=">AAACi3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKRRhqa+gaatraxwsoG+gZgIECJsMQylBmgIKAfIHtDDEMKQz5DMkMpQy5DKkMeQwlQHYOQyJDMRBGMxgyGDAUAMViGaqBYkVAViZYPpWhloELqLcUqCoVqCIRKJoNJNOBvGioaB6QDzKzGKw7GWhLDhAXAXUqMKgaXDVYafDZ4ITBaoOXBn9wmlUNNgPklkognQTRm1oQz98lEfydoK5cIF3CkIHQhdfNJQxpDBZgt2YC3V4AFgH5Ihmiv6xq+udgqyDVajWDRQavge5faHDT4DDQB3llX5KXBqYGzcbjniSgW4AhBowgQ/TowGSEGekZmukZB5ooOzhBo4qDQZpBiUEDGB/mDA4MHgwBDKHgeJjEMJthDhMvkzGTFZMNRCkTI1SPMAMKYHIFACGukUw=</latexit>
1 + ( 1) =?
<latexit sha1_base64="iSckyhI7u8UoVI74y32YMTkgc7c=">AAACj3ichVHLSsNAFD3Gd3201Y3gRiwVQSiTVmwrqEU3umutVUFFkjitoWkSkmlBiz/gB+jChQ9wIX6AH+DGH3DhJ4jLCm5ceJsGREW9YTJnztxz58xc1TZ0VzD21Ca1d3R2dff0Bvr6BwaDofDQumtVHY0XNMuwnE1Vcbmhm7wgdGHwTdvhSkU1+IZaXmrub9S44+qWuSYObL5TUUqmXtQ1RRC1LcdT6fhUOpVIzi3shiIsxrwY+wlkH0TgR9YK3WEbe7CgoYoKOEwIwgYUuPRtQQaDTdwO6sQ5hHRvn+MIAdJWKYtThkJsmf4lWm35rEnrZk3XU2t0ikHDIeUYouyR3bAGe2C37Jm9/1qr7tVoejmgWW1pub0bPB7Jv/2rqtAssP+p+tOzQBEpz6tO3m2Pad5Ca+lrh6eN/OxqtD7BrtgL+b9kT+yebmDWXrXrHF89+8OPSl7oxahB8vd2/ATr8Zg8E0vkpiOZRb9VPRjFOCapH0lksIwsClTfxgnOcSGFpaQ0L2VaqVKbrxnGl5BWPgDiqZJ1</latexit>
12892 + 9837 =?
<latexit sha1_base64="3cCCvxejuSwhH3gAGZntIRJA/Ck=">AAAChHichVHNLgNRFP46/uunxUZi02iIhTRnWtWOhTRsLFsUCSIz42LS6cxkZtqExguwJRZWJBbiATyAjRew8AhiSWJj4cy0IhY4N/fec79zvnO/e4/mmIbnEz1FpLb2js6u7p5ob1//QCw+OLTq2TVXF2XdNm13XVM9YRqWKPuGb4p1xxVqVTPFmlZZCOJrdeF6hm2t+AeO2Kqqe5axa+iqz1ApvR1PUkrJZ7OZfIJSREpaybGjKIqckxMyI4El0bKiHb/DJnZgQ0cNVQhY8Nk3ocLjsQEZBIexLTQYc9kzwrjAEaLMrXGW4AyV0Qqve3zaaKEWn4OaXsjW+RaTp8vMBMbpkW7olR7olp7p49dajbBGoOWAd63JFc527Hhk+f1fVpV3H/vfrD81+9hFPtRqsHYnRIJX6E1+/fD8dXl2abwxQVf0wvov6Ynu+QVW/U2/Lomliz/0aKyFf4wb9NWFxO/Oajolz6QypelkYb7Vqm6MYgyT3I8cClhEEWWuL3CCU5xJndKUlJGyzVQp0uIM44dJc58RFJAW</latexit>
2
<latexit sha1_base64="dtxzgykiOKrKKToFg2z7LT9XiS0=">AAAChHichVHNLgNRFP6Mv/ovNhKbRkMspDnTqnYsRNhYaikSRGbGbU1MZyYz0ybVeAG2xMKKxEI8gAew8QIWfQSxJLGxcGZaEQucm3vvud8537nfvUdzTMPziRptUntHZ1d3pKe3r39gcCg6PLLh2RVXFwXdNm13S1M9YRqWKPiGb4otxxVqWTPFpna4HMQ3q8L1DNta92uO2C2rJcsoGrrqM5SjvWicEko2nU5lY5QgUpJKhh1FUeSMHJMZCSyOlq3a0XvsYB82dFRQhoAFn30TKjwe25BBcBjbRZ0xlz0jjAsco5e5Fc4SnKEyeshriU/bLdTic1DTC9k632LydJkZwyQ90S290iPd0TN9/FqrHtYItNR415pc4ewNnYytvf/LKvPu4+Cb9admH0VkQ60Ga3dCJHiF3uRXjy5e1+bzk/UpuqYX1n9FDXrgF1jVN/0mJ/KXf+jRWAv/GDfoqwux352NZEKeS6Rys/HFpVarIhjHBKa5HxksYgWrKHB9gVOc4VzqkmaklJRupkptLc4ofpi08AkM1JAU</latexit>
0
<latexit sha1_base64="P9nwiNSxLZKIKI1oebJys0uGCoU=">AAACiHichVHLLgRBFD3ae7wGG4nNxIRYTW63R09bydhYjscgQaS7FSp6ujvdPZMw8QM2logViYX4AB9g4wcsfIJYkthYuN0zIha4laq6de49t07VtXxHhhHRU5PS3NLa1t7Rmerq7untS/cPrIReJbBFyfYcL1izzFA40hWlSEaOWPMDYZYtR6xa+3NxfLUqglB67nJ04IvNsrnryh1pmxFDJU3TNWMrnaWckZ+amshnKEdkaIbOjmEYqq5mVEZiy6JhRS99hw1sw4ONCsoQcBGx78BEyGMdKgg+Y5uoMRawJ5O4wBFSzK1wluAMk9F9Xnf5tN5AXT7HNcOEbfMtDs+AmRmM0iPd0Cs90C0908evtWpJjVjLAe9WnSv8rb7joaX3f1ll3iPsfbP+1BxhB/lEq2TtfoLEr7Dr/Orh6evSzOJobYyu6IX1X9IT3fML3Oqbfb0gFi/+0GOxFv4xbtBXFzK/OytaTp3OTSxMZmcLjVZ1YBgjGOd+6JjFPIoocX2JE5zhXEkppOiKUU9VmhqcQfwwpfAJa7yREg==</latexit>
22729
train
test
(the case where a clear answer and underlying logic exist)

/ 35
12
RF says 1 + 1 = 5.15 and 1 - 1 = 5.15 and 12892 + 9837 = 12.75
MLP better? But anyway it’s totally wrong.
2-10-5-1 Feed Forward NN (MLP, Multi-Layer Perceptron)
adder
x1
x2
x1 + x2
1 + 3 = 4
2 + 5 = 7
5 + 9 = 14
3 + 10 = 13
5 + 6 = 11
1 + 1 =?
1 + ( 1) =?
12892 + 9837 =?
2
0
22729
train
test
(the case where a clear answer and inductive logic exist)
Try to teach ML “arithmetic addition” only by examples.

/ 35
13
Linear regression rocks! 😆
LR says 1 + 1 = 2 and 1 - 1 = 0 and 12892 + 9837 = 22729
adder
x1
x2
x1 + x2
1 + 3 = 4
2 + 5 = 7
5 + 9 = 14
3 + 10 = 13
5 + 6 = 11
1 + 1 =?
1 + ( 1) =?
12892 + 9837 =?
2
0
22729
train
test
(Perfect Answers!!)

/ 35
13
Linear regression rocks! 😆
LR says 1 + 1 = 2 and 1 - 1 = 0 and 12892 + 9837 = 22729
All are because of “inductive bias” intrinsically encoded in the model.
LR with two input variables is just
ﬁtting a plane
<latexit sha1_base64="iFKbOGXhRe3/O37wsq+JtGPu+ec=">AAAClnichVHLSsNAFD2Nr1pfrW4EN8FSEYQyUVFxIaKILvuwraBSkjitoWkSkrRaiz/gD7gQFwoq4gf4AW78ARf9BHFZwY0Lb9OAqKg3JPfMmXtuzsxVLF1zXMYaAaGjs6u7J9gb6usfGBwKR4azjlmxVZ5RTd20txTZ4bpm8IyruTrfsmwulxWd55TSams/V+W2o5nGpluz+G5ZLhpaQVNll6h8OCKLh3lJnBIVytOU1Xw4yuLMC/EnkHwQhR8JM3yPHezBhIoKyuAw4BLWIcOhZxsSGCzidlEnziakefscxwiRtkJVnCpkYkv0LdJq22cNWrd6Op5apb/o9NqkFBFjT+yWNdkju2PP7P3XXnWvR8tLjbLS1nIrP3Qymn77V1Wm7GL/U/WnZxcFLHheNfJueUzrFGpbXz06baYXU7H6BLtkL+T/gjXYA53AqL6qV0meOvvDj0Je6MZoQNL3cfwE2em4NBefSc5Gl1f8UQUxhnFM0jzmsYwNJJCh/gc4xzVuhFFhSVgT1tulQsDXjOBLCIkPAxKUnQ==</latexit>
ax1 + bx2 + c
Any three instances are enough to have
<latexit sha1_base64="57verNgszw6m8+AhOWc3Hu105bk=">AAACp3ichVFNS9xQFD2mftX6MW03BTfBwaIow42WtgiC6MadOjrjgMrwEp/jw0wSkjdT7eBe/AMuXCm4KAW3unfjH+jCn1C6VHDjojeZgKioN+S+88675+a8XDtwVaSJrlqMN61t7R2db7vedff09mXefyhGfi10ZMHxXT8s2SKSrvJkQSvtylIQSlG1Xblsb83E58t1GUbK95b0TiDXqqLiqQ3lCM1UOTMgJq1R046TM0mreVXZ1CIM/R/mdtkyRziPlTNZylES5lNgpSCLNOb9zBlWsQ4fDmqoQsKDZuxCIOJnBRYIAXNraDAXMlLJucQuulhb4yrJFYLZLc4V3q2krMf7uGeUqB3+istvyEoTg/SHftE1XdJv+kt3z/ZqJD1iLzu82k2tDMp9+58Wb19VVXnV2LxXvehZYwPfE6+KvQcJE9/CaerrPw+uFyfyg43PdEz/2P8RXdEF38Cr3zgnCzJ/+IIfm73wH+MBWY/H8RQUx3LW19z4wpfs1HQ6qk70YwBDPI9vmMIs5lHg/ns4xRnOjWFjzigapWap0ZJqPuJBGOI/dKSbIg==</latexit>
a = 1, b = 1, c = 0 ) x1 + x2
MLP has partial “linearity” inside and that’s why MLP is better than RF is.
RF is “piecewise constant” and only returns values between sample min and max.
to points in 3D
adder
x1
x2
x1 + x2
1 + 3 = 4
2 + 5 = 7
5 + 9 = 14
3 + 10 = 13
5 + 6 = 11
1 + 1 =?
1 + ( 1) =?
12892 + 9837 =?
2
0
22729
train
test
(Perfect Answers!!)

/ 35
14
“Inductive biases” (often unintendedly) defines the interspace of instances, and then
also defines the prediction for unseen areas (crucial for out-of-distribution predictions).
Random Forest
Linear Regression Extra Trees (no bootstrap)
Carefully designed inputs
(input only confidently relevant
info into ML)
Conservative models + Strong
inductive biases that best fit to
the given problem
General models having a large number
of parameters + Generalizable
inductive biases
(enables zero-shot/few-shot transfer?)
Kitchen-sink inputs (input all
potentially relevant info into ML)
Use any “physics-informed” conditions to further constrain or regularize
the model space sounds a good idea indeed
Complex
Simple
Small Large
Input-Output Correlation for Target
Data Required To Make ML Work

/ 35
15
A recent news: OGB Large-Scale Challenge @ KDDCup 2021
Current ML is too data-hungry (and purportedly vulnerable to any data bias)
Gaps between technical interests and reality
Modern ML can learn any input-output mappings in theory, but more data is needed
when the input-output correlation is weak.
PCQM4M-LSC predicting DFT-calculated HOMO-LUMO energy gap of molecules given their
2D molecular graphs. (3,803,453 graphs from PubChemQC; cf. 133,885 graphs for QM9)
1st place: 10 GNNs (12-Layer Graphormer) + 8 ExpC*s (5-Layer ExpandingConv)
73 GNNs (11-Layer LiteGEMConv with Self-Supervised Pretraining)
20 GNNs (32-Layer GN with Noisy Nodes)
Test MAE 0.1200 (eV)
2nd place: Test MAE 0.1204 (eV)
3rd place: Test MAE 0.1205 (eV)
Our reality. Practical “open-end” cases
we can only have very limited data relative to the astronomically vast search space.
Our technical interests. we’re very excited to explore ML over large data (for pretraining +
transfer) with generalizable modular structures: CNNs vs Transformers vs GNNs vs MLPs

/ 35
16
Today’s talk
Reactants
(Gas)
Catalysts
(Solid)
Nano-particle
surface
Adsorption
Diffusion
Dissociation
Recombination
Desorption
God made the bulk;

/ 35
17
The base dataset
http://www.fhi-berlin.mpg.de/acnew/department/pages/ocmdata.html
https://www.nature.com/articles/s41467-019-08325-8#Sec19
Oxidative coupling of methane (OCM) reactions
Methane (CH4) is partially oxidized to C2 hydrocarbons such as ethane (C2H6) and ethylene (C2H4) in a single step
Elemental composition of catalyst (mol%) Process parameters + Preparation Catalytic performance
• Zavyalova, U.; Holena, M.; Schlögl, R.; Baerns, ChemCatChem 2011.
• Followup:
Kondratenko, E. V.; Schlüter, M.; Baerns, M.; Linke, D.; Holena, M.
Catal. Sci. Technol. 2015.
• Renalysis with Corrections & Outlier Removal
Schmack, R.; Friedrich, A.; Kondratenko, E. V.; Polte, J.; Werwatz, A.;
Kraehnert, R. Nat Commun 2019.
1866 catalyst records from 421 reports

/ 35
18
Problem #1: It’s underspecified
• Each catalyst was mostly measured at different reaction conditions.
• Only a few are measured under multiple conditions
158 compositions > 2 conditions
La:100.0 x 24
Mg:100.0 x 18
Ca:100.0 x 18
Sm:100.0 x 13
Nd:100.0 x 10
Ba:100.0 x 9
Y:100.0 x 9
Ce:100.0 x 9
Zr:100.0 x 9
Sr:100.0 x 7
Si:100.0 x 7
Gd:100.0 x 7
Na:8.9 Si:83.1 Mn:3.5 W:4.5 x 7
Pr:100.0 x 6
Eu:100.0 x 6
Yb:100.0 x 5
Al:100.0 x 4
Li:10.0 Mg:90.0 x 4
Mg:90.9 La:9.1 x 4
Li:9.1 Mg:90.9 x 4
Tb:100.0 x 4
Li:20.0 Cl:20.0 Zr:60.0 x 4
Na:20.0 Cl:20.0 Zr:60.0 x 4
Cl:20.0 K:20.0 Zr:60.0 x 4
Cl:20.0 Rb:20.0 Zr:60.0 x 4
Cl:20.0 Zr:60.0 Cs:20.0 x 4
• No replicates in the same conditions
• But as we see later, reaction conditions are quite inﬂuential. Because of
this, “no generally valid correlation between a catalyst’s composition,
its structure and its OCM performance has been established yet.”
Strong limitation of observational data (just passively acquired)
Observational study Interventional study

/ 35
19
Problem #2: It’s sparse
74 elements All pairwise comparisons
Mostly, arbitrary pairs of catalysts don’t even have any common elements.
Can we meaningfully compare 'Na:33.2 Ti:0.5 Mn:66.3’ and 'Zn:77.8 Ce:22.2’ …?
['Na:33.2 Ti:0.5 Mn:66.3', 'Zn:77.8 Ce:22.2']
['C:32.7 K:65.4 Pb:1.9', 'Y:100.0']
['Na:66.7 Mo:33.3', 'Al:94.5 Mo:5.5']
['C:4.0 Na:4.0 Ce:92.0', 'Y:70.0 Bi:30.0']
['Si:98.2 Cs:1.8', 'Ti:50.0 Gd:50.0']
['Na:9.1 Si:82.8 Cr:3.6 W:4.5', 'Na:20.0 Mg:80.0']
['Y:66.7 Ba:33.3', 'Al:77.0 Ag:18.0 Ba:5.0']
['Al:87.0 Cl:8.0 Fe:1.0 Sr:4.0', 'Na:33.2 Mn:66.3 Ta:0.5']
['Na:1.0 La:99.0', 'Li:9.1 Ca:90.9']
['Fe:100.0', 'Sr:50.0 Nd:50.0’]
['Al:75.0 Cl:16.0 Sr:8.0 Rh:1.0', 'Na:4.5 Si:79.2 Mn:16.3']
['S:2.9 K:5.7 Ca:91.4', 'Sm:100.0']
['P:34.5 Sr:65.5', 'Na:58.3 Cl:25.0 Mo:16.7']
['Li:23.0 Si:73.2 W:3.8', 'Si:33.3 Ca:66.6 Pb:0.1']
['Al:90.5 Ag:8.5 Pr:1.0', 'Na:66.7 Mo:33.3']
['Cl:20.0 Ba:10.0 Nd:70.0', 'Mg:90.9 La:9.1']
['Gd:100.0', 'Mn:50.0 Mo:50.0']
['Na:76.9 Nb:23.1', 'La:90.0 Pb:10.0']
['Li:6.5 S:3.2 Ca:90.3', 'P:34.5 Sr:65.5']
['Na:5.0 Si:72.0 Cl:5.0 Mn:18.0', 'P:34.0 S:7.5 Ca:51.0 Pb:7.5']

/ 35
20
Problem #3: It’s biased
Unavoidable Human-Caused Biases
“most chemical experiments are planned by human scientists and therefore are subject
to a variety of human cognitive biases, heuristics and social inﬂuences.”
Jia, X.; Lynch, A.; Huang, Y.; Danielson, M.; Lang’at, I.; Milder, A.; Ruby, A. E.; Wang, H.; Friedler, S. A.;
Norquist, A. J.; Schrier, J. Nature 2019, 573 (7773), 251–255.
Catalyst such as LaO3, Li/MgO, and
Mn/Na2WO4/SiO2 extensively studied.

/ 35
21
Our solutions
Solution to Problem #1 (Underspeciﬁcation)
Tree ensemble regressors with prediction variances are used to make robust and less
risky prediction as well as to quantify how uncertain each ML prediction is.
Solution to Problem #2 (Sparsity)
The catalyst representation called SWED (Sorted Weighted Elemental Descriptors) is
developed to represent catalysts not in a one-hot fashion but by elemental descriptors.
Solution to Problem #3 (Strong Bias)
On the top of the above two, sequential model-based optimization with SWED only by 3
descriptors (electronegativity, density, and ΔHfus) as well as 8 descriptors are explored.
Also, for suggested candidates to be worth testing, SHAP interpretations are provided.
OCM Dataset Update, Reanalysis, Exploration
The original dataset (1866 catalyst records from 421 reports until 2009) is extended to
4559 catalyst records from 542 reports from 2010 to 2019, and reanalyzed.

/ 35
22
#1. Tree ensemble regression with uncertainty
Tree ensemble regressors with prediction variances are used to make robust and less
risky prediction as well as to quantify how uncertain each ML prediction is.
Gradient Boosted Trees
Random Forest
sample max
sample min
GradientBoostingRegressor
LGBMRegressor
RandomForestRegressor
ExtraTreesRegressor
By quantile
regression
to .16, .5, .84
quantiles
Naturally by
the law of
total variance
bounded
prediction
Avoid the risk of unintended extrapolation?
(High-dimensional feature spaces can be
counterintuitive…)

/ 35
23
#2. SWED representation of catalysts
The catalyst representation called SWED (Sorted Weighted Elemental Descriptors) is
developed to represent catalysts not in a one-hot fashion but by elemental descriptors.
Key Idea
one-hot-like features below are statistically incomparable
so represent catalysts instead by any elemental descriptors
to represent arbitrary (used/unused) elements in a common
ground, considering chemical similarities.

/ 35
24
SWED (Sorted Weighted Elemental Descriptors)
• Product terms can represent interaction effects between variables (e.g. probabilistic gating, attention, …)
and furthermore, they can zero out the feature when the corresponding element is 0%.
• Sorted concatenation is lossless, and was better than weighted sum or weighted max.
• SWED lose the exact composition. To compensate, we also developed a SWED→composition estimator.
• We tried many other things (matrix decomposition, Aitchison geometry, GNN, etc) that didn’t work.
+

/ 35
25
#3. Optimism in the face of uncertainty
<latexit sha1_base64="Nr2ImZk9Gvp+/1cBWPwAK6yswp0=">AAACiXichVFNLwNRFD0dX1VfxUZi02iIVXNHBOmq0Y1lP7QkiMyMh9H5ysy0UU3/gJWdYEViIX6AH2DjD1j0J4gliY2F2+kkguBO3rzzzrvnvvPeVR1D93yiVkTq6u7p7Yv2xwYGh4ZH4qNjZc+uupooabZhu+uq4glDt0TJ131DrDuuUEzVEGtqJdveX6sJ19Nta9WvO2LLVPYsfVfXFJ+p8qZqNg6b2/EkpSiIxE8ghyCJMHJ2/A6b2IENDVWYELDgMzagwONvAzIIDnNbaDDnMtKDfYEmYqytcpbgDIXZCv/3eLURshav2zW9QK3xKQYPl5UJTNMj3dALPdAtPdH7r7UaQY22lzrPakcrnO2R44ni278qk2cf+5+qPz372MVS4FVn707AtG+hdfS1o9OXYrow3ZihK3pm/5fUonu+gVV71a7zonDxhx+VvfCLcYPk7+34CcpzKXkhRfn5ZGY5bFUUk5jCLPdjERmsIIcS1z/ACc5wLg1IsrQkpTupUiTUjONLSNkPIVKSSQ==</latexit>
x
<latexit sha1_base64="Qqfga3YaBoUI0Yb7fQVzumPP1Sw=">AAAChHichVHLSsNAFD1GrbW+qm4EN8WiuJBy6xsXUnTjsg9rC1VKEqc1mCYhSQu1+AO6VVy4UnAhfoAf4MYfcNFPEJcKblx4mwZEi3rDZM6cuefOmbmKpWuOS9Tskrp7egN9wf7QwODQ8Eh4dGzHMau2KrKqqZt2XpEdoWuGyLqaq4u8ZQu5ougipxxutvZzNWE7mmlsu3VL7FXksqGVNFV2mUrVi+EoxciLSCeI+yAKP5Jm+B672IcJFVVUIGDAZaxDhsNfAXEQLOb20GDOZqR5+wLHCLG2ylmCM2RmD/lf5lXBZw1et2o6nlrlU3QeNisjmKYnuqVXeqQ7eqaPX2s1vBotL3WelbZWWMWRk4nM+7+qCs8uDr5Uf3p2UcKq51Vj75bHtG6htvW1o4vXzFp6ujFD1/TC/q+oSQ98A6P2pt6kRPryDz8Ke+EX4wbFf7ajE+zMx+LLMUotRhMbfquCmMQUZrkfK0hgC0lkub7AKc5wLgWkOWlBWmqnSl2+ZhzfQlr/BNcbj/U=</latexit>
y
sample max
sample min
We would like to ﬁnd X better than (hopefully) the currently known best .
<latexit sha1_base64="g1U7Tzgl+Lj3TSOL2Mnl/fuxw90=">AAACi3ichVHLSsNAFL2Nr1qtrboR3BRLxVW5UVEpLooiuOzDPqAtJYljDc2LJC3W0B9w6cZF3Si4ED/AD3DjD7joJ4jLCm5ceJsGRIv1hsmcOXPPnTNzRUORLRux6+PGxicmp/zTgZnZ4FwoPL+Qt/SGKbGcpCu6WRQFiymyxnK2bCusaJhMUEWFFcT6fn+/0GSmJevakd0yWEUVapp8IkuCTVSxLKrOWbvKV8NRjKMbkWHAeyAKXqT08COU4Rh0kKABKjDQwCasgAAWfSXgAcEgrgIOcSYh2d1n0IYAaRuUxShDILZO/xqtSh6r0bpf03LVEp2i0DBJGYEYvuA99vAZH/AVP/+s5bg1+l5aNIsDLTOqoYul7Me/KpVmG06/VSM923ACO65XmbwbLtO/hTTQN8+vetlEJuas4i2+kf8b7OIT3UBrvkt3aZbpjPAjkhd6MWoQ/7sdwyC/Hue34pjejCb3vFb5YRlWYI36sQ1JOIQU5Nw+XEIHrrkgt8EluN1BKufzNIvwI7iDL6QMku0=</latexit>
x1
<latexit sha1_base64="HrABz/ySZPgA1Ybj81k3XV8tH68=">AAACi3ichVHNSgJRFD5Of2aZVpugjSRGKzlaVEgLKYKW/uQPqMjMdLXB+WNmlGzwBVq2aWGbghbRA/QAbXqBFj5CtDRo06LjOBAl2Rnu3O9+93znfvceQZcl00LsebiJyanpGe+sb27evxAILi7lTa1piCwnarJmFAXeZLKkspwlWTIr6gbjFUFmBaFxMNgvtJhhSpp6bLV1VlH4uirVJJG3iCqWBcU+61Tj1WAYo+hEaBTEXBAGN1Ja8BHKcAIaiNAEBRioYBGWgQeTvhLEAEEnrgI2cQYhydln0AEfaZuUxSiDJ7ZB/zqtSi6r0npQ03TUIp0i0zBIGYIIvuA99vEZH/AVP/+sZTs1Bl7aNAtDLdOrgYuV7Me/KoVmC06/VWM9W1CDXcerRN51hxncQhzqW+dX/WwiE7HX8RbfyP8N9vCJbqC23sW7NMt0x/gRyAu9GDUo9rsdoyAfj8a2o5jeCif33VZ5YRXWYIP6sQNJOIIU5Jw+XEIXrjk/t8kluL1hKudxNcvwI7jDL6Ysku4=</latexit>
x2
<latexit sha1_base64="qj6YHRPaHOekeYfY3235sFbWBxQ=">AAAChHichVHNLgNRFP6M/98WG4lNoyEW0pzpn9YCYWOpqDYpkZlxMel0ZjIzbULjBdgSCysSC/EAHsDGC1h4BLEksbFwZloRi3Ju7r3nfud85373HtU2dNcjem6T2js6u7p7evv6BwaHQuHhkU3XqjqayGuWYTlFVXGFoZsi7+meIYq2I5SKaoiCWl7244WacFzdMje8Q1tsV5R9U9/TNcVjKLewE45SLJtJUzIeoRhRJk5pdlIkZ+VsRGbEtyiatmqF77GFXVjQUEUFAiY89g0ocHmUIINgM7aNOmMOe3oQFzhGH3OrnCU4Q2G0zOs+n0pN1OSzX9MN2BrfYvB0mBnBJD3RLb3RI93RC322rFUPavhaDnlXG1xh74ROxtY//mVVePdw8MP6U7OHPWQCrTprtwPEf4XW4NeOLt7W59Ym61N0Ta+s/4qe6YFfYNbetZucWLv8Q4/KWvjHuEHfXYi0djbjMTkdS+SS0cWlZqt6MI4JTHM/ZrGIFawiz/UFTnGGc6lLmpESUqqRKrU1OaP4ZdL8F+zrkAQ=</latexit>
?
?
?
?
?
What next location of is likely to give higher ?
<latexit sha1_base64="GizvUUdXn2Kx+/exfuoNL7eSBEo=">AAACiXichVG7TgJBFD2sL8QHqI2JDZFgrMgsEh5WRBtLHgImYMzuOuDqvrK7EJX4A1Z2Rq00sTB+gB9g4w9Y8AnGEhMbCy8LxliodzM7Z87cc+fMXNnSVMdlrOMThoZHRsf844GJyanpYGhmtuyYTVvhJcXUTHtLlhyuqQYvuaqr8S3L5pIua7wiH6z39istbjuqaWy6Rxbf1qWGodZVRXKJKtdkvX14shOKsFg8kxRTK2EPJFKiB+JiOhMWY8yLCAaRM0MPqGEXJhQ0oYPDgEtYgwSHvipEMFjEbaNNnE1I9fY5ThAgbZOyOGVIxB7Qv0Gr6oA1aN2r6XhqhU7RaNikDCPKntkd67Inds9e2MevtdpejZ6XI5rlvpZbO8HT+eL7vyqdZhd736o/PbuoI+15Vcm75TG9Wyh9fev4vFtcLUTbS+yGvZL/a9Zhj3QDo/Wm3OZ54eoPPzJ5oRejBn11Ifw7KMdjYjK2kk9EsmuDVvmxgEUsUz9SyGIDOZSo/j7OcIFLYUIQhbSw2k8VfAPNHH6EsP4J1yOSpA==</latexit>
x <latexit sha1_base64="fNJj/X2guXDQcypavSpjd2rZHL0=">AAAChHichVG7SgNBFD1ZXzG+ojaCzWJQLGSZTWISLUS0sczDqKAiu+skLu6L3U0gBn9AW8XCSsFC/AA/wMYfsMgniKWCjYV3NxGxUO8yO2fO3HPnzFzVMXTPZ6wVEbq6e3r7ov2xgcGh4ZH46NiGZ9dcjZc127DdLVXxuKFbvOzrvsG3HJcrpmrwTfVwNdjfrHPX021r3W84fNdUqpZe0TXFJ6rQ2IsnmJRcyMjZlBiCdFYOQVLOLYiyxMJIoBN5O36PHezDhoYaTHBY8AkbUODRtw0ZDA5xu2gS5xLSw32OY8RIW6MsThkKsYf0r9Jqu8NatA5qeqFao1MMGi4pRUyzJ3bLXtkju2PP7OPXWs2wRuClQbPa1nJnb+RkovT+r8qk2cfBt+pPzz4qyIVedfLuhExwC62trx9dvJYWi9PNGXbNXsj/FWuxB7qBVX/Tbgq8ePmHH5W80ItRg766IP4ONpKSnJFShXRieaXTqigmMYVZ6kcWy1hDHmWqz3GKM5wLvcKckBLm26lCpKMZx48Qlj4BjPuQUA==</latexit>
y
Exploitation Make the best decision given current information
Exploration Gather more information by probing uncertain areas
A fundamental choice: exploitation-exploration tradeoff
x1
x2
exploitative choice
explorative choice
<latexit sha1_base64="TEyxI6Ffe+b+OEFRulxYJ0sUB8w=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBGBdk8IVxRXTjkoc8EkTS1gEbStu0hQSJP2DiVhauNHFh/AA/wI0/4IJPMC4xcePCS2lilIi3mc6ZM/fcOTNXMlTFshnreISR0bHxCe+kb2p6ZtYfmJvPWnrdlHlG1lXdzEuixVVF4xlbsVWeN0wu1iSV56Tqfm8/1+Cmpejaod00eLEmVjSlrMiiTVS6ebxWCoRYhDkRHARRF4TgRkIPPOIIJ9Aho44aODTYhFWIsOgrIAoGg7giWsSZhBRnn+McPtLWKYtThkhslf4VWhVcVqN1r6blqGU6RaVhkjKIMHth96zLntkDe2Wff9ZqOTV6Xpo0S30tN0r+i8X0x7+qGs02Tr9VQz3bKGPH8aqQd8NhereQ+/rGWbub3k2FWyvslr2R/xvWYU90A63xLt8leep6iB+JvNCLUYOiv9sxCLLrkeh2ZCO5GYrvua3yYgnLWKV+xBDHARLIUP0KLnGFtuAVIsKWEOunCh5Xs4AfIcS/AEWBkJQ=</latexit>
y⇤
<latexit sha1_base64="ivVWRQUyNt8f9PjHjZnGRvohU5E=">AAAChnichVG5TsNAEH2YK9wBGiSaiCgIUURjbqgiaChDQg6JS7ZZgoVjW7YTKUT8ABItFFQgUSA+gA+g4Qco8gmIMkg0FEwcI6BImNXuzr6ZN/t2R7UN3fWIah1SZ1d3T2+or39gcGh4JDw6lnWtkqOJjGYZlpNXFVcYuikynu4ZIm87QimqhsipJxuNeK4sHFe3zG2vYou9olIw9SNdUzyG0pX92YNwlOLkW+SXs0jy6pIckQMkisCSVvgRuziEBQ0lFCFgwmPfgAKXxw5kEGzG9lBlzGFP9+MCZ+hnbomzBGcojJ7wWuDTToCafG7UdH22xrcYPB1mRhCjF7qnOj3TA73SZ8taVb9GQ0uFd7XJFfbByPlE+uNfVpF3D8c/rLaaPRxhxdeqs3bbRxqv0Jr88ulVPb2WilWn6ZbeWP8N1eiJX2CW37W7LZG6bqNHZS38Y9yg7y5EWjvZubi8FJ/fWogm1oNWhTCJKcxwP5aRwCaSyHD9Ai5wiSspJMWlRWm5mSp1BJxx/DEp8QVx95Cq</latexit>
y⇤
Random choice (e.g. random design) or
evenly spaced sampling (e.g. full factorial
design) can also work for lower dimensional
exploration.

/ 35
26
x
y
sample max
sample min
ML fits a function to minimize the average errors, and as a result, ML functions go
through the center (mean) of sample output values.
the currently known best
We would like to find X better than (hopefully) the currently known best.
Now we would like to use ML for the goal.
y⇤
goes through here
predictions will be groundless
when it goes beyond this area
unintended extrapolation
When ML is rightly fitted, the predicted values are
never larger than the known best, which is
inconsistent with the goal.

/ 35
26
x
x1
x2
y
sample max
sample min
ML fits a function to minimize the average errors, and as a result, ML functions go
through the center (mean) of sample output values.
the currently known best
We would like to find X better than (hopefully) the currently known best.
Now we would like to use ML for the goal.
y⇤
goes through here
predictions will be groundless
when it goes beyond this area
unintended extrapolation
When ML is rightly fitted, the predicted values are
never larger than the known best, which is
inconsistent with the goal.

/ 35
27
x
x1
x2
y
sample max
sample min
This is why we need a criterion taking uncertainty into consideration instead of direct
use of ML predicted values to guide exploration.
It’ll be nice to gather more
information around here
even though the mean is
not so high (the predictions
have a large variance)
<latexit sha1_base64="i+Yum6tkJlYfyXr5MYELEoyKzkM=">AAAChHichVFNLwNRFD3GV3222EhsGk3FQpo7rZZaSMPGUlGVIDIzHiadzkxmpk2q8QfYEgsrEgvxA/wAG3/Awk8QSxIbC3dmKmKBO3nzzjvvnvvOe1e1Dd31iJ7apPaOzq7uSE9vX//AYDQ2NLzuWjVHEyXNMixnQ1VcYeimKHm6Z4gN2xFKVTVEWa0s+vvlunBc3TLXvIYttqvKvqnv6ZriMVVs7MQSlErnc2nKx0OQzYVAzqfjcoqCSKAVy1bsDlvYhQUNNVQhYMJjbECBy98mZBBs5rbRZM5hpAf7AkfoZW2NswRnKMxW+L/Pq80Wa/Lar+kGao1PMXg4rIwjSY90Q6/0QLf0TB+/1moGNXwvDZ7VUCvsnejx6Or7v6oqzx4OvlV/evawh9nAq87e7YDxb6GF+vrh+evq3EqyOUFX9ML+L+mJ7vkGZv1Nuy6KlYs//KjshV+MG/TVhfjvYD2dknOpTHE6UVhotSqCMYxjkvsxgwKWsIwS1xc4wSnOpC5pSspI2TBVamtpRvAjpPlPiwOQTw==</latexit>
y

/ 35
27
x
x1
x2
y
sample max
sample min
This is why we need a criterion taking uncertainty into consideration instead of direct
use of ML predicted values to guide exploration.
y <latexit sha1_base64="Qqfga3YaBoUI0Yb7fQVzumPP1Sw=">AAAChHichVHLSsNAFD1GrbW+qm4EN8WiuJBy6xsXUnTjsg9rC1VKEqc1mCYhSQu1+AO6VVy4UnAhfoAf4MYfcNFPEJcKblx4mwZEi3rDZM6cuefOmbmKpWuOS9Tskrp7egN9wf7QwODQ8Eh4dGzHMau2KrKqqZt2XpEdoWuGyLqaq4u8ZQu5ougipxxutvZzNWE7mmlsu3VL7FXksqGVNFV2mUrVi+EoxciLSCeI+yAKP5Jm+B672IcJFVVUIGDAZaxDhsNfAXEQLOb20GDOZqR5+wLHCLG2ylmCM2RmD/lf5lXBZw1et2o6nlrlU3QeNisjmKYnuqVXeqQ7eqaPX2s1vBotL3WelbZWWMWRk4nM+7+qCs8uDr5Uf3p2UcKq51Vj75bHtG6htvW1o4vXzFp6ujFD1/TC/q+oSQ98A6P2pt6kRPryDz8Ke+EX4wbFf7ajE+zMx+LLMUotRhMbfquCmMQUZrkfK0hgC0lkub7AKc5wLgWkOWlBWmqnSl2+ZhzfQlr/BNcbj/U=</latexit>
y
<latexit sha1_base64="uHwbcp1I7cfAojOCGzVZ2+fOJ7M=">AAACh3ichVE7TsNAEH2Yb/glQINEExGBoAnjACGh4tNQQiCAFKLINhuwcGzLdiKFiAtQ0AaJCiQKxAE4AA0XoMgRECVINBSMnSBEEZjV7s6+mTf7dke1Dd31iBodUmdXd09vX6h/YHBoOBwZGd11rbKjiaxmGZazryquMHRTZD3dM8S+7QilpBpiTz1Z9+N7FeG4umXueFVb5EvKkakXdU3xfMieqc4WIjGKp1NJWkhEKU6USlCSnUWS03I6KjPiWwwt27QiDzjAISxoKKMEARMe+wYUuDxykEGwGcujxpjDnh7EBc7Qz9wyZwnOUBg94fWIT7kWavLZr+kGbI1vMXg6zIxiip7pjt7oie7phT7b1qoFNXwtVd7VJlfYhfD5+PbHv6wS7x6Of1h/avZQRCrQqrN2O0D8V2hNfuW0/ra9nJmqTdMNvbL+a2rQI7/ArLxrt1sic/WHHpW18I9xg767EG3v7CbicjI+v7UQW1lrtaoPE5jEDPdjCSvYwCayXP8YF6jjUgpJc1JSSjVTpY4WZwy/TFr9AnUXkR0=</latexit>
p(y)
<latexit sha1_base64="xAYAr5eatCRkt85fJM86k8bLQd0=">AAAChnichVE7TsNAEH2YXwi/AA0STUQEQhTWOCQhoYqgoSSEABI/2WYTLBzbsp1IIeICSLRQUIFEgTgAB6DhAhQcAVGCREPB2AlCFMCsdnf2zbzZtzuaYxqeT/TUIXV2dff0Rvqi/QODQ8OxkdF1z665uijptmm7m5rqCdOwRMk3fFNsOq5Qq5opNrTDpSC+UReuZ9jWmt9wxE5VrVhG2dBVn6FiY3d2L5YgOZfNUCoZJ5kom6QMO2lSckourjASWAJtW7Fjd9jGPmzoqKEKAQs++yZUeDy2oIDgMLaDJmMue0YYFzhGlLk1zhKcoTJ6yGuFT1tt1OJzUNML2TrfYvJ0mRnHFD3SDb3SA93SM338WqsZ1gi0NHjXWlzh7A2fjBff/2VVefdx8M36U7OPMrKhVoO1OyESvEJv8etH56/FhdWp5jRd0Qvrv6QnuucXWPU3/bogVi/+0KOxFv4xbtBXF+K/O+tJWcnIc4VUIr/YblUEE5jEDPdjHnksYwUlrl/BKc5wLkUkWUpL861UqaPNGcMPk/Kf1ZuQ2g==</latexit>
y⇤
<latexit sha1_base64="Cx8vtQI6wD6rFemnJtzP0VxR74M=">AAACp3ichVE9T9tQFD0Y2vJRSqALUheLiCphiK5TGhIkKlSWboSPhEh8RLbzQp9wbMt+ieRG7FX/QIdOrdQBIXWFnYU/wMBPqDqCxMLQaydVxQBc6/mdd9499533ruU7MlRElwPa4NCTp8+GR0bHno+/mEhNTlVDrx3YomJ7jhfULDMUjnRFRUnliJofCLNlOWLLOliJ97c6Igil526qyBe7LXPflU1pm4qpemqmnIneRXtz2aUd6ap6l+HhHsOminQ/E2X1RlRPpSlXKhZoPq9TjqiYpwKDt2SUjJJuMBNHGv0oe6kT7KABDzbaaEHAhWLswETI3zYMEHzmdtFlLmAkk32BQ4yyts1ZgjNMZg/4v8+r7T7r8jquGSZqm09xeASs1DFLF3REV3ROx/Sbbu+t1U1qxF4inq2eVvj1iS/TGzePqlo8K3z8r3rQs0ITxcSrZO9+wsS3sHv6zqevVxuL67Pd1/SD/rD/73RJZ3wDt3Nt/1wT698e8GOxF34xbtC/Luj3g2o+ZxRyb9bm08vv+60axivMIMP9WMAyPqCMCtf/jF84wamW1Va1qlbrpWoDfc1L3AnN/AuqB50Q</latexit>
P(y > y⇤
) =
Z 1
y⇤
p(y)dy
<latexit sha1_base64="svoh2wRjAJ9IgkLWLNY4QZ6NPX0=">AAACwnichVHLTttAFL24vMsjlA0Sm1EjKlpF0XWANEECIWilLnk0gBSHyHYmMOBX7UkkY/ID/AALVi3qouoH9AO6abetWPAJqEsqsWHRayeoYgFcazxnztxz58xcw7NEIBEvupQn3T29ff0Dg0+HhkdGU2PPNgO34Zu8ZLqW628besAt4fCSFNLi257Pdduw+JZxsBLvbzW5HwjXeS9Dj1dsfdcRdWHqkqhqakGzdblnGNHbVlnLhEzLHGkZFi6GO68qC5pwZDUi2NohWJchowSz5krmTYcvWS2sptKYLRbyOJtjmEUs5DBPYA7VolpkKjFxpKETq27qG2hQAxdMaIANHByQhC3QIaCvDCogeMRVICLOJySSfQ4tGCRtg7I4ZejEHtB/l1blDuvQOq4ZJGqTTrFo+KRkMIXn+AWv8Ad+xUu8ubdWlNSIvYQ0G20t96qjxxMb14+qbJol7P1XPehZQh0KiVdB3r2EiW9htvXNw5Orjfn1qegFfsI/5P8jXuB3uoHT/Gt+XuPrpw/4McgLvRg16LYL7H6wmcuq+ezM2mx6abnTqn6YhOcwTf14DUvwDlahRPXP4Cf8gt/KG2Vf+aAE7VSlq6MZhzuhHP0DArmnyA==</latexit>
E[ y | y > y⇤
] =
Z 1
y⇤
y · p(y)dy
Probability of
improvement (PI)
Expected
Improvement (EI)
Upper Conﬁdence
Bound (UCB)
Optimization objectives under uncertainty
(aka acquisition function)
It’ll be nice to gather more
information around here
even though the mean is
not so high (the predictions
have a large variance)
<latexit sha1_base64="i+Yum6tkJlYfyXr5MYELEoyKzkM=">AAAChHichVFNLwNRFD3GV3222EhsGk3FQpo7rZZaSMPGUlGVIDIzHiadzkxmpk2q8QfYEgsrEgvxA/wAG3/Awk8QSxIbC3dmKmKBO3nzzjvvnvvOe1e1Dd31iJ7apPaOzq7uSE9vX//AYDQ2NLzuWjVHEyXNMixnQ1VcYeimKHm6Z4gN2xFKVTVEWa0s+vvlunBc3TLXvIYttqvKvqnv6ZriMVVs7MQSlErnc2nKx0OQzYVAzqfjcoqCSKAVy1bsDlvYhQUNNVQhYMJjbECBy98mZBBs5rbRZM5hpAf7AkfoZW2NswRnKMxW+L/Pq80Wa/Lar+kGao1PMXg4rIwjSY90Q6/0QLf0TB+/1moGNXwvDZ7VUCvsnejx6Or7v6oqzx4OvlV/evawh9nAq87e7YDxb6GF+vrh+evq3EqyOUFX9ML+L+mJ7vkGZv1Nuy6KlYs//KjshV+MG/TVhfjvYD2dknOpTHE6UVhotSqCMYxjkvsxgwKWsIwS1xc4wSnOpC5pSspI2TBVamtpRvAjpPlPiwOQTw==</latexit>
y

/ 35
28
Identifying local peaks of EI of the ML model
Expected
improvement
Local peaks of EI would be nice
candidates having locally maximal EIs.
But they are not at given sample points, and
the following local search is designed.
multistart from given
sample points
adding small random
perturbation, and update
position when EI increases
stop when local
perturbation doesn’t
change the EI value
any more.
run clustering over ﬁnal
candidates, and suggest K
candidates having locally
maximal EI values.
Every time SWED is changed, the
corresponding composition is
estimated by our algorithm, and
then recalculate valid SWED from it.
Partly because tree ensemble
regression functions are locally
bumpy, this clustering is effective
x

/ 35
29
Explorative Search with SWED
SWED represent every element with respect to a given set of elemental descriptors.
So we can focus only on the selected elemental properties to explore catalysts.
74 elements
Compositional
(onehot-like)
Catalyst: Mg 83.46, Li 16.53
SWED-8
83.46 × 16.53 × 0.00 0.00
…
SWED-3
83.46 × 16.53 × 0.00 0.00
…
SWED-3 features: electronegativity, density, enthalpy of fusion
SWED-8 features: SWED-3 features + atomic weight, atomic radius, m.p., b.p., ionization enegy
Each user’s intention and focus for catalyst exploration can be design through the
elemental descriptor choice.
can control
speciﬁcity &
focus

/ 35
30
Our updated dataset
The original dataset:
1866 catalyst records
from 421 reports (1982 - 2009)
Mine, S.; Takao, M.; Yamaguchi, T.; Toyao, T.*; Maeno, Z.; Hakim Siddiki, S. M. A.; Takakusagi, S.; Shimizu, K.*; Takigawa, I.* ChemCatChem 2021.
https://doi.org/10.1002/cctc.202100495.
4559 catalyst records from 542 reports
The update dataset:
4559 catalyst records
from 542 reports (2010 - 2019)

/ 35
31
ML Predictions of C2 yields
1. Conventional: composition + condition
2. Proposed(Exploitative): composition + SWED + condition
3. Proposed(Explorative): SWED + condition w/ SWED→composition estimator
RFR (Random Forest); ETR (ExtraTrees); XGB (XGBoost)
SWED-3 features: electronegativity, density, enthalpy of fusion
SWED-8 features: SWED-3 features + atomic weight, atomic radius, m.p., b.p., ionization enegy

/ 35
32
Top 20 highest-EI candidates based on SWED-3
As appeared
not included in
the data
Fs, Se, Os, Bm
infrequent elements
also observed
though these are
toxic and impractical
but explorative
suggestions were able
be made

/ 35
33
Post analysis for models and suggested catalysts
1st: (1) Mn: 72.3 (2) Li: 27.7 2nd: (1) Sr:50.0 (2) Ce:45.0 (3) Yb:5.0
With SHAP, feature importance/permutation importance, dependency plot, etc.

/ 35
34
Today’s talk
Reactants
(Gas)
Catalysts
(Solid)
Nano-particle
surface
Adsorption
Diffusion
Dissociation
Recombination
Desorption
God made the bulk;

/ 35
35
Acknowledgements
Ken-ichi
SHIMIZU
Satoru
TAKAKUSAGI
Takashi
TOYAO
Zen
MAENO
Keisuke SUZUKI
Motoshi TAKAO
Shinya MINE
Taichi YAMAGUCHI
S. M. A. Hakim Siddiki

A machine-learning view on heterogeneous catalyst design and discovery

More Related Content

What's hot (20)

Similar to A machine-learning view on heterogeneous catalyst design and discovery (20)

More from Ichigaku Takigawa (20)

Recently uploaded (20)

A machine-learning view on heterogeneous catalyst design and discovery