Applications of Machine Learning for Materials Discovery at NREL
- 1. Applications of Machine Learning for Materials Discovery at NREL Caleb Phillips, Ph.D. Data Analysis and Visualization Computational Sciences Center National Renewable Energy Laboratory
- 2. NREL | 2 Modern “Full Stack” Materials Science Synthesis Characterization Computation Golden, Colorado
- 3. NREL | 3 Skeptics Allowed “I’ll admit it, there may be something to this ‘big data’ and ‘machine learning’ thing everyone keeps talking about.” - Anonymous cynic (2017) What changed? • Computational power • Deep Neural Networks • Cheap storage, big data • Increasing adoption/investment
- 4. NREL | 4 Setting Realistic Expectations Machine learning & Deep Learning Image source: Gartner.com, Aug. 2017
- 5. NREL | 5 Overview of the talk Compelling examples of materials-oriented machine learning at work at NREL: • Improving the throughput of experimentation: • Interpretation: Accelerate the data->knowledge path • Automation: Replace onerous manual tasks • Prediction: Predict properties not measured • Augmenting or replacing DFT simulations in candidate screening • Prediction: End-to-end deep learning on molecular and atomistic structures Will cover applications at a high level – talk to or email me for more info } Do work faster with more insight } Focus work, avoid some altogether
- 6. NREL | 6 But first, the Data The Materials Project http://materials.nrel.gov http://organiceletronics.nrel.gov http://htem.nrel.gov { { { Experimental Theoretical Both
- 7. NREL | 7 Example: Experimental Materials Discovery Taylor et al. Adv. Funct. Mater. 18, 3169 (2008) % In Conductivity of Annealed InZnO Goal: Make a PhD Thesis Amount of Analysis a Routine Activity Composition Structure Property Process Slide credit: John Perkins
- 8. NREL | 8 Application Driven (High Throughput) Materials Discovery at NREL Input: Theoretical calculations Combinatorial synthesis Spatially resolved characterization Output: Application driven optimization
- 9. NREL | 9 Application Driven (High Throughput) Materials Discovery at NREL Input: Theoretical calculations Combinatorial synthesis Spatially resolved characterization Output: Application driven optimization AI/ML Opportunities Improve fidelity Guided search Faster screening Automation Visualization Property Prediction{
- 10. NREL | 10 Initial motivating problem: Accelerate Slow Analysis Tasks 880 Unanalyzed XRD Patterns (Data) 1 Structure Phase Map (Knowledge) Slow
- 11. NREL | 11 Clustering by structure and composition Samples Results Extracted Data Set ~ 1000 XRD patterns Spectral Clustering NNLS Decomposition Apply Machine Learning to Determine Clusters in XRD Patterns Fast: ~ 30 seconds on a laptop Calculated XRD Patterns
- 12. NREL | 12 Automatic band gap calculation Goal: replace highly subjective manual Process with something scalable, automated, and (more) accurate. Combining experimental and theoretical data compare properties across a wide landscape of materials systems and synthesis conditions. Schwarting et al., Materials Discovery (2018)
- 13. NREL | 13 Application Driven (High Throughput) Materials Discovery at NREL Input: Theoretical calculations Combinatorial synthesis Spatially resolved characterization Output: Application driven optimization AI/ML Opportunities Improve fidelity Guided search Faster screening Automation Visualization Property Prediction{
- 14. NREL | 14 High throughput screening using computational results Constraints Molecule Generator Predictive (Machine Learning) Model Simulation on Supercomputer $$$ Results Database OR Best candidates All candidates (sequentially) Visualization & Analysis Materials Synthesis $$$$$ Measurement and Validation New Materials Theoretical Experimental Training on Past Results Phillips et al. CoDA (2016)
- 15. NREL | 15 Predict opto-electric properties of molecules Support Vector Regression (SVR) performance when predicting calculated band gap. Residual error is linear and normally distributed. Median error is effectively zero, RMSE is 0.25 EV or less for most scenarios. First try: learn using molecular descriptors (traditional feature engineering) 2 million candidates
- 16. NREL | 16 End-to-end Learning: Skip the feature extraction Image Recognition: Convolutional Neural Networks (CNNs) O Message Passing Blocks Node Recurrent Units Node Embedding Layer Graph Output Layer(s) Dense Regression Layers Predictions Input Graph (Molecule) Molecular Graphs: Message Passing Neural Networks (MPNNs)Gilmer et al., CoRR (2017) Key hypothesis: model can learn which features are important directly from structure.
- 17. NREL | 17 End-to-end Learning: Skip the feature extraction Duplicate 1 (DFT) MachineLearningPrediction 3-5x improvement over manually engineered features. Accuracy approaching repeated-measures accuracy of DFT. Gap 0.90 HOMO 1.05 LUMO 0.89 Spectral overlap 1.28 Polymer HOMO 1.24 Polymer LUMO 1.03 Polymer gap 1.19 Polymer optical LUMO 1.02 !"#$("&'ℎ)*+ ,+&-*)*.) !"#$(012 3456)'&7+8) St. John et al. https://arxiv.org/abs/1807.10363. (2018)
- 18. NREL | 18 Transfer learning and training set size St. John et al. https://arxiv.org/abs/1807.10363. (2018)
- 19. NREL | 19 End-to-end learning for crystalline materials Represent crystal structure as a graph to allow end-to-end learning. Kamdar. 2018. NREL/US DOE CSGF.
- 20. NREL | 20 Thanks to Many Collaborators (and many funding sources) Theory Stephan Lany Vladan Stevonvic Aaron Holder @ LBNL Gerd Ceder Kristin Persson Data Robert White Kristin Munch Peter Graf @ NIST Zachary Trautt Robert Hanisch Experiment Andriy Zakutayev John Perkins Philip Parilla David Ginley Bill Tumas Sebastian Siol Lauren Garten Elisabetta Arca Matthew Taylor @ NIST Martin Green Jae Hattrick-Simpers Nam Nguyen @ SLAC Apurva Mehta @ ANL Debbie Myers AI/ML Jacob Hinkle Marcus Schwarting Peter St. John @ Harvard Harshil Kamdar Slide credit: John Perkins
- 21. NREL | 21 Selected Publications Peter C. St. John, Caleb Phillips, Travis W. Kemper, A. Nolan Wilson, Michael F. Crowley, Mark R. Nimlos, Ross E. Larsen. Message-passing neural networks for high-throughput polymer screening. In submission. ArXiv preprint: https://arxiv.org/abs/1807.10363 Marcus Schwarting, Sebastian Siol, Kevin Talley, Andriy Zakutayev, Caleb Phillips. Automated algorithms for band gap analysis from optical absorption spectra. Materials Discovery, April 18, 2018. https://doi.org/10.1016/j.md.2018.04.003 Andriy Zakutayev, Nick Wunder, Marcus Schwarting, John Perkins, Robert White, Kristin Munch, William Tumas, and Caleb Phillips. An open experimental database for exploring inorganic materials. Nature. Scientific Data. April 3, 2018. https://www.nature.com/articles/sdata201853 Caleb Phillips, Ross Larson, Kristin Munch, Nikos Kopidakis. Guided Search for Organic Photovoltaic Materials Using Predictive Data Modeling. Conference on Data Analysis (CoDA) 2016. March 2-4, 2016. Santa Fe, New Mexico.
- 22. www.nrel.gov Thank you This work was authored by the National Renewable Energy Laboratory, operated by Alliance for Sustainable Energy, LLC, for the U.S. Department of Energy (DOE) under Contract No. DE-AC36-08GO28308. Funding provided by U.S. Department of Energy Office of Energy Efficiency and Renewable Energy. The views expressed in the article do not necessarily represent the views of the DOE or the U.S. Government. The U.S. Government retains and the publisher, by accepting the article for publication, acknowledges that the U.S. Government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this work, or allow others to do so, for U.S. Government purposes. caleb.phillips@nrel.gov
- 23. NREL | 23 Save work: predict not-measured properties • Electrical conductivity prediction using random forest model • Training variables: chemical composition, XRD peak count, deposition conditions • Training process: 10-fold cross-validation by withdrawing 25% sample libraries • Training set: 16K data points varying by 9-10 orders of magnitude Predicted vs Measured Conductivity Prediction accuracy for Conductivity Prediction accuracy of 1-2 orders of magnitude, reasonable for semiconductors Zakutayev et al. Scientific Data 5 180053 (2018)
- 24. NREL | 24 What’s in my database? tSne model can group 70K samples based on similarity of their chemical compositions t-distributed stochastic neighbor embedding (tSne) dimensionality reduction model Zakutayev et al. Scientific Data 5 180053 (2018)