AI that/for matters
2 likes630 views
This document provides an overview of recent advances in applying artificial intelligence and machine learning techniques to matters and materials. It discusses several key ideas and approaches, including: - Using graph neural networks and message passing algorithms to model molecules as graphs and predict molecular properties. - Generative models like variational autoencoders and generative adversarial networks to represent molecules in a continuous latent space and generate new molecular structures. - Reinforcement learning approaches for predicting chemical reactions and planning chemical syntheses. - Directed generation of molecular graphs using graph variational autoencoders to overcome limitations of string-based representations. The document outlines many promising directions for using deep learning to tackle important problems in chemistry, materials science
AI that/for matters
- 1. Source: rdn consulting AI FOR MATTERS Molecules and Materials 23/01/2019 1 Hanoi, Jan 2019 Truyen Tran Deakin University @truyenoz truyentran.github.io truyen.tran@deakin.edu.au letdataspeak.blogspot.com goo.gl/3jJ1O0
- 2. 23/01/2019 2
- 3. 23/01/2019 3 Reconstructs particle tracks from 3D points left in the silicon detectors. https://www.kaggle.com/c/trackml-particle-identification Agrawal, A., & Choudhary, A. (2016). Perspective: Materials informatics and big data: Realization of the “fourth paradigm” of science in materials science. Apl Materials, 4(5), 053208. Jim Gray, Turing Award 1998 (1944-2007) Honoured as father of The 4th Paradigm
- 4. AI Research: dream 23/01/2019 4 tendirectionszen.org Among the most challenging scientific questions of our time are the corresponding analytic and synthetic problems: • How does the brain function? • Can we design a machine which will simulate a brain? -- Automata Studies, 1956.
- 5. AI Research: reality 23/01/2019 5 Vietnam News ICLR’19
- 6. What makes AI? Perceiving Learning Reasoning Planning 23/01/2019 6 Acting Robotics Communicating Consciousness Automated discovery Modern AI is mostly data-driven, as opposed to classic AI, which is mostly expert-driven.
- 7. What you can’t design, learn! (aka variational method) Filling the slot In-domain (intrapolation), e.g., an alloy with a given set of characteristics Out-domain (extrapolation), e.g., weather/stock forecasting Classsification, recognition, identification Action, e.g., driving Mapping space, e.g., translation Replacing expensive simulations 23/01/2019 7 Estimating semantics, e.g., concept/relation embedding Assisting experiment designs Finding unknown, causal relation, e.g., disease-gene Predicting experiment results, e.g., alloys phase diagrams material characteristics
- 8. Machine learning settings Supervised learning (mostly machine) A B Unsupervised learning (mostly human) Will be quickly solved for “easy” problems (Andrew Ng) 23/01/2019 8 Anywhere in between: semi- supervised learning, reinforcement learning, lifelong learning, meta- learning, few-shot learning, knowledge-based ML
- 9. Idea: Over-represent and select $3M Prize, 3 years 170K patients, 4 years worth of data Predict length-of-stay next year Not deep learning yet (early 2013), but strong ensemble needed suggesting dropout/batch-norm 9 This is me!
- 10. Idea: Filter stacking Integrate-and-fire neuron andreykurenkov.com Feature detector Block representation23/01/2019 10
- 11. Idea: Repeated refinement Classification Image captioning Sentence classification Neural machine translation Sequence labelling Source: http://karpathy.github.io/assets/rnn/diags.jpeg 23/01/2019 11
- 12. Idea: Convolution & composition adeshpande3.github.io 23/01/2019 12
- 13. Galaxy Zoo challenge: Categorization of galaxy images 23/01/2019 13
- 14. Idea: Attention 23/01/2019 14 Show, Attend and Tell: Neural Image Caption Generation with Visual Attention, K. Xu , J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhutdinov, R. Zemel, Y. Bengio
- 15. Idea: Message passing 23/01/2019 15 Thin column Relation graph Stacked learning Column nets#REF: Pham, Trang, et al. "Column Networks for Collective Classification." AAAI. 2017.
- 16. Generative models 23/01/2019 16 Many applications: • Text to speech • Simulate data that are hard to obtain/share in real life (e.g., healthcare) • Generate meaningful sentences conditioned on some input (foreign language, image, video) • Semi-supervised learning • Planning
- 17. Variational Autoencoder (Kingma & Welling, 2014) Two separate processes: generative (hidden visible) versus recognition (visible hidden) http://kvfrans.com/variational-autoencoders-explained/ Gaussian hidden variables Data Generative net Recognising net
- 18. Generative adversarial networks (Adapted from Goodfellow’s, NIPS 2014) 23/01/2019 18
- 19. Progressive GAN: Generated images 23/01/2019 19 Karras, T., Aila, T., Laine, S., & Lehtinen, J. (2017). Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196.
- 20. Deep learning vs electronics Neuron as feature detector SENSOR, FILTER Multiplicative gates AND gate, Transistor, Resistor Attention mechanism SWITCH gate Memory + forgetting Capacitor + leakage Skip-connection Short circuit Computational graph Circuit Compositionality Modular design 23/01/2019 20
- 21. AI for molecules 23/01/2019 21 https://pubs.acs.org/doi/full/10.1021/acscentsci.7b00550 • Represent atom/molecular space • Predict molecular properties • Estimate chem-chem interaction • Predict chemical reaction • Fast search for new molecules • Plan chemical synthesis
- 22. Molecular properties prediction Traditional techniques: Graph kernels (ML) Molecular fingerprints (Chemistry) Modern techniques Molecule as graph: atoms as nodes, chemical bonds as edges 23/01/2019 22 #REF: Penmatsa, Aravind, Kevin H. Wang, and Eric Gouaux. "X- ray structure of dopamine transporter elucidates antidepressant mechanism." Nature 503.7474 (2013): 85-90.
- 23. Graph memory networks (GMN) 23/01/2019 23 Controller … Memory Drug molecule Task/query Bioactivities 𝒚𝒚 query #Ref: Pham, Trang, Truyen Tran, and Svetha Venkatesh. "Graph Memory Networks for Molecular Activity Prediction." ICPR’18. Message passing as refining atom representation
- 24. 23/01/2019 24 GMN on molecular bioactivities
- 25. Approximating DFT DFT = Density Functional Theory Gilmer, Justin, et al. "Neural message passing for quantum chemistry." arXiv preprint arXiv:1704.01212 (2017). 23/01/2019 25 𝒚𝒚
- 26. Predict multiple properties 23/01/2019 26 #REF: Do, Kien, et al. "Attentional Multilabel Learning over Graphs-A message passing approach." Machine Learning, 2019.
- 27. #REF: Do, Kien, et al. "Attentional Multilabel Learning over Graphs-A message passing approach." Machine Learning 2019.
- 28. Chemical-chemical interaction via Relational Dynamic Memory Networks 23/01/2019 28 𝑴𝑴1 … 𝑴𝑴𝐶𝐶 𝒓𝒓𝑡𝑡 1 …𝒓𝒓𝑡𝑡 𝐾𝐾 𝒓𝒓𝑡𝑡 ∗ Controller Write𝒉𝒉𝑡𝑡 Memor y Graph Query Output Read heads #REF: Pham, Trang, Truyen Tran, and Svetha Venkatesh. "Relational dynamic memory networks." arXiv preprint arXiv:1808.04247(2018).
- 29. Results on STITCH database 23/01/2019 29
- 30. Chemical reaction product prediction as reinforcement learning Input: Molecules Output: Molecules Model: Graph morphism Method: Graph transformation policy network (GTPN) 23/01/2019 30 Do, Kien, Truyen Tran, and Svetha Venkatesh. "Graph Transformation Policy Network for Chemical Reaction Prediction." arXiv preprint arXiv:1812.09441 (2018).
- 32. Molecular generation: Case of drug The space of drugs is estimated to be 1e+23 to 1e+60 Only 1e+8 substances synthesized thus far. It is impossible to model this space fully. The current technologies are not mature for graph generations. But approximate techniques do exist. 23/01/2019 32 Source: pharmafactz.com
- 33. Combinatorial chemistry Generate variations on a template Returns a list of molecules from this template that Bind to the pocket with good pharmacodynamics? Have good pharmacokinetics? Are synthetically accessible? 23/01/2019 33 #REF: Talk by Chloé-Agathe Azencott titled “Machine learning for therapeutic research”, 12/10/2017
- 34. AI approach to molecule design Representing molecules using fingerprints Representing graph as string, and use sequence VAEs or GANs. Graph VAE & GAN Model nodes & interactions Model cliques Sequences Iterative methods Reinforcement learning Discrete objectives Any combination of these + memory.
- 35. Molecule fingerprints Input of the encoder : the fingerprint of a molecule The decoder outputs the predicted fingerprint . The generative model generates a vector E, which is then discriminated from the latent vector of the real molecule by the discriminator. Kadurin, Artur, et al. "The cornucopia of meaningful leads: Applying deep adversarial autoencoders for new molecule development in oncology." Oncotarget 8.7 (2017): 10883.
- 36. Source: wikipedia.org Molecule string Using SMILES representation of molecule, to convert a molecular graph into a string SMILES = Simplified Molecular-Input Line-Entry System Then using sequence-to-sequence + VAE/GAN to model the continuous space that encodes/decodes SMILES strings Allow easy optimization on the continuous space 23/01/2019 36 #REF: Gómez-Bombarelli, Rafael, et al. "Automatic chemical design using a data-driven continuous representation of molecules." arXiv preprint arXiv:1610.02415 (2016).
- 37. VAE for molecular space modelling 23/01/2019 37 Uses VAE for sequence-to-sequence. Gómez-Bombarelli, Rafael, et al. "Automatic chemical design using a data-driven continuous representation of molecules." ACS Central Science (2016).
- 38. Source: wikipedia.org Drawbacks of string representation String graphs is not unique! Lots of string are invalid Precise 3D information is lost Short range in graph may become long range in string A better way is to encode/decode graph directly. 23/01/2019 38 #REF: Gómez-Bombarelli, Rafael, et al. "Automatic chemical design using a data-driven continuous representation of molecules." arXiv preprint arXiv:1610.02415 (2016).
- 39. Better approach: Generating molecular graphs directly No regular, fixed-size structures Graphs are permutation invariant: #permutations are exponential function of #nodes The probability of a generated graph G need to be marginalized over all possible permutations Multiple objectives: Diversity of generated graphs Smoothness of latent space Agreement with or optimization of multiple design objectives
- 40. GraphVAE Handles irregular structures Predict the whole adjacency matrix, node types and edge types Deals with variable size graph Bounded by the size of the largest graph in training data. Handles permutation invariance Matching every pair of nodes in 2 graphs Partially promotes diversity 23/01/2019 40 #REF: Simonovsky, M., & Komodakis, N. (2018). GraphVAE: Towards Generation of Small Graphs Using Variational Autoencoders. arXiv preprint arXiv:1802.03480.
- 41. 23/01/2019 41 Adjacency matrix Edge types Node types k>n The graph size are bounded Latent vector for whole graph #REF: Simonovsky, M., & Komodakis, N. (2018). GraphVAE: Towards Generation of Small Graphs Using Variational Autoencoders. arXiv preprint arXiv:1802.03480.
- 42. Junction tree VAE 23/01/2019 42 Jin, W., Barzilay, R., & Jaakkola, T. (2018). Junction Tree Variational Autoencoder for Molecular Graph Generation. ICML’18. Junction tree is a way to build a “thick-tree” out of a graph Cluster vocab: rings bonds atoms
- 43. 23/01/2019 43 Jin, W., Barzilay, R., & Jaakkola, T. (2018). Junction Tree Variational Autoencoder for Molecular Graph Generation. ICML’18.
- 44. AI for materials 23/01/2019 44 • Characterise the space of materials • Represent crystals • Map alloy composition phase diagram • Inverse design: Map phase diagram alloy composition • Generate alloys • Optimize processing parameters Agrawal, A., & Choudhary, A. (2016). Perspective: Materials informatics and big data: Realization of the “fourth paradigm” of science in materials science. Apl Materials, 4(5), 053208.
- 45. Bayesian optimization for short polymer fiber 23/01/2019 45 Li, Cheng, David Rubín de Celis Leal, Santu Rana, Sunil Gupta, Alessandra Sutti, Stewart Greenhill, Teo Slezak, Murray Height, and Svetha Venkatesh. "Rapid Bayesian optimisation for synthesis of short polymer fiber materials." Scientific reports 7, no. 1 (2017): 5683.
- 46. Crystal as a graph neural net 23/01/2019 46 Xie, Tian, and Jeffrey C. Grossman. "Crystal Graph Convolutional Neural Networks for an Accurate and Interpretable Prediction of Material Properties." Physical review letters 120.14 (2018): 145301.
- 47. Computing materials similarity 23/01/2019 47 Strong features are highly but nonlinearly predictive of properties (e.g., formation energy) The relationship between features & properties can be locally linear Materials that share the same feature- property relationship may be functionally similar
- 48. Alloy space exploration ● Scientific innovations are expensive ● One search per specific target ● Availability of growing data #REF: Nguyen, P., Tran, T., Gupta, S., Rana, S., & Venkatesh, S. Incomplete Conditional Density Estimation for Fast Materials Discovery. SDM’19
- 49. Inverse design ● Leverage the existing data and query the simulators in an offline mode ● Avoid the global optimization by learning the inverse design function f -1(y) ● Predict design variables in a single step
- 51. Full phase specification 50% phase missing
- 53. Recent context NIPS Workshop on Molecular and Materials Sciences (2017, 2018) The International Workshop on Machine Learning for Materials Science 2018 2018 Workshop on Machine Learning in Materials Science (April 2018) AI-driven Acceleration Platform for Materials (Jan 2018, CIFAR Report) Workshop on Deep Learning for Physical Sciences (Dec 2017) Machine learning for materials research: Bootcamp & workshop (June 2017) International Workshop on Machine Learning for Materials Science (March 2017, Finland) Understanding Many-Particle Systems with Machine Learning (Sept-Dec 2016) 23/01/2019 53
- 55. Machine learning a physics theory? Expressiveness Can represent the complexity of the world Can compute anything computable Learnability Have mechanism to learn from the training signals Generalizability Work on unseen data 23/01/2019 55
- 56. AI as physics Intelligence as self-organizing phenomena: reducing ignorance/entropy Neural networks as a statistical mechanical system Learning as variational optimization Inference in probabilistic graphical models as Bethe free-energy minimization Phase transition may occur in AI systems Ultimate AI must solve the consciousness problem, which may require quantum physics (or a new physics) 23/01/2019 56 Life has low entropy
- 57. Physics of Restricted Boltzmann machines Boltzmann machines as a generalization of Ising model Restricted Boltzmann machine as a simplified version, but with hidden variables to denote underlying unobserved processes Stack of RBMs is akin to renormalization trick in physics 23/01/2019 57 energy Restricted Boltzmann Machine (~1994, 2001)
- 58. RBM for n-body problem Carleo, Giuseppe, and Matthias Troyer. "Solving the quantum many-body problem with artificial neural networks." Science355.6325 (2017): 602-606. Torlai, Giacomo, et al. "Many-body quantum state tomography with neural networks." arXiv preprint arXiv:1703.05334 (2017). Nomura, Yusuke, et al. "Restricted Boltzmann machine learning for solving strongly correlated quantum systems." Physical Review B 96.20 (2017): 205152. Gao, Xun, and Lu-Ming Duan. "Efficient representation of quantum many-body states with deep neural networks." Nature communications 8.1 (2017): 662. Chen, Jing, et al. "On the equivalence of restricted Boltzmann machines and tensor network states." arXiv preprint arXiv:1701.04831 (2017). Rao, Wen-Jia, et al. "Identifying product order with restricted Boltzmann machines." Physical Review B 97.9 (2018): 094207. 23/01/2019 58
- 59. Quantum machine learning 23/01/2019 59 Quantum Machine Learning, Jacob Biamonte, Peter Wittek, Nicola Pancotti, Patrick Rebentrost, Nathan Wiebe and Seth Lloyd, Nature 549, 195–202 14 September 2017 Read more at: https://phys.org/news/2017-09- quantum-machines.html#jCp
- 61. Prediction versus understanding We can predict well without understanding (e.g., planet/star motion Newton). Guessing the God’s many complex behaviours versus knowing his few universal laws. 23/01/2019 61
- 62. Some open challenges Can AI/ML discover a new phase of matter ? Would AI/ML discover new algorithms for us ? Would it be possible for us to make progress on fermion sign problem? Non-stochastic, or better ways for optimizing, renormalizing, and evolving Neural Networks. Information pattern aware structure learning of neural networks 23/01/2019 62 #REF: Liu, Jin-Guo, Shuo-Hui Li, and Lei Wang. "Lecture Note on Deep Learning and Quantum Many-Body Computation." (2018).