Brains@Bay Meetup: The Effect of Sensorimotor Learning on the Learned Representations in Deep Neural Networks - Viviane Clay
0 likes371 views
- The representations learned through self-supervised exploration in an embodied agent encode meaningful information about concepts without being explicitly taught. - Concepts can be learned from very few examples (above chance accuracy with one example) by analyzing neuron correlations across examples. - This concept mapping approach does not require retraining or large labeled datasets, and has no problem with catastrophic forgetting, since it relies on the encoded representations.
Brains@Bay Meetup: The Effect of Sensorimotor Learning on the Learned Representations in Deep Neural Networks - Viviane Clay
- 1. RTG Computational Cognition RTG Computational Cognition Sensorimotor Learning in ANNs Viviane Clay The Effect of Sensorimotor Learning on the Learned Representations in Deep Neural Networks Viviane Clay
- 2. RTG Computational Cognition Differences in Computational Abilities Viviane Clay Current ANNs are vulnerable to adversarial attacks, have trouble to continually learn without catastrophic forgetting, extrapolate knowledge to new tasks, learn causal models, and learn efficiently from few labeled examples. (Hendrycks et al., 2019) (Goodfellow et. al., 2015)
- 3. RTG Computational Cognition Learning more like Children Viviane Clay I investigate the effect of supervision and embodiment as well as gradual knowledge acquisition on learning. How Humans Learn How Machines Learn Sensorimotor Preoperational Concrete Operational Formal Operational <2 years 2-7 years 7-11 years >11 years Piaget’s Stages of Cognitive Development
- 4. RTG Computational Cognition Learning more like Children Viviane Clay “Instead of trying to produce a program to simulate the adult mind, why not rater try to produce one which simulates the childs? If this were then subjected to an appropriate course of education one would obtain the adult brain.” – Alan Turing, 1950
- 5. RTG Computational Cognition Learning Through Interaction Viviane Clay The idea is to let an agent learn through interaction with the world and then take a look at what kind of representation is learned.
- 6. RTG Computational Cognition Learning Through Interaction Viviane Clay Sensorimotor learning takes place in a randomly generated 3D maze environment with interactable objects.
- 7. RTG Computational Cognition Experimental Conditions Viviane Clay Perception Reconstruction Action Perception Perception Classification 1. Representation Learning Embodied Agent Autoencoder Classifier Control Conditions
- 8. RTG Computational Cognition Analysis of the Learned Representations Viviane Clay
- 9. RTG Computational Cognition Analysis of the Learned Representations Viviane Clay The representations of the embodied agent are correlated with actions and action-relevant objects such as doors.
- 10. RTG Computational Cognition Analysis of the Learned Representations Viviane Clay
- 11. RTG Computational Cognition Analysis of the Learned Representations Viviane Clay The agent learns a sparse encoding of the high dimensional visual input. This is a very efficient and robust way to code information (Ahmad & Scheinkman, 2019) and can be observed in nature (Perez-Orive et al., 2002; Laurent, 2002; Young & Yamane, 1992; Brecht & Sakmann, 2002) Avg # active each frame # neurons used Active in more that 40% of frames Most active neuron active in 12.6 (4.9%) 182 (71%) 3 (1.2%) 68.3% 239 (93.4%) 239 (93.4%) 239 (93.4%) 100% 175.2 (68.5%) 256 (100%) 233 (91%) 100%
- 12. RTG Computational Cognition Analysis of the Learned Representations Viviane Clay When measuring the sparsity using the Gini index one can see that the embodied agent gets more and more sparse over time.
- 13. RTG Computational Cognition Part I - Summary Viviane Clay Next question: Can we build on these encodings of the world and use them for tasks such as few- shot object detection? Summary: • The representations learned through very weakly supervised or self- supervised exploration are structured and meaningful • They encode action-oriented information in very sparse activation patterns • The encodings learned through interaction differ significantly from the encodings learned without interaction
- 14. RTG Computational Cognition Fast Mapping Viviane Clay Children (and adults) can make an object-word association with as little as one example. This is called Fast Mapping. How Humans Learn How Machines Learn Sensorimotor Preoperational Concrete Operational Formal Operational <2 years 2-7 years 7-11 years >11 years Piaget’s Stages of Cognitive Development
- 15. RTG Computational Cognition Fast Concept Mapping Viviane Clay We want to transition from self-supervised learning through interaction to learning complex concepts from few examples.
- 16. RTG Computational Cognition Fast Concept Mapping Viviane Clay 1. Perception Reconstruction Action Perception Perception Classification
- 17. RTG Computational Cognition Fast Concept Mapping Viviane Clay 2.
- 18. RTG Computational Cognition Fast Concept Mapping Viviane Clay ?
- 19. RTG Computational Cognition Fast Concept Mapping Viviane Clay Take N examples of the concept
- 20. RTG Computational Cognition Fast Concept Mapping Viviane Clay Give them to the trained Agent and see how they are represented
- 21. RTG Computational Cognition Fast Concept Mapping Viviane Clay Look at which Neurons fire together in each Example
- 22. RTG Computational Cognition Fast Concept Mapping Viviane Clay Sum up the correlations
- 23. RTG Computational Cognition Fast Concept Mapping Viviane Clay Use the M strongest connections to define the concept
- 24. RTG Computational Cognition Fast Concept Mapping Viviane Clay The strength of a connection defines its weight in the concept
- 25. RTG Computational Cognition Fast Concept Mapping Viviane Clay Do this for any concept you like to extract
- 26. RTG Computational Cognition Fast Concept Mapping Viviane Clay Inference: Compare the activations of a new input to the concept definition
- 27. RTG Computational Cognition Fast Concept Mapping Viviane Clay Inference: Add up the amount of evidence for the concept and check if it is above a threshold Evidence: 0.66
- 28. RTG Computational Cognition Fast Concept Mapping Viviane Clay Inference: Do this for all concepts of interest Evidence: 0.66 Evidence >= TH Evidence < TH
- 29. RTG Computational Cognition Fast Concept Mapping Viviane Clay It works with surprisingly few examples!
- 30. RTG Computational Cognition Fast Concept Mapping Viviane Clay About equally well as with the classifier representations, which were optimized for exactly these concepts.
- 31. RTG Computational Cognition Fast Concept Mapping Viviane Clay Why? The idea is that in a good encoder few examples of a class should already be representative of the overall pattern used to encode this class.
- 32. RTG Computational Cognition Fast Concept Mapping Viviane Clay FCM works better on some concepts than on others. It can show which information is encoded well in the networks.
- 33. RTG Computational Cognition Fast Concept Mapping Viviane Clay You can add as many concepts as you want! If the concept is encoded in the representation, it can be extracted without retraining or huge amounts of labeled data.
- 34. RTG Computational Cognition Part II - Summary Viviane Clay Making to computational aspects of learning more natural can lead to more brain-like abilities. Summary: • The representations learned through very weakly supervised or self-supervised exploration encode information about concepts which were never explicitly taught • These concepts can be learned with very few examples (above chance accuracy already with one example) • Concept mapping has no problem with catastrophic forgetting • There is no need for a big labeled data set (there often is in few-shot approaches) • Learning more naturally (embodied & self/weakly-supervised) leads to abilities that we also find in humans such as fast-mapping
- 35. RTG Computational Cognition Takeaways Viviane Clay 1) Our brains are sensorimotor systems, and if we want to model them, our models should be as well. 2) What a model learns is a reflection of its environment and task. 3) If we want to understand cognition and our brains, we need to understand the tasks that need to be solved. If we want our artificial models to learn brain-like representations, we need to let them learn tasks that brains need to solve.
- 36. RTG Computational Cognition Embodied AI Viviane Clay Thanks for your attention! Any questions? Ideas?
- 37. RTG Computational Cognition Publications Viviane Clay https://doi.org/10.1016/j.neunet.2020.11.004 https://arxiv.org/abs/2102.02153 (currently under review @ journal)
- 38. RTG Computational Cognition Differences in Information Processing Viviane Clay “An algorithm is likely to be understood more readily by understanding the nature of the problem being solved than by examining the mechanism (and the hardware) in which it is embodied. In a similar vein, trying to understand perception by studying only neurons is like trying to understand bird flight by studying only feathers: It just cannot be done. In order to understand bird flight, we have to understand aerodynamics; only then do the structure of feathers and the different shapes of birds’ wings make sense.” (Marr,2010, Chapter1, p.27)
- 39. RTG Computational Cognition Differences in Information Processing Viviane Clay
- 40. RTG Computational Cognition Looking for Inspiration Viviane Clay In nature certain solution have developed independently in different species which means they are at least local optima. Using solutions for information processing that evolution came up with does not seem like a bad start. © Arizona Board of Regents / ASU Ask A Biologist
- 41. RTG Computational Cognition The Importance of Embodiment Viviane Clay Self-generated movements with concurrent visual feedback seem necessary in order to display visually guided behavior and to perform tasks that require depth perception.
- 42. RTG Computational Cognition Emerging Sparsity and its Mysterious Causes Viviane Clay Observations Action Encoding Encoded State Encoded Next State Predicted Next State Predicted Next Action Forward Loss Inverse Loss Encoding Encoded State Observations Encoding Encoded State Decoding Reconstruction Reconstruction Error Observations Encoding Encoded State Error Object Classification True Label Encoding Encoded State Encoding Encoded State Encoded State
- 43. RTG Computational Cognition Test Concepts & Data Distribution No door Level door Green door Key door Other door Nothing 150 200 150 150 200 850 Key 150 0 50 50 0 250 Orb 50 50 50 50 50 250 350 250 250 250 250
- 44. RTG Computational Cognition My PhD Project – A Birds Eye View Viviane Clay 0 1 0 0 0 0 21 10 8 No Key 1 Key 2 Keys 3 Keys 4 Keys 5 Keys Time Left Level Walk Turn Body Jump Turn Camera Observations Actions (168x168x3)
- 45. RTG Computational Cognition Analysis of the Learned Representations Viviane Clay Agents with all types of rewards have a very action-oriented encoding of the sensory input.
- 46. RTG Computational Cognition Fast Concept Mapping from Embodied Representations Viviane Clay Concept logic: This can improve performance. The classifier already implicitly makes use of this. OR AND AND
- 47. RTG Computational Cognition Bring the Concepts Back into the World Idea: Add extracted concepts as additional information for decision making. This will also refine the activations to match the concepts (hopefully).
- 48. RTG Computational Cognition Conventional RL (simplified) Viviane Clay Rewards from the environment are used to optimize the policy. Observation φ Representation Action Value Estimate Environment Rewards 𝜋 Actor (policy gradient) & Critic (value function) Loss
- 49. RTG Computational Cognition Exploration by Curiosity Viviane Clay Can learn without external rewards by using the prediction error as internal reward. Observation φ Representation Action Value Estimate Environment 𝜋 Φ’ Next Observation Forward Model Inverse Model Predicted Action Predicted Φ(next obs) Inverse Loss Internal Reward Actor & Critic Loss Next Observation Forward Loss (Pathak et. al., 2017)
- 50. RTG Computational Cognition Exploration by Disagreement Viviane Clay The inverse loss can be seen as an auxiliary task for feature learning. Φ’ can also be random features or learned with other loss like VAE. Observation φ Representation Action Value Estimate Environment 𝜋 Φ’ Next Observation Forward Model Inverse Model Predicted Action Predicted Φ(next obs) Inverse Loss Forward Loss Internal Reward (Pathak et. al., 2019)
- 51. RTG Computational Cognition Exploration by Disagreement Viviane Clay The inverse loss can be seen as an auxiliary task for feature learning. Φ’ can also be random features or learned with other loss like VAE. Observation φ Representation Action Value Estimate Environment 𝜋 Φ’ Next Observation Forward Model Auxiliary Task/Loss Predicted Φ(next obs) Forward Loss Internal Reward (Pathak et. al., 2019)
- 52. RTG Computational Cognition Exploration by Disagreement Viviane Clay We can train an ensemble of forward models. Observation φ Representation Action Value Estimate Environment 𝜋 Φ’ Next Observation Forward Model Auxiliary Task/Loss Predicted Φ(next obs) Forward Loss Forward Model Predicted Φ(next obs) Forward Model Predicted Φ(next obs) Forward Model Predicted Φ(next obs) (Pathak et. al., 2019)
- 53. RTG Computational Cognition Exploration by Disagreement Viviane Clay As an intrinsic reward we can then take the disagreement between models. Observation φ Representation Action Value Estimate Environment 𝜋 Φ’ Next Observation Forward Model Auxiliary Task/Loss Predicted Φ(next obs) Forward Loss Forward Model Predicted Φ(next obs) Forward Model Predicted Φ(next obs) Forward Model Predicted Φ(next obs) Internal Reward Variance/ Disagreement = (Pathak et. al., 2019)
- 54. RTG Computational Cognition My PhD Project – A Birds Eye View Viviane Clay The agent shows more spike like activations while AE and C are more continuously active with changes in activation magnitude.
- 55. RTG Computational Cognition Fast Concept Mapping from Embodied Representations Viviane Clay
- 56. RTG Computational Cognition Comparison Between Agent Representations – Consistent Threshold and Pattern Complexity
- 57. RTG Computational Cognition How well can Concepts be Extracted using Single Neurons or Triplets? TH: 20%
- 58. RTG Computational Cognition How does this Compare to an SVM? TH: 20% Fitting a support vector machine on 5 positive and 5 negative examples (only positive is not possible here) does not lead to increased accuracy.
- 59. RTG Computational Cognition Fast Concept Mapping from Embodied Representations Viviane Clay 5 Examples
- 60. RTG Computational Cognition Fast Concept Mapping from Embodied Representations Viviane Clay 250 Examples