(deep) reinforcement learning - CAB420
4 likes656 views
The document provides an overview of Juxi Leitner's work in robotic vision and reinforcement learning at the Queensland University of Technology, including his research on intelligent agents and robotic systems. It outlines various methods for training robots using reinforcement learning and introduces concepts such as policies, rewards, and state representations in Markov decision processes. Additionally, it discusses challenges and applications of reinforcement learning in robotics, highlighting the importance of understanding the limitations of current systems and the potential for integrating deep learning techniques.
![understanding limita,ons of deep nets,
reinforcement learning and transfer of knowledge
deep learning visual control
[Tow et al, ACRA 2015]](https://image.slidesharecdn.com/2016-reinforcementlearninglecture-170426063517/85/deep-reinforcement-learning-CAB420-33-320.jpg)
![http://roboticvision.org/
[Zhang et al, ACRA 2015]
deep learning visual control](https://image.slidesharecdn.com/2016-reinforcementlearninglecture-170426063517/85/deep-reinforcement-learning-CAB420-34-320.jpg)
![http://roboticvision.org/
[Zhang et al, arxiv.org]
deep learning visual control
understanding limita,ons of deep nets,
reinforcement learning and transfer of knowledge](https://image.slidesharecdn.com/2016-reinforcementlearninglecture-170426063517/85/deep-reinforcement-learning-CAB420-35-320.jpg)
![deep learning visual servoing
Perception Module Control Module
Conv1 Conv2 Conv3 FC_c2 FC_c3FC_c1
Q-values
7×7conv+ReLU
stride2
4×4conv+ReLU
stride2
3×3conv+ReLU
stride1
64 lters 64 lters 64 lters
fullyconn.
300units
9units
84×84
400units
fullyconn.
fullyconn.+ReLU
fullyconn.+ReLU
I BN
5units
θ
Bottleneck
Or
Occlusion
A B C ED
Occlusion Occlusion
Occlusion
[Zhang et al, arxiv.org]
understanding limita,ons of deep nets,
reinforcement learning and transfer of knowledge](https://image.slidesharecdn.com/2016-reinforcementlearninglecture-170426063517/85/deep-reinforcement-learning-CAB420-37-320.jpg)
(deep) reinforcement learning - CAB420
- 1. Juxi Leitner arc centre of excellence for robotic vision queensland university of technology <j.leitner@qut.edu.au> http://Juxi.net reinforcement learning Juxi Guest Lecture, CAB420 Machine Learning (deep)
- 3. Dalle Molle Institute for AI (IDSIA) Work Juxi Leitner PhD Informatics / Intelligent Systems MSc Space Robotics & Automation BSc Information & Software Engineering Intelligent (Space) Robots European Space Agency (ESA) Erasmus Intelligent Systems Work (Humanoid) Robot Vision Instituto Superior Técnico (IST) Mobility Intelligent Space Systems Laboratory About Me Current Robotic Vision and Actions Queensland University of Technology (QUT) arc centre of excellence for robotic vision | qut juxi.net | roboticvision.org | bne-robotics.net | brisbane.ai
- 5. http://roboticvision.org/ create agent that see and interact with the world http://Juxi.net/aboutme
- 8. BRISBANE.AI defining AI study of "intelligent agents”: any device that perceives its environment and takes actions that maximize its chance of success at some goal
- 10. http://roboticvision.org/ IM-CLeVeR Teaser https://www.youtube.com/watch?v=OyfonCDxUiU full video: https://vimeo.com/51011081
- 16. http://roboticvision.org/ robot RL Konidaris et al. Autonomous Skill Acquisition on a Mobile Manipulator https://www.youtube.com/watch?v=yUICAkSQTZY
- 17. http://roboticvision.org/ Jan Peters et al. Motor Skill Learning from Demonstration robot RL https://www.youtube.com/watch?v=qtqubguikMk
- 18. http://roboticvision.org/ foundations a policy, a reward signal, a value func,on, and, op,onally, a model of the environment http://cs.stanford.edu/people/karpathy/reinforcejs/http://karpathy.github.io/2016/05/31/rl/
- 19. http://roboticvision.org/ policy Policy defines the actions to be taken per state
- 20. http://roboticvision.org/ value function Value function is a prediction of future reward per state
- 21. http://roboticvision.org/ mdp An information state (a.k.a. Markov state) contains all useful information from the history. i.e. the state is a sufficient statistic of the future pomdp what if: robot with camera vision isn’t told its absolute location agent state != environment state Formally this is a partially observable Markov decision process (POMDP)
- 23. http://roboticvision.org/ Example 3.2: Pick-and-Place Robot Consider using reinforcement learning to control the mo4on of a robot arm in a repe44ve pick-and-place task. If we want to learn movements that are fast and smooth, the learning agent will have to control the motors directly and have low-latency informa4on about the current posi4ons and veloci4es of the mechanical linkages. The ac4ons in this case might be the voltages applied to each motor at each joint, and the states might be the latest readings of joint angles and veloci4es. The reward might be +1 for each object successfully picked up and placed. To encourage smooth movements, on each 4me step a small, nega4ve reward can be given as a func4on of the moment- to-moment “jerkiness” of the mo4on.
- 28. http://roboticvision.org/ on/off policy Q-learning (Watkins, 1989), h7ps://studywolf.wordpress.com/2013/07/01/reinforcement-learning-sarsa-vs-q-learning/
- 29. BRISBANE.AI deephype Dartmouth1956 Initial Hype AI Winter AI Winter Expert Systems Hype AI Spring Deep Learning Hype
- 30. http://roboticvision.org/ deep RL Sergey Levine / Google Brain DeepMind QUT
- 31. http://roboticvision.org/ deep RL DeepMind idea: learn a policy to play a computer game using only visual information state: ? actions: ? reward: ?
- 32. http://roboticvision.org/ deep RL DeepMind idea: learn a policy to play a computer game using only visual information state: ? actions: ? reward: ? game screen button press game score problems?
- 33. understanding limita,ons of deep nets, reinforcement learning and transfer of knowledge deep learning visual control [Tow et al, ACRA 2015]
- 34. http://roboticvision.org/ [Zhang et al, ACRA 2015] deep learning visual control
- 35. http://roboticvision.org/ [Zhang et al, arxiv.org] deep learning visual control understanding limita,ons of deep nets, reinforcement learning and transfer of knowledge
- 36. http://roboticvision.org/ transfer visual control from simulation to real world reward issue percep,on issue noise issue explora,on issue
- 37. deep learning visual servoing Perception Module Control Module Conv1 Conv2 Conv3 FC_c2 FC_c3FC_c1 Q-values 7×7conv+ReLU stride2 4×4conv+ReLU stride2 3×3conv+ReLU stride1 64 lters 64 lters 64 lters fullyconn. 300units 9units 84×84 400units fullyconn. fullyconn.+ReLU fullyconn.+ReLU I BN 5units θ Bottleneck Or Occlusion A B C ED Occlusion Occlusion Occlusion [Zhang et al, arxiv.org] understanding limita,ons of deep nets, reinforcement learning and transfer of knowledge
- 38. ARC Centre of Excellence for Robotic Vision roboticvision.org limita,ons of current robo,c systems reproducible research on TASKS not datasets picking benchmark http://Juxi.net/dataset/acrv-picking-benchmark/ https://arxiv.org/abs/1609.05258
- 39. http://roboticvision.org/ deep RL Sergey Levine, Peter Pastor et al. (Google Brain)
- 40. http://roboticvision.org/ artificial curiosity Reward the reward-optimizing controller for actions yielding data that cause improvements of the adaptive predictor or data compressor! idea: similar: intrinsic motivation
- 43. http://roboticvision.org/ Is NeuroEvolution coming back? some recent papers: "Evolving Deep Neural Networks" by Miikkulainen et al (Sentient Technologies) "Large-scale Evolution of Image Classifiers" by Real et al (Google Brain) "PathNet: Evolution Channels Gradient Descent in Super Neural Networks" by Fernando et al. (DeepMind) "Evolution Strategies as a Scalable Alternative to Reinforcement Learning" by Salimans et al (OpenAI) evolution and RL Neuroevolution of augmented topologiesNEAT (2002):
- 44. http://roboticvision.org/ RL applications Motion Planning Grasping End-to-end Vision (?)
- 45. http://roboticvision.org/ challenges Curiosity? Exploration/Exploitation? Oracles? discreet —> continuous? real-world reward? https://github.com/aikorea/awesome-rl#theory
- 46. BRISBANE.AI new developments arxiv-sanity, twitter & get your hands dirty come to Brisbane.AI meetups! :) how to keep in the loop? http://Juxi.net/workshop/deep-learning-rss-2017/ Tools and toolboxes Neuroscience vs Deep Learning & Evolutionary approaches Generative Adversarial Networks Unsupervised Learning, Embodied Learning
- 47. BRISBANE.AI Jürgen ‘Juxi’ Leitner arc centre of excellence for robotic vision | qut juxi.net | roboticvision.org | bne-robotics.net | brisbane.ai In which we try to explain why we consider ar,ficial intelligence to be a subject most worthy of study, and in which we try to decide what exactly it is, this being a good thing to decide before embarking. TUTORIAL ONE BRISBANE ARTIFICIAL INTELLIGENCE http://Juxi.net <juxi.leitner@gmail.com>
- 49. for listening thank you ICRA 2018 Brisbane,Australia j.leitner@roboticvision.org http://Juxi.net/projects Juxi