Asynchronous Methods for Deep Reinforcement Learning
0 likes589 views
The document discusses asynchronous methods for deep reinforcement learning, highlighting the limitations of experience replay and proposing a framework that executes multiple agents in parallel across environments. This approach enhances training stability, reduces computational costs, and shows significant performance improvements in various tasks compared to traditional methods. Experimental results demonstrate robust efficiencies and effective training using reduced resources on a single machine.
![Asynchronous Methods for Deep Reinforcement Learning
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Tim Harley, Timothy P., David Silver, Koray Kavukcuoglu
Google DeepMind
Montreal Institute for Learning Algorithms (MILA), University of Montreal
Journal reference: ICML 2016
Cite as: arXiv:1602.01783 [cs.LG]
(or arXiv:1602.01783v2 [cs.LG] for this version)
資應所 105065702 李思叡
1](https://image.slidesharecdn.com/ppdlpresentationforisa-190317085322/85/Asynchronous-Methods-for-Deep-Reinforcement-Learning-1-320.jpg)
Asynchronous Methods for Deep Reinforcement Learning
- 1. Asynchronous Methods for Deep Reinforcement Learning Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Tim Harley, Timothy P., David Silver, Koray Kavukcuoglu Google DeepMind Montreal Institute for Learning Algorithms (MILA), University of Montreal Journal reference: ICML 2016 Cite as: arXiv:1602.01783 [cs.LG] (or arXiv:1602.01783v2 [cs.LG] for this version) 資應所 105065702 李思叡 1
- 2. Outline Introduction Related Work Reinforcement Learning Background Asynchronous RL Framework Experiments Conclusions and Discussion 2
- 3. Outline Introduction Related Work Reinforcement Learning Background Asynchronous RL Framework Experiments Conclusions and Discussion 3
- 4. Introduction Online RL algorithm with DNN is fundamentally unstable - sequence of observed data encounter by an online RL agent is non-stationary - online RL updates are strongly correlated - solution idea → experience replay (experience replay achieved unprecedented success) Drawbacks of experience replay: - uses more memory and computation per real interaction - requires off-policy learning algorithm that can update from data generated by an older policy 4
- 5. Introduction In this paper: Asynchronously execute multiple agents in parallel on multiple environments. (instead of experience replay) Benefits: - decorrelate the agents’ data into a more stationary process - can apply on both on-policy and off-policy RL algorithm - GPU or massively distributed machines → multi core CPU (single machine) - cost far less time than GPU-based algorithms - use far less resources than massively distributed approach. 5
- 6. Outline Introduction Related Work Reinforcement Learning Background Asynchronous RL Framework Experiments Conclusions and Discussion 6
- 7. Related Work The General Reinforcement Learning Architecture (Gorila) of performs asynchronous training of reinforcement learning agents in a distributed setting. (Nair et al., 2015) Map Reduce framework with linear function approximation. (Li & Schuurmans, 2011) Parallel version of the Sarsa uses multiple separate actor-learners. (Grounds & Kudenko, 2008) Convergence properties of Q-learning in the asynchronous optimization setting. (Tsitsiklis, 1994) The related problem of distributed dynamic programming. (Bertsekas, 1982) 7
- 8. Related Work: Gorila Architecture Perform asynchronous training of reinforcement learning agents in a distributed setting. Actor - acts in its own copy of environment - separate replay memory Learner - samples data from replay memory - computes gradients of the DQN loss with respect to the policy parameters 8
- 9. Related Work: Gorila Architecture Gradients are asynchronously sent to a central parameter server which updates a central copy of the model. The updated policy parameters are sent to the actor-learners at fixed interval. Setting: - 100 separate actor-learners - 30 parameter servers instance - 130 machines in total Performance: - outperform DQN over 49 Atari games - 20 times faster than DQN 9
- 10. Related Work Map Reduce framework to parallelizing batch reinforcement learning with linear function approximation - Parallelism was used to speed up large matrix operations - Not to parallelize the collection of experience or stabilize learning Parallel version of Sarsa algorithm - Multiple separate actor-learners to accelerate training - Learns separately and periodically send updates to weights that have changed significantly to the other learners using peer-to-peer (P2P) communication. 10
- 12. Outline Introduction Related Work Reinforcement Learning Background Asynchronous RL Framework Experiments Conclusions and Discussion 12
- 13. Reinforcement Learning Background one-step Q-learning (TD(0)) one-step Sarsa n-step Q-learning (TD(0) → TD(λ)) actor-critic 13
- 14. RL Background: Q-learning and Sarsa Q-learning: Sarsa: 14
- 15. RL Background: Actor-Critic Actor-Critic follow an approximate policy gradient: With advantage function: 15 (use the TD(0) error for example)
- 16. Outline Introduction Related Work Reinforcement Learning Background Asynchronous RL Framework Experiments Conclusions and Discussion 16
- 17. Asynchronous RL Framework Copy global network parameters Worker interacts with environment Accumulate gradients Update global network with gradients Copy global network parameters Worker interacts with environment .... source: https://medium.com/emergent- future/simple-reinforcement-learning-with- tensorflow-part-8-asynchronous-actor-critic- agents-a3c-c88f72a5e9f2 17Framework of A3C loop
- 18. Asynchronous RL Framework 18source: Naruto comic (315th, 617th episode)
- 19. Asynchronous RL Framework source: https://github.com/coreylynch/async-rl 19
- 20. Two main ideas of practice Similarly to the Gorila framework, but: - separate machines → multiple CPU threads on a single machine - removes the communication costs of sending gradients and parameters - use Hogwild! (Recht et al., 2011) style updates for training Multiple learners running in parallel - exploring different parts of the environment - maximize the diversity - be less correlated in time (decorrelate) - do not use replay memory (→ be able to use on-policy RL to train neural networks in a stable way) 20
- 21. Asynchronous RL Framework Asynchronous one-step Q-learning Asynchronous one-step Sarsa Asynchronous n-step Q-learning Asynchronous advantage actor-critic (A3C) 21 one-step Q-learning one-step Sarsa n-step Q-learning actor-critic
- 22. Algo: DQN v.s. Asynchronous one-step Q-learning 22 Deep Q Network (DQN) Asynchronous one-step Q-learning
- 23. Algo: Asynchronous Advantage Actor-Critic 23 can add entropy regularization (H): +
- 24. Optimization 1. SGD with momentum 2. RMSProp without shared statistics 3. RMSProp with shared statistics (mode robust) → RMSProp where statistics g are shared across threads 24
- 25. Outline Introduction Related Work Reinforcement Learning Background Asynchronous RL Framework Experiments Conclusions and Discussion 25
- 26. Experiments Atari 2600 Games (experiment on 57 games) TORCS Car Racing Simulator (3D game) MuJoCo Physics Simulator (Continuous Action Control) Labyrinth (3D maze game) 26
- 27. Experimental setup (A3C on Atari and TORCS) - number of threads: 16 (on a single machine and no GPUs) - updates every 5 actions (𝑡"#$ = 5 and 𝐼()*#+, = 5) - optimization: shared RMSProp - network architecture: 2 Conv layers and 1 FC layer (followed by ReLU) - input preprocessing and network architecture as (Mnih et al., 2015; 2013) - discount factor 𝛾 = 0.99, RMSProp decay factor 𝛼 = 0.99, entropy weight 𝛽 = 0.01 - learning rate: sample from a 𝐿𝑜𝑔𝑈𝑛𝑖𝐹𝑜𝑟𝑚(10>? , 10>A ) distribution (for more details, see paper section 8 & 9) 27
- 28. Learning speed comparison (5 Atari games) DQN: train on Nvidia K40 GPU Asynchronous methods: train on 16 CPU cores 28
- 29. Score result on 57 Atari games Fix all hyperparameters for all 57 games A3C, LSTM: add 256 LSTM cells after final hidden layer (for more compare) Mean, median: human-normalized scores on 57 Atari games 29
- 30. A3C on other environments TORCS 3D car racing game: https://youtu.be/0xo1Ldx3L5Q MuJoCo physics simulator: https://youtu.be/Ajjc08-iPx8 Labyrinth: https://youtu.be/nMR5mjCFZCw 30
- 31. Scalability and Data Efficiency - superlinear speedups (especially on one-step methods) 31 the speedups average over 7 Atari games
- 32. Robustness and Stability 50 different learning rates and random initializations Result: Robust to the choice of learning rate and random initialization Stable and do not collapse or diverage once they are learning (4 asynchronous methods have the same conclusion) 32
- 33. Comparison of three optimization methods 50 experiments on n-step Q and A3C with 50 different random learning rates and initializations 1. Momentum SGD 2. RMSProp 3. Shared RMSProp 33
- 34. Outline Introduction Related Work Reinforcement Learning Background Asynchronous RL Framework Experiments Conclusions and Discussion 34
- 35. Conclusions and Discussion In this framework - Stable training of NN is possible in many situations (value/policy-base, on/off-policy, discrete/continuous) - Reduce the comsumption of time - Could be potentially improved by using other ways of estimating advantage funciton - A number of complementary improvements to the NN architecture are possible. 35
- 36. Thanks for listening :) 36
- 37. Q & A 37