Hierarchical Reinforcement Learning with Option-Critic Architecture
0 likes437 views
This document describes hierarchical reinforcement learning using the option-critic architecture. The option-critic architecture allows for online, end-to-end learning of options in continuous state and action spaces by learning the initiation, intra-option policy, and termination policy of options using deep reinforcement learning techniques like policy gradients. The option-critic architecture extends the options framework by allowing options to be represented by neural networks and learned online through actor-critic methods.
![Reinforcement Learning
MDP: S, A, p(s |s, a), r(s, a)
Policy π(a|s) : S × A → [0, 1]
3](https://image.slidesharecdn.com/presentation-190118043653/85/Hierarchical-Reinforcement-Learning-with-Option-Critic-Architecture-9-320.jpg)
![Reinforcement Learning
MDP: S, A, p(s |s, a), r(s, a)
Policy π(a|s) : S × A → [0, 1]
3](https://image.slidesharecdn.com/presentation-190118043653/85/Hierarchical-Reinforcement-Learning-with-Option-Critic-Architecture-10-320.jpg)
![Reinforcement Learning
MDP: S, A, p(s |s, a), r(s, a)
Policy π(a|s) : S × A → [0, 1]
Goal: an optimal policy π∗ that maximizes E
∞
t=0
γt
rt|s0 = s
3](https://image.slidesharecdn.com/presentation-190118043653/85/Hierarchical-Reinforcement-Learning-with-Option-Critic-Architecture-11-320.jpg)
![References i
Bacon, Pierre-Luc, Jean Harb, and Doina Precup (2017). “The
Option-Critic architecture”. In: Proceedings of the Thirty-First
AAAI Conference on Artificial Intelligence, February 4-9, 2017,
San Francisco, California, USA. Pp. 1726–1734.
Bellman, Richard (1952). “On the theory of dynamic
programming”. In: Proceedings of the National Academy of
Sciences 38.8, pp. 716–719. doi: 10.1073/pnas.38.8.716.
Dilokthanakul, N., C. Kaplanis, N. Pawlowski, and M. Shanahan
(2017). “Feature Control as Intrinsic Motivation for Hierarchical
Reinforcement Learning”. In: ArXiv e-prints. arXiv:
1705.06769 [cs.LG].
26](https://image.slidesharecdn.com/presentation-190118043653/85/Hierarchical-Reinforcement-Learning-with-Option-Critic-Architecture-66-320.jpg)
Hierarchical Reinforcement Learning with Option-Critic Architecture
- 1. Hierarchical Reinforcement Learning with Option-Critic Architecture Oğuz Şerbetci April 4, 2018 Modelling of Cognitive Processes TU Berlin
- 2. Reinforcement Learning Hierarchical Reinforcement Learning Demonstration Resources Appendix 1
- 6. Reinforcement Learning Agent Environment Action atState st Reward rt 2
- 7. Reinforcement Learning MDP: S, A, p(s |s, a), r(s, a) 3
- 8. Reinforcement Learning MDP: S, A, p(s |s, a), r(s, a) 3
- 9. Reinforcement Learning MDP: S, A, p(s |s, a), r(s, a) Policy π(a|s) : S × A → [0, 1] 3
- 10. Reinforcement Learning MDP: S, A, p(s |s, a), r(s, a) Policy π(a|s) : S × A → [0, 1] 3
- 11. Reinforcement Learning MDP: S, A, p(s |s, a), r(s, a) Policy π(a|s) : S × A → [0, 1] Goal: an optimal policy π∗ that maximizes E ∞ t=0 γt rt|s0 = s 3
- 12. Problems 4
- 13. Problems • lack of planning and commitment 4
- 14. Problems • lack of planning and commitment • inefficient exploration 4
- 15. Problems • lack of planning and commitment • inefficient exploration • temporal credit assignment problem 4
- 16. Problems • lack of planning and commitment • inefficient exploration • temporal credit assignment problem • inability to divide-and-conquer 4
- 17. 5
- 19. Temporal Abstractions Icons made by Smashicons and Freepick from Freepick 6
- 20. Temporal Abstractions Icons made by Smashicons and Freepick from Freepick 6
- 21. Temporal Abstractions Icons made by Smashicons and Freepick from Freepick 6
- 22. Temporal Abstractions Icons made by Smashicons and Freepick from Freepick 6
- 23. 7
- 24. Options Framework (Sutton, Precup, et al. 1999) SMDP: S, A, p(s , k |s, a), r(s, a) (Sutton, Precup, et al. 1999) 8
- 25. Options Framework (Sutton, Precup, et al. 1999) SMDP: S, A, p(s , k |s, a), r(s, a) Option: Iω: initiation set πω: intra-option-policy βω: termination-policy (Sutton, Precup, et al. 1999) 8
- 26. Options Framework (Sutton, Precup, et al. 1999) SMDP: S, A, p(s , k |s, a), r(s, a) Option: Iω: initiation set πω: intra-option-policy βω: termination-policy (Sutton, Precup, et al. 1999) 8
- 27. Options Framework (Sutton, Precup, et al. 1999) SMDP: S, A, p(s , k |s, a), r(s, a) Option: Iω: initiation set πω: intra-option-policy βω: termination-policy πΩ: option-policy (Sutton, Precup, et al. 1999) 8
- 28. 9
- 29. Option-Critic (Bacon et al. 2017) Given the number of options Option-Critic learns βω, πω, πΩ in an end-to-end & online fashion. Allows non-linear function approximators, enabling continuous state and action spaces. • online, end-to-end learning of options in continuous state/action space • allows using non-linear function approximators (deep RL) 10
- 30. Value Functions The state value function: Vπ(s) = E ∞ t=0 γt rt|s0 = s 11
- 31. Value Functions The state value function: Vπ(s) = E ∞ t=0 γt rt|s0 = s = a∈As π(s, a) r(s, a) + γ s p(s |s, a)V(s ) Bellman Equations (Bellman 1952) 11
- 32. Value Functions The state value function: Vπ(s) = E ∞ t=0 γt rt|s0 = s = a∈As π(s, a) r(s, a) + γ s p(s |s, a)V(s ) The action value function: Qπ(s, a) = E ∞ t=0 γt rt|s0 = s, a0 = a Bellman Equations (Bellman 1952) 11
- 33. Value Functions The state value function: Vπ(s) = E ∞ t=0 γt rt|s0 = s = a∈As π(s, a) r(s, a) + γ s p(s |s, a)V(s ) The action value function: Qπ(s, a) = E ∞ t=0 γt rt|s0 = s, a0 = a = r(s, a) + γ s p(s |s, a) a ∈A π(s , a )Q(s , a ) Bellman Equations (Bellman 1952) 11
- 34. Value methods TD Learning: Qπ (s, a) ←Qπ (s, a) + α TD target r + γVπ (s ) −Qπ (s, a) TD error 12
- 35. Value methods TD Learning: Qπ (s, a) ←Qπ (s, a) + α TD target r + γVπ (s ) −Qπ (s, a) TD error Vπ (s ) = max a Q(s , a) Q-Learning 12
- 36. Value methods TD Learning: Qπ (s, a) ←Qπ (s, a) + α TD target r + γVπ (s ) −Qπ (s, a) TD error Vπ (s ) = max a Q(s , a) Q-Learning π(a|s) = argmax a Q(s, a) greedy policy 12
- 37. Value methods TD Learning: Qπ (s, a) ←Qπ (s, a) + α TD target r + γVπ (s ) −Qπ (s, a) TD error Vπ (s ) = max a Q(s , a) Q-Learning π(a|s) = argmax a Q(s, a) greedy policy π(a|s) = random with probability argmaxa Q(s, a) with probability 1 − - greedy policy 12
- 38. Option Value Functions The option value function: QΩ(s, ω) = a πω(a|s)QU (s, w, a) 13
- 39. Option Value Functions The option value function: QΩ(s, ω) = a πω(a|s)QU (s, w, a) The action value function: QU (s, ω, a) = r(s, a) + γ s p(s |s, a)U(ω, s ) 13
- 40. Option Value Functions The option value function: QΩ(s, ω) = a πω(a|s)QU (s, w, a) The action value function: QU (s, ω, a) = r(s, a) + γ s p(s |s, a)U(ω, s ) The state value function upon arrival: U(ω, s ) = (1 − βω(s ))QΩ(s , ω) + βω(s )VΩ(s ) 13
- 41. Policy Gradient Methods π(a|s) = argmax a Q(s, a) 14
- 42. Policy Gradient Methods π(a|s) = argmax a Q(s, a) vs. π(a|s, θ) = softmax a h(s, a, θ) 14
- 43. Policy Gradient Methods π(a|s) = argmax a Q(s, a) vs. π(a|s, θ) = softmax a h(s, a, θ) J(θ) 14
- 44. Policy Gradient Methods π(a|s) = argmax a Q(s, a) vs. π(a|s, θ) = softmax a h(s, a, θ) J(θ) = V∗ (s) 14
- 45. Policy Gradient Methods π(a|s) = argmax a Q(s, a) vs. π(a|s, θ) = softmax a h(s, a, θ) J(θ) = V∗ (s) θJ(θ) 14
- 46. Policy Gradient Methods π(a|s) = argmax a Q(s, a) vs. π(a|s, θ) = softmax a h(s, a, θ) J(θ) = V∗ (s) θJ(θ) Policy Gradient Theorem (Sutton, McAllester, et al. 2000) 14
- 47. Policy Gradient Methods π(a|s) = argmax a Q(s, a) vs. π(a|s, θ) = softmax a h(s, a, θ) J(θ) = V∗ (s) θJ(θ) Policy Gradient Theorem (Sutton, McAllester, et al. 2000) θJ(θ) = s µπ(s) a Qπ(s, a) θπ(a|s, θ) 14
- 48. Policy Gradient Methods π(a|s) = argmax a Q(s, a) vs. π(a|s, θ) = softmax a h(s, a, θ) J(θ) = V∗ (s) θJ(θ) Policy Gradient Theorem (Sutton, McAllester, et al. 2000) θJ(θ) = s µπ(s) a Qπ(s, a) θπ(a|s, θ) θJ(θ) = E γt a Qπ(s, a) θπ(a|s, θ) 14
- 49. Actor-Critic (Sutton 1984) θ ← θ + αγt δ TD-Error θ log π(a|s, θ) Taken from Pierre-Luc Bacon 15
- 50. Option-Critic (Bacon et al. 2017) Taken from (Bacon et al. 2017) 16
- 51. Option-Critic (Bacon et al. 2017) The gradient wrt. intra-option-policy parameters θ: θQΩ(s, ω) take better primitives inside options. 17
- 52. Option-Critic (Bacon et al. 2017) The gradient wrt. intra-option-policy parameters θ: θQΩ(s, ω) = E ∂ log πω,θ(a|s) ∂θ QU (s, ω, a) take better primitives inside options. 17
- 53. Option-Critic (Bacon et al. 2017) The gradient wrt. intra-option-policy parameters θ: θQΩ(s, ω) = E ∂ log πω,θ(a|s) ∂θ QU (s, ω, a) take better primitives inside options. The gradient wrt. termination-policy parameters ϑ: ϑU(ω, s ) shorten options with bad advantage. 17
- 54. Option-Critic (Bacon et al. 2017) The gradient wrt. intra-option-policy parameters θ: θQΩ(s, ω) = E ∂ log πω,θ(a|s) ∂θ QU (s, ω, a) take better primitives inside options. The gradient wrt. termination-policy parameters ϑ: ϑU(ω, s ) = E − ∂βω,ϑ(s ) ∂ϑ QΩ(s , ω) − VΩ(s , ω) e.g. maxω Q(s,w) shorten options with bad advantage. 17
- 55. Demonstration
- 56. 18
- 57. 19
- 58. 20
- 59. 20
- 60. Complex Environment i (Bacon et al. 2017) 21
- 61. Complex Environment ii (Harb et al. 2017) 22
- 62. But... i 23
- 63. But... ii (Dilokthanakul et al. 2017) 24
- 64. Resources
- 65. • Sutton & Barto, Reinforcement Learning: An Introduction, Second Edition Draft • David Silver’s Reinforcement Learning Course 25
- 66. References i Bacon, Pierre-Luc, Jean Harb, and Doina Precup (2017). “The Option-Critic architecture”. In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA. Pp. 1726–1734. Bellman, Richard (1952). “On the theory of dynamic programming”. In: Proceedings of the National Academy of Sciences 38.8, pp. 716–719. doi: 10.1073/pnas.38.8.716. Dilokthanakul, N., C. Kaplanis, N. Pawlowski, and M. Shanahan (2017). “Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning”. In: ArXiv e-prints. arXiv: 1705.06769 [cs.LG]. 26
- 67. References ii Harb, Jean, Pierre-Luc Bacon, Martin Klissarov, and Doina Precup (2017). “When waiting is not an option: Learning options with a deliberation cost”. In: arXiv: 1709.04571. Sutton, Richard S (1984). “Temporal credit assignment in reinforcement learning”. AAI8410337. PhD thesis. Sutton, Richard S, David A McAllester, Satinder P Singh, and Yishay Mansour (2000). “Policy gradient methods for reinforcement learning with function approximation”. In: Advances in Neural Information Processing Systems, pp. 1057–1063. 27
- 68. References iii Sutton, Richard S, Doina Precup, and Satinder Singh (1999). “Between MDPs and Semi-MDPs: A framework for temporal abstraction in reinforcement learning”. In: Artificial Intelligence 112.1-2, pp. 181–211. doi: 10.1016/S0004-3702(99)00052-1. 28
- 69. Appendix
- 70. Option-Critic (Bacon et al. 2017) i procedure train(α, NΩ) s ← s0 choose ω ∼ πΩ(ω|s) Option-policy repeat choose a ∼ πω,θ(a|s) Intra-option-policy take the action a in s, observe s and r 1. Options evaluation g ← r TD-Target if s is not terminal then g ← g + γ(1 − βω,ϑ(s ))QΩ(s , ω) + γ βω,ϑ(s ) max ω QΩ(s , ω) 29
- 71. Option-Critic (Bacon et al. 2017) ii 2. Critic improvement δU ← g − QU (s, ω, a) QU (s, ω, a) ← QU (s, ω, a) + αU δU 3. Intra-option Q-learning δΩ ← g − QΩ(s, ω) QΩ(s, ω) ← QΩ(s, ω) + αΩδΩ 4. Options improvement θ ← θ + αθ ∂ log πω,θ(a|s) ∂θ QU (s, ω, a) ϑ ← ϑ + αϑ ∂βω,ϑ(s ) ∂ϑ (QΩ(s , ω) − maxω QΩ(s , ω) + ξ) 30
- 72. Option-Critic (Bacon et al. 2017) iii if terminate ∼ βω,ϑ(s ) then Termination-policy choose ω ∼ πΩ(ω|s ) s ← s . until s is terminal 31