Reinforcement Learning: a Brief Overview
1 like304 views
This document provides an introduction to reinforcement learning (RL). It discusses key concepts in RL including supervised vs reinforcement learning, the multi-armed bandit problem, exploration vs exploitation, and challenges in stationary vs non-stationary problems. Action values and optimal action values are estimated using sample averages and updated using a weighted average. Preliminary results show an RL strategy outperforming other strategies on a non-stationary problem.
![Es:ma:on
Qk+1 =
=
1
k +1
ri
i=1
k+1
∑
=
1
k +1
[rk+1 + ri
i=1
k
∑ ]
=
1
k +1
[rk+1 + kQk +Qk −Qk ]
=
1
k +1
[rk+1 +(k +1)Qk −Qk ]
= Qk +
1
k +1
[rk+1 −Qk ]
New es:ma:on= Old es:ma:on+ Step size x [Target- Old es:ma:on]](https://image.slidesharecdn.com/reinforcementlearninggroupmeeting-160731173724/85/Reinforcement-Learning-a-Brief-Overview-10-320.jpg)
![Sta:onary and non-Sta:onary
Qk =
= Qk−1 +α[rk −Qk−1]
=αrk +(1−α)Qk−1
=αrk +(1−α)αrk−1 +(1−α)2
Qk−2
=αrk +(1−α)αrk−1 +(1−α)2
αrk−2 +...+(1−α)k−1
αr1 ++(1−α)K
Q0
= (1−α)k
Q0 + α(1−
i=1
k
∑ α)k−i
ri
0 <α ≤1
Weight
§ Introducing the weight (α):
considers the recent rewards greater than long- past
ones.
This method is called exponen:al, recency-weighted average.](https://image.slidesharecdn.com/reinforcementlearninggroupmeeting-160731173724/85/Reinforcement-Learning-a-Brief-Overview-13-320.jpg)
Reinforcement Learning: a Brief Overview
- 3. Mo:va:on • Typical AcGve Learning 0 50 100 150 0.7 0.75 0.8 0.85 0.9 0.95 1 Mean Absolute Error # of iterations MAE Strategy 1 Strategy 2 More info on Ac:ve Learning: Rubens, Neil; Elahi, Mehdi; Sugiyama, Masashi; Kaplan, Dain; Ac:ve Learning in Recommender Systems, Recommender Systems Handbook, Springer US (2015)
- 4. Mo:va:on • AdapGve AcGve Learning 0 50 100 150 0.7 0.75 0.8 0.85 0.9 0.95 1 # of iterations MAE Mean Absolute Error Strategy 1 Strategy 2 Adaptive Strategy Switching point More info on Adap:ve Ac:ve Learning: Elahi, Mehdi, Francesco Ricci, and Neil Rubens. "A survey of ac:ve learning in collabora:ve filtering recommender systems." Computer Science Review (2016).
- 5. Mo:va:on • AdapGve AcGve Learning 0 50 100 150 0.7 0.75 0.8 0.85 0.9 0.95 1 # of iterations MAE Mean Absolute Error Adaptive Strategy More info on Adap:ve Ac:ve Learning: Elahi, Mehdi, Francesco Ricci, and Neil Rubens. "A survey of ac:ve learning in collabora:ve filtering recommender systems." Computer Science Review (2016).
- 6. n-Armed Bandit § Slot machine with n-arms, each of them will give different reward § In every play, we should find the best arm to maximized the total reward Predict the next reward Choose the best arm Learn from the reward
- 7. Example § Example: 1st play 2nd play 3rd play § Every play is an Ac:on (a) § Then the system make transiGon to the next State (s) § In every play a reward (r) is given based on the chosen arm § How to play is a Policy (π) which maps states to ac9ons
- 8. Ac:on Value § AcGon value - Qt(a) : the esGmated value of an acGon (a) § This Method is called sample-average AcGon value at Gme t Rewards Number of Gmes acGon a is chosen
- 10. Es:ma:on Qk+1 = = 1 k +1 ri i=1 k+1 ∑ = 1 k +1 [rk+1 + ri i=1 k ∑ ] = 1 k +1 [rk+1 + kQk +Qk −Qk ] = 1 k +1 [rk+1 +(k +1)Qk −Qk ] = Qk + 1 k +1 [rk+1 −Qk ] New es:ma:on= Old es:ma:on+ Step size x [Target- Old es:ma:on]
- 11. Challenges § StaGonary and non-StaGonary § ExploraGon and ExploitaGon § Reinforcement Comparison
- 12. Sta:onary and non-Sta:onary § If the rewards are fixed we have a sta:onary problem. § You can keep what you have learned § But in many cases the rewards are changing over the Gme. This is called is non-sta:onary problem. § This means that once you learned, you can not keep it for ever Number of plays (t) Example of non-staGonary problem Rewards (r)
- 13. Sta:onary and non-Sta:onary Qk = = Qk−1 +α[rk −Qk−1] =αrk +(1−α)Qk−1 =αrk +(1−α)αrk−1 +(1−α)2 Qk−2 =αrk +(1−α)αrk−1 +(1−α)2 αrk−2 +...+(1−α)k−1 αr1 ++(1−α)K Q0 = (1−α)k Q0 + α(1− i=1 k ∑ α)k−i ri 0 <α ≤1 Weight § Introducing the weight (α): considers the recent rewards greater than long- past ones. This method is called exponen:al, recency-weighted average.
- 15. Explora:on and Exploita:on § ξ-Greedy: § SoSmax: Example: Boltzmann Method at * = argmax a Qt (a) at = at * at ≠ at * at * with probability 1 − ε random action with probability ε{at = Choosing the best acGon (Exploita:on) Choosing the not-the-best acGon (Explora:on) The best acGon The next acGon Maximum acGon value
- 17. Boltzmann Distribu:on e Qt (a) T e Qt (b) T b=1 n ∑ Temperature Lower Temperature More Greedy acGon selecGon More Exploita5on Higher Temperature Less Greedy acGon selecGon More Explora5on Gives the probability of choosing the acGon a with at the play t
- 18. Reinforcement Comparison § We know that in RL: AcGons with large rewards should be followed more likely than acGons with small rewards § If the reward is 5, is it large or small? § Natural reference reward is the average of previously received rewards Larger rewards > reference reward Small rewards < reference reward § Method based on this idea are called Reinforcement Comparison § This method: Introduces the fact probability of choosing an ac:on in ac:on selec:on process: Indicates that high rewards should increase the probability of reselec:ng the ac:on were taken
- 20. Op:mis:c Ini:al Value § Imagine we set the iniGal acGon value very high (say +5 instead of 0) § Whatever acGon is chosen, the next acGon value would be less than +5 § System may be DISAPPOINTED!! § It will end up with a temporary explora:on ac:ons Well-suited for sta9onary problems But not for non-sta9onary problems
- 22. Preliminary Result 0 50 100 150 0.7 0.75 0.8 0.85 0.9 0.95 1 # of iterations MAE Mean Absolute Error Strategy 1 Strategy 2 RL Strategy No averaging ξ=0.1 α=0.9 ExploraGon Non-StaGonary problem
- 23. 0 50 100 150 0.7 0.75 0.8 0.85 0.9 0.95 1 # of iterations MAE Mean Absolute Error Strategy 1 Strategy 2 RL Strategy Preliminary Result 5-fold averaging ξ=0.1 α=0.9 Non-StaGonary problem
- 24. 0 50 100 150 0.7 0.75 0.8 0.85 0.9 0.95 1 # of iterations MAE Mean Absolute Error Strategy 1 Strategy 2 RL Strategy Preliminary Result 10-fold averaging ξ=0.1 α=0.9 Non-StaGonary problem
- 25. RL References • R. S. Sueon et. al., Reinforcement Learning, an introduc:on, The MIT press , Cambridge • B. Bakker, Decision Making in Intelligence Systems, Lecture 2, UVA, Amsterdam • C. Rothkopf, N-Armed bandit problems, FIAS