Dynamic Programming and Reinforcement Learning applied to Tetris Game
2 likes1,398 views
The document discusses the application of dynamic programming and reinforcement learning to the game Tetris, highlighting the challenges of maximizing scores in a complex environment with numerous configurations. It emphasizes the importance of building effective evaluation functions using feature-based methods to improve decision-making and performance. Additionally, it explores the computational efficiencies gained through dynamic programming techniques in solving sequential decision-making problems under uncertainty.
![Dictionary m
m[0]=0, m[1]=1
integer fib(n)
if m[n] == null
m[n] = fib(n-1)+ fib(n-2)
return m[n]
Fibonacci Sequence - Dynamic
Programming](https://image.slidesharecdn.com/ia-apresentacao-160314192221/85/Dynamic-Programming-and-Reinforcement-Learning-applied-to-Tetris-Game-39-320.jpg)
Dynamic Programming and Reinforcement Learning applied to Tetris Game
- 1. Dynamic Programming and Reinforcement Learning applied to Tetris game Suelen Goularte Carvalho Inteligência Artificial 2015
- 2. Tetris
- 3. Tetris ✓ Board 20 x 10 ✓ 7 types of tetronimos (pieces) ✓ Move to down, left or right ✓ Rotation pieces
- 4. Tetris One-Piece Controller Player knows: ✓ board ✓ current piece.
- 5. Tetris Two-Piece Controller Player knows: ✓ board ✓ current piece ✓ next piece
- 6. Tetris Evaluation One-Piece Controller Two-Piece Controller
- 7. How many possibilities do we have just here?
- 8. Tetris indeed contains a huge number of board configurations. Finding the strategy that maximizes the average score is an NP-Complete problem! — Building Controllers for Tetris, 2009 7.0 × 2 ≃ 5.6 × 10 199 59
- 10. Tetris is a problems of sequential decision making under uncertainty. In the context of dynamic programming and stochastic control, the most important object is the cost-to-go function, which evaluates the expected future cost from current state. — Feature-Based Methods for Large Scale Dynamic Programming
- 11. 7000 3000 2500 1000 4000Si 5000 7000 3000 2500 1000 4000 best immediate reward Si immediate reward future reward 13000 9000 immediate reward vs. 5000 best future reward best immediate reward Immediate reward Future reward
- 12. 7.0 × 2 ≃ 5.6 × 10 199 59 Essentially impossible to compute, or even store, the value of the cost-to-go function at every possible state. — Feature-Based Methods for Large Scale Dynamic Programming
- 13. Compact representation alleviate the computational time and space of dynamic programming, which employs an exhaustive look-up table, storing one value per state. — Feature-Based Methods for Large Scale Dynamic Programming S {s1, s2, …, sn} V {v1, v2, …, sm} where m < n
- 14. For example, if the state i represents the number of customers in a queueing system, a possible and often interesting feature f is defined by f(0) = 0 and f(i) = 1 if i > 0. Such a feature focuses on whether a queue is empty or not. — Feature-Based Methods for Large Scale Dynamic Programming
- 15. — Feature-Based Methods for Large Scale Dynamic Programming Feature-bases method S {s1, s2, …, sn} V {v1, v2, …, sm} where m < n
- 16. — Feature-Based Methods for Large Scale Dynamic Programming Features: ★ Height of the current wall. ★ Number of holes. H = {0, ..., 20}, L = {0, ..., 200}. Feature extraction F : S ~ H x L 10 X 20 Feature-bases method
- 17. Using a feature-based evaluation function works better than just choosing the move that realizes the highest immediate reward. — Building Controllers for Tetris, 2009
- 18. Example of features — Building Controllers for Tetris, 2009
- 19. ...The problem of building a Tetris controller comes down to building a good evaluation function. Ideally, this function should return high values for the good decisions and low values for the bad ones. — Building Controllers for Tetris, 2009
- 20. Reinforcement Learning context, algorithms aim at tuning the weights such that the evaluation function approximates well the optimal expected future score from each state. — Building Controllers for Tetris, 2009
- 22. Reinforcement Learning by The Big Bang Theory https://www.youtube.com/watch?v=tV7Zp2B_mt8&list=PLAF3D35931B692F5C
- 23. Reinforcement Learning Imagine disputar um novo jogo cuja as regras você não conhece, depois de aproximadamente uma centena de movimentos, seu oponente anuncia: “Você perdeu!”. Em resumo, isso é aprendizagem por reforço.
- 24. Supervised Learning input 1 2 3 4 5 6 7 8 …. output 1 2 9 16 25 36 49 64 …. y = f(x) -> function approximation https://www.youtube.com/watch?v=Ki2iHgKxRBo&list=PLAwxTw4SYaPl0N6-e1GvyLp5-MUMUjOKo Map inputs to output f(x) = x labels scores well 2
- 25. Unsupervised Learning x x x xx x x xx x o o o o o o o o f(x) -> clusters description o o x x x x x x x x x x o o o o o o o oo o type clusters scores well
- 27. Reinforcement Learning ✓ Agents take actions in an environment and receive rewards ✓ Goal is to find the policy π that maximizes rewards ✓ Inspired by research into psychology and animal learning
- 28. Reinforcement Learning Model Given: S set of states, A set of actions, T(s, a, s') ~ P(s’ | s, a) transitional model, R reward function 5000 7000 3000 2500 1000 4000Si immediate reward future reward 13000 9000 Find: π(s) = a policy that maximizes
- 29. Needs higher computation, processing and memory.
- 31. Dynamic Programming Solving problems by breaking it down into simpler subproblems. Solving each subproblems just once, and storing their solutions. https://en.wikipedia.org/wiki/Dynamic_programming
- 32. A G caminho ótimo A B caminho ótimo G caminho ótimo Support Property: Optimal Substructure
- 33. Fibonacci Sequence 0 1 1 2 3 5 8 13 21 The sum of two numbers before results in the follow number.
- 34. 0 1 1 2 3 5 8 13 21 f(n) = f(n-1) + f(n-2) Recursive Formula: v = 0 1 2 3 4 5 6 7 8n = Fibonacci Sequence
- 35. Fibonacci 0 1 1 2 3 5 8 13 21 0 1 2 3 4 5 6 7 8 f(6) = f(6-1) + f(6-2) f(6) = f(5) + f(4) f(6) = 5 + 3 f(6) = 8 v = n =
- 36. Fibonacci Sequence - Normal computation 6 5 4 4 3 2 2 1 1 0 2 1 2 2 1 3 3 1 0 1 0 1 0 1 0 f(n) = f(n-1) + f(n-2)
- 37. 6 5 4 4 3 2 2 1 1 0 2 1 2 2 1 3 3 1 0 1 0 1 0 1 0 Fibonacci Sequence - Normal computation O(n )2
- 38. 18 of 25 Nodes Are Repeated Calculations!
- 39. Dictionary m m[0]=0, m[1]=1 integer fib(n) if m[n] == null m[n] = fib(n-1)+ fib(n-2) return m[n] Fibonacci Sequence - Dynamic Programming
- 40. Fibonacci Sequence - Dynamic Programming 5 4 3 3 2 1 1 0 2 index value 0 1 2 3 4 5 0 1
- 41. 5 4 3 3 2 1 1 0 2 index value 0 1 2 3 4 5 0 1 1 1+0=1 Fibonacci Sequence - Dynamic Programming
- 42. 5 4 3 3 2 1 1 0 2 index value 0 1 2 3 4 5 0 1 1 2 1+0=1 1+1=2 Fibonacci Sequence - Dynamic Programming
- 43. 5 4 3 3 2 1 1 0 2 index value 0 1 2 3 4 5 0 1 1 2 3 1+0=1 1+1=2 2+1=3 Fibonacci Sequence - Dynamic Programming
- 44. 5 4 3 3 2 1 1 0 2 O(1) memory O(n) running time index value 0 1 2 3 4 5 0 1 1 2 3 5 1+0=1 1+1=2 2+1=3 3+2=5 Fibonacci Sequence - Dynamic Programming
- 45. 100 games played 31 Some scores from time… Tsitsiklis and van Roy (1996) Bertsekas and Tsitsiklis (1996)3200100 games played Kakade (2001) applied without specifying how many game scores are averaged though6800 Farias and van Roy (2006) 90 games played. 4700 — Building Controllers for Tetris, 2009
- 46. Two-piece controller with some original features of which the weights were tuned by hand. Only 1 game was played and this took a week. One-piece controller 56 games played. Tuned by hand. 660Mil 7,2Mi Currents best! Dellacherie (Fahey, 2003) Dellacherie (Fahey, 2003) — Building Controllers for Tetris, 2009
- 47. Experiment…
- 48. Experiment — Feature-Based Methods for Large Scale Dynamic Programming Experienced human Tetris player would take about 3 minutes to eliminate 30 rows.
- 49. 20 jogadores. 3 jogadas cada. 3 minutos cada jogada. Experiment cont. 30 Média obtida: 24 score
- 50. Jogador 7 (eu) jogada 1 1000 scores ~ 1 row Experiment cont.
- 51. • Média 24 score a cada 3 minutos. • Ou seja, 5.760 a cada 12h de jogo contínuo. • Um ser-humano jogando começa a ficar próximo a performance dos algoritmos, após algumas otimizações, após mais ou menos 8h de jogo contínuo. Experiment cont.
- 52. Conclusão…
- 53. Dynamic Programming Reinforcement Learning Tetris Otimiza a utilização do poder computacional. Otimiza peso utilizado nas features. Utiliza feature-based para maximizar o score.
- 54. Dúvidas? Suelen Goularte Carvalho Inteligência Artificial 2015