Reinforcement Learning 1. Introduction
- 1. Chapter 1: Introduction Seungjae Ryan Lee
- 2. ● Learning by interacting with the environment ● Goal: maximize a numerical reward signal by choosing correct actions ○ Trial and error: learner is not told the best action ○ Delayed rewards: actions can affect all future rewards Reinforcement Learning
- 3. ● No external supervisor / teacher ○ No training set with labeled examples (answers) ○ Need to interact with environment in uncharted territories ● Different goals ○ Supervised Learning: Generalize existing data to minimize test set error ○ Reinforcement Learning: Maximize reward through interactions ○ Unsupervised Learning: Find hidden structure → Reinforcement Learning is a new paradigm of Machine Learning vs. Supervised and Unsupervised Learning
- 4. ● Interactions between agent and environment ● Uncertainty about the environment ○ Effects of actions cannot be fully predicted ○ Monitor environment and react appropriately ● Defined goal ○ Judge progress through rewards ● Present affects the future ○ Effect can be delayed ● Experience improves performance Characteristics of Reinforcement Learning
- 5. ● Complex sequence of interactions to achieve goal ● Need to observe and react to the uncertainty of the environment ○ Grab different bowl if current bowl is dirty ○ Stop pouring if the bowl is about to overflow ● Actions have delayed consequences ○ Failing to get spoon does not matter until you start eating ● Experience improves performance Example: Preparing Breakfast Get cereal Get bowl Get spoon Get milk Pour cereal Pour milk Grab spoon
- 6. ● Exploration: Try different actions ● Exploitation: Choose best known action ● Need both to obtain high reward Exploration vs Exploitation Try a new cereal? Eat the usual cereal?Agent
- 7. ● Policy defines the agent’s behavior ● Reward Signal defines the goal of the problem ● Value Function indicates the long-term desirability of state ● Model of the environment mimics behavior of environment Elements of Reinforcement Learning
- 8. ● Mapping from observation to action ● Defines the agent’s behavior ● Can be stochastic Policy s1 s2 s3 s4 a1 a2 S A
- 9. ● Reward ○ Immediate reward of action ○ Defines good/bad events for the agent ○ Given by the environment ● Value Function ○ Sum of future rewards from a state ○ Long-term desirability of states ○ Difficult to estimate ○ Primary basis of choosing action Reward Signal vs. Value Function
- 10. ● Mimics the behavior of environment ● Allow planning a future course of actions ● Not necessary for all RL methods ○ Model-based methods use the model for planning ○ Model-free methods only use trial-and-error Model https://worldmodels.github.io/
- 11. ● Assume imperfect opponent ● Agent needs to find and exploit imperfections Example: Tic Tac Toe
- 12. Tic Tac Toe with Reinforcement Learning ● Initialize value functions to 0.5 (except terminal states) ● Learn by playing games ○ Move greedily most times, but explore sometimes ● Incrementally update value functions by playing games ● Decrease learning rate over time to converge
- 13. ● Minimax algorithm ○ Assumes best play for opponent → Cannot exploit opponent ● Classic optimization ○ Require complete specification of opponent → Impractical ○ ex. Dynamic Programming ● Evolutionary methods ○ Finds optimal algorithm ○ Ignores useful structure of RL problems ○ Works best when good policy can be found easily Tic Tac Toe with other algorithms
- 14. ● Can be applied to: ○ more complex games (ex. Backgammon) ○ problems without enemies (“games against nature”) ○ problems with partially observable environments ○ non-episodic problems ○ continuous-time problems Reinforcement Learning beyond Tic-Tac-Toe https://blog.openai.com/evolution-strategies/
- 15. Thank you! Original content from ● Reinforcement Learning: An Introduction by Sutton and Barto You can find more content in ● github.com/seungjaeryanlee ● www.endtoend.ai