Understanding Dynamic Programming through Bellman Operators
2 likes679 views
The document presents a detailed examination of dynamic programming using Bellman operators, focusing on the value function vector spaces and the behaviors of policy and optimality operators. It describes key concepts such as contraction mapping, policy evaluation, and improvement, illustrating how these elements lead to optimal policies through iterative methods like policy and value iteration. The findings establish that deterministic greedy policies derived from optimal value functions successfully achieve optimal outcomes.
![Vector Space of Value Functions
Assume State pace S consists of n states: {s1, s2, . . . , sn}
Assume Action space A consists of m actions {a1, a2, . . . , am}
This exposition extends easily to continuous state/action spaces too
We denote a stochastic policy as π(a|s) (probability of “a given s”)
Abusing notation, deterministic policy denoted as π(s) = a
Consider a n-dim vector space, each dim corresponding to a state in S
A vector in this space is a specific Value Function (VF) v: S → R
With coordinates [v(s1), v(s2), . . . , v(sn)]
Value Function (VF) for a policy π is denoted as vπ : S → R
Optimal VF denoted as v∗ : S → R such that for any s ∈ S,
v∗(s) = max
π
vπ(s)
Ashwin Rao (Stanford) Bellman Operators January 14, 2019 3 / 11](https://image.slidesharecdn.com/bellmanoperators-190115051140/85/Understanding-Dynamic-Programming-through-Bellman-Operators-3-320.jpg)
![Some more notation
Denote Ra
s as the Expected Reward upon action a in state s
Denote Pa
s,s as the probability of transition s → s upon action a
Define
Rπ(s) =
a∈A
π(a|s) · Ra
s
Pπ(s, s ) =
a∈A
π(a|s) · Pa
s,s
Denote Rπ as the vector [Rπ(s1), Rπ(s2), . . . , Rπ(sn)]
Denote Pπ as the matrix [Pπ(si , si )], 1 ≤ i, i ≤ n
Denote γ as the MDP discount factor
Ashwin Rao (Stanford) Bellman Operators January 14, 2019 4 / 11](https://image.slidesharecdn.com/bellmanoperators-190115051140/85/Understanding-Dynamic-Programming-through-Bellman-Operators-4-320.jpg)
Understanding Dynamic Programming through Bellman Operators
- 1. Understanding (Exact) Dynamic Programming through Bellman Operators Ashwin Rao ICME, Stanford University January 14, 2019 Ashwin Rao (Stanford) Bellman Operators January 14, 2019 1 / 11
- 2. Overview 1 Vector Space of Value Functions 2 Bellman Operators 3 Contraction and Monotonicity 4 Policy Evaluation 5 Policy Iteration 6 Value Iteration 7 Policy Optimality Ashwin Rao (Stanford) Bellman Operators January 14, 2019 2 / 11
- 3. Vector Space of Value Functions Assume State pace S consists of n states: {s1, s2, . . . , sn} Assume Action space A consists of m actions {a1, a2, . . . , am} This exposition extends easily to continuous state/action spaces too We denote a stochastic policy as π(a|s) (probability of “a given s”) Abusing notation, deterministic policy denoted as π(s) = a Consider a n-dim vector space, each dim corresponding to a state in S A vector in this space is a specific Value Function (VF) v: S → R With coordinates [v(s1), v(s2), . . . , v(sn)] Value Function (VF) for a policy π is denoted as vπ : S → R Optimal VF denoted as v∗ : S → R such that for any s ∈ S, v∗(s) = max π vπ(s) Ashwin Rao (Stanford) Bellman Operators January 14, 2019 3 / 11
- 4. Some more notation Denote Ra s as the Expected Reward upon action a in state s Denote Pa s,s as the probability of transition s → s upon action a Define Rπ(s) = a∈A π(a|s) · Ra s Pπ(s, s ) = a∈A π(a|s) · Pa s,s Denote Rπ as the vector [Rπ(s1), Rπ(s2), . . . , Rπ(sn)] Denote Pπ as the matrix [Pπ(si , si )], 1 ≤ i, i ≤ n Denote γ as the MDP discount factor Ashwin Rao (Stanford) Bellman Operators January 14, 2019 4 / 11
- 5. Bellman Operators Bπ and B∗ We define operators that transform a VF vector to another VF vector Bellman Policy Operator Bπ (for policy π) operating on VF vector v: Bπv = Rπ + γPπ · v Bπ is a linear operator with fixed point vπ, meaning Bπvπ = vπ Bellman Optimality Operator B∗ operating on VF vector v: (B∗v)(s) = max a {Ra s + γ s ∈S Pa s,s · v(s )} B∗ is a non-linear operator with fixed point v∗, meaning B∗v∗ = v∗ Define a function G mapping a VF v to a deterministic “greedy” policy G(v) as follows: G(v)(s) = arg max a {Ra s + γ s ∈S Pa s,s · v(s )} BG(v)v = B∗v for any VF v (Policy G(v) achieves the max in B∗) Ashwin Rao (Stanford) Bellman Operators January 14, 2019 5 / 11
- 6. Contraction and Monotonicity of Operators Both Bπ and B∗ are γ-contraction operators in L∞ norm, meaning: For any two VFs v1 and v2, Bπv1 − Bπv2 ∞ ≤ γ v1 − v2 ∞ B∗v1 − B∗v2 ∞ ≤ γ v1 − v2 ∞ So we can invoke Contraction Mapping Theorem to claim fixed point We use the notation v1 ≤ v2 for any two VFs v1, v2 to mean: v1(s) ≤ v2(s) for all s ∈ S Also, both Bπ and B∗ are monotonic, meaning: For any two VFs v1 and v2, v1 ≤ v2 ⇒ Bπv1 ≤ Bπv2 v1 ≤ v2 ⇒ B∗v1 ≤ B∗v2 Ashwin Rao (Stanford) Bellman Operators January 14, 2019 6 / 11
- 7. Policy Evaluation Bπ satisfies the conditions of Contraction Mapping Theorem Bπ has a unique fixed point vπ, meaning Bπvπ = vπ This is a succinct representation of Bellman Expectation Equation Starting with any VF v and repeatedly applying Bπ, we will reach vπ lim N→∞ BN π v = vπ for any VF v This is a succinct representation of the Policy Evaluation Algorithm Ashwin Rao (Stanford) Bellman Operators January 14, 2019 7 / 11
- 8. Policy Improvement Let πk and vπk denote the Policy and the VF for the Policy in iteration k of Policy Iteration Policy Improvement Step is: πk+1 = G(vπk ), i.e. deterministic greedy Earlier we argued that B∗v = BG(v)v for any VF v. Therefore, B∗vπk = BG(vπk )vπk = Bπk+1 vπk (1) We also know from operator definitions that B∗v ≥ Bπv for all π, v B∗vπk ≥ Bπk vπk = vπk (2) Combining (1) and (2), we get: Bπk+1 vπk ≥ vπk Monotonicity of Bπk+1 implies BN πk+1 vπk ≥ . . . B2 πk+1 vπk ≥ Bπk+1 vπk ≥ vπk vπk+1 = lim N→∞ BN πk+1 vπk ≥ vπk Ashwin Rao (Stanford) Bellman Operators January 14, 2019 8 / 11
- 9. Policy Iteration We have shown that in iteration k + 1 of Policy Iteration, vπk+1 ≥ vπk If vπk+1 = vπk , the above inequalities would hold as equalities So this would mean B∗vπk = vπk But B∗ has a unique fixed point v∗ So this would mean vπk = v∗ Thus, at each iteration, Policy Iteration either strictly improves the VF or achieves the optimal VF v∗ Ashwin Rao (Stanford) Bellman Operators January 14, 2019 9 / 11
- 10. Value Iteration B∗ satisfies the conditions of Contraction Mapping Theorem B∗ has a unique fixed point v∗, meaning B∗v∗ = v∗ This is a succinct representation of Bellman Optimality Equation Starting with any VF v and repeatedly applying B∗, we will reach v∗ lim N→∞ BN ∗ v = v∗ for any VF v This is a succinct representation of the Value Iteration Algorithm Ashwin Rao (Stanford) Bellman Operators January 14, 2019 10 / 11
- 11. Greedy Policy from Optimal VF is an Optimal Policy Earlier we argued that BG(v)v = B∗v for any VF v. Therefore, BG(v∗)v∗ = B∗v∗ But v∗ is the fixed point of B∗, meaning B∗v∗ = v∗. Therefore, BG(v∗)v∗ = v∗ But we know that BG(v∗) has a unique fixed point vG(v∗). Therefore, v∗ = vG(v∗) This says that simply following the deterministic greedy policy G(v∗) (created from the Optimal VF v∗) in fact achieves the Optimal VF v∗ In other words, G(v∗) is an Optimal (Deterministic) Policy Ashwin Rao (Stanford) Bellman Operators January 14, 2019 11 / 11