![Safety Challenge
RL expected reward :
• Suppose R(s) [-1,1] for “typical events” ( +1,
-1)
• p << 1 policy (p = 10-8
)
• expected reward pr, r reward
• , -r << -1/p . reward .
• variance . Variance :
Claim: The variance of the policy gradient estimator grows with 1/p](https://image.slidesharecdn.com/safemultiagentrlforautonomousedriving4-170508165059/85/Safe-Multi-Agent-Reinforcement-Learning-for-Autonomous-Driving-23-320.jpg)
![Safety Challenge
learned policy ?
RL expected reward :
• Suppose R(s) [-1,1] for “typical events” ( +1,
-1)
• p << 1 policy (p = 10-8
)
• expected reward pr, r reward
• , -r << -1/p . reward .
• variance . Variance :
Claim: The variance of the policy gradient estimator grows with 1/p](https://image.slidesharecdn.com/safemultiagentrlforautonomousedriving4-170508165059/85/Safe-Multi-Agent-Reinforcement-Learning-for-Autonomous-Driving-24-320.jpg)
[한국어] Safe Multi-Agent Reinforcement Learning for Autonomous Driving
- 1. Presentation on “Safe, Multi-Agent, RL for Autonomous Driving” Kiho Suh Modulabs( ), May 8th 2017
- 2. About Paper • 2016 10 • Mobileye (1999 ) • Developing vision-based advanced driver-assistance systems • 2017 3 $15.3 billion( 17 ) • Shai Shaked reddit hacker news “Failures of Deep Learning”
- 3. ? • • “multi-agent” game. • agent , . • : https://www.bloomberg.com/news/articles/ 2015-12-18/humans-are-slamming-into-driverless-cars-and-exposing-a-key-flaw
- 5. Three Elements of Autonomous Driving Sensing Mapping Driving Policy (Planning) Environmental Model 360 awareness Localization at High Accuracy (10cm) Drivable Path Negotiating in a Multi-Agent game Strategy
- 6. Three Elements of Autonomous Driving Sensing Mapping Driving Policy (Planning) Environmental Model 360 awareness Localization at High Accuracy (10cm) Drivable Path Negotiating in a Multi-Agent game Strategy !
- 7. Driving Policy ? • : policy • Multi-Agent Game • agent : ? ? ? • Defensive/Aggressive Tradeoff: • : . • • Sensing uncertainty: radar • Blind zone: ?
- 10. Some Approaches to RL • Imitation Learning (behavior Cloning) • Learn from pairs {(si,ai)}. where ai is the action chosen by a good (e.g. human driver) agent at state si. One can then use supervised learning to learn a policy π such that π(si) ≈ at • Direct Policy Optimization • Express policy π in parametric form and directly optimize it using SGD • Value based Learning • Learn the Q or V functions. Solely context of MDP. • Model based Learning and Planning • Learn the probability of state transitions and solve the optimization problem of finding the optimal Q. Clearly rely on the Markov Assumption.
- 11. Some Approaches to RL • Imitation Learning (behavior Cloning) • Learn from pairs {(si,ai)}. where ai is the action chosen by a good (e.g. human driver) agent at state si. One can then use supervised learning to learn a policy π such that π(si) ≈ at • Direct Policy Optimization • Express policy π in parametric form and directly optimize it using SGD • Value based Learning • Learn the Q or V functions. Solely context of MDP. • Model based Learning and Planning • Learn the probability of state transitions and solve the optimization problem of finding the optimal Q. Clearly rely on the Markov Assumption. In practice: !
- 12. Markov Decision Process (MDP) The Markovian Assumption: st+1 is conditional independent of the past given st, at • multi-agents • Markovian Assumption , “ ” ( hidden state in POMDPs) . “ ” ? • Markovity . • An agnostic RL setting (The optimal policy st ). • Value function (V and Q Makovian Assumption ). : “ ”
- 13. RL without the Markovian Assumption • O Imitation learning (behavior cloning) • ? Direct policy optimization • X Value based learning • X Model based learning and planning st Markovian Assumption ! agent dynamics system Markov Assumption ! !
- 15. Policy Stochastic Gradient without Markovian Assumption parameter θ ! action πθ
- 16. Policy Stochastic Gradient without Markovian Assumption parameter θ ! action πθ parameter θ
- 18. Variance Reduction • SGD variance • Value function variance . • Markovian Assumption variance reduction • SVRG (Johnson and Zhang) SDCA(Shalev-Shwartz and Zhang) , variance .
- 19. Variance Reduction Lemma ξ : random variable RL . Bt,i baseline, ( Value function , expected future reward, state ). Qθ hat (expected future reward, action ) Advantage function . Actor Critic . ! Non- Markovian . environment assumption !
- 20. Imitation Learning Policy Gradient • Imitation learning πθ • SGD πθ • Markovian assumptions
- 21. Imitation Learning Policy Gradient (video) https://www.dropbox.com/s/egfcghug4y612fp/mobileye3.mp4?dl=0
- 22. safe RL!
- 23. Safety Challenge RL expected reward : • Suppose R(s) [-1,1] for “typical events” ( +1, -1) • p << 1 policy (p = 10-8 ) • expected reward pr, r reward • , -r << -1/p . reward . • variance . Variance : Claim: The variance of the policy gradient estimator grows with 1/p
- 24. Safety Challenge learned policy ? RL expected reward : • Suppose R(s) [-1,1] for “typical events” ( +1, -1) • p << 1 policy (p = 10-8 ) • expected reward pr, r reward • , -r << -1/p . reward . • variance . Variance : Claim: The variance of the policy gradient estimator grows with 1/p
- 25. Safe RL • policy π : S -> A • policy function : π = π(T) ο πθ (D) • .
- 26. πθ (D) • . , . • πθ (D): S -> D state parameters set desires set . • This function is parameterized by θ and is being learned. • The desires produced by πθ (D) are translated into a cost function over driving trajectories.
- 27. π(T) • • π(T): S -> D state desires set action (continuous trajectory). • This function is implemented by solving an optimization problem with hard constraints on safety and a cost that is defined by the “desires” • hard constraints framework . not being learned ( θ )
- 28. π = π (T) ο πθ (D) • double-merge scenario ( ) • . • . • heuristic brute-force .
- 29. The Double-Merge Scenario (video) https://www.dropbox.com/s136nbndtdyehtgidoubleMerge.m4v?dl=0
- 30. Double-merge Scenario Desire -> desired target speed of the host vehicle desired lateral position in lanes units classification labels assigned to each of the n other vehicles g: give way( ) t: take way( ) o: maintain offset distance to it( )
- 33. Desires ( ?)
- 34. Markovity and Safety • Markovity: • Markov Assumption agnostic RL . • Variance Reduction Markovian Assumption . • Safety: “desire” rule-based “hard constraints” . safety : • • defensive/aggressive • .
- 35. ? • Curse of Dimensionality” • Semantic Abstraction: driving policy options graph . • Continuous Trajectory Planner: we use a dynamic programming planning module that optimizes the “desires” subject to the “hard constraints” on safety. • : self-play . • : policies .
- 36. Options Graph • Desires . • children child policy . • state space Desires subset output space . • Reducing variance and sample complexity of the learning problem. possible option graph for the double merge scenario
- 37. Conclusions • Driving in realistic dense traffic scenes calls • agent • agent : defensive/aggressive tradeoff • “The rules of breaking the rules …” • • • safety component • Agnostic RL with no Markovian assumption for the Multi-Agent setting • Semantic abstraction using the Options Graph • Self-play •
- 38. Reference • Reinforcement learning: An introduction, Barto and Sutton • https://www.youtube.com/watch?v=n8T7A3wqH3Q • www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html • https://www.bloomberg.com/news/articles/2015-12-18/humans-are- slamming-into-driverless-cars-and-exposing-a-key-flaw • https://media.nips.cc/Conferences/2016/Slides/6198-Slides.pdf • https://www.cs.huji.ac.il/~shais/ • https://www.dropbox.com/s136nbndtdyehtgidoubleMerge.m4v?dl=0