![Different solutions for behaviors in games
11
Code Learn
Machine Learning AI
“Teach the machines
to do [stuff]”
— Imitation
Learning
— Reinforcement
Learning
Reactive AI
“Write a lot of explicit
code”
— C#
— Behavior Trees
— State Machines
Unity
ML-Agents](https://image.slidesharecdn.com/w19061910webinar-ml-agents-190620045559/85/weekly-AI-tech-talk-85-ml-agents-Enabling-Learned-Behaviors-with-Reinforcement-Learning-11-320.jpg)
weekly AI tech talk #85 ml-agents Enabling Learned Behaviors with Reinforcement Learning
- 1. GenerativeArt—MadewithUnity Enabling Learned Behaviors with Reinforcement Learning Ervin Teng Research Engineer, Machine Learning
- 2. Advances in Machine Learning and Gaming DOTA 2 Valve Corporation Starcraft II Blizzard Entertainment Go (Game) 2
- 3. Advances in Machine Learning and Gaming DOTA 2 Valve Corporation Starcraft II Blizzard Entertainment Go (Game) 3
- 5. Behaviors 5
- 7. Behaviors 7 NPC and PC Squad of NPCs and PCs
- 8. Behaviors 8 NPC and PC Squad of NPCs and PCs Game world (storytelling, interactive narration) Player Assistant
- 9. Behaviors 9 NPC and PC Squad of NPCs and PCs Game world (storytelling, interactive narration) Game objects Player Assistant
- 10. Different solutions for behaviors in games 10 Reactive AI “Write a lot of explicit code” — C# — Behavior Trees — State Machines Code
- 11. Different solutions for behaviors in games 11 Code Learn Machine Learning AI “Teach the machines to do [stuff]” — Imitation Learning — Reinforcement Learning Reactive AI “Write a lot of explicit code” — C# — Behavior Trees — State Machines Unity ML-Agents
- 12. What are Agents? Agents perform actions in their environment. At any given moment, the environment is in a certain state. From the state, Agents collect observations. From observations, Agents take actions according to their policy. 12
- 13. What are Agents? Agents perform actions in their environment. At any given moment, the environment is in a certain state. From the state, Agents collect observations. From observations, Agents take actions according to their policy. Behavior 13
- 14. Illustration of Agents taking action Observations Location Velocity Rotation Camera Ammo Count Enemy’s Proximity Policy Actions Move Jump Turn Shoot Duck Repeat14
- 15. How can Agents be used in games? 15
- 16. How can ML-Agents be used in games? Game Testing Game Balancing PvP Bots 3 Types of Agents Uses in Gaming v 1 Player Agent v From the POV of the player playing the game 16
- 17. How can ML-Agents be used in games? Game Testing Game Balancing 3 Types of Agents Uses in Gaming v 1 Player Agent v From the POV of the player playing the game v 2 Non Player Agent v Unplayable characters or objects in the game Enemies Companions Passerby Characters 17
- 18. How can ML-Agents be used in games? Game Testing Game Balancing PvP Bot 3 Types of Agents Uses in Gaming v 1 Player Agent v From the POV of the player playing the game v 2 Non Player Agent v Unplayable characters or objects in the game v 3 Invisible Agent v Scene itself or other experiences not seen by the player Enemies Companions Passerby Characters Content Generation Difficulty Tuning Player Engagement 18
- 19. How do we create Agents? How do we create the Policy? 19
- 20. How do we create Agents? How do we create the Policy? Unity Machine Learning Agents An Open Source Toolkit for training intelligent agents using the latest in Deep Reinforcement Learning. 20
- 21. How do we put this into a game? 21
- 22. How do we put this into a game? The Unity Inference Engine Run any ML-Agents neural network model on the CPU or GPU on any Unity supported platform 22
- 23. ML-Agents + Inference Engine Integrate Train Embed 23
- 24. Integrate Train Embed 24 ML-Agents + Inference Engine
- 25. Integrate ML-Agents into your Unity project Agent A1 Agent A2 Brain External Communicator Python APIScene or level Academy 25
- 26. Different ways to set up Single Brain Controlling Multiple Clones of the Same Type of Agent Brain 26
- 27. Different ways to set up Brain Brain Brain Multiple Brains Controlling Multiple Clones of Different Types of Agents 27
- 28. Different ways to set up Brain Brain Brain Brain 28
- 29. Integrate Train Embed 29 ML-Agents + Inference Engine
- 30. Training methods v 1 v 2Reinforcement Learning Imitation Learning 30
- 31. Training methods v ● Agents learn through EXPERIENCE 1 v 2Reinforcement Learning Imitation Learning 31
- 32. Training methods v ● Agents learn through EXPERIENCE ● A lot of trial and error until optimal 1 v 2Reinforcement Learning Imitation Learning 32
- 33. Training methods v ● Agents learn through EXPERIENCE ● A lot of trial and error until optimal ● Simulation can be sped up greatly 1 v 2Reinforcement Learning Imitation Learning 33
- 34. Training methods v ● Agents learn through EXPERIENCE ● A lot of trial and error until optimal ● Simulation can be sped up greatly ● “Super-human” abilities 1 v 2Reinforcement Learning Imitation Learning 34
- 36. Training methods v 2 Imitation Learning 36 v ● Agents learn through EXPERIENCE ● A lot of trial and error until optimal ● Simulation can be sped up greatly ● “Super-human” abilities 1 Reinforcement Learning
- 37. Training methods v ● Agents learn through DEMONSTRATIONS 2 Imitation Learning 37 v ● Agents learn through EXPERIENCE ● A lot of trial and error until optimal ● Simulation can be sped up greatly ● “Super-human” abilities 1 Reinforcement Learning
- 38. Training methods v ● Agents learn through DEMONSTRATIONS ● Mimics behavior quickly 2 Imitation Learning 38 v ● Agents learn through EXPERIENCE ● A lot of trial and error until optimal ● Simulation can be sped up greatly ● “Super-human” abilities 1 Reinforcement Learning
- 39. Training methods v ● Agents learn through DEMONSTRATIONS ● Mimics behavior quickly ● Recorded demonstrations needed 2 Imitation Learning 39 v ● Agents learn through EXPERIENCE ● A lot of trial and error until optimal ● Simulation can be sped up greatly ● “Super-human” abilities 1 Reinforcement Learning
- 40. Training methods v ● Agents learn through DEMONSTRATIONS ● Mimics behavior quickly ● Recorded demonstrations needed ● “Human-like” abilities 2 Imitation Learning 40 v ● Agents learn through EXPERIENCE ● A lot of trial and error until optimal ● Simulation can be sped up greatly ● “Super-human” abilities 1 Reinforcement Learning
- 42. Other algorithms, methods and examples 42 • Curiosity • Long-term / Short-term Memory • Curriculum Learning • On-Demand Decision Making • Concurrent Unity instances • Supports OpenAI Gym • Supports Google Dopamine • Supports AWS, Azure, GCP, Docker • …And more coming soon! All included in the Unity Machine Learning Agents Toolkit
- 43. Integrate Train Embed 43 ML-Agents + Inference Engine
- 47. After training Brain Agent A1 Agent A2 Agent A2 47
- 48. But what is under the hood? Brain 48 ?
- 49. But what is under the hood? Brain 49 v ● ~100K of parameters Neural Network ● Large Matrix Operations ● Multiple Agents Execution
- 50. Why is it hard to embed the Brain? 50 Brain
- 51. Why is it hard to embed the Brain? Brain ● Sample Brain used by ML-Agents uses 26 matrix operations 51
- 52. Why is it hard to embed the Brain? Brain ● Sample Brain used by ML-Agents uses 26 matrix operations ● Game needs to run graphics, physics and other systems in the game 52
- 53. Why is it hard to embed the Brain? Brain ● Sample Brain used by ML-Agents uses 26 matrix operations ● Game needs to run graphics, physics and other systems in the game ● FPS minimum threshold, hardware support, etc … 53
- 55. Unity Inference Engine 5555 v Burst compiler IL2CPP Compute shaders High Performance
- 56. Unity Inference Engine 5656 v Burst compiler IL2CPP Compute Shaders High Performance Generic platform-agnostic API
- 57. Unity Inference Engine 5757 v Burst compiler IL2CPP Compute Shaders High Performance CPU/GPU inference Generic platform-agnostic API
- 58. Unity Inference Engine 5858 v Burst compiler IL2CPP Compute Shaders High Performance Optimized for different platforms CPU/GPU inference Generic platform-agnostic API
- 59. Unity Inference Engine 5959 v Burst compiler IL2CPP Compute Shaders High Performance Optimized for different platforms CPU/GPU inference Fully integrated with ML-Agents Generic platform-agnostic API
- 63. Integrate Train Embed 63 ML-Agents + Inference Engine
- 64. 64
- 65. Jam City Snoopy Pop 65
- 66. Jam City Snoopy Pop + ML-Agents 66
- 67. Jam City Snoopy Pop + ML-Agents 67
- 68. Jam City Snoopy Pop + ML-Agents 68
- 69. github.com/Unity-Technologies/ml-agents Contact Us ML-Agents@Unity3d.com Thank You! ervin@unity3d.com