Practical AI in Games

Editor's Notes

• #2: “Practical AI in Games” talk is an introductory material for students and programmers aspiring for developing AI for games. Talk is split into 2 parts - first part provides an overview on popular AI approaches in games and second part gives an outlook on technology that might be relevant for games AI in the future. I gave this talk at Vilnius University as a guest speaker in the late October 2016. Thanks Žilvinas Ledas for invitation. Lingo: NPC is a Non-Player Character. Synonymous to Actor.
• #4: Of course it is a controversial statement and does not apply to all games. It is here to grab your attention…
• #5: … and to draw line between “scientific AI”, human intelligence and what we actually use in games.
• #6: This talk is split into 2 parts. First part is more practical and covers set of approaches that are most used in today games.
• #7: Second part aims to give an outlook about future of AI and what young professionals should keep an eye.
• #8: Traditional games like Chess and Go have well defined discreet states. In theory computer AI could beat the game by looking through all possible states and finding the best solution. In practice however it is extremely computationally expensive and memory hungry task. Search space is number of all possible moves (actions) in such game. However state in most modern games is far from discreet (unit can move just “a little bit”) and have complex rules - dramatically increasing search space.
• #9: Although task at hand is extremely complex - complete and optimal “solutions” belong more to the domain of “scientific AI”. Instead as game developers we need to entertain the player!
• #10: “Glued to Games: How Video Games Draw Us In and Hold Us Spellbound” is a great book disassembling what does fun actually mean. https://www.amazon.com/Glued-Games-Video-Spellbound-Directions/dp/0313362246
• #11: Take a look at the image - the player have thrown grenade just a couple of frames before, however NPC (Non-Player Characters) are reacting already - jumping aside or hiding! This is a good example of responsive NPC behavior. Rather simple, but well timed behavior is better than smart, but late one.
• #12: Players must relate to and understand behavior of NPCs to fully appreciate it. Animations, audio and visual clues are at least as important as AI code behind NPC decisions.
• #14: In the end it is all about making good game. If your game designer (1) does not understand what AI code can achieve, (2) thinks AI is not predictable enough, (3) has insufficient tools or (4) code is not solving tasks at hand - he will not be able to tap into the full potential of AI. You wasted your time!
• #15: ms - millisecond, is one thousandth of a second. You can realistically expect to spend on AI somewhere between 1 and 5 milliseconds during a single frame. Not more than that! In other words, if you would turn off everything and just keep AI code running - your game should be running between 1000 and 200 frames per second. Unless you are making turn-based strategy game of course :) Why? Keeping constant framerate is very important for player immersion, reaction time and for perceived quality of the game. And it is even worse with VR - people will literally puke when framerate drops. Therefore tasks that are necessary to run and complete during each frame are of highest priority, such are - graphics, physics, animations, audio, input processing, network, etc. Note that in order to keep Graphics Processors (GPU) busy lots of work needs to be executed on the main CPU first (even with modern graphics APIs like DirectX12, Vulkan or Metal). Modern engines will strive to distribute the work across all cores too. In other words CPU with all its cores is quite busy in games. Good news for AI some decisions can be spread over several frames to execute. Slow human brains often won’t notice.
• #16: For the sake of broadening our horizon - let’s look at AI “stack” in real-life scenarios from Robitics and self-driving Cars. Different layers are executed at different frequencies (for performance reasons). Lower layers that require immediate reaction and are critical for survival are executed more often. Higher layers are responsible for long term planning. If condition changes - elk jumps in front of the car - lower layers will react immediately and might trigger higher layers to re-evaluate situation / re-plan further actions. Navigation - responsible for finding efficient route to the gaol - decides on which road intersection to turn. Does not concern itself with details like driving on a straight road however. Behavior - responsible for optimizing driving style like gas consumption vs speed. Driver - execute physical actions. Motion Planner - plans ahead how to drive in the next intersection, when to change lanes, etc. Trajectory Planner - calculates actual trajectory given car speed, physical parameters. Reacts to sudden changes like appearance of unpredicted obstacles.
• #17: AI “stack” in games is somewhat similar. Behavior - gameplay level decisions: attack, seek cover, reload, etc. Navigation - finds route to the next location as commanded by Behavior. Animation - play animations accordingly to NPC actions and intentions. Trajectory - moves NPC from frame to frame as commanded by Navigation and in accordance with Animations layer. Reacts to dynamic obstacles. Next we are going to look at each layer separately.
• #18: Navigation aka Pathfinding - finds route between two locations. There are different ways how to subdivide world into set of discrete interconnected locations. Read more: https://en.wikipedia.org/wiki/Pathfinding
• #19: Grid is a 2D array covering the world. Grid locations can be blocked or open for navigation.
• #20: NavMesh or Navigation Mesh is a simplified geometrical representation of the world. Polygons represent locations accessible for navigation. Actor can move on top of NavMesh polygon freely without encountering any obstacles. Read more: https://en.wikipedia.org/wiki/Navigation_mesh
• #21: Waypoint graph - nodes represent locations accessible for navigation. Edges connecting nodes represent paths between locations that are obstacle free. Read more: https://en.wikipedia.org/wiki/Waypoint
• #22: Grid naturally suit 2D or 2.5D worlds. Dynamic obstacles can be represented relatively easy by changing state of grid elements as occupied by NPC (Non-Player Character). Bigger NPC units usually will cover several elements of the grid.
• #23: NavMesh naturally suit 3D worlds. Notice on the image that NavMesh (blue color) is narrower than the actual passages - size of the actor has to be implicitly “baked” into the NavMesh, otherwise actor would stick through the walls while walking. Producing NavMesh from dense world geometry can be a complex task, luckily there are robust libraries like: http://masagroup.github.io/recastdetour/
• #24: Waypoints naturally suit 3D worlds. Waypoints provide easy control for designers to define accessible areas for NPC, but that of course means more manual work to setup. Nowadays waypoints are falling out of favor and are usually replaced with NavMesh due to automated process.
• #26: All described approaches work best for (1) static world, (2) where all NPCs are roughly the same size and (3) can turn on a spot easily.
• #27: It is quite common to combine several data structures in one game. For instance in RPG game NavMesh approach can be used for navigation inside the houses, but Waypoints could be used to represent the outside world. For Strategy game multiple grids of different resolution can overlap in order to handle dynamic obstacles where it matters more, etc.
• #28: Read more: http://www.red3d.com/cwr/steer/gdc99/
• #29: Navigation layer alone provides us with very rough path (red line). If actor would follow it blindly it would look very unnatural and “robotic”.
• #30: One task for Steering layer is to take information from Navigation and create more believable path (green line).
• #31: Another task is to avoid dynamic obstacles that Navigation does not want to deal with. It is important to remember that while steering can avoid dynamic obstacles it can lead NPC far away from the path planned by Navigation. When that happens Steering will usually trigger Navigation to reevaluate the path starting from the current location. For both Path simplification and Dynamic obstacle avoidance engine will fire additional (short) ray-queries to check, if path is blocked.
• #32: Example of Steering for NPC character following the player (“Buddy follow”) while staying behind him, but visible through the camera. Yellow spheres depict possible locations, red arrows point to locations that are blocked by the rock. Worth reading: http://allenchou.net/2016/05/a-brain-dump-of-what-i-worked-on-for-uncharted-4/
• #34: Read: https://en.wikipedia.org/wiki/Finite-state_machine FSM is a graph where: nodes are states of NPC “brain” edges are conditions that will make “brain” jump (transition) from one state to another NPC actions are associated with both states and transitions FSM is a simple “map of the brain”. What makes NPC act - actions assigned to every state or transition. As AI designer you would layout FSM, implement various actions with script and assign those actions to FSM states and transitions.
• #36: Example of script code that is attached to FSM and drives actual behavior - in this case spawning new particle and pushing it away with force. OnStateEnter, OnStateExit - functions that execute once when FSM enters/exits particular state OnStateUpdate - function is executed every frame while FSM is in a particular state.
• #37: Example of FSM from a real game.
• #38: FSM are great as a starting point in games AI. Hierarchical FSM: http://aigamedev.com/open/article/hfsm-gist/
• #39: Lets take classic Pacman game as an example and see how AI is implemented there. Watch Pacman gameplay: https://www.youtube.com/watch?v=uswzriFIf_k
• #40: More about Pacman gamedesign, ghost behavior, etc: http://gameinternals.com/post/2072558330/understanding-pac-man-ghost-behavior http://www.gamasutra.com/view/feature/3938/the_pacman_dossier.php https://www.khanacademy.org/computer-programming/pac-man-ghost-ai-visualisation/6163767378051072
• #41: Although Pacman is small and old game - it is a great example of the good design principles for AI. Keep close attention to ghosts in Pacman: just couple of frames before making turn, they will turn their eyes towards intended direction - that gives a glimpse of intent ghosts reverse direction when important state of the “brain” changes - that gives opportunity for player to know of change in behavior they change speed of movement and color when they become “eatable” - player knows whenever ghosts are dangerous or not
• #43: Ghosts in Pacman have 4 states of the “brain”. In “Scatter” state they go towards predefined corners and stay there for a short period of time.
• #44: In “Attack” state every ghost has slightly different behavior that affects Navigation. Navigation in Pacman is extremely simple, just a single goal that ghost is trying to reach. In case of a red ghost Navigation goal is player itself. In case of a pink ghost - goal is always slightly ahead of the player, etc. Steering in Pacman is extremely simple too - at every intersection ghosts turn in the direction that would bring them closer to the goal commanded by Navigation. Read: http://gameinternals.com/post/2072558330/understanding-pac-man-ghost-behavior
• #45: Visualization of FSM behind ghost “brain”.
• #48: Read: https://en.wikipedia.org/wiki/Behavior_tree BT is a tree where: branches are conditions leaves are actions. With BT we no longer operate with predefined “brain” states like we did with FSM. Instead BT “brain” will pick the best possible action to execute based on the hierarchy of conditions. This hierarchy of conditions are parent nodes of any particular action.
• #49: BT execution is started from the root node and descends down the tree. Depending on the type of the node - either (Selector) only one of its children or (Sequence) all children will be executed. Leaf nodes (Action) will make NPC act. In the image below grey nodes are conditions that were false, therefore actions (“Seek” action in this case) will not be executed. Green nodes are under conditions that are true during this frame, therefore actions below them will be executed. NPC is going to “Flee”.
• #50: This example will execute sequence of actions starting with selecting random point to travel and then wait until the movement succeeds.
• #52: BT is a very common approach in games today.
• #53: Read: http://www.gameaipro.com/GameAIPro/GameAIPro_Chapter09_An_Introduction_to_Utility_Theory.pdf Utility is a list of actions and associated utility scores. Utility scorer simply tells us how useful particular action is at the given moment. “Brain” will pick and execute the best possible action with highest utility score.
• #54: To simplify Utility AI is a more generic and flexible representation than BT. BT can be transformed into Utility AI representation. This example shows how leafs of BT become list of actions in Utility AI representation…
• #55: … and conditions become utility scorers. Scorers are implemented as small code snippets than evaluate state of our NPC and world around and return single value - score. Read more: http://www.gamasutra.com/blogs/JakobRasmussen/20160427/271188/Are_Behavior_Trees_a_Thing_of_the_Past.php
• #56: Example: http://apexgametools.com/learn/apex-utility-ai-documentation/apex-utility-ai-unity-survival-shooter/
• #57: Utility AI are gaining popularity in the game development. Utility AI allows to create more modular and emergent behavior than FSM or BT.
• #58: For more detail comparison: http://intrinsicalgorithm.com/IAonAI/2012/11/ai-architectures-a-culinary-guide-gdmag-article/
• #59: All discussed approaches purely react to current situation. Can we make AI to anticipate future and plan for it? It is called “deliberate” approach. More on planning: http://aigamedev.com/open/review/planning-in-games/
• #60: Influence Maps can be seen as a simple way to add some planning ahead into otherwise reactive AI. Influence Maps modifies how Navigation is searching for the best path. The best path becomes not the shortest path, but path which satisfies specific NPC goal better.
• #61: Excerpt from the great presentation about Killzone AI: https://www.guerrilla-games.com/read/killzones-ai-dynamic-procedural-tactics Killzone talk perfectly explains the concept - I decided to “borrow” their slides instead of making poor copy of this great example.
• #71: Read more: http://www.gamedev.net/page/resources/_/technical/artificial-intelligence/the-core-mechanics-of-influence-mapping-r2799
• #75: Make a plan of actions towards desired goal. Execute and re-plan when necessary. Idea should remind you pathfinding, except looking for path in action space instead of geometrical space. Read more: http://alumni.media.mit.edu/~jorkin/goap.html Approach that GOAP (Goal Oriented Action Planning) is based upon is called STRIPS in scientific literature: https://en.wikipedia.org/wiki/STRIPS. Slightly more advanced version of STRIPS is called SHOP: https://www.cs.umd.edu/projects/shop/description.html
• #77: It is not easy however to evaluate how particular action will affect the state of the world - search space can “explode” (see early slide on Chess/Go/Starcraft). This is somewhat less explored area in games - we need to do more work here.
• #78: So far we have looked at “engineer” approaches to AI - rational, step-by-step, hardcoded. They do not learn anything or even try to analyze opponents play style. Unsurprisingly that is NOT how our brain works :) Our brains can learn, adapt to situations and find new ingenious solutions to problems. Can we learn from brain to build better machines and game AI? Read more: https://en.wikipedia.org/wiki/Cerebral_cortex
• #79: Read more: https://en.wikipedia.org/wiki/Neuron
• #80: Magnified and colored slice of human brain - ~2 mm deep slice. Fat dots are “bodies” of neuron cells, lines are connections between neurons.
• #81: Computer generated reconstruction of connections between neurons in cerebral cortex.
• #82: Why study neuroscience? It will teach you about different type of computations - highly parallel, highly scalable with emphasize on learning instead of hardcoding. Meanwhile modern day sequential code that executes just on handful of cores is going to be less and less relevant. In the future hardcoding or as we call it today “programming” for chips based on classical Von Neumann architectures will become as prized and eventful as soldiering radio sets today!
• #83: Read more: https://en.wikipedia.org/wiki/Artificial_neuron
• #84: Lets look at development of human built systems that take some inspiration from brains. Photo - Facebook GPU based computer purposed for Machine Learning tasks. More: https://code.facebook.com/posts/1687861518126048/facebook-to-open-source-ai-hardware-design/
• #85: Google took GPU design and removed “parts” unnecessary for Machine Learning. More: https://en.wikipedia.org/wiki/Tensor_processing_unit
• #87: FPGA - not new hardware design, but recently gaining more interest from Machine Learning. Hard to program though. More: https://en.wikipedia.org/wiki/Field-programmable_gate_array
• #88: Currently implemented on top of FPGA. One step towards turning FPGA type of design into Machine Learning friendly. More: http://www.clement.farabet.net/research.html#neuflow
• #89: Neuromorphic chips build neurons directly in hardware. Building neurons in hardware is not hard, however interconnecting lots of them is the main challenge. Sometimes called - “network on chip”. Read more: https://en.wikipedia.org/wiki/TrueNorth Neuromorphic approach in general: https://en.wikipedia.org/wiki/Neuromorphic_engineering
• #91: Single chip has 4096 cores. Each core simulates one neuron, stores information about its connection and has network router to communicate with other neurons.
• #92: TEDxCERN talk by Madeline Lancaster: https://www.youtube.com/watch?v=EjiWRINEatQ Cerebral Organoid is a tiny brain grown in the lab. Neurons in cerebral organoids seems to be functional, however connection and activity seems to be chaotic comparing with “real” brains. Some promising breakthroughs are happening in the last 2 years in this area!
• #97: So far cerebral organoids are not fully developed brains comparing to alive organisms. Nor they are connected to any body - do not get any meaningful information and do not do any meaningful computations. I would guess that in the future (10 years?) we will learn how to augment cerebral organoids with computers - closing feedback loop and training them. Biological AI co-processor, if you wish. Most likely in the process we will discover more about how biological brain learn. And improve our Machine Learning approaches.
• #98: 2016… but really shift was happening in the last 4-5 years.