World of Tanks* 1.0+: Enriching Gamers Experience with Multicore Optimized Physics and Graphics

Editor's Notes

• #3: Let me to introduce our presenters today. My name is Philipp Gerasimov, I work at Intel’s Developer Relations Division in GAME team, helping game developers worldwide to enable Intel Platforms and optimize their games. Here is my colleague Alexei Fedorov, he works at Intel ® TBB team within Developer Products Division and he’s one of the key Intel TBB developers. And last but not least our special guest from Wargaming.net, Bronislav Sviglo, who is the Rendering Team Lead for World of Tanks.
• #4: Here is the agenda for our today presentation.
• #5: We going to start with looking at the modern CPUs.
• #9: Hi! My name is Bronislav. I am a lead graphics engineer at Wargaming Minsk Studio and I’ve been working there for about 10 years now. Before we start talking about the near to distance future of multi-core computing in World of Tanks game…. …I’m glad to present to you our biggest ever World of Tanks update in terms of graphics and technology. It’s 1.0 version and it’s based on our own in house engine – Core Engine ©. Let’s take a look at our launch trailer of this huge update (World of Tanks 1.0).
• #11: In terms of graphics our game supports wide range of hardware. I’d like to remind you that 2004 is actually the year when such significant games for the whole industry like DooM 3, HL2, FarCry1 were released. So, we’re still capable to launch on such an outdated hardware which is by itself in my opinion very impressive . We have two graphics pipelines to support this range of hardware and, unfortunately, we have to support D3D9 and WinXP (China market is still huge). Despite this facts our game has faced several important graphics evolution points: - 0.8.x version brought an additional advanced render pipeline with deferred shading, decals, HDR rendering, better post-processing and many more. - 0.9.x version is very important for us because it brought PB rendering approach, better temporal anti-aliasing, various important improvements into post-processing pipeline and other. In the 1.0 update we’ve made all of the content from scratch so players will see a completely new game in terms of visual aspects & technology.
• #12: Before we move forward I’d like to show you some pretty pictures where you can see the difference between the old map and new one. I’d also like to emphasize that in terms of gameplay there is no difference. The map basically is the same.
• #13: The fact that right now, 60% of our audience has only 2 physical cores requires us to be very careful with the process of adding new computationally heavy features. And we also have to be very passionate about having very well optimized single threaded code including vector CPU instructions like SSE, AVX, and so on. Nowadays, our engine is heavily optimized for 2 to 4 cores machines and logical cores are also very helpful for such ongoing activities like IO, streaming, render thread and so on. But our game is a long running product and it should be prepared for the future of widespread multi-core machines among our players. I mean 4+ cores machines. This is the reason why we’re working on it right now…
• #14: Being able to use all available CPU cores liberates lots of performance which does not always need to lead to just better frame rate. Good, stable performance is ok, but we often can use this additional budget to improve the overall picture quality or make more advanced simulations. … And today we’re going to talk about R&D features which heavily rely on multi-core hardware. They all are in progress and may be a part of next releases: In the first topic I’m going to briefly overview how we’re doing destructions in our game Second topic is how players with multicore machine, may have more robust and accurate tank tread simulation And of course how we make our graphics engine perform concurrently
• #15: There are lots of ways to make your engine concurrent: Old approaches with set of dedicated threads for each major engine sub-system (audio, render, physics, simulation, etc.) Task based approach where there is no main execution thread, all work is subdivided into small tasks and they are executed on the pool of threads. Mix of those approaches, where on the one hand, we have job system for task execution and, on the other, we have long running dedicated threads for ongoing activities like streaming, IO, render thread etc. We’ve selected the third one. Next step is to chose good threading API and job system…
• #16: Of course there are lots of job systems or task schedulers in public domain but they all have some limitations, either bad API or lack of some advanced features. But for our project we’d like to have easy to use, feature rich job system with two main types of parallelism (concurrency): functional (task) and data (I’ll speak later on this) And of course an ability to have a good support is essential. Fortunately, we’ve found such library and it’s Intel’s Threading Building Blocks. Mike is going to give you a quick overview of this library…
• #17: And now we’ll take a look at technical demo.
• #23: We’re going to start with quick explanation how we’re doing destructions in our game.
• #24: Our destructions solution is based on Havok third party library and to be correct on the destruction module. This physics engine serves as a physics simulation and collision detection system. And it has out of the box multithreading support which allows us to significantly speedup the performance of physics simulation.
• #25: To achieve good performance without sacrificing gameplay, our engine uses two different scenes inside havok library: for accurate collision queries and for destruction interaction. Collision scene uses compressed meshes of triangles and they almost identical to the actual model’s geometry. It gives very accurate collision results, but it’s too slow for the physics simulation. Destruction scene uses simplified convex decomposed models, this approach is very fast for physical interaction and decomposed collision models rather lightweight because of their inaccuracy. This separation gave much more freedom to our artists and lowered complexity of the content production.
• #26: This is the high level look at our destruction system architecture based on the havok library. We have main thread which invokes two threads: Input Output system and Simulation Simulation is divided into number of tasks which are executed on the number of internal havok threads. Collision may be performed at any time and completely thread safe
• #27: On this video you can see lots of complex destructions. Some of them are already part of 1.0 update, but we’re working on even more advanced destructions.
• #28: To sum it up I’d say that we’re really happy with the quality of our simulation and this feature is a good example how to discover the potential of multi-core machines. In the future we plan to substitute internal havok scheduler with our own based on TBB. This should help minimize the problem with threads oversubscriptions.
• #29: The next topic is especially popular for our game about heavy machinery. High quality simulation of tank treads is very important when you’re driving a tank.
• #30: Currently we have two major types of treads in the 1.0 release. On the minimal settings we just render a skinned mesh of the tread, and scroll the texture along it when the tank moves. Nothing special. “Spline tread” approach uses artist defined control points to construct a spline, and then places along it tread segments. Thus, we achieve the next level of geometric quality of the tread. Tread animation is also improved because we can vibrate the spline’s control points between wheels to simulate the change of the tread’s tension.
• #31: Spline tracks look very good compared to the just skinned mesh, but, unfortunately, they have several flaws we want to address: The shape of a spline is hard to control. Because of that, when we vibrate its control points some spline’s curves could penetrate the terrain and other obstacles Spline also doesn’t allow us to create hard edges to properly place the tread on blocky objects We vibrate spline control points using invisible springs. And the way the springs vibrate is quite unpredictable when you try to tune it. This results in a very difficult tread setup process for the artist.
• #32: These problems motivated us to create a tread which would be physically simulated and produces better visual quality. We did several experiments with various physical models, considering quality and performance cost. The choice was to use the spring chain. Springs were less prone to jitter in our collision environment. Of course they may stretch, which is not physically correct, but that stretch is gradual along the tread, thus visually not noticeable. A nice bonus of the springs is that they naturally produces animations when a tank swings. This gives treads a much more natural look than we could achieve with the spline. The artists will also be free from tuning the tread animation, because it’s fully procedural. And last but not least, during the simulation we could collide joints of the springs with the collision scene, producing much more detailed collision effects.
• #33: Eventual solution is the following: The spring chain doesn’t rotate around the tank. We snap it at two points at the front and back wheels. So it only swings with the tank and collides with the obstacles, producing the shape along which we scroll visible tread segments. The different parts of the track have different tension depending the tank moves forward or backward. To simulate this effect we divide the tread in 4 parts and change length of springs for each part separately to control the tension along the whole tread. At the collision phase we…: Calculate 1D height field underneath each tread by doing several collisions tests with the environment Then we test each spring joint against it Collisions with the wheels are just a point against a disc The result is stable enough.
• #34: Each tread is simulated as a job for TBB. To increase concurrency we also exploit data parallelism by using parallel_for and parallel_reduce functionality. 1) So, due to heavy usage of multi-threading, tank treads simulation may even not affect resulting performance.
• #35: On the video you can see all of features of new tank treads like: Sagging effect, where tread may sag if there is an empty space beneath it Tension effect, where the different parts of the tank tread have different tension depending on the tank moves forward or bachward … and of course much more accurate collisions with the environment
• #36: In summary I’d like to say that we achieved high quality tank treads simulation which such cool features like ability to sag below the tank, better interaction with the environment and so on. In the future we plan to concentrate on tread simulation for some exotic tank types. And of course on better fine grain tuning of concurrent execution.
• #37: The last but not least topic is about concurrent rendering. Often rendering is one of the difficult things to make concurrent. It becomes even more challenging if you have to support wide range of hardware and various API, especially old one like DX9 which wasn’t created with multi-threading in mind at all. Or DX11 which was, but unsuccessfully, Deferred Context is very pure concept and it just didn’t work. Vulkan, Metal and DX12 is way better, but as I previously mentioned our audience is fragmented towards old low-end hardware and we have to deal with it. Here we’re going to talk about how to enable concurrent rendering while still being able to work on outdated hardware.
• #38: And this is our agenda about concurrent rendering: First of all we’re going to talk about our past even before 1.0 release. Then I’ll give you an overview about what is Abstract Rendering Interface (later ARI) and how it helps us with concurrency How we do concurrency in rendering And how different types of parallelism help us to achieve good performance and scalability
• #39: World of Tanks came out as a DirectX 9 game back in 2010 and it was powered by BigWorld engine. There was strict separation between tick and draw methods. And almost all of the subsystems directly called DirectX 9 API.
• #40: Optimal usage of the GPU was guaranteed only due to a correct order of execution of render sub systems. Material system was based on D3D Effect Framework which didn’t support concurrency in any way, but was very user friendly (we even decided to use a similar approach in our own effect framework called wgfx). And ability to add new Graphics API was possible only with overcomplicating the existing code base.
• #41: It took us a long time to overcome those limitations and it was challenging because World Of Tanks is an online game and meanwhile the project was evolving. Patch 9.15 was released with lots of under the hood optimizations and we moved away from direct calls of graphics API towards Abstract Rendering Interface. ARI allowed us to uniformly use different graphics API without losing performance. During the process of creating ARI we were inspired by upcoming at that moment low level graphics APIs like Mantle and DX12, and tried our best to make architecture of ARI very similar to them. (I’ll speak later on it) So, basically with ARI we were able to gather the render commands into the software command buffers. Those gathered command buffers then were submitted to render thread which in turns was in charge of processing them and making direct calls to an appropriate graphics API (at that moment they were either DX9 or DX11).
• #42: Wgfx effect framework was introduced as a faster, multicore aware alternative to D3D effect framework with lots of under the hood optimizations and customized API. Despite the fact, that we had separate render thread for processing software command buffers, we still gathered render commands only in the main thread. But even that gave us up to 30% boost in performance back then. Thus, the next step is to gather render commands concurrently from different threads.
• #43: As I previously said when we were creating the ARI framework we were inspired by the low-level graphics API like Mantle and DX12 at that moment. So, ARI is based on the following cornerstones: Command list contains a list of render commands like draw, clear, update, copy-resource, etc. which is supposed to be later executed on the render thread. Device Interface is basically our “driver” between our user code and specific graphics API (like DX9, DX11 or DX12) Resource describes any kind of graphics resource like buffer, texture, query, graphics pipeline state, and so on.
• #44: Concurrency is achieved by submitting a list of commands into separate render thread. To be able to record a list of commands for later execution on render thread, each command should be simple POD (plain old data) object which contains all required information for later usage and should not have any dependency on any other command. Command list preparation may be done concurrently from different threads because each command list doesn’t depend on each other. The correct result is guaranteed by a correct order of submitting command list into separate render thread. Example of commands may be draw, clear or query, and so on. I’d like to point out that there are some kind of commands like update or copy-resource which require to have manual memory management and life time control of the content you provide to be a part of these commands. So, when you try to upload texture on GPU you usually don’t want to copy the whole texture content right into the command, instead you just provide pointer or link on the memory to copy from, and guarantee that this memory remains valid till the end of this frame.
• #45: ARI interface for device access is divided into three categories based on the access pattern: Free-threaded or Thread- independent interface, is used for operations which may be executed any time from any thread. With this interface we can perform resource creation and deletion, receiving adapter state and so on. Single-threaded interface, is mostly related to the command list operations like filling list with commands, compiling it to native commands in case of DX12, and so on. Creation thread only interface, is a legacy thing and it’s about limitations of existing old graphics API like DX9, DX11 to perform special operations like present or reset on a render device.
• #46: Concurrent rendering in our game is achieved by TBB flow graph, which manages dependencies between render tasks and defines the actual order of submission gathered command lists into the render thread. Render tasks, which represent independent command lists for each render sub-system of the engine And separate render thread which consumes software command buffers and makes native calls to graphics APIs Notice, how significantly concurrent rendering increase usage of all available CPU cores.
• #47: On the high-level render frame is managed by TBB flow graph which acts as a render task scheduler. Flow graph consists of number of nodes and edges. Each node of the graph is a big chunk of render work within one sub system. Edges form dependencies between different sub-systems or key renderer stages. Here you can see various examples of the render nodes, so you can have an idea what they might be.
• #48: And on this diagram you can see example how high-level frame render graph may look like. Having such high level architecture of the frame is very helpful for everyone in the team to quickly understand what is the actual flow or order of execution and what sub-systems depends on each other. More over this code is really easy to modify and optimize because all high-level information about course grain tasks are located in one place. Task based or functional concurrency works well here because what you need to do here to enable concurrency is just wrap up each task and add dependency and that’s all.
• #49: As I previously mentioned our engine uses TBB flow graph to gather command list for various sub-systems in parallel and then flush them in a predefined order into the render thread. Unfortunately, if we do only one flush to the render thread per frame it will make render thread stay idle. 1) So, we have to perform intermediate flushes to give the render thread some work as soon as possible at any given time.
• #50: And now lets take a look at the average render frame on the Intel i5 CPU with 4 cores. Here we can see the main thread which gathers all the commands into the number of independent command lists one by one. And it takes a lot of time to do it (about 17 ms). And this is the render thread which is actually processing previous render frame. You may also notice that render thread is idle for pretty big amount of time. This is because our main thread can’t generate render commands fast enough.
• #51: And now we enable the concurrent command generation, but at first, do in parallel only two tasks. As you can see render tasks executed only in two threads. This is because of dependencies between tasks. You might also notice that the overall frame time is reduced and more over the idle time in render thread is lower.
• #52: If we add five more tasks, TBB flow graph starts executing them concurrently on different threads with respect to dependencies. It’s also interesting, that in terms of TBB flow graph there is no main thread and it acts like a usual worker thread.
• #53: When we put all the render tasks into job system overall frame rate is doubled. But there is some curious moment, this bubble is essentially shows that we have to be careful with two things: Size of the each individual render task And set of dependencies between this task and others … or it might lead to such big inefficiencies.
• #54: So, as you can see functional or task parallelism (concurrency) is pretty easy to implement. It requires us a minimal effort to write and support such code, and because we can take a high level view of our rendering pipeline, we can easily make modifications to increase parallelism in our engine.
• #55: Unfortunately, high level frame render graph requires us to reason about big chunks of work which is not always optimal. Just because not all tasks have the same size or the execution time. Or not all tasks may be executed in parallel because some of them may have dependencies on each other.
• #56: So this approach quickly leads to the critical execution path. You may think about minimizing amount of dependencies between tasks which is good BTW or subdividing task into smaller one which may significantly reduce the readability of the code. Ok, but how could we improve performance while still keep our code readable?
• #57: Before, we were using only functional (or task) parallelism and there is also data parallelism. So, what we can do is to exploit data parallelism inside each sub system independently, and be sure that TBB is going to do the rest. Thus, we still have easy to read and maintain high level frame render graph and each sub-system may produce additional tasks to better utilize all available CPU cores.
• #58: Let’s take a look at high level CPU like i7 with 6 physical cores. We have ongoing render thread and set of render tasks executed in main thread. The render frame takes about 12 ms
• #59: And now the same frame but with enabled concurrent rendering. As you can see the render frame is reduced significantly and more over all 6 cores are busy. 1) But to reduce the critical path, big tasks should be subdivided into smaller one (this is the future work).
• #60: And we still have bubbles where CPU cores are idle but the overall CPU utilization is much higher.
• #61: To reduce amount of bubbles we’re using data parallelism. Here you can see how VFX prepare task schedules additional sub-tasks to be executed in parallel on other threads. So, bubbles are eliminated or at least number of them is reduced.
• #62: In summary I’d like to say that concurrent rendering allows us significantly reduce time spent during commands generation. Despite the fact that we’ve added parallel command generation, the code is still simple and easy to modify. And sparse performance may be used for other awesome features like tank treads, more objects on the scene or even more destructions. There is still lots of work to be done, the renderer may be subdivided into even smaller tasks and each task in turn may be decomposed into number of sub-tasks by using parallel algorithms. Also we have to be very careful with the command submission pattern on outdated graphics API. and being able to always have enough work for render thread, so it won’t stay idle. DX12 and Vulkan allows us to process commands right inside the generation thread, so we plan to exploit it to reduce pressure on render thread in the future.
• #63: To sum it up I’d like to say that multi-core age is coming and right now even not so powerful hardware often have 2 or more cores. And today I’ve showed you how we at World of Tanks development team trying to leverage all available power form multi-core machines.
• #64: I’d like to give a big thanks to Render, R&D Art, Tools Teams… and of course to the entire World Of Tanks development team.
• #68: And now we’ll take a look at technical demo.
• #69: And of course we faced a bunch of problems to solve. A tank could have up to 250 segments in each tread. It’s quite a bit considering that we have up to 30 tanks simultaneously and springs should collide between the wheels and environment obstacles Also we have hundreds of different tanks with different suspensions. So the final algorithm should be pretty general to support such a variety When the tank destroys a building the debris could move freely and independently, even jitter to some extent. In case of the tread we can’t afford it, the springs are always in a highly constrained environment (between wheels and the terrain) and jitterings are really annoying In real life the treads influence where can the tank can move and how. But the server doesn’t consider this effect because of performance reasons. This results in frequent situations when there is no way you can place the tread between wheels and the terrain Also the setup process of the new tread should be easy enough for the artists
• #70: And the final feature of the tread – its ability to sag below the tank. But before we continue please remember that the server doesn’t consider the tread. Because of that we frequently can’t properly place the tread between the bottom wheels and the terrain. Thus natural ability of the spring chain to sag doesn’t work out of the box. For an average PC we disabled the effect, snapping the bottom part of the tread to the nearby wheels. But for high end we can afford to process additional algorithm to resolve the mentioned problem. At the first pass we calculate the bottom of the track snapping it to the nearby wheels as for an average PC. At the second pass we resimulate the bottom without snap constraints Now each bottom spring joint has two possible positions: from snapped and unsnapped passes For each joint we calculate how close it is to the terrain and obstacles and based on that we blend the final position between the two mentioned positions. If the joint is far away from any obstacle – we favor position from the second pass, in other case – from the first. This allows the tread to sag and we have no jitters when there is no way to place it between the bottom wheels and the terrain.