Developing and optimizing a procedural game: The Elder Scrolls Blades- Unite 2019

Editor's Notes

• #3: Intros
• #4: How difficult it is to build a procedural game with advanced visuals for mobile platforms? We will try to offer some insights Our experience is from working on The Elder Scrolls Blades – it reimagined TES for modern mobile platforms Specifically we will talk about: precomputed lighting in procedural context  some aspects of performance/optimization that we dealt with
• #6: This is an example of what a level looks like from an editor point of view Made of standalone blocks called rooms The red lines represent connections between rooms Different shapes and size but they all fit together Any room can be attached to any other rooms for the same theme. All fit on a grid that acts as a framework for the level generator To support longer levels without using too much memory rooms are streamed in while the player progresses through the level
• #7: Example of a single rooms Rooms are prefab that contain all the data we need to use that piece in a level
• #8: Unity render pipeline Scriptable render pipeline was still experimental when we started the project Realtime lighting Looks great with the PBR shader, especially with high definition normal maps Allows dynamic shadow in key areas like this window Baked global illumination Both lightmaps and light probes Add depth to scenes with indirect lighting Can add lot of baked lights without affecting performance
• #10: On this screenshot what you see is a harsh cut produced by lightmaps along the floor and wall were two different rooms connect. The problem here is that since lightmaps are generated separately for each room, we can get end up with dramatically different colors and brightness around room connection. Because of the way we bake global illumination per room, our global illumination isn’t really global. We also can’t predict which rooms will get connected together because they’re assembled randomly at runtime. Bake time solutions… We concluded that we needed a runtime solution to this
• #11: That runtime solution is lightmap blending. The concept here is rather simple, what if we could blend the lightmap on each side of a connection to make the seam disappear? Then we could start with a 50-50 blend at the connection and the fade it out based on distance. Well you need a few things to achieve this. One, we need to add a secondary lightmap for blending to the materials on each side. Each side needs the other side’s lightmap And since a blend like this needs to be applied per pixel, we need to assign a blend factor to each vertex. And we need a uv set to sample the secondary lightmap texture. Once we have that we simply edit the shader to sample both lightmaps and blend them based on the blending factor. All of this is setup can be done at runtime when attaching two rooms together
• #12: With this technique we only end up using the lightmap color at the seam and stretching it, so any local detail can get blow out of proportion which doesn’t look good. The problem is that we don’t have enough to blend that shadow correctly, we don’t know what that shadow in the blue room would have looked like if it had continued naturally. Well what if we could create that information?
• #13: This is exactly what we did! We’ve created a mesh that is precisely the shape of the connection but stretched based on the blending distance. We make it so that it gets lightmapped by Unity at bake time, but we never actually render it in game. We only use it’s lightmap data. With this we have more accurate data for blending so we don’t need to stretch anything anymore.
• #15: The harsh cut is gone and we don’t have any noticeable artifact. The light on the floor from that torch does fade pretty quickly at the connection. It’s not 100% accurate, but still an acceptable compromise.
• #16: Because of the blending we do have an additional texture fetch per pixel, but we only do that for static renderers that are near a connection. So it’s not very significant. The worst performance problem comes from computing the uv set for the secondary lightmap. It involves finding the closest point on the extension mesh for each vertex of the blended renderers. This can be quite heavy calculations but what we did is run all of that asynchronously in a background thread. When streaming a level, the new rooms are spawned outside of the view of the player so we don’t need to apply lightmap blending immediately. And because we modify meshes at runtime by adding an extra uv set this doesn’t work with Unity’s static batching. You’ll have to either remove the static batching option on meshes that are blended or disable the feature globally in your project settings.
• #17: So this takes care of our biggest lightmap problem, but I mentioned earlier that we also use light probes as part of our lighting pipeline.
• #18: Light probes are object that contain directional lighting information for a specific point in space. They’re a built-in feature in Unity. If you setup a LightProbeGroup component in your scene, Unity will generate light probe information when you bake the lighting for that scene. At runtime they allow the engine to sample baked lighting information for dynamic objects. They’re a complementary feature to lightmaps. Lightmap are for applying baked lightins to static object and light probes are for applying baked lighting to dynamic objects. The problem with light probes is that Unity’s implementation doesn’t support our procedural pipeline. The first reason is that light probe data is stored in scenes only, but we use prefabs for our rooms. The second and main reason is that probe positions are set at bake time and cannot be moved. A specific room’s position in world space is not fixed in our case. We have to be able to spawn that room wherever our level generator tell us to.
• #19: At a high level this is how the light probes pipeline is separated in Unity. We have the generation part that runs in the editor when baking lighting. There’s the renderer that samples light probes at runtime for each dynamic object. It finds the probes that surround the object’s position and interpolate their values and then there’s the shader that uses that information to render the object. The only part of this that’s not compatible with our procedural pipeline is the renderer. So what if we just replaced with our own custom implementation?
• #20: So how would do that? We write the light probes data generated by Unity to a binary file that we attach to each room prefab. When a new room is attached at runtime we deserialize the probes data and attach the probe to a global probes network for the level. Then every frame we sample the probes network to find probe information for each dynamic object based on their position. Then we send over that data to the shader.
• #21: How do we manage that data? Well once lighting an been baked the LightPorbes class contains a SphericalHarmonicsL2 object for each probe in your scene. Under the hood this is just an array of floats so it’s easy to serialize. The main problem we faced here is that the shader expects a different format and we didn’t know how to translate a SphericalHarmonicsL2 object into the format the shader expects. Well it turns out there’s a very important line about this in the Unity documentation. It says: “The Unity shader code for reconstruction is found in UnityCG.cginc and is using the method from Appendix A10 Shader/CPU code for Irradiance Environment Maps from Peter-Pikes paper.” That links to a paper called Stupid Spherical Harmonics tricks by Peter-Pike Sloan. Appendix 10 of that paper contains the precise formulas that you need to feed a SphericalHarmonicsL2 object to Unty’s Standard shader.
• #22: But how exactly do you feed that data to the shader. The standard way of filling in shader properties is through materials. But probe values could be different for every single object even if they share the same material. We don’t want to create a unique material for every object, that would prevent batching. Unity has a feature for cases like this called MaterialPropertyBlocks. This allows you to fill in shader properties per object without having to go through a material. When using this you also need to set the [PerRendererData] tag on you shader property so that the engine knows your property will be filled in through MaterialPropertyBlocks. For probes data specifically you should set the lightProbeUsage value of all your renderers to “Custom Provided” This tells Unity that you’re responsible for filling in spherical harmonics. That way Unity won’t override your values.
• #23: Overall replacing that system worked well for us, but here are a few considerations. Sampling and interpolating probes for each dynamic object each frame can be pretty expensive so you need a way to know which renderers will or won’t move. Objects that don’t move should sample the probes once when spawned and then never again. Objects that do move should not sample probes if they have not moved since the last sampling. In our case we fill rooms with probe based on a 3D grid which means we a regular layout for our probes. This is very helpful because it allows us to find the closest probes to any point in space in constant time. Without a regular layout it becomes a much more complex problem to find the surrounding probes for any position. Unity’s default system supports that with by tetrahedral tessellation, but we didn’t need to reimplement that on our side. Another thing to keep in mind is that the shader properties used for spherical harmonics also contain the scene’s ambient color, so you’ll need to handle that yourself by adding the ambient color with the probe’s values before sending them to the shader.
• #24: Not running at high enough frame rate Android might be more affected
• #25: CPU side rendering is costly and could often be a bottleneck While GPU side could be as well but Tile based GPU deal well with overdraw Generally shader performance is easier to deal with The number of draw calls is a big performance driver for both CPU and often GPU Control the number of passes Shadows Depth pass Pixel lights etc. Efficiency of batching: static and dynamic Efficiency of graphics API used…
• #26: Culling is significant in reducing the number of draw calls A procedural game cannot use precomputed visibility culling Dynamic visibility culling  interior strategy exterior strategy Dynamic occlusion culling for town environment Buildings are natural occluders for other buildings, vegetations, decorations, NPCs etc.
• #27: Graphics API is a huge performance driver What is faster OpenGL or Vulkan? OpenGL is a traditional API maintaining internal state Vulkan is much more modern with many facilities to reference states and rendering data
• #28: Performance depends on graphics drivers… Loading times and FPS We discovered shader cache on some OpenGL drivers Vulkan is generally faster… But could be slower compared to OpenGL on other devices
• #29: Dynamic graphics API selection to attempt the best of both world Unity cannot change graphics API in the run time But there is a provision for auto selection and fall back  Can also be initialized with a different API Requires custom Unity build To implement -- needs custom player Activity (java) Checking device or SOC name to pass a flag to unity
• #30: Vulkan is viable Not many graphics artefacts Faster overall Beware validation layer Helps shader compilation caching Test in your specific case Both frame performance And loading times Dynamic API selection when beneficial Works in debug out of the box May need a custom Unity build May make it into main line Thanks to Unity for their help working with us to implement
• #31: Continuing with Android:  How Unity distributes threads to different cores  Most modern Android devices are 8-cores There are big cores that are clocked higher And small cores that are slower but consume less power Unity game can easily have 30-40 threads running Most important ones are: UnityMain UnityGFXDeviceW – multithreaded rendering is presumed to be used Worker threads These will be used more and more with DOTs and Jobs system Some systems use these already  E.g.: skinning, physics, particles Use Systrace to find out how threads are distributed
• #32: Core jumping Contention between workers and main/rendering threads Little cores not used…
• #33: Putting everything on big cores might actually be ok These are faster Workers and not uber busy We experimented with alternative affinities
• #34: Reserve cores for UnityMain and UnityGFXDeviceW Allocate workers (and choreographer and audio etc.) to other cores big and small
• #35: This is what the result looks like Generally found this to be measurably faster (5-10%) Only 4 workers We asked Unity for a feature to allocate more workers
• #36: Nav mesh generation benefits from workers on little cores Need to code a custom plugin to discover thread ids and adjust affinities Plugins are very easy with IL2CPP Setaffinity syscall to set core mask Not a major performance improvement Yet measurable We asked Unity for more workers Systrace is very useful iOS has much better single core performance (fewer cores though) Thanks to Google for their help
• #37: Running out of memory iOS is more affected Generally less memory on iOS devices, many with only 2G
• #38: Tools give different reading Xcode widget is the most important reading – OS jettisons apps Other tools give smaller estimates  Modern mobile devices memory system is complex
• #39: Native memory for asset data: textures, meshes etc – Reference counted Managed memory for Unity Objects and C# objects – GC collected Doesn’t account for everything in a Unity app though Virtual memory space – large, exceeds physical space Physical wired memory is only a small part Consists of wired dirty memory, Reusable (GPU uses this for storage during rendering) External, encrypted code Some read-only assets can be unloaded Some dirty memory can be compressed
• #40: Use mach functionality to get detailed view
• #41: A practical view of memory types evolving over time Overall use of internal memory is important Managed heap state is important Compressed memory utilization is important
• #42: Xcode widget reflects what OS is using for decision making There are many memory types and several levels of allocators Tools compute memory differently and often conservatively Use mach functionality to get details
• #43: Heap growth is sizable steps Sometimes we are short of memory and have MM of space in unused managed heap slack
• #44: Unity uses Boehm-Demers GC Easy enough to write a plugin that externs some GC statics One to note is GC_free_space_divisor that controls how heap will expand Can increase the divisor to have heap expanding less
• #45: Unity is great of exposing GameObject hierarchy Tools can be used to automate finding GameObject leaks It is harder for native and managed memory There is a Unity memory profiler which is improving Allows to compare captures and drill down Custom tools can be build using low level memory API The API provides a snapshot of all memory blocks in native and managed memory
• #46: We had to build a custom tool to help finding C# level leaks Can take a snapshot that compare with previous snapshot based on type names Also built a feature to see which types refer to which types Often times it is not drilling down into individual object but high level view that helps figuring out leaks Here is an example? Perhaps a leak?
• #47: We have access to Boehm statics Exercise caution as with anything low level Thanks to the team