A Real-time Radiosity Architecture

Editor's Notes

• #2: Hello various rendering dudes. “Hello, I’m Per.” “Hello, I’m Sam.” Etc. We (DICE and Geomerics) have been working together to incorporate Enlighten2 into Frostbite engine. The experience shaped the development of both Enlighten and Frostbite, and the resulting architecture and workflows are what we intend to discuss today.
• #3: INTRODUCTION (Sam) Radiosity has always been an exciting topic! There are a massive number of papers on the subject. In recent years we have seen a focus on fully dynamic “real time” solutions, primarily GPU based ones (e.g. Crytech, LPB2 (Evans), VPL-based, “instant radiosity”, etc) Probably fair to say most focus has been on finding new novel algorithms. The sheer number of papers that exist gives a near-continuum of algorithmic options, in which we can see some common categories of algorithms emerging as clusters. (eg. Photon-mapping, VPLs, voxel-grid based methods). This talk is less about demonstrating an algorithm and more about discussing the surrounding architecture and its use in practice. What do I even mean by radiosity architecture? Structure within which the algorithms operate – independent of algorithm to some degree “Dual” to algorithmic category in some sense. Enlighten undergoes regular algorithmic changes, but structure static. Key point for this talk is that the architecture is itself a very power tool, which I will illustrate with examples Should become clear as I describe it. DICE and Geomerics have been working together since close to the birth of Enlighten. The integration into Frostbite showcases the use of real time radiosity. We will describe how our pipeline & runtime have been set up to work with Enlighten. Show the workflows we have created to use dynamic radiosity.
• #4: AGENDA (Quick slide - outline rest of talk.) Essentially me then Per. I do an overview and talk about Enlighten and its architecture. Per then demonstrates how Enlighten was integrated into Frostbite.
• #5: ENLIGHTEN OVERVIEW 1/2 Summary of what Enlighten is trying to achieve. This is reflected in the architectural decisions we made. If you change these you will get a different outcome. Current console cycle - constrained by hardware GPU not so powerful and over-utilitised already. Main feedback we got from pre-release versions of Enlighten was that the GPU wasn’t a viable resource Plus DX9-class hardware constrains algorithmic options Memory also very limited. Multi-core is clearly best target. Wide range of abilities between the 3 targets though – scalability is vital. Always trading off quality with flexibility Offline – great quality, terrible iteration time, not real time Real time without precomputation – low quality, great iteration and gameplay Enlighten2 is a midpoint Some pre-processing, so lighting does not fully reflect moving geometry. Focusing on good quality, scalability and flexibility, not fully dynamic at all costs. Frostbite Wanted a lighting solution with fast iteration times / support for dynamic environments. Previous static techniques gave great results, but painful to wait for lightmaps to build. Give artist more creative freedom... Art controls Many points at which artists can add value. End goal is always beautiful, emotive, believable images. Not physical realism. Allow control over all aspects of indirect lighting – not hardwired to direct lights. Per-light indirect scales/colouring Global direct/indirect balance and tonemapping
• #6: ENLIGHTEN DECISIONS These are the 4 key architectural features I’d like to dig into further today. Separate lighting pipeline: Radiosity calculated independently of and asyncronously to rendering engine. Lightmaps: Compact, controllable and cacheable indirect lighting representation Target geometry: Separate lighting resolution from geometric detail Single bounce: Big algorithmic simplification Will now walk through each in turn.
• #7: Will use this asset as an example. It’s similar to the sponza atrium, built internally and shipped in our SDK.
• #8: ENLIGHTEN INTRO 2/2 Pre-process static geometry Enlighten attempts to put as much magic into this as possible. Essentially a problem-distillation stage. Details to be aware of now: Break up scene into systems – locally solveable problems but provide global solution. 1 system == 1 output lightmap Also setup relighting information – will cover this in more detail Precomputing information is a tough compromise to make, but very important to getting near-to-offline lighting quality. Runtime Our runtime separates indirect lighting work from direct lighting work. Direct lighting is done on the GPU as usual Indirect lighting is done on the CPU (current best place for target platforms) Both can run asyncronously. Previous cached output is always available – GPU just composites with the latest results off the carousel. Think of two processes, whirring away asyncronously from each other. Separation is not just an optimisation – also allows a lot of creative freedom different integration options
• #9: LIGHTMAP OUTPUT 2/2 We shall walk through an example shot in our runtime pipeline. Note the split of direct and indirect lighting paths, with the dependence on a common description of the lighting in the scene.
• #10: This is the traditional direct lighting in this shot. It’s a simple cascade shadow map based directional light. The choice of light and its implementation is essentially arbitrary. Note the lack of direct environment lighting. Environment lighting (and other area lights) are handled entirely within Enlighten.
• #11: This is the corresponding input to Enlighten. The core runtime component of Enlighten maps a point sampled description of the lighting over the target mesh surface, to a lightmap or lightprobe output. Anything you can express in these terms is valid input to Enlighten. We provide several fast paths for common light types. Directional lights with precomputed visibility is one of these options, which we use in this example. This provides a fast efficient method of generating the input lighting that does not require any interaction with the GPU-rendered sunlight, although we could have chosen to point sample that lighting instead. You may note that as well as the directional light, there is also radiosity lighting in the point sampled data. This is the previous lightmap output being fed to enlighten as an input light to generate a second bounce. This is how we generate multiple bounces. I’ll return to this later on.
• #12: This shot shows the “target” mesh for this scene, with the point-sampled lightmap output generated for the previous input. This is in some sense the raw lightmap output from Enlighten, before any interpolation, albedo modulation or directional relighting is applied. Note how low resolution the output is. You can see the effect of the skylight (blue) and the bounced directional light. Although low detail this resolution captures the essence of the indirect lighting. Significant soft shadows are present, and the lighting gradients can be seen quite clearly.
• #13: This shows exactly the same lighting data, but applied to the detailed geometry, together with normal maps and indirect specular. Note how much the basic lightmap output gives you when taken together with these relighting additions. In particular, hardware interpolation gives you a lot as long as your output is very ‘clean’. This cleanness is important - we scale linearly with output texels, so each texel really has to work for it’s place in the lightmap. If you want a very dense output format, as we have, you can’t afford noise. Much of the detail is filled in by the off-axis relighting. Simple variation in normals gives you a lot of lighting variation. There are multiple output formats for Enlighten. This particular screenshot is using our “directional irradiance” technique which is a mid point between a full spherical output (e.g. Spherical harmonics – complete spherical lighting data) and direction-less irradiance (the other 2 output options). The specular effect is a simple ‘imaginary’ specular effect generated in the final shader. Think of it as a phong specular model parametised by the strongest lighting direction. You only get one specular highlight, but your eye is very intolerant to errors in specular term. Believable specular results are far easier to obtain than “accurate” ones. So we prefer to give as much control to the user/artist as possible here.
• #14: And here we see the final composite. This was a brief introduction to our runtime radiosity pipeline. Per will talk through its use in Frostbite later on. Want to move to the second of my architecture points: the modelling of a single bounce...
• #15: ENLIGHTEN SINGLE BOUNCE 1/1 This is a big algorithmic simplication, and when running in realtime, doesn’t actually lose you anything. Infinite bounces through feedback loop “The Right Way” at 60fps Key point All terms in lighting become separable Easy dynamic albedo Much simpler to compute and compress There are some very elegant methods for computing the fully converged integral (based around S = (1-T)^-1), and if you never intended to update some aspect of that solution, this might be the best option. But in practice you are always generating more than one solution. With one bounce you don’ t have any cross terms. For instance, the surface albedo is now separable – essentially just a diagonal matrix you multiply your input lighting by. So any precomputation you do is now independent of the surface albedo, so you are free to update it in realtime. Similarly, the transfer coefficients are much simpler to calculate. This also leave the door open to recalculating them on the fly. Convergence is quick. 4 bounces at 60fps =>4*1000/60 = 66 ms. You can’t see this in practice. The decoupled direct lighting also helps you. Even if there is a lag, your direct lights are always lag-free. It’s harder to perceive the lag in indirect lighting in this setting, and only becomes apparent with momentary light sources (grenades, muzzle flash, etc).
• #16: ENLIGHTEN LIGHTMAPS 1/2 The word “Lightmap” does tend to conjure up images of large pages of memory-hogging textures. Enlighten lightmaps are actually really very small and very densely packed – there is very little wastage. Memory footprint and bandwidth are low. Primarily comes from only storing indirect lighting, and using target geometry to simplify the uv surface area. The target mesh lightmap is no longer a 1-1 mapping of the original detailed surface. The low numbers of charts allow us to get very high coverage. We support a number of different output texture formats that all share the same layout. So you can swap algorithms easily. As well as cheap interpolation, the key properly lightmaps give us a cacheable output. This is in contrast to many gpu solutions that are temporary or view-frustum only. This property gives us a wide range of scalability options. This is the real key point. Extremely valuable in practice. First if you consider the different kinds of lighting environment you might encounter in a video game it becomes more apparent why this is helpful: Outdoor time-of-day lighting Sun motion triggers global lighting update Sun moves slowly – stagger updates across frames (lots of work but do it gradually) Intimate indoor lighting More detailed, rapidly updating lighting Restricted visibility – only update visible areas (need lower latency but achievable by culling) A zero-order integration Only update lightmaps in offline editor Target any platform that supports textures (wii, etc) “Parametised” lighting descriptions Used in some MMOs Compute radiosity on load, or only update on world events (Just doing the work when required)
• #17: TARGET GEOMETRY 2/4 Lets take a closer look at the target/detail mesh projection. Want to capture lighting at the resolution we choose, then relight other geometry from there. This is the target geometry authored for our arches scene. It’s very basic. The key thing is to get a mesh that has a simple uv surface area (low chart counts). The tri count is not important. Collision geometry or a low LOD is usually a good starting point.
• #18: TARGET GEOMETRY 3/4 This is the detail geometry with the lighting from the target mesh lifted onto it. The lifting operation is actually an offline mesh-to-mesh projection. Deliberately simple. There is no need to author uvs for the detail geometry. These are generated during the projection. Rather cool property – easy to author simple target geometry. Skip uv authoring on complex detail geometry. Extra authoring overhead, but simple geometry and we also provide uv tools. Not an exact science - reasonably tolerant to mistakes.
• #19: TARGET GEOMETRY 4/4 Here’s an example projection. The yellow chart is from the detail mesh and has been projected to the pink/red target mesh. Note the generated uvs in the 2D inset correspond to the 3D location. --- cut ---- The projection itself is done as part of the precompute. Here we can see the packed uvs for the target mesh with one particular chart highlighted (in pink in 3D and red in 2D), and a single detail chart (in yellow) also highlighted. The detail mesh has uvs that sample a small section of the target mesh uvs. By computing a projection from the detail mesh vertex positions to the target mesh geometry we can generate a set of uvs for the detail mesh. It’s a (fairly simple) 3D projection based on the shape of each piece of geometry. The UV projection is then inferred from the 3D projection. Note that no actual simplified geometry is generated during the process – no CSG ops, mesh reduction or other complex stuff. We are deliberately keeping it as simple as possible. Also note that “lifting” lighting from one mesh to another is just an offline uv generation problem. At runtime there is no difference between rendering a detail or target mesh – they both sample from the same data. There are further implementation detais regarding handling of overhanging geometry, mesh instances, authoring controls and so on. Which we won’t have time to cover today.
• #20: To recap, these are the 4 architectural features we’ve just covered. <summarise key points of each> I’ll now hand over Per to discuss their use in Frostbite...
• #21: So I’m going to go through how we’ve set up the Frostbite Engine to run with Enlighten and how our pipeline and runtime support this architecture. But first I’m going to talk about why we started to work with Geomerics, and why we decided to go for a real-time radiosity solution in Frostbite.
• #22: So why do we want real-time radiosity in a game engine? For us the main argument was to improve the workflows and iteration times. We’ve been working with traditional lightmap techniques at Dice, and even if the results can look amazing in the end, it was just a very painful way of creating content . It’s not unusual that artists spend hours waiting for lightmap renders to finish. So we though, if artist can spend their time working on actually lighting the game instead of waiting for lightmaps to compute, then perhaps the end results will look more interesting? Another main argument is to support for dynamic environments. Video games are becoming more dynamic, so if we change the lighting in the game, we should also be able to update the bounce light dynamically. And finally, the architecture that came out of integrating Enlighten into Frostbite turned out to be pretty flexible. The direct and indirect lighting pipeline is completely separate, so the architecture is pretty robust to general changes to the rendering engine.
• #23: Before we can run the game with dynamic radiosity, we have to do a few things in our precompute pipeline. I’ll go through these steps one by one.
• #24: Enlighten provides two ways of representing the bounce light. Either via lightmaps, or via lightprobes. The first thing we do is to decide how each object should be lit. Static geometry is parameterized and lit with dynamic lightmaps. This geometry can bounce light, and is typically large objects that don’t move. Dynamic objects can only receive bounce light by sampling lightprobes, so they are typically moving around in the scene or just small and don’t affect the lighting to much themselves. In this scene you can see how we’ve separated static and dynamic objects. The underlying geometry is used to bounce light, which is then transferred to all objects in the scene.
• #25: One of the key features in Enlighten is to group objects into systems. A system is a collection of meshes that can be processed independently, and this really makes the radiosity a more local problem. Each system can be processed in parallel, the precompute can be distributed and runtime updates can be separated. We automatically define input dependencies to each system system, which is a way to put restrictions on the light transport. So when we update the yellow system here, we only read bounce light from the green systems and we can forget about the rest. We also use systems to control update performance. Large systems will have many pixels and it will take longer to compute, so by creating many small systems, we can spread out radiosity updates on several frames if we like. We typically update one system per CPU core every frame.
• #26: We need to parameterize the world to get lightmaps uvs, and we do this semi-automatically. For each static mesh we also have a low-poly target mesh that we use for lighting. The target mesh is manually parametrized, and we project the detail mesh onto the target mesh to generate uvs for the detail mesh. We also pack a uv atlas for each system. Each system atlas is independent of all other systems, so we end up with one lightmap per system.
• #27: When we have generated systems and parameterized the geometry, we can generate runtime data in our precompute pipeline. The runtime data has information about geometry and form factors that we need to update the radiosity in real time. There’s one data set per system, which is very nice if we want to stream data from disk. All systems can be processed in parallel, so we use Incredibuild’s XGI to distribute this build step. This is the only time consuming step of the precompute, but it scales pretty well with incredibuild. A typical final game level takes about 10 – 30 min to precompute. Since this data only contains geometry information, so we only have to regenerate it when we change the geometry, not the lighting or the colors in the scene.
• #28: Let’s take a look at the runtime. A key thing with this architecture is that we have a separate render pipeline for direct and indirect radioisty. In fact, we update the radiosity on the CPU and we do the direct light & compositing on the GPU. Frostbite uses deferred rendering, so can we can render many light sources every frame. Each light source is fed to Enlighten and part of the radiosity bounce light. Another thing we do is to separate the rendering of lightmaps and lightprobes. Lightmaps are rendered in the forward pass, but lightprobes are added to 3D textures so we can render them deferred in screen. The reason we do this is so we don’t have to upload a unique lightprobe for every object we render, which tends to quite a few objects if you consider foliage, vegetation, particle effect and decals, so adding lightprobes deferred in screenspace is just simpler for us.
• #29: The complete render pipeline looks like this. First, we update lightmaps & lightprobes on the CPU, and we lift lightprobes into 3d textures. These 3d textures cover the entire scene. Next, we run the geometry pass, where we add bounce light from the lightmaps to a separate g-buffer which we LogLuv encode. We also use the stencil buffer to mask out all dynamic objects, so we know what to light with lightprobes later. Finally, in the light pass, we first render all deferred lights, we then add the lightmaps from the g-buffer, and finally we add the lightprobe 3d textures deferred in screen space.
• #30: Let’s take a look at an example scene. This is the lightmaps and lightprobes generated on the CPU.
• #31: First we render the direct light sources deferred.
• #32: Then we add the lightmap bounce light.
• #33: Then we add bounce light from the lightprobes. We add them all together, to get the final compposite.
• #34: Final composite
• #37: Lightmaps The initial constraint of targeting the current console cycle makes lightmaps the output of choice But key properly is that it’s cacheable. 2D nature is not vital. Lazy eval of more “3D” structure expected on next cycle and PC. Laziness important to avoid O(n^3) scaling. Just not viable on consoles at the moment. Currently very task based Ok to a point. Incremental update vs fixed cost Enlighten costs as much to bounce black light as it does complex lighting. Might appear to be a good optimisation to make, but the tradeoffs are not obvious. This requires more thought. Might make more sense when moving towards a fully dynamic solution. Temporal optimisations can be tricky... Splitting lighting by distance Another opportunity exposed from single bounce model Maths does work out. Radiosity is different across scales. Question is: how useful is it to exploit?