![Short Hair / Beard
• Considered many methods
[Tariq08] [Neulander98] [Assarson09] [Nguyen06] [Neulander01]
• Ended up with camera aligned triangle strips
• Reuse of existing code (e.g. mesh skinning)
• Reuse of existing art pipeline
• Move Vertices in the Vertex Shader (VS)
VS “code” to move the vertices Before VS After VS
10](https://image.slidesharecdn.com/gdc2011epicsamaritan-110518214200-phpapp01/85/The-Technology-Behind-the-DirectX-11-Unreal-Engine-Samaritan-Demo-10-320.jpg)
![Rendering hair
• How to shade the pixel?
• Alpha Test / clip -> Problems with Aliasing
• Alpha Blend -> Problems with Sorting, fogging, Depth of Field
• Alpha To Coverage (A2C) -> Problems with many layers
• Order Independent Transparency [Gruen10] -> Too many layers?
• Our choice:
• Render to MSAA buffer
-> Depth for DOF/Fog/Shadow receiving
• Stick to binary occlusion (per MSAA sample)
• SSAA (Alpha Test per MSAA sample)
-> Anti-aliasing for individual hairs
12](https://image.slidesharecdn.com/gdc2011epicsamaritan-110518214200-phpapp01/85/The-Technology-Behind-the-DirectX-11-Unreal-Engine-Samaritan-Demo-12-320.jpg)
![Deferred Rendering [Hargreaves04]
• UnrealEngine 3 is primarily a forward renderer
• Geometry detail * MSAA * Complex shaders * Many lights
-> too slow in forward, too many shader permutations
• Added more GBuffer properties
– Albedo + Specular color, Specular Power
– Spec + Diffuse normal (Wet material is 2 layered)
– Subsurface scattering
• Some forward rendering remains (skin, hair and translucency)](https://image.slidesharecdn.com/gdc2011epicsamaritan-110518214200-phpapp01/85/The-Technology-Behind-the-DirectX-11-Unreal-Engine-Samaritan-Demo-15-320.jpg)
![Screen Space Subsurface Scattering
(SSSSS)
• Idea is to gather lighting contributions in screen space
[Mikkelsen10] [Jimenez09]
• Gather 16 samples in a disc, randomize per pixel and in time
• Artist can define SSS color and world space scatter radius
• Takes Depth and Normal input into account
• Hides shadow sampling artifacts
• Doesn’t work with ear
16 samples
16 samples per pixel randomized
19](https://image.slidesharecdn.com/gdc2011epicsamaritan-110518214200-phpapp01/85/The-Technology-Behind-the-DirectX-11-Unreal-Engine-Samaritan-Demo-19-320.jpg)
![Cubemaps
• only far reflections
Planar reflection
• fixed camera position
• fixed reflection plane
• good for dynamic ground reflection
… [Buerger07]
“Billboard reflections”
• Many textured quads (billboards)
• Placement like any other static object
• Can move/rotate/scale dynamically
• No limitations on the reflecting surface
22](https://image.slidesharecdn.com/gdc2011epicsamaritan-110518214200-phpapp01/85/The-Technology-Behind-the-DirectX-11-Unreal-Engine-Samaritan-Demo-22-320.jpg)
![Static Reflection Shadows
• Ray-casting a distance field
– Jump over empty areas
– Stored in a volume texture
– Distance also allows arbitrary blurred shadows
– Half resolution (bilateral up-sampling) [Shopf09] [Tomasi98]
Shadows visualized (brightened sky) Distance Field visualized
27](https://image.slidesharecdn.com/gdc2011epicsamaritan-110518214200-phpapp01/85/The-Technology-Behind-the-DirectX-11-Unreal-Engine-Samaritan-Demo-27-320.jpg)
![Point Light Reflections
• Phong or Blinn-Phong specular wasn’t giving the look we wanted
• Anyone have a “wet street BRDF” ?
• We added a new specular type
– More “lengthy”
– Shadowed like Billboard reflections
– Energy preserving [CodeItNow09]
– Distance attenuated but not distance bound
Photo reference Real-time (without rain drops)
29](https://image.slidesharecdn.com/gdc2011epicsamaritan-110518214200-phpapp01/85/The-Technology-Behind-the-DirectX-11-Unreal-Engine-Samaritan-Demo-29-320.jpg)
![What is Bokeh?
• Bokeh is the name of the shape that can be seen in photos
or movies when objects are out of focus.
• Contributes to the filmic look
• Shape depends on the camera and lens
• Many Depth of Field algorithms blur objects out of focus
without the desired shape. [Lefohn10]
Photo Reference Different Bokeh shapes (real-time)
31](https://image.slidesharecdn.com/gdc2011epicsamaritan-110518214200-phpapp01/85/The-Technology-Behind-the-DirectX-11-Unreal-Engine-Samaritan-Demo-31-320.jpg)
![Bokeh Depth of Field
Render a Bokeh textured quad for each pixel
[LostPlanetD3D10][3DMark]
• Quad size and opacity depends on the Circle of Confusion (CoC) radius
• CoC radius is computed from the pixel depth
• Accumulate pixel color and opacity weighted by the Bokeh texture
• Splitting the content into layers avoids occlusion artifacts
Foreground (blurred) In Focus (Full Resolution) Background (blurred)
32](https://image.slidesharecdn.com/gdc2011epicsamaritan-110518214200-phpapp01/85/The-Technology-Behind-the-DirectX-11-Unreal-Engine-Samaritan-Demo-32-320.jpg)
![References 1/2
• [Buerger07] GPU Rendering of Secondary Effects
http://wwwcg.in.tum.de/Research/data/Publications/vmv07.pdf
• [Tariq08] Real-Time Hair Rendering on the GPU
http://developer.nvidia.com/object/siggraph-2008-hair.html
• [Lauritzen10] Deferred Rendering for Current and Future Rendering Pipelines
http://visual-computing.intel-research.net/art/publications/deferred_rendering/
• [CodeItNow09] Energy Conservation In Games
http://www.rorydriscoll.com/2009/01/25/energy-conservation-in-games
• [Lefohn10] Advanced Real-Time Depth of Field Techniques
https://graphics.stanford.edu/wikis/cs448s-
10/FrontPage?action=AttachFile&do=get&target=CS448s-10-10-depthOfFieldForWeb.pdf
• [Jimenez09] Screen-Space Perceptual Rendering of Human Skin
http://giga.cps.unizar.es/~diegog/ficheros/pdf_papers/TAP_Jimenez_LR.pdf
• [Shopf09] Mixed Resolution Rendering
• http://developer.amd.com/gpu_assets/ShopfMixedResolutionRendering.pdf
• [Tomasi98] Bilateral Filtering for Gray and Color Images
• http://www.cs.duke.edu/~tomasi/papers/tomasi/tomasiIccv98.pdf
• [Hargreaves] Deferred Shading
• http://read.pudn.com/downloads160/sourcecode/game/724029/DeferredShading.pdf
36 /35](https://image.slidesharecdn.com/gdc2011epicsamaritan-110518214200-phpapp01/85/The-Technology-Behind-the-DirectX-11-Unreal-Engine-Samaritan-Demo-36-320.jpg)
![References 2/2
• [Gruen10] OIT and Indirect Illumination using DX11 Linked Lists
http://developer.amd.com/gpu_assets/OIT%20and%20Indirect%20Illumination%20using%20DX11
%20Linked%20Lists_forweb.ppsx
• Robust Multiple Specular Reflections and Refractions
http://http.developer.nvidia.com/GPUGems3/gpugems3_ch17.html
• [Neulander01] Hair Rendering (Ivan Neulander, Rhythm & Hues Studios)
http://www.rhythm.com/~ivan/hairRender.html
• [Nguyen06] GPU Gems2: Chapter 23. Hair Animation and Rendering in the Nalu Demo
http://http.developer.nvidia.com/GPUGems2/gpugems2_chapter23.html
• [Assarson09] Siggraph 2009: GPU Primitives-Case Study: Hair Rendering
http://s09.idav.ucdavis.edu/talks/07-Ulf-GPU-Prims-and-Hair-course-slides.pdf
• [Neulander98] Rendering Generalized Cylinders with Paintstrokes
http://www.rhythm.com/~ivan/pdfs/gi98.pdf
• [LostPlanetD3D10] Lost Planet D3D10 Parallel Rendering
http://meshula.net/wordpress/?p=124
• [3DMark] 3DMark11 Whitepaper
http://www.3dmark.com/wp-content/uploads/2010/12/3DMark11_Whitepaper.pdf
• [Buerger07] GPU Rendering of Secondary Effects
http://wwwcg.in.tum.de/Research/data/Publications/vmv07.pdf
• [Mikkelsen10] Cross Bilateral Filters for Skin Shading
http://jbit.net/~sparky/subsurf/cbf_skin.pdf
37](https://image.slidesharecdn.com/gdc2011epicsamaritan-110518214200-phpapp01/85/The-Technology-Behind-the-DirectX-11-Unreal-Engine-Samaritan-Demo-37-320.jpg)
The Technology Behind the DirectX 11 Unreal Engine"Samaritan" Demo
- 1. Game Developer Conference 2011 The Technology Behind the DirectX 11 Unreal Engine "Samaritan" Demo Martin Mittring Bryan Dudash Senior Graphics Architect Developer Technology Martin.Mittring@EpicGames.com bdudash@nvidia.com Epic Games, Inc. NVIDIA
- 2. Overview • About • Real-time demo • Technical Part: – Tessellation (NVIDIA) – Hair – Deferred + MSAA – Subsurface Scattering – Reflections – Depth of Field
- 3. Demo Goals • Ready for GDC 2011 • Real-time on High-end PC (off the shelf hardware) • Engine improvements: – Add Direct3D 11 support in Unreal Engine 3 – Implement features needed for next-gen quality • Research: – New hardware features like Tessellation – Advanced render techniques – Content creation / workflow 3
- 4. Storyboard to define the scope => Near shots, faces, hair, harsh lighting, rain 4
- 5. Derived Technology needs Direct3D 11 • Tessellation (NVIDIA) Filmic look • Quality Harsh lighting, night scene • Dynamic Shadows Rain • Reflections • Particles • Animated water surface • Wet material shading 5
- 6. Derived Technology needs Close ups • Depth of Field • Facial expressions Short scalp hair and beard • Hair • Simple animation • Rather simple shading Coat ”Realistic and Interactive Clothing in Epic Games Samaritan Demo Using NVIDIA APEX” Thursday 4:30- 5:30 Room 110, North Hall 6
- 7. Video / Real-time demo
- 8. Rendering Tessellation Hair Deferred + MSAA Subsurface Scattering Reflections Depth of Field 8
- 9. Rendering Tessellation Hair Deferred + MSAA Subsurface Scattering Reflections Depth of Field 9
- 10. Short Hair / Beard • Considered many methods [Tariq08] [Neulander98] [Assarson09] [Nguyen06] [Neulander01] • Ended up with camera aligned triangle strips • Reuse of existing code (e.g. mesh skinning) • Reuse of existing art pipeline • Move Vertices in the Vertex Shader (VS) VS “code” to move the vertices Before VS After VS 10
- 11. Hair creation • Generate very thin triangle strips in 3ds Max (Plug-in “Hair Farm”) • ~5000 splines -> ~16000 triangles • Texture contains 36 individual hairs Hair and head mesh in 3ds Max Texture 11
- 12. Rendering hair • How to shade the pixel? • Alpha Test / clip -> Problems with Aliasing • Alpha Blend -> Problems with Sorting, fogging, Depth of Field • Alpha To Coverage (A2C) -> Problems with many layers • Order Independent Transparency [Gruen10] -> Too many layers? • Our choice: • Render to MSAA buffer -> Depth for DOF/Fog/Shadow receiving • Stick to binary occlusion (per MSAA sample) • SSAA (Alpha Test per MSAA sample) -> Anti-aliasing for individual hairs 12
- 13. SSAA 4x MSAA 4x MSAA per pixel clip (AlphaTest) per sample clip (SSAA) 13 /35
- 14. Rendering Tessellation Hair Deferred + MSAA Subsurface Scattering Reflections Depth of Field 14
- 15. Deferred Rendering [Hargreaves04] • UnrealEngine 3 is primarily a forward renderer • Geometry detail * MSAA * Complex shaders * Many lights -> too slow in forward, too many shader permutations • Added more GBuffer properties – Albedo + Specular color, Specular Power – Spec + Diffuse normal (Wet material is 2 layered) – Subsurface scattering • Some forward rendering remains (skin, hair and translucency)
- 16. Anti-Aliasing • 4x MSAA for forward rendering • Deferred rendering requires special attention • Per fragment shading only where needed: 1. Clear stencil, Set stencil write 2. Pass 1: if heuristic(depth/normal) do discard otherwise shade per pixel 3. Activate stencil test MSAA per pixel MSAA per fragment 4. Pass 2: shade per fragment Deferred shadows (notice the bright halo) 16
- 17. Rendering Tessellation Hair Deferred + MSAA Subsurface Scattering Reflections Depth of Field 17
- 18. Human skin Subsurface Scattering (SSS) • Important effect to render believable faces in dynamic lighting • Many skin layers contribute to the final look • Human eye is trained to recognize details in faces • Human skin is a special case that allows approximations No SSS SSS only With SSS 18
- 19. Screen Space Subsurface Scattering (SSSSS) • Idea is to gather lighting contributions in screen space [Mikkelsen10] [Jimenez09] • Gather 16 samples in a disc, randomize per pixel and in time • Artist can define SSS color and world space scatter radius • Takes Depth and Normal input into account • Hides shadow sampling artifacts • Doesn’t work with ear 16 samples 16 samples per pixel randomized 19
- 20. Rendering Hair Deferred + MSAA Subsurface Scattering Reflections Depth of Field 20
- 21. Image Based Lighting (IBL) • To compute incoming light at given position and direction • How it works? – Function that maps position and direction to an image point (5D->2D) – Image with HDR content representing all incoming light • Complex lighting • Blurry reflections • Diffuse lighting 21
- 22. Cubemaps • only far reflections Planar reflection • fixed camera position • fixed reflection plane • good for dynamic ground reflection … [Buerger07] “Billboard reflections” • Many textured quads (billboards) • Placement like any other static object • Can move/rotate/scale dynamically • No limitations on the reflecting surface 22
- 23. One Billboard reflection • Each billboard is textured (Color and Alpha for Opacity) • Ray / quad intersection is simple math – Ray start position: surface point we want to shade – Ray direction: reflected eye vector Notice the reflection on the right 23
- 24. Glossy reflections • Isotropic reflections • Anisotropic “lengthy” reflections 24
- 25. Many Billboard reflections • Many Billboard can occlude each other • Iterate through all billboards • Store n (~3) nearest hits (z, color, opacity) • Composite n layers with alpha blending • TextureArrays to index a texture in the shader -> Same size and format 25
- 26. Reflection Shadows Without shadows With shadows Notice that without shadows light leaks through the building 26
- 27. Static Reflection Shadows • Ray-casting a distance field – Jump over empty areas – Stored in a volume texture – Distance also allows arbitrary blurred shadows – Half resolution (bilateral up-sampling) [Shopf09] [Tomasi98] Shadows visualized (brightened sky) Distance Field visualized 27
- 28. Dynamic Reflection Shadows • Crucial for grounding objects • Method assumes single plane reflection (ground) • We generate an image from the reflected eye position (similar to planar reflections), storing depth Without • Final mask is generated by rendering quads for each occluding Texel • The quad size is computed from the stored depth With 28
- 29. Point Light Reflections • Phong or Blinn-Phong specular wasn’t giving the look we wanted • Anyone have a “wet street BRDF” ? • We added a new specular type – More “lengthy” – Shadowed like Billboard reflections – Energy preserving [CodeItNow09] – Distance attenuated but not distance bound Photo reference Real-time (without rain drops) 29
- 30. Rendering Tessellation Hair Deferred + MSAA Subsurface Scattering Reflections Depth of Field 30
- 31. What is Bokeh? • Bokeh is the name of the shape that can be seen in photos or movies when objects are out of focus. • Contributes to the filmic look • Shape depends on the camera and lens • Many Depth of Field algorithms blur objects out of focus without the desired shape. [Lefohn10] Photo Reference Different Bokeh shapes (real-time) 31
- 32. Bokeh Depth of Field Render a Bokeh textured quad for each pixel [LostPlanetD3D10][3DMark] • Quad size and opacity depends on the Circle of Confusion (CoC) radius • CoC radius is computed from the pixel depth • Accumulate pixel color and opacity weighted by the Bokeh texture • Splitting the content into layers avoids occlusion artifacts Foreground (blurred) In Focus (Full Resolution) Background (blurred) 32
- 33. Bokeh Depth of Field Rendering Scatter pass (Geometry Shader): • Setup viewports to render to background/foreground layers • For each pixel: – Compute the Circle of Confusion (CoC) radius – Compute viewport (foreground / backgound) – Setup a quad with the Bokeh texture 1 (RGB: Bokeh*scene color, A: Bokeh) – Render quad with additive blending 2 Resolve pass: Render Target with • Reconstruct the layer color (RGB divided by A) two viewports • Blend layers by the accumulated occlusion (background, in focus, foreground) 33
- 34. Bokeh Depth of Field Optimizations • Vertex / Triangle count: • Input image is the half resolution scene (Color + Depth) • Fill rate: – Input image is the half resolution scene (Color + Depth) – Output image is half resolution and recombined later with full resolution – For each 2x2 input block: depending on heuristic (CoC radius, color and depth difference), spawn 1 or 4 quads (GS) Red: 4 quads Green: 1 quad 34
- 35. Bokeh Depth of Field: Translucency • Problem: Fog / particles / smoke / lens flares • Make some effects not affected by Depth of Field – Artists can specify which material – Composed after/without Depth of Field • New Shader graph node – To give artist control (fade out or blend to blurry version) 35
- 36. References 1/2 • [Buerger07] GPU Rendering of Secondary Effects http://wwwcg.in.tum.de/Research/data/Publications/vmv07.pdf • [Tariq08] Real-Time Hair Rendering on the GPU http://developer.nvidia.com/object/siggraph-2008-hair.html • [Lauritzen10] Deferred Rendering for Current and Future Rendering Pipelines http://visual-computing.intel-research.net/art/publications/deferred_rendering/ • [CodeItNow09] Energy Conservation In Games http://www.rorydriscoll.com/2009/01/25/energy-conservation-in-games • [Lefohn10] Advanced Real-Time Depth of Field Techniques https://graphics.stanford.edu/wikis/cs448s- 10/FrontPage?action=AttachFile&do=get&target=CS448s-10-10-depthOfFieldForWeb.pdf • [Jimenez09] Screen-Space Perceptual Rendering of Human Skin http://giga.cps.unizar.es/~diegog/ficheros/pdf_papers/TAP_Jimenez_LR.pdf • [Shopf09] Mixed Resolution Rendering • http://developer.amd.com/gpu_assets/ShopfMixedResolutionRendering.pdf • [Tomasi98] Bilateral Filtering for Gray and Color Images • http://www.cs.duke.edu/~tomasi/papers/tomasi/tomasiIccv98.pdf • [Hargreaves] Deferred Shading • http://read.pudn.com/downloads160/sourcecode/game/724029/DeferredShading.pdf 36 /35
- 37. References 2/2 • [Gruen10] OIT and Indirect Illumination using DX11 Linked Lists http://developer.amd.com/gpu_assets/OIT%20and%20Indirect%20Illumination%20using%20DX11 %20Linked%20Lists_forweb.ppsx • Robust Multiple Specular Reflections and Refractions http://http.developer.nvidia.com/GPUGems3/gpugems3_ch17.html • [Neulander01] Hair Rendering (Ivan Neulander, Rhythm & Hues Studios) http://www.rhythm.com/~ivan/hairRender.html • [Nguyen06] GPU Gems2: Chapter 23. Hair Animation and Rendering in the Nalu Demo http://http.developer.nvidia.com/GPUGems2/gpugems2_chapter23.html • [Assarson09] Siggraph 2009: GPU Primitives-Case Study: Hair Rendering http://s09.idav.ucdavis.edu/talks/07-Ulf-GPU-Prims-and-Hair-course-slides.pdf • [Neulander98] Rendering Generalized Cylinders with Paintstrokes http://www.rhythm.com/~ivan/pdfs/gi98.pdf • [LostPlanetD3D10] Lost Planet D3D10 Parallel Rendering http://meshula.net/wordpress/?p=124 • [3DMark] 3DMark11 Whitepaper http://www.3dmark.com/wp-content/uploads/2010/12/3DMark11_Whitepaper.pdf • [Buerger07] GPU Rendering of Secondary Effects http://wwwcg.in.tum.de/Research/data/Publications/vmv07.pdf • [Mikkelsen10] Cross Bilateral Filters for Skin Shading http://jbit.net/~sparky/subsurf/cbf_skin.pdf 37
- 38. Thanks • Our partner: NVIDIA • NVIDIA: Johnny Costello, Bryan Dudash, Jon Jansen, Ignacio Llamas, John McDonald, David Schoemehl • Entire Epic team • Everyone that contributed to the demo • Epic: Daniel Wright, Andrew Scheidecker, Jordan Walker 38
- 40. Questions? • Is this a game? No. This is just a technology demo. • Is that in UnrealEngine 3? These features are available now to UE3 licensees and will be in the March UDK. ”Realistic and Interactive Clothing in Epic Games Samaritan Demo Using NVIDIA APEX” Thursday 4:30- 5:30 Room 110, North Hall
- 41. NVIDIA @ GDC 2011 CAN’T GET ENOUGH? MORE WAYS TO LEARN: NVIDIA GAME TECHNOLOGY THEATER Fri, March 4th @ NVIDIA Booth Open to all attendees. Featuring talks and demos from leading developers at game studios and more, covering a wide range of topics on the latest in GPU game technology. MORE DEVELOPER TOOLS & RESOURCES Available online 24/7 @ developer.nvidia.com NVIDIA Booth South Hall #1802 Details on schedule and to download copies of presentations visit www.nvidia.com/gdc2011
- 42. Bonus Slide • Barely documented but very useful: – HLSL Semantic SV_SampleIndex Used as input causes the shader to run per MSAA sample. Can be used in texture.Load(float2(u,v),SampleIndex) or EvaluateAttributeAtSample(Interpolator,SampleIndex) – HLSL Semantic SV_Coverage uint, MSAA bit mask, PS input and output • How to index a texture in the shader? – 2D Texture Atlas -> Size limits, Border and Precision issues – Sample array (D3D9/10/11) -> Only for constant index / unroll able loops – Dynamic branching -> Slow – Texture array (D3D10/11) -> Same size and format, CPU update performance? • Energy preserving Specular images (material varies Glossiness): Specular tweaked for glossy areas Specular tweaked for dull areas Energy preserving 42