FMX2013: Butterfly Effect

Butterﬂy Effect:
Workﬂow, Rendering & Shading
Renaldas Zioma
Unity Technologies
@__ReJ__
Monday, April 29, 13
This talk will cover workflows and techniques that we adopted while working on the
Butterfly Effect.
•

Unite template2
What is it?
Real-time animated short
Collaboration between Unity, Passion Pictures and NVIDIA
If you haven’t seen Butterfly Effect is realtime animated short, developed in collaboration
between Unity, Passion Pictures and NVIDIA
•

Opportunity to Learn
• Passion Pictures
• New world of RealTime
• No prior experience with Unity
• Unity Technologies
• CG Pipeline
• Huge amounts of Data
3
Passion Pictures is a film and animation production company with broad experience in
CG, stop-motion, documentary and traditional media. Passion Pictures have developed
visual elements for animated rock band Gorillaz, cinematics for Rock Band video game
and infamous (in UK) Meerkat advertisements.
Prior to this collaboration Passion Pictures had NO experience with Real Time
rendering / Game technologies or any experience working with Unity tools.
We approached Passion Pictures in late Fall of 2011 looking for collaboration on a real-
time short.
Real-time was a completely new area for Passion Pictures thus presented a learning
opportunity to adopt RealTime for internal pre-visualization and leverage experience for
working with Interactive projects (studio is working on number of such projects since).
Passion Pictures: “We felt we had an opportunity to do something special that we hadn’t
done before, get experience with the Unity real time engine and help drive Unity in a
direction that could integrate it with our studio.”

Project Outline
• 30GB of source Assets
• 5GB of runtime Data
• 137 shots
• 40 “scenes” in Editor
4

Project Organization
• Team of ~10 - ﬁrst month
• pre production
• research / tools
• Team of ~30 - next 2 months
• content
• tools / engine modiﬁcations
• Team of 5 - last 3.5 months
• polish
• optimizations
5

Unite template6
Passion Pictures presented a number of short film ideas and once Dan Sumich’s
“Butterfly Effect” pitch was chosen - work started.
Storyboard was developed from original pitch. Next storyboard was put into Animatronic
- animated sequence of images in sync with music too capture flow and timing.
Example of the storyboards for Scooter section in development.

Unite template7
Color Boards helped to capture mood and lighting.
It is worth noting how well Story and Color Boards resemble the final look of the short.

Previz
• One Sequence
• Goals
• Initial timings
• Visualize Space
• Prime lenses
• 10, 14, 18, 21, 24, 27, 35, 40, 50, 75, 100, 200
• Zoom lens
• 18 to 150
8
First pre-visualization was done in traditional to Passion Pictures way - directly in
Softimage XSI.
Only standard camera settings based on physically cameras and lenses as used in Live
shoots (Prime/Zoom lenses) were used to capture the filmic feel.

Pipeline
• Connect Passion pipeline with Unity Editor
• Main Goal
Minimal changes to Artist workﬂow!
9
Very short production cycle (traditional to CG industry) required minimal changes to
artist workflow.
Both Passion pipeline (XSI scripts) and Unity Editor scripting capabilities proved to be
flexible enough.
Passion Pictures: “After the creative challenge of finding the right director and script for
the film, the two immediate challenges were making sure that we had the right tools in
Unity for the job so that we could take our pipeline into Unity, and also making sure that
we could still work in the way we normally do as a studio.
If we break down our pipeline into animation, simulation, rendering and compositing
then we wanted to be able to bring these as much as possible into Unity working with
our existing content creation tools as normal.”

Passion Pipeline
• No Change for Artists
• Use Existing Asset Pipeline
• Sequences SB, BU
• Shots SB_01_00, SB_02_00
• Assets [Character/Camera/Layout]
• Export Models and Publish Shots as normal
• Add Unity support
10
Standard Passion Pictures asset pipeline could be mapped to Unity asset representation
quite easily. Multiple shots were grouped into a single Unity scene - some scenes
contained one shot, some multiple. Sequence of shots (SB - SkateBoard, BU - Bus)
became a group of Unity scenes sharing same folder.
Standard Passion Picture pipeline (set of Softimage XSI scripts) was augmented with
“publishing” shots to Unity project file structure and to incorporate SVN version control.
SVN was a new tool for artists.

Unite template
Character Rig
• XSI Softimage Rig
• Bones
• Envelope
• Face Shapes
• Controls
• Unity Rig
• Bones
• Blend Shapes
• Materials
11

Unite template
Facial Rig
• Bones
• Head, neck, jaw, eyes
• Blend shapes
• Face
• Mocap shoot
• Face markers
• Cubic motion
• Retarget facial animation onto XSI controls
12
Facial animations in Softimage XSI / Unity: combination of bones and blend shapes.
Face markers were used for facial motion capture, retargeted on controls in Softimage
XSI.

Unite template13
Mocap facial shoot on the right. Animation transferred onto Softimage XSI character rig
on the left.
We worked with Centroid for the motion capture shoot. Passion Pictures provided them
with a FK rig containing just the bones and skin for our Hero character. Centroid sent
FBX files for our FK rig send back FBX files of the FK rig from Centroid.
Using an intermediate rig Passion Pictures transferred the FK animation back onto our
IK character rig in Softimage XSI. By keeping the shoot rig live in the scene, we were
able to either keep the motion capture, animate on top of it or replace it with new
animation, depending what worked in the shot.

Unite template14
Facial animation controls in Softimage XSI after retargeting.

Non-linear time
• Treat Sequences as Live Shoot
• Cover the action from different angles
• Each sequence needs multiple cameras
15
Action packed Butterfly Effect storyboard pushed Passion Pictures to treat film
almost as a Live shoot. In traditional Live shoots multiple cameras are used to
cover action from different angles. During the montage footage might overlap
or jump in time according to Directors whim.
Film was broken down into sequences which correspond to a continuous action
(such as Skateboard sequence).
Description about every camera was created in Final Cut and exported to
custom XML format (based on Edit Decision List or EDL file format of Final Cut).

Unite template
Non-linear time
• Shot Description
• Final Cut edit
• Export EDL as XML file
• Camera in/out points
• Timeline in Unity
• Sequence Preview in RV
• Using EDL from Final Cut
• Play whole preview sequence in RV
• Instant Feedback for Animators
16
Action packed Butterfly Effect storyboard pushed Passion Pictures to treat film
almost as a Live shoot. In traditional Live shoots multiple cameras are used to
cover action from different angles. During the montage footage might overlap
or jump in time according to Directors whim.
Film was broken down into sequences which correspond to a continuous action
(such as Skateboard sequence).
Description about every camera was created in Final Cut and exported to
custom XML format (based on Edit Decision List or EDL file format of Final Cut).

Unite template
Timeline in Unity
17
Next this data could be read in Unity and used to control cameras, animated
objects and physical simulations. Custom timeline viewer was developed in
Unity using Editor scripting. Changes to the ﬁlm edit, jumping back and forward
in time and reordering of the shots became possible without need to rebake
animations in Softimage XSI.

Unite template
Timeline in Unity
18
Next this data could be read in Unity and used to control cameras, animated
objects and physical simulations. Custom timeline viewer was developed in
Unity using Editor scripting. Changes to the ﬁlm edit, jumping back and forward
in time and reordering of the shots became possible without need to rebake
animations in Softimage XSI.

Unite template
Non-linear Challenges
• Animation
• Particles
• Physics
• Rigid Body simulation
19

Unite template
Point Cache
• Per-vertex Deformations
• Cloth
• Hair
20
Two types of vertex animation: BlendShapes and Point Cache were introduced in Unity
for Butterfly Effect.
Passion Pictures: “In our standard pipeline we create point caches for each animated
model in the scene using .pc2 or alembic files. This creates a break in the pipeline so
that when we’re rendering we don’t need all the animation rig in the scene. For software
rendering the cache sizes and data throughput aren’t an issue but in a real time game
engine we have to keep the enveloped characters and bake the animation on the bones
per frame.”

21
Point Cache
Point Cache used for extreme face deformation

Unite template
Tessellation
• CatmullClark
• Now: on CPU
• Future: OpenSubdiv library
22
(Mostly due to multiple layers of resting cloth) CatmullClark tessellation was used to
match DCC tools exactly. It is problematic to implement CatmullClark using DirectX11
hardware tessellation because of recursive nature of certain patches. Instead of
approximation we went for exact CPU implementation for 1st level of subdivision.

Unite template
Hair
• Guide “strands” for modeling
• GPU Tesselation and “Ampliﬁcation”
•
23
Guide hair strands provide excellent control for animators.

Unite template24
Hair
Final look of the hair.

25
MentalRay Architectural
(MIA) Shaders
• Familiar
• Small set of parameters
• Physically Based
• Micro facet
• Energy conserving
We aimed to provide PassionPictures artists with familiar tools, hence decision to
implement realtime counterparts for MentalRay architectural shaders.

Unite template26
1 parameter for Diffuse
Between Clay and Plastic.
Screenshot from the Unity Editor.

Unite template27
2 parameters for Specular
Roughness and Reflectivity

Unite template28
2 parameters for Reﬂectivity
Reflectivity for depending on angle of incidence. Defined as pair of parameters - for ray
hitting at Grazing angle (90) and for ray perpendicular to the surface.

Unite template
MIA Implementation
• Oren-Nayar
• Diffuse
• Cook-Torrance
• Specular
• Geometric + Roughness + Fresnel term
• Ward distribution for Roughness
• I think... ;)
• Direct + Indirect illumination
29

Unite template
Unity Implementation
• Oren-Nayar
• very expensive
• OK approximations
• Cook-Torrance
• Normalized Blinn-Phong instead of Ward
• Roughness to Phong exponent (n):
• n = 2 / roughness2 - 2
• Tweaked to match MIA look:
• n = 2 / roughness4 - 2
• Schlick approximation for Fresnel
• Only direct illumination at RunTime
30

Unite template31
MentalRay Unity
Realtime implementation matches offline MentalRay implementation quite well. Note
some small differences in size of specular and very glossy surfaces.

Unite template32
Most of materials where setup by Passion Pictures in Softimage XSI and were
transferred to Unity. Tweaks were necessary to achieve final look in Unity, but relative
compatibility of the MIA and Unity shaders saved a bulk of the manual work for us.

Unite template33
Indirect Illumination (GI)
• Point Clouds for Dynamic Geometry
• Spherical Harmonics for Diffuse Bounce
• Cube Textures for Reﬂections
For indirect lighting we went for very practical approach - storing secondary bounces in
the low-resolution textures / Spherical Harmonics probes.

Unite template34
• Textures for Static Geometry
• Diffuse Bounce
• AO

Unite template35
• Calculated offline with Autodesk Beast
• Path Tracer + Final Gather
• IBL for Sky
• Fake Caustics
Beast setup:
. 500 Final Gather rays (1 bounce)
. Path Tracer (4 bounces)
. 1024 IBL sky rays
Final quality scene bake would take between 20 minutes and 8 hours depending on the
size of the scene. Bake quality for all scenes in the short was carried on over-night on 4
machines.

Unite template36
Real Time Shadows

Unite template37
Baked Shadows
• Of course baked only for static environments

38
Ambient Occlusion
• HBAO - Screen Space
Horizon-Based Ambient Occlusion - developed by NVIDIA.

39
Ambient Occlusion
• Too Dark & Dirty

40
Ambient Occlusion
• Applied ONLY to Indirect Illumination
• Colored to match Sky better
Screen Space Ambient Occlusion information is calculated at the start of the frame and
passed into the shaders, where it is applied to (occludes / darkens) indirect lighting only.
Ambient Occlusion is tinted towards the sky to achieve more plausible look.

Unite template
Skin
• Texture Space Diffusion
• 3 Layers
41
Subsurface only scattering of the skin on the left side. Lighting in texture (unwrapped
UV) space on the right.

Unite template
Back Scattering
• Translucency Map
• Adds intensity to sub-
dermal layer
42

R
43
2 Specular lobes

Unite template
R
44
Final look of the skin.

Unite template45
Depth Of Field
• Antialiasing aware
• FXAA based
• With Splatted Bokeh Texture

46
High variation of depth causes Depth of Field to produce noticeable aliasing on the
edges of geometry. We married Depth Of Field filter with additional FXAA based
antialiasing technique driven by radius of Circle of Confusion to blur neighboring pixels.
Example of aliasing caused by discontinuities in depth buffer on the left side. Our
approach on the right side.

Depth Of Field using “standard” blur - top image.
Detecting bright pixels and “splattering” bokeh sprites on top - bottom image.

48
Thank you!
Talented people, come join our team working on Showcases for Unity!
http://unity3d.com/jobs

References
49
 [Bavoil08] Image-Space Horizon-Based Ambient Occlusion. L. Bavoil, M. Sainz, 2008
 [Catmull78] Recursively generated B-spline surfaces on arbitrary topological meshes. E. Catmull, J.
Clark, 1978
 [CookTorrance82] A Reflectance Graphics Model for Computer graphic. R. L. Cook, K. E. Torrance,
1982
 [Hoffman10] Crafting Physically Motivated Shading Models for Game Development. N. Hoffman,
2010
 [Lottes09] FXAA. T. Lottes, 2009
 [Mental07] mental ray Architectural and Design Visualization Shader Library. mental images GmbH,
2007
 http://www.mentalimages.com/fileadmin/user_upload/PDF/arch_and_design.pdf
 [Mikkelsen09] Microfacet Based Bidirectional Reflectance Distribution Function. M. Mikkelsen, 2009
 [Nießner12] Feature-Adaptive GPU Rendering of Catmull-Clark Subdivision Surfaces. M. Nießner,
C. Loop, M. Meyer, T. Derose, 2012
 [Olano10] LEAN Mapping. M. Olano, D. Baker, 2010
 [Schüler11] The Blinn-Phong Normalization Zoo. C. Schüler, 2011
 http://www.thetenthplanet.de/archives/255
 [Zhukov98] An ambient light illumination model. S. Zhukov, A. Inoes, G. Kronin, 1998

FMX2013: Butterfly Effect

More Related Content

What's hot (18)

Similar to FMX2013: Butterfly Effect (20)

More from Renaldas Zioma (8)

Recently uploaded (20)

FMX2013: Butterfly Effect