GPU Rigid Body Simulation GDC 2013

Welcome to this talk about GPU Rigid Body Dynamics. My name is Erwin Coumans and I
am leading the Bullet Physics SDK project. I have been working on Bullet 100% of my
time for about 10 years now. First as a Sony Computer Entertainment US R&D employee
and currently as AMD employee.
2

Our initial work on GPGPU acceleration in Bullet was cloth simulation. AMD contributed
OpenCL and DirectX 11 DirectCompute accelerated cloth simulation. This is available in
the Bullet 2.x SDK at http://bullet.googlecode.com
Here is a screenshot of the Samurai Demo with Bullet GPU cloth integrated.
3

More recently one of the engineers in my team worked on GPU accelerated hair
simulation. This work was presented at the Vriphys 2012 conference by Dongsoo Han
and integrated in the latest Tombraider 2013 game under the name TressFX.
4

We started exploring GPU rigid body simulation at Sony Computer Entertainment US
R&D. My SCEA colleague Roman Ponomarev started the implementation in CUDA and
switched over to OpenCL once the SDKs and drivers became available. I’ll briefly discuss
the first generation GPU rigid body pipeline.
When I joined AMD, my colleague Takahiro Harada implemented most of the second
generation GPU rigid body pipeline, around 2010.
From 2011 onwards I’m working on the third generation GPU rigid body pipeline, with
pretty much the same quality as the Bullet 2.x CPU rigid body simulation. This is work in
progress, but the results are already useful.
One of the GPU rigid body demos uses a simplified triangle mesh of the Samurai Cloth
demo, as you can see in this screenshot. The triangle mesh is reduced to around
100.000 triangles (there is no instancing used).
5

Before going into details, the most essential parts of a rigid body are the world space
position and orientation of rigid bodies. In our case the position defines the center of
mass, using a float3. A 4rd unused w-component is often added to make this structure
16 byte aligned. Such alignment can help SIMD operations.
The orientation defines the inertial basis frame of a rigid body and it can be stored as a
3x3 matrix or as a quaternion. For Bullet 2.x we used a 3x3 matrix, but in Bullet 3.x we
switched over to quaternions because it is more memory efficient.
6

The rigid body bodies are simulated forward in time using the linear and angular
velocity, using an integrator such as Euler integration. Both linear and angular velocity is
stored as a 3-component float.
7

If we only look at the position and linear velocity, the C/C++ version is just a one liner.
This would be sufficient for most particle simulations.
8

As you can see the OpenCL version of this kernel is almost identical to the C/C++ version
in previous slide. Instead of passing the body index through an argument, the OpenCL
work item can get its index using the get_global_id() API.
9

We can apply the gravity in this kernel as well. Alternatively the gravity can be applied
inside the constraint solver, for better quality.
10

When we introduce angular velocity and the update of the orientation, the code
becomes a bit more complicated. Here is the OpenCL kernel implementation, with
identical functionality to the Bullet 2.x position and orientation update.
We have the option to apply some damping and to clamp the maximum angular
velocity. For improved accuracy in case of small angular velocities, we use a Taylor
expansion to approximate the sinus function.
11

Here is the host code to setup the arguments and the call to execute the kernel on the
OpenCL device.
If we would have no constraints or collisions, we would be finished with our rigid body
simulation,
12

Once we add collision detection and constraint solving a typical discrete rigid body
simulation pipeline looks as in this slide. The white stages below are trivial to parallelize,
or embarrassingly parallel, so we don’t spend further time on it in this presentation.
The detection of potential overlapping pairs in blue, the contact point generation and
reduction in yellow and constraint solving stages in orange are the main stages that we
need to deal with.
13

Here is an overview of the 50 kernels of our current third generation GPU rigid body
pipeline. Note that 100% of the rigid body simulation is executed on the OpenCL
GPU/device.
The green kernels are general useful parallel primitives, such as a parallel radix sort, a
prefix scan and such. Often they are provided by a hardware vendor, such as AMD, Intel
or NVIDIA. Bullet includes its own implementation for parallel primitives, that are
reasonably well optimized to work well on most OpenCL GPU devices.
The kernels in blue are related to finding the potential overlapping pairs, also known as
broadphase collision detection. The yellow kernels are related to contact computation
and reduction, also known as narrowphase collision detection, as well as bounding
volume hieraries to deal with triangle meshes, often called midphase. The constraint
solver kernels are colored orange.
There is still more work to be done in various areas, such as supporting more collision
shape types and constraint types aside from contact constraints.
14

Before going into details how we optimized the various rigid body pipeline stages, here
is a quick overview of a system with Host and Device.
Generally the host will upload the data from Global Host Memory to Global Device
memory, upload the compiled kernel code to the device and enqueue the work
items/groups so that the GPU executes them in the right order.
The connection between host and device, here the orange arrow, is a PCI express bus in
current PCs. This connection will improve a lot in upcoming devices.
When using an integrated GPU we the global memory might be shared and unified
between host and device.
15

A typical GPU device contains multiple Compute Units, where each Compute Unity can
execute many threads, or work items, in parallel. For an AMD GPU there are usually 64
threads executed in parallel for each Compute Unit, we call this a wavefront. On NVIDIA
GPUs there are usually 32 threads or work items executed in parallel for each Compute
Unity, they call this parallel group of 32 threads a warp.
Each thread, or work item, has private memory. Private Memory on a 7970 is a 64kb
register file pool, each work item gets some registers allocated.
Each Compute Unit has shared local memory, that can accessed by all threads in the
Compute Unit. Shared Local Memory or LDS in AMD terminology, is 64kb on a 7970.
Local atomics can synchronize operations between threads in a Compute Unit on shared
local memory
The Global Device Memory can be access by all Compute Units. You can use global
atomic operations on Global Device Memory.
Shared Local Memory is usually an order of magnitude faster than global device
memory, so it is important to make use of this for an efficient GPU implementation.
The programmer will distribute work into work groups, and the GPU has the
responsibility to map the Work Groups to Compute Units. This is useful to make the
solution scalable: we can process our work groups on small devices such as cell phone
16

with just a single or few Compute Units, or we can execute the same program on a high-
end discrete GPU with many Compute Units.
16

Here is a quick view of the GPU hardware as presented by the operating system. On the
right we see a more detailed view from the GPU hardware through the OpenCL API. We
can make use of the low level information to optimize our program.
17

The same information is available on Mac OSX or Linux or other OpenCL devices.
18

It appears that some OpenCL implementation is available as “easter egg” as parts of
some OpenGL ES drivers in Android mobile devices such as the Nexus 4 or Nexus 10.
The Playstation 4 seems also suitable for GPGPU development: The system is also set up
to run graphics and GPGPU computational code in parallel, without suspending one to
run the other.
Chris Norden says that Sony has worked to carefully balance the two processors to
provide maximum graphics power of 1.843 teraFLOPS at an 800Mhz clock speed while
still leaving enough room for computational tasks. The GPU will also be able to run
arbitrary code, allowing developers to run hundreds or thousands of parallelized tasks
with full access to the system's 8GB of unified memory.
19

Here is the first generation of our GPU rigid body pipeline. A lot of simplification is used,
so we could re-use a lot of work that was already available in GPU particle simulation.
20

The uniform grid as already available in some OpenCL sdks, so we used this as a starting
point for the GPU rigid body pipeline.
21

We use parallel primitives a lot when implementing GPU friendly algorithms, for
example a parallel radix sort and prefix scan is used in a parallel uniform grid solution.
22

The contact point generation was also simplified a lot, by approximating collision shapes
using a collection of spheres. The process of creating a sphere collection, or voxelization,
could be performed on CPU or GPU.
23

Here are some screenshots of a simulation of some Stanford bunnies dropping on a
plane.
24

We use an iterative constraint solver based on Projected Gauss Seidel. In a nutshell, the
velocity of bodies needs to be adjusted to avoid violations in position (penetration) and
velocity. This is done by sequential application of impulses.
Pairwise constraints, such as contact constraints between two bodies need to be solved
sequentially in a Gauss Seidel style algorithm, so that the most up-to-date velocity is
available for each constraint.
An alternative would be to use Jacobi style constraint solving, where all constraints use
“old” velocities, so we can trivially parallelize it. The drawback is that Jacobi doesn’t
converge as well as Gauss Seidel.
25

We cannot trivially update the velocities for bodies in each constraint in parallel,
because we would have write conflicts. For example the contact constraint 1 in the
picture tries to write velocity of object B (and A), while contact constraint 2 tries to
update the same body B (and C). To avoid such write conflicts, we can sort the
constraints in batches, also known as graph coloring. All the constraints in each batch
are guaranteed not to access the same dynamic rigid body.
Note that non-movable or static rigid bodies, with zero mass, can be ignored: their
velocity is not updated.
26

We first implemented the graph coloring or batch creation on the CPU host side. The
code is very simple, as you can see. We iterate over all unprocessed constraints, and
keep track of the used bodies for this batch. If the two bodies in a constraint are still
available for this batch, we assign the batch index to the contraints. Otherwise the
constraint will be processed in another batch.
There is a small optimization that we can clear the used bodies when we reach the
device SIMD width, because only 32 or 64 threads are processed in parallel on current
GPU OpenCL devices.
27

We can execute the same code on the GPU in a single Compute Unit, and use local
atomics to synchronize between the threads. It is not optimal, but a good starting point.
In a later pipeline we have improved the batch creation.
28

The uniform grid works best if all objects are of similar size. In the second GPU rigid
body pipeline became more general, so we can deal with objects of different sizes.
Also we use an improved collision shape representation, instead of a sphere collection,
and we can use the full GPU capacity for constraint batch creation and solving.
29

Once we use other shapes then just spheres, it becomes useful to use a simplified
bounding volume for the collision shape. We use axis aligned bounding boxes on the
GPU, just like we do on CPU.
30

We can use the support mapping function to generate local space axis aligned bounding
boxes.
31

See the book "Collision Detection in Interactive 3D Environments", 2004, Gino Van Den
Bergen for more information about the support mapping.
33

The GPU is very suitable to compute a the dot product for many vertices in parallel.
Alternatively we can use the GPU cube mapping hardware to approximage the bounding
volume computation.
34

Instead of re-computing the world space bounding volume each time, we can
approximate the world space bounding volume using the pre-computed local space
AABB.
35

Here is the kernel code to compute the world space AABB from the local space AABB.
Basically the absolute dot product using the rows of the rotation matrix and its extents
will produce the world space AABB.
Note that the rows of the 3x3 orientation matrix are basically the axis of the local space
AABB.
37

The uniform grid doesn’t deal well with varying object sizes, so our first attempt to deal
with mixed sized objects was to use multiple broadphase algorithms, dependent on the
object size.
Computation of potential overlapping pairs between small objects can be done using a
uniform grid, while a pair search that involves larger objects could be done on CPU (or
GPU) using a different algorithm.
38

If we revisit the GPU hardware, we note that Compute Units perform their work
independently. So it would be good to generate batches in a way so that we can use
multiple independent Compute Units.
43

We perform the batching of constraints in two stages: the first stage splits the
constraints so that they can be processed by independent Compute Units. We use a
uniform grid for this, where non-neighboring cells can be processed in parallel.
44

The second stage batching within a Compute Unit is similar to the first GPU rigid body
pipeline, but we added some optimizations. Takahiro Harada wrote a SIGGRAPH paper
with more details.
45

We typically use around 4 to 10 iterations in the solver, and in each iteration we have
around 10 to 15 batches. The large amount of kernel launches can become a
performance bottleneck, so we implemented a GPU side solution that only requires a
single kernel launch per iteration.
46

Our third generation GPU rigid body pipeline has the same quality and is as general as
the regular discrete Bullet 2.x rigid body pipeline on the CPU.
47

The regular 3-axis sweep and prune broadphase pair search algorithms incrementally
updates the sorted AABBs for each of the 3 world axis in 3D.
We sort the begin and end points for each AABB to their new position, one object at a
time, using swap operations. We incrementally add or remove pairs during those swap
operations.
This exploits spatial and temporal coherency: objects don’t move a lot between frames.
This process is difficult to parallelizing due to data dependencies: globally changing data
structure would require locking.
48

We modify the 3D axis sweep and prune algorithm to make it more suitable for GPU,
while keeping the benefits of the incremental pair search during the swaps.
49

Instead of performing the incremental sort and swap together, we perform the sorting in
parallel as one stage, and perform the swaps in a separate stage using read-only
operations.
We maintain the previous sorted axis and compare it with the updated sorted axis. Each
object can perform the swaps by traversing the previous and current sorted elements,
without modifying the data.
Although unusual, we can detect rare degenerate cases that would lead to too many
swaps in the 3-axis SAP algorithm, and do a fallback to another broadphase.
Such fallback is not necessary in most practical simulations. Still, generally it can be a
good idea to mix multiple broadphase algorithms to exploit the best properties out of
each broadphase.
50

For convex objects we use the separating axis test and Sutherland Hodgeman clipping
and contact reduction to generate contacts.
Concave compound shapes and concave triangle meshes produce pairs of convex
shapes, that are processed alongside the convex-convex pairs that are produced by the
broadphase.
51

The contact clipping can generate a lot of contact points. We reduce the number of
contacts between two convex polyhedral to a maximum of 4 points.
We always keep the contact point with the deepest penetration, to make sure the
simulation gets rid of penetrations.
We keep 3 other contact points with maximum or minimum projections along two
orthogonal axis in the contact plane.
54

The convex-convex contact generation can be used between convex objects, but also
between the convex child shapes in a compound or for convex against individual
triangles of a triangle mesh.
We unify the pairs generated by the broadphase pair search, compound child pair search
and concave triangle mesh pairs so they can use the same narrowphase convex collision
detection routines.
The code for the separating axis test, contact clipping and contact reduction is very
large. This leads to very inefficient GPU usage, because of tread divergence and register
spill.
We break the the convex-convex contact generation up into a pipeline that consists of
many stages (kernels).
55

When a convex or compound shape collides against a concave triangle mesh, we
perform a BVH query using the world space AABB of the convex or compound shape,
against the precomputed BVH tree of the triangle mesh.
We optimized the BVH tree traversal to make it GPU friendly. Instead of recursion, we
iteratively traverse over an array of nodes.
We divide the BVH tree into smaller subtrees that fit in shared local memory, so they
can be processed by multiple threads in a work group.
56

Instead of Projected Gauss Seidel, we can also use the iterative Jacobi scheme to solve
the constraints.
Jacobi converges slower than PGS. We can improve the convergence by splitting the
objects into multiple “split bodies” for each contact.
Then, after solving an iteration using the Jacobi solver applied to the split bodies, we
average the velocities for all the split bodies and update the “original” body.
The idea is explained in the “Mass Splitting for Jitter-Free Parallel Rigid Body Simulation”
SIGGRAPH 2012 paper.
We can improve upon this scheme by mixing PGS and Jacobi even further. We know that
within a Compute Unit, only 64 threads are executed in parallel,
so we can only apply Jacobi within the active threads of a wavefront/warp and use PGS
outside (by updating the rigid body positions using the averaged split bodies).
57

We use plain OpenGL 3.x and a simple user interface called GWEN (gui without
extravagant nonsense) with some cross-platform classes to open an OpenGL 3.x context
for Windows, Linux and Mac OSX.
58

We performed some preliminary timings to make sure that our GPU rigid body pipeline
works well on GPU hardware of varying vendors.
Note that those numbers don’t give a good indication, because we haven’t optimized
the GPU rigid body pipeline good enough yet.
60

Here are some preliminary performance timings. Note that we haven’t optimized the
GPU rigid body pipeline well, so the numbers can be improved a lot.
61

We test our GPU rigid body pipeline on Windows, Mac OSX and Linux using AMD,
NVIDIA and Intel GPUs. The source code will be available at the github repository.
62

GPU Rigid Body Simulation GDC 2013

More Related Content

What's hot (18)

Viewers also liked (7)

Similar to GPU Rigid Body Simulation GDC 2013 (20)

Recently uploaded (20)

GPU Rigid Body Simulation GDC 2013