Aiding the CUDA Compiler for Fun and Profit by Joe Rowell
0 likes199 views
The document discusses optimizing CUDA programming for GPUs, focusing on the architecture's parallel nature, including threads, warps, and shared memory. It highlights best practices for efficient memory access patterns, emphasizing the use of registers over slower memory types, and provides examples of inefficient versus optimized code. Additionally, it includes specific assembly code for demonstrating warp uniformity and other performance techniques.
![SASS assembler
XMAD.MRG R3, R2.reuse, c[0x0] [0x8].H1, RZ
XMAD R0, R2.reuse, c[0x0] [0x8], R0
XMAD.PSL.CBCC R2, R2.H1, R3.H1, R0](https://image.slidesharecdn.com/joerowellp99conf2024-241016192802-8f519c65/85/Aiding-the-CUDA-Compiler-for-Fun-and-Profit-by-Joe-Rowell-7-320.jpg)
![PTX assembly for warp uniformity
.visible .func (.param .b64 func_retval0) int* warp_uniform<int*>(int*)(
.param .b64 int* warp_uniform<int*>(int*)_param_0
)
{
ld.param.v2.u32 {%r1, %r2}, [int* warp_uniform<int*>(int*)_param_0];
mov.u32 %r4, 0;
mov.u32 %r5, 31;
mov.u32 %r6, -1;
shfl.sync.idx.b32 %r8|%p1, %r1, %r4, %r5, %r6;
shfl.sync.idx.b32 %r9|%p2, %r2, %r4, %r5, %r6;
mov.b64 %rd1, {%r8, %r9};
st.param.b64 [func_retval0+0], %rd1;
ret;
}](https://image.slidesharecdn.com/joerowellp99conf2024-241016192802-8f519c65/85/Aiding-the-CUDA-Compiler-for-Fun-and-Profit-by-Joe-Rowell-10-320.jpg)
![Example of inefficient summation
#define BLOCK_SIZE 1024
template<int size>
__device__ void sum(int* out, int* in) {
__shared__ int accum[BLOCK_SIZE];
accum[threadIdx.x] = 0;
__syncthreads();
for (int i = threadIdx.x; i < size; i += BLOCK_SIZE) {
accum[threadIdx.x] += in[i];
}
__syncthreads();
*out = *out + accum[threadIdx.x];
}](https://image.slidesharecdn.com/joerowellp99conf2024-241016192802-8f519c65/85/Aiding-the-CUDA-Compiler-for-Fun-and-Profit-by-Joe-Rowell-16-320.jpg)
![Example with registers
#define BLOCK_SIZE 1024
template<int size>
__device__ void sum(int* out, int* in) {
int accum = 0;
for (int i = threadIdx.x; i < size; i += BLOCK_SIZE) {
accum += in[i];
}
*out = *out + accum;
}](https://image.slidesharecdn.com/joerowellp99conf2024-241016192802-8f519c65/85/Aiding-the-CUDA-Compiler-for-Fun-and-Profit-by-Joe-Rowell-17-320.jpg)
![Example of a poor access pattern
#define BLOCK_SIZE 1024
template<int size>
__device__ void sum(int* out, int* in) {
int accum = 0;
int offset = BLOCK_SIZE * threadIdx.x;
for (int i = 0; i < BLOCK_SIZE; i++) {
accum += in[offset + i];
}
*out = *out + accum;
}](https://image.slidesharecdn.com/joerowellp99conf2024-241016192802-8f519c65/85/Aiding-the-CUDA-Compiler-for-Fun-and-Profit-by-Joe-Rowell-20-320.jpg)
![SASS for poor access pattern
LDG.E R28, [R2+-0x3c]
LDG.E R26, [R2+-0x38]
LDG.E R27, [R2+-0x34]
LDG.E R24, [R2+-0x30]
LDG.E R25, [R2+-0x2c]
LDG.E R20, [R2+-0x28]
LDG.E R21, [R2+-0x24]
LDG.E R4, [R2+-0x20]
LDG.E R5, [R2+-0x1c]
LDG.E R6, [R2+-0x18]
LDG.E R7, [R2+-0x14]
LDG.E R8, [R2+-0x10]](https://image.slidesharecdn.com/joerowellp99conf2024-241016192802-8f519c65/85/Aiding-the-CUDA-Compiler-for-Fun-and-Profit-by-Joe-Rowell-21-320.jpg)
![SASS for good access pattern
LDG.E R10, [R2+0x1000]
LDG.E R12, [R2+0x2000]
LDG.E R13, [R2+0x3000]
LDG.E R14, [R2+0x4000]
LDG.E R16, [R2+0x5000]
LDG.E R17, [R2+0x6000]
LDG.E R18, [R2+0x7000]
LDG.E R20, [R2+0x8000]
LDG.E R21, [R2+0x9000]
LDG.E R22, [R2+0xa000]
LDG.E R24, [R2+0xb000]
LDG.E R25, [R2+0xc000]](https://image.slidesharecdn.com/joerowellp99conf2024-241016192802-8f519c65/85/Aiding-the-CUDA-Compiler-for-Fun-and-Profit-by-Joe-Rowell-22-320.jpg)
Aiding the CUDA Compiler for Fun and Profit by Joe Rowell
- 1. A ScyllaDB Community Aiding the CUDA Compiler for Fun and Profit Joe Rowell Founding Engineer at poolside.ai
- 2. Joe Rowell (he/him) Founding Engineer at poolside.ai ■ ■ “Low level performance person” ■ ■ Cryptography background ■ ■ Love arcane hardware facts
- 3. “Joe, why don’t we just write assembler?”
- 4. A brief bit of software A CUDA GPU is inherently parallel from a software perspective. ■ The smallest logical unit is a thread. ■ Threads are grouped into blocks. ■ Blocks are grouped into grids.
- 5. A brief bit of hardware A CUDA GPU is inherently parallel from a hardware perspective. ■ Threads are scheduled in groups of 32 threads, called warps. ■ Warps execute on streaming multiprocessors, or SMs. ■ Multiple SMs make up your GPU.
- 6. PTX assembler mov.u32 %r3, %ntid.x; mov.u32 %r4, %ctaid.x; mov.u32 %r5, %tid.x; mad.lo.s32 %r1, %r3, %r4, %r5; setp.ge.s32 %p1, %r1, %r2;
- 7. SASS assembler XMAD.MRG R3, R2.reuse, c[0x0] [0x8].H1, RZ XMAD R0, R2.reuse, c[0x0] [0x8], R0 XMAD.PSL.CBCC R2, R2.H1, R3.H1, R0
- 8. Write code for warps, not for threads
- 9. Warp uniform variables template <typename T> __device__ T warp_uniform(T value) { struct { union { T value; // Assume sizeof(T) <= 8 struct { uint32_t lowInt; uint32_t highInt; }; }; } p; p.value = value; p.lowInt = __shfl_sync(0xffffffff, (unsigned)p.lowInt, 0); p.highInt = __shfl_sync(0xffffffff, (unsigned)p.highInt, 0); return p.value; }
- 10. PTX assembly for warp uniformity .visible .func (.param .b64 func_retval0) int* warp_uniform<int*>(int*)( .param .b64 int* warp_uniform<int*>(int*)_param_0 ) { ld.param.v2.u32 {%r1, %r2}, [int* warp_uniform<int*>(int*)_param_0]; mov.u32 %r4, 0; mov.u32 %r5, 31; mov.u32 %r6, -1; shfl.sync.idx.b32 %r8|%p1, %r1, %r4, %r5, %r6; shfl.sync.idx.b32 %r9|%p2, %r2, %r4, %r5, %r6; mov.b64 %rd1, {%r8, %r9}; st.param.b64 [func_retval0+0], %rd1; ret; }
- 11. SASS assembly for warp uniformity
- 12. Warp shuffle for computing the minimum template<typename T> __device__ T warp_min(T val) { for (int offset = 16; offset > 0; offset /= 2) { T tval = __shfl_down_sync(0xFFFFFFFF, val, offset); if (tval < val) { val = tval; } } return val; }
- 13. Use registers where you can
- 14. Access costs ■ Registers are ~1 cycle ■ Shared memory or cache are ~5 cycles ■ Global memory is ~500 cycles
- 15. The compiler will never do this for you.
- 16. Example of inefficient summation #define BLOCK_SIZE 1024 template<int size> __device__ void sum(int* out, int* in) { __shared__ int accum[BLOCK_SIZE]; accum[threadIdx.x] = 0; __syncthreads(); for (int i = threadIdx.x; i < size; i += BLOCK_SIZE) { accum[threadIdx.x] += in[i]; } __syncthreads(); *out = *out + accum[threadIdx.x]; }
- 17. Example with registers #define BLOCK_SIZE 1024 template<int size> __device__ void sum(int* out, int* in) { int accum = 0; for (int i = threadIdx.x; i < size; i += BLOCK_SIZE) { accum += in[i]; } *out = *out + accum; }
- 18. Be aware of unfriendly access patterns
- 19. CUDA memory accesses under the hood ■ Accesses in CUDA come in three varieties: 32 bytes, 64 bytes, and 128 bytes. ■ For optimal performance, you must ensure that the memory accesses are contiguous, and ideally aligned.
- 20. Example of a poor access pattern #define BLOCK_SIZE 1024 template<int size> __device__ void sum(int* out, int* in) { int accum = 0; int offset = BLOCK_SIZE * threadIdx.x; for (int i = 0; i < BLOCK_SIZE; i++) { accum += in[offset + i]; } *out = *out + accum; }
- 21. SASS for poor access pattern LDG.E R28, [R2+-0x3c] LDG.E R26, [R2+-0x38] LDG.E R27, [R2+-0x34] LDG.E R24, [R2+-0x30] LDG.E R25, [R2+-0x2c] LDG.E R20, [R2+-0x28] LDG.E R21, [R2+-0x24] LDG.E R4, [R2+-0x20] LDG.E R5, [R2+-0x1c] LDG.E R6, [R2+-0x18] LDG.E R7, [R2+-0x14] LDG.E R8, [R2+-0x10]
- 22. SASS for good access pattern LDG.E R10, [R2+0x1000] LDG.E R12, [R2+0x2000] LDG.E R13, [R2+0x3000] LDG.E R14, [R2+0x4000] LDG.E R16, [R2+0x5000] LDG.E R17, [R2+0x6000] LDG.E R18, [R2+0x7000] LDG.E R20, [R2+0x8000] LDG.E R21, [R2+0x9000] LDG.E R22, [R2+0xa000] LDG.E R24, [R2+0xb000] LDG.E R25, [R2+0xc000]
- 23. Thank you! Let’s connect. Joe Rowell joe@poolside.ai