Accelerating microbiome research with OpenACC
0 likes119 views
The document discusses revolutionizing microbiome research by leveraging OpenACC for GPU computing, improving the performance of computing Unifrac distance matrices. It outlines the significant speedups achieved by transitioning from CPU to GPU, noting a 20x speedup on modest datasets and highlights the optimizations necessary for maximizing efficiency. The work was funded by various grants from the NSF and NIH, and suggestions for enhancing multi-GPU and non-Linux support are included.
![Computing UniFrac
• Most compute localized
in a tight loop
• Operating on
a stripe range
for(unsigned int stripe = start;
stripe < stop; stripe++) {
dm_stripe = dm_stripes[stripe];
for(unsigned int j = 0;
j < n_samples / 4; j++) {
int k = j * 4;
double u1 = emb[k];
double u2 = emb[k+1];
double v1 = emb[k + stripe + 1];
double v2 = emb[k + stripe + 2];
…
dm_stripe[k] += (u1-v1)*length;
dm_stripe[k+1] += (u2-v2)*length;
}
}
Invoked many times with distinct emp[:] buffers](https://image.slidesharecdn.com/openacc20unifrac-200904224619/85/Accelerating-microbiome-research-with-OpenACC-6-320.jpg)
![Computing UniFrac
• Most compute localized
in a tight loop
• Operating on
a stripe range
for(unsigned int stripe = start;
stripe < stop; stripe++) {
dm_stripe = dm_stripes[stripe];
for(unsigned int j = 0;
j < n_samples / 4; j++) {
int k = j * 4;
double u1 = emb[k];
double u2 = emb[k+1];
double v1 = emb[k + stripe + 1];
double v2 = emb[k + stripe + 2];
…
dm_stripe[k] += (u1-v1)*length;
dm_stripe[k+1] += (u2-v2)*length;
}
}
Intel Xeon E5-2680 v4 CPU
(using all 14 cores)
800 minutes (13 hours)
Modest size EMP dataset
Invoked many times with distinct emp[:] buffers](https://image.slidesharecdn.com/openacc20unifrac-200904224619/85/Accelerating-microbiome-research-with-OpenACC-7-320.jpg)
![Porting to GPU
• OpenACC makes it trivial to
port to GPU compute.
• Just decorate with a pragma.
• But needed minor refactoring
to have a unified buffer.
(Was array of pointers)
#pragma acc parallel loop collapse(2)
present(emb,dm_stripes_buf)
for(unsigned int stripe = start;
stripe < stop; stripe++) {
for(unsigned int j = 0;
j < n_samples / 4; j++) {
int idx =(stripe-start_idx)*n_samples;
double *dm_stripe =dm_stripes_buf+idx;
int k = j * 4;
double u1 = emb[k];
double u2 = emb[k+1];
double v1 = emb[k + stripe + 1];
double v2 = emb[k + stripe + 2];
…
dm_stripe[k] += (u1-v1)*length;
dm_stripe[k+1] += (u2-v2)*length;
}
}
Invoked many times with distinct emp[:] buffers](https://image.slidesharecdn.com/openacc20unifrac-200904224619/85/Accelerating-microbiome-research-with-OpenACC-8-320.jpg)
![Porting to GPU
#pragma acc parallel loop collapse(2)
present(emb,dm_stripes_buf)
for(unsigned int stripe = start;
stripe < stop; stripe++) {
for(unsigned int j = 0;
j < n_samples / 4; j++) {
int idx =(stripe-start_idx)*n_samples;
double *dm_stripe =dm_stripes_buf+idx;
int k = j * 4;
double u1 = emb[k];
double u2 = emb[k+1];
double v1 = emb[k + stripe + 1];
double v2 = emb[k + stripe + 2];
…
dm_stripe[k] += (u1-v1)*length;
dm_stripe[k+1] += (u2-v2)*length;
}
}
Modest size EMP dataset
NVIDIA Tesla V100
(using all 84 SMs)
92 minutes (1.5 hours) Was 13h on CPU
• OpenACC makes it trivial to
port to GPU compute.
• Just decorate with a pragma.
• But needed minor refactoring
to have a unified buffer.
(Was array of pointers)
Invoked many times with distinct emp[:] buffers](https://image.slidesharecdn.com/openacc20unifrac-200904224619/85/Accelerating-microbiome-research-with-OpenACC-9-320.jpg)
![Optimization step 1
Modest size EMP dataset
92 mins before, was 13h on CPU
• Cluster reads and minimize writes
• Fewer kernel invocations
• Memory writes much more
expensive than memory reads.
• Also undo manual unrolls
• Were optimal for CPU
• Bad for GPU
• Properly align
memory buffers
• Up to 5x slowdown
when not aligned
#pragma acc parallel loop collapse(2)
present(emb,dm_stripes_buf,length)
for(unsigned int stripe = start;
stripe < stop; stripe++) {
for(unsigned int k = 0;
k < n_samples ; k++) {
…
double my_stripe = dm_stripe[k];
#pragma acc loop seq
for (unsigned int e=0;
e<filled_embs; e++) {
uint64_t offset = n_samples*e;
double u = emb[offset+k];
double v = emb[offset+k+stripe+ 1];
my_stripe += (u-v)*length[e];
}
…
dm_stripe[k] += (u1-v1)*length;
}
}
NVIDIA Tesla V100
(using all 84 SMs)
33 minutes
Invoked fewer times due to batched emp[:] buffers](https://image.slidesharecdn.com/openacc20unifrac-200904224619/85/Accelerating-microbiome-research-with-OpenACC-10-320.jpg)
![Optimization step 2
• Reorder loops to maximize
cache reuse.
#pragma acc parallel loop collapse(3)
present(emb,dm_stripes_buf,length)
for(unsigned int sk = 0;
sk < sample_steps ; sk++) {
for(unsigned int stripe = start;
stripe < stop; stripe++) {
for(unsigned int ik = 0;
ik < step_size ; ik++) {
unsigned int k = sk*step_size + ik;
…
double my_stripe = dm_stripe[k];
#pragma acc loop seq
for (unsigned int e=0;
e<filled_embs; e++) {
uint64_t offset = n_samples*e;
double u = emb[offset+k];
double v = emb[offset+k+stripe+ 1];
my_stripe += (u-v)*length[e];
}
…
dm_stripe[k] += (u1-v1)*length;
}
}
}
NVIDIA Tesla V100
(using all 84 SMs)
12 minutes Was 33 mins
Modest size
EMP dataset](https://image.slidesharecdn.com/openacc20unifrac-200904224619/85/Accelerating-microbiome-research-with-OpenACC-11-320.jpg)
![Optimization step 2
• Reorder loops to maximize
cache reuse.
• CPU code
also benefitted
from
optimization
#pragma acc parallel loop collapse(3)
present(emb,dm_stripes_buf,length)
for(unsigned int sk = 0;
sk < sample_steps ; sk++) {
for(unsigned int stripe = start;
stripe < stop; stripe++) {
for(unsigned int ik = 0;
ik < step_size ; ik++) {
unsigned int k = sk*step_size + ik;
…
double my_stripe = dm_stripe[k];
#pragma acc loop seq
for (unsigned int e=0;
e<filled_embs; e++) {
uint64_t offset = n_samples*e;
double u = emb[offset+k];
double v = emb[offset+k+stripe+ 1];
my_stripe += (u-v)*length[e];
}
…
dm_stripe[k] += (u1-v1)*length;
}
}
}
NVIDIA Tesla V100
(using all 84 SMs)
12 minutes Was 33 mins
Modest size
EMP dataset
Xeon E5-2680 v4 CPU
(using all 14 cores)
193minutes (~3 hours)
Originally 13h on the same CPU](https://image.slidesharecdn.com/openacc20unifrac-200904224619/85/Accelerating-microbiome-research-with-OpenACC-12-320.jpg)
Accelerating microbiome research with OpenACC
- 1. Accelerating microbiome research with OpenACC Igor Sfiligoi – University of California San Diego in collaboration with Daniel McDonald and Rob Knight OpenACC Summit 2020 – Sept 2020
- 3. We are what we eat Studies demonstrated clear link between • Gut microbiome • General human health https://www.biotechniques.com/multimedia/archive/00252/microbiome2_252150a.jpg
- 4. UniFrac distance Need to understand how similar pairs of microbiome samples are with respect to the evolutionary histories of the organisms. UniFrac distance matrix • Samples where the organisms are all very similar from an evolutionary perspective will have a small UniFrac distance. • On the other hand, two samples composed of very different organisms will have a large UniFrac distance. Lozupone and Knight Applied and environmental microbiology 2005
- 5. Computing UniFrac • Matrix can be computed using a striped pattern • Each stripe can be computed independently • Easy to distribute over many compute units CPU1 CPU2
- 6. Computing UniFrac • Most compute localized in a tight loop • Operating on a stripe range for(unsigned int stripe = start; stripe < stop; stripe++) { dm_stripe = dm_stripes[stripe]; for(unsigned int j = 0; j < n_samples / 4; j++) { int k = j * 4; double u1 = emb[k]; double u2 = emb[k+1]; double v1 = emb[k + stripe + 1]; double v2 = emb[k + stripe + 2]; … dm_stripe[k] += (u1-v1)*length; dm_stripe[k+1] += (u2-v2)*length; } } Invoked many times with distinct emp[:] buffers
- 7. Computing UniFrac • Most compute localized in a tight loop • Operating on a stripe range for(unsigned int stripe = start; stripe < stop; stripe++) { dm_stripe = dm_stripes[stripe]; for(unsigned int j = 0; j < n_samples / 4; j++) { int k = j * 4; double u1 = emb[k]; double u2 = emb[k+1]; double v1 = emb[k + stripe + 1]; double v2 = emb[k + stripe + 2]; … dm_stripe[k] += (u1-v1)*length; dm_stripe[k+1] += (u2-v2)*length; } } Intel Xeon E5-2680 v4 CPU (using all 14 cores) 800 minutes (13 hours) Modest size EMP dataset Invoked many times with distinct emp[:] buffers
- 8. Porting to GPU • OpenACC makes it trivial to port to GPU compute. • Just decorate with a pragma. • But needed minor refactoring to have a unified buffer. (Was array of pointers) #pragma acc parallel loop collapse(2) present(emb,dm_stripes_buf) for(unsigned int stripe = start; stripe < stop; stripe++) { for(unsigned int j = 0; j < n_samples / 4; j++) { int idx =(stripe-start_idx)*n_samples; double *dm_stripe =dm_stripes_buf+idx; int k = j * 4; double u1 = emb[k]; double u2 = emb[k+1]; double v1 = emb[k + stripe + 1]; double v2 = emb[k + stripe + 2]; … dm_stripe[k] += (u1-v1)*length; dm_stripe[k+1] += (u2-v2)*length; } } Invoked many times with distinct emp[:] buffers
- 9. Porting to GPU #pragma acc parallel loop collapse(2) present(emb,dm_stripes_buf) for(unsigned int stripe = start; stripe < stop; stripe++) { for(unsigned int j = 0; j < n_samples / 4; j++) { int idx =(stripe-start_idx)*n_samples; double *dm_stripe =dm_stripes_buf+idx; int k = j * 4; double u1 = emb[k]; double u2 = emb[k+1]; double v1 = emb[k + stripe + 1]; double v2 = emb[k + stripe + 2]; … dm_stripe[k] += (u1-v1)*length; dm_stripe[k+1] += (u2-v2)*length; } } Modest size EMP dataset NVIDIA Tesla V100 (using all 84 SMs) 92 minutes (1.5 hours) Was 13h on CPU • OpenACC makes it trivial to port to GPU compute. • Just decorate with a pragma. • But needed minor refactoring to have a unified buffer. (Was array of pointers) Invoked many times with distinct emp[:] buffers
- 10. Optimization step 1 Modest size EMP dataset 92 mins before, was 13h on CPU • Cluster reads and minimize writes • Fewer kernel invocations • Memory writes much more expensive than memory reads. • Also undo manual unrolls • Were optimal for CPU • Bad for GPU • Properly align memory buffers • Up to 5x slowdown when not aligned #pragma acc parallel loop collapse(2) present(emb,dm_stripes_buf,length) for(unsigned int stripe = start; stripe < stop; stripe++) { for(unsigned int k = 0; k < n_samples ; k++) { … double my_stripe = dm_stripe[k]; #pragma acc loop seq for (unsigned int e=0; e<filled_embs; e++) { uint64_t offset = n_samples*e; double u = emb[offset+k]; double v = emb[offset+k+stripe+ 1]; my_stripe += (u-v)*length[e]; } … dm_stripe[k] += (u1-v1)*length; } } NVIDIA Tesla V100 (using all 84 SMs) 33 minutes Invoked fewer times due to batched emp[:] buffers
- 11. Optimization step 2 • Reorder loops to maximize cache reuse. #pragma acc parallel loop collapse(3) present(emb,dm_stripes_buf,length) for(unsigned int sk = 0; sk < sample_steps ; sk++) { for(unsigned int stripe = start; stripe < stop; stripe++) { for(unsigned int ik = 0; ik < step_size ; ik++) { unsigned int k = sk*step_size + ik; … double my_stripe = dm_stripe[k]; #pragma acc loop seq for (unsigned int e=0; e<filled_embs; e++) { uint64_t offset = n_samples*e; double u = emb[offset+k]; double v = emb[offset+k+stripe+ 1]; my_stripe += (u-v)*length[e]; } … dm_stripe[k] += (u1-v1)*length; } } } NVIDIA Tesla V100 (using all 84 SMs) 12 minutes Was 33 mins Modest size EMP dataset
- 12. Optimization step 2 • Reorder loops to maximize cache reuse. • CPU code also benefitted from optimization #pragma acc parallel loop collapse(3) present(emb,dm_stripes_buf,length) for(unsigned int sk = 0; sk < sample_steps ; sk++) { for(unsigned int stripe = start; stripe < stop; stripe++) { for(unsigned int ik = 0; ik < step_size ; ik++) { unsigned int k = sk*step_size + ik; … double my_stripe = dm_stripe[k]; #pragma acc loop seq for (unsigned int e=0; e<filled_embs; e++) { uint64_t offset = n_samples*e; double u = emb[offset+k]; double v = emb[offset+k+stripe+ 1]; my_stripe += (u-v)*length[e]; } … dm_stripe[k] += (u1-v1)*length; } } } NVIDIA Tesla V100 (using all 84 SMs) 12 minutes Was 33 mins Modest size EMP dataset Xeon E5-2680 v4 CPU (using all 14 cores) 193minutes (~3 hours) Originally 13h on the same CPU
- 13. 20x speedup on modest EMP dataset E5-2680 v4 CPU Original New GPU V100 GPU 2080TI GPU 1080TI GPU 1080 GPU Mobile 1050 fp64 800 193 12 59 77 99 213 fp32 - 190 9.5 19 31 36 64 Using fp32 adds additional boost, especially on gaming and mobile GPUs 20x V100 GPU vs Xeon CPU + 4x from general optimization
- 14. 140x speedup on cutting edge 113k sample Per chip (in minutes) 128x CPU E5-2680 v4 Original New 128x GPU V100 4x GPU V100 16x GPU 2080TI 16x GPU 1080TI fp64 415 97 14 29 184 252 fp32 - 91 12 20 32 82 Aggregated (in chip hours) 128x E5-2680 v4 CPU Original New 128x GPU V100 4x GPU V100 16x GPU 2080TI 16x GPU 1080TI fp64 890 207 30 1.9 49 67 fp32 - 194 26 1.3 8.5 22 140x V100 GPUs vs Xeon CPUs + 4.5x from general optimization
- 15. 140x V100 GPUs vs Xeon CPUs + 4.5x from general optimization 140x speedup on cutting edge 113k sample Per chip (in minutes) 128x CPU E5-2680 v4 Original New 128x GPU V100 4x GPU V100 16x GPU 2080TI 16x GPU 1080TI fp64 415 97 14 29 184 252 fp32 - 91 12 20 32 82 Aggregated (in chip hours) 128x E5-2680 v4 CPU Original New 128x GPU V100 4x GPU V100 16x GPU 2080TI 16x GPU 1080TI fp64 890 207 30 1.9 49 67 fp32 - 194 26 1.3 8.5 22 Largish CPU cluster Single node
- 16. 22x speedup on consumer GPUs Per chip (in minutes) 128x CPU E5-2680 v4 Original New 128x GPU V100 4x GPU V100 16x GPU 2080TI 16x GPU 1080TI fp64 415 97 14 29 184 252 fp32 - 91 12 20 32 82 Aggregated (in chip hours) 128x E5-2680 v4 CPU Original New 128x GPU V100 4x GPU V100 16x GPU 2080TI 16x GPU 1080TI fp64 890 207 30 1.9 49 67 fp32 - 194 26 1.3 8.5 22 22x 2080TI GPUs vs Xeons CPU, 4.5x from general optimizationConsumer GPUs slower than server GPUs but still faster than CPUs (Memory bound)
- 17. Desiderata • Support for array of pointers • Was able to work around it, but annoying • Better multi-GPU support • Currently handled with multiple processes + final merge • (Better) AMD GPU support • GCC theoretically has it, but performance in tests was dismal • Non-Linux support • Was not able to find an OpenACC compiler for MacOS or Windows
- 18. Conclusions • OpenACC made porting UniFrac to GPUs extremely easy • With a single code base • Some additional optimizations were needed to get maximum benefit • But most were needed for the CPU-only code path, too • Performance on NVIDIA GPUs great • But wondering what to do for AMD GPUs and GPUs on non-linux systems
- 19. Acknowledgments This work was partially funded by US National Science Foundation (NSF) grants OAC-1826967, OAC-1541349 and CNS-1730158, and by US National Institutes of Health (NIH) grant DP1-AT010885.