Persistent RNNs: Stashing Recurrent Weights On-Chip
1 like486 views
The presentation by Gregory Diamos discusses how persistent RNNs can enhance machine learning performance by efficiently utilizing GPU resources, particularly addressing the issues of data-parallelism and optimization limits. It emphasizes the potential of scaling RNNs with upcoming GPU technology while encouraging the development of more efficient algorithms. The speaker advocates for pushing performance boundaries towards achieving significant computational outputs with minimal energy consumption.
Persistent RNNs: Stashing Recurrent Weights On-Chip
- 1. Persistent RNNs (stashing recurrent weights on-chip) Presenter: Gregory Diamos Silicon Valley AI Lab Baidu Jun 20, 2016 Presenter: Gregory Diamos Persistent RNNs
- 2. Machine learning has benefited greatly from faster computer systems. GPUs in particular, have delivered a step forward. Presenter: Gregory Diamos Persistent RNNs
- 3. Imagine the problems that you could solve with even faster systems. Presenter: Gregory Diamos Persistent RNNs
- 4. HPC is an opportunity 10,000x TitanX GPU Fastest supercomputer Presenter: Gregory Diamos Persistent RNNs
- 5. Limits of data-parallelism Presenter: Gregory Diamos Persistent RNNs
- 6. Hardware limits wall-clocktimetoconvergence mini-batch size inefficient hardware Hardware becomes less efficient at small batch sizes. Presenter: Gregory Diamos Persistent RNNs
- 7. Optimization limits wall-clocktimetoconvergence mini-batch size inefficient optimization Optimization algorithms perform more work at large batch sizes. Presenter: Gregory Diamos Persistent RNNs
- 8. Mini-batch limits wall-clocktimetoconvergence mini-batch size inefficient hardware inefficient optimization These effects combine to limit the maximum number of GPUs. Presenter: Gregory Diamos Persistent RNNs
- 9. Persistent RNNs Open source CUDA implementation: https://github.com/baidu-research/persistent-rnns Presenter: Gregory Diamos Persistent RNNs
- 10. Persistent RNN Details Presenter: Gregory Diamos Persistent RNNs
- 11. Persistent RNNs weights GEMM GEMM GEMM GEMM Persistent RNN weights weights weights weights data0 data1 data2 data3 data4 data0 data1 data2 data3 data4 RNNs built on GEMM routines reload the weights each timestep. However, the weights are constant, and this is wasteful. Presenter: Gregory Diamos Persistent RNNs
- 12. Cache weights in registers weights GPU thread registers datapath Presenter: Gregory Diamos Persistent RNNs
- 13. A global barrier data0 GPU data1 GPU barrier Presenter: Gregory Diamos Persistent RNNs
- 14. Experiments Presenter: Gregory Diamos Persistent RNNs
- 15. Scaling to 128 GPUs Presenter: Gregory Diamos Persistent RNNs
- 16. Exploring deep residual RNNs Presenter: Gregory Diamos Persistent RNNs
- 17. Pascal and future Future GPUs will enable bigger and faster RNN layers. Presenter: Gregory Diamos Persistent RNNs
- 18. Three challenges Presenter: Gregory Diamos Persistent RNNs
- 19. Close the gap with the fastest supercomputers. Presenter: Gregory Diamos Persistent RNNs
- 20. Do not settle for inefficient algorithms. Presenter: Gregory Diamos Persistent RNNs
- 21. Push performance to the edge of physical limits. 10 PetaFlops in 300 Watts. 150 ExaFlops in 25 MegaWatts. Presenter: Gregory Diamos Persistent RNNs