OpenCL caffe IWOCL 2016 presentation final

Editor's Notes

• #2: This is Hugh Perkins. Take the opportunity to Say something about your OpenCL DNN work! I am delivering the talk on behalf of the author Junli. She could not come to make the presentation due to visa issues. When the authors submitted the paper, Junli was at AMD. But now she is working at Tesla. The project if an open sourced project. So the talk is not on behalf of any company at this point.
• #3: Deep learning applications has posed great challenges to system design Deep learning applications are both compute intensive and memory intensive, Due to the high complexicity of DNN models, and Big Data input. The past 3 years, DNN models keep increase the number of the parameters, increase from millions to billons. The Big data samples to train a model can be from hundreds of millions to hundreds of billions depending on different applications. Yet the data is said to increase 10x per year. Thus Deep learning applications has posed great challenges to system design.
• #4: The transition of DNN systems DNN has becoming the hottest internet trends in the past two years. The current trend indicates that DNN applications will be everywhere. Cross platform compatibility is becoming a challenge. I see this as opportunity for OpenCL.. Let me first introduce the DNN process a little bit, a DNN model will be trained on supercomputers with Big Data input, later deployed on could for online recognition, these days, DNN apps are also deployed on mobiles and wearable. I have listed the scale of each category and opportunity. Offline training supercomputers will be small scale, in hundreds of CPU + GPU machines, Performance will critical for internet players, but our performance is currently not competitive. online cloud will be much bigger scale, from 100k to 1M, and is more tolerate with performance, there is potential opportunity for AMD to win with better performance per dollar. Timing is right. the online cloud is in transition form CPU cluster into CPU + GPU cluster. Mobile we have no products.
• #6: Due to DNN models' high complexity, we use a two-phase strategy. First we introduce the OpenCL porting strategies that guarantee algorithm convergence; Then we analyze OpenCL's performance bottlenecks in DNN domain and propose a few optimization techniques improve hardware resources utilization and boost OpenCL runtime eciency.
• #8: Layer1: C++ interfaces (for domain experts) Layer2: OpenCL wrapper hides hardware details (for systems) Layer3: Underlying GPU kernels (for device level optimizations
• #9: DNN is a deep layered structure, if we port all layers together, the algorithm convergence can be easily destroyed and it is very painful to debug, due to millions of parameters and stochastic gradients. The challenges of gradient is a well know challenge for any machine learning framework development. For porting from CUDA to OpencL, because the hardware abstraction details, data transfer and other aspects are all different, we
• #14: Optimal batch size 16 or 32 on AMD GPUs
• #15: Even if you have fast math librarie, DNN has this nature of workload imbalance for each layers. As shown in the following figure, Alexnet model, each conv layer has different number of channels, as the layer goes deeper, feature maps size reduces, these factors decide that the amount of computing for each layer are all different When we map this layer to the GPU, this will result some layer keeps the GPU busy, while other layers may use only part of the GPU. So, how to balance GPU utilizations? Luckily, we can make use of model parallelism. Can you explain what model parallelism is? Model parallelism refers to divide the whole model into smaller chunks or branches and run it on different computing nodes. This is very useful when the whole model can not be fit into the same GPU or accelerate speed by mapping to different computing nodes. This same algorithm parallelism can be used to balance the workload imbalance between layers. Usually the DNN frameworks will compute each black in sequential manner. We utilize this to apply multiple command queues. These days, as DNN models start to show more braches in the structure, for example, the google inception network, most layers has 3 or 4 branches. this technique can be used to run each branch concurrently to improve utlization Performance effect:for small Cifar, speed by 10%.　for large ImageNet, 12% for minibatch size of 10; for large minibatch sizes, we sometimes observe a slow down due to interfering between two queues and extra synchronization overheads．　Final Performance improvement depends on layer structure, data size and hardware resources.
• #17: Need continuous community efforts to improve It is hard to achieve high utlization of hardware without low level hardware languanges and optimization Cross platform compibility can be got for free though OpenCL But you could not achieve the best performance for free. Most efficient implemenation has to always come from customized low level optimizations.