«Training Deep Learning Models on Multi-GPUs Systems», Dmitry Spodarets.
Download as PPTX, PDF1 like213 views
This document discusses training deep learning models using multi-GPU systems. It introduces distributed training using data or model parallelism and synchronous vs asynchronous training. Horovod is presented as a distributed training framework that can plug into TensorFlow, Keras and other frameworks to enable efficient multi-GPU and multi-node training using MPI for communication. Examples are given for using Horovod with Keras to distribute training across multiple GPUs on a single machine or multiple machines.
![Horovod Example - Keras
import keras
from keras import backend as K
import tensorflow as tf
import horovod.keras as hvd
# Initialize Horovod.
hvd.init()
# Pin GPU to be used to process local rank (one GPU per process)
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
config.gpu_options.visible_device_list = str(hvd.local_rank())
K.set_session(tf.Session(config=config))
# Build model…
model = ...
opt = keras.optimizers.Adadelta(1.0)
# Add Horovod Distributed Optimizer.
opt = hvd.DistributedOptimizer(opt)
model.compile(loss=keras.losses.categorical_crossentropy, optimizer=opt, metrics=['accuracy'])
# Broadcast initial variable states from rank 0 to all other processes.
callbacks = [hvd.callbacks.BroadcastGlobalVariablesCallback(0)]
model.fit(x_train, y_train, callbacks=callbacks, epochs=10, validation_data=(x_test, y_test))](https://image.slidesharecdn.com/spodarets-ml-meetup-odessa-27-190228104759/85/Training-Deep-Learning-Models-on-Multi-GPUs-Systems-Dmitry-Spodarets-11-320.jpg)
«Training Deep Learning Models on Multi-GPUs Systems», Dmitry Spodarets.
- 1. Training Deep Learning Models on Multi-GPUs Systems Dmitry Spodarets Odessa / Machine Learning Meetup / 27.02.2019
- 4. Distributed training Data parallel vs model parallel Faster or larger models? Distributed TensorFlow training https://www.youtube.com/watch?v=bRMGoPqsn20
- 5. Distributed training Asynchronous vs Synchronous Fast or precise? Keras - multi-GPU training is not automatic :( https://keras.io/utils/#multi_gpu_model
- 6. Bottlenecks RAM / CPU I/O Connections
- 9. Distributed training framework for TensorFlow, Keras, PyTorch, and MXNet
- 10. Horovod Stack ● Plugs into TensorFlow via custom op mechanism ● Uses MPI for worker discovery and reduction coordination ● Uses NVIDIA NCCL for actual reduction on the server and across servers
- 11. Horovod Example - Keras import keras from keras import backend as K import tensorflow as tf import horovod.keras as hvd # Initialize Horovod. hvd.init() # Pin GPU to be used to process local rank (one GPU per process) config = tf.ConfigProto() config.gpu_options.allow_growth = True config.gpu_options.visible_device_list = str(hvd.local_rank()) K.set_session(tf.Session(config=config)) # Build model… model = ... opt = keras.optimizers.Adadelta(1.0) # Add Horovod Distributed Optimizer. opt = hvd.DistributedOptimizer(opt) model.compile(loss=keras.losses.categorical_crossentropy, optimizer=opt, metrics=['accuracy']) # Broadcast initial variable states from rank 0 to all other processes. callbacks = [hvd.callbacks.BroadcastGlobalVariablesCallback(0)] model.fit(x_train, y_train, callbacks=callbacks, epochs=10, validation_data=(x_test, y_test))
- 12. Running Horovod To run on a machine with 4 GPUs: $ mpirun -np 4 -H localhost:4 -bind-to none -map-by slot -x NCCL_DEBUG=INFO -x LD_LIBRARY_PATH -x PATH -mca pml ob1 -mca btl ^openib python train.py To run on 4 machines with 4 GPUs each: $ mpirun -np 16 -H server1:4,server2:4,server3:4,server4:4 -bind-to none -map-by slot -x NCCL_DEBUG=INFO -x LD_LIBRARY_PATH -x PATH -mca pml ob1 -mca btl ^openib python train.py
- 14. Which GPU to Get for Deep Learning http://timdettmers.com/2018/11/05/which-gpu-for-deep-learning/