Scaling Deep Learning with MXNet

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Mu Li, Alex Smola and Joseph Spisak
AWS Machine Learning
Scaling Deep Learning with MXNet
Wifi: IMPACT DEEP / deep2017

Modules
1. Introduction to Deep Learning
2. Getting Started with MXNet
• Cloud - AMIs and CloudFormation Template
• Laptop - build from source
3. MXNet
• NDArrays and Symbols
• Training Deep Networks
4. Examples
• Computer Vision
• Natural Language Processing
• Recommender Systems

1. Introduction to Deep Learning
• Machine Learning
• Shallow and Deep Learning
• Models (fully connected, convolutions, sequences)
• Training
image credit - wikipedia

Programming FizzBuzz
Code
var o='';
for (var i=1; i<=100; i++) {
i%3 || (o+='Fizz ');
i%5 || (o+='Buzz ');
!(i%3 && i%5) || (o+=(i+' '));
}
console.log(o);
Programming with Data
• Generate training data  
(9,’Fizz’),(10,’Buzz’),(2,’2’),
(15,’Fizz Buzz’) …
• Extract input features 
x -> (x%3, x%5, x%15)
• Train a classifier  
mapping input x to output y
using training data
Inspired by joelgrus.com

That was silly.
Why would you ever do this?

Programming with Data
• Given some input x we want to estimate output y
• If it’s a simple deterministic rule, probably don’t bother
• Often quite complicated, vague but lots of examples
• Impossible to write code but can train estimator
cat dog dogcat

Edible Poisonous
Truffle 0 10
Death cap 1,000,000 0
Machine Learning 101
• Training data (patterns, labels)
• Images (pictures / cat, dog, dinosaur, chicken, human)
• Text (email / spam, ham)
• Sound (recording / recognized speech)
• Video (sequence / break-in, OK)
• Images (X-ray image / size of cancer)
• Loss function
• Estimator  
Learns mapping data into outputs

• Training Error 
Error on the dataset that estimator uses to find f
• Test Error 
Error on a new, unseen test set. This can be way off.  
Due to overfitting. E.g. see predictions for election.
• Training 
Find parameters for f, e.g. by loss minimization
• Model Selection 
Adjust tuning parameters, network layers, trees, kernels …

Real Neural Networks
Soma (CPU) 
Cell body - combines signals
Dendrite (input bus) 
Combines inputs from other neurons
Synapse (interface) 
Interface and parameter store between neurons
Axon (cable) 
Up to 1m long. Send activation signal other neurons

Artificial Neuron x1 x2 x3 xn. . .
output
w1 wn
synaptic
weights
• Input 
Vector of training data x
• Output 
Linear function of inputs
• Nonlinearity 
Transform output into desired range
of values, e.g. for classification we
need probabilities [0, 1]
• Training 
Learn the weights w and bias b f(x) = (hw, xi + b)

Shallow Learning
• Extract clever features 
(preprocessing)
• Map into feature space 
(kernel methods)
• Set of rules 
(decision tree)
• Combine multiple estimates 
(boosting)
Deep Learning
• Many simple neurons
• Specialized layers  
(images, text, audio, …)
• Stack layers 
(hence deep learning)
• Optimization is difficult
• Backpropagation
• Stochastic gradient descent
usually simple to learn better accuracy

Multilayer Perceptron 
and Optimization

Perceptron
x1 x2 x3 xn. . .
output
w1 wn
synaptic
weights
y(x) = (hw, xi)

Multilayer Perceptron
y1i(x) = (hw1i, xi)
y2(x) = (hw2, y1i)

y1i(x) = (hw1i, xi)
y2i(x) = (hw2i, y1i)
y3(x) = (hw3, y2i)

Layer Representation
Alternate between linear function  
Wx and nonlinear function σ(x)
Loss function 
to measure quality of estimate
yi = Wixi
xi+1 = (yi)
x1
x2
x3
x4
y
W1
W2
W3
W4
l(y, yi)

Multilayer Perceptron Training
Layer Representation 
 
Gradient descent 
 
• Stochastic gradient descent 
(use only one sample)
• Minibatch (small subset)
Wi Wi ⌘@Wi
l(y, yn)
x1
x2
x3
x4
y
W1
W2
W3
W4
yi = Wixi
xi+1 = (yi)

Multilayer Perceptron Training
Layer Representation
Change in Objective
Chain Rule
yi = Wixi
xi+1 = (yi)
gj = @Wj
l(y, yi)
@x [f2 f1] (x) = [@f1
f2 f1] (x) [@xf1] (x)
x1
x2
x3
x4
y
W1
W2
W3
W4
effect of input
second layer

Backpropagation
Layers & Gradients 
 
The chain rule (aka BackProp)
@xi
yi = Wi
@Wi
yi = xi
@yi
xi+1 = 0
(yi)
=) @xi
xi+1 = 0
(yi)W>
i
@xi
l(y, yn) = @xi+1
l(y, yn)@xi
xi+1
= @xi+1 l(y, yn) 0
(yi)W>
i
@Wi l(y, yn) = @yi l(y, yn)@Wi yi
= @xi+1 l(y, yn) 0
(yi)x>
i
yi = Wixi
xi+1 = (yi)
matrix-vector multiply

Rectified Linear Unit (ReLu)
Forward mapping 
 
 
 
Nonlinearity between layers
• Gradients vanish at tails for sigmoid
• Replace by max(0,x) 
Derivative is in {0; 1}, sparse signal
x2
x3
W2
yi = Wixi
xi+1 = (yi)

Example - Word Vectors
• Full vocabulary (e.g. 30k words)
• One-hot encoding for words  
(0,0,0,0,1,0,0,0) (0,0,0,0,0,0,1,0) 
 
• Map words into vector space via y = Wx 
 
 
• Postprocess as normal
cat
dinosaur
catdog
laptop
dog

That was complicated!
MXNet takes care of this for you …
fc = mx.symbol.FullyConnected(data=input, num_hidden=4096)
relu = mx.symbol.Activation(data=fc, act_type="relu")
dropout = mx.symbol.Dropout(data=relu, p=0.5)

Where is Waldo?

Where is Waldo?
No need to learn a different Waldo
model for every location.

Convolutional Layers
• Images have shift invariance
• Low level mostly edge and  
feature detectors
• Key Idea 
Replace matrix multiplication 
with local computation
• Backprop automatic in MXNet
image credit - Le Cun et al, 1998, Olshausen and Field, 1998
yij =
X
a=
X
b=
Wabxi+a,j+b

Subsampling & MaxPooling
• Multiple convolutions blow up dimensionality 
 
 
 
 
 
• Subsampling - average over patches (works OK)
• MaxPooling - pick the maximum over patches (much better)
image credit - Le Cun et al, 1998
LeNet ~1995

LeNet in R (using MXNet)
get_symbol <- function(num_classes = 1000) {
data <- mx.symbol.Variable('data')
# first conv
conv1 <- mx.symbol.Convolution(data = data, kernel = c(5, 5), num_filter = 20)
tanh1 <- mx.symbol.Activation(data = conv1, act_type = "tanh")
pool1 <- mx.symbol.Pooling(data = tanh1, pool_type = "max", kernel = c(2, 2), stride = c(2, 2))
# second conv
conv2 <- mx.symbol.Convolution(data = pool1, kernel = c(5, 5), num_filter = 50)
tanh2 <- mx.symbol.Activation(data = conv2, act_type = "tanh")
pool2 <- mx.symbol.Pooling(data = tanh2, pool_type = "max", kernel = c(2, 2), stride = c(2, 2))
# first fullc
flatten <- mx.symbol.Flatten(data = pool2)
fc1 <- mx.symbol.FullyConnected(data = flatten, num_hidden = 500)
tanh3 <- mx.symbol.Activation(data = fc1, act_type = "tanh")
# second fullc
fc2 <- mx.symbol.FullyConnected(data = tanh3, num_hidden = num_classes)
# loss
lenet <- mx.symbol.SoftmaxOutput(data = fc2, name = 'softmax')
return(lenet)
}

Fancy structures
• Compute different filters
• Compose one big vector from all of them
• Layer them iteratively
Szegedy et al. arxiv.org/pdf/1409.4842v1.pdf

Inception v3 in Python (using MXNet)
def get_symbol(num_classes=1000):
# data
data = mx.symbol.Variable(name="data")
# stage 1
conv1 = ConvFactory(data=data, num_filter=64, kernel=(7, 7), stride=(2, 2), pad=(3, 3), name='1')
pool1 = mx.symbol.Pooling(data=conv1, kernel=(3, 3), stride=(2, 2), name='pool_1', pool_type='max')
# stage 2
conv2red = ConvFactory(data=pool1, num_filter=64, kernel=(1, 1), stride=(1, 1), name='2_red')
conv2 = ConvFactory(data=conv2red, num_filter=192, kernel=(3, 3), stride=(1, 1), pad=(1, 1), name='2')
pool2 = mx.symbol.Pooling(data=conv2, kernel=(3, 3), stride=(2, 2), name='pool_2', pool_type='max')
# stage 2
in3a = InceptionFactoryA(pool2, 64, 64, 64, 64, 96, "avg", 32, '3a')
in3b = InceptionFactoryA(in3a, 64, 64, 96, 64, 96, "avg", 64, '3b')
in3c = InceptionFactoryB(in3b, 128, 160, 64, 96, '3c')
# stage 3
in4a = InceptionFactoryA(in3c, 224, 64, 96, 96, 128, "avg", 128, '4a')
in4b = InceptionFactoryA(in4a, 192, 96, 128, 96, 128, "avg", 128, '4b')
in4c = InceptionFactoryA(in4b, 160, 128, 160, 128, 160, "avg", 128, '4c')
in4d = InceptionFactoryA(in4c, 96, 128, 192, 160, 192, "avg", 128, '4d')
in4e = InceptionFactoryB(in4d, 128, 192, 192, 256, '4e')
# stage 4
in5a = InceptionFactoryA(in4e, 352, 192, 320, 160, 224, "avg", 128, '5a')
in5b = InceptionFactoryA(in5a, 352, 192, 320, 192, 224, "max", 128, '5b')
# global avg pooling
avg = mx.symbol.Pooling(data=in5b, kernel=(7, 7), stride=(1, 1), name="global_pool", pool_type='avg')
# linear classifier
flatten = mx.symbol.Flatten(data=avg, name='flatten')
fc1 = mx.symbol.FullyConnected(data=flatten, num_hidden=num_classes, name='fc1')
softmax = mx.symbol.SoftmaxOutput(data=fc1, name='softmax')
return softmax

Sequence Models
Schmidhuber & Hochreiter, 1998

Sequence Models
• Sequence of observations 
Purchases, likes, app use, e-mails, queries, ratings, words, dialogue
• Hidden state to explain behavior
• Navigational, informational queries in search
• Interest distributions for users over time
• Trajectory and location modeling
• Content and meaning in text
Action
Explanation

Latent Variable Models
• Temporal sequence of observations 
Purchases, likes, app use, e-mails, queries, ratings, words, dialogue
• Latent state to explain behavior
• Plain deep network = RNN
• Deep network with attention = LSTM / GRU … 
(learn when to update state, how to read out)
x
h

Long Short Term Memory
x
h
Schmidhuber and Hochreiter, 1998
it = (Wi(xt, ht) + bi)
ft = (Wf (xt, ht) + bf )
zt+1 = ft · zt + it · tanh(Wz(xt, ht) + bz)
ot = (Wo(xt, ht, zt+1) + bo)
ht+1 = ot · tanh zt+1
input
forgetting
output
state

Long Short Term Memory
x
h
(zt+1, ht+1, ot) = LSTM(zt, ht, xt)
Treat it as a black box

LSTM (Long Short Term Memory)
• Group LSTM cells into layers
• Can model different scales of dynamics

Natural Language Translation
Le, Sutskever and Vinyals, 2014

Summary
Perceptron Multilayer Perceptron Backpropagation
Convolutional Nets LSTM MXNet
x1
x2
x3
x4
y
W
W
W
W4
@xi
l(y, yn) = @xi+1
l(y, yn)@xi
xi+1
= @xi+1 l(y, yn) 0
(yi)W>
i
@Wi l(y, yn) = @yi l(y, yn)@Wi yi
= @xi+1 l(y, yn) 0
(yi)x>
i

2. Running MXNet
• Amazon Machine Images (AMI)
• Cloud Formation Templates
• Local install & resources

Amazon Machine Image for
Deep Learning
http://bit.ly/deepami
Deep Learning any way you want on AWS

Getting started with Deep Learning
• For data scientists and developers
• Setting up a DL system takes (install) time & skill
• Keep packages up to date and compile
• Install all dependencies (licenses, compilers, drivers, etc.)
• NVIDIA Drivers for G2 and P2 servers
• Intel MKL linked for everything else (C5 coming soon)
http://bit.ly/deepami

Getting started with Deep Learning
• Drivers 
CUDA / CUDNN / CUFFT / CUSPARSE
• Development tools 
Python 2 and 3, Anaconda, Jupyter notebooks, Graphviz
• Deep Learning Platforms (compiled & tested)
• MXNet, Tensorflow, CNTK 
multi-GPU, multi-machine (MXNet recommended)
• Caffe, Theano, Torch, Keras
Always up to date (less than 1 month), optimized & tested on AWS

Getting started
acbc32cf4de3:image-classification smola$ ssh ec2-user@54.210.246.140
Last login: Fri Nov 11 05:58:58 2016 from 72-21-196-69.amazon.com
=============================================================================
__| __|_ )
_| ( / Deep Learning AMI for Amazon Linux
___|___|___|
This is beta version of the Deep Learning AMI for Amazon Linux.
The README file for the AMI ➜➜➜➜➜➜➜➜➜➜➜➜➜➜➜➜➜➜➜➜ /home/ec2-user/src/README.md
Tests for deep learning frameworks ➜➜➜➜➜➜➜➜➜➜➜➜ /home/ec2-user/src/bin
=============================================================================
7 package(s) needed for security, out of 75 available
Run "sudo yum update" to apply all updates.
Amazon Linux version 2016.09 is available.
[ec2-user@ip-172-31-55-21 ~]$ cd src/
[ec2-user@ip-172-31-55-21 src]$ ls
anaconda2 bazel caffe cntk keras mxnet OpenBLAS README.md Theano
anaconda3 bin caffe3 demos logs Nvidia_Cloud_EULA.pdf opencv tensorflow torch

AWS CloudFormation Template
for Deep Learning
http://bit.ly/deepcfn

AWS CloudFormation Templates
• Hardware as Code
• Define compute resources
• Define network
• Define OS installation and configuration
• Deep Learning
• Many toolkits barely deal with multiple GPUs (DGX-1)
• BigData needs massive compute resources
• Hardware availability and software support

Scaling Deep Learning with MXNet

AWS CloudFormation Components
• VPC in the customer account.
• The requested number of worker instances in an Auto Scaling
group within the VPC. Workers are launched in a private subnet.
• Master instance in a separate Auto Scaling group that acts as a
proxy to enable connectivity to the cluster via SSH.
• Two security groups that open ports on the private subnet for
communication between the master and workers.
• IAM role that allows users to access and query Auto Scaling
groups and the private IP addresses of the EC2 instances.
• NAT gateway used by instances within the VPC to talk to the
outside.

Launching an MXNet job
MNIST digit classification demo 
../../tools/launch.py -n  
$DEEPLEARNING_WORKERS_COUNT -H  
$DEEPLEARNING_WORKERS_PATH  
python train_mnist.py  
--gpus $(seq -s , 0 1 $ 
(($DEEPLEARNING_WORKER_GPU_COUNT - 1)))  
--network lenet --kv-store dist_sync
parameter
server
network
choice
update
policy

Installation Pointers
http://mxnet.io/get_started/setup.html
(many packages - Python, graphviz, OpenCV, OpenMP, CUDA)
• MXNet with Docker
• Installing MXNet on the Cloud (AWS AMI)
• Installing MXNet on Ubuntu
• Installing MXNet on Amazon Linux
• Installing MXNet on OS X (Mac)
• Installing MXNet on Windows

Ubuntu & Python
# download and install CUDA 8 and CUDNN 8
git clone https://github.com/dmlc/mxnet.git ~/mxnet --recursive
# If building with GPU, add configurations to config.mk file:
cd ~/mxnet
cp make/config.mk .
echo "USE_CUDA=1" >> config.mk
echo "USE_CUDA_PATH=/usr/local/cuda" >> config.mk
echo "USE_CUDNN=1" >> config.mk
# Install MXNet for Python with all required dependencies
cd ~/mxnet/setup-utils
bash install-mxnet-ubuntu-python.sh
# We have added MXNet Python package path in your ~/.bashrc.
# Run the following command to refresh environment variables.
source ~/.bashrc

Mac and Scala
• Install prerequisites 
Graphviz, Python, OpenCV, Jupyter
• Build general MXNet package, then
make scalapkg
make scalainstall
… enjoy …

Windows and R
• Prebuilt binary libmxnet.dll for download (CUDA / Intel) 
https://github.com/dmlc/mxnet/releases
• Build from source in VisualStudio 
(needs lots dependences - OpenCV, GraphViz, …)
• Install prebuilt binary for R 
install.packages("drat", repos="https://cran.rstudio.com") 
drat:::addRepo("dmlc")  
install.packages(“mxnet”)
• Install from source …

Any language. Anywhere. Native Performance
Linux
Windows
OS X
Docker
NVIDIA
Intel
Python
R
Scala
Julia
Android, iOS, IoT … ARM … JavaScript Torch, Caffe …

3. MXNet
• Imperative and Declarative Programming
• Language Support
• Backend and Automatic Parallelization

Caffe
Torch
Theano
Tensorflow
CNTK
Keras
Paddle
Chainer
SINGA
DL4J
Why yet another deep networks tool?
image credit - Banksy/wikipedia

Why yet another deep networks tool?
• Frugality & resource efficiency 
Engineered for cheap GPUs with smaller memory, slow networks
• Speed
• Linear scaling with #machines and #GPUs
• High efficiency on single machine, too (C++ backend)
• Simplicity 
Mix declarative and imperative code
single implementation of
backend system and
common operators
performance guarantee
regardless which frontend
language is used
frontend
backend

MXNet Highlights
Mixed Programming API
Auto Parallel Scheduling
Distributed Computing
Memory Optimization
Runs Everywhere
Multi-Language Support
Flexibility Eﬃciency Portability
Front-
end
Back-
end
{
{

Imperative Programs
import numpy as np
a = np.ones(10)
b = np.ones(10) * 2
c = b * a
print c
d = c + 1 Easy to tweak
with python
codes
Pro
• Straightforward and flexible.
• Take advantage of language native
features (loop, condition, debugger)
Con
• Hard to optimize

Declarative Programs
A = Variable('A')
B = Variable('B')
C = B * A
D = C + 1
f = compile(D)
d = f(A=np.ones(10),
B=np.ones(10)*2)
Pro
• More chances for optimization
• Cross different languages
Con
• Less flexible
A B
1
+
⨉
C can share memory with D,
because C is deleted later

Imperative vs. Declarative for Deep Learning
Computational Graph
of the Deep Architecture
forward backward
Needs heavy optimization,
fits declarative programs
Needs mutation and more
language native features, good for
imperative programs
Updates and Interactions
with the graph
• Iteration loops
• Parameter update 
• Beam search
• Feature extraction …
w w ⌘@wf(w)

MXNet: Mix the Flavors Together
Imperative
NDArray API
Declarative
Symbolic Executor
>>> import mxnet as mx
>>> a = mx.nd.zeros((100, 50))
>>> b = mx.nd.ones((100, 50))
>>> c = a + b
>>> c += 1
>>> print(c)
>>> import mxnet as mx
>>> net = mx.symbol.Variable('data')
>>> net = mx.symbol.FullyConnected(data=net, num_hidden=128)
>>> net = mx.symbol.SoftmaxOutput(data=net)
>>> texec = mx.module.Module(net)
>>> texec.forward(data=c)
>>> texec.backward() Imperative NDArray can be
set as input to the graph

Mixed API for Quick Extensions
• Runtime switching between different graphs depending on input
• Useful for sequence modeling and image size reshaping
• Use of imperative code in Python, 10 lines of additional Python code
BucketingVariable length sentences

3D Image Construction
Deep3D
100 lines of Python code
https://github.com/piiswrong/deep3d

What We Heard from Users
• Python is nice, but I like R/Julia/
Matlab more
• I want Scala to work with the
Spark pipeline
• I need C++ interface to run on
embedded systems
• I prefer Javascript to run on user
browsers
Programming Languages: Frameworks:
• I used Torch for 7 years
• All my codes are in Caffe
• I started deep learning with
Tensorflow
• I only used Numpy before,
how should I start?

single implementation
of backend system and
common operators
performance guarantee
regardless of which
frontend language is used
frontend
backend
Multiple Programming Languages

Bringing Caffe to MXNet
import mxnet as mx
data = mx.symbol.Variable('data')
fc1 = mx.symbol.CaffeOp(data_0=data, num_weight=2, prototxt= 
"layer{type:"InnerProduct" inner_product_param{num_output: 128} }")
act1 = mx.symbol.CaffeOp(data_0=fc1, prototxt="layer{type:"TanH"}")
fc2 = mx.symbol.CaffeOp(data_0=act1, num_weight=2, prototxt= 
"layer{type:"InnerProduct" inner_product_param{num_output: 10}}")
mlp = mx.symbol.SoftmaxOutput(data=fc3)
Caffe is widely used in computer vision 
Call Caffe Operators in MXNet

Bringing Torch to MXNet
Tensor Computation
Modules (Layers)
Torch is a popular Lua framework for
both scientiﬁc computing and deep learning
import mxnet as mx
x = mx.th.randn(2, 2, ctx=mx.gpu(0))
y = mx.th.abs(x)
print y.asnumpy()
import mxnet as mx
data = mx.symbol.Variable(‘data’)
fc = mx.symbol.TorchModule(data_0=data, lua_string=‘nn.Linear(784, 128)’,…
mlp = mx.symbol.TorchModule(data_0=fc, lua_string=‘nn.LogSoftMax()’,…

MXNet Highlights
Mixed Programming API
Auto Parallel Scheduling
Distributed Computing
Memory Optimization
Runs Everywhere
Multi-Language Supports
Flexibility Eﬃciency Portability
Front-
end
Back-
end
{
{

Deploy Everywhere
Beyond
BlindTool by Joseph Paul Cohen, demo on Nexus 4
• Fit the core library with
all dependencies into a
single C++ source ﬁle
• Easy to compile on
…
Amalgamation
Runs in browser  
with Javascript
The ﬁrst image for
search “dog” at
images.google.com
Outputs “beagle”
with prob = 73%
within 1 sec

TX1 on Flying Drone
TX1 with customized board
Drone
Realtime detection and tracking on TX1
~10 frame/sec with 640x480 resolution

Backend
a b
2
⨉
+
c
fullc
softmax
weight
bias
Back-end
import mxnet as mx
a = mx.nd.ones((100, 50))
b = mx.nd.ones((100, 50))
c = a + b
c *= 2
import mxnet as mx
net = mx.symbol.Variable('data')
net = mx.symbol.FullyConnected(
data=net, num_hidden=128)
net = mx.symbol.SoftmaxOutput(data=net)
texec = mx.module.Module(net)
texec.forward(data=c)
texec.backward()
Front-end
NNVM
A lot of optimizations
happen here
∂fullc
∂softmax

Traverse the computation graph to reduce the memory footprint  
with linear time complexity
Memory Optimization
a
b c
dnow a is
deletable
aliveness analysis
b ca
shared space between
variables
share a and b

Results for Deep CNNs onmemory(GB)
0
2.25
4.5
6.75
9
alexnet inception vgg
baseline mxnet
Training
2.6x
1.8x
memory(GB)
0
2.25
4.5
6.75
9
alexnet inception vgg
baseline mxnet
Prediction
4.4x
4x
3.2x1.8x

Neural Art
1M pixels
GTX 980 TI 6G
(in 20x speed)

Trading Computation for Memory
ResNet
• 1000 layers
• batch size 32
LSTM
• 4 layers
• 1000 hidden size
• 1000 unroll
• batch size 32
Before After
Resnet 130 GB 4 GB
LSTM 270 GB 2.5 GB

Writing Parallel Programs is Painful
Each forward-backward-update
involves O(num_layer), which is often
100—1,000, tensor computations and
communications
data = next_batch()data[gpu0].copyfrom(data[0:50])
_, fc1_wgrad[gpu0] =
FullcBackward(fc1_ograd[gpu0] ,
fc1_ograd[gpu0], fc2_wgrad[gpu0] =
fc2_weight[gpu0])
fc2_ograd[gpu0] = LossGrad(fc2[gpu0],
label[0:50])
fc2[gpu0] = FullcForward(fc1[gpu0],
fc2_weight[gpu0])
fc1[gpu0] = FullcForward(data[gpu0],
fc2_wgrad[cpu] =
fc2_wgrad[gpu0] + fc2_wgrad[gpu1]
fc2_weight[cpu].copyto(
fc2_weight[gpu0] , fc2_weight[gpu1])
fc2_weight[cpu] -= lr*fc12_wgrad[gpu0]
fc1_weight[cpu] -= lr * fc1_wgrad[gpu0]
fc1_wgrad[cpu] =
fc1_wgrad[gpu0] + fc1_wgrad[gpu1]
fc1_weight[cpu].copyto(
data[gpu0].copyfrom(data[51:100])
_, fc1_wgrad[gpu1] =
fc1_ograd[gpu1], fc2_wgrad[gpu1] =
fc2_ograd[gpu1] = LossGrad(fc2[gpu1],
fc2[gpu1] = FullcForward(fc1[gpu1],
fc2_weight[gpu1])
fc1[gpu1] = FullcForward(data[gpu1],
fc1_weight[gpu1])
Dependency graph for 2-layer neural
networks with 2 GPUs

Auto Parallelization
Write serial programs Run in parallel
import mxnet as mx
A = mx.nd.ones((2,2)) *2
C = A + 2
B = A + 1
D = B * C
A = 2
C = A + 2 B = A + 1
D = B ⨉ C

Data Parallelism
key-value store
examples
1. Read a data partition
2. Pull the parameters
3. Compute the gradient
4. Push the gradient
5. Update the parameters

Scalability on Multiple GPUs
AWS EC2 P2.16xlarge
8 Nvidia Tesla K80
(16 GPUs)
Synchronized SGD
Speedup(x)
1
4.75
8.5
12.25
16
# GPUs
1 4.75 8.5 12.25 16
Resnet 152
Inception V3
Alexnet
Ideal

Implementation
PCIe Switch
GPU
GPU
GPU
GPU
CPU
Network Switch
63 GB/s
4 PCIe 3.0 16x
15.75 GB/s
PCIe 3.0 16x
1.25 GB/s
10 Gbit Ethernet
Level-1 Servers
Workers
Level-2 Servers
✦ Hierarchical parameter servers

Distributed Experiments
• Google Inception v3
• Increasing machines
from 1 to 47
• 2x faster than
TensorFlow if using
more than 10
machines

Distributed Training Speedup
• Cloud formation with
Deep Learning AMI
• 16x P2.16xlarge
• Mounted on EFS
• ImageNet  
1.2M images  
1K classes
• 152-layer ResNet 
5.4d on 4x K80s
(1.2h per epoch) 
0.22 top-1 error

Distributed Training Convergence

Fine tuning for distributed inference

4. Examples
• NDArray, Symbols
• Parallel Training
• Computer Vision
• Natural Language Processing
• Recommender Systems

Python notebooks
https://github.com/dmlc/mxnet-notebooks
Basic concepts
• NDArray - multi-dimensional array computation
• Symbol - symbolic expression for neural networks
• Module - neural network training and inference
Applications
• MNIST: recognize handwritten digits
• Check distributed training results
• Predict with pre-trained models
• LSTMs for sequence learning
• Recommender systems

Notebook - MNIST and LeNet
Le Cun et al, 1998
fc1 = mx.symbol.FullyConnected(data = data, name='fc1', num_hidden=128)
act1 = mx.symbol.Activation(data = fc1, name='relu1', act_type="relu")
fc2 = mx.symbol.FullyConnected(data = act1, name = 'fc2', num_hidden = 64)
act2 = mx.symbol.Activation(data = fc2, name='relu2', act_type="relu")
fc3 = mx.symbol.FullyConnected(data = act2, name='fc3', num_hidden=10)
mlp = mx.symbol.SoftmaxOutput(data = fc3, name = 'softmax')
Notebook
https://github.com/dmlc/mxnet/tree/master/example/image-classification

Training a machine learning model

Training
Data
All
Labelled
Data
75% 25%
.

Notebook - Long Short Term Memory
x
h
it = (Wi(xt, ht) + bi)
ft = (Wf (xt, ht) + bf )
zt+1 = ft · zt + it · tanh(Wz(xt, ht) + bz)
ot = (Wo(xt, ht, zt+1) + bo)
ht+1 = ot · tanh zt+1
input
forgetting
output
state

Recommender Systems
Users u, movies m (or projects)
Function class
Loss function for recommendation (Yelp, Netflix) 
 
rum = hvu, wmi + bu + bm
X
u⇠m
(hvu, wmi + bu + bm yum)
2

User-Item Ratings Matrix
very sparse
99.5% unknown

Recommender Systems
Regularized Objective 
 
Update operations
Very simple SGD algorithm (random pairs)
This should be cheap …
X
u⇠m
(hvu, wmi + bu + bm + b0 rum)
2
+
2
h
kUk
2
Frob + kV k
2
Frob
i
vu (1 ⌘t )vu ⌘twm (hvu, wmi + bu + bm + b0 rum)
wm (1 ⌘t )wm ⌘tvu (hvu, wmi + bu + bm + b0 rum)

Matrix Factorization as Deep Network

Adding Features
• One-hot encoding for classical  
recommenders
• Add features to one-hot vector 
 
 
 
 
• Images - via a Convolutional Network (e.g. ResNet)
• Text - LSTM or (much simpler) via Bag of Words

Recommender systems, not recommender archaeology
users
items
time
NOW
predict that
(future)
use this
(past)
don’t predict this
(archaeology)
Wu et al., 2017 WSDM

The Netﬂix contest
got it wrong …

Getting it right
change in
taste and
expertise
change in
perception
and novelty
LSTM
LSTM
Wu et al, WSDM’17

Scaling Deep Learning with MXNet

More Related Content

What's hot (17)

Similar to Scaling Deep Learning with MXNet (20)

More from AI Frontiers (20)

Recently uploaded (20)

Scaling Deep Learning with MXNet