TensorFlow for IITians
Download as PPTX, PDF1 like187 views
This document provides an overview of TensorFlow presented by Ashish Agarwal and Ashish Bansal. The key points covered include: - TensorFlow is an open-source machine learning framework for research and production. It allows models to be deployed across different platforms. - TensorFlow models are represented as dataflow graphs where nodes are operations and edges are tensors flowing between operations. Placeholders, variables, and sessions are introduced. - Examples demonstrate basic linear regression and logistic regression models built with TensorFlow. Layers API and common neural network components like convolutions and RNNs are also covered. - Advanced models like AlexNet, Inception, ResNet, and neural machine translation with attention are briefly overviewed.
![import tensorflow as tf
# Input (x)
x = tf.placeholder(tf.float32)
# Model parameters
W = tf.Variable([.3], dtype=tf.float32)
b = tf.Variable([-.3], dtype=tf.float32)
# Prediction
prediction = W * x + b
Example: Linear Regression](https://image.slidesharecdn.com/tensorflowforiitians-190809065743/85/TensorFlow-for-IITians-15-320.jpg)
![import tensorflow as tf
# Input (x)
x = tf.placeholder(tf.float32)
# Model parameters
W = tf.Variable([.3], dtype=tf.float32)
b = tf.Variable([-.3], dtype=tf.float32)
# Prediction
prediction = W * x + b
# Label(y) and loss
y = tf.placeholder(tf.float32)
loss = tf.reduce_sum(tf.square(prediction - y))
Example: Linear Regression](https://image.slidesharecdn.com/tensorflowforiitians-190809065743/85/TensorFlow-for-IITians-16-320.jpg)
![import tensorflow as tf
# Input (x)
x = tf.placeholder(tf.float32)
# Model parameters
W = tf.Variable([.3], dtype=tf.float32)
b = tf.Variable([-.3], dtype=tf.float32)
# Prediction
prediction = W * x + b
# Label(y) and loss
y = tf.placeholder(tf.float32)
loss = tf.reduce_sum(tf.square(prediction - y))
Example: Linear Regression](https://image.slidesharecdn.com/tensorflowforiitians-190809065743/85/TensorFlow-for-IITians-17-320.jpg)
![import tensorflow as tf
# Input (x)
x = tf.placeholder(tf.float32)
# Model parameters
W = tf.Variable([.3], dtype=tf.float32)
b = tf.Variable([-.3], dtype=tf.float32)
# Prediction
prediction = W * x + b
# Label(y) and loss
y = tf.placeholder(tf.float32)
loss = tf.reduce_sum(tf.square(prediction - y))
Example: Linear Regression
# Initialization, gradients and SGD nodes
initializer = tf.global_variables_initializer()
optimizer = tf.train.GradientDescentOptimizer(0.01)
training_step = optimizer.minimize(loss)](https://image.slidesharecdn.com/tensorflowforiitians-190809065743/85/TensorFlow-for-IITians-18-320.jpg)
![import tensorflow as tf
# Input (x)
x = tf.placeholder(tf.float32)
# Model parameters
W = tf.Variable([.3], dtype=tf.float32)
b = tf.Variable([-.3], dtype=tf.float32)
# Prediction
prediction = W * x + b
# Label(y) and loss
y = tf.placeholder(tf.float32)
loss = tf.reduce_sum(tf.square(prediction - y))
Example: Linear Regression
# Initialization, gradients and SGD nodes
initializer = tf.global_variables_initializer()
optimizer = tf.train.GradientDescentOptimizer(0.01)
training_step = optimizer.minimize(loss)
# Training data
x_train = ...
y_train = ...
# Runtime setup
sess = tf.Session()](https://image.slidesharecdn.com/tensorflowforiitians-190809065743/85/TensorFlow-for-IITians-19-320.jpg)
![import tensorflow as tf
# Input (x)
x = tf.placeholder(tf.float32)
# Model parameters
W = tf.Variable([.3], dtype=tf.float32)
b = tf.Variable([-.3], dtype=tf.float32)
# Prediction
prediction = W * x + b
# Label(y) and loss
y = tf.placeholder(tf.float32)
loss = tf.reduce_sum(tf.square(prediction - y))
Example: Linear Regression
# Initialization, gradients and SGD nodes
initializer = tf.global_variables_initializer()
optimizer = tf.train.GradientDescentOptimizer(0.01)
training_step = optimizer.minimize(loss)
# Training data
x_train = ...
y_train = ...
# Runtime setup
sess = tf.Session()
# Initialization and training loop
sess.run(initializer)
for i in range(1000):
sess.run(training_step,
feed_dict={x: x_train, y: y_train})](https://image.slidesharecdn.com/tensorflowforiitians-190809065743/85/TensorFlow-for-IITians-20-320.jpg)
![import tensorflow as tf
# Input (x)
x = tf.placeholder(tf.float32)
# Model parameters
W = tf.Variable([.3], dtype=tf.float32)
b = tf.Variable([-.3], dtype=tf.float32)
# Prediction
prediction = W * x + b
# Label(y) and loss
y = tf.placeholder(tf.float32, shape=[4, 1])
loss = tf.reduce_sum(tf.square(prediction - y))
Example: Linear Logistic Regression
loss = tf.reduce_sum(
tf.nn.sigmoid_cross_entropy_with_logits(
labels=y,
logits=logits))
logits = W * x + b
prediction = tf.sigmoid(logits)](https://image.slidesharecdn.com/tensorflowforiitians-190809065743/85/TensorFlow-for-IITians-21-320.jpg)
![import tensorflow as tf
# Input (x)
x = tf.placeholder(tf.float32)
# Model parameters
W = tf.Variable([.3], dtype=tf.float32)
b = tf.Variable([-.3], dtype=tf.float32)
# Prediction
prediction = W * x + b
# Label(y) and loss
y = tf.placeholder(tf.float32, shape=[4, 1])
loss = tf.reduce_sum(tf.square(prediction - y))
Example: Linear Regression using tf.layers
prediction = tf.layers.dense(x, units=1,
use_bias=True, activation=None)](https://image.slidesharecdn.com/tensorflowforiitians-190809065743/85/TensorFlow-for-IITians-22-320.jpg)
![import tensorflow as tf
# Input (x)
x = tf.placeholder(tf.float32, shape=[4, 1])
# Prediction
prediction = tf.layers.dense(x, units=1,
use_bias=True, activation=None)
# Label(y) and loss
y = tf.placeholder(tf.float32, shape=[4, 1])
loss = tf.reduce_sum(tf.square(prediction - y))
Example: Linear Regression
l1 = tf.layers.dense(x, units=100,
use_bias=True, activation=relu)
l2 = tf.layers.dense(l1, …)
...
prediction = tf.layers.dense(ln, …)
Non-
^](https://image.slidesharecdn.com/tensorflowforiitians-190809065743/85/TensorFlow-for-IITians-23-320.jpg)
![Putting it together: AlexNet
# Convolution and pooling layers.
conv1 = tf.layers.conv2d(input,
filters=64, kernel_size=[11, 11], stride=4)
pool1 = tf.layers.max_pooling2d(conv1,
kernel_size=[3, 3], stride=2)
conv2 = tf.layers.conv2d(pool1,
filters=192, kernel_size=[5, 5])
pool2 = tf.layers.max_pooling2d(conv2,
kernel_size=[3, 3], stride=2)
conv3 = tf.layers.conv2d(pool2,
filters=384, kernel_size=[3, 3])
conv4 = tf.layers.conv2d(conv3,
filters=384, kernel_size=[3, 3])
conv5 = tf.layers.conv2d(conv4,
filters=256, kernel_size=[3, 3])
pool5 = tf.layers.max_pooling2d(conv5,
kernel_size=[3, 3], stride=2)
reshaped_pool5 = tf.reshape(pool5, [-1, 5 * 5 * 256])
# Fully connected layers with dropout.
fc6 = tf.layers.dense(reshaped_pool5, units=4096)
drp6 = tf.layers.dropout(fc6, keep_prob=0.5)
fc7 = tf.layers.dense(drp6, units=4096)
drp7 = tf.layers.dropout(fc7, keep_prob=0.5)
fc8 = tf.layers.dense(drp7, units=1000,
activation_fn=None)
# Calculating the loss.
loss = tf.nn.softmax_cross_entropy_with_logits(
labels=labels, logits=fc8)](https://image.slidesharecdn.com/tensorflowforiitians-190809065743/85/TensorFlow-for-IITians-29-320.jpg)
![Simple RNN
def rnn(cell, input_list, initial_state):
state = initial_state
outputs = []
for inp in input_list:
output, state = cell(inp, state)
outputs.append(output)
return outputs, state](https://image.slidesharecdn.com/tensorflowforiitians-190809065743/85/TensorFlow-for-IITians-40-320.jpg)
![Dynamic RNN
# 8 layer LSTM with residual connections
# Each layer is on a separate GPU
cell = MultiRNNCell(
[DeviceWrapper(ResidualWrapper(LSTMCell(num_units=512)),
device='/gpu:%d' % i)
for i in range(8)]))
outputs, states = dynamic_rnn(cell, inputs, sequence_length)](https://image.slidesharecdn.com/tensorflowforiitians-190809065743/85/TensorFlow-for-IITians-41-320.jpg)
![Cloud Based Serving
First two steps (training, saving model) is the
same
Option 1: Run the web/api on a cloud
compute instance. Works across
Azure/AWS/GoogleCloud
Option 2: [Google Cloud] Deploy saved
model to Cloud ML Engine
Option 3: [AWS] Through SageMaker (only
oythin 2.7) or custom EC2/Lambda
Option 4: [Azure] Not able to see
any specific things - looks like plain
ol’ compute](https://image.slidesharecdn.com/tensorflowforiitians-190809065743/85/TensorFlow-for-IITians-55-320.jpg)
![Long Short-Term Memory (LSTMs):
Make Your Memory Cells Differentiable
[Hochreiter & Schmidhuber, 1997]
MX YMX Y
WRITE? READ?
FORGET?
W R
F
Sigmoids](https://image.slidesharecdn.com/tensorflowforiitians-190809065743/85/TensorFlow-for-IITians-61-320.jpg)
TensorFlow for IITians
- 1. TensorFlow For IITians Ashish Agarwal, Google Ashish Bansal, Capital One Dec, 2017
- 3. A Quick Comparison Source: https://svds.com/getting-started-deep- learning/
- 4. Machine Learning repository on GitHub #1
- 5. TensorFlow: Democratize ML ❖ An open-source machine learning platform for everyone ❖ Fast, flexible, and production-ready ❖ Scales from research to production
- 6. Talk Overview ❖ Tensorflow Introduction ➢ Execution model ➢ Writing your first model ❖ Image models ❖ Sequence models ❖ Tensorflow Deployment
- 7. Mul Add biases weights inputs labels Xent Nodes are Operations Dataflow Graph: Operations inputs * weights + biases
- 8. Mul Add biases weights inputs labels Xent Tensors flow along edges Dataflow Graphs: Tensors m = inputs * weights a = m + biases
- 9. Mul Add biases weights inputs labels Xent Values “fed” at execution times Dataflow Graph: Placeholders placeh older inputs = tf.placeholder(tf.float32)
- 10. Mul Add biases weights inputs labels Xent Variables represent persistent mutable state Dataflow Graph: Variables Variabl e weights = tf.Variable(0.3)
- 11. Interface to runtime: Sessions sess = tf.Session(...) sess.run(“f:0”, feed_dict={“b”: ...})
- 13. Example: Linear Regression using Gradient Descent
- 14. import tensorflow as tf # Input (x) x = tf.placeholder(tf.float32) Example: Linear Regression
- 15. import tensorflow as tf # Input (x) x = tf.placeholder(tf.float32) # Model parameters W = tf.Variable([.3], dtype=tf.float32) b = tf.Variable([-.3], dtype=tf.float32) # Prediction prediction = W * x + b Example: Linear Regression
- 16. import tensorflow as tf # Input (x) x = tf.placeholder(tf.float32) # Model parameters W = tf.Variable([.3], dtype=tf.float32) b = tf.Variable([-.3], dtype=tf.float32) # Prediction prediction = W * x + b # Label(y) and loss y = tf.placeholder(tf.float32) loss = tf.reduce_sum(tf.square(prediction - y)) Example: Linear Regression
- 17. import tensorflow as tf # Input (x) x = tf.placeholder(tf.float32) # Model parameters W = tf.Variable([.3], dtype=tf.float32) b = tf.Variable([-.3], dtype=tf.float32) # Prediction prediction = W * x + b # Label(y) and loss y = tf.placeholder(tf.float32) loss = tf.reduce_sum(tf.square(prediction - y)) Example: Linear Regression
- 18. import tensorflow as tf # Input (x) x = tf.placeholder(tf.float32) # Model parameters W = tf.Variable([.3], dtype=tf.float32) b = tf.Variable([-.3], dtype=tf.float32) # Prediction prediction = W * x + b # Label(y) and loss y = tf.placeholder(tf.float32) loss = tf.reduce_sum(tf.square(prediction - y)) Example: Linear Regression # Initialization, gradients and SGD nodes initializer = tf.global_variables_initializer() optimizer = tf.train.GradientDescentOptimizer(0.01) training_step = optimizer.minimize(loss)
- 19. import tensorflow as tf # Input (x) x = tf.placeholder(tf.float32) # Model parameters W = tf.Variable([.3], dtype=tf.float32) b = tf.Variable([-.3], dtype=tf.float32) # Prediction prediction = W * x + b # Label(y) and loss y = tf.placeholder(tf.float32) loss = tf.reduce_sum(tf.square(prediction - y)) Example: Linear Regression # Initialization, gradients and SGD nodes initializer = tf.global_variables_initializer() optimizer = tf.train.GradientDescentOptimizer(0.01) training_step = optimizer.minimize(loss) # Training data x_train = ... y_train = ... # Runtime setup sess = tf.Session()
- 20. import tensorflow as tf # Input (x) x = tf.placeholder(tf.float32) # Model parameters W = tf.Variable([.3], dtype=tf.float32) b = tf.Variable([-.3], dtype=tf.float32) # Prediction prediction = W * x + b # Label(y) and loss y = tf.placeholder(tf.float32) loss = tf.reduce_sum(tf.square(prediction - y)) Example: Linear Regression # Initialization, gradients and SGD nodes initializer = tf.global_variables_initializer() optimizer = tf.train.GradientDescentOptimizer(0.01) training_step = optimizer.minimize(loss) # Training data x_train = ... y_train = ... # Runtime setup sess = tf.Session() # Initialization and training loop sess.run(initializer) for i in range(1000): sess.run(training_step, feed_dict={x: x_train, y: y_train})
- 21. import tensorflow as tf # Input (x) x = tf.placeholder(tf.float32) # Model parameters W = tf.Variable([.3], dtype=tf.float32) b = tf.Variable([-.3], dtype=tf.float32) # Prediction prediction = W * x + b # Label(y) and loss y = tf.placeholder(tf.float32, shape=[4, 1]) loss = tf.reduce_sum(tf.square(prediction - y)) Example: Linear Logistic Regression loss = tf.reduce_sum( tf.nn.sigmoid_cross_entropy_with_logits( labels=y, logits=logits)) logits = W * x + b prediction = tf.sigmoid(logits)
- 22. import tensorflow as tf # Input (x) x = tf.placeholder(tf.float32) # Model parameters W = tf.Variable([.3], dtype=tf.float32) b = tf.Variable([-.3], dtype=tf.float32) # Prediction prediction = W * x + b # Label(y) and loss y = tf.placeholder(tf.float32, shape=[4, 1]) loss = tf.reduce_sum(tf.square(prediction - y)) Example: Linear Regression using tf.layers prediction = tf.layers.dense(x, units=1, use_bias=True, activation=None)
- 23. import tensorflow as tf # Input (x) x = tf.placeholder(tf.float32, shape=[4, 1]) # Prediction prediction = tf.layers.dense(x, units=1, use_bias=True, activation=None) # Label(y) and loss y = tf.placeholder(tf.float32, shape=[4, 1]) loss = tf.reduce_sum(tf.square(prediction - y)) Example: Linear Regression l1 = tf.layers.dense(x, units=100, use_bias=True, activation=relu) l2 = tf.layers.dense(l1, …) ... prediction = tf.layers.dense(ln, …) Non- ^
- 25. Convolutions output = tf.layers.conv2d( input, filters=2, kernel_size=(4, 4), strides=(1, 1)) Credit: @martin_gorner’s slides
- 26. Downsampling: Avg/Max Pooling output = tf.layers.max_pooling2d( input, pool_size=(4, 4), strides=2) Credit: @martin_gorner’s slides
- 27. Regularization: Dropouts output = tf.layers.dropout( input, keep_prob) Credit: @martin_gorner’s slides
- 28. Putting it together: AlexNet ImageNet Classification with Deep Convolutional Neural Networks Alex Krizhevsky, Ilya Sutskever and Geoffrey E. Hinton
- 29. Putting it together: AlexNet # Convolution and pooling layers. conv1 = tf.layers.conv2d(input, filters=64, kernel_size=[11, 11], stride=4) pool1 = tf.layers.max_pooling2d(conv1, kernel_size=[3, 3], stride=2) conv2 = tf.layers.conv2d(pool1, filters=192, kernel_size=[5, 5]) pool2 = tf.layers.max_pooling2d(conv2, kernel_size=[3, 3], stride=2) conv3 = tf.layers.conv2d(pool2, filters=384, kernel_size=[3, 3]) conv4 = tf.layers.conv2d(conv3, filters=384, kernel_size=[3, 3]) conv5 = tf.layers.conv2d(conv4, filters=256, kernel_size=[3, 3]) pool5 = tf.layers.max_pooling2d(conv5, kernel_size=[3, 3], stride=2) reshaped_pool5 = tf.reshape(pool5, [-1, 5 * 5 * 256]) # Fully connected layers with dropout. fc6 = tf.layers.dense(reshaped_pool5, units=4096) drp6 = tf.layers.dropout(fc6, keep_prob=0.5) fc7 = tf.layers.dense(drp6, units=4096) drp7 = tf.layers.dropout(fc7, keep_prob=0.5) fc8 = tf.layers.dense(drp7, units=1000, activation_fn=None) # Calculating the loss. loss = tf.nn.softmax_cross_entropy_with_logits( labels=labels, logits=fc8)
- 30. Batch Normalization: Reduce Internal Covariate Shift output = tf.layers.batch_normalization(input)
- 31. Multi-scale and 1X1 convolutions “Going Deeper with Convolutions” Christian Szegedy, Wei Liu, Yangqing Jia et. al.
- 32. The Inception Architecture (GoogLeNet, 2015) model = tf.keras.applications.InceptionV3(...) model.train_on_batch(...) model.predict(...)
- 33. Skip Connections: Residual Blocks Deep Residual Learning for Image Recognition, Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
- 34. Resnet Architecture model = tf.keras.applications.Resnet50(...) model.train_on_batch(...) model.predict(...)
- 35. Data Augmentation: Adversarial Examples
- 36. Healthy Diseased Hemorrhages No DR Mild DR Moderate DR Severe DR Proliferative DR
- 37. Sequence Prediction with RNNs IITs are the best आईआईटी सबसे अच्छे हैं
- 38. Looping constructs: RNN Credits: Olah’s “Understanding LSTMs”
- 39. LSTM: Long Short-Term Memory Credits: Olah’s “Understanding LSTMs”
- 40. Simple RNN def rnn(cell, input_list, initial_state): state = initial_state outputs = [] for inp in input_list: output, state = cell(inp, state) outputs.append(output) return outputs, state
- 41. Dynamic RNN # 8 layer LSTM with residual connections # Each layer is on a separate GPU cell = MultiRNNCell( [DeviceWrapper(ResidualWrapper(LSTMCell(num_units=512)), device='/gpu:%d' % i) for i in range(8)])) outputs, states = dynamic_rnn(cell, inputs, sequence_length)
- 42. Language Models (Unsupervised) InputsIITs are bestthe Labelsare bestthe <EOS>
- 43. Credit: @martin_gorner’s slides Deep Literature
- 44. Supervised Translation Models Input sentence Target sentence Sequence to Sequence Learning with Neural Networks Sutskever, Vinyals, Le आईआईटी सबसे हैंअच्छे IITs are bestthe <EOS> आईआईटी सबसे अच्छे
- 45. Encoder LSTMs Decoder LSTMs <s> Y1 Y3 SoftMax Y1 Y2 </s> X3 X2 </s> 8 Layers Gpu1 Gpu2 Gpu2 Gpu3 + + + Gpu8 Attention + ++ Gpu1 Gpu2 Gpu3 Gpu8 Neural Machine Translation Model
- 46. Encoder LSTMs Decoder LSTMs <s> Y1 Y3 SoftMax Y1 Y2 </s> X3 X2 </s> 8 Layers Gpu1 Gpu2 Gpu2 Gpu3 + + + Gpu8 Attention + ++ Gpu1 Gpu2 Gpu3 Gpu8 Neural Machine Translation Model Go Deep!
- 47. Encoder LSTMs Decoder LSTMs <s> Y1 Y3 SoftMax Y1 Y2 </s> X3 X2 </s> 8 Layers Gpu1 Gpu2 Gpu2 Gpu3 + + + Gpu8 Attention + ++ Gpu1 Gpu2 Gpu3 Gpu8 Neural Machine Translation Model Residual Connections
- 48. Encoder LSTMs Decoder LSTMs <s> Y1 Y3 SoftMax Y1 Y2 </s> X3 X2 </s> 8 Layers Gpu1 Gpu2 Gpu2 Gpu3 + + + Gpu8 Attention + ++ Gpu1 Gpu2 Gpu3 Gpu8 Neural Machine Translation Model Bidirectional LSTM
- 49. Encoder LSTMs Decoder LSTMs <s> Y1 Y3 SoftMax Y1 Y2 </s> X3 X2 </s> 8 Layers Gpu1 Gpu2 Gpu2 Gpu3 + + + Gpu8 Attention + ++ Gpu1 Gpu2 Gpu3 Gpu8 Neural Machine Translation Model Attention
- 50. Magenta A ht Xt
- 52. Deploying TensorFlow Applications aka Inference
- 53. Deployment Options and Choices at a Glance
- 54. Python / C App Custom API Serving Basic model is run inference in context of an app that serves an API This app can be built in C/C++ or Python easily It can be deployed on-prem or in the cloud An alternate path is to use tensorflow-serving Train a Model Save the Model & export to protobuf Load lookups/ embeddings Load the Model Call tf.session/run
- 55. Cloud Based Serving First two steps (training, saving model) is the same Option 1: Run the web/api on a cloud compute instance. Works across Azure/AWS/GoogleCloud Option 2: [Google Cloud] Deploy saved model to Cloud ML Engine Option 3: [AWS] Through SageMaker (only oythin 2.7) or custom EC2/Lambda Option 4: [Azure] Not able to see any specific things - looks like plain ol’ compute
- 56. Mobile Deployment Preparation of model for mobile requires additional steps - Consider reducing size of models by removing nodes not useful for inference - Quantization: https://www.tensorflow.org/performance/quantization - Usually embedded in an app - TensorFlow Lite and TensorFlow Mobile Qq: Do you really need it in an app? Similar process for Raspberry Pi
- 57. Thank You!
- 59. Putting it together: Convolutional Network conv2D max_pooling2d conv2D max_pooling2d dense dense 28 X 28 dropout Logits softmax_cross_entropy_with_logits Labels
- 60. Putting it together: Deep Mnist # First convolutional layer. x = tf.layers.conv2D(input, filters=32, kernel_size=5, activation=tf.nn.relu) # Pooling layer: down-samples by 2X. x = tf.layers.max_pooling2d(x, pool_size=2, strides=2) # Second convolutional layer. x = tf.layers.conv2D(x, filters=64, kernel_size=5, activation=tf.nn.relu) # Second pooling layer: down-samples by 2X. x = tf.layers.max_pooling2d(x, pool_size=2, strides=2) # Flatten the last three dimensions to allow # applying FC layers. x = tf.layers.flatten(x) # Fully connected layer with 1024 units. x = tf.layers.dense(x, units=1024, activation=tf.nn.relu) # Dropout. keep_prob = tf.placeholder(tf.float32) x = tf.layers.dropout(x, keep_prob) # Fully connected layer with 10 units. logits = tf.layers.dense(x, units=10)
- 61. Long Short-Term Memory (LSTMs): Make Your Memory Cells Differentiable [Hochreiter & Schmidhuber, 1997] MX YMX Y WRITE? READ? FORGET? W R F Sigmoids
- 62. TensorFlow Distributed Execution Engine CPU GPU Android iOS ... C++ FrontendPython Frontend ... Layers Estimator Models in a box Train and evaluate models Build models Keras Model Canned Estimators
Editor's Notes
- #3: [Ashish Bansal will intro] Theano is dead. TF is the de-facto standard.
- #4: Intro by Ashish Bansal
- #12: Runtime can be remote and distributed
- #15: Python frontend for graph generation
- #29: 16.4% compared to second result of 26.17% (map @5) ILSVRC2012 Now p@1 ~85%. p@5 ~2% ?
- #32: Multi-scale convolutions Dimension reduction using 1*1 convolutions
- #37: Diabetic retinopathy detection using retina scans
- #41: python: reduce haskell: foldl
- #51: In Google's Project Magenta, an open source project, we use ML to look at text and images, and analyze what's in them- which is extremely useful for our products and users. Just trying to guess what the next note should be. It does this by looking at 10K songs. You can train your networks on music from Radiohead to Bach. All we’re trying to do is predict the next note via a recurrent neural network, all written in TensorFlow.
- #53: This is Ashish Bansal’s section
- #54: What does deployment mean? In this context, we are talking about inference, not training.
- #62: The key idea behind LSTMs is to take that model and make it differentiable by replacing the gates with sigmoids. A LSTM cell has an input, which you multiply by a value gated by a sigmoid, which either squashes it or lets it flow into the memory. it has an output, which equivalently will be multiplied by a value between 0 and 1 depending on the sigmoid. And a ‘forget gate’ which controls whether the data remains in that memory cell. All of these gates are controlled by pieces of the neural network that connect to them, so you can backpropagate through them just like anything else and learn the rules governing theopening and closing of those gates. This is a very powerful paradigm in general: turn logic into something differentiable you can backprop through. We’ll re-use this later.