Developing a ML model using TF Estimator
2 likes988 views
This document discusses developing machine learning models. It covers using tf.estimator for building models, training pipelines with TensorFlow Records and tf.data for efficient input pipelines, and distributed training with GPUs and TPUs. The document provides an example of building a MNIST classifier with tf.estimator and discusses steps like creating estimators, model functions, and training specifications. It also covers creating custom estimators and using premade estimators.
![#### 1 INFERENCE MODEL
input_layer = tf.reshape(features["x"], [-1, 28, 28, 1])
conv1 = _conv(input_layer,kernel=[5,5,1,64],name='conv1',log=params['log'])
conv2 = _conv(conv1,kernel=[5,5,64,64],name='conv2',log=params['log'])
dense = _dense(conv2,size_in=7*7*64,size_out=params['dense_units'],
name='Dense',relu=True,log=params['log'])
if mode==tf.estimator.ModeKeys.TRAIN:
dense = tf.nn.dropout(dense,params['drop_out'])
logits = _dense(dense,size_in=params['dense_units'],
size_out=10,name='Output',relu=False,log=params['log'])
#1 Inference() and #2 Calculations and Metrics
#1 Inference()](https://image.slidesharecdn.com/arzfekjkt3sil4srxuya-signature-f0fc63a7c04fd0200d9f4814cbe3cd2d81aef06c50d366a726c0226f63af14fe-poli-190304135013/85/Developing-a-ML-model-using-TF-Estimator-25-320.jpg)
![#### 2 CALCULATIONS AND METRICS
predictions = {"classes": tf.argmax(input=logits,axis=1),
"logits": logits,
"probabilities": tf.nn.softmax(logits,name='softmax')}
export_outputs = {'predictions': tf.estimator.export.PredictOutput(predictions)}
if (mode==tf.estimator.ModeKeys.TRAIN or mode==tf.estimator.ModeKeys.EVAL):
loss = tf.losses.sparse_softmax_cross_entropy(labels=labels,logits=logits)
accuracy = tf.metrics.accuracy(
labels=labels, predictions=tf.argmax(logits,axis=1))
metrics = {'accuracy':accuracy}
tf.summary.scalar('accuracy',accuracy[1])
#1 Inference() and #2 Calculations and Metrics #2 Calculations](https://image.slidesharecdn.com/arzfekjkt3sil4srxuya-signature-f0fc63a7c04fd0200d9f4814cbe3cd2d81aef06c50d366a726c0226f63af14fe-poli-190304135013/85/Developing-a-ML-model-using-TF-Estimator-26-320.jpg)
![#### 4 MODE = TRAIN
if mode == tf.estimator.ModeKeys.TRAIN:
learning_rate = tf.train.exponential_decay(
params['learning_rate'],tf.train.get_global_step(),
decay_steps=100000,decay_rate=0.96)
optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)
if params['replicate']==True:
optimizer = tf.contrib.estimator.TowerOptimizer(optimizer)
train_op = optimizer.minimize(loss=loss,global_step=tf.train.get_global_step())
tf.summary.scalar('learning_rate', learning_rate)
return tf.estimator.EstimatorSpec(
mode=mode, loss=loss, train_op=train_op)
#3 Predict, #4 Train, and #5 Eval #4 MODE = TRAIN](https://image.slidesharecdn.com/arzfekjkt3sil4srxuya-signature-f0fc63a7c04fd0200d9f4814cbe3cd2d81aef06c50d366a726c0226f63af14fe-poli-190304135013/85/Developing-a-ML-model-using-TF-Estimator-28-320.jpg)
![dataset = tf.data.TFRecordDataset(c['files'],num_parallel_reads=c['threads'])
tf.data input_fn()](https://image.slidesharecdn.com/arzfekjkt3sil4srxuya-signature-f0fc63a7c04fd0200d9f4814cbe3cd2d81aef06c50d366a726c0226f63af14fe-poli-190304135013/85/Developing-a-ML-model-using-TF-Estimator-35-320.jpg)
![dataset = dataset.map(parse_tfrecord,num_parallel_calls=c['threads'])
dataset = dataset.map(lambda x,y: (image_scaling(x),y),num_parallel_calls=c['threads'])
tf.data input_fn()](https://image.slidesharecdn.com/arzfekjkt3sil4srxuya-signature-f0fc63a7c04fd0200d9f4814cbe3cd2d81aef06c50d366a726c0226f63af14fe-poli-190304135013/85/Developing-a-ML-model-using-TF-Estimator-36-320.jpg)
![if c['mode']==tf.estimator.ModeKeys.TRAIN:
dataset = dataset.map(lambda x,y: (distort(x),y),num_parallel_calls=c['threads'])
dataset = dataset.shuffle(buffer_size=c['shuffle_buff'])
tf.data input_fn()](https://image.slidesharecdn.com/arzfekjkt3sil4srxuya-signature-f0fc63a7c04fd0200d9f4814cbe3cd2d81aef06c50d366a726c0226f63af14fe-poli-190304135013/85/Developing-a-ML-model-using-TF-Estimator-37-320.jpg)
![dataset = dataset.repeat()
dataset = dataset.batch(c['batch'])
dataset = dataset.prefetch(2*c['batch'])
tf.data input_fn()](https://image.slidesharecdn.com/arzfekjkt3sil4srxuya-signature-f0fc63a7c04fd0200d9f4814cbe3cd2d81aef06c50d366a726c0226f63af14fe-poli-190304135013/85/Developing-a-ML-model-using-TF-Estimator-38-320.jpg)
![gpu:0
Output Reduction
graph
mat
mul
add
gpu:1
graph
mat
mul
add
X split
[ grad(gpu:0) , ypred(gpu:0) , loss(gpu:0) ]
[ grad(gpu:1) , ypred(gpu:1) , loss(gpu:1) ]](https://image.slidesharecdn.com/arzfekjkt3sil4srxuya-signature-f0fc63a7c04fd0200d9f4814cbe3cd2d81aef06c50d366a726c0226f63af14fe-poli-190304135013/85/Developing-a-ML-model-using-TF-Estimator-47-320.jpg)
![gpu:0
Output Reduction
graph
mat
mul
add
gpu:1
graph
mat
mul
add
X split
[ grad(gpu:0) , ypred(gpu:0) , loss(gpu:0) ]
[ grad(gpu:1) , ypred(gpu:1) , loss(gpu:1) ]
avg
con-
cat
sum
[ grad(model) , ypred(model) , loss(model) ]
Update
Model
Params](https://image.slidesharecdn.com/arzfekjkt3sil4srxuya-signature-f0fc63a7c04fd0200d9f4814cbe3cd2d81aef06c50d366a726c0226f63af14fe-poli-190304135013/85/Developing-a-ML-model-using-TF-Estimator-48-320.jpg)
Developing a ML model using TF Estimator
- 1. Developing Machine Learning Models Kevin Tsai
- 3. GOTO Chicago 2018 Developing Machine Learning Models Kevin Tsai, Google
- 4. Agenda Machine Learning with tf.estimator Performance pipelines with TFRecords and tf.data Distributed Training with GPUs and TPUs
- 5. Machine Learning with tf.estimator
- 6. Model Serving Training Pipeline Inference Pipeline Predictions Model Building Machine Learning Pipeline Transform & Feature Engineering Transform & Feature Engineering Training Data Inference Requests
- 7. Model Serving Training Pipeline Inference Pipeline Predictions Machine Learning Pipeline train() evaluate() Model split() Transform & Feature Engineering Transform & Feature Engineering Training Data Inference Requests
- 8. Model Serving Training Pipeline Inference Pipeline Predictions Machine Learning Pipeline train() evaluate() Model split() Logs Transform & Feature Engineering Transform & Feature Engineering Training Data Inference Requests
- 9. Model Serving Training Pipeline Inference Pipeline Predictions Machine Learning Pipeline train() export() evaluate() Model split() Logs Transform & Feature Engineering Transform & Feature Engineering Training Data Inference Requests
- 10. tf.estimator estimator model_fn Model Serving Training Pipeline Inference Pipeline Transform & Feature Engineering Transform & Feature Engineering Predictions Training Data Inference Requests Machine Learning Pipeline with tf.estimator
- 11. tf.estimator estimator model_fn train() evaluate() predict() Model Serving Training Pipeline Inference Pipeline Transform & Feature Engineering Transform & Feature Engineering Predictions Training Data Inference Requests Machine Learning Pipeline with tf.estimator
- 12. tf.estimator estimator model_fn train_input_fn eval_input_fn predict_input_fn train() evaluate() predict() Model Serving Training Pipeline Inference Pipeline Transform & Feature Engineering Transform & Feature Engineering Predictions Training Data Inference Requests Machine Learning Pipeline with tf.estimator
- 13. tf.estimator estimator model_fn train_input_fn eval_input_fn predict_input_fn train() evaluate() predict() Model Serving Training Pipeline Inference Pipeline Transform & Feature Engineering Transform & Feature Engineering Predictions Logs Training Data Inference Requests Machine Learning Pipeline with tf.estimator
- 14. tf.estimator estimator model_fn train_input_fn eval_input_fn predict_input_fn train() evaluate() predict() serving_input_rec_fn export_savedmodel() Model Serving Training Pipeline Inference Pipeline Transform & Feature Engineering Transform & Feature Engineering Predictions Logs Training Data Inference Requests Machine Learning Pipeline with tf.estimator
- 15. Estimator in TensorFlow TensorFlow Distributed Execution Engine Python Layers Canned estimators Estimator Keras model Models in a box Train and evaluate models Build models C++ Java Go ... CPU GPU TPU Android iOS ...
- 16. Example: MNIST ? 1 MNIST: Yann LeCunn
- 17. Training: 1. Import data 2. Create input functions 3. Create estimator 4. Create train and evaluate specs 5. Run train and evaluate Test Inference: 6. Test run predict() 7. Export model Steps to Run Training on Premade Estimator
- 18. Premade Estimators DNNClassifier DNNRegressor LinearClassifier LinearRegressor DNNLinearCombinedClassifier DNNLinearCombinedRegressor + tf.contrib.estimator Premade and Custom Estimators or Custom Estimator
- 21. Reshape Convolution ReLU Max Pool Convolution ReLU Max Pool (-1,784) (-1,28,28,1) (-1,14,14,64) (-1,7,7,64) #1 Inference() #1 Inference()
- 22. Reshape Convolution ReLU Max Pool Reshape Dense ReLU Convolution ReLU Max Pool (-1,784) (-1,28,28,1) (-1,14,14,64) (-1,7,7,64) (-1,1024) (-1,3136) #1 Inference() Dropout #1 Inference()
- 23. Reshape Convolution ReLU Max Pool Reshape Dense ReLU Softmax Convolution ReLU Max Pool (-1,784) (-1,28,28,1) (-1,14,14,64) (-1,7,7,64) (-1,1024) (-1,10) (-1,3136) (-1,10) #1 Inference() Dropout Dense #1 Inference()
- 24. Reshape Convolution ReLU Max Pool Reshape Dense ReLU Softmax Convolution ReLU Max Pool (-1,784) (-1,28,28,1) (-1,14,14,64) (-1,7,7,64) (-1,1024) (-1,10) (-1,3136) (-1,10) log tf.name_scope tf.nn.conv2d tf.nn.relu tf.nn.max_pool tf.summary.histogram( tf.summary.histogram tf.summary.histogram log relu=False tf.name_scope tf.add tf.matmul tf.nn.relu tf.summary.histogram tf.summary.histogram tf.summary.histogram #1 Inference() Dropout Dense Hands-on TensorBoard: Dandelion Mane #1 Inference()
- 25. #### 1 INFERENCE MODEL input_layer = tf.reshape(features["x"], [-1, 28, 28, 1]) conv1 = _conv(input_layer,kernel=[5,5,1,64],name='conv1',log=params['log']) conv2 = _conv(conv1,kernel=[5,5,64,64],name='conv2',log=params['log']) dense = _dense(conv2,size_in=7*7*64,size_out=params['dense_units'], name='Dense',relu=True,log=params['log']) if mode==tf.estimator.ModeKeys.TRAIN: dense = tf.nn.dropout(dense,params['drop_out']) logits = _dense(dense,size_in=params['dense_units'], size_out=10,name='Output',relu=False,log=params['log']) #1 Inference() and #2 Calculations and Metrics #1 Inference()
- 26. #### 2 CALCULATIONS AND METRICS predictions = {"classes": tf.argmax(input=logits,axis=1), "logits": logits, "probabilities": tf.nn.softmax(logits,name='softmax')} export_outputs = {'predictions': tf.estimator.export.PredictOutput(predictions)} if (mode==tf.estimator.ModeKeys.TRAIN or mode==tf.estimator.ModeKeys.EVAL): loss = tf.losses.sparse_softmax_cross_entropy(labels=labels,logits=logits) accuracy = tf.metrics.accuracy( labels=labels, predictions=tf.argmax(logits,axis=1)) metrics = {'accuracy':accuracy} tf.summary.scalar('accuracy',accuracy[1]) #1 Inference() and #2 Calculations and Metrics #2 Calculations
- 27. #### 3 MODE = PREDICT if mode == tf.estimator.ModeKeys.PREDICT: return tf.estimator.EstimatorSpec( mode=mode, predictions=predictions, export_outputs=export_outputs) #3 Predict, #4 Train, and #5 Eval #3 MODE = PREDICT
- 28. #### 4 MODE = TRAIN if mode == tf.estimator.ModeKeys.TRAIN: learning_rate = tf.train.exponential_decay( params['learning_rate'],tf.train.get_global_step(), decay_steps=100000,decay_rate=0.96) optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate) if params['replicate']==True: optimizer = tf.contrib.estimator.TowerOptimizer(optimizer) train_op = optimizer.minimize(loss=loss,global_step=tf.train.get_global_step()) tf.summary.scalar('learning_rate', learning_rate) return tf.estimator.EstimatorSpec( mode=mode, loss=loss, train_op=train_op) #3 Predict, #4 Train, and #5 Eval #4 MODE = TRAIN
- 29. #### 5 MODE = EVAL if mode == tf.estimator.ModeKeys.EVAL: return tf.estimator.EstimatorSpec( mode=mode,loss=loss,eval_metric_ops=metrics) #3 Predict, #4 Train, and #5 Eval #5 MODE = EVAL
- 31. (Inefficient) Pipeline # Load Data mnist = tf.contrib.learn.datasets.load_dataset("mnist") train_data = mnist.train.images train_labels = np.asarray(mnist.train.labels, dtype=np.int32) eval_data = mnist.test.images eval_labels = np.asarray(mnist.test.labels, dtype=np.int32) # Create Input Functions train_input_fn = tf.estimator.inputs.numpy_input_fn( x={"x": train_data},y=train_labels,batch_size=100,num_epochs=None,shuffle=True) eval_input_fn = tf.estimator.inputs.numpy_input_fn( x={"x": eval_data},y=eval_labels,num_epochs=1,shuffle=False) pred_input_fn = tf.estimator.inputs.numpy_input_fn( x={"x": eval_data},num_epochs=1,shuffle=False)
- 32. TFRecord train004.tfrecords tf.train.Example tf.train.Example tf.train.Example tf.train.Example tf.train.Example ...
- 33. train004.tfrecords tf.train.Example tf.train.Example tf.train.Example tf.train.Example tf.train.Example ... tf.train.Example tf.train.Features tf.train.Feature tf.train.Feature tf.train.Feature ... tf.python_io.TFRecordWriter tf.train.Feature tf.train.Feature tf.train.Example tf.train.Features TFRecord
- 35. dataset = tf.data.TFRecordDataset(c['files'],num_parallel_reads=c['threads']) tf.data input_fn()
- 36. dataset = dataset.map(parse_tfrecord,num_parallel_calls=c['threads']) dataset = dataset.map(lambda x,y: (image_scaling(x),y),num_parallel_calls=c['threads']) tf.data input_fn()
- 37. if c['mode']==tf.estimator.ModeKeys.TRAIN: dataset = dataset.map(lambda x,y: (distort(x),y),num_parallel_calls=c['threads']) dataset = dataset.shuffle(buffer_size=c['shuffle_buff']) tf.data input_fn()
- 38. dataset = dataset.repeat() dataset = dataset.batch(c['batch']) dataset = dataset.prefetch(2*c['batch']) tf.data input_fn()
- 39. iterator = dataset.make_one_shot_iterator() images, labels = iterator.get_next() features = {'images': images} return features, labels tf.data input_fn()
- 40. train_spec = tf.estimator.TrainSpec(input_fn=lambda: dataset_input_fn(train_params)) eval_spec = tf.estimator.EvalSpec(input_fn=lambda: dataset_input_fn(eval_params)) tf.data input_fn()
- 42. Parallelism is Not Automatic
- 43. graph mat mul add X ypred = WX + b W b ypred Data and Model Parallelism
- 44. gpu:0 graph mat mul add X ypred = WX + b W b ypred graph mat mul add gpu:1 graph mat mul add X Data Parallelism split Data and Model Parallelism
- 45. gpu:1 gpu:0 graph mat mul add X ypred = WX + b W b ypred graph mat mul add gpu:1 graph mat mul add gpu:0 graph mat mul add X Data Parallelism Model Parallelism X split Data and Model Parallelism
- 46. gpu:1 gpu:0 graph mat mul add X ypred = WX + b W b ypred graph mat mul add gpu:1 graph mat mul add gpu:0 graph mat mul add X Data Parallelism Model Parallelism X gpu:3gpu:2 graph mat mul add split split Data and Model Parallelism
- 47. gpu:0 Output Reduction graph mat mul add gpu:1 graph mat mul add X split [ grad(gpu:0) , ypred(gpu:0) , loss(gpu:0) ] [ grad(gpu:1) , ypred(gpu:1) , loss(gpu:1) ]
- 48. gpu:0 Output Reduction graph mat mul add gpu:1 graph mat mul add X split [ grad(gpu:0) , ypred(gpu:0) , loss(gpu:0) ] [ grad(gpu:1) , ypred(gpu:1) , loss(gpu:1) ] avg con- cat sum [ grad(model) , ypred(model) , loss(model) ] Update Model Params
- 49. Output Reduction gpu:3 gpu:1 gpu:0 gpu:2gpu:3 gpu:1 gpu:0 gpu:2 cpu/gpu Naive Consolidation Ring All-Reduce Baidu All-Reduce: Andrew Ng
- 50. Distributed Accelerator Options replicate_model_fn (TF1.5+; Naive Consolidation; Single Server) distribute.MirroredStrategy (TF1.8+; All-Reduce; Single Server for now) TPUEstimator (TF1.4+; All-Reduce; 64-TPU pod) Horovod (TF1.8+; All-Reduce; Multi Server) multi_gpu_model (Keras; Naive Consolidation; Single Server) Meet Horovod: Alex Sergeev, Mike Del Balso
- 51. TPU / TPU Pod TPUs Batch Size Time to 90 Epochs Accuracy 1 256 23:22 76.6% 4 1024 05:48 76.3% 16 4096 01:30 76.5% 32 8192 45 min 76.1% 64 16384 22 min 75.0% 64 8192 ->16384 29.5 min 76.1% ImageNet is the New MNIST
- 52. Summary tf.estimator ● Premade Estimators for quick and dirty jobs (5-part recipe) ● Custom models w/o the plumbing (5-part function) ● Keras Functional API for Inference() in custom model_fn() ● Use Checkpoints/Preemptible VMs/Cloud Storage to reduce cost ● Scalable model building from laptop to GPUs to TPU Pods TFRecords & tf.data for pipeline performance ● Parallelize creation of TFRecord files ~ 100-150MB ● Use tf.data input_fn() Distributed Accelerator options ● Input pipeline before bigger/faster/more accelerators ● Scale Up before Out
- 53. Thank you.