On-device ML with TFLite
1 like427 views
This document discusses on-device machine learning using TensorFlow Lite. It introduces why on-device ML is important, options for implementing on-device ML like TensorFlow Lite, ML Kit and MediaPipe. It then provides an end-to-end example of training a model on MNIST data using TensorFlow Keras, converting it to TFLite format and deploying it on Android. It also discusses optimizing models for mobile and edge devices and running inference on microcontrollers and Coral Edge TPUs.
![@margaretmz | #ML | #GDE
Validate TFLite model after conversion
31
Protip: validate the tflite model in python after conversion -
31
TensorFlow result TFLite result Compare results
# Test the TensorFlow model on random
Input data.
tf_result = model(tf.constant(input_data))
# Load TFLite model and allocate tensors.
interpreter = tf.lite.Interpreter(model_path="converted_model.tflite")
interpreter.allocate_tensors()
# Get input and output tensors.
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
# Test model on random input data.
input_shape = input_details[0]['shape']
input_data = np.array(np.random.random_sample(input_shape),
dtype=np.float32)
interpreter.set_tensor(input_details[0]['index'], input_data)
interpreter.invoke()
tflite_result = interpreter.get_tensor(output_details[0]['index'])
# Compare the result.
for tf_result, tflite_result in zip(tf_result, tflite_result):
np.testing.assert_almost_equal(tf_result,
tflite_result,
decimal=5)](https://image.slidesharecdn.com/20200212on-devicemlwithtflite-200213134116/85/On-device-ML-with-TFLite-31-320.jpg)
![@margaretmz | #ML | #GDE
Dependencies
Update build.gradle to include tensorflow lite
android {
// Make sure model doesn't get compressed when app is compiled
aaptOptions {
noCompress "tflite"
}
}
dependencies {
….
// Add dependency for TensorFlow Lite
compile 'org.tensorflow:tensorflow-lite:[version-number]’
}
Place the mnist.tflite model file under /assets folder
33](https://image.slidesharecdn.com/20200212on-devicemlwithtflite-200213134116/85/On-device-ML-with-TFLite-33-320.jpg)
On-device ML with TFLite
- 1. On-device ML with Lite Margaret Maynard-Reid, 2/12/2020 @margaretmz
- 2. @margaretmz | #ML | #GDE Topics ● Why on-device ML? ● On-device ML options ● E2E tf.Keras to TFLite to Android ○ train a model from scratch ○ convert to TFLite ○ deploy to mobile and IoT ● TFLite on microcontroller & Coral Edge TPU 2
- 3. @margaretmz | #ML | #GDE 3 Intro Why On-device ML? ● Access to more data ● Faster user interaction ● Preserve privacy Unique constraints: ● Less compute power ● Limited memory ● Battery consumption
- 4. @margaretmz | #ML | #GDE TensorFlow for mobile & edge devices 4 2015 TF open sourced 2016 TF mobile 2017 TF Lite developer preview 2018 ML Kit 2019 - New ML Kit features - TF Mobile deprecated - New TFLite features!!!
- 5. @margaretmz | #ML | #GDE TFLite on 3b+ devices! Source: Tensorflow Lite team 5
- 6. @margaretmz | #ML | #GDE Dance Like @I/O 2019 Segmentation, Pose, GPU on-device 6
- 7. @margaretmz | #ML | #GDE TensorFlow Lite ● Converter - convert to TFLite file format ● Interpreter - execute inference & optimized for small devices ● Ops/Kernel - limited ops ● Interface to hardware acceleration ○ NN API ○ Edge TPU 7
- 8. Optimization 1. Reduce model size TFLite model optimization toolkit ● Quantization - convert 32 bit floating point to fixed point (e.g. 8-bit int) ○ Post-training quantization ○ Quantization-aware training ● Pruning - eliminating unnecessary values in the weight tensor 8 2. Speed up inference On Android: ● GPU delegate ● Android NNAPI
- 9. On-device ML What are your options? Media Pipe 9
- 10. @margaretmz | #ML | #GDE On-device ML Options 10 What / how Who Where Native Android (iOS) apps ● Direct deploy to Android ● With ML Kit ● With MediaPipe ● Fritz.ai Android (or iOS) developers React Native Web developers TFLite / TF micro Embedded Microcontrollers Edge TPUs
- 11. @margaretmz | #ML | #GDE React Native Support ● Use TF.js ML directly inside React Native with WebGL acceleration ● Load models from the web, or compile into your application Link to demo video | Link to github 11
- 12. @margaretmz | #ML | #GDE Base APIs (Out of the box) Custom models ● Dynamic model downloads ● A/B testing (via Firebase remote Configuration) ● Model conversion (from TensorFlow to TFLite) Learn more about ML Kit 👉 g.co/mlkit Image labelling OCR Face detection Barcode scanning Landmark detection Smart reply Object detection & Tracking Translation (56 languages) AutoML Google ML Kit 12
- 13. @margaretmz | #ML | #GDE Why use ML Kit? 13 Convert to Bytebuffer/bit map Calibration Java Native Frame Scheduler (Image Timestamp) Convert to byte array Output Results Pipeline config Convert to Grayscale Resize/Rotate Tracker Frame Selection Convert to RGB/Resize/R otate Detector (TF Lite model) Object Manager Image Validation Resize Pipeline Classifier ( TF Lite model) Source: ML Kit team
- 14. @margaretmz | #ML | #GDE ● Firebase console ● AutoML - train model ● Download TFLite ● Mobile & edge https://firebase.google.com/docs/ml-kit/automl-image-labeling Google ML Kit - AutoML 14
- 15. @margaretmz | #ML | #GDE MediaPipe A cross-platform AI pipeline framework by Google Research: ● TensorFlow & TFLite ● Desktop, web, mobile, Coral Edge TPUs ● Fast & realtime ● GPU ● WebGL 15 Source: MediaPipe Github
- 16. @margaretmz | #ML | #GDE Two talks on Media Pipe @AI Nextcon 2/13 1PM @Google Seattle 2/13 5PM ● Google MediaPipe @Seattle by Ming Yong 16
- 17. @margaretmz | #ML | #GDE Fritz.ai Mobile ML made easy... ● Supports Android & iOS ● Features: Image labelling & segmentation, object detection, style transfer, pose estimation… ● Analytics, custom model hosting, perf monitoring… ● Free up to certain usage 17 Source: Embrace your new look with Fritz Hair Segmentation
- 18. Datasets Train model (Convert to TFLite) Deploy for inference End to End Model training to inference With TensorFlow 2.0 18
- 19. @margaretmz | #ML | #GDE End to end: model training to inference in TF 2.0 19 Model ● tf.Keras (TensorFlow) ● Python libraries: Numpy, Matplotlib etc SavedModel or Keras model Serving ● Cloud ● Web ● Mobile ● IoT ● Micro controllers ● Edge TPU Training Inference Data
- 20. @margaretmz | #ML | #GDE Data ● Existing datasets ○ Part of the deep learning framework: ■ MNIST, CIFAR10, FASHION_MNIST, IMDB movie reviews etc ○ Open datasets: ■ MNIST, MS-COCO, IMAGENet, CelebA etc ○ Kaggle datasets: https://www.kaggle.com/datasets ○ Google Dataset search tool: https://toolbox.google.com/datasetsearch ○ TF 2.0: TFDS ● Collect your own data 20
- 21. @margaretmz | #ML | #GDE Models Options of getting a model: ● Download a pre-trained model (here): Inception-v3, mobilenet etc. ● Transfer learning with a pre-trained model ○ Feature extraction or fine tuning on pre-trained model ○ TensorFlow hub (https://www.tensorflow.org/hub/) ● Train your own model from scratch (example in this talk) 21
- 22. @margaretmz | #ML | #GDE Model saving, conversion, deployment ● Model saving - SavedModel or Keras model ● Model conversion ○ Convert the model to tflite format ○ Validate the converted model before deploy ● Deploy TFLite for inference 22
- 23. @margaretmz | #ML | #GDE End to End: tf.Keras to TFLite to Android 23
- 24. @margaretmz | #ML | #GDE MNIST dataset ● 60,000 train set and 10,000 test set ● 28x28x1 grayscale images ● 10 classes: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 ● Popular for computer vision ○ “hello world” tutorial or ○ benchmarking ML algorithms 24
- 25. @margaretmz | #ML | #GDE Training the model in Colab Launch sample code on Colab → mnist_tfkeras_to_tflite.ipynb 1. Import data 2. Define model architecture 3. Train the model 4. Model saving & conversion ○ Save a Keras model ○ convert to tflite format 25
- 26. @margaretmz | #ML | #GDE A typical CNN model architecture MNIST example: ● Convolutional layer (definition) ● Pooling layer (definition) ● Dense (fully-connected layer) definition 26 input conv pool conv pool conv pool Dense 0 1 2 3 4 5 6 7 8 9
- 27. @margaretmz | #ML | #GDE Inspect the model - in python code In python code, after defining the model architecture, use model.summary() to show the model architecture 27
- 28. @margaretmz | #ML | #GDE Virtualize model Use a visualization tool: ● TensorBoard ● Netron (https://github.com/lutzroeder/Netron) Drop the .tflite model into Netron and see the model visually Note: model metadata a new TFLite tool (to be launched) will allow you to inspect the model & modify the metadata 28
- 29. @margaretmz | #ML | #GDE Model saving When to save as SavedModel or a Keras model? Note: In TensorFlow 2.0 , tf.keras.Model.save() and tf.keras.models.save_model() default to the SavedModel format (not HDF5). (link to doc) 29 SavedModel Keras Model Share pre-trained models and model pieces on TensorFlow Hub Train with tf.Keras and you know your deploy your target When you don’t know the deploy target
- 30. @margaretmz | #ML | #GDE Model conversion (with TFLite converter) 30 Command line Python code (recommended) SavedModel tflite_convert --saved_model_dir=/tmp/my_saved_model --output_file=/tmp/my_model.tflite Keras Model --keras_model_file=/tmp/my_keras_model.h5 --output_file=/tmp/my_model.tflite # Create a converter converter = tf.contrib.lite.TFLiteConverter.from_keras_model_file(keras_model) from_keras_model(model) # Set quantize to true (optional) converter.post_training_quantize=True # Convert the model tflite_model = converter.convert() # Create the tflite model file tflite_model_name = "my_model.tflite" open(tflite_model_name, "wb").write(tflite_model)
- 31. @margaretmz | #ML | #GDE Validate TFLite model after conversion 31 Protip: validate the tflite model in python after conversion - 31 TensorFlow result TFLite result Compare results # Test the TensorFlow model on random Input data. tf_result = model(tf.constant(input_data)) # Load TFLite model and allocate tensors. interpreter = tf.lite.Interpreter(model_path="converted_model.tflite") interpreter.allocate_tensors() # Get input and output tensors. input_details = interpreter.get_input_details() output_details = interpreter.get_output_details() # Test model on random input data. input_shape = input_details[0]['shape'] input_data = np.array(np.random.random_sample(input_shape), dtype=np.float32) interpreter.set_tensor(input_details[0]['index'], input_data) interpreter.invoke() tflite_result = interpreter.get_tensor(output_details[0]['index']) # Compare the result. for tf_result, tflite_result in zip(tf_result, tflite_result): np.testing.assert_almost_equal(tf_result, tflite_result, decimal=5)
- 32. @margaretmz | #ML | #GDE Tflite on Android Android sample code DigitRecognizer, step by step: ● Place tf.lite model under assets folder ● Update build.gradle dependencies ● Input image - custom view, gallery or camera ● Data preprocessing ● Classify with the model ● Post processing ● Display result in UI 32
- 33. @margaretmz | #ML | #GDE Dependencies Update build.gradle to include tensorflow lite android { // Make sure model doesn't get compressed when app is compiled aaptOptions { noCompress "tflite" } } dependencies { …. // Add dependency for TensorFlow Lite compile 'org.tensorflow:tensorflow-lite:[version-number]’ } Place the mnist.tflite model file under /assets folder 33
- 34. @margaretmz | #ML | #GDE Input - image data Input to the classifier is an image, your options: ● Draw on canvas from custom View ● Get image from Gallery or a 3rd party camera ● Live frames from Camera2 API Make sure the image dimensions (shape) matches what your classifier expects ● 28x28x1- MNIST or FASHION_MNIST gray scale image ● 299x299x3 - Inception V3 ● 256x256x3 - MobileNet 34
- 35. @margaretmz | #ML | #GDE Image preprocessing ● Convert Bitmap to ByteBuffer ● Normalize pixel values to be a certain range ● Convert from color to grayscale, if needed 35
- 36. @margaretmz | #ML | #GDE Run inference Load the model file located under the assets folder Use the TensorFlow Lite interpreter to run inference on the input image 36
- 37. @margaretmz | #ML | #GDE Post processing The output is an array of probabilities, each correspond to a category Find the category with the highest probability and output result to UI 37
- 38. @margaretmz | #ML | #GDE Summary ● Training with tf.Keras is easy ● Model conversion to TFLite is easier ● Android implementation is getting better: ○ Validate tflite model before deploy to Android ○ Image pre-processing ○ Input tensor shape? ○ Color or grayscale? ○ Post processing My blog post: E2E tf.Keras to TFLite to Android 38
- 39. @margaretmz | #ML | #GDE New TFLite features Announced at TensorFlow World: 1. New TFLite support library (link) 2. Model metadata (not yet launched) 3. Model repository pre-converted to tflite format (link to models w/ examples | link to hosted models) 4. Transfer learning made easy - model customization API (link) 5. Ready to use end-to-end tutorials and full example apps (link) 6. TFLite course on Udacity (link) 39
- 40. @margaretmz | #ML | #GDE TFLite classification demo app Check out the classification Demo app in TensorFlow repo 40
- 41. @margaretmz | #ML | #GDE Inference with GPU ● Face contour detection ● Link to blog post: TensorFlow Lite Now Faster with Mobile GPUs 41
- 42. @margaretmz | #ML | #GDE Posenet example ● PoseNet model on Android ● Camera live frames ● Display key body parts in real time ● Link to blog post: Track human poses in real-time on Android with TensorFlow Lite 42
- 43. @margaretmz | #ML | #GDE More TFLite examples 43
- 44. @margaretmz | #ML | #GDE On device ML training is finally here! ● Train with ~20 images ● Use transfer learning ● Quantized MobileNetV2 ● Android device (5.0+) Link to blog | Android sample 44
- 45. @margaretmz | #ML | #GDE TFLite on microcontroller ● Tiny models on tiny computers ● Consumes much less power than CPUs - days on a coin battery ● Tiny RAM and Flash available ● Opens up voice interface to devs More info here - ● Doc - https://www.tensorflow.org/lite/guide/microcontroller ● Code lab - https://g.co/codelabs/sparkfunTF ● Purchase - https://www.sparkfun.com/products/15170 45
- 46. @margaretmz | #ML | #GDE Coral edge TPU (beta) - hardware for on-device ML acceleration Link to codelab: https://codelabs.developers.google.com/codelabs/edgetpu-classifier/index.html#0 ● Dev board (+ camera module) ● USB Accelerator (+ camera module + Raspberry Pi) Coral Edge TPU 46
- 47. @margaretmz | #ML | #GDE Coral Edge TPU MobileNet SSD model running on TPU Inference time: < ~20 ms > ~60 fps 47
- 48. @margaretmz | #ML | #GDE Coral Edge TPU demo MobileNet SSD model running on CPU Inference time > ~390ms ~ 3fps 48
- 49. @margaretmz | #ML | #GDE On-device ML trends ● Why the future of machine learning is tiny? - Pete Warden ● Deploying to mobile and IoT will get much easier ● TFLite will have many more features ● Federated learning ● On device training 49
- 50. @margaretmz | #ML | #GDE Awesome TFLite 😎 bit.ly/awesome-tflite - please star ⭐ the repo if you find it useful! 50
- 51. @margaretmz | #ML | #GDE Thank you! 51 Follow me on Twitter, Medium or GitHub to learn more about deep learning, TensorFlow and on-device ML @margaretmz @margaretmz margaretmz