Educating your app – adding ML edge to your apps - Maoz Tamir
1 like189 views
The document discusses adding machine learning capabilities to mobile applications. It describes several cloud ML services that can be used, as well as the Firebase ML Kit which provides pre-trained models for tasks like face detection, barcode scanning, and image labeling that can run both on-device and in the cloud. The document also covers techniques for retraining existing models for custom purposes and optimizing models for mobile with tools like TensorFlow Lite. Live demos are provided of using Firebase ML Kit for image labeling and retraining models.
![Retrain
Set your input images
[any_path]/my_own_classifier/data/dalek
[any_path]/my_own_classifier/data/cyberman
[any_path]/my_own_classifier/data/doctor
Run the retrain
python -m scripts.retrain
--bottleneck_dir=tf_files/bottlenecks
--how_many_training_steps=500
--model_dir=tf_files/models/
--summaries_dir=tf_files/training_summaries/"${ARCHITECTURE}"
--output_graph=tf_files/retrained_graph.pb
--output_labels=tf_files/retrained_labels.txt
--image_dir=tf_files/data
<droidcon Tel Aviv>](https://image.slidesharecdn.com/droidcon-creatingsmartapplicationtfandmlkit-181023190205/85/Educating-your-app-adding-ML-edge-to-your-apps-Maoz-Tamir-14-320.jpg)
![Back to Android
private static final String MODEL_FILE = "file:///android_asset/graph.pb";
private static final String LABEL_FILE = "file:///android_asset/labels.txt";
// load the model into a TensorFlowInferenceInterface.
c.inferenceInterface = new TensorFlowInferenceInterface(
assetManager, MODEL_FILE );
// This model can be retrained with different numbers of output classes output array must be in
the same size
// Get the tensorflow node
final Operation operation = c.inferenceInterface.graphOperation(outputName);
// Inspect its shape
final int numClasses = (int) operation.output(0).shape().size(1);
// Build the output array with the correct size.
c.outputs = new float[numClasses];
<droidcon Tel Aviv>](https://image.slidesharecdn.com/droidcon-creatingsmartapplicationtfandmlkit-181023190205/85/Educating-your-app-adding-ML-edge-to-your-apps-Maoz-Tamir-18-320.jpg)
Educating your app – adding ML edge to your apps - Maoz Tamir
- 1. Creating Smarter Applications Maoz Tamir <droidcon Tel Aviv>
- 2. Seeing the Whole Picture <droidcon Tel Aviv>
- 3. In the beginning .. Classic image processing If … if … Specific solution for a specific problem Rule Engine– By clusters. More of the same Classic ML – Define features Deep Neural Networks Generic solution for a specific problem <droidcon Tel Aviv>
- 4. Add ML to your App Adding personalization into your App. Gather information and providing immediate feedback. Provides an efficient searching experience. Detect And Control <droidcon Tel Aviv>
- 5. Cloud ML Microsoft Cognitive Services Amazon Machine Learning Services IBM Watson Firebase ML Kit Alibaba Cloud services <droidcon Tel Aviv>
- 6. Firebase ML KIT Face Detection Barcode Scanning Landscape Detection Image Labeling Text Recognition Custom TensorFlow Model <droidcon Tel Aviv>
- 7. Firebase ML KIT Almost any application uses it . Build in ML Models. Both Cloud and on Device. Scalable. <droidcon Tel Aviv>
- 8. Live Demo <droidcon Tel Aviv>
- 9. Coding time – Vision label detection FirebaseVisionLabelDetector detector = FirebaseVision.getInstance)( . getVisionLabelDetector )(; //Or, to Use the cloud //FirebaseVisionCloudLabelDetector detector = FirebaseVision.getInstance)( //. getVisionCloudLabelDetector)options( ; FirebaseVisionImage image = FirebaseVisionImage.fromBitmap)bitmap( ; Task<List<FirebaseVisionLabel>> result= detector.detectInImage)image( . addOnSuccessListener( new OnSuccessListener<List<FirebaseVisionLabel <<)({ @Override public void onSuccess)List<FirebaseVisionLabel> labels({ // Task completed successfully } }) . addOnFailureListener( new OnFailureListener )({ @Override public void onFailure)@NonNull Exception e){ // Task failed with an exception } });
- 10. Creating something different oRecognize Something specific oRun Offline oResponse time oCost <droidcon Tel Aviv>
- 11. ML Frameworks TensorFlow Scikit-learn Caffe Torch Spark Mahout CNTK 2 (Microsoft Cognitive Toolkit) <droidcon Tel Aviv>
- 12. Retrain an Image Classifier for New Categories COLLECT AND PREPARE DATA RETRAIN OPTIMIZE TO MOBILE RUN ON DEVICE Fatkun
- 13. retraining Inception V3 for custom image classification Available models <droidcon Tel Aviv>
- 14. Retrain Set your input images [any_path]/my_own_classifier/data/dalek [any_path]/my_own_classifier/data/cyberman [any_path]/my_own_classifier/data/doctor Run the retrain python -m scripts.retrain --bottleneck_dir=tf_files/bottlenecks --how_many_training_steps=500 --model_dir=tf_files/models/ --summaries_dir=tf_files/training_summaries/"${ARCHITECTURE}" --output_graph=tf_files/retrained_graph.pb --output_labels=tf_files/retrained_labels.txt --image_dir=tf_files/data <droidcon Tel Aviv>
- 16. Optimize to Mobile – TF-Mobile or TF Lite TF lite Using TensorFlow Lite Converter TF for mobile Optimize for inference python -m tensorflow.python.tools.optimize_for_inference Remove or not needed nodes and handling unsupported operations Quantize the network weights python -m scripts.quantize_graph reduce neural network weights the graph size from 32bit to 8bit . <droidcon Tel Aviv>
- 17. Quantize the network weights <droidcon Tel Aviv>
- 18. Back to Android private static final String MODEL_FILE = "file:///android_asset/graph.pb"; private static final String LABEL_FILE = "file:///android_asset/labels.txt"; // load the model into a TensorFlowInferenceInterface. c.inferenceInterface = new TensorFlowInferenceInterface( assetManager, MODEL_FILE ); // This model can be retrained with different numbers of output classes output array must be in the same size // Get the tensorflow node final Operation operation = c.inferenceInterface.graphOperation(outputName); // Inspect its shape final int numClasses = (int) operation.output(0).shape().size(1); // Build the output array with the correct size. c.outputs = new float[numClasses]; <droidcon Tel Aviv>
- 19. Back to Android // Copy the input data into TensorFlow. inferenceInterface.feed( inputName, // The name of the node to feed. floatValues, // The array to feed 1, inputSize, inputSize, 3 ); // The shape of the array inferenceInterface.run( outputNames, // Names of all the nodes to calculate. logStats); // Bool, enable stat logging. inferenceInterface.fetch( outputName, // Fetch this output. outputs); // Into the prepared array. <droidcon Tel Aviv>
- 20. Live Demo <droidcon Tel Aviv>
- 21. Lesson learned Handle your data with care. Use the framework that fits you (TensorFlow , Torch, Caffe , Spark ….) Don’t be afraid to play with parameters. Learn to use Tensorboard. Use Linux or Linux based OS. Keep up with the research. <droidcon Tel Aviv>
- 22. Questions ? <droidcon Tel Aviv>