Who's afraid of ML -V2- Hello MLKit
0 likes117 views
The document discusses implementing machine learning models on mobile devices using TensorFlow Lite. It provides an example of building an image classification model that can identify images of fruits. The steps include loading a pre-trained model and label file from assets, preprocessing input images, running inference on the model to obtain predictions, and displaying the most likely labels. The document also introduces some common machine learning models available in TensorFlow Lite for tasks like text recognition, face detection, and barcode scanning.
![Image
Classifier
[strawberry, apple, ...]
labels.txt](https://image.slidesharecdn.com/whosafraid-v2-180823160207/85/Who-s-afraid-of-ML-V2-Hello-MLKit-59-320.jpg)
![Image
Classifier
[ [0..6] , [ 0.1 ] , ...]
[strawberry, apple, ...]
probArray
labels.txt](https://image.slidesharecdn.com/whosafraid-v2-180823160207/85/Who-s-afraid-of-ML-V2-Hello-MLKit-66-320.jpg)
![probArray =
{
[0.7],
[0.3],
[0],
[0],
}
labelList =
{
strawberry,
apple,
pineapple,
banana
}
0.3](https://image.slidesharecdn.com/whosafraid-v2-180823160207/85/Who-s-afraid-of-ML-V2-Hello-MLKit-67-320.jpg)
![ImageClassifier.java
model = loadModelFile();
tflite = new Interpreter(model);
labelList = loadLabelList();
probArray = new float[1][labelList.size()];](https://image.slidesharecdn.com/whosafraid-v2-180823160207/85/Who-s-afraid-of-ML-V2-Hello-MLKit-68-320.jpg)
![Image
Classifier
[......] [ [0..6] , [ 0.1 ] , ...]
[strawberry, apple, ...]
ByteBuffer probArray
labels.txt](https://image.slidesharecdn.com/whosafraid-v2-180823160207/85/Who-s-afraid-of-ML-V2-Hello-MLKit-69-320.jpg)
![ImageClassifier.java
model = loadModelFile();
tflite = new Interpreter(model);
labelList = loadLabelList();
probArray = new float[1][labelList.size()];
imgData =
ByteBuffer.allocateDirect(
DIM_IMG_SIZE_X * DIM_IMG_SIZE_Y * DIM_PIXEL_SIZE);
imgData.order(ByteOrder.nativeOrder());](https://image.slidesharecdn.com/whosafraid-v2-180823160207/85/Who-s-afraid-of-ML-V2-Hello-MLKit-70-320.jpg)
![classifier
.classify(bitmap)
Image
Classifier
[......] [ [0..6] , [ 0.1 ] , ...]
[strawberry, apple, ...]
ByteBuffer probArray
labels.txt](https://image.slidesharecdn.com/whosafraid-v2-180823160207/85/Who-s-afraid-of-ML-V2-Hello-MLKit-72-320.jpg)
![void convertBitmapToByteBuffer(Bitmap bitmap) {
//...
bitmap.getPixels(intValues, 0, bitmap.getWidth(),
0, 0,bitmap.getWidth(), bitmap.getHeight());
int pixel = 0;
for (int i = 0; i < DIM_IMG_SIZE_X; ++i) {
for (int j = 0; j < DIM_IMG_SIZE_Y; ++j) {
final int val = intValues[pixel++];
imgData.put((byte) ((val >> 16) & 0xFF));
imgData.put((byte) ((val >> 8) & 0xFF));
imgData.put((byte) (val & 0xFF));
}
}
}](https://image.slidesharecdn.com/whosafraid-v2-180823160207/85/Who-s-afraid-of-ML-V2-Hello-MLKit-75-320.jpg)
![private String getTopKLabels() {
for (int i = 0; i < labelList.size(); ++i) {
sortedLabels.add(
new AbstractMap.SimpleEntry<>(labelList.get(i), (labelProbArray[0][i] &
0xff) / 255.0f));
if (sortedLabels.size() > RESULTS_TO_SHOW) {
sortedLabels.poll();
}
}
String textToShow = "";
final int size = sortedLabels.size();
for (int i = 0; i < size; ++i) {
Map.Entry<String, Float> label = sortedLabels.poll();
textToShow = String.format("n%s: %4.2f", label.getKey(), label.getValue())
+ textToShow;
}
return textToShow;
}](https://image.slidesharecdn.com/whosafraid-v2-180823160207/85/Who-s-afraid-of-ML-V2-Hello-MLKit-78-320.jpg)
![Links
- Tensorflow
- https://www.tensorflow.org/
- Tensorflow lite
- https://www.tensorflow.org/mobile/tflite/
- Codes labs
- codelabs.developers.google.com/codelabs/tensorflow-for-poets-2-tflite/
- Google’s Machine Learning Crash Course
- developers.google.com/machine-learning/crash-course/
- [Dr. Joe Dispenza]](https://image.slidesharecdn.com/whosafraid-v2-180823160207/85/Who-s-afraid-of-ML-V2-Hello-MLKit-81-320.jpg)
Who's afraid of ML -V2- Hello MLKit
- 1. Who’s afraid of Machine Learning? Britt Barak
- 2. Britt Barak Google Developer Expert - Android Women Techmakers Israel Britt Barak @brittBarak
- 12. In a machine...
- 20. 0.64 0.75 0.4 Input Red Seeds pattern Top leaves 0.5 0.8 0.3 0.5 * 0.64 + 0.8 * 0.75 + 0.3 * 0.4
- 21. 0.64 0.75 0.4 Input Red Seeds pattern Top leaves 0.5 0.8 0.3 0.5 * 0.64 + 0.8 * 0.75 + 0.3 * 0.4 ___________ 1.04
- 22. 0.64 0.75 0.4 Red Seeds pattern Top leaves 0.5 0.8 0.3 0.5 * 0.64 + 0.8 * 0.75 + 0.3 * 0.4 ___________ 1.04 + 0.7 ___________ Input
- 23. 0.64 0.75 0.4 Red Seeds pattern Top leaves 1.74 0.5 0.8 0.3 0.5 * 0.64 + 0.8 * 0.75 + 0.3 * 0.4 ___________ 1.04 + 0.7 ___________ 1.74 Input
- 26. 0.64 0.75 0.4 Red Seeds pattern Top leaves 1.02 1.74 0.97 Output Strawberry Not Strawberry Input
- 32. Strawberry Not Not StrawberryNot Not Strawberry NotNot 0.5 * 0.64 + 0.8 * 0.75 + 0.3 * 0.4 ___________ 1.04 + 0.7 ___________ 1.74
- 33. Training
- 37. Data science
- 38. TensorFlow - Open source - Widely used - Flexible for scale: - 1 or more CPUs / GPUs - desktop, server, mobile device
- 39. Strawberry
- 40. Strawberry
- 41. TensorFlow Mobile - Speech Recognition - Image Recognition - Object Localization - Gesture Recognition - Translation - Text Classification - Voice Synthesis
- 42. Lightweight Fast Cross platform
- 46. 1. Add assets
- 48. labels.txt strawberry orange lemon fig pineapple banana jackfruit custard apple pomegranate hay carbonara chocolate sauce dough meat loaf pizza
- 49. 2. Add TensorFlow Lite
- 50. repositories { maven { url 'https://google.bintray.com/tensorflow' } } dependencies { // ... implementation 'org.tensorflow:tensorflow-lite:+' } build.gradle
- 51. android { aaptOptions { noCompress "tflite" noCompress "lite" } } build.gradle
- 53. Image Classifier
- 54. ImageClassifier.java tflite = new Interpreter();
- 55. ImageClassifier.java model = loadModelFile(); tflite = new Interpreter(model);
- 56. MappedByteBuffer loadModelFile() { AssetFileDescriptor descriptor= getAssets().openFd(MODEL_PATH); }
- 57. MappedByteBuffer loadModelFile() { AssetFileDescriptor descriptor= getAssets().openFd(MODEL_PATH); FileInputStream inputStream = new FileInputStream(descriptor.getFileDescriptor()); FileChannel channel = inputStream.getChannel(); }
- 58. MappedByteBuffer loadModelFile() { AssetFileDescriptor descriptor= getAssets().openFd(MODEL_PATH); FileInputStream inputStream = new FileInputStream(descriptor.getFileDescriptor()); FileChannel channel = inputStream.getChannel(); long start = descriptor.getStartOffset(); long length = descriptor.getDeclaredLength(); return channel.map(FileChannel.MapMode.READ_ONLY, start, length); }
- 60. ImageClassifier.java model = loadModelFile(); tflite = new Interpreter(model); labelList = loadLabelList();
- 61. labels.txt strawberry orange lemon fig pineapple banana jackfruit custard apple pomegranate hay carbonara chocolate sauce dough meat loaf pizza
- 62. List<String> loadLabelList() throws IOException { ` InputStreamReader inputStream = new InputStreamReader(getAssets().open(LABEL_PATH)); }
- 63. List<String> loadLabelList() throws IOException { ` InputStreamReader inputStream = new InputStreamReader(getAssets().open(LABEL_PATH)); BufferedReader reader = new BufferedReader(inputStream); }
- 64. List<String> loadLabelList() throws IOException { ` InputStreamReader inputStream = new InputStreamReader(getAssets().open(LABEL_PATH)); BufferedReader reader = new BufferedReader(inputStream); List<String> labelList = new ArrayList<String>(); String line; while ((line = reader.readLine()) != null) { labelList.add(line); } }
- 65. List<String> loadLabelList() throws IOException { ` InputStreamReader inputStream = new InputStreamReader(getAssets().open(LABEL_PATH)); BufferedReader reader = new BufferedReader(inputStream); List<String> labelList = new ArrayList<String>(); String line; while ((line = reader.readLine()) != null) { labelList.add(line); } reader.close(); return labelList; }
- 66. Image Classifier [ [0..6] , [ 0.1 ] , ...] [strawberry, apple, ...] probArray labels.txt
- 67. probArray = { [0.7], [0.3], [0], [0], } labelList = { strawberry, apple, pineapple, banana } 0.3
- 68. ImageClassifier.java model = loadModelFile(); tflite = new Interpreter(model); labelList = loadLabelList(); probArray = new float[1][labelList.size()];
- 69. Image Classifier [......] [ [0..6] , [ 0.1 ] , ...] [strawberry, apple, ...] ByteBuffer probArray labels.txt
- 70. ImageClassifier.java model = loadModelFile(); tflite = new Interpreter(model); labelList = loadLabelList(); probArray = new float[1][labelList.size()]; imgData = ByteBuffer.allocateDirect( DIM_IMG_SIZE_X * DIM_IMG_SIZE_Y * DIM_PIXEL_SIZE); imgData.order(ByteOrder.nativeOrder());
- 71. 4. Run the model / classify
- 72. classifier .classify(bitmap) Image Classifier [......] [ [0..6] , [ 0.1 ] , ...] [strawberry, apple, ...] ByteBuffer probArray labels.txt
- 73. ImageClassifier.java String classify(Bitmap bitmap) { convertBitmapToByteBuffer(bitmap); }
- 74. void convertBitmapToByteBuffer(Bitmap bitmap) { //... bitmap.getPixels(intValues, 0, bitmap.getWidth(), 0, 0,bitmap.getWidth(), bitmap.getHeight()); } }
- 75. void convertBitmapToByteBuffer(Bitmap bitmap) { //... bitmap.getPixels(intValues, 0, bitmap.getWidth(), 0, 0,bitmap.getWidth(), bitmap.getHeight()); int pixel = 0; for (int i = 0; i < DIM_IMG_SIZE_X; ++i) { for (int j = 0; j < DIM_IMG_SIZE_Y; ++j) { final int val = intValues[pixel++]; imgData.put((byte) ((val >> 16) & 0xFF)); imgData.put((byte) ((val >> 8) & 0xFF)); imgData.put((byte) (val & 0xFF)); } } }
- 76. ImageClassifier.java String classify(Bitmap bitmap) { convertBitmapToByteBuffer(bitmap); tflite.run(imgData, probArray);
- 77. ImageClassifier.java String classify(Bitmap bitmap) { convertBitmapToByteBuffer(bitmap); tflite.run(imgData, probArray); String textToShow = getTopLabels(); return textToShow; }
- 78. private String getTopKLabels() { for (int i = 0; i < labelList.size(); ++i) { sortedLabels.add( new AbstractMap.SimpleEntry<>(labelList.get(i), (labelProbArray[0][i] & 0xff) / 255.0f)); if (sortedLabels.size() > RESULTS_TO_SHOW) { sortedLabels.poll(); } } String textToShow = ""; final int size = sortedLabels.size(); for (int i = 0; i < size; ++i) { Map.Entry<String, Float> label = sortedLabels.poll(); textToShow = String.format("n%s: %4.2f", label.getKey(), label.getValue()) + textToShow; } return textToShow; }
- 79. Strawberry - 0.87 Apple - 0.13 Tomato - 0.01
- 81. Links - Tensorflow - https://www.tensorflow.org/ - Tensorflow lite - https://www.tensorflow.org/mobile/tflite/ - Codes labs - codelabs.developers.google.com/codelabs/tensorflow-for-poets-2-tflite/ - Google’s Machine Learning Crash Course - developers.google.com/machine-learning/crash-course/ - [Dr. Joe Dispenza]
- 82. Thank you! Keep in touch! Britt Barak @brittBarak
- 83. BETA
- 84. Key capabilities ● Production-ready for common use cases ● On-device or in the cloud ● Deploy custom models
- 87. 1. Add the sdk Quickly include the SDK using Gradle or CocoaPods.
- 88. 2. Prepare input data For example, if you're using a vision feature, capture an image from the camera and generate the necessary metadata such as image rotation, or prompt the user to select a photo from their gallery.
- 89. 3. Apply the ML model to your data By applying the ML model to your data, you generate insights such as the emotional state of detected faces or the objects and concepts that were recognized in the image, depending on the feature you used. Use these insights to power features in your app like photo embellishment, automatic metadata generation, or whatever else you can imagine.
- 90. Ready to use models
- 91. Ready to use models: 1. Text Recognition
- 92. Ready to use models: 2. Face Detection
- 93. Ready to use models: 3. Barcode Scanning
- 94. Corners (87,87) (612,87) (612,612) (87,612) Raw value WIFI:S:SB1Guest;P:12345;T:WEP;; WiFi information
- 95. Ready to use models: 4. Image Labeling