DevFest SG 2024 - What’s new in On-device Generative AI

What’s new in
on-device
Generative AI
Singapore
Hassan Abid
GDE Android

About myself
GDE
- Google Developers Expert for
Android Since 2016
- CTO at BELIVE TECHNOLOGY
- Mentor for AI First Singapore
startup program
- Mentor for Solution Challenge
2023 & 2024
- Speaker at GDG Events across
Asia Pacific (GDG Singapore
since 2018)

Gen AI on Android : Use Cases
On-Device AI
Gemini Cloud Inference for Android
Gemini in Android Studio
Summary
Agenda

1) GenAI on Android:
use cases
Singapore

Smartphone’s smaller screens constraints:
● Information consumption is harder than
on other platforms
● Text input via software keyboard can be
tedious
Mobile constraints

● Text & audio summarization
● Translation: Automatic translation of podcasts into
50 languages.
● Document Q&A
● Classification and categorization of media content
GenAI for content consumption

● Smart reply
● Text rewrite
● Smart compose
GenAI for input assistance

The camera you always have with
you can be used as input to
augment prompts.
Camera as input

Vision for GenAI on mobile
Integrated in apps as
the central feature
Will enable new
core user journeys
in apps
Will augment every
user interaction
and unlock in apps
personalization and
intelligence
Today Soon Finally
e.g: Chatbot apps, AI-generated profile
apps
e.g: Google Photo Magic Editor, GMail
email summarization, Messages Smart
rewrite
AI involved with potentially any user
interaction

Benefits of on-device execution
Offline
availability
Local
processing
No inference
cost
Potentially
reduced latency

There are several options for on-device…
MLKit
Mobile SDK that simplifies
adding machine learning to
Android apps.
Ready-to-use APIs for tasks
like text recognition and face
detection
developers.google.com/
ml-kit
MediaPipe
Open-source framework that
simplifies building and
deploying machine learning
pipelines.
Ideal for processing
time-series data like video
and audio on various
platforms.
https://ai.google.dev/
edge/mediapipe
Gemini Nano
Cutting-edge language
models developed by
Google
Specifically designed to
run efficiently on mobile
devices
d.android.com/ai/aicore
LiteRT
(formerly TFLite)
On-device ML model
runtime.
Ideal for ML models
performing classification,
detection or segmentation
tasks.
ai.google.dev/edge/
litert

On-device GenAI Models
Fine-tuning may be critical in
order to get good accuracy
Not good at all use-cases
Implications
Lower parameter size
(compared to cloud models)
Smaller context window
Less generalized model
Limitations

Android’s foundation model of choice
for building on-device GenAI applications
Integrated to the Android OS via AICore in Android 14+
Google AI Edge SDK provides access to APIs
Available on:
- Pixel 8 series
- Pixel 9 series
- Samsung S24 series
- Samsung Flip 6 and Fold 6
- Motorola Razr Plus 2024
Gemini Nano
Nano
with
Multimodality

Customize model to perform specific tasks
LoRA allows for app-specific optimizations while
maintaining a single copy of model
LoRA powered all production applications
of Gemini Nano so far
Fine-tuning
AI Core
LoRA
Gemini Nano
Safety features

A Closer Look: Pixel Screenshots
Leverages Gemini Nano w/ Multimodality to:
● Save - Generates a title and summary of
screenshots & offers suggested actions based
on the content (eg. add details to Calendar)
● Organize - create custom collections within
the Pixel Screenshots app to easily group
screenshots around different ideas or topics.
● Recall - Ask for information you want to find in
a conversational way – like “Package tracking
number?” – and get a conversational answer in
return – “Your tracking number is 1234” in
addition to including the original image it came
from.

● Early Access Preview / Experimental Access announced in 2023/2024.
● Check more here
https://developer.android.com/ai/gemini-nano/experimental
Gemini Nano developer access

DevFest SG 2024 - What’s new in On-device Generative AI

● Get the sample app at https://github.com/android/ai-samples
● Install Prerequisites
○ Join the aicore-experimental Google group
○ Opt in to the Android AICore testing program
○ Update APK’s for AI Core (Beta) & Private Compute Services
Get started with Gemini Nano

build.gradle file
Create GenerateConfig object

Run the inference and pass in your prompt

Run Gemma-2 2B model on device
1. The LLM Inference API lets
you run large language
models (LLMs) completely
on-device
Reference link
Sample Android App : [link]
2. Features :
a. Text-to-text generation
b. LLM selection
c. LoRA Support

1. Use conversion scripts to format the model to be compatible
with MediaPipe.
a. E.g : Phi-2, Falcon, or StableLM
b. Non-Kaggle models
2. The model conversion process requires the MediaPipe PyPI
package
Convert Models to Mediapipe
Format [link]

Download LiteRT(TFLite) models from
Kaggle / Hugging Face [link]

1. Mediapipe LLM inference API can be configured to support LoRA for
LLM models
2. Convert the model weights into a tensorFlow lite Flatbuffer using
MediaPipe python package (ConversionConfig)
3. The Web, Android and iOS LLM Inference API are updated to support
LoRA model inference.
a. .setLoraPath('<path to LoRA model>')
Support fine-tuned LoRA
models

1. Fine tuned model on Hugging Face :
https://huggingface.co/monsterapi/gemma-2b-lora-maths-orca-200k
2. Data Set used to fine tune model :
https://huggingface.co/datasets/microsoft/orca-math-word-problems-20
0k/viewer/default/train?p=2000&row=200032
3. Python Notebook to run model on Colab Enterprise :
https://github.com/hassanabidpk/mediapipe_custom_llm_inference/blob/main/gemma-2
b-lora-maths-orca-200k.ipynb
Fine Tuned model example

Example : Gemma-2 2B model
without fine tuning for
Maths problems
Q: There were 12 roses, 2 orchids, and 5 lilies in
the vase. Jessica cut some more roses, orchids,
and lilies from her flower garden. There are now
10 roses, 21 orchids, and 15 lilies in the vase. How
many roses, orchids, and lilies did she cut from
her garden?
Response from basic model : -27 Flowers

Example : Gemma 2B model with fine tuning for
Maths problems [link]
There were 12 roses, 2 orchids, and 5 lilies in the
vase. Jessica cut some more roses, orchids, and
lilies from her flower garden. There are now 10
roses, 21 orchids, and 15 lilies in the vase. How
many roses, orchids, and lilies did she cut from
her garden?
Response from fine-tuned model : -2 roses, 19
orchids, and 10 lilies.

Example : Gemma 2B model with fine tuning for
Maths problem [link]

Singapore
Fine tuning can help
improve model responses
for specific use-cases

Example : Results from Gemini 1.5 Pro
Similar to fine tuned model (But provided negative value)

● Google's high-performance runtime for on-device
AI.
● Convert and run TensorFlow, PyTorch, and JAX
models to the TFLite format
● Multiplatform Support (iOS, Android, Embedded
Linux etc.)
● More details here : https://ai.google.dev/edge/litert
LiteRT (TensorFlow Lite)

3) Gemini Cloud
inference for
Android
Singapore

Gemini 1.5 Pro and Gemini 1.5 Flash are
ideal for scaling across a wide range of tasks:
● Multimodal capabilities (text, image,
video, audio)
● Adhere to specific output format (JSON,
CSV, Markdown, YAML, etc…)
Vertex AI for Firebase

Vertex AI for Firebase
Gemini 1.5 Flash
Gemini 1.5 Pro
Vertex AI
your
Android
app
Android SDK
Easily access Gemini Cloud models to your Android app:
Cloud
Android
device
>10 lines of code
integration…

Context window
What does 1M tokens get you?
Text
750 000 words
(4 Harry Potter books)
Audio
11 hours
(~1.5 Harry Potter audiobooks)
Video
1 hour
Gemini 1.5 Pro: 2M tokens Gemini 1.5 Flash: 1M tokens
The amount of text data the model with consider at one time when creating the response.

Function calling
Define custom Kotlin functions (or tools) and provide them to the
model to augment its capabilities.
Weather forecast
Order history
Currency conversion
etc…

Go to the Firebase documentation to
learn more about VertexAI in
Firebase.
firebase.google.com/docs/vertex-ai

4) Gemini in
Android Studio
Singapore

Coding Assistant
Ask android questions to an AI
assistance directly in Android
Studio.

Generate Android Code
Multi-line code completions
– hit tab to accept.

Upcoming Features
Coming Soon
Multimodal Image Uploads
● Upload images alongside your Gemini chat
sessions, utilizing Gemini 1.5's multimodal
input support.
● Use cases:
UI generation
"Given this wireframe, implement a
Compose UI using material3"
Refer to Slide no 11 onwards :
https://bit.ly/4bynBzy

d.android.com/ai
To learn more AI and Android:

Singapore
The ideal approach for
building Gen AI apps
combines on-device and
cloud-based AI.

DevFest SG 2024 - What’s new in On-device Generative AI

More Related Content

Similar to DevFest SG 2024 - What’s new in On-device Generative AI (20)

More from Hassan Abid (17)

Recently uploaded (20)

DevFest SG 2024 - What’s new in On-device Generative AI