Using CoreML to Push the Limits of your App

Core MLandyourApp
Howto use Core MLto pushthe limits
Mugunth Kumar, Itty Bitty Apps, Melbourne

Agenda
This talk explains how aspiring app developers/
companies can take full advantage of Core ML to push
their apps to the next level.

Agenda
I'll introduce you to
- The fundamentals of Core ML
- What is possible with machine learning in iOS 11
- What is not possible with machine learning in iOS
11

Agenda
I'll introduce you to
- The fundamentals of Core ML
- What is possible with machine learning in iOS 11
- What is not possible with machine learning in iOS 11
Lastly, What could potentially be possible in a future
release of iOS (12 and above )

Agenda

Agenda
» What is Core ML

Agenda
» What is Core ML
» Machine Learning in iOS 11
» Vision
» Natural Language Processing

Agenda
» What is Core ML
» Vision
» Features not yet available
» Retraining models (iOS 12 )

Agenda
» What is Core ML
» Vision
» Features not yet available
» Retraining models (iOS 12 )
» Questions

Core ML
What is Core ML

Core ML
What is Core ML
» Core ML is a framework newly introduced in iOS 11
SDK

Core ML
What is Core ML
SDK
» Once considered complicated

Core ML
What is Core ML
SDK
» Once considered CPU intensive

Core ML
What is Core ML
SDK
» Once considered impossible to be done on the
client

Core ML
What is Core ML
SDK
» Once considered impossible to be done on the
client
» Made easy in iOS 11
!

Featuresavailabletodayin iOS 11

» Vision
» Object Tagging and Recognition

» Vision
» Object Tagging and Recognition
» Sentiment Analysis
» Language Tagging
» Parts of Speech Analysis

Thistalk
This talk is exclusively on NLP and text processing
for two reasons.

Thistalk
for two reasons.
» Plenty of libraries and blogs already exist on
vision and Object Tagging.

Thistalk
for two reasons.
» Vision tagging model ﬁles are large (30MB to 200+
MB) and the "side" beneﬁt you get using Core ML
gets defeated.

Thistalk
for two reasons.
» Vision tagging model files are large (30MB to 200+
MB) and the "side" benefit you get using Core ML
gets defeated.
» As of iOS 11, you can't on the fly download a Core
ML model on the fly like on-demand resource.

Thistalk
With Core ML and NLP, your app could do

Thistalk

Thistalk
» Parts of Speech Analysis/Language Tagging

Thistalk
» Parts of Speech Analysis/Language Tagging
» Combining Both

Who can be beneﬁtted bySentimentAnalysis

» Apps that show reviews

» Social media account classiﬁers

» Social media account classiﬁers
» Pretty much any app that deals with lots of text
from e-book readers to magazine apps

Getting Started
You need a Core ML Model. You can download from
Or you can convert using coremltools

Getting Started
» https://developer.apple.com/machine-learning/

Getting Started
» https://github.com/likedan/Awesome-CoreML-Models

Getting Started
» https://github.com/likedan/Awesome-CoreML-Models
» https://pypi.python.org/pypi/coremltools

SentimentAnalysis Demo

» For the demo, I'll be using one such model from
the Awesome-CoreML-Models repo.

» Speciﬁcally SentimentPolarity.mlmodel by Vadym
Markov.

» Speciﬁcally SentimentPolarity.mlmodel by Vadym
Markov.
» It's a very light weight model weighing in at
275Kb.

The Code!
func predictSentiment(from text: String) -> Sentiment {
do {
var inputFeatures = features(from: text)
guard inputFeatures.count > 1 else { throw Error.featuresMissing }
let output = try model.prediction(input: inputFeatures)
let positiveProb = output.classProbability["Pos"]!
let negativeProb = output.classProbability["Neg"]!
return Sentiment(positiveProbablity: positiveProb, negativeProbablity: negativeProb)
} catch {
preconditionFailure("Error occurred. Model ﬁle might be corrupted.")
}
}

More code!
func features(from text: String) -> [String: Double] {
var wordCounts = [String: Double]()
tagger.string = text
let range = NSRange(location: 0, length: text.utf16.count)
tagger.enumerateTags(in: range, scheme: .nameType, options: options) { _, tokenRange, _, _ in
let token = (text as NSString).substring(with: tokenRange).lowercased()
guard token.count >= 2 else { return }
if let value = wordCounts[token] {
wordCounts[token] = value + 1.0
} else {
wordCounts[token] = 1.0
}
}
return wordCounts
}

Performance

Performance
» Reasonably fast.

Performance
» Reasonably fast.
» With more text, performance can take a hit

Performance
» Reasonably fast.
» With more text, performance can take a hit
» As long as you avoid the main thread, you should
be good to go

Parts ofSpeechAnalysis

» Parts of speech analysis (and Language Tagging)
was introduced since iOS 5.

» All those calendar apps you saw were using this
feature to convert your natural language input
into a calendar or a reminder event.

» You don't need a model ﬁle to do this. Everything
you need is built into the SDK.

» You don't need a model ﬁle to do this. Everything
you need is built into the SDK.
» So why is this section in this talk?

» In iOS 11, Parts of Speech Analysis was completely
reimplemented on top of Core ML framework

» Language Tagging wasn't very accurate with some
Eastern European and Asian languages.

» Language Tagging wasn't very accurate with some
Eastern European and Asian languages.
» This is mostly ﬁxed with Core ML in iOS 11

Lets getour hands dirty!
func tags(from text: String) -> [[String: String]] {
var tagsToReturn = [[String: String]]()
tagger.string = text
let range = NSRange(location: 0, length: text.utf16.count)
// Tokenize and count the sentence
tagger.enumerateTags(in: range, scheme: .lexicalClass, options: options)
{ tag, tokenRange, _, _ in
let token = (text as NSString).substring(with: tokenRange).lowercased()
let tag = (tag?.rawValue ?? "")
tagsToReturn.append([token : tag])
}
return tagsToReturn
}

Parts ofSpeechAnalysis Demo

Mixingthemallup
There are plenty of research articles that argues
sentiment analysis is better done by emphasising
adverbs and adjectives.
Words like beautiful, useless, hopeless, gigantic
convey more "sentiment" information compared to
objects in a sentence like "Australia", "iPhone" or
subjects like "I", "You".
Ex: I can clearly say, Australia is one of the most
beautiful countries on the planet.

Lets getour hands dirty!
func featuresUsingTaggingAlgorithm(from text: String) -> [String: Double] {
let parsedTags = tags(from: text)
var wordCounts = [String: Double]()
for tag in parsedTags {
let (token, tag) = (tag.keys.ﬁrst!, tag.values.ﬁrst!)
var valueToAdd = Double(1.0)
if tag == "Adverb" {
valueToAdd = Double(30.0)
}
if tag == "Adjective" {
valueToAdd = Double(35.0)
}
if let value = wordCounts[token] {
wordCounts[token] = value + valueToAdd
} else {
wordCounts[token] = valueToAdd
}
}
return wordCounts
}

MIX UPALGORITHM DEMO

Features notyetavailable

» Retraining models is not possible (iOS 12 )
» Face Recognition can't be performed as of now
» Face Recognition ﬁrst involves detection of
faces (available since iOS 6)
» Pass these as training data to a classiﬁer
» Face Recognition is however available for the
native Photos app. So it is only a matter of
time.

SARCASM DETECTION!

Whatyou should knowbynow

» Apple has very elegantly abstracted out the inner
workings that, most of Core ML is fairly straight
forward (codewise: under 10 lines) if you know the
foundations.

foundations.
» Understanding the foundations will let you apply
Core ML in your app easily.

foundations.
» Understanding the foundations will let you apply
Core ML in your app easily.
» Using CoreML innovatively will help make your app
stand out from the crowd.

That's ME!
I'm a

That's ME!
I'm a
» Developer

That's ME!
I'm a
» Developer
» Author of iOS Programming
Pushing the Limits

That's ME!
I'm a
» Developer
» Author of iOS Programming
Pushing the Limits
» Currently baking some cool
apps @ Itty Bitty Apps,
Melbourne

Questions?

Using CoreML to Push the Limits of your App

More Related Content

What's hot (11)

Similar to Using CoreML to Push the Limits of your App (20)

More from Mugunth Kumar (6)

Recently uploaded (20)

Using CoreML to Push the Limits of your App