Use NLP to Solve Business Problems
2 likes657 views
The document discusses how Annie Flippo used natural language processing (NLP) techniques to solve AwesomenessTV's business problem of managing similar video assets across different platforms. NLP was used to identify similar videos by processing video titles and descriptions, vectorizing them, and measuring cosine similarity. Specifically, it discusses data processing techniques like tokenization, stemming, lemmatization and vectorization to transform text into numeric vectors that can then be compared to identify similar videos. It also describes how bi-grams were later used to improve results by capturing word pairs instead of individual words.
![Tokenize
Count distinct words from a corpus
“The quick brown fox jumped over the lazy dogs”
becomes
[‘the’, ‘quick’, ‘brown’, ‘fox’, ‘jump’, ‘over’, ‘lazy’, ‘dog’]](https://image.slidesharecdn.com/use-nlp-to-solve-business-problems-161024000106/85/Use-NLP-to-Solve-Business-Problems-11-320.jpg)
![Vectorize
Count occurrences from distinct word
vector.
“The quick brown fox jumped over the lazy dogs”
Tokenized to:
[‘the’, ‘quick’, ‘brown’, ‘fox’, ‘jump’, ‘over’, ‘lazy’, ‘dog’]
Vectorized to:
[2, 1, 1, 1, 1, 1, 1, 1]](https://image.slidesharecdn.com/use-nlp-to-solve-business-problems-161024000106/85/Use-NLP-to-Solve-Business-Problems-12-320.jpg)
![Bag-of-Words Comparison
Doc 1: “The quick brown fox jumped over the lazy dogs”
Doc 2: “The quick fox ran away from the dog”
After processing, the corpus attribute vector is:
[‘quick’, ‘brown’, ‘fox’, ‘jump’, ‘over’, ‘lazy’,
‘dog’, ‘run’, ‘away’, ‘from’]
Two documents vectorize to:
Doc 1: [1, 1, 1, 1, 1, 1, 1, 0, 0, 0]
Doc 2: [1, 0, 1, 0, 0, 0, 1, 1, 1, 1]
Sentences are transformed into numeric vectors!
brown
fox
lazy
run
quick
dog](https://image.slidesharecdn.com/use-nlp-to-solve-business-problems-161024000106/85/Use-NLP-to-Solve-Business-Problems-13-320.jpg)
![Bi-Gram Comparison
Due to the same words used across our videos, the bag-of-words
similarity resulted high false positive matches.
The solution is to use a Bi-Gram algorithm where 2 consecutive
words are extracted as one feature:
“The quick brown fox jumped over the lazy dogs”
becomes:
[‘the quick’, ‘quick brown’, ‘brown fox’, ‘fox jump’,
‘jump over’, ‘over the’, ‘the lazy’, ‘lazy dog’]](https://image.slidesharecdn.com/use-nlp-to-solve-business-problems-161024000106/85/Use-NLP-to-Solve-Business-Problems-15-320.jpg)
Use NLP to Solve Business Problems
- 6. Why use NLP? Top goals for Natural Language Processing are: 1. Document Similarity (search engine query) 2. Topic Modeling (Twitter/Blog Analysis) 3. Sentiment Analysis (movie or restaurant reviews)
- 7. Data Processing Why perform text processing? • To rid of messiness of free-from text • To group words with the same meaning • Convert text to numeric features • Model on equivalent numeric features
- 8. Data Processing Titles and descriptions get scrubbed • Remove punctuation, non-ascii, carriage returns • Remove stop words (i.e. it, this, and, that) • Stemming • Lemmatize • Tokenize • Vectorize
- 13. Bag-of-Words Comparison Doc 1: “The quick brown fox jumped over the lazy dogs” Doc 2: “The quick fox ran away from the dog” After processing, the corpus attribute vector is: [‘quick’, ‘brown’, ‘fox’, ‘jump’, ‘over’, ‘lazy’, ‘dog’, ‘run’, ‘away’, ‘from’] Two documents vectorize to: Doc 1: [1, 1, 1, 1, 1, 1, 1, 0, 0, 0] Doc 2: [1, 0, 1, 0, 0, 0, 1, 1, 1, 1] Sentences are transformed into numeric vectors! brown fox lazy run quick dog
- 16. Limitations Certain phrases such as “Behind the scenes” are found frequently. This creates an artificially high similarity score even if the videos are dissimilar. Possible solutions: • Perform more custom data scrubbing • Double-check by matching duration of videos • Have the matches verified by a human