Data Science Weekend 2017. CleverDATA. Text mining of beauty blogs: о чем говорят женщины?

Text mining of Beauty Blogs:
Text mining of Beauty Blogs:
О чем говорят женщины?
Артем Просветов
Data Scientist, CleverDATA

cleverdata.ru | info@cleverdata.ru
Raw blog data
Raw data: 98,496 pages in format of ~ 1,000,000 files.
Ready for analysis: 58,719 English pages (59.6%)
40.4% data: empty pages and pages with errors, not English pages
(23,461), photo/video pages without text (2,315), articles from
techcrunch.com (3,402)

From 60k of pages → ~2000 authors.
Pages → Authors

Mean blog post size (in words)
One can distinguish 2 populations
of bloggers:
• twitter style' authors with short
posts (~20%)
• full-length bloggers with 200-
500 mean words per post
(~80%)

Used APIs and services:
- Sentity (https://sentity.io/)
- Twinword (https://www.twinword.com/)
- Textualinsights (http://www.textualinsights.com/)
- VivekN (https://github.com/vivekn/sentiment-web)
Sentiment analysis

Sentiment analysis
• - the resulting sentiment rate is based
on 4 independent rate systems.
• - the majority of the blogs have positive
emotion rate.
• - the mean sentiment rate is «positive
warm» 0.72.
• - all this results are intuitively consistent
and are in a good agreement with
manual tests

We used a few traffic rank systems:
Estimation of blog efficiency
• Alexa Rank, that basically audits and makes public the frequency of
visits on various Web sites.
• Yandex Thematic Citation Index (TIC), that determines the
“credibility” of Internet resources based on a qualitative assessment
of links to other sites.
• Google Page Rank, that works by counting the number and quality
of links to blog to determine a rough estimate of how important the
website is.

Content relevance rate is based on fuzzy string matching:
- Every company product name was string matched with all amount of blogs.
- String matching is based on Levinstein's metric.
- Pages with 90% matching rate were marked up.
- Tests with direct brand name matching showed that we get about 90-100%
accuracy on each product name deppends on words in title.
- The result relevance rate for each author is summed from all marks of
his/hers pages.
Relevance Rate

Levenshtein distance is a string metric for measuring the difference between
two sequences.
Informally, the Levenshtein distance between two words is the minimum
number of single-character edits (i.e. insertions, deletions or substitutions)
required to change one word into the other.
Levinshtein distance between 'beer' and 'bread' is 44/100
Levenshtein distance

The most active authors
write with sentiment
rate in short range:
0.74 +/- 0.03
Sentiment rate
Blogsize(pages)
Sentiments vs Blog size

The most discussed
blogs have middle-
size authors.
Log(Blog size)
Meandiscussion
Discussion vs Blog size

Again, 2 kinds of bloggers:
- 'twitter style' authors
with short posts
- full-length bloggers
Log(mean words per page)
Log(Blogsize)
Words vs Pages

f you want to make a big
discussion, you should
praise something.
All highly discussed
authors are sentiment
positive (>=0.4)
Sentiment rate
Meandiscussion
Discussion vs Sentiments

We use Klout service to rank authors
according to online social influence.
Klout measures the size of a user's
social media network and correlates the
content created to measure how other
users interact with that content.
- the median Klout score is 40.1
Using of Klout score for bloggers

One can distinguish a population
of beginner bloggers with low
Klout score, that have tendency
to amplification of sentiments.
Sentiment rate
Kloutscore
Sentiments vs Klout score

• Amount of blog pages
• Mean discussion size
• AlexaRank + YandexTIC + Google PageRank
• Relevance rate
• Sentiment rate
• Klout score
Final Author Rating is based on

4 independent sentiment
rating systems are combined
Alexa Rank
Yandex Thematic Citation Index
Google PageRank
list of most PR effective authors
Pragmatic statistical information
key recommendations for blogger
resulting sentiment rate is
fully consistent with tests
Blog
efficiency
rating
Blog
relevance
rating
Sentiment
analysis
Make your data clever
Based on fuzzy string
matching
Blog rating in
accordance to
mentions of company
products in text

Name Url Sentiment Pages Mean
Comments
Hayley Carr http://www.londonbeautyqueen.com 0.71 229 10.9
Luzanne http://pinkpeonies.co.za 0.77 66 68.3
Allison http://www.neversaydiebeauty.com 0.70 182 42.9
Mica Kelly, Beth,
Jessica Diner
http://blog.birchbox.co.uk 0.74 196 0.26
Poonam http://beautyandmakeupmatters.com 0.78 142 4.3
Silvie http://mysillylittlegang.com 0.74 571 0.64
TOP Rated Authors

Testing the result
Hayley Carr (Top Rated Author):
“BlaBlaBla is definitely a brand to be reckoned with... All of the
BlaBlaBla products have multiple purposes, as well as smelling
and feeling fabulous; the packaging is clean and fresh whilst
still looking great in your bathroom, as well as having unique
application methods that only aid the product performance...
It's definitely worth checking out this growing brand, before it
starts taking over the world. “

Authors ←→ Products

In order to associate a blogger
with a product we must:
• Find products for promotion
• Find main topics of each blogger
• Match topics of each blogger with product names
• Find best combinations of blogger and product

Finding the most perspective
for promotion products

Let's build document-term
matrix, where each row is a
document, each term is a
column and a color intensity
indicates that a term appears in
a document at least once.
We can use TF-IDF method
to get document-term matrix.
Finding topics:
the document-term matrix

Finding topics: TF - IDF
• Term frequency TF(t,d) is the number of times that term t
occurs in document d.
• The inverse document frequency (IDF) is a measure of how
much information the word provides, that is, whether the
term is common or rare across all documents.
• Term frequency–inverse document frequency, is a numerical
statistic that is intended to reflect how important a word is
to a document in a collection or corpus.

• NMF is a variant of Matrix
Factorization where we start
with a matrix D with document-
term matrix, and constrain the
elements of W and T to be non-
negative.
• Lets us interpret each row of the
T matrix as a topic.
Topic extraction: NMF

• For each author we build document-term matrix.
• For each document-term matrix we perform matrix
factorization and find main topics
• For each product we match product name with
main topics of author and find the rate of intensity.
• If author have exact product name in one of
his/hers titles, we set the rate of intensity to 0 (the
author has already made review of the the
product).
Topic extraction

Thus for each pair of author-product we find rate of intensity and we can
visualize it in form of heatmap where products are sorted by mean rate of
intensity and authors are sorted by author rating:
Note: the most rated authors are highly intensive on matrix
The intensity matrix

Next we extract the most resonance peaks from product-author matrix of intensity.
After each peak extraction the column with a peak is dropped, so for each author
we get only one product.
We need to build recommendations only for 4 products and we can select 40
best rated authors for this task.
The intensity matrix

In order to associate a blogger
with a product we must:
• Find products for promotion
• Find main topics of each blogger
• Match topics of each blogger with product names
• Find best combinations of blogger and product
• Profit!

BlaBlaBla Body Oil Allison http://www.neversaydiebeauty.com
BlaBlaBla Wrinkle
Repair
Cindy Batchelor http://mystylespot.net
BlaBlaBla Face Serum Marie Papachatzis http://iamthemakeupjunkie.blogspot.ru
BlaBlaBla Face Oil Emily - Style Lobster http://stylelobster.com
The resulting associations

Data Science Weekend 2017. CleverDATA. Text mining of beauty blogs: о чем говорят женщины?

Data Science Weekend 2017. CleverDATA. Text mining of beauty blogs: о чем говорят женщины?

More Related Content

Viewers also liked (13)

More from Newprolab (8)

Recently uploaded (20)

Data Science Weekend 2017. CleverDATA. Text mining of beauty blogs: о чем говорят женщины?