SlideShare a Scribd company logo
Different types of analysis Sentiment analysis Stopword removal Next meetings
Big Data and Automated Content Analysis
Week 4 – Monday
»Sentiment Analysis«
Damian Trilling
d.c.trilling@uva.nl
@damian0604
www.damiantrilling.net
Afdeling Communicatiewetenschap
Universiteit van Amsterdam
27 February 2017
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Today
1 Different types of analysis
What can we do?
Systematizing analytical approaches
2 Data analysis 1: Sentiment analysis
What is it?
Bag-of-words approaches
Advanced approaches
A sentiment analysis tailored to your needs!
3 NLP-preview: Stopword removal
Natural language processing
A simple algorithm
4 Take-home message, next meetings, & exam
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
What we already can do
with regard to data collection:
• query a (JSON-based) API (GoogleBooks, Twitter)
• handle CSV files
• handle JSON files
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
What we already can do
with regard to data collection:
• query a (JSON-based) API (GoogleBooks, Twitter)
• handle CSV files
• handle JSON files
with regard to analysis:
Not much. We counted some frequencies and calculated some
averages.
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
What can we do?
Data analysis: Overview
What can we do?
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
What can we do?
Data analysis: Overview
What can we do?
What do you think? What are interesting methods to
analyze large data sets (like, e.g., social media data? What
questions can they answer?)
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
What can we do?
What else can we do?
For example
• sentiment analysis
• automated coding with regular expressions
• natural language processing
• supervised and unsupervised machine learning
• network analysis
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
What can we do?
What else can we do?
Or ideally. . .
. . . a combination of these techniques.
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Systematizing analytical approaches
Overview
Systematizing analytical approaches
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Systematizing analytical approaches
Systematizing analytical approaches
Taking the example of Twitter:
Analyzing the structure
• Number of Tweets over time
• singleton/retweet ratio
• Distribution of number of Tweets per user
• Interaction networks
Bruns, A., & Stieglitz, S. (2013). Toward more systematic Twitter analysis: metrics for tweeting activities.
International Journal of Social Research Methodology. doi:10.1080/13645579.2012.756095
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Systematizing analytical approaches
Systematizing analytical approaches
Taking the example of Twitter:
Analyzing the structure
• Number of Tweets over time
• singleton/retweet ratio
• Distribution of number of Tweets per user
• Interaction networks
⇒ Focus on the amount of content and on the question who
interacts with whom, not on what is said
Bruns, A., & Stieglitz, S. (2013). Toward more systematic Twitter analysis: metrics for tweeting activities.
International Journal of Social Research Methodology. doi:10.1080/13645579.2012.756095
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Systematizing analytical approaches
Systematizing analytical approaches
Taking the example of Twitter:
Analyzing the content
• Sentiment analysis
• Word frequencies, searchstrings
• Co-word analysis (⇒frames)
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Systematizing analytical approaches
Systematizing analytical approaches
Taking the example of Twitter:
Analyzing the content
• Sentiment analysis
• Word frequencies, searchstrings
• Co-word analysis (⇒frames)
⇒ Focus on what is said
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Systematizing analytical approaches
Systematizing analytical approaches
⇒ It depends on your reserach question which approach is
more interesting!
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Systematizing analytical approaches
Automated Content Analysis
Methodological approach
deductive inductive
Typical research interests
and content features
Common statistical
procedures
visibility analysis
sentiment analysis
subjectivity analysis
Counting and
Dictionary
Supervised
Machine Learning
Unsupervised
Machine Learning
frames
topics
gender bias
frames
topics
string comparisons
counting
support vector machines
naive Bayes
principal component analysis
cluster analysis
latent dirichlet allocation
semantic network analysis
Boumans, J.W., & Trilling, D. (2016). Taking stock of the toolkit: An overview of relevant automated content
analysis approaches and techniques for digital journalism scholars. Digital Journalism, 4, 1. 8–23.
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
What is it?
Data analysis 1:
Sentiment analysis
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
What is it?
What is sentiment analysis?
Extracting subjective information from texts
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
What is it?
What is sentiment analysis?
Extracting subjective information from texts
• the author’s attitude towards the topic of the text
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
What is it?
What is sentiment analysis?
Extracting subjective information from texts
• the author’s attitude towards the topic of the text
• polarity: negative—positive
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
What is it?
What is sentiment analysis?
Extracting subjective information from texts
• the author’s attitude towards the topic of the text
• polarity: negative—positive
• subjectivity: neutral—subjective *
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
What is it?
What is sentiment analysis?
Extracting subjective information from texts
• the author’s attitude towards the topic of the text
• polarity: negative—positive
• subjectivity: neutral—subjective *
• advanced approaches: different emotions
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
What is it?
What is sentiment analysis?
Extracting subjective information from texts
• the author’s attitude towards the topic of the text
• polarity: negative—positive
• subjectivity: neutral—subjective *
• advanced approaches: different emotions
* Less sophisticated approaches do not see this as a sperate dimension but
simply calculate objectivity = 1 − (negativity + positivity)
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
What is it?
Applications
Who uses it?
• Companies
• especially for Web Analytics
• Social Scientists
• applications in data journalism, politics, . . .
Many references to examples in Mostafa (2013).
⇒ Cases in which you have a huge amount of data or real-time
data and you want to get an idea of the tone.
Mostafa, M. M. (2013). More than words: Social networks’ text mining for consumer brand sentiments. Expert
Systems with Applications, 40(10), 4241– 4251. doi:10.1016/j.eswa.2013.01.019
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
What is it?
Example
1 >>> sentiment("Great service by @NSHighspeed")
2 (0.8, 0.75)
3 >>> sentiment("Bad service by @NSHighspeed")
4 (-0.6166666666666667, 0.6666666666666666)
(polarity, subjectivity) with
−1 ≤ polarity ≤ +1
0 ≤ subjectivity ≤ +1 )
This is the module pattern.nl, available for Python 2 only.
De Smedt, T., & Daelemans W. (2012). Pattern for Python. Journal of Machine Learning Research, 13, 2063-2067.
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Bag-of-words approaches
Data analysis 1: Sentiment analysis
Bag-of-words approaches
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Bag-of-words approaches
Bag-of-words approaches
How does it work?
• We take each word of a text and look if it’s positive or
negative.
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Bag-of-words approaches
Bag-of-words approaches
How does it work?
• We take each word of a text and look if it’s positive or
negative.
• Most simple way: compare it with a list of negative words and
with a list of positive words (That’s what Mostafa (2013) did)
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Bag-of-words approaches
Bag-of-words approaches
How does it work?
• We take each word of a text and look if it’s positive or
negative.
• Most simple way: compare it with a list of negative words and
with a list of positive words (That’s what Mostafa (2013) did)
• More advanced: look up a subjectivity score from a table
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Bag-of-words approaches
Bag-of-words approaches
How does it work?
• We take each word of a text and look if it’s positive or
negative.
• Most simple way: compare it with a list of negative words and
with a list of positive words (That’s what Mostafa (2013) did)
• More advanced: look up a subjectivity score from a table
• e.g., add up the scores and average them.
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Bag-of-words approaches
How to do this
If you were to run an analyis like the one by Mostafa (2013), how
could you do this?
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Bag-of-words approaches
How to do this
(given a string tekst that you want to analyze and two lists of strings with negative
and positive words, lijstpos=["great","fantastic",...,"perfect"] and
lijstneg)
1 sentiment=0
2 for woord in tekst.split():
3 if woord in lijstpos:
4 sentiment=sentiment+1 #same as sentiment+=1
5 elif woord in lijstneg:
6 sentiment=sentiment-1 #same as sentiment-=1
7 print (sentiment)
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Bag-of-words approaches
Do we need to have the lists in our program itself?
No.
You could have them in a separate text file, one per row, and then
read that file directly to a list.
1 poslijst=open("filewithonepositivewordperline.txt").read().splitlines()
2 neglijst=open("filewithonenegativewordperline.txt").read().splitlines()
Big Data and Automated Content Analysis Damian Trilling
BDACA1617s2 - Lecture4
Different types of analysis Sentiment analysis Stopword removal Next meetings
Bag-of-words approaches
More advanced versions
• CSV files or similar tables with weights
• Or some kind of dict?
Big Data and Automated Content Analysis Damian Trilling
BDACA1617s2 - Lecture4
BDACA1617s2 - Lecture4
Different types of analysis Sentiment analysis Stopword removal Next meetings
Bag-of-words approaches
Mustafa 2013: Interpreting the output
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Bag-of-words approaches
Mustafa 2013: Interpreting the output
Your thoughts?
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Bag-of-words approaches
Mustafa 2013: Interpreting the output
Your thoughts?
• each word
counts equally
(1)
• many tweets
contain no
words from the
list. What does
this mean?
• Ways to
improve BOW
approaches?
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Bag-of-words approaches
Bag-of-words approaches
e.g., Schut, L. (2013). Verenigde Staten vs. Verenigd Koningrijk: Een automatische inhoudsanalyse naar
verklarende factoren voor het gebruik van positive campaigning en negative campaigning door vooraanstaande
politici en politieke partijen op Twitter. Bachelor Thesis, Universiteit van Amsterdam.
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Bag-of-words approaches
Bag-of-words approaches
pro
• easy to implement
• easy to modify:
• add or remove words
• make new lists for other languages, other categories (than
positive/negative), . . .
• easy to understand (transparency, reproducability)
e.g., Schut, L. (2013). Verenigde Staten vs. Verenigd Koningrijk: Een automatische inhoudsanalyse naar
verklarende factoren voor het gebruik van positive campaigning en negative campaigning door vooraanstaande
politici en politieke partijen op Twitter. Bachelor Thesis, Universiteit van Amsterdam.
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Bag-of-words approaches
Bag-of-words approaches
con
• simplistic assumptions
• e.g., intensifiers cannot be interpreted ("really" in "really
good" or "really bad")
• or, even more important, negations.
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Advanced approaches
Data analysis 1: Sentiment analysis
Advanced approaches
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Advanced approaches
Improving the BOW approach
Example: The Sentistrenght algorithm
• −5 . . . − 1 and +1 . . . + 5
• spelling correction
• "booster word list" for strengthening/weakening the effect of
the following word
• interpreting repeated letters ("baaaaaad"), CAPITALS and !!!
• idioms
• negation
• ldots
Thelwall, M., Buckley, K., & Paltoglou, G. (2012). Sentiment strength detection for the social Web. Journal of the
American Society for Information Science and Technology, 63(1), 163-173.
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Advanced approaches
Advanced approaches
Take the structure of a text into account
• Try to apply linguistics concepts to identify sentence structure
• can identify negations
• can interpret intensifiers
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Advanced approaches
Example
1 from pattern.nl import sentiment
2 >>> sentiment("Great service by @NSHighspeed")
3 (0.8, 0.75)
4 >>> sentiment("Really")
5 (0.0, 1.0)
6 >>> sentiment("Really Great service by @NSHighspeed")
7 (1.0, 1.0)
(polarity, subjectivity) with
−1 ≤ polarity ≤ +1
0 ≤ subjectivity ≤ +1 )
Unlike in pure bag-of-words approaches, here, the overall sentiment is not just
the sum or the average of its parts!
De Smedt, T., & Daelemans W. (2012). Pattern for Python. Journal of Machine Learning Research, 13, 2063-2067.
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Advanced approaches
Advanced approaches
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Advanced approaches
Advanced approaches
pro
• understand intensifiers or negation
• thus: higher accuracy
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Advanced approaches
Advanced approaches
pro
• understand intensifiers or negation
• thus: higher accuracy
con
• Black box? Or do we understand the algorithm?
• Difficult to adapt to own needs
• really much better results?
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
A sentiment analysis tailored to your needs!
Data analysis 1: Sentiment analysis
A sentiment analysis tailored to your needs!
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
A sentiment analysis tailored to your needs!
A sentiment analysis tailored to your needs!
Identifying suicidal texts
• Bag-of-words-approach with very specific dictionary
• added negation
• added regular expression search for key phrases
• Very specific design requirements: False positives are OK,
false negatives not!
Huang, Y.-P., Goh, T., & Liew, C.L. (2007). Hunting suicide notes in web 2.0 – preliminary findings. Ninth IEEE
International Symposium on Multimedia. Retrieved from
http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4476021
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
A sentiment analysis tailored to your needs!
Already this still relatively simple approach seems to work
satisfactory, but if 106 scientists from 24 competing teams (!)
work on it, they can
Pestian, J.P.; Matykiewicz, P., Linn-Gust, M., South, B., Uzuner, O., Wiebe, J., Cohen, K.B., Hurdle, J., & Brew,
C. (2012). Sentiment analysis of suicide notes: A shared task. Biomedical Informatics Insights, 5(1), p. 3-16.
Retrieved from http://europepmc.org/articles/PMC3299408?pdf=render
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
A sentiment analysis tailored to your needs!
Already this still relatively simple approach seems to work
satisfactory, but if 106 scientists from 24 competing teams (!)
work on it, they can
group suidide notes by these characteristics:
• swear
• family
• friend
• positive emotion
• negative emotion
• anxiety
• anger
• sad
• cognitive process
• biology
• sexual
• ingestion
• religion
• death
Pestian, J.P.; Matykiewicz, P., Linn-Gust, M., South, B., Uzuner, O., Wiebe, J., Cohen, K.B., Hurdle, J., & Brew,
C. (2012). Sentiment analysis of suicide notes: A shared task. Biomedical Informatics Insights, 5(1), p. 3-16.
Retrieved from http://europepmc.org/articles/PMC3299408?pdf=render
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
A sentiment analysis tailored to your needs!
Sidenote
An alternative state-of-the-art approach:
Use supervised machine learning
• Instead of defining rules, hand-code (“annotate”) the
sentiment of some tweets manually and let the computer find
out which words or characters (“features”) predict sentiment
• Then use this model to predict sentiment for other tweets
• Essentially the same like what you know since the second year
of your Bachelor: regression analysis (but now with DV
sentiment and IV’s word occurrences)
• ⇒ week 7
Gonzalez-Bailon, S., & Paltoglou, G. (2015). Signals of public opinion in online communication: A comparison of
methods and data sources. The ANNALS of the American Academy of Political and Social Science, 659(1),
95-–107.
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Natural Language Processing preview:
Stopword removal
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Natural Language Processing preview:
Stopword removal
Why now? — Because the logic of the algorithm is very much
related to the one of our first simple sentiment analysis
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Natural language processing
Stopword removal: What and why?
Why remove stopwords?
• If we want to identify key terms (e.g., by means of a word
count), we are not interested in them
• If we want to calculate document similarity, it might be
inflated
• If we want to make a word co-occurance graph, irrelevant
information will dominate the picture
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
A simple algorithm
Stopword removal: How
1 testo=’He gives her a beer and a cigarette.’
2 testonuovo=""
3 stopwords=[’and’,’the’,’a’,’or’,’he’,’she’,’him’,’her’]
4 for verbo in testo.split():
5 if verbo not in stopwords:
6 testonuovo=testonuovo+verbo+" "
What do we get if we do:
1 print (testonuovo)
Can you explain the algorithm?
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
A simple algorithm
We get:
1 >>> print (testonuovo)
2 ’He gives beer cigarette. ’
Why is "He" still in there?
How can we fix this?
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
A simple algorithm
Stopword removal
1 testo=’He gives her a beer and a cigarette.’
2 testonuovo=""
3 stopwords=[’and’,’the’,’a’,’or’,’he’,’she’,’him’,’her’]
4 for verbo in testo.split():
5 if verbo.lower() not in stopwords:
6 testonuovo=testonuovo+verbo+" "
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Take-home message
Mid-term take-home exam
Next meetings
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Take-home messages
What you should be familiar with:
• You should have completely understood last week’s exercise.
Re-read it if neccessary.
• Approaches to the analysis (e.g., structure vs. content)
• Types of sentiment analysis, application areas, pros and cons
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Mid-term take home exam
Week 5: Wednesday, 8 March, to Sunday, 12 March
• You get the exam on Wednesday at the end of the meeting
• Answers have to be handed in no later than Sunday, 23.59
• 30% of final grade
• 3 questions:
1 Literature question: E.g., different methods (“Explain how. . . is
done”) and/or epistemological or theoretical implications
(“What does this mean for social-scientific research?”)
2 Empirical question (conceptual)
3 Empirical question (actual programming task)
If you fully understood all exercises until now, it shouldn’t be
difficult and won’t take too long. But give yourself a lot of
buffer time!!!
Big Data and Automated Content Analysis Damian Trilling
Different types of analysis Sentiment analysis Stopword removal Next meetings
Next meetings
Wednesday, 1–3
We work together on the examples in Chapter 6. You can bring
your own data (you will probably learn more!), but then already
think about how to write some script to read the data (as we did
last week and as described in Chapter 5).
Big Data and Automated Content Analysis Damian Trilling

More Related Content

What's hot (20)

PDF
Analyzing social media with Python and other tools (1/4)
PDF
Introduction to Text Mining and Visualization with Interactive Web Application
PDF
How Graph Databases used in Police Department?
PPTX
Using the search engine as recommendation engine
PDF
Bytewise approximate matching, searching and clustering
PPTX
The Web of Data: do we actually understand what we built?
PDF
Query Processing with k-Anonymity
PDF
Text Analysis: Latent Topics and Annotated Documents
PDF
Developing A Big Data Search Engine - Where we have gone. Where we are going:...
PDF
Privacy Protectin Models and Defamation caused by k-anonymity
PDF
Sybrandt Thesis Proposal Presentation
PPTX
K anonymity for crowdsourcing database
DOC
Advanced c c++
Analyzing social media with Python and other tools (1/4)
Introduction to Text Mining and Visualization with Interactive Web Application
How Graph Databases used in Police Department?
Using the search engine as recommendation engine
Bytewise approximate matching, searching and clustering
The Web of Data: do we actually understand what we built?
Query Processing with k-Anonymity
Text Analysis: Latent Topics and Annotated Documents
Developing A Big Data Search Engine - Where we have gone. Where we are going:...
Privacy Protectin Models and Defamation caused by k-anonymity
Sybrandt Thesis Proposal Presentation
K anonymity for crowdsourcing database
Advanced c c++
Ad

Viewers also liked (10)

PDF
EWMA 2013 - Ep500 - The Role of the Pharmacist as a Member of a Multidiscipli...
PPT
Water Centers
TXT
PDF
Data Science: Case "Political Communication 1/2"
PDF
InterTech is a Qatar contractor company
PPTX
Proposal skripsi Jurusan Agribisnis Fakultas Pertanian Universitas Jambi 2016
PDF
Ringkasan jurnal deby
PDF
ΣΕΒ: Εβδομαδιαίο Δελτίο
EWMA 2013 - Ep500 - The Role of the Pharmacist as a Member of a Multidiscipli...
Water Centers
Data Science: Case "Political Communication 1/2"
InterTech is a Qatar contractor company
Proposal skripsi Jurusan Agribisnis Fakultas Pertanian Universitas Jambi 2016
Ringkasan jurnal deby
ΣΕΒ: Εβδομαδιαίο Δελτίο
Ad

Similar to BDACA1617s2 - Lecture4 (20)

PPTX
Sentiment analysis using ml
PDF
Kishaloy Haldar and Wenqiang Lei - WESST - Sentiment Analysis of Social Media
PDF
Sentiment Analysis with NVivo 11 Plus
PPTX
A presentation on Sentiment Analysis....
PDF
110917_0900_Karimi.pdf
PPTX
Sentimental Analysis - Naive Bayes Algorithm
PPTX
A review on sentiment analysis and emotion detection.pptx
PPTX
Social Media Sentiments Analysis
PDF
Sentimental analysis
PDF
Sentiment Analysis on Twitter Dataset using R Language
PPTX
Sentiment analysis
PDF
PDF
Detailed Investigation of Text Classification and Clustering of Twitter Data ...
PDF
A Review On Sentiment Analysis And Emotion Detection From Text
PPTX
Presentation on Sentiment Analysis
PDF
SENTIMENT ANALYSIS APPROACH IN NATURAL LANGUAGE PROCESSING FOR DATA EXTRACTION
PDF
IRJET- Real Time Sentiment Analysis of Political Twitter Data using Machi...
PDF
Sentiment Analysis using Machine Learning.pdf
PPTX
Application-Of-NLP-Sentimental-Analysis.pptx
Sentiment analysis using ml
Kishaloy Haldar and Wenqiang Lei - WESST - Sentiment Analysis of Social Media
Sentiment Analysis with NVivo 11 Plus
A presentation on Sentiment Analysis....
110917_0900_Karimi.pdf
Sentimental Analysis - Naive Bayes Algorithm
A review on sentiment analysis and emotion detection.pptx
Social Media Sentiments Analysis
Sentimental analysis
Sentiment Analysis on Twitter Dataset using R Language
Sentiment analysis
Detailed Investigation of Text Classification and Clustering of Twitter Data ...
A Review On Sentiment Analysis And Emotion Detection From Text
Presentation on Sentiment Analysis
SENTIMENT ANALYSIS APPROACH IN NATURAL LANGUAGE PROCESSING FOR DATA EXTRACTION
IRJET- Real Time Sentiment Analysis of Political Twitter Data using Machi...
Sentiment Analysis using Machine Learning.pdf
Application-Of-NLP-Sentimental-Analysis.pptx

More from Department of Communication Science, University of Amsterdam (13)

Recently uploaded (20)

PDF
Insiders guide to clinical Medicine.pdf
PDF
STATICS OF THE RIGID BODIES Hibbelers.pdf
PDF
Black Hat USA 2025 - Micro ICS Summit - ICS/OT Threat Landscape
PDF
Abdominal Access Techniques with Prof. Dr. R K Mishra
PDF
O5-L3 Freight Transport Ops (International) V1.pdf
PDF
Saundersa Comprehensive Review for the NCLEX-RN Examination.pdf
PPTX
PPT- ENG7_QUARTER1_LESSON1_WEEK1. IMAGERY -DESCRIPTIONS pptx.pptx
PPTX
Pharmacology of Heart Failure /Pharmacotherapy of CHF
PPTX
Introduction_to_Human_Anatomy_and_Physiology_for_B.Pharm.pptx
PPTX
PPH.pptx obstetrics and gynecology in nursing
PPTX
Cell Structure & Organelles in detailed.
PPTX
Final Presentation General Medicine 03-08-2024.pptx
PDF
Basic Mud Logging Guide for educational purpose
PDF
Sports Quiz easy sports quiz sports quiz
PPTX
Lesson notes of climatology university.
PPTX
Pharma ospi slides which help in ospi learning
PDF
Microbial disease of the cardiovascular and lymphatic systems
PPTX
school management -TNTEU- B.Ed., Semester II Unit 1.pptx
PDF
TR - Agricultural Crops Production NC III.pdf
PDF
102 student loan defaulters named and shamed – Is someone you know on the list?
Insiders guide to clinical Medicine.pdf
STATICS OF THE RIGID BODIES Hibbelers.pdf
Black Hat USA 2025 - Micro ICS Summit - ICS/OT Threat Landscape
Abdominal Access Techniques with Prof. Dr. R K Mishra
O5-L3 Freight Transport Ops (International) V1.pdf
Saundersa Comprehensive Review for the NCLEX-RN Examination.pdf
PPT- ENG7_QUARTER1_LESSON1_WEEK1. IMAGERY -DESCRIPTIONS pptx.pptx
Pharmacology of Heart Failure /Pharmacotherapy of CHF
Introduction_to_Human_Anatomy_and_Physiology_for_B.Pharm.pptx
PPH.pptx obstetrics and gynecology in nursing
Cell Structure & Organelles in detailed.
Final Presentation General Medicine 03-08-2024.pptx
Basic Mud Logging Guide for educational purpose
Sports Quiz easy sports quiz sports quiz
Lesson notes of climatology university.
Pharma ospi slides which help in ospi learning
Microbial disease of the cardiovascular and lymphatic systems
school management -TNTEU- B.Ed., Semester II Unit 1.pptx
TR - Agricultural Crops Production NC III.pdf
102 student loan defaulters named and shamed – Is someone you know on the list?

BDACA1617s2 - Lecture4

  • 1. Different types of analysis Sentiment analysis Stopword removal Next meetings Big Data and Automated Content Analysis Week 4 – Monday »Sentiment Analysis« Damian Trilling d.c.trilling@uva.nl @damian0604 www.damiantrilling.net Afdeling Communicatiewetenschap Universiteit van Amsterdam 27 February 2017 Big Data and Automated Content Analysis Damian Trilling
  • 2. Different types of analysis Sentiment analysis Stopword removal Next meetings Today 1 Different types of analysis What can we do? Systematizing analytical approaches 2 Data analysis 1: Sentiment analysis What is it? Bag-of-words approaches Advanced approaches A sentiment analysis tailored to your needs! 3 NLP-preview: Stopword removal Natural language processing A simple algorithm 4 Take-home message, next meetings, & exam Big Data and Automated Content Analysis Damian Trilling
  • 3. Different types of analysis Sentiment analysis Stopword removal Next meetings What we already can do with regard to data collection: • query a (JSON-based) API (GoogleBooks, Twitter) • handle CSV files • handle JSON files Big Data and Automated Content Analysis Damian Trilling
  • 4. Different types of analysis Sentiment analysis Stopword removal Next meetings What we already can do with regard to data collection: • query a (JSON-based) API (GoogleBooks, Twitter) • handle CSV files • handle JSON files with regard to analysis: Not much. We counted some frequencies and calculated some averages. Big Data and Automated Content Analysis Damian Trilling
  • 5. Different types of analysis Sentiment analysis Stopword removal Next meetings What can we do? Data analysis: Overview What can we do? Big Data and Automated Content Analysis Damian Trilling
  • 6. Different types of analysis Sentiment analysis Stopword removal Next meetings What can we do? Data analysis: Overview What can we do? What do you think? What are interesting methods to analyze large data sets (like, e.g., social media data? What questions can they answer?) Big Data and Automated Content Analysis Damian Trilling
  • 7. Different types of analysis Sentiment analysis Stopword removal Next meetings What can we do? What else can we do? For example • sentiment analysis • automated coding with regular expressions • natural language processing • supervised and unsupervised machine learning • network analysis Big Data and Automated Content Analysis Damian Trilling
  • 8. Different types of analysis Sentiment analysis Stopword removal Next meetings What can we do? What else can we do? Or ideally. . . . . . a combination of these techniques. Big Data and Automated Content Analysis Damian Trilling
  • 9. Different types of analysis Sentiment analysis Stopword removal Next meetings Systematizing analytical approaches Overview Systematizing analytical approaches Big Data and Automated Content Analysis Damian Trilling
  • 10. Different types of analysis Sentiment analysis Stopword removal Next meetings Systematizing analytical approaches Systematizing analytical approaches Taking the example of Twitter: Analyzing the structure • Number of Tweets over time • singleton/retweet ratio • Distribution of number of Tweets per user • Interaction networks Bruns, A., & Stieglitz, S. (2013). Toward more systematic Twitter analysis: metrics for tweeting activities. International Journal of Social Research Methodology. doi:10.1080/13645579.2012.756095 Big Data and Automated Content Analysis Damian Trilling
  • 11. Different types of analysis Sentiment analysis Stopword removal Next meetings Systematizing analytical approaches Systematizing analytical approaches Taking the example of Twitter: Analyzing the structure • Number of Tweets over time • singleton/retweet ratio • Distribution of number of Tweets per user • Interaction networks ⇒ Focus on the amount of content and on the question who interacts with whom, not on what is said Bruns, A., & Stieglitz, S. (2013). Toward more systematic Twitter analysis: metrics for tweeting activities. International Journal of Social Research Methodology. doi:10.1080/13645579.2012.756095 Big Data and Automated Content Analysis Damian Trilling
  • 12. Different types of analysis Sentiment analysis Stopword removal Next meetings Systematizing analytical approaches Systematizing analytical approaches Taking the example of Twitter: Analyzing the content • Sentiment analysis • Word frequencies, searchstrings • Co-word analysis (⇒frames) Big Data and Automated Content Analysis Damian Trilling
  • 13. Different types of analysis Sentiment analysis Stopword removal Next meetings Systematizing analytical approaches Systematizing analytical approaches Taking the example of Twitter: Analyzing the content • Sentiment analysis • Word frequencies, searchstrings • Co-word analysis (⇒frames) ⇒ Focus on what is said Big Data and Automated Content Analysis Damian Trilling
  • 14. Different types of analysis Sentiment analysis Stopword removal Next meetings Systematizing analytical approaches Systematizing analytical approaches ⇒ It depends on your reserach question which approach is more interesting! Big Data and Automated Content Analysis Damian Trilling
  • 15. Different types of analysis Sentiment analysis Stopword removal Next meetings Systematizing analytical approaches Automated Content Analysis Methodological approach deductive inductive Typical research interests and content features Common statistical procedures visibility analysis sentiment analysis subjectivity analysis Counting and Dictionary Supervised Machine Learning Unsupervised Machine Learning frames topics gender bias frames topics string comparisons counting support vector machines naive Bayes principal component analysis cluster analysis latent dirichlet allocation semantic network analysis Boumans, J.W., & Trilling, D. (2016). Taking stock of the toolkit: An overview of relevant automated content analysis approaches and techniques for digital journalism scholars. Digital Journalism, 4, 1. 8–23. Big Data and Automated Content Analysis Damian Trilling
  • 16. Different types of analysis Sentiment analysis Stopword removal Next meetings What is it? Data analysis 1: Sentiment analysis Big Data and Automated Content Analysis Damian Trilling
  • 17. Different types of analysis Sentiment analysis Stopword removal Next meetings What is it? What is sentiment analysis? Extracting subjective information from texts Big Data and Automated Content Analysis Damian Trilling
  • 18. Different types of analysis Sentiment analysis Stopword removal Next meetings What is it? What is sentiment analysis? Extracting subjective information from texts • the author’s attitude towards the topic of the text Big Data and Automated Content Analysis Damian Trilling
  • 19. Different types of analysis Sentiment analysis Stopword removal Next meetings What is it? What is sentiment analysis? Extracting subjective information from texts • the author’s attitude towards the topic of the text • polarity: negative—positive Big Data and Automated Content Analysis Damian Trilling
  • 20. Different types of analysis Sentiment analysis Stopword removal Next meetings What is it? What is sentiment analysis? Extracting subjective information from texts • the author’s attitude towards the topic of the text • polarity: negative—positive • subjectivity: neutral—subjective * Big Data and Automated Content Analysis Damian Trilling
  • 21. Different types of analysis Sentiment analysis Stopword removal Next meetings What is it? What is sentiment analysis? Extracting subjective information from texts • the author’s attitude towards the topic of the text • polarity: negative—positive • subjectivity: neutral—subjective * • advanced approaches: different emotions Big Data and Automated Content Analysis Damian Trilling
  • 22. Different types of analysis Sentiment analysis Stopword removal Next meetings What is it? What is sentiment analysis? Extracting subjective information from texts • the author’s attitude towards the topic of the text • polarity: negative—positive • subjectivity: neutral—subjective * • advanced approaches: different emotions * Less sophisticated approaches do not see this as a sperate dimension but simply calculate objectivity = 1 − (negativity + positivity) Big Data and Automated Content Analysis Damian Trilling
  • 23. Different types of analysis Sentiment analysis Stopword removal Next meetings What is it? Applications Who uses it? • Companies • especially for Web Analytics • Social Scientists • applications in data journalism, politics, . . . Many references to examples in Mostafa (2013). ⇒ Cases in which you have a huge amount of data or real-time data and you want to get an idea of the tone. Mostafa, M. M. (2013). More than words: Social networks’ text mining for consumer brand sentiments. Expert Systems with Applications, 40(10), 4241– 4251. doi:10.1016/j.eswa.2013.01.019 Big Data and Automated Content Analysis Damian Trilling
  • 24. Different types of analysis Sentiment analysis Stopword removal Next meetings What is it? Example 1 >>> sentiment("Great service by @NSHighspeed") 2 (0.8, 0.75) 3 >>> sentiment("Bad service by @NSHighspeed") 4 (-0.6166666666666667, 0.6666666666666666) (polarity, subjectivity) with −1 ≤ polarity ≤ +1 0 ≤ subjectivity ≤ +1 ) This is the module pattern.nl, available for Python 2 only. De Smedt, T., & Daelemans W. (2012). Pattern for Python. Journal of Machine Learning Research, 13, 2063-2067. Big Data and Automated Content Analysis Damian Trilling
  • 25. Different types of analysis Sentiment analysis Stopword removal Next meetings Bag-of-words approaches Data analysis 1: Sentiment analysis Bag-of-words approaches Big Data and Automated Content Analysis Damian Trilling
  • 26. Different types of analysis Sentiment analysis Stopword removal Next meetings Bag-of-words approaches Bag-of-words approaches How does it work? • We take each word of a text and look if it’s positive or negative. Big Data and Automated Content Analysis Damian Trilling
  • 27. Different types of analysis Sentiment analysis Stopword removal Next meetings Bag-of-words approaches Bag-of-words approaches How does it work? • We take each word of a text and look if it’s positive or negative. • Most simple way: compare it with a list of negative words and with a list of positive words (That’s what Mostafa (2013) did) Big Data and Automated Content Analysis Damian Trilling
  • 28. Different types of analysis Sentiment analysis Stopword removal Next meetings Bag-of-words approaches Bag-of-words approaches How does it work? • We take each word of a text and look if it’s positive or negative. • Most simple way: compare it with a list of negative words and with a list of positive words (That’s what Mostafa (2013) did) • More advanced: look up a subjectivity score from a table Big Data and Automated Content Analysis Damian Trilling
  • 29. Different types of analysis Sentiment analysis Stopword removal Next meetings Bag-of-words approaches Bag-of-words approaches How does it work? • We take each word of a text and look if it’s positive or negative. • Most simple way: compare it with a list of negative words and with a list of positive words (That’s what Mostafa (2013) did) • More advanced: look up a subjectivity score from a table • e.g., add up the scores and average them. Big Data and Automated Content Analysis Damian Trilling
  • 30. Different types of analysis Sentiment analysis Stopword removal Next meetings Bag-of-words approaches How to do this If you were to run an analyis like the one by Mostafa (2013), how could you do this? Big Data and Automated Content Analysis Damian Trilling
  • 31. Different types of analysis Sentiment analysis Stopword removal Next meetings Bag-of-words approaches How to do this (given a string tekst that you want to analyze and two lists of strings with negative and positive words, lijstpos=["great","fantastic",...,"perfect"] and lijstneg) 1 sentiment=0 2 for woord in tekst.split(): 3 if woord in lijstpos: 4 sentiment=sentiment+1 #same as sentiment+=1 5 elif woord in lijstneg: 6 sentiment=sentiment-1 #same as sentiment-=1 7 print (sentiment) Big Data and Automated Content Analysis Damian Trilling
  • 32. Different types of analysis Sentiment analysis Stopword removal Next meetings Bag-of-words approaches Do we need to have the lists in our program itself? No. You could have them in a separate text file, one per row, and then read that file directly to a list. 1 poslijst=open("filewithonepositivewordperline.txt").read().splitlines() 2 neglijst=open("filewithonenegativewordperline.txt").read().splitlines() Big Data and Automated Content Analysis Damian Trilling
  • 34. Different types of analysis Sentiment analysis Stopword removal Next meetings Bag-of-words approaches More advanced versions • CSV files or similar tables with weights • Or some kind of dict? Big Data and Automated Content Analysis Damian Trilling
  • 37. Different types of analysis Sentiment analysis Stopword removal Next meetings Bag-of-words approaches Mustafa 2013: Interpreting the output Big Data and Automated Content Analysis Damian Trilling
  • 38. Different types of analysis Sentiment analysis Stopword removal Next meetings Bag-of-words approaches Mustafa 2013: Interpreting the output Your thoughts? Big Data and Automated Content Analysis Damian Trilling
  • 39. Different types of analysis Sentiment analysis Stopword removal Next meetings Bag-of-words approaches Mustafa 2013: Interpreting the output Your thoughts? • each word counts equally (1) • many tweets contain no words from the list. What does this mean? • Ways to improve BOW approaches? Big Data and Automated Content Analysis Damian Trilling
  • 40. Different types of analysis Sentiment analysis Stopword removal Next meetings Bag-of-words approaches Bag-of-words approaches e.g., Schut, L. (2013). Verenigde Staten vs. Verenigd Koningrijk: Een automatische inhoudsanalyse naar verklarende factoren voor het gebruik van positive campaigning en negative campaigning door vooraanstaande politici en politieke partijen op Twitter. Bachelor Thesis, Universiteit van Amsterdam. Big Data and Automated Content Analysis Damian Trilling
  • 41. Different types of analysis Sentiment analysis Stopword removal Next meetings Bag-of-words approaches Bag-of-words approaches pro • easy to implement • easy to modify: • add or remove words • make new lists for other languages, other categories (than positive/negative), . . . • easy to understand (transparency, reproducability) e.g., Schut, L. (2013). Verenigde Staten vs. Verenigd Koningrijk: Een automatische inhoudsanalyse naar verklarende factoren voor het gebruik van positive campaigning en negative campaigning door vooraanstaande politici en politieke partijen op Twitter. Bachelor Thesis, Universiteit van Amsterdam. Big Data and Automated Content Analysis Damian Trilling
  • 42. Different types of analysis Sentiment analysis Stopword removal Next meetings Bag-of-words approaches Bag-of-words approaches con • simplistic assumptions • e.g., intensifiers cannot be interpreted ("really" in "really good" or "really bad") • or, even more important, negations. Big Data and Automated Content Analysis Damian Trilling
  • 43. Different types of analysis Sentiment analysis Stopword removal Next meetings Advanced approaches Data analysis 1: Sentiment analysis Advanced approaches Big Data and Automated Content Analysis Damian Trilling
  • 44. Different types of analysis Sentiment analysis Stopword removal Next meetings Advanced approaches Improving the BOW approach Example: The Sentistrenght algorithm • −5 . . . − 1 and +1 . . . + 5 • spelling correction • "booster word list" for strengthening/weakening the effect of the following word • interpreting repeated letters ("baaaaaad"), CAPITALS and !!! • idioms • negation • ldots Thelwall, M., Buckley, K., & Paltoglou, G. (2012). Sentiment strength detection for the social Web. Journal of the American Society for Information Science and Technology, 63(1), 163-173. Big Data and Automated Content Analysis Damian Trilling
  • 45. Different types of analysis Sentiment analysis Stopword removal Next meetings Advanced approaches Advanced approaches Take the structure of a text into account • Try to apply linguistics concepts to identify sentence structure • can identify negations • can interpret intensifiers Big Data and Automated Content Analysis Damian Trilling
  • 46. Different types of analysis Sentiment analysis Stopword removal Next meetings Advanced approaches Example 1 from pattern.nl import sentiment 2 >>> sentiment("Great service by @NSHighspeed") 3 (0.8, 0.75) 4 >>> sentiment("Really") 5 (0.0, 1.0) 6 >>> sentiment("Really Great service by @NSHighspeed") 7 (1.0, 1.0) (polarity, subjectivity) with −1 ≤ polarity ≤ +1 0 ≤ subjectivity ≤ +1 ) Unlike in pure bag-of-words approaches, here, the overall sentiment is not just the sum or the average of its parts! De Smedt, T., & Daelemans W. (2012). Pattern for Python. Journal of Machine Learning Research, 13, 2063-2067. Big Data and Automated Content Analysis Damian Trilling
  • 47. Different types of analysis Sentiment analysis Stopword removal Next meetings Advanced approaches Advanced approaches Big Data and Automated Content Analysis Damian Trilling
  • 48. Different types of analysis Sentiment analysis Stopword removal Next meetings Advanced approaches Advanced approaches pro • understand intensifiers or negation • thus: higher accuracy Big Data and Automated Content Analysis Damian Trilling
  • 49. Different types of analysis Sentiment analysis Stopword removal Next meetings Advanced approaches Advanced approaches pro • understand intensifiers or negation • thus: higher accuracy con • Black box? Or do we understand the algorithm? • Difficult to adapt to own needs • really much better results? Big Data and Automated Content Analysis Damian Trilling
  • 50. Different types of analysis Sentiment analysis Stopword removal Next meetings A sentiment analysis tailored to your needs! Data analysis 1: Sentiment analysis A sentiment analysis tailored to your needs! Big Data and Automated Content Analysis Damian Trilling
  • 51. Different types of analysis Sentiment analysis Stopword removal Next meetings A sentiment analysis tailored to your needs! A sentiment analysis tailored to your needs! Identifying suicidal texts • Bag-of-words-approach with very specific dictionary • added negation • added regular expression search for key phrases • Very specific design requirements: False positives are OK, false negatives not! Huang, Y.-P., Goh, T., & Liew, C.L. (2007). Hunting suicide notes in web 2.0 – preliminary findings. Ninth IEEE International Symposium on Multimedia. Retrieved from http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4476021 Big Data and Automated Content Analysis Damian Trilling
  • 52. Different types of analysis Sentiment analysis Stopword removal Next meetings A sentiment analysis tailored to your needs! Already this still relatively simple approach seems to work satisfactory, but if 106 scientists from 24 competing teams (!) work on it, they can Pestian, J.P.; Matykiewicz, P., Linn-Gust, M., South, B., Uzuner, O., Wiebe, J., Cohen, K.B., Hurdle, J., & Brew, C. (2012). Sentiment analysis of suicide notes: A shared task. Biomedical Informatics Insights, 5(1), p. 3-16. Retrieved from http://europepmc.org/articles/PMC3299408?pdf=render Big Data and Automated Content Analysis Damian Trilling
  • 53. Different types of analysis Sentiment analysis Stopword removal Next meetings A sentiment analysis tailored to your needs! Already this still relatively simple approach seems to work satisfactory, but if 106 scientists from 24 competing teams (!) work on it, they can group suidide notes by these characteristics: • swear • family • friend • positive emotion • negative emotion • anxiety • anger • sad • cognitive process • biology • sexual • ingestion • religion • death Pestian, J.P.; Matykiewicz, P., Linn-Gust, M., South, B., Uzuner, O., Wiebe, J., Cohen, K.B., Hurdle, J., & Brew, C. (2012). Sentiment analysis of suicide notes: A shared task. Biomedical Informatics Insights, 5(1), p. 3-16. Retrieved from http://europepmc.org/articles/PMC3299408?pdf=render Big Data and Automated Content Analysis Damian Trilling
  • 54. Different types of analysis Sentiment analysis Stopword removal Next meetings A sentiment analysis tailored to your needs! Sidenote An alternative state-of-the-art approach: Use supervised machine learning • Instead of defining rules, hand-code (“annotate”) the sentiment of some tweets manually and let the computer find out which words or characters (“features”) predict sentiment • Then use this model to predict sentiment for other tweets • Essentially the same like what you know since the second year of your Bachelor: regression analysis (but now with DV sentiment and IV’s word occurrences) • ⇒ week 7 Gonzalez-Bailon, S., & Paltoglou, G. (2015). Signals of public opinion in online communication: A comparison of methods and data sources. The ANNALS of the American Academy of Political and Social Science, 659(1), 95-–107. Big Data and Automated Content Analysis Damian Trilling
  • 55. Different types of analysis Sentiment analysis Stopword removal Next meetings Natural Language Processing preview: Stopword removal Big Data and Automated Content Analysis Damian Trilling
  • 56. Different types of analysis Sentiment analysis Stopword removal Next meetings Natural Language Processing preview: Stopword removal Why now? — Because the logic of the algorithm is very much related to the one of our first simple sentiment analysis Big Data and Automated Content Analysis Damian Trilling
  • 57. Different types of analysis Sentiment analysis Stopword removal Next meetings Natural language processing Stopword removal: What and why? Why remove stopwords? • If we want to identify key terms (e.g., by means of a word count), we are not interested in them • If we want to calculate document similarity, it might be inflated • If we want to make a word co-occurance graph, irrelevant information will dominate the picture Big Data and Automated Content Analysis Damian Trilling
  • 58. Different types of analysis Sentiment analysis Stopword removal Next meetings A simple algorithm Stopword removal: How 1 testo=’He gives her a beer and a cigarette.’ 2 testonuovo="" 3 stopwords=[’and’,’the’,’a’,’or’,’he’,’she’,’him’,’her’] 4 for verbo in testo.split(): 5 if verbo not in stopwords: 6 testonuovo=testonuovo+verbo+" " What do we get if we do: 1 print (testonuovo) Can you explain the algorithm? Big Data and Automated Content Analysis Damian Trilling
  • 59. Different types of analysis Sentiment analysis Stopword removal Next meetings A simple algorithm We get: 1 >>> print (testonuovo) 2 ’He gives beer cigarette. ’ Why is "He" still in there? How can we fix this? Big Data and Automated Content Analysis Damian Trilling
  • 60. Different types of analysis Sentiment analysis Stopword removal Next meetings A simple algorithm Stopword removal 1 testo=’He gives her a beer and a cigarette.’ 2 testonuovo="" 3 stopwords=[’and’,’the’,’a’,’or’,’he’,’she’,’him’,’her’] 4 for verbo in testo.split(): 5 if verbo.lower() not in stopwords: 6 testonuovo=testonuovo+verbo+" " Big Data and Automated Content Analysis Damian Trilling
  • 61. Different types of analysis Sentiment analysis Stopword removal Next meetings Take-home message Mid-term take-home exam Next meetings Big Data and Automated Content Analysis Damian Trilling
  • 62. Different types of analysis Sentiment analysis Stopword removal Next meetings Take-home messages What you should be familiar with: • You should have completely understood last week’s exercise. Re-read it if neccessary. • Approaches to the analysis (e.g., structure vs. content) • Types of sentiment analysis, application areas, pros and cons Big Data and Automated Content Analysis Damian Trilling
  • 63. Different types of analysis Sentiment analysis Stopword removal Next meetings Mid-term take home exam Week 5: Wednesday, 8 March, to Sunday, 12 March • You get the exam on Wednesday at the end of the meeting • Answers have to be handed in no later than Sunday, 23.59 • 30% of final grade • 3 questions: 1 Literature question: E.g., different methods (“Explain how. . . is done”) and/or epistemological or theoretical implications (“What does this mean for social-scientific research?”) 2 Empirical question (conceptual) 3 Empirical question (actual programming task) If you fully understood all exercises until now, it shouldn’t be difficult and won’t take too long. But give yourself a lot of buffer time!!! Big Data and Automated Content Analysis Damian Trilling
  • 64. Different types of analysis Sentiment analysis Stopword removal Next meetings Next meetings Wednesday, 1–3 We work together on the examples in Chapter 6. You can bring your own data (you will probably learn more!), but then already think about how to write some script to read the data (as we did last week and as described in Chapter 5). Big Data and Automated Content Analysis Damian Trilling