SlideShare a Scribd company logo

BD-ACA week1b

Department of Communication Science, University of Amsterdam, profile picture
Department of Communication Science, University of Amsterdam

The document discusses the concept of big data, including its definitions, methods, tools, and previous research in automated content analysis. It outlines challenges such as the need for extensive computational power, the potential biases in data interpretations, and the ethical considerations when utilizing large datasets. Additionally, the document emphasizes the necessity of developing custom tools in Python for academic research to ensure transparency and reproducibility.

Education
Related topics:
Content Analysis Overview
1 of 56
Downloaded 29 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
What is Big Data? Methods What have others done? What can we do? The schedule Big Data and Automated Content Analysis Week 1 – Wednesday »Introduction« Damian Trilling d.c.trilling@uva.nl @damian0604 www.damiantrilling.net Afdeling Communicatiewetenschap Universiteit van Amsterdam 1 April 2014 Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Today 1 What is Big Data? Definitions Are we doing Big Data research? 2 Methods Which techniques? Which tools? 3 What have others done? Online news sharing Partisan asymmetries 4 What can we do? Considerations regarding feasibility Examples from last year 5 The schedule Next meetings Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Definitions What is Big Data? Big Data and Automated Content Analysis Damian Trilling
BD-ACA week1b
What is Big Data? Methods What have others done? What can we do? The schedule Definitions What is Big Data? What would you call Big Data? Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Definitions What is Big Data? Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Definitions What is Big Data? A simple technical definition could be: Everything that needs so much computational power and/or storage that you cannot do it on a regular computer. Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Definitions What is Big Data? Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Definitions What is Big Data? Vis, 2013 • “commercial” definition (Gartner): “’Big data’ is high-volume, -velocity and -variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making” Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Definitions What is Big Data? Vis, 2013 • boyd & Crawford definition: 1 Technology: maximizing computation power and algorithmic accuracy to gather, analyze, link, and compare large data sets. 2 Analysis: drawing on large data sets to identify patterns in order to make economic, social, technical, and legal claims. 3 Mythology: the widespread belief that large data sets offer a higher form of intelligence and knowledge that can generate insights that were previously impossible, with the aura of truth, objectivity, and accuracy. Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Definitions What is Big Data? Vis, 2013 • “commercial” definition (Gartner): “’Big data’ is high-volume, -velocity and -variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making” • boyd & Crawford definition: 1 Technology: maximizing computation power and algorithmic accuracy to gather, analyze, link, and compare large data sets. 2 Analysis: drawing on large data sets to identify patterns in order to make economic, social, technical, and legal claims. 3 Mythology: the widespread belief that large data sets offer a higher form of intelligence and knowledge that can generate insights that were previously impossible, with the aura of truth, objectivity, and accuracy. Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Definitions Implications & critizism Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Definitions Implications & critizism boyd & Crawford, 2012 1 Big Data changes the definition of knowledge 2 Claims to objectivity and accuracy are misleading 3 Bigger data are not always better data 4 Taken out of context, Big Data loses its meaning 5 Just because it is accessible does not make it ethical 6 Limited access to Big Data creates new digital divides Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Definitions APIs, researchers and tools make Big Data Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Definitions APIs, researchers and tools make Big Data Vis, 2013 Inevitable influences of: • APIs • filtering, search strings, . . . • changing services over time • organizations that provide the data Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Are we doing Big Data research? Are we doing Big Data research in this course? Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Are we doing Big Data research? Are we doing Big Data research in this course? Depends on the definition • Not if we take a definition that only focuses on computing power and the amount of data Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Are we doing Big Data research? Are we doing Big Data research in this course? Depends on the definition • Not if we take a definition that only focuses on computing power and the amount of data • But: We are using the same techniques. And they scale well. Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Are we doing Big Data research? Are we doing Big Data research in this course? Depends on the definition • Not if we take a definition that only focuses on computing power and the amount of data • But: We are using the same techniques. And they scale well. • Oh, and about that high-performance computing in the cloud: We actually do have access to that, so if someone has a really great idea. . . Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Methods Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Which techniques? What we will learn the next weeks Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Which techniques? What we will learn the next weeks 1. How to collect data APIs, scrapers and crawlers, feeds, databases, . . . Storage in different file formats Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Which techniques? What we will learn the next weeks 1. How to collect data APIs, scrapers and crawlers, feeds, databases, . . . Storage in different file formats 2. How to analyze data Sentiment analysis, automated content analysis, regular expressions, natural language processing, cluster analysis, machine learning, network analysis Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Which tools? The methods The tools we use for this Some existing tools, but mostly we will write our own tools (or modify existing tools) using the programming language Python (and make use of the huge amount of Python modules others already wrote). Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Which tools? The methods The tools we use for this Some existing tools, but mostly we will write our own tools (or modify existing tools) using the programming language Python (and make use of the huge amount of Python modules others already wrote). But aren’t there existing tools? Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Which tools? The methods The tools we use for this Some existing tools, but mostly we will write our own tools (or modify existing tools) using the programming language Python (and make use of the huge amount of Python modules others already wrote). But aren’t there existing tools? — sometimes. But look at this example: Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Which tools? Why program your own tool? Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Which tools? Why program your own tool? If the task would have been done with a (commercial) tool, we can only research what the tool allows us to do (⇒ our discussion from some minutes ago). Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Which tools? Why program your own tool? If the task would have been done with a (commercial) tool, we can only research what the tool allows us to do (⇒ our discussion from some minutes ago). Luckily, the problem is easily solved The task was done with a self-written Python program. We change the line lengte_list.append(len(row[textcolumn])) to lengte_list.append(len(row[textcolumn].split())) Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Which tools? Why program your own tool? Moreover, the tools we use can limit the range of questions that might be imagined, simply because they do not fit the affordances of the tool. Not many researchers themselves have the ability or access to other researchers who can build the required tools in line with any preferred enquiry. This then introduces serious limitations in terms of the scope of research that can be done. Vis, 2013 Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Which tools? More considerations Assuming that science should be transparent and reproducible Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Which tools? More considerations Assuming that science should be transparent and reproducible, we should use tools that are • platform-independent • free (as in beer and as in speech, gratis and libre) • which implies: open source Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Which tools? More considerations Assuming that science should be transparent and reproducible, we should use tools that are • platform-independent • free (as in beer and as in speech, gratis and libre) • which implies: open source This ensures it can our research (a) can be reproduced by anyone, and that there is (b) no black box that no one can look inside Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Which tools? Why program your own tool? [...] these [commercial] tools are often unsuitable for academic purposes because of their cost, along with the problematic ‘black box’ nature of many of these tools. Vis, 2013 [...] we should resist the temptation to let the opportunities and constraints of an application or platform determine the research question [...] Mahrt & Scharkow, 2013, p. 30 Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule What have others done? Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Online news sharing Online news sharing Castillo, El-Haddad, Pfeffer, & Stempeck, 2013 Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Online news sharing Online news sharing The question “We describe the interplay between website visitation patterns and social media reactions to news content. [. . . ] We also show that social media reactions can help predict future visitation patterns early and accurately.” (p.1) Castillo, El-Haddad, Pfeffer, & Stempeck, 2013 Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Online news sharing Online news sharing The method • data set 1: log files provided by Al Jazeera • data set 2: Facebook and Twitter API • analysis: link 1 and 2 and estimate the relationships Castillo, El-Haddad, Pfeffer, & Stempeck, 2013 Big Data and Automated Content Analysis Damian Trilling
BD-ACA week1b
What is Big Data? Methods What have others done? What can we do? The schedule Online news sharing Online news sharing The issues • You need that guy at Al Jazeera • You need the infrastructure to cope with the data • Very much tailored to one outlet Castillo, El-Haddad, Pfeffer, & Stempeck, 2013 Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Partisan asymmetries Partisan asymmetries Conover, Gonçalves, Flammini, & Menczer, 2012 Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Partisan asymmetries Partisan asymmetries The question How does the Twitter behavior differ between right-wing and left-wing users? Conover, Gonçalves, Flammini, & Menczer, 2012 Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Partisan asymmetries Partisan asymmetries The method Starting with two hashtags (one used by progressives, one used by conservatives), 55 co-occuring hashtags were identified. Identification of follower networks, retweet networks, mention networks within tweets with these hashtags. Conover, Gonçalves, Flammini, & Menczer, 2012 Big Data and Automated Content Analysis Damian Trilling
BD-ACA week1b
What is Big Data? Methods What have others done? What can we do? The schedule Partisan asymmetries Partisan asymmetries The issues • The two seeds #tcot and #p2 oversample (extremly) partisan content, inevitably leading to the structure in Figure 3. • But maybe no problem, as the rest of the paper aims to compare these groups. • Not an empirical problem, but still: What do we learn exactly from this study apart from the – interesting – case? Conover, Gonçalves, Flammini, & Menczer, 2012 Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Considerations regarding feasibility To which extent could we conduct such studies? Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Considerations regarding feasibility What can we do? Cool research, sure, but what can we do? • Dependency from third parties: scraping < API < server-side implementation • Restrictions (e.g., Twitter: sprinkler, garden hose, fire hose) ⇒ Vis, 2013: Data are made! • We can’t just trust the numbers. Some tasks require human coders – or a qualitative approach, at least as a pre-study. Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Considerations regarding feasibility What can we do? pros, cons, and feasibility • APIs (Twitter, Facebook, . . . ) • server-side implementations (⇒ Al Jazeera-example) • scraping • client-side log files • "traditional" methods (surveys etc.) Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Considerations regarding feasibility What can we do? What helps us answer our questions? 1 Draft a RQ. 2 Think of different ways to approach it 3 Think of different data sources. 4 Think of different analyses. Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Considerations regarding feasibility What can we do? What helps us answer our questions? 1 Draft a RQ. 2 Think of different ways to approach it 3 Think of different data sources. 4 Think of different analyses. And then: 1 Check what the technical possibilities are. (e.g., is there an API? How can we get the data?) 2 Re-evaluate all steps. Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Examples from last year Some examples from last year’s course Questions • Are (popular) apps for children in the Apple App Store framed in terms of education or entertainment? • How can house prices on funda.nl be predicted? • Where are those who edit Wikipedia-entries about companies geographically located, related to the company’s HQ? • How do embassies [or top universities] communicate on Twitter? • How is the tone in newspaper election coverage related to the tone on Twitter? Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule The schedule Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule The schedule Each week In general: A lecture (Monday) and a lab session (Wednesday). Each week one method. Examinations A mid-term take-home exam in week 5 and an individual research project on which you work during the whole course. Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Next meetings Next meetings Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Next meetings Week 2 No meeting on Monday (Easter) Wednesday, 8 April: Lab Session Getting to know Python better. Do not forget to finish the exercises in chapter 2 and 3 in advance. Big Data and Automated Content Analysis Damian Trilling
What is Big Data? Methods What have others done? What can we do? The schedule Next meetings Week 3: Data harvesting and storage Monday, 13–4 A conceptual overview of APIs, scrapers, crawlers, RSS-feeds, databases, and different file formats Wednesday, 15–4 Writing some first data collection scripts Big Data and Automated Content Analysis Damian Trilling

More Related Content

PDF
BDACA - Lecture1
Department of Communication Science, University of Amsterdam
 
PDF
BDACA1617s2 - Lecture 1
Department of Communication Science, University of Amsterdam
 
PDF
BDACA1516s2 - Lecture4
Department of Communication Science, University of Amsterdam
 
PDF
Introduction to Data Science and Large-scale Machine Learning
Nik Spirin
 
PPTX
Haystack 2019 - Search-based recommendations at Politico - Ryan Kohl
OpenSource Connections
 
PDF
From Big Data to AI
Maloy Manna, PMP®
 
PDF
BDACA - Lecture8
Department of Communication Science, University of Amsterdam
 
PDF
Clare Corthell: Learning Data Science Online
sfdatascience
 
BDACA - Lecture1
Department of Communication Science, University of Amsterdam
 
BDACA1617s2 - Lecture 1
Department of Communication Science, University of Amsterdam
 
BDACA1516s2 - Lecture4
Department of Communication Science, University of Amsterdam
 
Introduction to Data Science and Large-scale Machine Learning
Nik Spirin
 
Haystack 2019 - Search-based recommendations at Politico - Ryan Kohl
OpenSource Connections
 
From Big Data to AI
Maloy Manna, PMP®
 
BDACA - Lecture8
Department of Communication Science, University of Amsterdam
 
Clare Corthell: Learning Data Science Online
sfdatascience
 

What's hot (15)

PDF
Claudia Gold: Learning Data Science Online
sfdatascience
 
PDF
Big data and AI presentation slides
CloudxLab
 
PDF
How to become a Data Scientist?
HackerEarth
 
PPSX
Bigdata presentation
SatishAlerts
 
PPTX
Lessons Learned The Hard Way: 32+ Data Science Interviews
Gregory Kamradt
 
PPTX
Keynote - An overview on Big Data & Data Science - Dr Gregory Piatetsky-Shapiro
Data ScienceTech Institute
 
PPTX
Machine Learning for Non-technical People
indico data
 
PDF
Be a Data Scientist in 8 steps!
PromptCloud
 
PPTX
Lecture #01
Konpal Darakshan
 
PDF
Data_Scientist_Position_Description
Suman Banerjee
 
PPTX
How To Become a Data Scientist in Iran Marketplace
Mohamadreza Mohtat
 
PPTX
Big Data Courses in Pune
Classboat.com
 
PPTX
Machine Learning for Non-Technical People - Turing Fest 2019
Britney Muller
 
PPTX
Course Information for March 25th Batch
UpXAcademy
 
PDF
Quantitative Ethics - Governance and ethics of AI decisions
Nikita Lukianets
 
Claudia Gold: Learning Data Science Online
sfdatascience
 
Big data and AI presentation slides
CloudxLab
 
How to become a Data Scientist?
HackerEarth
 
Bigdata presentation
SatishAlerts
 
Lessons Learned The Hard Way: 32+ Data Science Interviews
Gregory Kamradt
 
Keynote - An overview on Big Data & Data Science - Dr Gregory Piatetsky-Shapiro
Data ScienceTech Institute
 
Machine Learning for Non-technical People
indico data
 
Be a Data Scientist in 8 steps!
PromptCloud
 
Lecture #01
Konpal Darakshan
 
Data_Scientist_Position_Description
Suman Banerjee
 
How To Become a Data Scientist in Iran Marketplace
Mohamadreza Mohtat
 
Big Data Courses in Pune
Classboat.com
 
Machine Learning for Non-Technical People - Turing Fest 2019
Britney Muller
 
Course Information for March 25th Batch
UpXAcademy
 
Quantitative Ethics - Governance and ethics of AI decisions
Nikita Lukianets
 
Ad

Similar to BD-ACA week1b (20)

PDF
BDACA1516s2 - Lecture1
Department of Communication Science, University of Amsterdam
 
PDF
Career in Data Science (July 2017, DTLA)
Thinkful
 
PDF
Thinkful DC - Intro to Data Science
TJ Stalcup
 
PPTX
Big data analytics
Isłém Jendoubi
 
PDF
Getting Started in Data Science
Thinkful
 
PDF
How to succeed at data without even trying!
Dylan
 
PDF
A Review of Big data for Social Policy Decision Making
Ridi Fe
 
PDF
Big Data, Big Opportunities
Arimo, Inc.
 
PDF
Tableau @ Facebook - Summer 2014
Andy Kriebel
 
PDF
Big Data Intoduction & Hadoop ArchitectureModule1.pdf
Sharmila Chidaravalli
 
PDF
Getting started in data science (4:3)
Thinkful
 
PDF
Getting started in data science (4:3)
Thinkful
 
PDF
Getting started in Data Science (April 2017, Los Angeles)
Thinkful
 
PDF
Thinkful - Intro to Data Science - Washington DC
TJ Stalcup
 
PPTX
Big Data and HR - Talk @SwissHR Congress
Marcel Blattner, PhD
 
PDF
Startds9.19.17sd
Thinkful
 
PDF
Getstarteddssd12717sd
Thinkful
 
PDF
Deck 92-146 (3)
Thinkful
 
PDF
2017 06-14-getting started with data science
Thinkful
 
PDF
Intro to Data Science
TJ Stalcup
 
BDACA1516s2 - Lecture1
Department of Communication Science, University of Amsterdam
 
Career in Data Science (July 2017, DTLA)
Thinkful
 
Thinkful DC - Intro to Data Science
TJ Stalcup
 
Big data analytics
Isłém Jendoubi
 
Getting Started in Data Science
Thinkful
 
How to succeed at data without even trying!
Dylan
 
A Review of Big data for Social Policy Decision Making
Ridi Fe
 
Big Data, Big Opportunities
Arimo, Inc.
 
Tableau @ Facebook - Summer 2014
Andy Kriebel
 
Big Data Intoduction & Hadoop ArchitectureModule1.pdf
Sharmila Chidaravalli
 
Getting started in data science (4:3)
Thinkful
 
Getting started in data science (4:3)
Thinkful
 
Getting started in Data Science (April 2017, Los Angeles)
Thinkful
 
Thinkful - Intro to Data Science - Washington DC
TJ Stalcup
 
Big Data and HR - Talk @SwissHR Congress
Marcel Blattner, PhD
 
Startds9.19.17sd
Thinkful
 
Getstarteddssd12717sd
Thinkful
 
Deck 92-146 (3)
Thinkful
 
2017 06-14-getting started with data science
Thinkful
 
Intro to Data Science
TJ Stalcup
 
Ad

More from Department of Communication Science, University of Amsterdam (20)

PDF
BDACA - Lecture7
Department of Communication Science, University of Amsterdam
 
PDF
BDACA - Lecture6
Department of Communication Science, University of Amsterdam
 
PDF
BDACA - Tutorial5
Department of Communication Science, University of Amsterdam
 
PDF
BDACA - Lecture5
Department of Communication Science, University of Amsterdam
 
PDF
BDACA - Lecture4
Department of Communication Science, University of Amsterdam
 
PDF
BDACA - Lecture3
Department of Communication Science, University of Amsterdam
 
PDF
BDACA - Lecture2
Department of Communication Science, University of Amsterdam
 
PDF
BDACA - Tutorial1
Department of Communication Science, University of Amsterdam
 
PDF
BDACA1617s2 - Lecture7
Department of Communication Science, University of Amsterdam
 
PDF
BDACA1617s2 - Lecture6
Department of Communication Science, University of Amsterdam
 
PDF
BDACA1617s2 - Lecture5
Department of Communication Science, University of Amsterdam
 
PDF
BDACA1617s2 - Lecture4
Department of Communication Science, University of Amsterdam
 
PDF
BDACA1617s2 - Lecture3
Department of Communication Science, University of Amsterdam
 
PDF
BDACA1617s2 - Lecture 2
Department of Communication Science, University of Amsterdam
 
PDF
BDACA1617s2 - Tutorial 1
Department of Communication Science, University of Amsterdam
 
PDF
Media diets in an age of apps and social media: Dealing with a third layer of...
Department of Communication Science, University of Amsterdam
 
PDF
Conceptualizing and measuring news exposure as network of users and news items
Department of Communication Science, University of Amsterdam
 
PDF
Data Science: Case "Political Communication 2/2"
Department of Communication Science, University of Amsterdam
 
PDF
Data Science: Case "Political Communication 1/2"
Department of Communication Science, University of Amsterdam
 
PDF
BDACA1516s2 - Lecture8
Department of Communication Science, University of Amsterdam
 
BDACA - Lecture7
Department of Communication Science, University of Amsterdam
 
BDACA - Lecture6
Department of Communication Science, University of Amsterdam
 
BDACA - Tutorial5
Department of Communication Science, University of Amsterdam
 
BDACA - Lecture5
Department of Communication Science, University of Amsterdam
 
BDACA - Lecture4
Department of Communication Science, University of Amsterdam
 
BDACA - Lecture3
Department of Communication Science, University of Amsterdam
 
BDACA - Lecture2
Department of Communication Science, University of Amsterdam
 
BDACA - Tutorial1
Department of Communication Science, University of Amsterdam
 
BDACA1617s2 - Lecture7
Department of Communication Science, University of Amsterdam
 
BDACA1617s2 - Lecture6
Department of Communication Science, University of Amsterdam
 
BDACA1617s2 - Lecture5
Department of Communication Science, University of Amsterdam
 
BDACA1617s2 - Lecture4
Department of Communication Science, University of Amsterdam
 
BDACA1617s2 - Lecture3
Department of Communication Science, University of Amsterdam
 
BDACA1617s2 - Lecture 2
Department of Communication Science, University of Amsterdam
 
BDACA1617s2 - Tutorial 1
Department of Communication Science, University of Amsterdam
 
Media diets in an age of apps and social media: Dealing with a third layer of...
Department of Communication Science, University of Amsterdam
 
Conceptualizing and measuring news exposure as network of users and news items
Department of Communication Science, University of Amsterdam
 
Data Science: Case "Political Communication 2/2"
Department of Communication Science, University of Amsterdam
 
Data Science: Case "Political Communication 1/2"
Department of Communication Science, University of Amsterdam
 
BDACA1516s2 - Lecture8
Department of Communication Science, University of Amsterdam
 

Recently uploaded (20)

PPTX
1st Inaugural Professorial Lecture held on 19th February 2020 (Governance and...
kawisojames10
 
PDF
Supply Chain Operations Speaking Notes -ICLT Program
labyankof
 
PDF
Classroom Observation Tools for Teachers
Nicaramirez
 
PDF
Chinmaya Tiranga quiz Grand Finale.pdf
Nitin Suresh
 
PPTX
Lesson notes of climatology university.
sgladness67
 
PPTX
Orientation - ARALprogram of Deped to the Parents.pptx
JeromeTamayoSantiago1
 
PPTX
IMMUNITY IMMUNITY refers to protection against infection, and the immune syst...
shilpa939953
 
PDF
01-Introduction-to-Information-Management.pdf
PrinceIbarra
 
PPTX
Final Presentation General Medicine 03-08-2024.pptx
ZaheerAhmad228692
 
PPTX
Pharmacology of Heart Failure /Pharmacotherapy of CHF
Rajshri Ghogare
 
PPTX
human mycosis Human fungal infections are called human mycosis..pptx
shilpa939953
 
PDF
Trump Administration's workforce development strategy
Mebane Rash
 
PPTX
202450812 BayCHI UCSC-SV 20250812 v17.pptx
home
 
PDF
3rd Neelam Sanjeevareddy Memorial Lecture.pdf
Academy of Grassroots Studies and Research of India (AGRASRI)
 
PPTX
Pharma ospi slides which help in ospi learning
rameetkumar304
 
PDF
Microbial disease of the cardiovascular and lymphatic systems
KeyaSharma
 
PDF
OBE - B.A.(HON'S) IN INTERIOR ARCHITECTURE -Ar.MOHIUDDIN.pdf
ArchitectAFMMohiuddi
 
PDF
Yogi Goddess Pres Conference Studio Updates
LDMMia Reiki Lectures
 
PDF
RMMM.pdf make it easy to upload and study
sajedul2
 
PPTX
GDM (1) (1).pptx small presentation for students
shrawanbpkihs
 
1st Inaugural Professorial Lecture held on 19th February 2020 (Governance and...
kawisojames10
 
Supply Chain Operations Speaking Notes -ICLT Program
labyankof
 
Classroom Observation Tools for Teachers
Nicaramirez
 
Chinmaya Tiranga quiz Grand Finale.pdf
Nitin Suresh
 
Lesson notes of climatology university.
sgladness67
 
Orientation - ARALprogram of Deped to the Parents.pptx
JeromeTamayoSantiago1
 
IMMUNITY IMMUNITY refers to protection against infection, and the immune syst...
shilpa939953
 
01-Introduction-to-Information-Management.pdf
PrinceIbarra
 
Final Presentation General Medicine 03-08-2024.pptx
ZaheerAhmad228692
 
Pharmacology of Heart Failure /Pharmacotherapy of CHF
Rajshri Ghogare
 
human mycosis Human fungal infections are called human mycosis..pptx
shilpa939953
 
Trump Administration's workforce development strategy
Mebane Rash
 
202450812 BayCHI UCSC-SV 20250812 v17.pptx
home
 
3rd Neelam Sanjeevareddy Memorial Lecture.pdf
Academy of Grassroots Studies and Research of India (AGRASRI)
 
Pharma ospi slides which help in ospi learning
rameetkumar304
 
Microbial disease of the cardiovascular and lymphatic systems
KeyaSharma
 
OBE - B.A.(HON'S) IN INTERIOR ARCHITECTURE -Ar.MOHIUDDIN.pdf
ArchitectAFMMohiuddi
 
Yogi Goddess Pres Conference Studio Updates
LDMMia Reiki Lectures
 
RMMM.pdf make it easy to upload and study
sajedul2
 
GDM (1) (1).pptx small presentation for students
shrawanbpkihs
 

BD-ACA week1b

  • 1. What is Big Data? Methods What have others done? What can we do? The schedule Big Data and Automated Content Analysis Week 1 – Wednesday »Introduction« Damian Trilling d.c.trilling@uva.nl @damian0604 www.damiantrilling.net Afdeling Communicatiewetenschap Universiteit van Amsterdam 1 April 2014 Big Data and Automated Content Analysis Damian Trilling
  • 2. What is Big Data? Methods What have others done? What can we do? The schedule Today 1 What is Big Data? Definitions Are we doing Big Data research? 2 Methods Which techniques? Which tools? 3 What have others done? Online news sharing Partisan asymmetries 4 What can we do? Considerations regarding feasibility Examples from last year 5 The schedule Next meetings Big Data and Automated Content Analysis Damian Trilling
  • 3. What is Big Data? Methods What have others done? What can we do? The schedule Definitions What is Big Data? Big Data and Automated Content Analysis Damian Trilling
  • 5. What is Big Data? Methods What have others done? What can we do? The schedule Definitions What is Big Data? What would you call Big Data? Big Data and Automated Content Analysis Damian Trilling
  • 6. What is Big Data? Methods What have others done? What can we do? The schedule Definitions What is Big Data? Big Data and Automated Content Analysis Damian Trilling
  • 7. What is Big Data? Methods What have others done? What can we do? The schedule Definitions What is Big Data? A simple technical definition could be: Everything that needs so much computational power and/or storage that you cannot do it on a regular computer. Big Data and Automated Content Analysis Damian Trilling
  • 8. What is Big Data? Methods What have others done? What can we do? The schedule Definitions What is Big Data? Big Data and Automated Content Analysis Damian Trilling
  • 9. What is Big Data? Methods What have others done? What can we do? The schedule Definitions What is Big Data? Vis, 2013 • “commercial” definition (Gartner): “’Big data’ is high-volume, -velocity and -variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making” Big Data and Automated Content Analysis Damian Trilling
  • 10. What is Big Data? Methods What have others done? What can we do? The schedule Definitions What is Big Data? Vis, 2013 • boyd & Crawford definition: 1 Technology: maximizing computation power and algorithmic accuracy to gather, analyze, link, and compare large data sets. 2 Analysis: drawing on large data sets to identify patterns in order to make economic, social, technical, and legal claims. 3 Mythology: the widespread belief that large data sets offer a higher form of intelligence and knowledge that can generate insights that were previously impossible, with the aura of truth, objectivity, and accuracy. Big Data and Automated Content Analysis Damian Trilling
  • 11. What is Big Data? Methods What have others done? What can we do? The schedule Definitions What is Big Data? Vis, 2013 • “commercial” definition (Gartner): “’Big data’ is high-volume, -velocity and -variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making” • boyd & Crawford definition: 1 Technology: maximizing computation power and algorithmic accuracy to gather, analyze, link, and compare large data sets. 2 Analysis: drawing on large data sets to identify patterns in order to make economic, social, technical, and legal claims. 3 Mythology: the widespread belief that large data sets offer a higher form of intelligence and knowledge that can generate insights that were previously impossible, with the aura of truth, objectivity, and accuracy. Big Data and Automated Content Analysis Damian Trilling
  • 12. What is Big Data? Methods What have others done? What can we do? The schedule Definitions Implications & critizism Big Data and Automated Content Analysis Damian Trilling
  • 13. What is Big Data? Methods What have others done? What can we do? The schedule Definitions Implications & critizism boyd & Crawford, 2012 1 Big Data changes the definition of knowledge 2 Claims to objectivity and accuracy are misleading 3 Bigger data are not always better data 4 Taken out of context, Big Data loses its meaning 5 Just because it is accessible does not make it ethical 6 Limited access to Big Data creates new digital divides Big Data and Automated Content Analysis Damian Trilling
  • 14. What is Big Data? Methods What have others done? What can we do? The schedule Definitions APIs, researchers and tools make Big Data Big Data and Automated Content Analysis Damian Trilling
  • 15. What is Big Data? Methods What have others done? What can we do? The schedule Definitions APIs, researchers and tools make Big Data Vis, 2013 Inevitable influences of: • APIs • filtering, search strings, . . . • changing services over time • organizations that provide the data Big Data and Automated Content Analysis Damian Trilling
  • 16. What is Big Data? Methods What have others done? What can we do? The schedule Are we doing Big Data research? Are we doing Big Data research in this course? Big Data and Automated Content Analysis Damian Trilling
  • 17. What is Big Data? Methods What have others done? What can we do? The schedule Are we doing Big Data research? Are we doing Big Data research in this course? Depends on the definition • Not if we take a definition that only focuses on computing power and the amount of data Big Data and Automated Content Analysis Damian Trilling
  • 18. What is Big Data? Methods What have others done? What can we do? The schedule Are we doing Big Data research? Are we doing Big Data research in this course? Depends on the definition • Not if we take a definition that only focuses on computing power and the amount of data • But: We are using the same techniques. And they scale well. Big Data and Automated Content Analysis Damian Trilling
  • 19. What is Big Data? Methods What have others done? What can we do? The schedule Are we doing Big Data research? Are we doing Big Data research in this course? Depends on the definition • Not if we take a definition that only focuses on computing power and the amount of data • But: We are using the same techniques. And they scale well. • Oh, and about that high-performance computing in the cloud: We actually do have access to that, so if someone has a really great idea. . . Big Data and Automated Content Analysis Damian Trilling
  • 20. What is Big Data? Methods What have others done? What can we do? The schedule Methods Big Data and Automated Content Analysis Damian Trilling
  • 21. What is Big Data? Methods What have others done? What can we do? The schedule Which techniques? What we will learn the next weeks Big Data and Automated Content Analysis Damian Trilling
  • 22. What is Big Data? Methods What have others done? What can we do? The schedule Which techniques? What we will learn the next weeks 1. How to collect data APIs, scrapers and crawlers, feeds, databases, . . . Storage in different file formats Big Data and Automated Content Analysis Damian Trilling
  • 23. What is Big Data? Methods What have others done? What can we do? The schedule Which techniques? What we will learn the next weeks 1. How to collect data APIs, scrapers and crawlers, feeds, databases, . . . Storage in different file formats 2. How to analyze data Sentiment analysis, automated content analysis, regular expressions, natural language processing, cluster analysis, machine learning, network analysis Big Data and Automated Content Analysis Damian Trilling
  • 24. What is Big Data? Methods What have others done? What can we do? The schedule Which tools? The methods The tools we use for this Some existing tools, but mostly we will write our own tools (or modify existing tools) using the programming language Python (and make use of the huge amount of Python modules others already wrote). Big Data and Automated Content Analysis Damian Trilling
  • 25. What is Big Data? Methods What have others done? What can we do? The schedule Which tools? The methods The tools we use for this Some existing tools, but mostly we will write our own tools (or modify existing tools) using the programming language Python (and make use of the huge amount of Python modules others already wrote). But aren’t there existing tools? Big Data and Automated Content Analysis Damian Trilling
  • 26. What is Big Data? Methods What have others done? What can we do? The schedule Which tools? The methods The tools we use for this Some existing tools, but mostly we will write our own tools (or modify existing tools) using the programming language Python (and make use of the huge amount of Python modules others already wrote). But aren’t there existing tools? — sometimes. But look at this example: Big Data and Automated Content Analysis Damian Trilling
  • 27. What is Big Data? Methods What have others done? What can we do? The schedule Which tools? Why program your own tool? Big Data and Automated Content Analysis Damian Trilling
  • 28. What is Big Data? Methods What have others done? What can we do? The schedule Which tools? Why program your own tool? If the task would have been done with a (commercial) tool, we can only research what the tool allows us to do (⇒ our discussion from some minutes ago). Big Data and Automated Content Analysis Damian Trilling
  • 29. What is Big Data? Methods What have others done? What can we do? The schedule Which tools? Why program your own tool? If the task would have been done with a (commercial) tool, we can only research what the tool allows us to do (⇒ our discussion from some minutes ago). Luckily, the problem is easily solved The task was done with a self-written Python program. We change the line lengte_list.append(len(row[textcolumn])) to lengte_list.append(len(row[textcolumn].split())) Big Data and Automated Content Analysis Damian Trilling
  • 30. What is Big Data? Methods What have others done? What can we do? The schedule Which tools? Why program your own tool? Moreover, the tools we use can limit the range of questions that might be imagined, simply because they do not fit the affordances of the tool. Not many researchers themselves have the ability or access to other researchers who can build the required tools in line with any preferred enquiry. This then introduces serious limitations in terms of the scope of research that can be done. Vis, 2013 Big Data and Automated Content Analysis Damian Trilling
  • 31. What is Big Data? Methods What have others done? What can we do? The schedule Which tools? More considerations Assuming that science should be transparent and reproducible Big Data and Automated Content Analysis Damian Trilling
  • 32. What is Big Data? Methods What have others done? What can we do? The schedule Which tools? More considerations Assuming that science should be transparent and reproducible, we should use tools that are • platform-independent • free (as in beer and as in speech, gratis and libre) • which implies: open source Big Data and Automated Content Analysis Damian Trilling
  • 33. What is Big Data? Methods What have others done? What can we do? The schedule Which tools? More considerations Assuming that science should be transparent and reproducible, we should use tools that are • platform-independent • free (as in beer and as in speech, gratis and libre) • which implies: open source This ensures it can our research (a) can be reproduced by anyone, and that there is (b) no black box that no one can look inside Big Data and Automated Content Analysis Damian Trilling
  • 34. What is Big Data? Methods What have others done? What can we do? The schedule Which tools? Why program your own tool? [...] these [commercial] tools are often unsuitable for academic purposes because of their cost, along with the problematic ‘black box’ nature of many of these tools. Vis, 2013 [...] we should resist the temptation to let the opportunities and constraints of an application or platform determine the research question [...] Mahrt & Scharkow, 2013, p. 30 Big Data and Automated Content Analysis Damian Trilling
  • 35. What is Big Data? Methods What have others done? What can we do? The schedule What have others done? Big Data and Automated Content Analysis Damian Trilling
  • 36. What is Big Data? Methods What have others done? What can we do? The schedule Online news sharing Online news sharing Castillo, El-Haddad, Pfeffer, & Stempeck, 2013 Big Data and Automated Content Analysis Damian Trilling
  • 37. What is Big Data? Methods What have others done? What can we do? The schedule Online news sharing Online news sharing The question “We describe the interplay between website visitation patterns and social media reactions to news content. [. . . ] We also show that social media reactions can help predict future visitation patterns early and accurately.” (p.1) Castillo, El-Haddad, Pfeffer, & Stempeck, 2013 Big Data and Automated Content Analysis Damian Trilling
  • 38. What is Big Data? Methods What have others done? What can we do? The schedule Online news sharing Online news sharing The method • data set 1: log files provided by Al Jazeera • data set 2: Facebook and Twitter API • analysis: link 1 and 2 and estimate the relationships Castillo, El-Haddad, Pfeffer, & Stempeck, 2013 Big Data and Automated Content Analysis Damian Trilling
  • 40. What is Big Data? Methods What have others done? What can we do? The schedule Online news sharing Online news sharing The issues • You need that guy at Al Jazeera • You need the infrastructure to cope with the data • Very much tailored to one outlet Castillo, El-Haddad, Pfeffer, & Stempeck, 2013 Big Data and Automated Content Analysis Damian Trilling
  • 41. What is Big Data? Methods What have others done? What can we do? The schedule Partisan asymmetries Partisan asymmetries Conover, Gonçalves, Flammini, & Menczer, 2012 Big Data and Automated Content Analysis Damian Trilling
  • 42. What is Big Data? Methods What have others done? What can we do? The schedule Partisan asymmetries Partisan asymmetries The question How does the Twitter behavior differ between right-wing and left-wing users? Conover, Gonçalves, Flammini, & Menczer, 2012 Big Data and Automated Content Analysis Damian Trilling
  • 43. What is Big Data? Methods What have others done? What can we do? The schedule Partisan asymmetries Partisan asymmetries The method Starting with two hashtags (one used by progressives, one used by conservatives), 55 co-occuring hashtags were identified. Identification of follower networks, retweet networks, mention networks within tweets with these hashtags. Conover, Gonçalves, Flammini, & Menczer, 2012 Big Data and Automated Content Analysis Damian Trilling
  • 45. What is Big Data? Methods What have others done? What can we do? The schedule Partisan asymmetries Partisan asymmetries The issues • The two seeds #tcot and #p2 oversample (extremly) partisan content, inevitably leading to the structure in Figure 3. • But maybe no problem, as the rest of the paper aims to compare these groups. • Not an empirical problem, but still: What do we learn exactly from this study apart from the – interesting – case? Conover, Gonçalves, Flammini, & Menczer, 2012 Big Data and Automated Content Analysis Damian Trilling
  • 46. What is Big Data? Methods What have others done? What can we do? The schedule Considerations regarding feasibility To which extent could we conduct such studies? Big Data and Automated Content Analysis Damian Trilling
  • 47. What is Big Data? Methods What have others done? What can we do? The schedule Considerations regarding feasibility What can we do? Cool research, sure, but what can we do? • Dependency from third parties: scraping < API < server-side implementation • Restrictions (e.g., Twitter: sprinkler, garden hose, fire hose) ⇒ Vis, 2013: Data are made! • We can’t just trust the numbers. Some tasks require human coders – or a qualitative approach, at least as a pre-study. Big Data and Automated Content Analysis Damian Trilling
  • 48. What is Big Data? Methods What have others done? What can we do? The schedule Considerations regarding feasibility What can we do? pros, cons, and feasibility • APIs (Twitter, Facebook, . . . ) • server-side implementations (⇒ Al Jazeera-example) • scraping • client-side log files • "traditional" methods (surveys etc.) Big Data and Automated Content Analysis Damian Trilling
  • 49. What is Big Data? Methods What have others done? What can we do? The schedule Considerations regarding feasibility What can we do? What helps us answer our questions? 1 Draft a RQ. 2 Think of different ways to approach it 3 Think of different data sources. 4 Think of different analyses. Big Data and Automated Content Analysis Damian Trilling
  • 50. What is Big Data? Methods What have others done? What can we do? The schedule Considerations regarding feasibility What can we do? What helps us answer our questions? 1 Draft a RQ. 2 Think of different ways to approach it 3 Think of different data sources. 4 Think of different analyses. And then: 1 Check what the technical possibilities are. (e.g., is there an API? How can we get the data?) 2 Re-evaluate all steps. Big Data and Automated Content Analysis Damian Trilling
  • 51. What is Big Data? Methods What have others done? What can we do? The schedule Examples from last year Some examples from last year’s course Questions • Are (popular) apps for children in the Apple App Store framed in terms of education or entertainment? • How can house prices on funda.nl be predicted? • Where are those who edit Wikipedia-entries about companies geographically located, related to the company’s HQ? • How do embassies [or top universities] communicate on Twitter? • How is the tone in newspaper election coverage related to the tone on Twitter? Big Data and Automated Content Analysis Damian Trilling
  • 52. What is Big Data? Methods What have others done? What can we do? The schedule The schedule Big Data and Automated Content Analysis Damian Trilling
  • 53. What is Big Data? Methods What have others done? What can we do? The schedule The schedule Each week In general: A lecture (Monday) and a lab session (Wednesday). Each week one method. Examinations A mid-term take-home exam in week 5 and an individual research project on which you work during the whole course. Big Data and Automated Content Analysis Damian Trilling
  • 54. What is Big Data? Methods What have others done? What can we do? The schedule Next meetings Next meetings Big Data and Automated Content Analysis Damian Trilling
  • 55. What is Big Data? Methods What have others done? What can we do? The schedule Next meetings Week 2 No meeting on Monday (Easter) Wednesday, 8 April: Lab Session Getting to know Python better. Do not forget to finish the exercises in chapter 2 and 3 in advance. Big Data and Automated Content Analysis Damian Trilling
  • 56. What is Big Data? Methods What have others done? What can we do? The schedule Next meetings Week 3: Data harvesting and storage Monday, 13–4 A conceptual overview of APIs, scrapers, crawlers, RSS-feeds, databases, and different file formats Wednesday, 15–4 Writing some first data collection scripts Big Data and Automated Content Analysis Damian Trilling