BDACA - Lecture3

Last week’s exercise Data harvesting and storage Storing data Next meetings
Big Data and Automated Content Analysis
Week 3 – Monday
Data harvesting and storage
Anne Kroon & Damian Trilling
a.c.kroon@uva.nl
@annekroon
d.c.trilling@uva.nl
@damian0604
www.damiantrilling.net
Afdeling Communicatiewetenschap
Universiteit van Amsterdam
19 February 2018
Big Data and Automated Content Analysis Damian Trilling

Today
1 Last week’s exercise
Step by step
Concluding remarks
2 Data harvesting and storage
APIs
RSS feeds
Scraping and crawling
Parsing text ﬁles
3 Storing data
CSV tables
JSON and XML
4 Next meetings

Last week’s exercise
Discussing the code

Step by step
Reading a JSON ﬁle into a dict, looping over the dict
Task 1: Print all titles of all videos
1 import json
2
3 with open("/home/damian/pornexercise/xhamster.json") as fi:
4 data=json.load(fi)
5
6 for k,v in data.items()):
7 print (v["title"])

Step by step
1 import json
2
5
NB: You have to know (e.g., by reading the documentation of the dataset)
that the key is called title

Step by step
1 import json
2
5
NB: You have to know (e.g., by reading the documentation of the dataset)
that the key is called title
NB: data is in fact a dict of dicts, such that each value v is another dict.
For each of these dicts, we retrieve the value that corresponds to the key title

Step by step
What to do if you do not know the structure of the
dataset?
Inspecting your data: use the functions type() and len() and/or
the dictionary method .keys()
1 len(data)
2 type(data)
3 data.keys()

Step by step
dataset?
Inspecting your data: use the functions type() and len() and/or
the dictionary method .keys()
1 len(data)
2 type(data)
3 data.keys()
len() returns the number of items of an object; type() returns the type
object; .keys() returns a list of all available keys in the dictionary

Step by step
dataset?
Inspecting your data: use the module pprint
1 from pprint import pprint
2
3 pprint(data)

Step by step
For the sake of completeness. . .
.items() returns a key-value pair, that’s why we need to assign
two variables in the for statement.
These alternatives would also work:
1 for v in data.values()):
2 print(v["title"])
1 for k in data: #or: for k in data.keys():
2 print(data[k]["title"])
Do you see (dis-)advantages?

Step by step
Working with a subset of the data
What to do if you want to work with a smaller subset of the data?
Taking a random sample of 10 items in a dict:
1 import random
2 mydict_short = dict(random.sample(mydict.items(),10))
Taking the ﬁrst 10 elements in a list:
1 mylist_short = mylist[:10]

Step by step
Initializing variables, merging two lists, using a counter
Task 2: Average tags per video and most frequently used tags
1 from collections import Counter
2
3 alltags=[]
4 i=0
5 for k,v in data.items():
6 i+=1
7 alltags+=v["channels"] # or: alltags.extend(v["channels"])
8
9 print(len(alltags),"tags are describing",i,"different videos")
10 print("Thus, we have an average of",len(alltags)/i,"tags per video")
11
12 c=Counter(alltags)
13 print (c.most_common(100))
(there are other, more eﬃcient ways of doing this)

Step by step
Nesting blocks, using a defaultdict to count, error handling
Task 3: What porn category is most frequently commented on?
1 from collections import defaultdict
2
3 commentspercat=defaultdict(int)
5 for tag in v["channels"]:
6 try:
7 commentspercat[tag]+=int(v["nb_comments"])
8 except:
9 pass
10 print(commentspercat)
11 # if you want to print in a fancy way, you can do it like this:
12 for tag in sorted(commentspercat, key=commentspercat.get, reverse=True):
13 print( tag,"t", commentspercat[tag])
A defaultdict is a normal dict, with the diﬀerence that the type of each value is
pre-deﬁned and it doesn’t give an error if you look up a non-existing key

Step by step
Nesting blocks, using a defaultdict to count, error handling
Task 3: What porn category is most frequently commented on?
1 from collections import defaultdict
2
3 commentspercat=defaultdict(int)
5 for tag in v["channels"]:
6 try:
7 commentspercat[tag]+=int(v["nb_comments"])
8 except:
9 pass
10 print(commentspercat)
11 # if you want to print in a fancy way, you can do it like this:
12 for tag in sorted(commentspercat, key=commentspercat.get, reverse=True):
13 print( tag,"t", commentspercat[tag])
A defaultdict is a normal dict, with the diﬀerence that the type of each value is
pre-deﬁned and it doesn’t give an error if you look up a non-existing key
NB: In line 7, we assume the value to be an int, but the datasets sometimes contains
the string “NA” instead of a string representing an int. That’s why we need the
try/except construction

Step by step
Adding elements to a list, sum() and len()
Task 4: Average length of descriptions
1 length=[]
3 length.append(len(v["description"]))
4
5 print ("Average length",sum(length)/len(length))

Step by step
Merging vs appending
Merging:
1 l1 = [1,2,3]
2 l2 = [4,5,6]
3 l1 = l1 + l2
4 print(l1)
gives [1,2,3,4,5,6]
Appending:
1 l1 = [1,2,3]
2 l2 = [4,5,6]
3 l1.append(l2)
4 print(l1)
gives [1,2,3,[4,5,6]]
l2 is seen as one element to append to l1

Step by step
Tokenizing with .split()
Task 5: Most frequently used words
1 allwords=[]
3 allwords+=v["description"].split()
4 c2=Counter(allwords)
5 print(c2.most_common(100))
.split() changes a string to a list of words.
"This is cool”.split()
results in
["This", "is", "cool"]

Concluding remarks
Concluding remarks
Make sure you fully understand the code!
Re-read the corresponding chapters
PLAY AROUND!!!

An overview of APIs, scrapers, crawlers, RSS-feeds, and diﬀerent
ﬁle formats

APIs
Collecting data:
APIs

APIs
APIs

APIs
Querying an API
1 # contact the Twitter API
2 auth = OAuth(access_key, access_secret, consumer_key, consumer_secret)
3 twitter = Twitter(auth = auth)
4
5 # get all info about the user ’username’)
6 tweepinfo=twitter.users.show(screen_name=username)
7
8 # save his bio statement to the variable bio
9 bio=tweepinfo["description"])
10
11 # save his location to the variable location
12 location=tweepinfo["location"]
(abbreviated Python example of how to query the Twitter REST API)

APIs
Who oﬀers APIs?
The usual suspects: Twitter, Facebook, Google – but also Reddit,
Youtube, . . .

APIs
Who oﬀers APIs?
Youtube, . . .
If you ever leave your bag on a bus on Chicago

APIs
Who oﬀers APIs?
Youtube, . . .
If you ever leave your bag on a bus on Chicago
. . . but do have Python on your laptop, watch this:
https://www.youtube.com/watch?v=RrPZza_vZ3w.
That guy queries the Chicago bus company’s API to calculate
when exactly the vehicle with his bag arrives the next time at the
bus stop in front of his oﬃce.
(Yes, he tried calling the help desk before, but they didn’t know. He got his bag back.)

APIs
APIs
Pro
• Structured data
• Easy to process automatically
• Can be directy embedded in your script
Morstatter, F., Pfeﬀer, J., Liu, H., & Carley, K. M. (2013). Is the sample good enough? Comparing data from
Twitters Streaming API with Twitters Firehose. International AAAI Conference on Weblogs and Social Media.

APIs
APIs
Pro
• Structured data
• Easy to process automatically
• Can be directy embedded in your script
Con
• Often limitations (requests per minute, sampling, . . . )
• You have to trust the provider that he delivers the right
content (⇒ Morstatter e.a., 2013)
• Some APIs won’t allow you to go back in time!
Morstatter, F., Pfeﬀer, J., Liu, H., & Carley, K. M. (2013). Is the sample good enough? Comparing data from
Twitters Streaming API with Twitters Firehose. International AAAI Conference on Weblogs and Social Media.

APIs
So we have learned that we can access an API directly.
But what if we have to do so 24/7?

APIs
So we have learned that we can access an API directly.
But what if we have to do so 24/7?
Collecting tweets with a tool running on a server that does query
the API 24/7.

APIs

APIs
It queries the API and stores the result
It continuosly calls the Twitter-API and saves
all tweets containing speciﬁc hashtags to a
MySQL-database.
You tell it once which data to collect – and
wait some months.

APIs
Retrieving the data for analysis
You could access the MySQL-database directly.
Generally raw data needs to be cleaned and
parsed, before you can start analyzing and
visualizing

RSS feeds
Collecting data:
RSS feeds

RSS feeds
RSS feeds
What’s that?
• A structured (XML) format in which for example news sites
and blogs oﬀer their content

RSS feeds
RSS feeds
What’s that?
• You get only the new news items (and that’s great!)

RSS feeds
RSS feeds
What’s that?
• Title, teaser (or full text), date and time, link

RSS feeds
RSS feeds
What’s that?
• Title, teaser (or full text), date and time, link
http://www.nu.nl/rss

RSS feeds
RSS feed

RSS feeds
Parsing RSS feeds
1
2 tree = fromstring(htmlsource)
3
4 # parsing article teaser
5 try:
6 teaser=tree.xpath(’//*[@class="item-excerpt"]//text()’)[0]
7 except:
8 logger.debug("Could not parse article teaser.")
9 teaser=""
10
11 # parsing article text
12 try:
13 text=" ".join(tree.xpath(’//*[@class="block-wrapper"]/div[@class="
block-content"]/p//text()’)).strip()
14 except:
15 text = ""
16 logger.warning("Could not parse article text")
(abbreviated Python example of how to parse the RSS feed of NU.nl)

RSS feeds
RSS feeds
Pro
• One protocol for all services
• Easy to use
Con
• Full text often not included, you have to download the link
separately (⇒ Problems associated with scraping)
• You can’t go back in time! But we have archived a lot of RSS
feeds

Collecting data:

If you have no chance of getting already structured data via
one of the approaches above

• Download web pages, try to identify the structure yourself
• You have to parse the data

• Download web pages, try to identify the structure yourself
• You have to parse the data
• Can get very complicated (depending on the speciﬁc task),
especially if the structure of the web pages changes
Further reading:
http://scrapy.org
https:
//github.com/anthonydb/python-get-started/blob/master/5-web-scraping.py

Parsing text ﬁles
Collecting data:
Parsing text ﬁles

Parsing text ﬁles
For messy input data or for semi-structured data
Guiding question: Can we identify some kind of pattern?

Parsing text ﬁles
Examples
• Lewis, Zamith, & Hermida (2013) had a corrupt CSV-ﬁle
Lewis, S. C., Zamith, R., & Hermida, A. (2013). Content analysis in an era of Big Data: A hybrid approach to
computational and manual methods. Journal of Broadcasting & Electronic Media, 57(1), 3452.
doi:10.1080/08838151.2012.761702

Parsing text ﬁles
Examples
• LexisNexis gives you a chunk of text (rather than, e.g., a
structured JSON or XML object)

Parsing text ﬁles
Examples
• LexisNexis gives you a chunk of text (rather than, e.g., a
structured JSON or XML object)
But in both cases, as long as you can ﬁnd any pattern or structure
in it, you can try to write a Python script to parse the data.

1 tekst={}
2 section={}
3 length={}
4 ...
5 ...
6 with open(bestandsnaam) as f:
7 for line in f:
8 line=line.replace("r","")
9 if line=="n":
10 continue
11 matchObj=re.match(r"s+(d+) of (d+) DOCUMENTS",line)
12 if matchObj:
13 artikelnr= int(matchObj.group(1))
14 tekst[artikelnr]=""
15 continue
16 if line.startswith("SECTION"):
17 section[artikelnr]=line.replace("SECTION: ","").rstrip("n")
18 elif line.startswith("LENGTH"):
19 length[artikelnr]=line.replace("LENGTH: ","").rstrip("n")
20 ...
21 ...
22 ...
23
24 else:
25 tekst[artikelnr]=tekst[artikelnr]+line

CSV tables
Storing data:
CSV tables

CSV tables
CSV-ﬁles
Always a good choice
• All programs can read it
• Even human-readable in a simple text editor:
• Plain text, with a comma (or a semicolon) denoting column
breaks
• No limits regarging the size
• But: several dialects (e.g., , vs. ; as delimiter)

CSV tables
A CSV-ﬁle with tweets
1 text,to_user_id,from_user,id,from_user_id,iso_language_code,source,
profile_image_url,geo_type,geo_coordinates_0,geo_coordinates_1,
created_at,time
2 :-) #Lectrr #wereldleiders #uitspraken #Wikileaks #klimaattop http://t.
co/Udjpk48EIB,,henklbr,407085917011079169,118374840,nl,web,http://
pbs.twimg.com/profile_images/378800000673845195/
b47785b1595e6a1c63b93e463f3d0ccc_normal.jpeg,,0,0,Sun Dec 01
09:57:00 +0000 2013,1385891820
3 Wat zijn de resulaten vd #klimaattop in #Warschau waard? @EP_Environment
ontmoet voorzitter klimaattop @MarcinKorolec http://t.co/4
Lmiaopf60,,Europarl_NL,406058792573730816,37623918,en,<a href="http
://www.hootsuite.com" rel="nofollow">HootSuite</a>,http://pbs.twimg
.com/profile_images/2943831271/
b6631b23a86502fae808ca3efde23d0d_normal.png,,0,0,Thu Nov 28
13:55:35 +0000 2013,1385646935

JSON and XML
Storing data:
JSON and XML

JSON and XML
JSON and XML
Great if we have a nested data structure

JSON and XML
JSON and XML
• Items within feeds

JSON and XML
JSON and XML
• Personal data within authors within books

JSON and XML
JSON and XML
• Personal data within authors within books
• Tweets within followers within users

JSON and XML
A JSON object containing GoogleBooks data
1 {’totalItems’: 574, ’items’: [{’kind’: ’books#volume’, ’volumeInfo’: {’
publisher’: ’"O’Reilly Media, Inc."’, ’description’: u’Get a
comprehensive, in-depth introduction to the core Python language
with this hands-on book. Based on author Mark Lutzu2019s popular
training course, this updated fifth edition will help you quickly
write efficient, high-quality code with Python. Itu2019s an ideal
way to begin, whether youu2019re new to programming or a
professional developer versed in other languages. Complete with
quizzes, exercises, and helpful illustrations, this easy-to-follow,
self-paced tutorial gets you started with both Python 2.7 and 3.3
u2014 the
2 ...
3 ...
4 ’kind’: ’books#volumes’}

JSON and XML
An XML object containing an RSS feed
1 ...
2 <item>
3 <title>Agema doet aangifte tegen Samsom en Spekman</title>
4 <link>http://www.nu.nl/politiek/3743441/agema-doet-aangifte-
samsom-en-spekman.html</link>
5 <guid>http://www.nu.nl/politiek/3743441/index.html</guid>
6 <description>PVV-Kamerlid Fleur Agema gaat vrijdag aangifte doen
tegen PvdA-leider Diederik Samsom en PvdA-voorzitter Hans
Spekman wegens uitspraken die zij hebben gedaan over
Marokkanen. </description>
7 <pubDate>Thu, 03 Apr 2014 21:58:48 +0200</pubDate>
8 <category>Algemeen</category>
9 <enclosure url="http://bin.snmmd.nl/m/m1mxwpka6nn2_sqr256.jpg"
type="image/jpeg" />
10 <copyrightPhoto>nu.nl</copyrightPhoto>
11 </item>
12 ...

JSON and XML
It’s the same as our “dict of dicts”/“dict of lists”/. . . data
model!

Next meetings

Next meetings
Wednesday, 21–3
Writing some ﬁrst data collection scripts

BDACA - Lecture3

More Related Content

What's hot (20)

Similar to BDACA - Lecture3 (20)

More from Department of Communication Science, University of Amsterdam (17)

Recently uploaded (20)

BDACA - Lecture3