Session 03 acquiring data

Editor's Notes

• #2: Today we’re looking at the types of data that are hiding online, and how to bring them out of hiding and into your data science code.
• #3: So let’s begin. Here are the 6 things we’ll talk about today.
• #4: Your first problem is finding the data to help answer your questions.
• #5: A quick recap: these are some of the places where you can find data. Some of them are harder to process than others, but they all contain data.
• #6: And here are some places to find them - there’s a longer list in the references folder.
• #7: Development data isn’t always easy to obtain: you might have to create your own, by asking people to contribute information to you through crowdsourcing, in-person surveys, mobile surveys etc.
• #8: You might also need to generate data for your problem by using sensors.
• #9: Selection bias = non-random selection of individuals. One example of this is pothole reporting: potholes are more generally reported in more-affluent areas, by people who have both the smartphone apps and the time and energy to report. Missing data = data that you don’t have. You need to be aware of this, and take account of it. If you need more persuading, read about Wald and the bullethole problem.
• #10: There are many datafile types - here’s a guide to some of them.
• #11: Tables typically have rows and columns; relational data is typically hierarchical, e.g. can’t be easily converted into row-column form.
• #12: CSVs are the workhorse of datatypes: almost every data application can read them in.
• #13: Converting JSON to CSV: Use a conversion website (e.g. http://www.convertcsv.com/json-to-csv.htm) Write some Python code
• #14: Converting XML to CSV: Use a conversion website, e.g. http://www.convertcsv.com/xml-to-csv.htm Write code
• #15: One way to obtain data is through an application programming interface (API).
• #16: More about open APIs: https://en.wikipedia.org/wiki/Open_API
• #17: REST = Representational State Transfer; a human-readable way to ask APIs for information. At the top is a RESTful URL (web address); you can type this directly into an internet browser to get a datafile. This address has 3 parts to it: The base url, api.worldbank.org a description of what you’re looking for - in this case, the total rural population for all countries in the world Some more details, including filters (only data between 2000 and 2015) and data formats. Try this address, and try “&format=json” instead of “&format=csv” at the end.
• #18: REST = Representational State Transfer; a human-readable way to ask APIs for information. At the top is a RESTful URL (web address); you can type this directly into an internet browser to get a datafile. This address has 3 parts to it: The base url, api.worldbank.org a description of what you’re looking for - in this case, the total rural population for all countries in the world Some more details, including filters (only data between 2000 and 2015) and data formats. Try this address, and try “&format=json” instead of “&format=csv” at the end.
• #20: The Python requests library is useful for calling APIs from a python program (e.g. so you can then use or save the information returned from them). If anything goes wrong, try r.status_code You’re maybe wondering how to get this json data into a file. Here’s the code for that: import json fout = open('mynewdata.json', 'w') json.dump(jsondata, fout)
• #21: See https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
• #24: Here are places to look first: the website that data’s in, for file copies of the data the website that data’s in, for an api (http://api.theirsitename.com/, http://theirsitename.com/api, Google “site:theirsitename.com api”) related sites for file copies and apis Community warehouses (scraperwiki.com, datahub.io etc.) for other peoples’ scrapers
• #25: Big PDFs. And we’ll need to get the data out of them. This is where PDF scrapers come in.
• #29: Web scraping is the process of extracting data from webpages. If you open a webpage (e.g. https://en.wikipedia.org/wiki/List_of_U.S._states_and_territories_by_population) and click on “view source”, you’ll see the view that a computer has of that page. This is where the data is hiding…
• #31: The pattern for this is: =importHtml(“your-weburl”, “table”, yourtablenumber) More: www.mulinblog.com/basic-web-scraping-data-visualization-using-google-spreadsheets/
• #32: You’ve already used the Requests library to grab data from the web. Mechanise and Cookielib
• #34: Your exercises were all built into the class. But if you want more…
• #35: Most data science and visualisation programs can read CSV data, so if you can easily convert data to that, good. There are websites that will convert to csv; you can also do this by reading data in one format, and writing it out in another. The Pandas library is very helpful for reading in one format, and writing in another, if the data is row-column.
• #36: We’ll cover data cleaning later, but if you want to try next week’s visualisation techniques on your own data, it will need to at least be normalised. Here’s what we mean by this (and Tableau has a tool for doing this: see http://kb.tableau.com/articles/knowledgebase/denormalize-data).
• #37: Most data science and visualisation programs can read CSV data, so if you can easily convert data to that, good. There are websites that will convert to csv; you can also do this by reading data in one format, and writing it out in another. The Pandas library is very helpful for reading in one format, and writing in another, if the data is row-column.
• #38: Your exercises were all built into the class. But if you want more…