Data, data, data
- 1. Data data data Session III
- 2. I. What do we mean when we talk about data and where does data come from? II. What is data science? III. Where do you find the data you need? IV. Science Module: Niche Modelling
- 3. What do we mean when we talk about data? Session III
- 4. We ask many many questions about the world around us.
- 5. To answer those questions accurately requires a body of data and a set of tools to perform analyses that will lead us toward an answer
- 6. “ The goal is to transform data into information, and information into insight” Carly Fiorina
- 8. Data are the raw facts about our world
- 11. Thomas Nylen & Andrew Fountain (PSU), NASA, NSF
- 12. A lot of this data is available for you to use
- 13. Where does the data come from? Government http://www.data.gov / http://data.gov.uk/ http://www.census.gov/ Scientific research http://www.ncbi.nlm.nih.gov/genbank/ http://www.gbif.org/ http://earthengine.googlelabs.com Semi-automated and large scale collections http://eospso.gsfc.nasa.gov/ http://www.airquality.co.uk/autoinfo.php http://www.statistics.gov.uk/ For profit http://www.flickr.com/ http://www.google.com/trends http://www.facebook.com/data Citizens http://www.wikipedia.org / http://protectedplanet.net/ http://www.openstreetmap.org/
- 14. Direct data
- 15. Direct data Government http://www.data.gov / http://data.gov.uk/ http://www.census.g ov/ Scientific research http://www.ncbi.nlm.nih.gov/genbank/ http://www.gbif.org/ http://earthengine.googlelabs.com Semi-automated and large scale collections http://eospso.gsfc.nasa.gov / http://www.airquality.co.uk/autoinfo.php http://www.statistics.gov.uk/ For profit http://www.flickr.com/ http://www.google.com/trends http://www.facebook.com/data Citizens http://www.wikipedia.org / http://protectedplanet.net/ http://www.openstreetmap .org/
- 16. photo by solarnu on Flickr
- 17. Indirect data
- 18. Indirect data Government http://www.data.gov / http://data.gov.uk/ http://www.census.gov/ Scientific research http://www.ncbi.nlm.nih.gov/genbank/ http://www.gbif.org / http://earthengine.googlelabs.com Semi-automated and large scale collections http://eospso.gsfc.nasa.gov/ http://www.airquality.co.uk/autoinfo.php http://www.statistics.gov.uk/ For profit http://www.flickr.com / http://www.google.com/trends http://www.facebook.com/data Citizens http://www.wikipedia.org / http://protectedplanet.net/ http://www.openstreetmap.org/
- 19. Source: Google Flu Trends (http://www.google.org/flutrends)
- 20. There has been an almost incomprehensible growth in digital data
- 22. 5 Megabytes - a high resolution photo 5 Gigabytes - just more than a common DVD stores 1 Terabyte - size of common home computer harddrive 15 Petabyte - data produced at CERN each year 5 Exabytes ~ every word spoken by humans 1.2 Zettabytes ~ Digital Universe in 2010 1,180,591,620,717,411,303,424 bytes or 2 70
- 23. Okay, so there is a lot of data!
- 24. Is all this data free to use?
- 25. No
- 26. What you can do with data is largely dictated by license and copyright
- 27. Because of this many organizations have begun advocating and practicing 'open data' 'open data'
- 28. What is open data?
- 29. Free and unrestricted access to data
- 31. Kepler Data
- 32. '...survey a portion of our region of the Milky Way galaxy to discover dozens of Earth-size planets in or near the habitable zone and determine how many of the billions of stars in our galaxy have such planets...'
- 33. Launch a telescope into space for orbit around the sun Collect data from thousands of stars Spend 3 months trying to detect the presence of plants in orbit around these stars RELEASE THE DATA FOR USE BY ANYONE!!!
- 36. Different but related ideas Open Government data.gov, data.gov.uk Open Access plos.org Open Source Apache, Firefox
- 37. What is data science? Part II
- 38. Data analysis is a body of methods that help to describe facts, detect patterns, develop explanations, and test hypotheses. It is used in all of the sciences. It is used in business, in administration, and in policy. Levine, 1997, Introduction to Data Analysis: The Rules of Evidence
- 39. “ The goal is to transform data into information, and information into insight” Carly Fiorina
- 40. It is a set of skills performed often but not exclusively by scientists
- 41. The availability of data on the internet is making data analysis accessible to anyone
- 43. Part III Where do you find the data you need?
- 44. My community has a particular set of data that we rely on very often
- 49. During our class, to find the data you need...
- 50. Search First
- 51. Open government data, Large published datasets such as Wikipedia, Flickr, or public FusionTables, Natural sciences resources, Weather, Atmosphere and Geographic datasets
- 52. See the growing list of datasets our class will uncover https://github.com/andrewxhill/DMID/wiki/Datasets
- 53. Now you have found it, how do you get it?
- 54. Maybe you can download it directly from the website. Pay attention to what formats you find, some will be easier for you to use than others.
- 55. Sometimes, APIs, or application programming interfaces are available. These are probably for people with a bit more programming experience, but if you need the data ask the instructors and we might be able to help
- 57. Scraping Generally the hardest method, as it means programmatically pulling data from sources not necessarily designed to have data pulled from.
- 59. Linked data This is an evolution of both scraping and APIs, where many web resources are now designed to be both human readable and programatically navigable.
- 61. Remember that all of these data sources have different formats and potential sources of error, we will have a full session on data preparation, cleaning, and analysis
- 62. Remember that data comes in many shapes and sizes, be aware of what you find General formats - png, jpg, xls, doc Categorical and storage - csv, sql Geographic - shp, tif, asc
- 63. XLS VS CSV
- 64. How can we join datasets together?
- 65. Afghanistan 3 Bolivia 2 Guyana 1 Palau 4 GUY 1.03 BOL 1.34 PLW 0 AFG 19.03
- 66. Afghanistan 3 Bolivia 2 Guyana 1 Palau 4 GUY 1.03 BOL 1.34 PLW 0.20 AFG 1.93 Afghanistan AFG Bolivia BOL Guyana GUY Palau AFG
- 67. If you start working with data, really interesting things can appear.
- 70. GBIF.org
- 71. from Eric Fisher on Flickr
Editor's Notes
- #2: \n
- #3: \n
- #4: \n
- #5: Humans are naturally curious about our world. \nIn addition, we have social, economic, and personal motivations to understand how and why the world around us changes\n
- #6: \n
- #7: CEO HP\n
- #8: 1993 David Vaughan British Anta Survey\nPredicted breaking in 30yrs\n2008 he conceded that his estimates had been to conservative\n
- #9: \n
- #10: \n
- #11: \n
- #12: Automated weather station \nLake Vida Antarctica\n19 Meters of Ice\n2500 years\n\n
- #13: Automated weather station \nLake Vida Antarctica\n19 Meters of Ice\n2500 years\n\n
- #14: \n
- #15: \n
- #16: \n
- #17: \n
- #18: \n
- #19: \n
- #20: \n
- #21: \n
- #22: Genbank\n
- #23: \n
- #24: \n
- #25: \n
- #26: \n
- #27: \n
- #28: \n
- #29: \n
- #30: \n
- #31: \n
- #32: \n
- #33: \n
- #34: \n
- #35: Not everyone has the means to take this data and study it in a sophisticated analysis\nBut a lot of people are interested in space, astronomy, and our universe\nSo how could Kepler insure that these people could help them and have fun\n
- #36: \n
- #37: \n
- #38: \n
- #39: \n
- #40: \n
- #41: \n
- #42: \n
- #43: \n
- #44: \n
- #45: \n
- #46: \n
- #47: \n
- #48: \n
- #49: \n
- #50: \n
- #51: \n
- #52: \n
- #53: \n
- #54: \n
- #55: \n
- #56: \n
- #57: \n
- #58: \n
- #59: \n
- #60: You will not likely encounter this anytime over the next couple of weeks\nbut it is best to be aware of\n
- #61: \n
- #62: \n
- #63: \n
- #64: XLS might be easier for you to navigate\nopen it in excel, sort columns, search for what you want\nbut CSV will almost always be easier to use anyplace other than excel\nsmaller, compact, but easily parsable\n
- #65: This is not the linked data\n
- #66: ISO - International organization for standards\n\n
- #67: \n
- #68: \n
- #69: \n
- #70: \n
- #71: \n
- #72: \n