Big data and open access: a collision course for science
2 likes1,167 views
This document summarizes key points from a keynote talk on big data and open access given at the 2nd International LSDMA Symposium. It discusses how open access to data, along with open cleaning and analysis of data, can help address major societal challenges related to climate change, food security, and new economies. However, rapid growth in data and expectations is creating tensions between enabling new discoveries and addressing privacy concerns. Technical solutions exist but cultural and policy changes may also be needed to fully realize the benefits of open data while protecting privacy. The document provides several examples of open data initiatives related to text analysis, urban science, and atmospheric observations.
Big data and open access: a collision course for science
- 1. Keynote talk at 2nd Int’l LSDMA Symposium – The Challenge of Big Data in Science, Karlsruhe, Germany, Sept 2013 Big data and open access: on track for collision of cosmic proportions? Beth Plale, PhD, MBA Director, Data To Insight Center School of Informatics and Computing Indiana University
- 2. Open access, open cleaning, open data yields greatest degree of science advancement on grand societal ques�ons we face
- 3. Open Access “Data is the New Gold” Title of Opening Remarks, Neelie Kroes, VP of EU Commission responsible for Digital Agenda, Press Conference on Open Data Strategy, Dec 2011
- 4. Applied Forces Open access ini�a�ves by federal governments Big Data
- 5. Applied Force Distorts Object Enables societal grand challenges addressed in: à Climate change à Food security à New economies Open access ini�a�ves by federal governments à Grows concerns about privacy of personal data Big Data
- 6. Negative form of tension (tension I) Chilling effect on data sharing where social phenomena involved Social pressure to privacy overwhelm and spill over to non-‐personal data
- 7. Exponential Growth in Data Production
- 8. Similar growth in societal expectations that large societal problems will be solved by more data
- 9. Tension II: Rapid growth in data and expectations yields impossible-toreach success
- 10. Technical barriers to easing tensions but first … DRIVING APPLICATIONS: LIBRARY TEXTS; URBAN SCIENCE; WIND AND WATER
- 11. Hathi Trust Research Center Text mining at scale #HTRC #HathiTrust #HTRC #HathiTrust
- 12. à HathiTrust is large corpus providing opportunity for new forms of computation investigation. à The bigger the data, the less able we are to move it to a researcher’s desktop machine à Future research on large collections will require computation moves to the data, not vice versa
- 13. HTRC Partners Indiana University School of Informatics and Computing Indiana Universities Libraries University of Illinois Graduate School of Library and Information Science University of Illinios Libraries Brandies University Library University of Michigan http://www.hathitrust.org/htrc #HTRC #HathiTrust
- 14. HTRC Non-Consumptive Research Paradigm No action or set of actions on part of users, either acting alone or in cooperation with other users over duration of one or multiple sessions can result in sufficient information gathered from collection of copyrighted works to reassemble pages from collection. Definition disallows collusion between users, or accumulation of material over time. Differentiates human researcher from proxy which is not a user. Users are human beings. #HTRC #HathiTrust
- 15. Topic modeling on author Two topics with identical centralities but separate themes
- 16. Yearly values of a ratio between two wordlists in three different genres. 4,275 volumes. 1700-1899. Underwood et al. Research
- 17. Computation moves to data REST based Web services architecture and protocols Registry of services and algorithms Solr full text index noSQL store as volume store openID authentication Portal front-end, programmatic access SEASR text mining algos 2/4/14 17
- 18. Portal Blacklight SEASR analy�cs service Agent framework Agent instance Agent instance WSO2 registry services, collec�ons, data capsule images HTRC Data API v0.1 WS02 Iden�ty Server Agent instance Agent instance Solr index Task deployment Meandre Orchestra�on Non-consumptive Data capsules NCSA local resources Volume store Volume store (Cassandra) Volume store (Cassandra) (Cassandra) rsync NSF XSEDE Big Red II/IU Quarry Programma�c access e.g., HathiTrust corpus Page/volume tree (file system) 18 University of Michigan
- 19. HTRC: Open Data, Open Access, Open Cleaning? HathiTrust collection (69%) is not open data Constrained by authors who hold copyright to the books Computational analysis is by all accounts “fair use” under US copyright
- 20. HTRC: Open Data, Open Access, Open Cleaning? “Open cleaning” – enhancing OCR and MARC metadata HTRC is opening data and “cleaning” as fully as we can to make the collection useful to scholarly and scientific investigation
- 21. Wind and Water: the hydrologist’s (atmospheric) observational data dilemma Thanks to Jerry Brotzge, PhD meteorology, Oklahoma University * Credit/blame for �tle goes to Beth Plale
- 22. Atmospheric Observing Systems Recent addition of plethora of new observing systems to national US atmosphere observing infrastructure Improves ability to analyze current state of atmosphere, thus allowing new applications in hydrology and biology Challenges in: Data access; unique sensing requirements Data quality, calibrations, and errors Complex and non-uniform metadata
- 23. Use Case Use observational data from 3 different radars: FAA TDWR, WSR-88D, and local X-band (CASA) Feed data through OU-custom QA/calibration workflow. Feed into Vflow hydrological model. Note that Vflow is able to operate on (ingest) the “raw” reflectivity data directly. That is, it does not require the data to be turned into gridded precipitation data. Vflow is unique among hydrology models because of this ability. Done in real time, that is, continuously ingesting data over fixed interval.
- 24. List of Issues for Flood Forecasting using Radar data Problem Cause Poten�al Solu�on Hail contamina�on Assumes high rainfall rate Use of dual-‐pol, QC Bright band Ice at mid-‐levels biases dBZ Real-‐�me QC, 2 radar beams Ground clu�er Wind farms, blockage Use of Neural Net, velocity Radar a�enua�on High-‐frequency radars Real-‐�me QC model, fix Anomalous propaga�on High stable environment Use of Level 1, velocity Velocity de-‐aliasing High velocity returns Real-‐�me QC Radar calibra�on Poor maintenance Post QC Over/under es�ma�on below beam Radar too far from area of interest; undersampled Improved radar sampling; addi�onal sfc input Poor �me sampling Radar 5-‐min volume sampling Improved temporal sampling ET under beam Lack of surface informa�on Addi�onal surface data Spa�al interpola�on Polar to Cartesian coordinates Interpola�on algorithm Use of Reflec�vity Does not measure rain directly Calibra�on against sfc data
- 25. Example Workflow Quality Control Other radar systems (TDWR, CASA) Clear-‐air echoes removed Anomalous propaga�on (AP) removed WSR-‐88D data Clu�er removal Interpola�on from polar to a common Cartesian grid Hail contamina�on removal Velocity de-‐ aliasing Radar calibra�on Mel�ng layer contamina�on removal Undersampling Representa�ve ness Convert radar reflec�vity dBZ to rainfall rate Radar merger (across same network and mul�ple networks) Integrate radar data with satellite, surface observa�ons on grid
- 26. Examine hail contamination in more detail Level II radar data that is widely available (through LDM tool of UCAR in US) has not been “cleaned” of effects of clean air echoes, hail, undersampling, and melting layer contamination Hail has effect of high reflectivity readings and these high readings can be misinterpreted as high rainfall Meteorologists can detect hail easily by eyeballing a visual plot of reflectivity intensities so can go back to Level II data and process by removing hail contamination Meteorologists solve problem through trained eye, and good in-house scripts. What does poor hydrologist do?
- 27. Meterology/Hydrology: Open Data, Open Access, Open Cleaning? Data is open, but how to handle cleaning? A: force all level II data through workflow. Hydrologist uses only processed data (i.e., gridded precipitation data). Advantage: hides details from hydrologist Disadvantage: black box approach reduces trust A: Make “raw” level II data and Q&A workflow tasks available to hydrologist. Advantage: hydrologist can develop high level of trust in data Disadvantage: current metadata not sufficiently described to capture the kinds of Q&A that have been applied
- 28. Urban Science Tag cloud of related tweet topics #smartcityjam thanks to Jennifer Belissent, PhD * Credit/blame for �tle goes to Beth Plale
- 29. Urban Science Harness data from disparate sources with goal of improving city life. Fuses physical, biological, and informational sensing of the city in-situ sensors for environment: light, temperature, pollution Video: pedestrian and vehicular traffic Personal sensors: Fitbit and Up wristbands Internet sources: Twitter feeds, blogs, news articles, crowdsourced sensing Two examples in US Center of Urban Science and Progress, New York University Urban Center for Computation and Data, University of Chicago
- 30. Urban Science Thanks to Physics Today, Sept 2013 Graphic courtesy NYU Center for Urban Science and Progress * Credit/blame for �tle goes to Beth Plale
- 31. Urban science: open data, open access, open cleaning? CUSP is cleaning its own data for integration. Is this being done in way that Chicago can use? Likely not. Temporal streams are relatively simple to understand with even bad metadata. They are observational-physical and observational-social data sources so come with relatively known trust and attribution. What happens when CUSP wants to integrate predictive weather forecasting model results? Weak metadata and attribution can significantly compromise accuracy of results.
- 32. Data Provenance Work of Data To Insight Center at IU, its affiliated faculty and students
- 33. Provenance Core (W3C PROV)
- 35. Provenance for situational analysis of agent based model used in social ecological systems research Village labor sharing for agriculture production in Africa
- 36. Provenance capture AMSR-E data processing pipeline Advanced Microwave Scanning Radiometer (AMSR-‐E) : sensor aboard Aqua satellite; passive microwave radiometer. Observes precipita�on, sea surface temperatures, ice concentra�ons, snow water equivalent, surface wetness, wind speed, atmospheric cloud water, and water vapor. 36 Aug 2013
- 37. NASA AMSR-‐E imagery ingest processing pipeline: provenanc e capture for anomaly detec�on
- 38. Dataset: D2I-AMSR-E-Provenance Dataset Owner and Creator: Data to Insight Center Size: 15MB The University of Alabama in Huntsville processes data from the NASA AMSR-E instrument. The Karma project at Indiana University instrumented the ingest processing system and captured provenance for 3,890 runs for the period of September 2 - October 4 2011. The details of the runs are in Figure III-16 below; the largest provenance graph is the monthly rain graph that, when represented as a XML is approximately 13MB. Luo, Yuan, Plale, Beth, Jensen, Scott, Cheah, You-Wei, Conover, Helen. 2012. Provenance of AMSR-E Data from the National Snow and Ice Data Center (NSIDC). OPM XML Ver. 1.1., Sep 2 - Oct 4, 2011. Bloomington, Indiana: Data to Insight Center. http://dx.doi.org/10.5967/M0F47M2D
- 39. Provenance History Layout Algorithm Provenance of 1 month processing of NASA satellite ingest processing pipeline. Can help tracing error back to its cause. Shows relationship between daily products (each clover flower in clover leaf chain) and final monthly products at leftend. Provenance of a seaIce daily workflow 39 Aug 2013
- 40. Provenance graph compare: failed runs Le�: complete provenance of successful execu�on. Right: failed run, because final data product (green on le�) cannot be matched. 40
- 41. Graph compare: dropped provenance Le�: successful execu�on. Right: although successful execu�on, shows dropped no�fica�ons in provenance capture, because all nodes except some edges in le� graph cannot be matched. 41
- 42. Role of provenance in Open Data, Open Access, Open Cleaning Key contribution of provenance is to data quality. We posit that quality of data provenance has 3 dimensions: Correctness Completeness Relevancy Assumption: provenance collection process is automated Assessment is focused on correctness and completeness of captured provenance Steps: 1) Detect ambiguities and conflicts in real and synthetic provenance traces 2) Complete portions of missing provenance traces 3) Validate provenance traces when possible 4) Score the quality of provenance traces 42
- 43. Provenance Quality Analysis Overview G : Graph level M-G : Multi-Graph (Multiple graphs) Level N / E : Node/Edge Level 43
- 44. Wrapping Up: Open Data, Open Cleaning, Open Access S�mula�ng new business opportunity on stable interfaces to open data Open interfaces Open cleaning Open data Who’s working on: Research Data Alliance How? e.g, Crea�ve Commons license Personal privacy respected
- 45. Applied Forces Come Together to Distort Object into New Space Open access ini�a�ves Fundamental advances in à Climate change, à Food security à à New economies Big Data Personal data privacy, social isues of sharing Research Data Alliance Maturity in provenance and metadata
- 46. plale@indiana.edu Our hosts RDA Plenary 1 Chalmers Univ, Gothenburg, Sweden Photo courtesy Leif Laaksonen