Data Publishing Workflows with Dataverse
Download as PPTX, PDF2 likes1,821 views
The document outlines the workflows for data publishing using Dataverse, emphasizing its integration with various tools and frameworks for data sharing, preservation, and citation. It highlights features such as sensitivity settings for datasets, continuous integration for R packages, and collaboration with institutions like Harvard and MIT. The new Dataverse 4.0 is set to release on June 23, showcasing improved user interface and data handling capabilities across multiple scientific disciplines.
Data Publishing Workflows with Dataverse
- 1. Data Publishing Workflows with Dataverse Mercè Crosas, Ph.D. Twitter: @mercecrosas Director of Data Science Institute for Quantitative Social Science, Harvard University MIT, May 6, 2014
- 2. Intro to our Data Science Team and Projects
- 3. Data Science at the Institute for Quantitative Social Science http://datascience.iq.harvard.edu
- 4. Combines Expertise Data Science Applications and Tools Researchers Software Engineers Information Scientists Statistical Innovation Tool Building & Computer Science Data Curation & Stewardship
- 5. With a Team of 20 Mercè Crosas, Director of Data Science Statistics and Analytics James Honaker Christine Choirat Vito d’Orazio Software Development Gustavo Durand Robert Treacy Ellen Kraffmiller Michael Bar-Sinai Leonid Andreev Phil Durbin Steve Kraffmiller Xiangqing Yang Raman Prasad (BARI) Data Curation and Archiving Sonia Barbosa Eleni Castro Dwayne Liburd QA Kevin Condon Elda Sotiri Usability and UI Elizabeth Quigley Michael Heppler Gary King, Director of IQSS Cris Rothfuss, Excutive Director
- 6. Two widely-Used Frameworks Developed in the last Decade A framework that allows analysts to use and interpret a large body of R statistical models from heterogeneous contributors through a common interface. A data publishing framework that allows researchers to share, preserve, cite and analyze data, while keeping control and gaining credit for their data.
- 7. New Tools that Integrate with our Initial Work An interactive web interface that allows users at all levels of statistical expertise to explore their data and appropriately construct statistical models. Integrates with Zelig and Dataverse. A framework that allows data contributors to set a level of sensitivity for their dataset based on legal regulations, which defines how the data can be stored and shared. Integrates with Dataverse. In collaboration with NSF Privacy Tools project
- 8. Expanding in other Areas A web application that assists researchers to discover new clusters to categorize large document sets, leveraging all the clustering methods in the literature. An application that provides a continuous integration build solution for R packages shared in Git to archived published code in CRAN.
- 9. Support Throughout the Research Cycle Develop Quantitative Methods Analyze Quantitative Datasets Analyze Unstructured Text Publish Data Cite Data from Published Results Explore, reanalyze and reuse dataShare Sensitive Data Develop > Analyze > Share > Explore > Validate & Reuse
- 10. Current Research Interests and Efforts Reproducible and Reusable Science: “encourage open data and methodological transparency, and promote and enable data citation” (with Dataverse, Zelig and SolaFide) Computationally Assisted Exploration: “with Consilience and SolaFide, assist researchers to understand and discover new insights from their data” Interdisciplinary Quantitative Scientific Scope: “our tools and research frameworks address broad methodological issues in quantitative science and are often employed in other domains” When Data are Not Open: “solutions to preserve privacy, while still providing science the fundamental ability to learn, access and replicate findings, with DataTags and PrivateZelig” Large-Scale Data Sets: “will handle large-scale data sets, as Big Data science reaches all disciplines: Consilience for millions of text documents, and Zelig and Dataverse to handle TB-PB-scale data sets.”
- 12. The Harvard Dataverse Repository In collaboration with the Harvard Library, Harvard hosts a Dataverse instance free and open to all researchers. It currently holds > 53,000 datasets, with 735,000 files. Find or deposit data at: http://thedata.harvard.edu
- 13. Collaborations with MIT Membership through the Harvard-MIT Data Center (e.g., statistics training, access to ICPSR collection) The MIT Libraries Dataverse disseminates data purchased by the MIT Libraries (with Kate McNeill): http://thedata.harvard.edu/dvn/dv/mit MIT faculty and research groups are already disseminating their data through the Harvard Dataverse Research collaborations (with Micah Altman): Integration of Publications with Data (Funded by Sloan): http://projects.iq.harvard.edu/ojs-dvn Privacy Tools for Sharing Research Data (Funded by NSF): http://privacytools.seas.harvard.edu/
- 14. Dataverse 4.0 Target release date: June 23 • New UI • New rich, faceted search • New data file ingest (excel, CSV, R, Stata, SPSS) • New metadata for social sciences, astronomy, biomedical sciences. • Integration with SolaFide.
- 15. SolaFide Demo
- 17. Data Publishing Guidelines Three pillars to Data Publishing: A trusted data repository to guarantee long-term access A formal data citation* Sufficient information to understand and reuse the data (metadata, documentation, code) * Data Citation Principles: https://www.force11.org/datacitation
- 18. A Rigorous Publishing Workflow Release Version 1 A Published Dataset cannot be deleted (only deaccession, if legally needed) Push Version 1.1: small metadata change; citation doesn’t change Push Version 2: big metadata change, or file change; citation changes Authors, Title, Year, DOI Repository, UNF, V1 Authors, Title, Year, DOI Repository, UNF, V2
- 19. Workflows that Integrate with Journals 1. Publish a dataset to your Dataverse, then provide the Data Citation to the journal. 2. Contribute to a journal Dataverse: 1. Add dataset to Journal Dataverse as a draft. 2. Journal Editor reviews it, and approves it for release. 3. Dataset is published with Data Citation and link from journal article to the data. 3. Seamless Integration between journal system and Dataverse.
- 20. OJS and Dataverse Integration Sloan funded project to integrate PKP’s Open Journal System with the Dataverse software. Pilot with ~ 50 journals OJS Dataverse plugin now available with latest OJS release http://projects.iq.harvard.edu/ojs-dvn
- 21. Detailed System Integration XML file: AtomPub "entry" with Dublin Core Terms (e.g., title, creator) Zip file: All data files associated with that dataset. HTTP header "In-Progress: false" to publish datasets. Support HTTP verbs: GET, PUT, POST, and DELETE. XML file: “Deposit Receipt” HTTP status code: 200, 201, 204, 404, 405, 406, 412, 415 Client can query repository (server) any time to get status
- 22. Deposit API based on SWORD Follows SWORD2 specifications SWORD is known and supported within academic publishing; a “profile” of the AtomPub standard. The SWORD project provides client libraries for Python, Java, Ruby, and PHP: OJS uses the PHP client library OSF uses the Python client library DataUp and DVN-R built a custom Dataverse client
- 23. How it differs from SWORD Dataverse does not use SWORD download API: Use own Data API Plan to add this support in the future Add XML attribute to pass article citation from client: Allow DCterms:isReferencedby to contain attributes such as HoldingsURI to link back to article from Dataverse This is now part of the SWORD PHP client library Use “In-Progress: false” to indicate that dataset is ready to be published (In SWORD spec means deposit complete)
- 24. Support for Metadata Standards A core or citation metadata that applies to all datasets – Supported currently by Data Deposit API Extensible metadata blocks for specific domains: Social sciences: Maps to DDI schema; file metadata extracted from tabular data file Astronomy: Maps to VO schema; partially extracted from FITS file Biomedical sciences: Maps to ISA-tab schema Controlled vocabularies maps to EFO, OBI, and Ontology of Clinical Research Extended and managed using SKOS (support taxonomies within the framework of the semantic web)
- 30. Upcoming
- 31. Expanding to Larger and More Types of Data Sharing sensitive data with DataTags and Secure Dataverse Integration with other systems: OSF DataUp WorldMap DataBridge ORCID DASH (at Harvard) Expand to Larger data sets
- 32. DataTags: For Sharing Sensitive Data