RO-Crate: A framework for packaging research products into FAIR Research Objects

Editor's Notes

• #7: Overcome fragmentation
• #8: Identifiers to locate things Organisation to structure & link things Annotations about the things @type: MUST be Dataset @id: MUST end with / and SHOULD be the string ./ name: SHOULD identify the dataset to humans well enough to disambiguate it from other RO-Crates description: SHOULD further elaborate on the name to provide a summary of the context in which the dataset is important. datePublished: MUST be a string in ISO 8601 date format and SHOULD be specified to at least the precision of a day, MAY be a timestamp down to the millisecond. license: SHOULD link to a Contextual Entity in the RO-Crate Metadata File with a name and description. MAY have a URI (eg for Creative Commons or Open Source licenses). MAY, if necessary be a textual description of how the RO-Crate may be used. This document specifies a method, known as RO-Crate (Research Object Crate), of organizing file-based data with associated metadata, using linked data principles, in both human and machine readable formats, with the ability to include additional domain-specific metadata. The core of RO-Crate is a JSON-LD file, the RO-Crate Metadata File, named ro-crate-metadata.json. This file contains structured metadata about the dataset as a whole (the Root Data Entity) and, optionally, about some or all of its files. This provides a simple way to, for example, assert the authors (e.g. people, organizations) of the RO-Crate or one its files, or to capture more complex provenance for files, such as how they were created using software and equipment. While providing the formal specification for RO-Crate, this document also aims to be a practical guide for software authors to create tools for generating and consuming research data packages, with explanation by examples. At the basic level, an RO-Crate is a collection of files and resources represented as a Schema.org Dataset, that together form a meaningful unit for the purposes of communication, citation, distribution, preservation, etc. The RO-Crate Metadata File describes the RO-Crate, and MUST be stored in the RO-Crate Root. While RO-Crate is well catered for describing a Dataset as files and relevant metadata that are contained by the RO-Crate in the sense of living within the same root directory, RO-Crates can also reference external resources which are stored or accessed separately, via absolute URIs. This is particularly recommended where some resources cannot be co-hosted for practical or legal reasons, or if the RO-Crate itself is primarily web-based. It is important to note that the RO-Crate Metadata File is not an exhaustive manifest or inventory, that is, it does not necessarily list or describe all files in the package. Rather it is focused on providing sufficient amount of metadata to understand and use the content, and is designed to be compatible with existing and future approaches that do have full inventories / manifest and integrity checks, e.g. by using checksums, such as BagIt and Oxford Common File Layout OCFL Objects. The intention is that RO-Crates can work well with a variety of archive file formats, e.g. tar, zip, etc., and approaches to capturing file manifests and file fixity, such as BagIt, OCFL and git (see also appendix Combining with other packaging schemes). An RO-Crate can also be hosted on the web or mainly refer to web resources, although extra care to ensure persistence and consistency should be taken for archiving such RO-Crates.
• #9: Data might @type: MUST be Dataset @id: MUST end with / and SHOULD be the string ./ name: SHOULD identify the dataset to humans well enough to disambiguate it from other RO-Crates description: SHOULD further elaborate on the name to provide a summary of the context in which the dataset is important. datePublished: MUST be a string in ISO 8601 date format and SHOULD be specified to at least the precision of a day, MAY be a timestamp down to the millisecond. license: SHOULD link to a Contextual Entity in the RO-Crate Metadata File with a name and description. MAY have a URI (eg for Creative Commons or Open Source licenses). MAY, if necessary be a textual description of how the RO-Crate may be used. This document specifies a method, known as RO-Crate (Research Object Crate), of organizing file-based data with associated metadata, using linked data principles, in both human and machine readable formats, with the ability to include additional domain-specific metadata. The core of RO-Crate is a JSON-LD file, the RO-Crate Metadata File, named ro-crate-metadata.json. This file contains structured metadata about the dataset as a whole (the Root Data Entity) and, optionally, about some or all of its files. This provides a simple way to, for example, assert the authors (e.g. people, organizations) of the RO-Crate or one its files, or to capture more complex provenance for files, such as how they were created using software and equipment. While providing the formal specification for RO-Crate, this document also aims to be a practical guide for software authors to create tools for generating and consuming research data packages, with explanation by examples. At the basic level, an RO-Crate is a collection of files and resources represented as a Schema.org Dataset, that together form a meaningful unit for the purposes of communication, citation, distribution, preservation, etc. The RO-Crate Metadata File describes the RO-Crate, and MUST be stored in the RO-Crate Root. While RO-Crate is well catered for describing a Dataset as files and relevant metadata that are contained by the RO-Crate in the sense of living within the same root directory, RO-Crates can also reference external resources which are stored or accessed separately, via absolute URIs. This is particularly recommended where some resources cannot be co-hosted for practical or legal reasons, or if the RO-Crate itself is primarily web-based. It is important to note that the RO-Crate Metadata File is not an exhaustive manifest or inventory, that is, it does not necessarily list or describe all files in the package. Rather it is focused on providing sufficient amount of metadata to understand and use the content, and is designed to be compatible with existing and future approaches that do have full inventories / manifest and integrity checks, e.g. by using checksums, such as BagIt and Oxford Common File Layout OCFL Objects. The intention is that RO-Crates can work well with a variety of archive file formats, e.g. tar, zip, etc., and approaches to capturing file manifests and file fixity, such as BagIt, OCFL and git (see also appendix Combining with other packaging schemes). An RO-Crate can also be hosted on the web or mainly refer to web resources, although extra care to ensure persistence and consistency should be taken for archiving such RO-Crates.
• #10: More flexible than XML Schemas More straightforward than OWL More flexible than XML Schemas More straightforward than OWL Linked Data “exotics” there for when the time is right if needed by the right people Portland Common Data Model for digital library & repository content Metadata is Flat JSON file Descriptions of the Objects Descriptions of the RO-Crate Descriptions relating the Objects human and machine readable formats, additional domain-specific metadata.
• #15: PARADISEC and The University of Melbourne University of Technology Sydney PARADISEC (the Pacific And Regional Archive for Digital Sources in Endangered Cultures) has operated a data curation service for endangered languages for 18 years. Currently consisting of 500,000 files in 28,624 items and 574 collections, there is over 90TB of data under management. The repository, whilst still functional and fit for purpose, is showing signs of its age. Using the Arkisto Platform, the Modern PARADISEC project is updating the catalog whilst also enabling a path for smaller regional repositories to exist and to easily move their content across collections and institutions.
• #16: NIH Data Commons transferring and archiving very large HTS datasets in a location-independent way keep the context of data content together when its scattered. Scalability 
• #20: ReproZip can automatically pack your research along with all necessary data files, libraries, environment variables and options into a self-contained bundle. https://www.reprozip.org/ One pressure is towards bundling objects within objects. There’s nothing in DRS for a structured manifest within a bundle. RO-Crate as interchange format and integration with Zenodo, using Describo Data Cubes for efficient and scalable structured data access and discovery Research Objects as the mechanism to manage scientific research activities and connect associated resources RO snapshot/releases published to EOSC repositories and scholarly communication platforms (e.g., B2Share, Zenodo
• #23: We started in theory, but we work in practice ROs sit in the middle
• #24: To be FAIR each digital object type has its own metadata requirements, and may have its own repositories and registries https://op.europa.eu/en/publication-detail/-/publication/d787ea54-6a87-11eb-aeb5-01aa75ed71a1/language-en/format-PDF/source-190308283 https://www.eoscsecretariat.eu/sites/default/files/eosc-interoperability-framework-v1.0.pdf Schwardmann (2020), Digital Objects – FAIR Digital Objects: Which Services Are Required? Data Science Journal EOSC Interoperability Framework Draft (2020) Hardisty A, et al (2020) Conceptual design blueprint for the DiSSCo digitization infrastructure RIO 6: e54280. DONA Digital Object Architecture Digital Object Interface Protocol (2018) https://fairdigitalobjectframework.org/
• #25: [Hardisty A, et al (2020) Conceptual design blueprint for the DiSSCo digitization infrastructure RIO 6: e54280. How to make this practical!
• #27: So let’s do a quick reminder about what are the key components of FAIR Digital Objects: Multiple levels of Digital objects: - A Persistent Identifier to locate the DO - A set of Operations to use the DO - Metadata about the DO – itself a DO - All of these represented as bit sequences that can be transferred or stored in repository - Finally a Collection is a type of DO that aggregates other DOs and of course has its own metadata
• #28: For RO-Crate, how do we approach FDO? We have implementations for each of these concepts: The RO-Crate dataset itself is of course what we can call a Collection. It aggregates a set of data entities, but also contextual entities. I’ll come back to those. We describe the RO-crate and the entities in a metadata file We can assign persistent identiers to the RO-Crate and parts of it We can store the bit sequences, ideally with something like BagIt, to ensure completeness And deposit all of this in regular repositories like Zenodo or GitHub
• #39: RO-Crate recommends how to structure metadata.  FDO RO-Crate profile What are the operations on RO-Crate metadata? Re-visit Wf4Ever Research Object APIs? (basically CRUD) RO-Crate as a Linked Data Platform (LDP) Container? Registration of RO-Crate types/profiles How can RO-Crate users use FDO to resolve PID/storage challenges?