Aggregation workflow
1 like1,055 views
The document describes Europeana's aggregation workflow including: - Europeana's aggregation team handles partner relationships and technical support. - Data is submitted according to Europeana's publication policy and on a monthly cycle with deadlines. - The ingestion process involves validation, mapping data to EDM, and publishing on Europeana's portal and API. - Guidance is provided to help partners meet acceptance criteria around rights, metadata quality, and the EDM schema. - Future plans aim to open up more of the ingestion workflow for partners to do mapping and validation themselves.
Aggregation workflow
- 1. Aggregation workflow Cécile Devarenne Operations Officer Metadata training, Europeana Sounds project Athens, 23rd/24th of October 2014
- 2. Content • Europeana's aggregation team • Europeana Publication Policy • Aggregation workflow • Submission deadlines • Ingestion processes and tools • Acceptance criteria and Europeana validation of data • Guidance and help – Europeana pro • Future plans for aggregation workflow
- 3. Europeana’s aggregation team: who are we?
- 4. Europeana’s aggregation team • Partner relationships, business development, administration • Henning Scholz, Joris Pekel, Gina Van der Linden • Technical support • Operations officers: content@europeana.eu • Data support, feedback and ingestion of your collections into Europeana portal and API
- 6. Europeana Publication Policy Clear criteria for acceptance or decline of metadata for publication and for take down of legacy metadata from the Europeana database •Ingestion workflow (deadlines, timelines, prioritization) •Content scope (what is a digital object? what content does Europeana aggregate?) •Technical validation of metadata quality (expected values) •Metadata licensing (CC0) •Rights Statements for digital objects • All digital objects with valid edm:rights chosen from http:// pro.europeana.eu/web/guest/available-rights-statements • Public Domain material labelled with the Public Domain mark in edm:rights • edm:rights & dc:rights not in contradiction
- 7. Aggregation workflow and submission deadlines: how does it work?
- 8. 8
- 9. Submission of data: preliminary steps for your project • (1) Data Exchange Agreement to Europeana (DEA) • Europeana Sounds project needs to submit the signed Data Exchange Agreements for each contributing data provider • The Europeana Data Exchange Agreement establishes the terms under which Europeana can make use of the previews and descriptive metadata provided by cultural institution • More information to be found here: http://pro.europeana.eu/ensuring- permissions-for-aggregators • (2) Data contribution form • One form for the whole project • General information on data to be submitted to Europeana • Schedule of data delivery: ingestion planning • (3) Submission of data samples and feedback taken into account
- 10. Submission of data: (4) publication cycles • Operations officers work on a monthly cycle • Submission of data in the form of datasets: a coherent batch of records, for the Europeana Sounds project, probably one dataset for each of your data providers • A dataset takes on average 40 mins to process • Around 200 datasets are processed by the Operations officers for each cycle of publication • Datasets go through a full flow of operations before they are production ready • Datasets need to be submitted on time in order for this production cycle to work • Datasets are submitted by the technical/content coordinators of your project • The earlier you submit datasets the more feedback we can give!
- 11. Submission of data: new provider timeline
- 12. Submission of data: regular ingestion cycle timeline
- 13. Ingestion processes and tools: what happens to your data when submitted to Europeana?
- 14. Europeana’s set of ingestion tools • Unified Ingestion Manager (UIM): orchestrator of data flows triggered in various tools and plugins • SugarCRM (Customer Relationship Management): reference entries for datasets and organisations • REPOX: harvester to get the collections uploaded into Europeana • Europeana’s instance of Mint (Metadata INTeroperability): mapping and editing tool for ingested datasets • Data plugins • Itemization, Europeana identifiers generation • Dereferencing • Enrichment • Redirects • Extraction of hierarchies • Thumbnails caching
- 15. Europeana ingestion data flows
- 16. Steps to get data ingested From the moment your data was submitted: • Checks on raw xml (Browser) • Prior to harvesting • Identification of key issues • Creation/update of dataset information, checks on validity of the supplied harvesting information (SugarCRM - REPOX) • Harvesting (REPOX) • Mapping/editing of datasets (Europeana Mint) • Mapping tool for all datasets • Adapted for Europeana in order to process multiple formats (EDM, ESE, any metadata standard with provided XSLT) • Drag and drop appropriate elements • Quality checks and data cleaning if necessary • Transformation and validation of records according to EDM schema and schematron rules • EDM Internal data: Europeana ready material
- 17. Steps to get data ingested • Operations on data following transformation: • Itemization and creation/management of Europeana identifiers for permalinks to your records in Europeana • Extraction of hierarchies for datasets including EDM hierarchies • Thumbnails caching • Enrichments of data: • From links to linked data exposed ontologies, generation of additional contextual data (dereferencing) • From analysis of the provided data, automated semantic enrichment (Europeana enrichment) • If necessary (when a change of identifiers was communicated to Europeana), creation of redirections between previous and newly generated identifiers • Data ready! monthly deploy on Europeana portal and API
- 18. Acceptance criteria: how exactly is the Publication Policy implemented?
- 19. Acceptance criteria • Data Exchange Agreement to Europeana • Datasets submitted via OAI-PMH protocol, FTP or file • Metadata are accepted for publication after the feedback of the Europeana Operations Officers • EDM schema and guidelines • Rights labeling • Datasets are prioritized for publication if the edm:rights in the majority of the metadata of the dataset is PDM, CC0, CC BY or CC BY-SA
- 20. Automatic validation: • Validation according to the EDM schema • Validation of the mandatory properties • Unique identifiers within a dataset • Metadata records that don’t meet this validation are invalidated or discarded • Providers can fix issues first and resubmit or let Europeana ingest the records that are valid, and fix the invalid records at a later stage • Validation of urls for thumbnail creation (ImageMagick) Europeana validation
- 21. Applicable class Mandatory Properties (or alternatives) Aggregation edm:dataProvider Aggregation edm:isShownAt or edm:isShownBy Aggregation edm:provider Aggregation edm:rights Aggregation edm:aggregatedCHO Aggregation edm:ugc (when applicable) ProvidedCHO dc:title or dc:description ProvidedCHO dc:language for text objects ProvidedCHO dc:subject or dc:type or dc:coverage or dcterms:spatial ProvidedCHO edm:type Mandatory properties
- 22. Validation by the Operations officers: • Feedback is according to the EDM schema and guidelines • Checks on the connections between the EDM classes and the general structure of the data • Correct use of vocabularies, recommendations to include geolocations • Checks on the types of values: literals vs resources (e.g. a thumbnail always need to be a valid url) • Checks on links to digital representations of the objects; if direct links to a file, check that they are of reasonable size • Provision of thumbnails highly recommended • Feedback on (near) duplicate records • Feedback on rights statements in edm:rights and dc:rights • Feedback on any other metadata quality related matters (duplication of properties, encoding in the data, wrongly mapped properties, etc.) Europeana validation
- 23. • The data is represented according to expectations for both sides • Users can search and retrieve rich content: • Developers can make the best use of the API • Objects are clicked through and re-used from the Europeana portal Happy ingestion :-)
- 26. Guidance and help Europeana Professional: http://pro.europeana.eu/provide-data Content inbox – for all ingestion & metadata related matters content@europeana.eu
- 27. Questions?
- 28. Future plans for aggregation workflow
- 29. Future plans for aggregation workflow • Future plans to open up part of the Europeana ingestion workflow to providers • Providers can log-in the Europeana ingestion suite, identify the aggregator/project they work for • Providers can select the datasets they want to update, or add new datasets • Providers can upload their data (OAI-PMH and FTP protocols) • Providers can map their data to EDM, or edit data that is already EDM • Providers can validate the data against the EDM schema and preview them prior to submission • Other processes being considered for refactoring: semantic validation, link checking, thumbnail caching, enrichment
- 30. Future plans for aggregation workflow • Benefits for providers: • Possibility to map to EDM • Validation according to the EDM schema (with schematron rules we implemented) • Preview before publication • Self service, less dependent on Europeana, saving time (you can do many steps yourself, and you spot errors earlier) • Benefits for Europeana: • Operations scaled up – amount of projects, aggregators and therefore datasets has grown exponentially in the last years • More focus on EDM modeling and metadata related questions • Ingestion process transparent and more connected to the process at aggregators side
- 31. Thank you! Cécile Devarenne cecile.devarenne@europeana.eu or content@europeana.eu