Setting up Dataverse repository for research data

This project is funded from the EU Horizon 2020 Research and Innovation Programme (2014-2020) under Grant Agreement No. 823782
SSHOC
Setting up Dataverse repository for
research data
Slava Tykhonov, Senior Information Scientist
DANS-KNAW, The Royal Netherlands Academy of
Arts and Sciences
LIBSENSE webinar
10 March 2021

About me: DANS-KNAW projects (2016-2021)
● CLARIAH+ (ongoing)
● EOSC Synergy (ongoing)
● SSHOC Dataverse (ongoing)
● CESSDA DataverseEU 2018
● Time Machine Europe Supervisor at DANS-KNAW
● PARTHENOS Horizon 2020
● CESSDA PID (Personal Identifiers) Horizon 2020
● CLARIAH
● RDA (Research Data Alliance) PITTS Horizon 2020
● CESSDA SaW H2020-EU.1.4.1.1 Horizon 2020
2
Source: LinkedIn

About DANS-KNAW
DANS is the Dutch national centre of expertise and repository for research data.
We help researchers make their data available for reuse. This allows researchers
to use the data for new research and makes published research verifiable and
reproducible. With more than 150,000 datasets and a staff of 60, DANS is one of
the leading repositories in Europe.
Three pillars of DANS 2021-2025 programme: ‘Focus on FAIR”
•Centre of expertise for FAIR research data
•Versatile data repository: DANS data stations
•Active collaborator

What is Dataverse?
● Open source project developed by IQSS of Harvard University and published
on github
● Great product with very long history (from 2006)
● Very dynamic and experienced development team working in the Agile
environment (community call scheduled once in two weeks)
● Clear vision and understanding of research communities requirements,
public roadmap
● Strong community behind of Dataverse is helping to improve the basic
functionality and develop it further
● Dataverse has been selected as a data repository infrastructure by countries
from all continents
● Well developed architecture with rich API endpoints to build application
layers around Dataverse

Federated Dataverse data repositories worldwide
Source: Merce Crosas, Harvard Data Commons

DANS Dataverse 3.x migration (2016)
Basic DataverseNL services:
• Federated login for Netherlands
institutions
• Persistent Identifier Services (DOI and
handle)
• Integration with archival systems
Applications:
• Modern and historical world maps
visualisations
• Data API and Geo API services for
projects with data
• Panel datasets constructor
• Time series plot
• Treemaps
• Pie and chart visualizations
• Descriptive statistics tools

DataverseNL collaborative data network
Source: https://dataverse.nl

DataverseNL partners

Cooperative Model DataverseNL
DANS provides and manages the system & storage, organizes the meetings
(back-office)
•Research institutions run their own RDM-support for their researchers /end users
(front-office)
•Every partner institute is responsible for their own data
•Shared costs (service membership + storage)
•Advisory Board consisting of partner representatives decides on the general
policy
•Administrators committee to discuss technical and functional issues
•Cooperation agreement, Service Level Agreement, Processor agreement
(GDPR), General Terms of use (end users)
Source: Marion Wittenberg, DataverseNL

DANS Data Stations - Future Data Services
Dataverse is API based and a key framework for Open Innovation!

Major challenges to provide services for researchers
● Maintenance concerns - who will be in charge after project is finished?
● Infrastructure problems - how to install and run tools for researchers?
● Various Interoperability issues - how to leverage data exchange between
different systems and services
Software updates and bug fixing, licences, technical staff training, legal aspects
and so on...

Dataverse Installation Manual
https://guides.dataverse.org/en/latest/installation/index.html

Dataverse Docker module (CESSDA Dataverse, 2018)
Source: https://github.com/IQSS/dataverse-docker

Dataverse Kubernetes
Project maintained by Oliver Bertuch (FZ Julich) and available in Global
Dataverse Community Consortium github (GDCC)
Google Cloud, Amazon AWS, Microsoft Azure platforms supported
Open Source, community pull requests are welcome
http://github.com/IQSS/dataverse-kubernetes

SSHOC project task 5.2 Hosting and sharing data repositories
• Makes use of Dataverse software
• 4 ERICs: DARIAH, CLARIN, EHRIS and CESSDA
• Building mature infrastructure based on requirements of involved communities
• Developing external applications integrated with Dataverse (Dataverse Store)
• Investigating sustainable governance models
• Training Service Providers and institutes how to use Dataverse as a service

Data Commons is essential for integrations
Source: Merce Crosas, “Harvard Data Commons”

FAIR Dataverse
Source:
Mercè Crosas,
“FAIR principles and
beyond:
implementation in
Dataverse”

Our goals to increase Dataverse interoperability
Provide a custom FAIR metadata schema for European research communities:
● CESSDA metadata (Consortium of European Social Science Data Archives)
● Component MetaData Infrastructure (CMDI) metadata from CLARIN
linguistics community
Connect metadata to ontologies and CVs:
● link metadata fields to common ontologies (Dublin Core, DCAT)
● define semantic relationships between (new) metadata fields (SKOS)
● select available external controlled vocabularies for the specific fields
● provide multilingual access to controlled vocabularies

SKOSMOS framework to discover ontologies
20
● SKOSMOS is developed in
Europe by the National Library
of Finland (NLF)
● active global user community
● search and browsing interface
for SKOS concept
● multilingual vocabularies
support
● used for different use cases
(publish vocabularies, build
discovery systems, vocabulary
visualization)

External CV support: metadata field could be linked to many ontologies
Language switch in Dataverse will change the language terms!

Dataverse App Store
Let’s build different services out of tools!
Data preview: DDI Explorer, Spreadsheet/CSV, PDF, Text files, HTML,
Images, video render, audio, JSON, GeoJSON/Shapefiles/Map, XML
Interoperability: external controlled vocabularies (CESSDA CV Manager)
Data processing: NESSTAR DDI migration tool
Linked Data: RDF compliance (FAIR Data Point)
Federated login as a service (OAuth/Shibboleth in the same installation)

Dataverse Spreadsheet Previewer

Dataverse and CLARIN tools integration

Make Data Count metrics
Make Data Count is part of a broader Research Data Alliance (RDA) Data Usage Metrics
Working Group which helped to produce a specification called the COUNTER Code of
Practice for Research Data.
The following metrics can be downloaded directly from the DataCite hub for datasets hosted
by Dataverse installations:
● Total Views for a Dataset
● Unique Views for a Dataset
● Total Downloads for a Dataset
● Downloads for a Dataset
● Citations for a Dataset (via Crossref)
Dataverse Metrics API is a powerful source for BI tools used for the Data Landscape
monitoring.

Dataverse Metrics from 30+ repositories
Source: Metrics

Multilingual support
Dataverse SSHOC will run Weblate as a service for the user interface,
metadata schema and SOLR translation.
We’ve developed an experimental but adjustable pipeline for multilingual
support that allows to download and synchronize all translations available in
Dataverse Consortium github and provides easy access for translators to
keep all properties up-to-date.

Dataverse localization with Weblate
● service to connect files to Weblate in order to
translate them in a structured way
● several options for project visibility: accept
translations by the crowd, or only give access
to a select group of translators.
● Weblate indicates untranslated strings,
strings with failing checks, and strings that
need approval.
● when new strings are added with an upgrade
of Dataverse, Weblate can indicate which
strings are new and untranslated.

GUI translation with Weblate as a service
Source: SSHOC Weblate

Services in European Open Science Cloud (EOSC)
● EOSC requires the level 8 of maturity (at
least)
● we need the highest quality of software to
be accepted as a service
● clear and transparent evaluation of
services is essential
● the evidence of technical maturity is the
key to success
● the limited warranty will allow to stop out-
of-warranty services

Applications maturity level
Every software package should follow the same CESSDA Maturity Model to
be accepted as a service.
Must have: k8s infrastructure with upstream Docker images, warranty
statement, documentation, unit tests, Selenium tests, jenkins pipeline
Running demonstration service will allow to create the connection to your
own Dataverse.

CI/CD pipeline with SQAaaS (S)
1
2
3
git
push
Push GCP
container
registry
webhook
Create
docker
image
Kubernetes
Deployment
git clone
Jenkins pipeline (Jenkinsfile)
9
7
Run SQA
S 8
1. Developer pushes code to GitHub
2. Jenkins receives notification - build trigger
3. Jenkins clones the workspace
4. (S) Runs SQA tests and does FAIRness check
5. (S) Issuing digital badge according to the results
6. (S) SQAaaS API triggers appropriate workflow
7. Creates docker image if success
8. Pushes new docker image to container registry
9. Updates the kubernetes deployment
32
Source: EOSC Synergy project

Who is going to benefit from SSHOC Dataverse project?
• (SSH) institutes and researchers will be offered a Dataverse installation
on the cloud
• (SSH) institutes will be offered a Dataverse archive in a box solution for
their own purposes
• Many of the features to be developed in SSHOC will benefit also other
Dataverse installations / communities
All developments will be available for Dataverse community members!

Questions?
Slava Tykhonov (DANS-KNAW)
Senior Information Scientist
vyacheslav.tykhonov@dans.knaw.nl
Co-Chair:
Dataverse Working Group (WG) on Controlled Vocabularies and Ontologies
Dataverse WG on Registries
dataverse.org

Setting up Dataverse repository for research data

More Related Content

What's hot (20)

Similar to Setting up Dataverse repository for research data (20)

More from vty (7)

Recently uploaded (20)

Setting up Dataverse repository for research data