The Rise of Data Publishing in the Digital World (and how Dataverse and DataTags Help)

The Rise of Data Publishing
in the Digital World
(and how Dataverse and DataTags help)
Mercè Crosas, Ph.D.
Chief Data Science and Technology Ofﬁcer
Institute for Quantitive Social Science
Harvard University
@mercecrosas
NDSR 2016 Symposium

From 1665 to late 20th century:
A steady increase in size and
complexity of research output

The number of journals doubles every 20 years
since 1750s, with growth of number of scientists
1665 1765 1865 1965
100
10000
Mabe, 2003

1700: 3 journals
1665 1765 1865 1965
100
10000
Mabe, 2003

1700: 3 journals
1800: ~10 journals
1665 1765 1865 1965
100
10000
Mabe, 2003

1700: 3 journals
1800: ~10 journals
1900: ~400 journals
1665 1765 1865 1965
100
10000
Mabe, 2003

1700: 3 journals
1800: ~10 journals
1900: ~400 journals
2000: ~14,000 journals
(peer-reviewed)
1665 1765 1865 1965
100
10000
Mabe, 2003

Data Tables andVisuals Become Increasingly
Common, and part of the Scientiﬁc Argument
1665 1765 1865 1965
100
10000

a few tables &
visuals, as part of
the text
1665 1765 1865 1965
100
10000

a few tables &
visuals, as part of
the text 50% cite previous
work
1665 1765 1865 1965
100
10000

a few tables &
visuals, as part of
the text 50% cite previous
work
First Line Graphs
and bar charts
(Playfair, 1786)
1665 1765 1865 1965
100
10000

a few tables &
visuals, as part of
the text
50% of articles have
tables & ﬁgures
50% cite previous
work
First Line Graphs
and bar charts
(Playfair, 1786)
1665 1765 1865 1965
100
10000

a few tables &
visuals, as part of
the text
tables & ﬁgures
50% cite previous
work
method sections
appear
First Line Graphs
and bar charts
(Playfair, 1786)
1665 1765 1865 1965
100
10000

a few tables &
visuals, as part of
the text
tables & ﬁgures
50% cite previous
work
method sections
appear
First Line Graphs
and bar charts
(Playfair, 1786)
First Scatterplots
(Hershel,1833;
Galton 1896)
1665 1765 1865 1965
100
10000

a few tables &
visuals, as part of
the text
tables & ﬁgures
most articles
have tables &
ﬁgures, often
standalone
50% cite previous
work
method sections
appear
First Line Graphs
and bar charts
(Playfair, 1786)
First Scatterplots
(Hershel,1833;
Galton 1896)
1665 1765 1865 1965
100
10000

a few tables &
visuals, as part of
the text
tables & ﬁgures
most articles
have tables &
ﬁgures, often
standalone
50% cite previous
work
100% with citations
(1 per 100 words)
part of scholarly credit
method sections
appear
First Line Graphs
and bar charts
(Playfair, 1786)
First Scatterplots
(Hershel,1833;
Galton 1896)
1665 1765 1865 1965
100
10000

Scholarly Publishing Adapts to the
Increase of Cognitive Complexity (Gross et al 2001)

• 18th century:

• 18th century:
• formal components appear in articles (introduction,
conclusions, table, ﬁgures, citations)

• 18th century:
• 19th century:

• 18th century:
• 19th century:
• explain data instead of establish observations of facts

• 18th century:
• 19th century:
• wide use of visuals, high citation density, methods section

• 18th century:
• 19th century:
• 20th century:

• 18th century:
• 19th century:
• 20th century:
• structured quantitative data with increased use of statistics

• 18th century:
• 19th century:
• 20th century:
• wide range of data types with new technologies

• 18th century:
• 19th century:
• 20th century:
• Number of scientists increases from 100s to a few millions

• 18th century:
• 19th century:
• 20th century:
• Science becomes extremely specialized:

• 18th century:
• 19th century:
• 20th century:
• from 1 journal to 14,000 peer-reviewed journals

• 18th century:
• 19th century:
• 20th century:
• from 1 journal to 14,000 peer-reviewed journals
• one new journal for each 150 authors, read by 500

In the last decades, more
and more publications
and data

A Steeper Growth of Scholarly Output
Since 1950, the total number of journals doubles every ~15 years
2010: 80,000 journals
2010: 33,000 peer-reviewed

An Outburst of Research Data and Specialization,
Results into > 1000 Community Repositories

1920 - 1950s

First Social Science
Data Archives
(ODUM, ICPSR, ...)
1920 - 1950s

Data Archives
(ODUM, ICPSR, ...)
1920 - 1950s 1970 - 1980s

Data Archives
(ODUM, ICPSR, ...)
First Biomedical
Databases
(PDB, GenBank, ...)
1920 - 1950s 1970 - 1980s

Data Archives
(ODUM, ICPSR, ...)
First Biomedical
Databases
(PDB, GenBank, ...)
1920 - 1950s 1970 - 1980s 2016

Data Archives
(ODUM, ICPSR, ...)
A wide range of
Research Data
Repositories
First Biomedical
Databases
(PDB, GenBank, ...)
1920 - 1950s 1970 - 1980s 2016

Data Archives
(ODUM, ICPSR, ...)
A wide range of
Research Data
Repositories
First Biomedical
Databases
(PDB, GenBank, ...)
1500 repositories listed in re3data.org
1920 - 1950s 1970 - 1980s 2016

Data Publishing Emerges as the Union of
Scholarly Publishing and Data Archiving

Scholarly publishing:
Distribute research output

• Attribution and credit

• Dissemination

• Dissemination
• Finding & Reuse

• Dissemination
• Finding & Reuse
Data Archiving:
Long-term access to data

• Dissemination
• Finding & Reuse
Data Archiving:
• Accessibility

• Dissemination
• Finding & Reuse
Data Archiving:
• Accessibility
• Preservation

• Dissemination
• Finding & Reuse
Data Archiving:
• Accessibility
• Preservation
• Finding & Reuse

Why Data Publishing now?
Extending Gross et al. thesis, data publishing accommodates the
complexity of research input and output in the digital world.

• Data (and software) have become common input and
output of research

output of research
• A scholarly article cannot hold or describe accurately these
vast amounts of data and software

output of research
• A scholarly article cannot hold or describe accurately these
vast amounts of data and software
• As input and output of research, data must be citable and
accessible to enable validation and reuse, with attribution

What is needed for FAIR Data Publishing
FAIR = Findable Accessible Interoperable Reusable

Data Citation

Data Citation
• Persistent id to
reference data uniquely

Data Citation
• Support for versions
and ﬁxity

Data Citation
and ﬁxity
• Attribution to authors
and repository

Data Citation
and ﬁxity
and repository
Metadata

Data Citation
and ﬁxity
and repository
Metadata
• Catalog to discover and
locate the data

Data Citation
and ﬁxity
and repository
Metadata
locate the data
• Sufﬁcient information to
understand and reuse the
data

Data Citation
and ﬁxity
and repository
Metadata
locate the data
data
Repository

Data Citation
and ﬁxity
and repository
Metadata
locate the data
data
Repository
• Digital access to metadata
and data

Data Citation
and ﬁxity
and repository
Metadata
locate the data
data
Repository
and data
• Archive and preservation for
long-term access

Data Citation
and ﬁxity
and repository
Metadata
locate the data
data
Repository
and data
• Archive and preservation for
long-term access
• Interoperability through
standards and APIs

The Rise of Data Publishing in the Digital World (and how Dataverse and DataTags Help)

A data repository system that serves as a
solution for publishing FAIR research data

Around the World
Dataverse repositories serve a community, an institution, an archive, ...

Around the World
Harvard Dataverse:
Generic data repository open
to researchers world wide
Dataverse repositories serve a community, an institution, an archive, ...

Dataverses contain datasets,
datasets contain metadata and data ﬁles

Data Citation in Dataverse
Published
Year
Dataset
Title
Global
Persistent
Identiﬁer
Repository
= Data Publisher
Version (or
time range)
Authors

Data Citation Basics
Force11, Joint Declaration of Data Citation Principles; Starr et al, 2015

Data Citation Basics
Force11, Joint Declaration of Data Citation Principles; Starr et al, 2015
The dataset landing page is accessible and guaranteed by the repository
(or data publisher), even when data are restricted or deaccessioned

Metadata In Dataverse
Citation Metadata
author, title, repository,
year published, version,
etc
• Dublin Core
• DataCite
Domain-speciﬁc
Metadata
data collection info
(methods, organism,
observation, survey,
experiment, etc)
• DDI (social sciences)
• ISA-Tab BioCaddie (biomed)
• Virtual Observatory (astro)
• + Custom metadata blocks
File-level Metadata
metadata inside the data
ﬁle (variables, instrument
details, geospatial info,
etc)
• DDI (for variables),
• + more to be determined
Fields StandardsMetadata Level

Metadata In Dataverse
Citation Metadata
author, title, repository,
year published, version,
etc
• Dublin Core
• DataCite
Domain-speciﬁc
Metadata
data collection info
(methods, organism,
observation, survey,
experiment, etc)
• DDI (social sciences)
• ISA-Tab BioCaddie (biomed)
• Virtual Observatory (astro)
• + Custom metadata blocks
File-level Metadata
metadata inside the data
ﬁle (variables, instrument
details, geospatial info,
etc)
• DDI (for variables),
• + more to be determined
Fields StandardsMetadata Level
DataverseJSONSchema

Information Extraction:Tabular Files

RData
Stata
SPSS
Excel
CSV
var 1 var 2 var 3
obs 1 2 a 0
obs 2 4 c 0
obs 3 6 b 1
obs 4 1 e 0
obs 5 2 a 1
obs 6 3 b 1

RData
Stata
SPSS
Excel
CSV
var 1 var 2 var 3
obs 1 2 a 0
obs 2 4 c 0
obs 3 6 b 1
obs 4 1 e 0
obs 5 2 a 1
obs 6 3 b 1
Variable Metadata:
Variable name, label,
type, stats, geospatial
coordinates

RData
Stata
SPSS
Excel
CSV
var 1 var 2 var 3
obs 1 2 a 0
obs 2 4 c 0
obs 3 6 b 1
obs 4 1 e 0
obs 5 2 a 1
obs 6 3 b 1
Variable Metadata:
coordinates
2 a 0
4 c 0
6 b 1
1 e 0
2 a 1
3 b 1
DataValues:
Independent of format

RData
Stata
SPSS
Excel
CSV
var 1 var 2 var 3
obs 1 2 a 0
obs 2 4 c 0
obs 3 6 b 1
obs 4 1 e 0
obs 5 2 a 1
obs 6 3 b 1
Variable Metadata:
coordinates
2 a 0
4 c 0
6 b 1
1 e 0
2 a 1
3 b 1
DataValues:
Independent of format
Universal Numerical Fingerprint (UNF):
checksum on data values, from canonical format

Information Extraction: FITS (astro) Files

Header Metadata:
coordinates (R.A.,
declination),
photometric info, ...

Header Metadata:
coordinates (R.A.,
declination),
photometric info, ...
Data Objects:
•Image Files
•Spectra
•Data cubes
•Tables
•...

In addition to data citation and
metadata features, Dataverse
has a rich set of features that
facilitate data publishing

Tiered Access
Open (default):
CC0
Open Open Click to Download
GuestBook Open Open
Fill in guestbook before
download
Terms of Use Open Open
Click through terms of
use before download
Data Restricted Open Restricted Request Access via
click through
Data Restricted Open Restricted
Request Access via
application
Metadata Files How to Access

Data Publishing Workﬂows
Create Dataset
(landing page
restricted)

Create Dataset
(landing page
restricted)
Review
(collaborators or
anonymous reviewers)

Create Dataset
(landing page
restricted)
Publish v. 1
Review
(collaborators or

Create Dataset
(landing page
restricted)
Publish v. 1
Review
(collaborators or
Minor change
(metadata only)

Create Dataset
(landing page
restricted)
Publish v. 1
Review
(collaborators or
Minor change
(metadata only)
Publish v. 1.1

Create Dataset
(landing page
restricted)
Publish v. 1
Review
(collaborators or
Minor change
(metadata only)
Publish v. 1.1
Major change
(might include new
data ﬁle)

Create Dataset
(landing page
restricted)
Publish v. 1
Review
(collaborators or
Minor change
(metadata only)
Publish v. 1.1
Major change
(might include new
data ﬁle)
Publish v. 2

And more at dataverse.org guides ...

Biomedical Dataverse addresses data
publication of large ﬁles: SBGridData

The Biomedical Dataverse at Harvard Medical School -
also tested as a persistent repository for LINCS data
(NIH Library of Integrated Network based Cellular Signatures)
Collaboration with Piotr Sliz and Caroline Shamu (HMS)
(NIH Library of Integrated Network-based Cellular Signatures)

An additional challenge
for data publishing:
Sensitive Data

“User
Uploads
must
be
void
of
all
iden4ﬁable

informa4on,
such
that
re-‐iden4ﬁca4on
of
any
subjects

from
the
amalgama4on
of
the
informa4on
available

from
all
of
the
materials
(across
datasets
and

dataverses)
uploaded
under
any
one
author
and/or

user
should
not
be
possible.”

“SubmiCer
represents
and
warrants
that
the
Content

does
not
contain
any
informa4on
(i)
which
iden4ﬁes,
or

which
can
be
used
in
conjunc4on
with
other
publicly

available
informa4on
to
personally
iden4fy,
any

individual;”

“If
you
are
submiHng
human
sequences
to
GenBank,

do
not
include
any
data
that
could
reveal
the
personal

iden4ty
of
the
source.
It
is
our
assump4on
that
you

have
received
any
necessary
informed
consent

authoriza4ons
that
your
organiza4ons
require
prior
to

submiHng
your
sequences.”
GenBank

How can we maximize
publishing sensitive data while
being mindful of privacy?

Sweeney
L,
Crosas
M,
Bar-‐Sinai
M.
Sharing
Sensi4ve
Data
with
Conﬁdence:
The
DataTags
System.
Technology
Science.
2015101601.

October
16,
2015.
hCp://techscience.org/a/2015101601
The DataTags System

A datatag is a set of security features and access
requirements for ﬁle handling

A datatag is a set of security features and access
requirements for ﬁle handling
A datatags repository is one that stores and shares
data ﬁles in accordance with a standardized and
ordered levels of security and access requirements

Datatags&Levels&
Tag$Type$ Descrip-on$ Security$Features$ Access$Requirements$
Blue$ Public& Clear&storage&
Clear&transmission&
&
Open&
Green$ Controlled$
public&
Clear&storage&
Clear&transmission&
Email,&OAuth&veriﬁed&
registra:on&
Yellow$ Accountable& Clear&storage&
Encrypted&transmit&
Password,&Registered&,&
Approval,&Click&DUA&
Orange$ More$
accountable&
Encrypted&storage&
Encrypted&transmit&
Password,&Registered,&
Approval,&Signed&DUA&
Red$ Fully$
accountable&
Encrypted&storage&
Encrypted&transmit&
TwoDfactor&authen:ca:on,&
Crimson$ Maximally$
restricted&
Mul:Encrypt&store&
Encrypted&transmit&
TwoDfactor&authen:ca:on,&

DataTags Workﬂow in a Dataverse Repository
(under development)
Data$File$
Inges-on$
Sensi-ve$
Dataset$
Direct$
Access$
Privacy$
Preserving$
Access$
Automa-c$
Interview$$
Review$Board$
Approval$
hCp://datatags.org
hCp://privacytools.seas.harvard.edu
Two-‐factor

Authen4ca4on;
Signed
DUA

Thanks!
And join us to this year’s
Dataverse Community Meeting

References
• http://dataverse.org
• http://dataverse.harvard.edu
• http://datatags.org
• Sweeney L, Crosas M, Bar-Sinai M. 2015, Sharing
Sensitive Data with Conﬁdence:The DataTags System.
Technology Science, hCp://techscience.org/a/2015101601
• Gross Harmon, Reidy, 2001, Communicating Science
• Mabe,
2003,
The
Growth
and
Number
of
Journals
• Friendly,
2006,
A
Brief
History
of
Data
Visualiza4on

The Rise of Data Publishing in the Digital World (and how Dataverse and DataTags Help)

More Related Content

Similar to The Rise of Data Publishing in the Digital World (and how Dataverse and DataTags Help) (20)

More from Merce Crosas (20)

Recently uploaded (20)

The Rise of Data Publishing in the Digital World (and how Dataverse and DataTags Help)