Assessing Perceived Usability of the Data Curation Profiles Toolkit Using the Technology Acceptance Model

Assessing Perceived Usability of the
Data Curation Profiles Toolkit
Using the Technology Acceptance Model
Tao Zhang, D. Scott Brandt
Purdue University
USA
Lisa Zilinski
Carnegie Mellon University
USA
Jake Carlson
University of Michigan
USA
IDCC 2015, London

The Data Curation Profiles Toolkit (DCPT)
 Created at Purdue for librarians to engage
researchers in discussion about data
 Interview protocol
 Capture information about a dataset across lifecycle
 Explore how data are used and managed
 Identify data curation needs
 Document generated from data interview
2

Using the DCPT
 Three-stage process
3
Preparation Interviews
Constructing
DCP
 DCPs as community resource
 Understand researcher needs with data
 Inform development of data services
 Data Curation Profiles Directory
http://docs.lib.purdue.edu/dcp/

Assessing the DCPT
 Various data collected on how the DCPT has been
used
 Study on effectiveness of the DCPT
 Carlson (2013): Increased users’ confidence in
discussing data sharing, but time and effort for
developing DCP as barrier to use the DCPT
 Brandt & Carlson (2013): Users recognized utility and
impact, and strongly suggested further enhancement
for data curation needs (as opposed to data
management)
4

Motivation
 Formal and structured assessment of the DCPT to
reveal factors affecting:
 User perception
 Intention to use
 Difficulties
 Areas to be improved
 Challenge for usability evaluation
 Time limit
 Task performance measures
5

Technology Acceptance Model (TAM)
 Perceived usability (PU and PEOU)
 Predict user acceptance and actual usage of technical
systems/tools
 Critical for overall user experience
6
Source: Davis (1989)
Perceived Usability

Methodology
 Survey of potential determinants of perceived usability
(28 questions measured in 5-point Likert scale)
 “I have experience in conducting one-and-one interviews.”
 “It takes __ time to learn the DCP Toolkit.”
 “I can adjust the questions in the DCP Toolkit for use in
different situations.”
 PU, PEOU, and Intention to Use the DCPT
 Questionnaire measures adopted from Davis (1989)
 Open-ended questions
 Difficulties and obstacles
 Areas that user liked and could be improved
7

Methodology
 Survey sent to 895 registered users of the DCPT
website in Dec. 2013
 28 measures of determinants
 221 responses (24.7%) in a month
 Most respondents are professional librarians with data
management related responsibilities
 Data analysis
 Likert ratings of determinants-> Exploratory Factor
Analysis -> Regression Analysis
 Qualitative analysis of open-ended responses
8

Exploratory Factor Analysis
9
Measure 1
Measure 2
Measure 3
Measure N
……
Factor 1
Measure 4
Factor 2
 Correlations between
variables result from
sharing of factors
 Uncover underlying
structure of a large set
of measured variablesFactor n
……

Factor Analysis Results
 Seven factors extracted for 84.2% of total variance
 Applicability
 Time
 Complexity
 Experience and Share
 Training and Help
 Extensibility
 Interviewee Requirements
 Factors in regression models to predict PU, PEOU,
and Intention to Use
10

Regression Results
 Multivariate, stepwise regression models
[PU PEOU Intention to Use] = Factors 1-7 + error
11
Perceived Usefulness Estimated β
Applicability 3.00
Experience and Share 2.38
Training and Help 1.17
Perceived Ease of Use Estimated β
Applicability 1.12
Time -1.11 *
Complexity -1.59
Interviewee Requirements -1.71
Intention to Use Estimated β
Applicability 0.60
Time -0.41
Complexity -0.38
Training and Help 0.29
Extensibility 0.33
*p-value = 0.064, sample size > 250 needed to
show significance

Regression Results
 Multivariate, stepwise regression models
[Intention to Use] = PU, PEOU + error
 All regression models R2 around 0.5
12
Intention to Use Estimated β
PU 0.156
PEOU 0.054 (p = 0.07 > 0.05)

Open-ended Questions
 Questions asked in the survey:
 “If you used the DCP Toolkit in the past, did you encounter any
difficulties? If yes, please explain.” (n=46)
 “If you plan to use the DCP Toolkit, what would be the obstacles
you may encounter?” (n=62)
 “What are the things you like about the DCP Toolkit?” (n=69)
 “What are the things you think should be improved in the DCP
Toolkit?” (n=54)
 Qualitative analysis method
 Two independent researchers reviewed and coded responses
 Two iterations, 66.8% agreement on initial coding and 100%
consensus on final coding results
13

Open-ended Responses
 Themes:
 Amount of time required to use the DCPT vs. depth of
information from completed DCP
 Structure and format of the toolkit
 Alignment of the DCPT with particular context
 Using the DCPT to engage faculty and library
community
14

Open-ended Responses
 Finding the right balance
 Time required for both researchers and interviewees
 Depth of information in DCP as good utility
 Applicability
 Adapting structure and format to contexts
 Making decisions based on results
 Extending the DCPT
 Compact, “lite version”; online tool
 Focus on particular data types or fields
 Community building based on DCPs
15

Quantitative vs. Qualitative Results
 The balance between time and value
 Interviewee Requirements affect Perceived Ease of Use
 Complexity and Time affect Intention to Use
 Time requirement vs. thoroughness in open-ended
responses
 Training and Help
 Significant in regressions on Perceived Usefulness, Intention
to Use
 Open-ended responses requested additional help on:
 Adaptations for different purposes
 Transforming collected information into DCPs and making
decisions
18

Conclusion
 Technology-Acceptance Model useful for
assessment
 Factor analysis & regressions
 Significant factors identified for Perceived Usability &
Intention to Use
 Open-ended responses complement quantitative
results
 Usability improvement of the DCPT
 Reduce time requirement
 Increase flexibility
 Training and help
19

DCP: What We’ve Learned & Going Forward
20
Preparation
& Training
Interview/
Interact
Profiles/
Best
Practices
Utilizing
Outputs
Preparation Interviews
Constructing
DCP
DCP 1.0
DCP 2.0
DCP 2.0 Roadmap Workshop,
Purdue University, June 2015
Paper-based
Tech-based

Thank you!
Tao Zhang, D. Scott Brandt
Purdue University
USA
zhan1022@purdue.edu
techman@purdue.edu
Lisa Zilinski
Carnegie Mellon University
USA
ldz@cmu.edu
Jake Carlson
University of Michigan
USA
jakecar@umich.edu
IDCC 2015, London

Assessing Perceived Usability of the Data Curation Profiles Toolkit Using the Technology Acceptance Model

More Related Content

What's hot (8)

Viewers also liked (8)

Similar to Assessing Perceived Usability of the Data Curation Profiles Toolkit Using the Technology Acceptance Model (20)

More from Tao Zhang (14)

Recently uploaded (20)

Assessing Perceived Usability of the Data Curation Profiles Toolkit Using the Technology Acceptance Model

Editor's Notes