But Were We Successful: Using Online Asynchronous Focus Groups to Evaluate Library Services

But Were We Successful?
Using Online Asynchronous Focus Groups to Evaluate
Library Services
Rebekah Cummings
rebekah.cummings@utah.edu
Lindsay Ozburn
lindsay.ozburn@usu.edu
Andrea Payant
andrea.payant@usu.edu
Mike Shelton
michael.shelton@usu.edu
Ryan Bushman
ryan.bushman@usu.edu
Utah Library Association Conference - May 20, 2021

Why Was RDMS Created?
Why did Utah State University create a research data management services
program?
RDMS was created to comply with federal mandates stemming from the
2013 OSTP Memo on “Increasing Access to the Results of Federally
Funded Scientific Research.”

2013 OSTP Memo
In 2013 the Office of Science and Technology Policy issued a memorandum
titled:
“Increasing Access to the Results of Federally Funded Scientific Research”
• “The Administration is committed to ensuring that, to the greatest extent
and with the fewest constraints possible and consistent with law and the
objectives set out below, the direct results of federally funded scientific
research are made available to and useful for the public, industry, and the
scientific community. Such results include peer-reviewed publications and
digital data.”
• Stipulated that federal agencies with over $100M in R&D had to develop
plans to make publications and data that originated from federally funded
research freely available to the public.

What This Meant for Researchers:
● The need to maintain and make their data publicly
accessible, generally through data repositories.
● The need for their publications to be open access
and to generally have them listed in the funder’s
public access repository.
● The need to include information about any data
and publications created with their annual reports
to their funders.
● The inclusion of a data management plan with the
award applications.

Utah State University recognized several needs necessary to help Primary Investigators (PI)
maintain compliance with funder mandates:
USU’s Response to the OSTP Memo:
• The need to create a Data Librarian position to assist researchers with their publications and data: Fulfilled in
Summer 2015.
• Bring campus stakeholders together to access needs and create a compliance program: In Autumn 2015 a cross-
campus group with members from the Office of Research and Graduate Studies (RGS), the Office of Information
Technology (IT), and the Merrill-Cazier Library started meeting to access how to best monitor and assist USU
researchers’ compliance.
• The needs to create and implement a University Research Data Policy and a process to create auditable records to
facilitate access to data and publications: By Autumn 2016 a plan was implemented that included the process for
award compliance tracking

USU’s Approach to Compliance:
USU decided to take an approach that facilitates the researchers’ ability to be in
Funder Compliance
USU worked to develop a compliance process that would:
• Demonstrate to Funders that the University is making good faith effort to comply with mandates.
• Emphasis to researchers that complying with funder mandates is an institutional priority.
• Demonstrate to researchers that resources are available to support them in meeting funder mandates to
make data openly accessible.

But Were We Successful: Using Online Asynchronous Focus Groups to Evaluate Library Services

Continued Support for RDMS from Administration
Both the Library and University administration recognizes that compliance to funder
mandates is an institutional priority. They continue to support RDMS in a variety of
ways:
• Monthly meetings of the University’s Research Data Management Committee, which is made up members from the Library
and the Office of Research. These meeting serve to discuss the needs of both RDMS and the researchers, news in the data
management field, and to develop instructional opportunities with the campus community.
• Once a semester the Library (via RDMS’s supervising Associate Dean) sponsors Data Club, a working lunch where PIs are
invited to visit with the RDMS staff and a representative from the Research Office to discuss their data concerns.
• In February 2020, the Library and Research office co-sponsored RDMS in conducting a half-day workshop called
“Datapalooza” geared toward teaching graduate students good data management practices. This was followed up in April
2021 with a second Zoom based Datapalooza event.

Establishing the Need for Program Assessment
•Association of American Universities (AAU) and the Association of Public & Land-Grant Universities
(APLU) working group conference
•Two members of RDMS, as well as the Interim Vice President of Research represented USU.
•Outcome: recognizing the need to assess the quality and effectiveness of the University’s
compliance program via RDMS’s efforts.

Assessing the compliance of federal mandates:
the library’s involvement
Early 2010’s…
• Federal organizations started requiring Data Management Plans to ensure research reproducibility
• Research reproducibility becomes a hot topic in LIS, with libraries examining what role they might play in
assisting their campus’ researchers with the reproducibility and accessibility of empirical research (Vitale,
2016)
o This led to a shift in academia = libraries become increasingly more engaged in campus research efforts and support.
• Emergence of RDMS programs and technical infrastructure as a support mechanism

Assessing the compliance of federal mandates:
the library’s involvement
Late 2010’s…
• Librarians and professional library orgs are more involved than ever in shaping policy and best practices for
RDMS in support of researchers
o Sparsely explored topic in LIS literature
• This research addresses the gap in literature and assessment of RDMS efforts

Our Assessment Methods
• Traditional survey coupled with an online asynchronous focus group (OAFG)
• Why two methods? Data triangulation
o Survey gave us a broad look at what works and what could be improved. OAFG dug
deeper into these questions to give us specifics for improvement
• OAFG discussion questions modeled after survey questions and results

Asynchronous Focus Groups as a Methodology: What are they?
• Focus groups conducted through online mediums where participants do not need
to be in the same place or online at the same time.
o Typically, they are given a time frame (2 day, for example) in which to participate
• Who uses them?
o Marketing- generating consumer insights that inform response to advertising
o Health - discussing healthcare issues with professionals across the globe (Han et al.,
2019; Ramo et al., 2019)

Pros and Cons of Asynchronous Focus Groups
Pros:
● Not bound in time and space as much; as
people can assemble to participate in
whatever time frame you give them
● Often, they have the potential to be
anonymous, which is helpful for sensitive
topics
● A lack of geographic boundaries helps increase
the diversity of a participant pool for specific
types of research projects
● In academia, busy faculty schedules are less of
a barrier for participation
Cons:
• Still dealing with the influence of social cues in
focus groups but in a much more presumptive
way (i.e., lack of vocal tone and eye contact can
lead to assumptions about someone’s answer).
• Previous forum posts can influence future posts
Why? While this is the case when speaking,
participants get more time to craft their answers
in online forums, leading to a higher potential
for influencing factors
• Ethical considerations of cyberspace: anonymity
could lead to people misrepresenting who they
are and/or sharing other participant’s responses
with others.

How did we use OAFG to assess our program?
• We used Canvas, an online learning platform that both USU and U of U had access
to.
• Of the team, we used the librarian at U of U as an outside moderator to encourage
honesty and to scrub data of identifiable information.
o Outside moderators are important with focus groups, in general, to prevent
unintentional biases in questions and to encourage honest answers from participants.

Hold Up.
You were assessing participants at one location
Why did you use an OAFG?

Qualities of Survey
• Population: 78 Researchers
• Sample: 28 Researchers
o Initial response of 32 researchers ( 41.03% of the population)
o Data cleaning determined 28 of respondents provided usable data (34.6% of population)
• Introduction of Bias
o We opted for a convenience sample rather than a randomized sample

The Survey
Two styles of question:
• Questions designed to determine the frequency of a behavior of the researcher often used the following
scale
o Never
o 0-3 times (a redundancy we would like to fix in further research projects)
o 4-7 times
o 8-10 times
o More than 10 times
• Questions designed to determines a researcher's response to USU’s efforts used the following scale
o Strongly Disagree
o Somewhat Disagree
o Neither Agree Nor Disagree
o Somewhat Agree
o Strongly Agree

Key Statistics
• 19 of the 27 (70.4%) respondents indicated they had three or fewer awards that
required deposit of publications into a public access repository
• 18 of the 27 (66.67%) respondents indicated they had three or fewer awards that
required deposit of data into a publicly accessible repository.
• “Could Sponsored Programs or the Library better facilitate efforts to meet agency
mandates?”
o 13 of 27 (48.15%) responded yes
o 14 (51.85%) responded no.
o When examined by years as a researcher, a higher percentage of respondents in the range of 11-20
years as a researcher responded “yes” to this question.

Key Statistics: Behavior of the Researcher
• Motive for making data publicly available
o 84.6% to comply with journal publishing requirements
o 92.6% to satisfy a funder
o 61.5% with the desire for their data to be public
• 92.59% of respondents indicated that they anticipated writing journal publications
up to 5 years after the close of their award
• 88.46% of respondents anticipated needing up to 4 years after the close of their
award to have final data ready for deposit in a public repository

Key Statistics: Response to USU’s Efforts
• The email from SPO was helpful
o 20 (74.07%) strongly agree
o 5 (18.52%) somewhat agree
• Researchers understand how to respond to the email
from SPO
o 5 (18.52%) somewhat disagree
• The email from SPO explains how they could find more
help
• The email from DL was helpful
o 5 (29.41 % somewhat agree
o 1 (5.88%) somewhat disagrees
• Researchers understand how to respond to email from
DL
o 2 (11.76%) neither agree nor disagree
• The email from DL explains how they could find more
help

Online Asynchronous Focus
Groups (OAFG)
SOCIALLY DISTANCED BEFORE IT
WAS COOL

• SECURE
• FAMILIAR
• ALLOWED FOR DISCUSSION
• RESPOND IN LINE
• TECHNICAL SUPPORT
Technology

Five Discussions
Introductions Response to funder
mandates
Researcher data
practices &
publication deposit
Research
Office and Library
program
What would help?

First Attempt
• Selected 15 participants at random
• VP for Research invited the fifteen participants
to join the OAFG at their convenience the
following day
• With only a single day’s notice, only two of the
fifteen participants responded, and they,
inexplicably, couldn’t get into the Canvas site.

Second Attempt
• Moved Canvas site from University of Utah to USU
• Gave participants several weeks of advance notice
• Confirmed attendance with five participants, but only three
showed up
• Good news! The three participants gave excellent, detailed
feedback, interacted with one another, and didn’t have any
technical issues

Qualitative Data Analysis
1. Praise for the Data Librarian, the library, and the research office (16
statements)
2. Data sharing challenges (6 statements)
3. Data resources (such as DMP Tool, FigShare, Digital Commons,
Metavist,Morpho) (10 statements)
4. Appropriate time to make data available (6 statements)
5. Open Access (7 statements)
6. Actionable steps USU can do to help researchers (6 statements)

Findings – Participants don’t love data sharing
“Honestly, when I first heard that Federal agencies required
researchers to make all the raw data available to the public I
thought: “Really? This is useless and opens the door for a lot of
plagiarism.” I still feel that way a little bit. It’s one thing to have open
access journals where everybody has the chance to download the
paper, but making data available (even after the paper has been
published) is a little bit too risky… I don’t make any data available
until after the paper has been published.”

Findings – Participants appreciate current services
“What helps me? Betty!!! She is great! As
[another participant] mentioned, when I have
to upload anything, Betty is my first point of
contact. I know that I get excellent information
from her.”

Findings – What could help?
• Meet with the Data Librarian after receiving an award
• Checking to see what publications can be uploaded and updating PI records
accordingly
• Depositing data and publications on behalf of researchers
• Custom support for any agency requirements
• Sending reminders beyond the close of the award, until the PI notified the Data
Librarian that all data and publications related to the award have been deposited
• Help paying open access fees.

Summary
• Pioneering the USU compliance tracking program
• Identifying the need for a program assessment
o Goals
 Understanding researcher behavior
 Evaluating response to USU’s compliance tracking program
o Methods
 Survey
 OAFG
• Qualitative data analysis and findings

Lessons Learned
• Survey
o Data representative of a portion of researchers
 Convenience vs. Randomized – taking a randomized sample makes the data more representative of
the population
o Work with someone that has statistical expertise
• OAFG
o Timing and buy in
o Platform needs

How our results will shape future services
• Clarification or requirements expected by individual funders
o Continue updating to data management services website
o Continue to tailor reminder emails to include links to resources for each PI’s agency information about
requirements
o The library is currently changing the workflow to include additional help post close-out of awards
o Improved communication from the USU Libraries and Sponsored Programs
• Further investigation and research
o Improving interaction with researchers less inclined to think Sponsored Programs/USU Libraries can help
them meet agency requirements to deposit publications and data

Future of the Project
• Developing a culture of responsible data stewardship
o Understanding research hesitancy to share data
o Further sessions
 When data should be released
 Data citation
 Data licensing
 Effective methods for describing research data

Read our article:
Rebekah Cummings, Lindsay Ozburn, Andrea Payant, Betty Rozum, Michael Shelton & Ryan Bushman
(2020) Assessing Research Compliance for Federally Funded Projects: The Good, the Bad, and the
Publicly Accessible, Journal of Library Administration, 60:7, 726-751, DOI:
10.1080/01930826.2020.1786985
Learn More

Thanks!
Do you have any questions?
Rebekah Cummings
rebekah.cummings@utah.edu
Lindsay Ozburn
lindsay.ozburn@usu.edu
Andrea Payant
andrea.payant@usu.edu
Mike Shelton
michael.shelton@usu.edu
Ryan Bushman
ryan.bushman@usu.edu

But Were We Successful: Using Online Asynchronous Focus Groups to Evaluate Library Services

More Related Content

What's hot (20)

Similar to But Were We Successful: Using Online Asynchronous Focus Groups to Evaluate Library Services (20)

More from Andrea Payant (14)

Recently uploaded (20)

But Were We Successful: Using Online Asynchronous Focus Groups to Evaluate Library Services