Building a Culture of Assessment: What I've Learned from Research, Trial, and (a lot of) Error

BUILDING A
CULTURE OF
ASSESSMENT
WHAT I'VE LEARNED
FROM RESEARCH, TRIAL,
AND (A LOT OF) ERROR
Meredith Farkas
Portland Community College

ABOUT ME
➤ Faculty Librarian at Portland Community College (OR)
➤ Previously:
➤ Head of Instructional Services, Portland State University
➤ Head of Instructional Initiatives, Norwich University
➤ Significant research and publishing on what it takes to build
an assessment culture
➤ Former psychotherapist interested in what it takes to really
change culture

“A culture of assessment is an
organizational environment in which
decisions are based on facts, research,
and analysis, and where services are
planned and delivered in ways that
maximize positive outcomes and
impacts for customers and stakeholders.
-Amos Lakos, Shelley Phipps, and Betsy Wilson
Lakos, A. & Phipps, S. E. (2004). "Creating a culture of assessment: A catalyst
for organizational change." portal: Libraries and the Academy, 4(3), 345-361.

MY DEFINITION
➤ Assessment is the norm - a
regular part of practice
➤ User-focused culture
➤ Learning and curiosity driven
➤ Assessment is done for
improvement
➤ Changes and decisions are
made based on what is learned
➤ Planning for new services
always includes planning for
assessment

ASSESSMENT AT MANY INSTITUTIONS
➤ Focused on doing what is
needed to satisfy accreditors or
administrators
➤ Coercive and top-down
➤ Little support
➤ Coalition of the willing
➤ Data is gathered, but rarely
used
➤ Not seen as an integral part of
teaching and learning
➤ Often a lot of cynicism around
assessment

MY EXPERIENCE:
ADVENTURES WITH
COORDINATOR SYNDROME

“
The word ‘Coordinator’ in a position title. Beware it, especially in
combination with a new or non-traditional job niche…
The problem with these jobs is that as often as not, there’s nothing
actually to coordinate. No budget. No dedicated staff. No
established service. ‘Coordinate’ all too often means ‘try to establish
a beachhead by begging your new colleagues to vouchsafe you a few
minutes of their time now and then, knowing that their supervisors
won’t tell them to and you have no authority whatever to demand
anything of them.’
Oh, but the buck will stop with you, Coordinator, on all matters
regarding your service. Be aware, also, that because what you’re
doing is new, you must hit a home run with it, quickly; slow and
steady gains will not do. If you don’t, two damaging assumptions
arise: that the service is a waste of effort, and that you’re a lousy
librarian. You’re a coordinator now; congratulations!
-Gavia Libraria, “The C-word"

YOU SAY YOU NEED SUPPORT? I’VE GOT
YOUR SUPPORT RIGHT HERE.

➤ Farkas, Meredith G. "Building and sustaining a culture of
assessment: best practices for change leadership." Reference services
review 41.1 (2013): 13-31.
➤ Farkas, Meredith G., and Lisa J. Hinchliffe. "Library faculty and
instructional assessment: Creating a culture of assessment
through the high performance programming model of
organizational transformation." Collaborative Librarianship 5.3
(2013): 177-188.
➤ Farkas, Meredith Gorran, Lisa Janicke Hinchliffe, and Amy Harris
Houk. "Bridges and barriers: factors influencing a culture of
assessment in academic libraries." College & Research Libraries
(2014).
➤ Farkas, Meredith. "Accountability vs. Improvement: Seeking
Balance in the Value of Academic Libraries Initiative." OLA
Quarterly 19.1 (2014): 4-7.

“For a scholarship of assessment to thrive, we
must align faculty culture, institutional
structures, and leadership for change.
The importance of this point cannot be
overstated. A meaningful assessment program
is more than just a new activity to be
undertaken, it is a change in how we think
about what we do in higher education.
-Don Haviland “Leading Assessment, from Faculty
Resistance to Faculty Engagement.”

THESE STUDIES SUGGEST SPECIFIC FACILITATING/HINDERING FACTORS
1. Oakleaf, M. & Hinchliffe, L. J. (2008). Assessment cycle or circular file:
Do academic librarians use information literacy assessment data? In
Proceedings of the 2008 Library Assessment Conference: Building Effective,
Sustainable, Practical Assessment.
2. Ndoye, A. & Parker, M. A. (2010). "Creating and sustaining a culture of
assessment." Planning for Higher Education, 38(2), 28-39.
3. Hiller, S., Kyrillidou, M., & Self, J. (2008). "When the evidence is not
enough: Organizational factors that influence effective and successful library
assessment." Performance Measures and Metrics, 9(3), 223-230.
4. Wright, S. & White, L. (2007). SPEC Kit 303, Library Assessment.
Washington, D.C.: Association of Research Libraries.

FACILITATING FACTORS SUGGESTED IN THE LITERATURE
➤ Assessment is a library priority (4)
➤ Commitment from library leadership (1, 3, 4)
➤ Commitment from faculty/staff (2, 3, 4)
➤ Organizational culture (3, 4)
➤ Staff with needed expertise and time (1, 3, 4)
➤ Staff development (4)
➤ Evidence-based decision-making (1, 2, 3, 4)

DEMONSTRATING THAT ASSESSMENT IS A
LIBRARY PRIORITY
➤ Shared understanding of
assessment exists within the
organization
➤ Clear expectations or concrete
goals for assessment
➤ Building an assessment plan

COMMITMENT FROM ADMINISTRATION
➤ Articulates the value of assessment
➤ Assessment is built into ALL planning
➤ Assessment is built into reward structures
➤ Supports library faculty/staff engaging in assessment
➤ Uses assessment data to inform decisions

EXPERTISE AND TIME
➤ Staff need dedicated
time to learn, try, fail,
and learn more
➤ Find ways to capitalize
on expertise outside of
the library

STAFF DEVELOPMENT
➤ Don’t just send one person to
a conference
➤ Opportunities for staff to learn
together
➤ Build a culture of reflective
practice and learning
communities

WHY DON’T LIBRARIANS USE
ASSESSMENT RESULTS?
➤ Too busy with other work
➤ Won’t be rewarded for assessment
work
➤ Lack of expertise
➤ Lack of centralized support within
the library
➤ Not valued by their library
Oakleaf and Hinchliffe. (2008).
“Assessment Cycle or Circular File: Do
Academic Librarians Use Information
Literacy Assessment Data?” Proceedings
of the 2008 Library Assessment Conference.

FACILITATING FACTORS - EXPANDED
1. Clear expectations for assessment
in the library
2. Assessment is a priority of
administration
3. Library has adopted learning
outcomes
4. Library has an assessment plan
5. Library has a shared
understanding of the purpose of
assessment
6. Library faculty/staff are supported
in doing assessment
7. Education and training about
assessment are supported
8. Library has assessment expertise
in-house
9. Library has systems/technologies
to support assessment
10. Library leadership offers explicit
support to get staff doing assessment
11. Data is available to interested
parties
12. Leadership uses assessment data
in decision-making
13. Librarians use results to improve
practice
14. Library culture is user-focused

ULTIMATELY, THIS IS
ABOUT CHANGING
CULTURE

JOHN KOTTER’S LEADING CHANGE: THE ORGANIZATIONAL CHANGE BIBLE

CREATE A SENSE OF URGENCY
➤ Urgency must be driven by
opportunities, not by fear
➤ Appeal to the things that
your audience cares about
(student learning, being
better instructors, etc.)
➤ Sometimes urgency comes
from outside (mandates
from administrators,
outside agencies, etc.)

FORM A GUIDING COALITION
(NOT JUST A COORDINATOR)
➤ Form a group to lead and support
assessment efforts
➤ Membership based on skills,
interest, and politics
➤ They will need admin support
➤ Examples:
➤ Northwestern University
libraryassessment.org/
bm~doc/
4macayeallightingtalk.pdf
➤ Portland Community College
www.pcc.edu/edserv/learning-
assessment/LACAboutUs.html

MacAyeal & Ryan. (2014). Support System: Establishing, Sustaining,and
Growing a Frameworkfor Assessment. Library Assessment Conference.

PCC LEARNING ASSESSMENT COUNCIL

CREATE AND COMMUNICATE A
VISION
➤ Common understanding of
the purpose of our
assessment program
➤ Develop an assessment
plan
OR
➤ Develop concrete
expectations for what
progress in assessment
looks like

EMPOWER OTHERS TO ACT ON
THE VISION
➤ Remove barriers
➤ Time
➤ Lack of knowledge/education
on assessment
➤ Fear of failure
➤ Incentivize assessment
➤ Grants, rewards, recognition
➤ Give staff/faculty the freedom to
determine what and how they
will assess
➤ Demonstrate that assessment
results will be used

CREATE SHORT-TERM WINS (OR “THE
PERFECT IS THE ENEMY OF THE GOOD”)
➤ Start small and simple
➤ Assessments that are easy
to do but yield valuable
learning
➤ What are we curious about?
➤ Debrief together and learn
from each experience
➤ Talk about changes made
based on assessment results

CONSOLIDATE IMPROVEMENTS AND
INSTITUTIONALIZE NEW APPROACHES
➤ Develop formal mechanisms
for reporting and sharing
assessment results
➤ Build mechanisms for
discussing assessment results
within the library
➤ Demonstrate evidence-based
decision-making
➤ Build assessment into
planning
➤ Build assessment into reward
structures (tenure?)

THERE ARE LIMITATIONS WITH
THE LITERATURE WE RELY ON TO
MAKE CLAIMS ABOUT BUILDING
AN ASSESSMENT CULTURE

1. Oakleaf, M. & Hinchliffe, L. J. (2008). Assessment cycle or circular file:
Do academic librarians use information literacy assessment data? In
Proceedings of the 2008 Library Assessment Conference: Building Effective,
Sustainable, Practical Assessment.
2. Ndoye, A. & Parker, M. A. (2010). "Creating and sustaining a culture of
assessment." Planning for Higher Education, 38(2), 28-39.
3. Hiller, S., Kyrillidou, M., & Self, J. (2008). "When the evidence is not
enough: Organizational factors that influence effective and successful library
assessment." Performance Measures and Metrics, 9(3), 223-230.
4. Wright, S. & White, L. (2007). SPEC Kit 303, Library Assessment.
Washington, D.C.: Association of Research Libraries.
Survey sent
to an infolit
listserv
Based on their experience with
research libraries that want to
build an assessment culture
Sent to an assessment
listserv
Survey of ARLs only

Our goal:
A systematic survey of culture of assessment and
investigation of factors that facilitate or hinder
libraries.
A survey of libraries, not librarians

THE SURVEY
➤ 60 Question Survey
➤ 10 descriptive questions about the institution or the
individual responding for the institution
➤ 47 yes/no questions
➤inquiry about presence/absence of 16 facilitating factors
named in other publications
➤does the presence/absence facilitate or hinder culture of
assessment
➤ 3 open-ended questions about what most facilitates or hinders
creation of a culture of assessment
➤ Respondents presented with different questions depending on
their answers to presence/absence of facilitating factors
(total questions for each respondent = 41 or 42)

FACILITATING FACTORS
1. Campus-wide assessment initiative
2. Library involvement in campus-
wide assessment initiative
3. Clear expectations for assessment
in the library
4. Assessment is a priority of
administration
5. Library has adopted learning
outcomes
6. Library has an assessment plan
7. Library has a shared understanding
of the purpose of assessment
8. Library faculty/staff are supported
in doing assessment
9. Education and training about
assessment are supported
10. Library has assessment expertise in-
house
11. Library has systems/technologies to
support assessment
12. Library leadership offers explicit
support to get staff doing assessment
13. Data is available to interested parties
14. Leadership uses assessment data in
decision-making
15. Librarians use results to improve
practice
16. Library culture is user-focused

Stage 1: Winter 2013
Libraries at all BA/MA/PhD-granting
institutions
Received a 42% response rate

Results presented at ACRL 2013 and
published in College & Research Libraries
Farkas, Meredith Gorran, Lisa Janicke Hinchliffe, and Amy Harris
Houk. "Bridges and Barriers: Factors Influencing a Culture of
Assessment in Academic Libraries." College & Research Libraries
76.2 (2015): 150-169.

BUT COMMUNITY
COLLEGES ARE DIFFERENT

Stage 2: Spring 2014
Libraries at all U.S. public community
colleges
Received a 32% response rate (n=292)

WOULD YOU DESCRIBE YOUR LIBRARY AS HAVING A
CULTURE OF ASSESSMENT, WHERE ASSESSMENT IS A
REGULAR PART OF YOUR INSTITUTIONAL PRACTICE?

THIS IS HIGHER THAN AT FOUR YEAR AND ABOVE
INSTITUTIONS?

IS THERE A CAMPUS-WIDE ASSESSMENT INITIATIVE AT
YOUR COLLEGE?
Also higher than at 4 year and above institutions (84.9%)
Only 80% of these libraries are involved in the campus-wide
assessment initiative

IS ASSESSMENT A PRIORITY OF LIBRARY
ADMINISTRATION?
Higher than at 4 year and above institutions (77.8%)

IS THERE A SHARED UNDERSTANDING OF THE PURPOSE OF
ASSESSMENT IN YOUR LIBRARY?

ARE THERE CLEAR EXPECTATIONS FOR ASSESSMENT IN
YOUR LIBRARY?

DOES YOUR LIBRARY HAVE AN ASSESSMENT PLAN?

ARE LIBRARY FACULTY/STAFF ADEQUATELY SUPPORTED IN
THEIR ASSESSMENT WORK?

DOES YOUR LIBRARY HAVE THE NECESSARY SKILLS IN-HOUSE
TO DEVELOP AND CONDUCT MEANINGFUL ASSESSMENTS AND
ANALYZE THE RESULTS?
Higher than at 4 year and above institutions (56%)

DOES THE LIBRARY LEADERSHIP USE ASSESSMENT DATA
SYSTEMATICALLY IN DECISION-MAKING AND PLANNING?

DOES THE LIBRARY STAFF USE ASSESSMENT DATA
SYSTEMATICALLY IN DECISION-MAKING AND PLANNING?
Higher than at 4 year and above institutions (71%)

ON WHICH FACILITATING FACTORS DID COMMUNITY
COLLEGES SCORE SIGNIFICANTLY (P<0.05) HIGHER?
➤Is there a shared understanding of the purpose of
assessment in your library?
➤Are library faculty/staff adequately supported in their
assessment work?
➤Does the library leadership use assessment data
systematically in decision-making and planning.

WHAT FACTORS ARE
ASSOCIATED WITH HAVING A
CULTURE OF ASSESSMENT?

REGIONAL ACCREDITING ASSOCIATION IS ASSOCIATED
WITH A CULTURE OF ASSESSMENT

TENURE STATUS AND FACULTY STATUS IS ASSOCIATED
WITH A CULTURE OF ASSESSMENT
p=0.02
This was not the case in the study of four year and above institutions

HAVING A CAMPUS-WIDE ASSESSMENT INITIATIVE IS
ASSOCIATED WITH A CULTURE OF ASSESSMENT
P<0.001

ALL OF THE FACILITATING
FACTORS ARE SIGNIFICANTLY
ASSOCIATED WITH A CULTURE
OF ASSESSMENT

FACTORS WHOSE PRESENCE IS MOST STRONGLY
ASSOCIATED WITH A CULTURE OF ASSESSMENT
➤Clear expectations for assessment in the library (90%
who have it have an assessment culture)
➤Assessment plan (82%)
➤Shared understanding of the purpose of assessment in
the library (81%)
➤Library leadership uses assessment data (78%)
➤Library leadership offers explicit support (77%)

FACTORS MOST STRONGLY ASSOCIATED WITH NOT HAVING
A CULTURE OF ASSESSMENT
➤Assessment is not a priority of administration (77%
who report this do not have an assessment culture)
➤Librarians do not use assessment data (72%)
➤Assessment data is not available to interested parties
(74%)
➤Library leadership does not use assessment data
(70%)
➤No shared understanding of the purpose of assessment
(69%)

WHAT FACTORS ARE MOST OFTEN PERCEIVED AS
FACILITATING?
➤ Clear expectations for assessment in the library (94%
who perceive that this facilitates their assessment
culture have an assessment culture)
➤ Library has an assessment plan
➤ Librarians use assessment data to improve practice
➤ Library leadership uses assessment data
➤ Existence of learning outcomes

OPEN-ENDED RESPONSES: WHAT ARE THE MOST
IMPORTANT FACILITATING FACTORS?
➤Institution-wide emphasis on assessment
➤Administrative priority and/or support
➤Accreditation
➤Organizational culture
➤Information literacy as a campus-wide outcome

OPEN-ENDED RESPONSES: WHAT MOST HINDERS
CREATING AN ASSESSMENT CULTURE?
➤Lack of sufficient staffing
➤Lack of time
➤Faculty or staff have not made assessment a priority –
either due to lack of interest or resistance to
assessment
➤Lack of expertise/access to training
➤Institution doesn’t prioritize assessment

“CULTURE OF ASSESSMENT” IS IN THE EYE OF THE BEHOLDER
http://www.flickr.com/photos/kylemay/1573158163/

WITHOUT AN INSTITUTIONAL COMMITMENT TO ASSESSMENT,
A CULTURE OF ASSESSMENT IS FAR LESS LIKELY TO EXIST
Institutions with a
campus-wide
66% of libraries report
having an assessment
culture
Institutions with no
campus-wide
36% of libraries report
having an assessment
culture

THERE ARE SIGNIFICANT ASSOCIATIONS WITH TENURE AND
FACULTY STATUS THAT ARE NOT PRESENT AT FOUR YEAR AND
ABOVE INSTITUTIONS

HAVING A PLAN IS CRITICAL
Fewer than half of libraries...
➤ Have clear expectations for assessment (52%)
➤ Have an assessment plan (45%)
Of libraries reporting clear expectations and an
assessment plan, 92% report having a culture of
assessment (versus 32% of those who report the
absence of both)

HOW ADMINISTRATION PROMOTES ASSESSMENT MAKES A
DIFFERENCE

Of those indicating assessment is a priority of administration
74% report having a culture of assessment
 
Of those indicating administration uses assessment data in decision-
making
 
 
Of those indicating administration adequately supports faculty/staff in
assessment work
 
 
Of those indicating a library has all three factors
 

HAVING INSTRUCTION AND ASSESSMENT LEADERS ALSO
HELPS

Of those academic libraries with an instruction coordinator
 
Of those with an assessment coordinator
 
 
Of those with an instruction committee
 
 
Of those with an assessment committee
 
Of those with none of these positions
 

MOST COMMUNITY COLLEGE
LIBRARIES ARE
ASSESSMENT LEADERS

CONSIDER
➤ What has helped your library move toward (or achieve!) a
culture of assessment?
➤ What has hindered your library in moving toward a culture of
assessment?
➤ Which of these facilitating factors seem most important for
you to work toward at your institution?

Questions? Comments? Challenges?
Meredith Farkas
Portland Community College
Email: meredith.farkas at pcc.edu
Twitter: librarianmer

Building a Culture of Assessment: What I've Learned from Research, Trial, and (a lot of) Error

More Related Content

What's hot (19)

Viewers also liked (6)

Similar to Building a Culture of Assessment: What I've Learned from Research, Trial, and (a lot of) Error (20)

More from Meredith Farkas (20)

Recently uploaded (20)

Building a Culture of Assessment: What I've Learned from Research, Trial, and (a lot of) Error