The Value of Evaluation

The Val ue of Eval uat i on
John J. Heldrich Center for Workforce Development
April 8, 2011
Presentation to the New Jersey County Colleges’
Institutional Advancement Affinity Group
William Mabe, PhD, Director of Research and Evaluation

John J. Heldrich Center for
Workforce Development
22
About the Heldrich Center
 The Heldrich Center is a research and policy institute
at Rutgers dedicated to raising the effectiveness of
the American workplace by strengthening workforce
education and training programs
 Founded in 1997, the center employs 18 full-time
professional staff and faculty representing an array of
disciplines, from economics and human services to
business and public policy

3
Heldrich Center Research and Evaluation
 Have evaluated over 30 education and workforce
programs
 Selected evaluation projects include:
– A profile of performance outcomes of New Jersey’s community
colleges
– Evaluation of Kessler Foundation funded grantees
– Evaluation of US Department of Education financed Parent
Information and Resource Centers

44
Benefits of Evaluation
• Learn what works and what does not work so
resources can be directed to most effective programs
• Identify barriers to success and program weaknesses
• Provide evidence for program sustainability
• Document best practices for replication

5
What Evaluation is NOT

66
Evaluation Is Not
• Auditing
• A gotcha
• Needs assessment
• Just about the numbers
• Customer satisfaction

77
Evaluation Is
• Systematic and objective process by which a
researcher assesses the quality, effectiveness, or
value of a program
 Systematic--follows established rules of scientific inquiry
 Objective--any neutral observer would arrive at the same
conclusions about the program if she were to use the
same methods
 Assess program quality by studying how the program
operates
 Effectiveness--whether the program achieves its intended
goal
 Value assessment places the effectiveness of the program
in the context of its costs

88
Types of Evaluations
• Process Evaluations:
 Identify barriers to successful implementation
 Identify which program components are effective and which are
ineffective at helping program achieve its goals
 Recommend strategies for modifying program to strengthen it
 Tend to use more qualitative data
• Outcome Evaluations:
 How well is the program achieving it goals? Is it effective?
 Tend to use more quantitative data

99
When Programs Should Be Evaluated
• Before the program begins. In the early stages of
developing any program, staff should plan the
program to accommodate evaluation
• Data collection should be ongoing
• Program should be evaluated at regular intervals

1010
Life-Cycle Model of Program Evaluation
• Like products, programs have life cycles:
• conceptualization
• piloting
• widespread implementation
• maturity
• (possibly) phase-out
• Type of evaluation to be conducted is a function of
where a program is in the life-cycle

1111
How to Approach Evaluation
• At what stage is the program in its life-cycle?
• What evidence would be needed to convince a
neutral observer that the program is effective?
That it’s being implemented well?
• If this program were successful what would I
expect to observe?
 Look for the observable implications of program success
 Observable implications can be measured, either qualitatively or
quantitatively
• Focus on both process and outcomes

1212
How to Evaluate
• Identify stakeholders
• Types of data: qualitative and quantitative
• Methodologies of data collection
• Individual interviews
• Focus groups
• Surveys
• Observation of program activities
• Collection of programmatic and administrative data
• Some methodologies can be used to collect
qualitative and quantitative data, e.g., surveys
• When to collect data
• Design program so that needed data can be collected from day
one

1313
Data Heldrich Center Has Used for
Program Evaluations
• New Jersey Unemployment Insurance Wage Record Data
• Quarterly census of earnings of over nearly all workers in NJ
• Heldrich Center has access to UI wage records through data sharing
agreement with NJ Dept of Labor and Workforce Development
• Permitted to use for approved research purposes
• Employment Service OSOS Data
• Records of people who received training through public workforce
system
• Heldrich has data sharing agreement with NJLWD
• Valuable for creating comparison groups
• New Jersey Student Unit Record Data System
• Centralized database of student enrollment and graduation records
that CHE collects from you and all public and some independent
institutions in the state
• Heldrich has data sharing agreement with NJCHE to use for approved
research purposes

14
Case Study #1: Essex County College Math
Initiative

15
Program Overview
Semester Key Program Elements
Summer I Pilot
Semester
•Linked developmentalpre-algebracourse and scientific
reasoningcourse
•STEM specific tutors
•Financialsupport
Fall2009 Semester &
Spring 2010 Semester
•Collegelevel Math course with a supported recitation
•STEMspecific tutors
•Financialsupport
•Access to a dedicatedcomputer lab
•Feedbackoriented homework structure.

16
Key Differences: STEM Algebra Course
versus College’s Regular Algebra Course
• Revised curriculum:
– Focus on concepts
– New textbook
– Coverage of more difficult topics
• Mandatory recitation
• Teaching Assistant
• Financial support for students
• Mandatory homework
• Different grading structure

17
Evaluation Design
• Process and outcome evaluations
• Use qualitative and quantitative data collection methods
• Interdisciplinary evaluation team
• Submitted interim reports to provide actionable information
in a timely manner
• Focused on understanding the culture and politics of the
school as part of the evaluation process
• Considered the views of all stakeholders

18
Qualitative Data Collection
Method Time Frame Number of Participants
Course Observations Three times per
course per
semester
Eighteen observations
Instructor Interviews Pre-Post Semester Twelve interviews with five different
instructors
Student Focus Groups Post Semester Five groups consisting of 5-8
students
Tutor Focus Groups Post Semester Four groups consisting of 5-10 tutors
Meetings, phone calls
and e-mails with
administrative
staff
Regularly throughout
the semester –
(Approx. two to three
times per month)
Two

19
Quantitative Data Collection
• Administrative Data
– Program level data is collected from the program administrator
– Includes program participant information
• Preliminary College level data is collected from the
college data administrator
– Includes grade information for ALL students enrolled in College
Algebra
– Retention information, and follow up grade information is
forthcoming

20
Notable Findings from Process Evaluation
• Increased social integration helped students establish a
sense of community.
• The STEM Program promoted small group interactivity and
encouraged students to prioritize self study.
• Students sought opportunities to interact with peers and
more advanced students as a learning strategy.
• Students began to see themselves and their peers as
resources and active participants in a dynamic learning
process
• Poor faculty collaboration and planning led to an ineffective
implementation of the interdisciplinary component of the
learning community.

21
Outcomes Comparison
• 76 percent of students in STEM Algebra course passed
in Spring 2010
• Compared with 54 percent of all non-STEM majors
• Compared with 56 percent of STEM majors

22
Study Limitations
• Results may be an indicator of the redesigned course’s
effectiveness, but it may also be a product of:
– Students self-selected into the STEM program
– Financial incentives may have impacted student behavior in
the STEM class
– The grading structure for the STEM section consisted of more
elements than the grading structure of the other MTH-100
courses
– The curriculum in the STEM section was more difficult than the
curriculum of the other MTH-100 courses

23
Case Study #2: Newark Pre-Apprenticeship
Program

24
Program Overview
• Program for low-income women and minorities in
Newark that prepares them for a union apprenticeship
in the construction industry
• Mostly basic skills preparation but also addresses
other barriers to employment, such as suspended
driver’s licenses
• Participants’ earnings low. Demographics:
predominantly (89 percent) black, some Latinos
• Research Question: Does this program increase
participants’ earnings?

25
How to Answer the Research Question?
 What we really want to know: what would have
happened to participants had they not participated
• Just look at what participants earn after the program?
– No way to know if that’s a “lot”
• See if they make more after the program than before?
– Many entered the program because they were unemployed
• See if they’re doing better than similar people who did
not participate in the program?
– Promising but potentially problematic: applicants more
motivated than non-applicants
– Need a way to compare to similarly motivated individuals
– Need a way to identify similar people

26
Creating a Valid Comparison Group
• Taking motivation into account:
– Comparison pool: individuals who graduated from training
programs through the One Stop in Newark at the same time
as program graduates
• Thousands of people completed training in Newark at
this time, which ones are most similar?
– Similar age
– Same race
– Same sex
– Similar employment and earnings history
• Use statistical software to match program graduates
to most similar completers of training programs in
Newark

27
Study Design
• Data
– New Jersey Unemployment Wage Record data
– Americas One-Stop Operating System
• Methodology
– Matched program participants by social security number with
their records in the NJ UI Wage Record and OSOS data
– Probabilistic matching to generate a comparison group
– Parametric statistical models to estimate the program’s effect
• Sample
– Treatment: 129 individuals who completed the program and
who were 22 years old or older at enrollment
– Comparison: Matched individuals from ASOS data

28
Work Histories of Program Graduates
• Program graduates tend to have:
– Limited employment histories
– Low average earnings
– Limited academic skills (as measured by TABE)
• Twelve percent of graduates had formerly
been incarcerated

29
• Program graduates earn more than similar
individuals who receive One-Stop training.
• Program graduates experience greater wage
growth after training than this comparison group.
• Subgroup analysis: Youth participants in program do
no better than comparison group
Program Graduates Out Earn Comparison
Group Members

30
Earnings Progression (Post-Matching)

31
• Study looked at graduates from 2004/2005 and
earnings through mid-2006, during real estate
boom
• Graduates versus participants
• Selective nature of program means cannot rule
out some differences, though pre-training
earnings help control for ability and motivation
Study Limitations

3232
Questions and Discussion

33
Heldrich Center Contact Information
William Mabe, PhD, John J. Heldrich Center for
Workforce Development, Rutgers University
– billmabe@rutgers.edu or 732-932-4100, ext. 6210
Heldrich Center Website: www.heldrich.rutgers.edu

The Value of Evaluation

More Related Content

What's hot (20)

Viewers also liked (18)

Similar to The Value of Evaluation (20)

Recently uploaded (20)

The Value of Evaluation

Editor's Notes