Beyond Smiley Sheets: Measuring the ROI of Learning and Development

Beyond Smiley Sheets:
Measuring the ROI of Learning
and Development

By: Keri Bennington
Account Director
UNC Executive Development

Tony Laffoley
Program Director
UNC Executive Development

All Content © UNC Executive Development 2012
Website: www.execdev.unc.edu |Phone: 1.800.862.3932 |Email: unc_exec@unc.edu

Beyond Smiley Sheets: Measuring the ROI of Learning and Development

Introduction

A recent report by the Chartered Institute of Personnel & Development found that
evaluation was a top priority among learning and development (L&D)
professionals (Personnel Today staff, 2012). Despite this fact, calculating return on
investment (ROI) on development programs is still rarely done; a McKinsey Quarterly
report found that only 8 percent of organizations actually evaluate the value of L&D
initiatives (Palmer, 2010). And too often, those who do track ROI rarely go beyond
asking for feedback from participants immediately after the event.

With today’s challenging economy, L&D budgets are receiving more scrutiny than
ever. Participant feedback forms (i.e., smiley sheets) administered immediately after a
learning program are no longer enough, and HR and talent management professionals
are feeling the pressure to look for more solid evidence to justify the investment in
their programs. This is particularly the case in leadership development programs,
where the focus is often on the development of intangible skills. Because L&D
programs often provide more long-term value rather than short-term effects, senior
leaders may consider eliminating them as an easy way to cut costs. Even if executive
sponsors are satisfied today, they may not be tomorrow. It makes sound fiscal sense
to go beyond smiley sheets and to establish robust measures that capture ROI so that
even the most critical of reviewers can see the value of L&D programs in an
organization.

In the mid-1990s, Laurie Bassi, then a researcher for the American Society for Training
and Development, discovered that the more a company invested toward developing
employees, the higher its stock value went the following year (McCann, 2011). Her
research put actual dollar figures to something L&D professionals have known for
years-- investing in employees pays off. Learning opportunities result in higher levels
of employee promotion, retention, satisfaction, skills and knowledge, and this
translates to better organizational performance. Yet connecting the dots by
demonstrating a real bottom-line ROI remains a continued challenge.

All Content © UNC Executive Development 2012 2|Pa ge


To help connect the dots and overcome the challenge of measuring the ROI of
development programs, this white paper:

Reviews the classic four-level model of evaluation.

Discusses the challenges in assessing value for new and existing development
initiatives.

Offers suggestions on how to ensure that L&D evaluations reflect what
executive leadership expects.

Provides steps to consider when evaluating the ROI of development programs.

Shares examples of companies that have effectively demonstrated the value of
their L&D programs.

Kirkpatrick’s Four Levels of Evaluation
D onald Kirkpatrick is perhaps best known for creating the classic four-level model
of evaluation. His simple paradigm—which still serves as an excellent framework
when evaluating development programs—identified four distinct evaluation tiers:

1. Reaction

2. Learning

3. Behavior

4. Results

The first level, reaction, captures participants’ satisfaction with the experience
immediately following the event. These smiley sheets gather participants’ thoughts
and feelings about the program, the moderator, the content, the venue, etc. Level two,
learning, captures the increase in knowledge, skills or capabilities as a result of the
experience.

Level three, behavior, assesses the transfer of learning—whether participants
successfully applied what they learned to their work. This level can also involve
assessing changes in behavior and attitudes that result from the experience. The final
level, results, assesses participants’ changes in performance or behavior and how
those changes have benefited the entire organization (i.e., bottom-line results).



Kirkpatrick argued that to obtain a meaningful evaluation, it was necessary to evaluate
the L&D experience at each level. This has proven easier said than done. Most
learning professionals find it a challenge to assess beyond the first level to
demonstrate long-term learning and organizational benefits, and it is this longer-term,
bottom-line impact that many senior leaders want to see.

The Challenges in Assessing the
Effectiveness of a Learning Program
N ot surprisingly, evaluating effectiveness is particularly challenging when the
targeted outcomes involve softer skills such as improved collaboration, decision
making, innovativeness and the ability to think strategically—common learning
objectives in many leadership development programs. It can be difficult to assign a
hard-dollar value to such skills, or to show a correlation between the learning initiative
and the acquisition of the targeted skills. It is also often a struggle to find the time to
follow-up appropriately after the program has occurred (a step that is important in
determining long-term impact and transfer of learning). Additionally, if the post-
program evaluation process and measures were not carefully planned at the program
development stage, such post-mortem evaluation may lack a budget and adequate
support.

These challenges can be overcome, however, and the acquisition of skills—even soft
skills—can be assessed.

Tips to Measuring ROI

1. Don’t go overboard. ROI need only demonstrate value beyond a reasonable
doubt. Find out what the executive sponsor identifies as success and stick to that.

2. Shift from a quality to a results mindset. When designing L&D programs, it’s all
about delivering a quality experience that encourages learning. When evaluating,
it’s about results. When calculating ROI, focus less on the quality of the
experience and more on the effect of learning.

3. Calculate ROI continuously. Always know how the L&D program is performing so
adjustments can be made. This not only helps improve the program, but can
justify how dollars are being spent at any time. (Continued…)



Tips to Measuring ROI (…continued)

4. Build a step-by-step case for ROI. Analyze organizational needs and develop
strategic learning plans, prioritize them and present them with sound justification—
based on anticipated ROI – about why senior leaders should support them.

5. Gather data beyond the program delivery and don’t forget data that is already
available. Evaluation and feedback should come from as many sources as feasibly
possible—from participants, their supervisors, peers and senior leaders.

6. ROI isn’t just about money. When analyzing results, consider such learning
measurements as quality, effectiveness, job impact and business results.

7. Be conservative in ROI calculations. To compensate for bias, self-reported ROI
should be factored down and follow-up evaluations should be weighed more than
evaluations reported immediately after the program.

8. Represent the money outlay as a per participant ratio. Personalize it. Show the per
participant cost (versus a total cost) to make the investment more palatable (e.g. for
this target population we are looking at a $7,000 investment in learning for an
employee responsible for, on average, $1million worth of business).

9. Communicate the story behind the numbers. This is where using anecdotal
information can be helpful in confirming the numbers. It never hurts to highlight
data with meaningful examples.

10. If the ROI numbers are low, don’t be discouraged. ROI is intended to assess what is
working and what should be shelved or revamped.
Source: Garvey, 2012.

Source: Garvey, 2012.
The Critical Conversation: Connecting
ROI to Expectations
I n an article published by Denmark-based management firm Mannaz, Scott Saslow
from the Institute of Executive Development emphasized that successful evaluation
starts well before the learning event occurs, at the pre-program stage. In fact, it should
begin at the needs assessment stage, as the organization explores existing and
anticipated skills gaps and identifies ways to close those gaps.



It is at this point that L&D managers should have a critical conversation with senior
leaders to answer the question, “What will constitute success for this learning
initiative?”

Mutual agreement on the evaluation criteria at this stage is critical, and L&D
professionals should come prepared to lead the discussion. Some questions to keep in
mind when speaking with executive sponsors:

What correlations will be considered valid measures of ROI? For example, one
could correlate a leadership development program that targets high
performance and strategic change with an increase in employee motivation
and engagement survey scores and/or retention rates.

When the learning program can’t take all of the credit, what attribution will it
receive? If it is agreed that the program will have a positive effect on a
particular organizational goal, but it cannot be attributed with 100 percent of
the achievement of that goal, delve further with senior leaders to agree on a
percentage of credit. For example, the learning program could receive partial
credit (i.e. 5 percent) for any operational cost savings found in the supply chain
process.

Baseline assessments may be taken during this phase if part of the evaluation will be a
“before and after” assessment. For example, if a goal is to improve collaboration
across departments, a baseline assessment among participants from each department
can occur before the program and again at appropriate intervals afterward (three
months, six months, and one year) to determine the level of improvement.

Palmer (2010) recommends five steps for HR and talent management professionals to
take when designing successful learning and development programs:

1. Know the organization’s strategic priorities.

2. Understand how the L&D function can contribute to those priorities.

3. Identify L&D programs that will support the organization’s strategic direction.

4. Build it with metrics.

5. Pitch it like you’re the CFO.

When designing a program from the ground up, these steps ensure that a focus on
ROI is directly connected to the organization’s strategic priorities.



Company Spotlight: Texas Health Resources

Senior leaders at Texas Health Resources
(THR) realized that they needed to cost-
effectively develop all of their employees
to deliver a comprehensive, integrated and
coordinated level of care with a “culture of excellence.” The organization developed
partnerships with local colleges to improve learning programs. THR evaluated the ROI
of this program by looking at the number of registered nurses (RNs) who passed their
licensure exams, decreased vacancy rates for RN positions, and faster time to
productivity. THR found that RNs participating in the program had an almost 100
percent pass rate on their licensure exams. Vacancy rates for RN positions fell from 11
percent to 2 percent, employee familiarity with equipment, facilities and hospital
policies increased, and employee orientation time fell from three to two weeks.

Source: Chief Learning Officer staff, 2010.

It’s Never Too Late to Have that Critical
Conversation and to Assess a Program’s
ROI
B ut what should L&D managers do when asked to provide more measurable ROI
data for established learning programs?

It is important to remember that it is never too late to have that critical conversation
about what constitutes a successful learning experience. HR and talent management
professionals should initiate this conversation, keeping in mind that even if senior
leaders express satisfaction with anecdotes and smiley sheets now, long-term
satisfaction and expectations can change with little notice. To stress the point again,
HR and talent management professionals should work with senior leaders to mutually
identify ROI measures (beyond smiley sheets) that are linked to the organization’s key
strategic objectives.



ROI measures should be related to performance after the L&D experience and,
according to some, tied to a dollar figure. For example, time saved or increased output
(or both) as a result of improved performance following participation in a development
program can then be compared to a dollar figure (Lang, 2012).

Kirkpatrick and Kirkpatrick, however, argue that tying ROI to a dollar figure may not
always be the best metric and suggest that calculating the return on expectations
(ROE) may be a better one. When considering ROE, HR and talent management
professionals should begin with an understanding of the desired learning outcomes,
and this starts by having critical conversations with all key stakeholders--senior
leaders, supervisors, peers and subordinates--all of whom can play a role in the
evaluation process.

Regardless of when that critical conversation occurs, asking the right questions to
gauge what stakeholders expect is vital. Some L&D professionals may make
inaccurate assumptions about key stakeholder expectations (Carder, 2012). This is
because they fail to ask the right questions about what the measurement outcomes
should reflect, and as a result, design measurements that are off the mark, too
complex or too impractical to execute. In other words, what do senior leaders want?
What do they consider “successful” when it comes to learning outcomes?

For some organizational cultures, executives may see feedback and anecdotes from
participants as sufficient to assess the value of an in-house leadership development
program. Again, it is important to remember that while this may be enough to assess
the learning in the short term, L&D managers should ask themselves if it will be
enough in the next three to five years and identify more robust measures that even a
chief financial officer would find hard to dispute.

Ways to Evaluate L&D Initiatives
T he metrics that could be used to evaluate L&D initiatives are extensive. Here is a
laundry list of measures that organizations often use to evaluate their learning
programs:

 Average change in performance appraisal ratings over time

 Customer satisfaction ratings

 Employee engagement survey scores



 Turnover rates

 Percentage of promotions

 Productivity rates over time

 Retention rates

These metrics can be used effectively for standard training and development
programs. The challenge, however, is to capture the less tangible, more higher-order
benefits of learning, such as revenue generation or cost-savings realized through
application exercises built into the program. Because leadership programs often focus
on the development of skills such as adaptability, collaboration, communication,
decision making, innovativeness and leading change, the intangible benefits are more
challenging to measure using conventional metrics (Kalman, 2012).

Linda O’Connell, principal of Learnologies, LLC, recommends blending ROI metrics
with participant and stakeholder feedback and anecdotes to better convey the total
value of leadership development programs. Jack Philips, chair of the ROI Institute
agrees. He recommends integrating anecdotal evidence at the same time traditional
data is collected. It can then be used to augment the data when it is reported to senior
leaders (Kalman, 2012).

What Fortune 500 Companies Are Using to Measure L&D Effectiveness

Employee replacement costs*

Turnover costs

Economic value of employee behaviors

Economic benefits of increased job satisfaction, organizational commitment
or similar job attributes

*According to the Society for Human Resource Management, the average
replacement cost of an employee is between 100 and 125 percent of the
employee’s annual salary.

Source: Green & Brainard, 2005.



Steps to Consider When Evaluating the
ROI of Development Programs
T he following steps can serve as
guidelines for L&D professionals Company Spotlight: Owens Corning
who want to create development
programs that effectively demonstrate Senior leaders
ROI. at Owens
Corning credit
Step 1: Have that Critical the company’s
Conversation culture of
innovation as
This cannot be emphasized enough.
a key reason
The first step in any evaluation
why the company remains an industry
process—regardless of when that
leader in the glass fiber industry.
process begins—is to initiate a
discussion with senior leaders to When the company approached UNC
identify what success for each learning
Executive Development, they wanted to
program means to them.
expand on that culture and foster
Step 2: Make Smiley Sheets innovation that lead to real business
results. Therefore, it was important that
More Robust the program be developed with measures
Smiley sheets can be biased due to the that could be reviewed at its conclusion.
immediacy effect. To minimize this
Owens Corning worked with the university
bias, re-assess participants three to six
months after the program and combine
to design and deliver a two-week
the data with concrete examples from “Innovation Boot Camp” for some of their
participants that outline how they have global business leaders with an objective
applied what they learned. that at the end of the experience,
participants would be ready to return to
If possible, request the same feedback
their offices with an innovative idea that
from the participants’ supervisor, peers
could be commercialized and developed
and subordinates. To build
into a new product, service or solution.
collaboration and to set expectations,
identify this goal during the planning
The first week of the program focused on
phase and conduct a baseline teaching teams business foundations.
assessment before the program. (Continued…)

All Content © UNC Executive Development 2012 10 | P a g e


Company Spotlight: Owens Corning (…continued)

The second week focused on the development of the innovative ideas identified by
participant teams. At the end of the program, teams presented their ideas to peers and
received a “go or no go” recommendation to determine whether the idea should be
presented to Owens Corning’s senior executive team. Owens Corning measured
program success by tracking the number of innovative ideas senior executives identified
as most viable and approved to develop further. The company has significantly
increased market share and revenue in targeted growth areas, offering two very
concrete measurements of the program's success.

Step 3: Include Real Business Challenges in the Program
Leadership programs often focus on helping high-potential employees think and act
more strategically. To measure such a program’s effectiveness, consider building
actual strategic business challenges into the design so participants can apply what
they are learning. For example, if the strategic challenge is global expansion,
challenge individuals or teams to craft a market-entry plan and have them present it to
senior leaders. Such a deliverable, if implemented, can represent significant ROI in the
eyes of executives.

Duke Energy has used this approach to good effect. Its two-week Strategic Leadership
Program, focuses on developing mid-level managers’ leadership skills, teaches
participants how to evaluate business decisions and how to execute business
strategies. The program culminates with participants working through an actual
strategic challenge identified by senior executives (Palmer, 2010).

Step 4: Integrate Learning Programs into the Organization’s
Performance Management System and Hold All Stakeholders
Accountable
To increase the probability that the targeted outcomes of a program remain a focus
area for participants, ensure performance appraisal goals reflect those targeted
outcomes for participants and their supervisors. This step may require gathering
information about an employee’s productivity before and after the program. In cases



where it is too late to gather pre-program data, consider using a control group of
equivalent employees, business units or organizations to compare against.

In cases where hard data is
difficult to identify, think Company Spotlight: J.C. Penny
creatively. At IBM Europe, for
example, a key aspect of a In 2008, J.C. Penney
leadership development program found itself among the
for high-potential women is a bottom of all retailers
robust mentorship program. To in customer satisfaction.
ensure the mentorship program is The organization
working as intended, the needed to engage and
company expects participants to educate its 155,000 associates in 1,100 stores,
be promoted within a year of the logistics centers and headquarters to raise its
start of the mentorship. Failure to
customer satisfaction rating from 43 percent.
obtain a promotion is seen as the
J.C. Penney therefore had a straightforward
sponsor’s failure, not the
measure for their program's success: improved
candidate’s (Cater & Silva, in
customer satisfaction ratings. The retailer
Kelly, 2012). While this may be
too radical a metric for some
educated its associates through a multi-faceted
organizations, the lesson here is approach that had distinct messages for each
not to be afraid to think differently targeted population (store leaders, store
when creating evaluation criteria. associates and home office associates). A key to
the initiative’s success was to obtain support
Also, do not overlook data that is
from all store leaders. Store managers then
often readily available when
received the learning so they could “champion”
planning evaluation. Retention
the message. All store associates then
rates of program participants
versus non-participants and
participated in learning sessions—delivered by
promotion and engagement store managers and the training supervisor
scores which translate to team--within three months of the store
increased employee participation manager’s learning program.
and productivity are areas where
Customer satisfaction ratings increased from 48
employers realize real value.
These should be tracked and
to 63 percent by the end of 2010, ranking J.C.
measured, not only for Penney as top in customer service.
participants, but for their
Source: Chief Learning Officer staff, 2010,
subordinates and supervisors too. Business Impact Division.



Step 5: Assign Participants Actual Projects after the Learning
Experience
This step can be particularly helpful when assessing the effectiveness of leadership
development programs. For example, if the goal of a leadership development
experience is to improve project management skills, assign participants an actual
project to manage after the program concludes, and establish check-point
measurements as the project progresses. Check-point measurements could include
the participant’s ability to assemble an effective strategy, the effectiveness of his or
her communication skills, the ability to acquire the necessary resources and the ability
to meet budgets and timelines.

Conclusion
E mployers today expect all business units to meet higher standards of
accountability, and the HR and talent management function is no exception. L&D
professionals cannot (and should not) accept that smiley sheets will be enough to
demonstrate the ROI of learning programs. Instead, they need to engage senior
leaders early to decide which measures should be used to determine the ROI for these
programs. This conversation is critical because L&D professionals need to
demonstrate the value of their learning programs based on metrics previously agreed
to with the ultimate decision makers to justify the investment in learning and
development.



About UNC Executive Development
Our approach to program design and delivery draws upon the power of real-world,
applicable experiences from our faculty and staff, integrated with the knowledge our
client partners share about the challenges they face.

We call this approach The Power of Experience. We combine traditional with
experiential and unique learning to ensure that all individuals gain relevant new skills
that they can easily implement within their own organizations. Through action learning
and business simulation activities, we challenge participants to think, reflect and make
decisions differently.

Our Approach: The Partnership

Our team customizes each leadership program through a highly collaborative process
that involves our clients, program directors, faculty and program managers. We are
dedicated to following-up with our clients and individual participants to ensure that
their learning experiences have been meaningful and impactful. This integrated
approach consistently drives strong outcomes.

Our Approach: The Results

Our executive education programs are designed with results in mind, and we are
focused on successfully meeting our clients' business and academic expectations.
Below are a few examples of the results our client partners have achieved:

Leadership refocused with new Products redefined
strategy and cohesive vision New markets targeted
Strategic plans created for the Cost-saving measures developed
global marketplace Silos leveled
Supply chains streamlined Teams aligned

Participants leave empowered to bring in new ideas, present different ways to grow
business and tackle challenges. The result is stronger individuals leading stronger
teams and organizations.

Contact Us

Website: www.execdev.unc.edu | Phone: 1.800.862.3932 | Email: unc_exec@unc.edu



Sources
Bersin & Associates staff (2012 January). The Corporate Learning Factbook 2012, Executive
Summary. Oakland, CA: Bersin & Associates.

Businessballs.com staff (n.d.). Kirkpatrick’s learning and training evaluation theory.
Businessballs.com. Retrieved from http://www.businessballs.com/kirkparicklearninge
valuationmodel.htm.

Carder, D. (2012 Spring). Six critical measurement mistakes and how to avoid them.
Training Industry Quarterly. Retrieved from http://www.trainingindustry.com/TIQ.

Chief Learning Officer staff (2010). 2010 Business Impact Division. Chief Learning Officer.
Retrieved from http://clomedia.com/articles/view/business-impact-division-1/print:1.

Chief Learning Officer staff (2010). 2010 Vanguard Reward. Chief Learning Officer.
Retrieved from http://clomedia.com/articles/view/3946.

Chief Learning Officer staff (2011). 2011 Vanguard Reward. Chief Learning Officer.
Retrieved from http://clomedia.com/articles/view/2011-vanguard-division-1/.

Garvey, A. (2012 September 24). Ten tips to measure training ROI. Trainingzone. Retrieved
from http://www.trainingzone.co.uk/topic/leadership/10-tips-measure-training-roi/179893.

Green, T. & Brainard, M. (2005 April 28). Calculating the return on investment (ROI) for
organizational development (OD) and human resource (HR) initiatives. Brainard Strategy.
Retrieved from http://www.apex-ps.com/articles/ODHHROI.pdf.

Kalman, F. (2012 January 27). Express learning’s ROI through storytelling. Chief Learning
Officer. Retrieved from http://clomedia.com/articles/view/express-learning-s-roi-through-
storytelling/print:1.

Kelly, K. (2012). The new business imperative: Recruiting, developing and retaining women
in the workplace. UNC Kenan-Flagler Business School. Retrieved from http://www.kenan-
flagler.unc.edu/executive-development/custom-programs/~/media/
3A15E5EC035F420690175C21F9048623.pdf.

Lang, A. (2012). Training program evaluation. Integral Defense Solutions. Retrieved from
http://www.integraldefensesolutions.com/wp-content/uploads/2012/06/Training-Program-
Evaluation.pdf.

McCann, D. (2011 June 1). Measured response. CFO.com. Retrieved from
http://www.cfo.com/printable/article.cfm/14577155.



Nielsen, L. (2012). Dispelling the touchy-feely stigma. OD Practitioner, 44, 1. Retrieved from
http://c.ymcdn.com/sites/www.plexusinstitute.org/research/resmgr/files/odp-
quantifying.pdf/.

Palmer, S. (2010). Making the business case for learning and development: 5 steps for
success. UNC Kenan-Flagler Business School. Retrieved from http://www.kenan-
flagler.unc.edu/executive-development/custom-programs/~/media/
5D2A2B1BFB794F8AAA023BD33D68844F.ashx.

Personnel Today staff (2012 April). Measuring the impact of learning: Quick-fire links.
Personnel Today. Retrieved from http://www.personneltoday.com/Articles/23/04/2012/
58483/measuring-the-impact-of-learning-quick-fire-links.htm.

Philips, J. (2007). Calculating the impact of ROI on HR programs. ROI Institute. Retrieved
from http://www.competentiebeleid.be/assets/waarom/Measuring_ROI.pdf.

Philips, J. (2008). Measuring ROI in the public sector. ROI Institute. Retrieved from
http://www.calhr.ca.gov/Documents/measuring-roi-in-the-public-sector-course-
materials.pdf.

Poulin, M. (2012 Spring). Assessing learning and performance. Training Industry Quarterly.
Retrieved from http://www.trainingindustry.com/TIQ.

Saslow, S. (n.d.). Measuring the impact and ROI in executive development programs.
Mannaz. Retrieved from http://www.mannaz.com/articles/impact-and-roi-in-executive-
development.

Yeo, K. (2009 July 22). Measuring organizational learning: Going beyond measuring
individual training programs. Human Resources iQ. Retrieved from
http://www.humanresourcesiq.com/metrics/articles/measuring-organizational-learning-
going-beyond-mea/.


Beyond Smiley Sheets: Measuring the ROI of Learning and Development

More Related Content

What's hot (20)

Similar to Beyond Smiley Sheets: Measuring the ROI of Learning and Development (20)

More from Kip Michael Kelly (20)

Recently uploaded (20)

Beyond Smiley Sheets: Measuring the ROI of Learning and Development