Evaluating Results and Benefits Week #6 Lecture 1The Impact of

Evaluating Results and Benefits
Week #6 Lecture 1
The Impact of Evaluation
This week we will continue our discussion pertaining to the
importance of evaluation and the role it plays within the
workplace. With the growing amount of competition in both the
workplace and industry, there is a growing need to improve
competitive standing to ensure growth. Because of this, it is
critical that employees consistently work to improve their
performance. Though, just because this is a stated need does not
mean that everyone will be on board. When management
conducts performance evaluation in an effective manner, they
can optimize productivity in the workplace to align with the
organization's strategy. This often will result in an effective
evaluation that may work for the benefit of both the employee
and the organization.
A successful manager knows and understands that a
performance evaluation is effective when improvement among
employees is seen after the evaluation has been conducted. A
manager should clearly communicate the expectations of the
employees throughout the evaluation process to ensure that
everyone is aware of the goals and expectations for the
workplace. Performance expectations should be specific so that
there is little room for doubt on the part of the employee. Both
strengths and weaknesses should be discussed during the
evaluation to ensure that the employee understands where they
need to improve and where they are making a significant
difference for the organization. Including positive
reinforcement and choosing positive strengths to provide to the
employee will be beneficial to foster a strong workplace morale.
In addition, a manager should be specific as to what
improvement expectations might be to help motivate the

employee in making the necessary changes.
Let’s discuss job satisfaction for a moment because I am
positive that the one goal you expect when taking a job is that
you will be happy there. This is a good attitude to start with,
and if all goes according to plan, this should be your attitude
moving forward. There is nothing better in the workplace than a
satisfied employee. Now, how does job satisfaction occur within
the workplace? For starters, strong retention and job loyalty is
important. With the growing of our economy and businesses
competing all over, retaining employees can be a challenge for
any workplace. There is competition and who offers the most is
often who wins the most.
However, that does not mean that the company will retain those
employees they recruit. The goal is to create realistic
expectations from start to finish. When creating a workplace
environment, it is important to consider what will keep your
employees comfortable. Some workplaces have implemented a
gym, others have revised their cafeteria options to provide
delicious and healthy food options. Some workplaces even go as
far as adding game rooms and exciting campuses (i.e. Google
and Microsoft). Though, once you have all of the benefits
established, now it is time to keep the employees moving.
Ultimately, despite all of the perks of a job, an employee who
believes they can grow within the organization is likely to show
loyalty to the company. Therefore, the use of evaluations will
prove useful in helping an employee to understand what they
need to do in order to scale the corporate ladder.
This brings us to our next discussion point - how do you move

up the ladder? When conducting a performance evaluation, it is
important that a trained manager conducts the evaluation. The
importance of having a skilled manager conduct these
interviews is the idea that they know how to work with various
personalities and can help to establish goals through the use of
the evaluation. Understanding what it takes to move forward
and setting realistic goals in order to do this will create a
motivated employee in both the short and long-term setting.
Remember that the key word is “realistic.” We all want to soar
to the top a week into our new career, however, the reality is
that it will take work and dedication. Moving up also takes
patience and support. Understanding what is expected and how
the goals can be met will create a positive work environment.
Finally, be sure to consider training within the workplace. I
know some of you silently (or maybe not so silently) just sighed
in disgust over the thought of training. I know that I have found
myself in training sessions that felt like a waste of time.
However, I have also been in training sessions that have
motivated me towards additional success. As a manager, the
goal is to develop a training program that is going to be
effective for your workplace. Just because a training program
worked well for one company does not mean that it will work
for yours. Research what is needed in terms of training within
your workplace, consider the types of employees you have, and
determine the best methods for training. Once the training has
been conducted, follow up! This is an important step that is
often forgotten. Take the time to discuss the training with your
employees and follow up with them to ensure that what was
learned is properly being applied to the workplace. Keep
following up over time, make changes, and continue to teach.
By doing this, you will ensure that the learning was not
forgotten and is truly being put to use to improve the workplace
overall.
Evaluation is critical to the success of both the employee and

the organization. This is a tool that needs to be taken seriously
and used often. Take the time to establish the evaluation tools
that are best suited for your workplace. You will be glad you
did.
Evaluating Results and Benefits
Week #6 Lecture 1
The Impact of Evaluation
This week we will continue our discussion pertaining to the
importance of evaluation and the role it plays within the
workplace. With the growing amount of competition in both the
workplace and industry, there is a growing need to improve
competitive standing to ensure growth. Because of this, it is
critical that employees consistently work to improve their
performance. Though, just because this is a stated need does not
mean that everyone will be on board. When management
conducts performance evaluation in an effective manner, they
can optimize productivity in the workplace to align with the
organization's strategy. This often will result in an effecti ve
evaluation that may work for the benefit of both the employee
and the organization.
A successful manager knows and understands that a
performance evaluation is effective when improvement among
employees is seen after the evaluation has been conducted. A
manager should clearly communicate the expectations of the
employees throughout the evaluation process to ensure that
everyone is aware of the goals and expectations for the
workplace. Performance expectations should be specific so that
there is little room for doubt on the part of the employee. Both
strengths and weaknesses should be discussed during the
evaluation to ensure that the employee understands where they
need to improve and where they are making a significant

difference for the organization. Including positive
reinforcement and choosing positive strengths to provide to the
employee will be beneficial to foster a strong workplace morale.
In addition, a manager should be specific as to what
improvement expectations might be to help motivate the
employee in making the necessary changes.
Let’s discuss job satisfaction for a moment because I am
positive that the one goal you expect when taking a job is that
you will be happy there. This is a good attitude to start with,
and if all goes according to plan, this should be your attitude
moving forward. There is nothing better in the workplace than a
satisfied employee. Now, how does job satisfaction occur within
the workplace? For starters, strong retention and job loyalty is
important. With the growing of our economy and businesses
competing all over, retaining employees can be a challenge for
any workplace. There is competition and who offers the most is
often who wins the most.
However, that does not mean that the company will retain those
employees they recruit. The goal is to create realistic
expectations from start to finish. When creating a workplace
environment, it is important to consider what will keep your
employees comfortable. Some workplaces have implemented a
gym, others have revised their cafeteria options to provide
delicious and healthy food options. Some workplaces even go as
far as adding game rooms and exciting campuses (i.e. Google
and Microsoft). Though, once you have all of the benefits
established, now it is time to keep the employees moving.
Ultimately, despite all of the perks of a job, an employee who
believes they can grow within the organization is likely to show
loyalty to the company. Therefore, the use of evaluations will
prove useful in helping an employee to understand what they
need to do in order to scale the corporate ladder.

This brings us to our next discussion point - how do you move
up the ladder? When conducting a performance evaluation, it is
important that a trained manager conducts the evaluation. The
importance of having a skilled manager conduct these
interviews is the idea that they know how to work with various
personalities and can help to establish goals through the use of
the evaluation. Understanding what it takes to move forward
and setting realistic goals in order to do this will create a
motivated employee in both the short and long-term setting.
Remember that the key word is “realistic.” We all want to soar
to the top a week into our new career, however, the reality is
that it will take work and dedication. Moving up also takes
patience and support. Understanding what is expected and how
the goals can be met will create a positive work environment.
Finally, be sure to consider training within the workplace. I
know some of you silently (or maybe not so silently) just sighed
in disgust over the thought of training. I know that I have found
myself in training sessions that felt like a waste of time.
However, I have also been in training sessions that have
motivated me towards additional success. As a manager, the
goal is to develop a training program that is going to be
effective for your workplace. Just because a training program
worked well for one company does not mean that it will work
for yours. Research what is needed in terms of training within
your workplace, consider the types of employees you have, and
determine the best methods for training. Once the training has
been conducted, follow up! This is an important step that is
often forgotten. Take the time to discuss the training with your
employees and follow up with them to ensure that what was
learned is properly being applied to the workplace. Keep
following up over time, make changes, and continue to teach.

By doing this, you will ensure that the learning was not
forgotten and is truly being put to use to improve the workplace
overall.
Evaluation is critical to the success of both the employee and
the organization. This is a tool that needs to be taken seriously
and used often. Take the time to establish the evaluation tools
that are best suited for your workplace. You will be glad you
did.
CHAPTER SIXTEEN
Understanding Context: Evaluation and Measurement in Not-
for-Profit Sectors
Dale C. Brandenburg
Many individuals associated with community agencies, health
care, public workforce development, and similar not-for-profit
organizations view program evaluation akin to a visit to the
dentist’s office. It’s painful, but at some point it cannot be
avoided. A major reason for this perspective is that evaluation
is seen as taking money away from program activities that
perform good for others, that is, intruding on valuable resources
that are intended for delivering the “real” services of the
organization (Kopczynski & Pritchard, 2004). A major reason
for this logic is that since there are limited funds available to
serve the public good, why must a portion of program delivery
be allocated to something other than serving people in need?
This is not an unreasonable point and one that program
managers in not-for-profits face on a continuing basis.
The focus of evaluation in not-for-profit organization has
shifted in recent years from administrative data to outcome
measurement, impact evaluation, and sustainability (Aspen

Institute, 2000), thus a shift from short-term to long-term
effects of interventions. Evaluators in the not-for-profit sector
view their world as the combination of technical knowledge,
communication skills, and political savvy that can make or
break the utility and value of the program under consideration.
Evaluation in not-for-profit settings tends to value the
importance of teamwork, collaboration, and generally working
together. This chapter is meant to provide a glimpse at a minor
portion of the evaluation efforts that take place in the not-for-
profit sector. It excludes, for example, the efforts in public
education, but does provide some context for workforce
development efforts.
CONTRAST OF CONTEXTS
Evaluation in not-for-profit settings tends to have different
criteria for the judgment of its worth than is typically found in
corporate and similar settings. Such criteria are likely to include
the following:
How useful is the evaluation?
Is the evaluation feasible and practical?
Does the evaluation hold high ethical principles?
Does the evaluation measure the right things, and is it accurate?
Using criteria such as the above seems a far cry from concepts
of return on investment that are of vital importance in the profit
sector. Even the cause of transfer of training can sometimes be
of secondary importance to assuring that the program is
described accurately. Another difference is the pressure of time.
Programs offered by not-for-profit organizations, such as an
alcohol recovery program, take a long time to see the effects
and, by the time results are viewable, the organization has
moved on to the next program. Instead we often see that
evaluation is relegated to measuring the countable, the numbers
of people who have completed the program, rather than the life-
changing impact that decreased alcohol abuse has on
participants. While the latter is certainly important, the typical

community-based organization (CBO) is limited in its resources
to perform the long-term follow-through needed to answer the
ultimate utility question. Thus, the choice of what is measured
tends to be the result of negotiation with stakeholders. The
broad goals of evaluation tend to be grouped among the
following:
Understanding and sharing what works with other organizations
and communities;
Building sustainability of programs and ensuring funding;
Strengthening the accountability of the programs with various
public constituencies;
Influencing the decisions of relevant policy makers and program
funders;
Building community capacity so that future engagements have
greater community participation; and
Understanding where the program is going so that it results in
reflecting on progress that can improve future programs.
These goals reflect some of the typical objectives for applying
evaluation in not-for-profit settings. The goals embody specific
activities that can be designed to collect evidence on a
program’s effectiveness, be accountable to stakeholders,
identify projects for improvement, clarify program plans, and
improve communication among all groups of stakeholders. The
types of programs or actions that are designed to improve
outcomes for particular individuals, groups, or communities
considered in this chapter include the following:
Direct service interventions: improve the nutrition of pre-school
children;
Research endeavors: determine whether race disparities in
emergency room care can be reduced;
Advocacy efforts: campaign to influence legislation on proper
use of infant car seats; and
Workforce training programs: job training program to reduce
unemployment among economically disadvantaged urban

residents.
The results of evaluation in not-for-profit settings are typically
designed to provide information for future decision making.
Most efforts can be grouped into three categories: process
evaluation, short-term evaluation, or long-term (outcome)
evaluation. In fact, it is rare to see a program evaluation effort
that does not include some process evaluation. Process
evaluation is considered important because it typically yields an
external view of how the program was conducted; in other
words, it provides a detailed description of the objectives,
activities, resources used, management of the program, and
involvement of stakeholders. It would be difficult to judge any
outcomes of a program without understanding the components
of the program in detail. That is why process evaluation is
important. Short-term evaluation deals with the accomplishment
of program objectives or the intermediate links between the
program activities and the long-term outcomes. Outcome
evaluation is associated with the long-term effects such as
health status or system changes that are often beyond the range
of a typical evaluation effort.
STAKEHOLDER ANALYSIS
One of the key tools used to structure the evaluation process in
not-for-profit settings is the stakeholder analysis. It is a key
initial stage to develop the primary issues and evaluation
questions from which other stages of the evaluation process can
be built. The stakeholder analysis is designed to identify the
needs and values of the separate stakeholders and combine the
results so that an adequate plan can be developed. It is rare to
find an evaluation effort in not-for-profit situations for which
there are at least three stakeholder groups with a major interest
in the effort. The stakeholder analysis is a means to organize
the political process and create an evaluation plan, as well as
satisfy the desires of the different perspectives and needs for
information. A first step in the process is to identify all groups
or individuals with a stake in the process, followed by a second

dividing the groups into primary and secondary members.
Primary members are those stakeholders who are likely to be
direct users of the evaluation results; whereas secondary
stakeholders may have an interest in the findings, but the results
of the evaluation are not likely to impact them directly.
For example, in the evaluation of a workforce development
project for urban adults, primary stakeholders would include the
sponsoring agency or funder, the manager of program delivery,
program developers, instructors, a third-party program
organizer, and partner organizations that might include
community-based organizations, a local business association,
and a representative from the city government. Secondary
stakeholders might include parents or relatives of program
participants, local welfare officials, advocacy groups, and the
participants themselves. A list of stakeholders can be
determined by answering the following questions:
Who pays for program staff time?
Who selects the participants?
Who champions the desired change?
Who is responsible for after-program behavior/performance?
Who determines success?
Who provides advice and counsel on program conditions?
Who provides critical guidance?
Who provides critical facts?
Who provides the necessary assistance?
Note that these questions do not identify any of the purposes of
the evaluation, only a means to distinguish who should be
involved in evaluation planning. The amount of stakeholder
involvement in the evaluation can be limited or it can be
substantial enough so that they assist in the design of the
overall process, including the development of data collection
protocols. The involvement of stakeholders should also lead to
increasing the value and usefulness of the evaluation (Greene,
1988) as well increasing the use of the findings. Such a process

increases the ownership in the report and possibly leads to a
common vision of collective goals (Kopczynski & Pritchard,
2004).
The second step of performing a stakeholder analysis involves
the identification of the requirements, or primary evaluation
purposes, and aligning those against the list of primary
stakeholders. The matrix shown in Table 16.1 is an example of
this analysis for a publicly funded workforce development
program for inner city adults.
The “requirements” or purposes of the evaluation can be
developed in a number of ways, but the major sources of the list
usually come from the published description of the program,
supplemented by interviews with the stakeholders. It would be
ideal if the matrix could be completed in a group setting, but it
is more likely that the evaluator develops the chart
independently and then shares the results in a program advisory
meeting. The “X” in a given box represents where a requirement
matches the possible use of that information by the selected
stakeholder. One can note, for example, that program Strengths
and Weaknesses are needed for staff internal to the development
and execution of the program but not of particular interest to
those outside that environment. Another outcome of this process
is at least a general expectation of reporting relationships that
would occur during the communication of the interim and final
results of the evaluation.
While it might seem that the stakeholder analysis might focus
on the program itself, there are other outputs that could be
entered into the analysis. Results of evaluation data are often
used to tell compelling stories that can increase the visibility
and marketing of the organization and increase accountability
with board members and community, as well as attract
additional sources of revenue. These findings were supported by
a national survey of local United Way organizations as reported

by Kopczynski & Pritchard (2004).
EVALUATION PURPOSES
While the stakeholder analysis establishes the initial phase of
understanding evaluation purposes, further definition is
supplied by a list of evaluation issues. These issues are
typically generated during the stakeholder analysis after further
understanding of the program description. Evaluation issues can
range from general questions about program impact to detailed
questions on the selection of service providers. A sample
(Gorman, 2001) set of questions or primary issues from a low -
income workforce development project that leverages resources
from a local housing authority are listed below:
Describe the major partners leading, managing, and staffing
each major activity area.
Describe the major program areas and the target population
service goals by area. What are the major strategies to be
employed to reach these service goals?
To what extent does the program assist in decreasing or
alleviating educational barriers to sustained employment? What
were the major instructional and training strategies employed to
attain the stated goals and objectives?
alleviating personal and social barriers to employment: poor
work histories, cultural barriers, substance abuse and
developmental disabilities, and limitations of transportation and
adequate childcare?
What were the participant impacts: wages, self-efficacy, and
employability?
Was there differential effectiveness of the program relative to
categories of the target population: public housing residents,
non-custodial parents, learning-disabled individuals, persons
with limited English proficiency, and other economically
disadvantaged groups?
Was there differential effectiveness relative to the categories of

high-demand sectors targeted by the program:
building/construction trades, health care, and hospitality?
What evidence indicates that the program management and its
partners can actively disseminate/replicate this program in other
regions via its current programs?
What are some key success stories that serve to illustrate
overall program value to participants and community?
How cost-effective was the project in general terms? By
program area? How did costs or efficiency of service change
over the course of the project?
What were the major lessons learned in the project? How do
these relate to self-sufficiency for the target population?
Community economic development? Leadership effectiveness?
These results were obtained from a stakeholder analysis and are
not yet grouped into evaluation components. Combining these
results with the stakeholder analysis assists in defining the
overall evaluation plan. It provides the framework for
developing the data requirements as well as the type of
measurement needed for instrumentation.
LOGIC MODELS
The final phase for planning the evaluation is the development
of a logic model, a description or map of how the major
components of an evaluation are aligned; that is, the connection
between how the program is designed and its intended results.
Logic models can be used in any phase of the program
development cycle (McLaughlin & Jordan, 2004) from initial
design to examining long-term impact. It is a means to make the
underlying theory behind the intervention more explicit and to
discover its underlying assumptions. Even the development of a
model, that is, mapping out all of its components, can be
instructive for program staff and other stakeholders. A logic
model is particularly useful for evaluators both as an advance
organizer as well as a planning tool for assessment development
(McLaughlin & Jordan, 2004). In many situations, such as
community or public health, the specification of a logic model

is a proposal requirement. Whether or not an evaluator builds on
an existing logic model or develops a preliminary version for
the evaluation plan, the map created is a visualization of how
the human and financial investments are intended to satisfy
program goals and lead to program improvements. Logic models
contain the theoretical and practical program concepts in a
sequence from input of resources to the ultimate impact.
Most logic models have a standard nomenclature (see the
Kellogg Foundation guidelines [W.K. Kellogg Foundation,
2007] and McLaughlin & Jordan) that contain the following
elements:
Resources: program inputs like needs assessment data and
capabilities (financial, human, organizational partnerships, and
community relationships) that can be allocated to the project.
Activities: the tasks or actions the program implements with its
resources to include events, uses of tools, processes, or
technology to perform actions to bring about intended changes
or results.
Outputs: the direct products of program activities or services
delivered by the program, even reports of findings that may be
useful to other researchers.
Outcomes: both short-term (one to three years) and longer-term
(four to six years) specific changes in the targeted individuals
or organizations associated with behavior, functioning,
knowledge, skills, or status within the community. Short-term
outcomes are those that are assumed to be “caused” by the
outputs; long-term outcomes are benefits derived from
intermediate outcomes.
Impact: the ultimate consequences or results of change that are
both intended and unintended for the individuals, organizations,
or communities that are part of the system, generally occurring
after program conclusion.
Whatever the process, such as a focus group, used to create a
logic model, another useful outcome is a listing of key

contextual factors not under the control of the program that
might have both positive and negative influences on the
program. These context factors can be divided into two
components—antecedent conditions and mediating variables
(McLaughlin & Jordan, 2004). Geography, economic conditions,
and characteristics of the target group to be served are example s
of the former; whereas staff turnover, new government
incentives, and layoffs for a local major employer are examples
of the latter.
The example shown in Figure 16.1 is a portion of a logic model
from public health and concerns a neighborhood violence
prevention program (Freddolino, 2005). The needs were derived
from data that showed that poverty and violence were
significant threats to the health of residents in Central County.
The general goals of the program were to increase the capacity
of local neighborhood groups to plan and implement local
prevention programs and work to establish links between
existing programs and agencies. Many logic models are likely
more complex than the example shown in that specific actions
are linked to given outputs and outcomes in order to
differentiate the various conceptual links in a program.
Logic models can be constructed in various ways to illustrate
various perspectives of a program. Figure 16.1 represents an
activities-based approach that concentrates on the
implementation of the program, and it is most useful for
program management and monitoring. A second approach based
on outcomes seeks to connect the resources to the desired
results and is specifically geared to subdivide the short-term,
long-term, and ultimate impact of the program. Such models are
most useful for designing evaluation and reporting strategies.
The logic model developed for a program should suggest the
type of measurement required in order to prove or improve the
model specified. Since the model is related to performance
objectives, it can also lead to assist in judging the merit or

worth of the outcomes observed.
A third type, the theory approach emphasizes the theoretical
constructs behind the idea for the program. Such models
concentrate on solution strategies and prior empirical evidence
that connect the selected strategies to potential activities and
assumptions. These are most useful for program planning and
design. Regardless of the type of model selected, each is useful
for the evaluator to provide more comprehensive descriptive
information that can lead to effective evaluation design and
planning.
Collaborative projects can also suffer from lack of
comparability in implementation due to inconsistent allocation
or availability of resources across partner organizations. This
lack of comparability in program resources often thwarts
attempts at gathering data to compare results across sites and
leads to misinterpretation of findings or no consistent findings.
An example evaluation plan (Gorman & Brandenburg, 2002)
from a multiple-site consortia project managed by a systems
integrator and funded by the U.S. Department of Labor is
provided in Table 16.3. In this case, the systems integrator,
American Community Partnerships, is an affiliated organization
associated with a large U.S. labor union whose goals, in part,
are to promote high-wage jobs in union roles for economically
disadvantaged urban residents. This particular program operated
in some of the very poorest neighborhoods in the respective
cities involved.
One can note that in complex arrangements represented in
Exhibit 16.1, that the management of internal and external
partner organizations is crucial to obtaining the needed services

for the target population as well as attempting to meet the long-
range goal of program sustainability. Initial sponsor-allocated
funds cover the start-up of the partnership, but the leveraging of
existing funds and other resources are required for the overall
effort to be successful. Other evaluation can be considerably
more complex when funded by two or more federal agencies.
The report provided by Hamilton and others (2001) is such an
example and provides a description of an evaluation funded
over five years by the U.S. Departments of Labor and
Education.
Foundation Guidelines
Numerous resources are available on outcome measurements
that are designed to assist not-for-profit organizations. A
current major source of information can be found through the
website of the Utica (New York) Public Library, associated with
the Foundation Center, a foundation collaborative. The site is
annotated and contains links to major foundations, professional
associations, government sponsors, technology uses, data
sources, statistical information, best practices, benchmarking,
and assessment tools.
Another set of evaluation models can be derived from an
examination of community change efforts. Two organizations
that have funded a considerable number of these actions include
the W. K. Kellogg Foundation and the Annie E. Casey
Foundation. Both organizations offer assistance in designing
evaluation efforts. These foundations, among others, want their
grantees to be successful in their efforts, so they have
developed evaluation guidelines for potential grantees to build
in evaluation efforts at the time of proposal writing. More
extensive guidelines exist for larger-scale efforts. Especially
instructive in this regard is Kellogg’s perspective on evaluating
social change in that they have constructed a framework
especially for external evaluators. Based on the dynamics of
social change and the context of the program, they detail four

types of evaluation designs: exploratory, predictive, self-
organizing, and initiative renewal. Given the complexity of the
type of program under review, evaluators are charged to pay
attention primarily to the major aspects of the system change
and disregard those features that are of minor consequence.
agencies and program staff in understanding the major issues to
be considered. These organizations can be quite effective
because they understand the local and regional context.
DATA COLLECTION AND MEASUREMENT
As can be concluded from some of the previous discussion, data
collection and measurement for evaluation in not-for-profit
settings can range from the deceptively simple to the complex.
Most data collection schemes tend to be customized to the
environment of the program being evaluated. There are a
number of reasons for this. The measurement of a program is
most often linked to stakeholder considerations and the needs of
the sponsoring organization, as opposed to creating an elegant
experimental design. The use of both quantitative and
qualitative data is applied to answer the evaluation questions
posed at the outset of the investigation.
Second, there is a strong bias for producing data that are useful
for all stakeholders, and many users, such as directors of
community-based organizations are not used to interpreting
sophisticated statistical analyses. Data on effectiveness of
programs often can be derived from solid descriptions of
program activities and survey data. This is not to say that
measurement is based on the lowest common denominator of
sophistication, but the end-result is to use the findings in a way
that can improve program activities. The availability of
resources within not-for-profits generally implies that program

staff collects data for monitoring purposes, that is, what is
countable from an administrative perspective (Kopczynski &
Pritchard, 2004). Other not-for-profit organizational
shortcomings may enter into the development of a plan for a
comprehensives data collection scheme, namely “the lack of
appreciation of data, lack of training, poorly developed
information systems, and high turnover rates” (Kopczynski &
Pritchard, 2004, p. 652), besides the issue of limited budgets.
A third reason to customize data collection is that they may be
limited by budget considerations. Elegant and rigorous designs
cost more to implement and can have an indirect effect on
program activities if certain target participants cannot
participate in the primary intervention. This is not to say that
rigorous designs are not applied in not-for-profit settings,
especially in the public health domain. One such effort
(Huffman & others, 2002) funded by the Packard Foundation
presents a wealth of information on outcome measurement,
valid evaluation constructs, and a set of principles to apply in
such settings. Subjects of workforce development programs, for
example, are often difficult to recruit, so allocating a portion of
participants to a control setting is not cost-effective in many
cases. Even more challenging would be follow-up data
collection efforts on a homeless population (Kopczynski &
Pritchard, 2004).
It is probably instructive at this point to introduce an example
of the type and range of data needed to satisfy requirements in a
large-scale workforce development effort. Using the same
program (Gorman & Brandenburg, 2002) from the example in
Table 16.3, Exhibit 16.1 represents a listing of the data
elements selected to provide a comprehensive evaluation.
REPORTING RESULTS

If evaluation reports are to be useful for all stakeholders, it is
important to consider the communication of evaluation results
early in the design of the evaluation process. Such
considerations are needed because the demand for clear and
credible information from not-for-profits is high (Kopczynski &
Pritchard, 2004). In general, it is prudent to negotiate what
results are to be presented to whom at the time the stakeholder
analysis is conducted. Final reports of evaluation are typically
reported in two stages: (1) a formal written report that contains
all details of the design, implementation, data collection, data
analysis, findings, and conclusion with possible
recommendations and (2) a second summary version, such as a
PowerPoint presentation that lists the highlights of the
investigation. Other reports, such as interim findings or data
from a single site in a multiple-site investigation, results of a
set of interviews or a survey, or case descriptions, may also be
added, depending on funding and stakeholder needs.
Regardless of the type of report, there is still a need to organize
reports so that stakeholders may be guided to make appropriate
decisions and plan future actions based on the findings. An
evaluation that contains as many data elements as depicted in
Exhibit 16.1 would need to be organized efficiently if
stakeholders were to be able to comprehend and sift through the
volume of data likely to be generated. Smock & Brandenburg
(1982) suggest a tool to aid in the process of organizing data
and presenting findings as portrayed in Table 16.4.
While the classification shown may be an oversimplification of
available data, it is nonetheless quite useful in presenting data
to unsophisticated users. The concept underlying its structure is
that the entirety of information may be ranked hierarchically
from its continuous mode into three levels. These three levels
represent an artificial trichotomy from very general (overall
success) to success or failure of specific program components to
very specific findings that have meaning to possibly a single

stakeholder. Level I information is the most general, must be
inferred from a variety of findings (rolled up across data
elements), permits a general summary of findings, can be
applied to most settings in which the question asked is limited
to a few statements, and is useful for general management “go
or no go” decisions, the kind of final judgment that a
sponsoring program officer would be interested in. This type of
data would be the primary target to use in a summary
presentation of the findings.
Level II information represents data useful for identifying the
strengths and weaknesses of a program by its representative
components, for example: the materials worked well,
instructional delivery could be improved, stakeholders were
satisfied, recruitment should be strengthened, or case work
referrals worked well. Comparisons across sites can be made
only in cases in which program elements are identical or very
similar and this comparison information, when legitimate, could
be included in the Level I report. The Level II information is
most useful for local program staff to identify areas in which
program development and delivery could be improved. It
provides a starting point to delve into specifics. Both Level I
and Level II information tend to be reliable due to the fact that
judgments made are cross-referenced and double-checked
during the data collection and analysis, whereas Level III
information is more site-specific and detailed.
Level III information is the most detailed, such as open-ended
comments on surveys or specific references to incidents made
during interviews, and while such data tend to be rich in detail,
they are also less reliable in that general judgments about
success or failure of a program component cannot be made. On
the other hand, they can provide very useful information for
improving specific aspects of a program as well as offer
explanations as to why Level II information yielded results in a
particular direction.

CONCLUSIONS
If one wishes to compare how evaluation work is done in not-
for-profit settings to how it is done in corporate training or
organizational development efforts, there are at least three
points of distinction. First, not-for-profit evaluation planning
tends to be more complex in that it requires a formal,
comprehensive stakeholder analysis. The Aspen Institute (2000)
study pointed out stakeholder analysis as the initial best
practice in not-for-profit program evaluation. The complexity is
driven more from the political side because there are likely to
be more stakeholders, with considerable variance in levels of
sophistication regarding evaluation methods, not all of whom
share the desire to even conduct the evaluation. Specific
attempts must be made to, first, identify all relevant
stakeholders, and then perform a process of consensus-building
to assure that all legitimate evaluation questions have been
posed. What happens during this initial stage has considerable
influence on the remainder of the evaluation process.
Second, evaluation methodologies employed tend to be dictated
by the sponsoring organizations, which can be quite varied in
their standards. With minor requirements, an evaluation
conducted under a U.S. Department of Labor or Housing and
Urban Development are quite flexible, whereas projects under
the Centers for Disease Control and Prevention, Health and
Human Services or a major grant from a private foundation are
more prescriptive. In fact, some foundations require a form of
“certification” for many of their external evaluators. Most
evaluation efforts are likely to assign to an external evaluator
due to the fact that most not-for-profit organizations do not
have the requisite capabilities to fulfill that role. This fact adds
to project complexity, but especially adds to the politics in
dealing with the recipient organization.
Third, evaluation efforts in not-for-profits can be as rigorous as

those in the private sector: employing sophisticated
experimental designs, requiring validated data collection tools,
and using complex analysis of qualitative data, but there is an
underlying requirement that the program must be described
adequately. This requirement that evaluation must address the
description of the target activities leads to the use of logic
models not often found in private or corporate evaluation
efforts. This author believes that a similar process might be
useful in those settings.
References
Annie E. Casey Foundation. (1997). Evaluating comprehensive
community change. Baltimore, MD: Author.
Aspen Institute. (2000, February). Current evaluation practice
in the nonprofit sector. Snapshots, Research Highlights for the
Nonprofit Sector Research Fund, 9.
Centers for Disease Control and Prevention. (1999). Framework
for program evaluation in public health (MMWR 48, No. RR–
11). Atlanta, GA: Author.

CHAPTER SEVENTEEN
Using New Technology to Create a User-Friendly Evaluation
Process
William J. Rothwell
Anita Pane Whiteford
Much interest has been focused on emerging technologies in
instructional design and delivery (see, for instance, Rothwell,
Butler, Maldonado, Hunt, Peters, Li, & Stern, 2004). However,
less attention has focused on applying new technolo gy to
evaluating instructional or other performance improvement
interventions (see Rothwell, 2005; Rothwell, Hohne, & King,
2007). After all, emerging technologies do have the potential to
expand the scope of evaluation in terms of number and
geographic distribution of participants; types of data that can be
collected; size and relational capabilities of storage and
retrieval systems; types of analysis conducted; and type of
reports generated. Emerging technologies also offer the promise
of exciting new ways to select or design data collection tools;
collect and analyze data; store and retrieve data and results; and
report evaluation results.
This chapter examines some of the new technology-based
evaluation tools. The chapter will answer the following
questions: (1) What technology-based evaluation tools are
available to human performance technology (HPT)
practitioners? (2) How can HPT practitioners use these tools to
evaluate performance interventions? (3) Are HPT practitioners
conducting evaluations of performance interventions? Why or
why not? (4) Why use technology-based tools? (5) What does
the future hold for HPT practitioners who use technology-based
tools to evaluate performance interventions? In addition, this
chapter will discuss ethical issues that could be posed by new

technology that could take HPT practitioners into uncharted
territory.
WHAT TECHNOLOGY-BASED EVALUATION TOOLS ARE
AVAILABLE TO HPT PRACTITIONERS?
Technology-based evaluation tools can be understood to mean
any electronically enabled applications that have experienced
accelerating growth the last few years in assisting HPT
practitioners and evaluators with ability to perform work faster,
better, and with increased skills. With new technology-based
applications appearing almost daily, it remains difficult to
catalog all the new technologies. Therefore, this section will
discuss only some technology-based tools that are expanding the
scope of evaluation, collecting evaluation data, storing and
retrieving evaluation data, analyzing evaluation data, and
reporting evaluation results.
General Characteristics: Multi-Tasking and Engagement
All evaluation tools that are available to HPT practitioners
expand the scope of evaluation in some way. Most tools have
the ability to allow users to multi-task: collect, store, retrieve,
analyze, and report data. Engagement theory is perhaps the most
well-applied theory to explain the reasons for the successes for
technology-based applications for HPT practitioners.
“Engagement theory is best applied because the theory is based
on three primary means to accomplish engagement: an emphasis
on collaborative effort, project-based assignments, and non-
academic focus” (Kearsley & Shneiderman, 1999, p. 5).
“Engagement theory is intended to be a conceptual framework
for technology-based learning and teaching because technology
can facilitate engagement in different ways that are difficult to
achieve without technology-based applications” (Kearsley &
Shneiderman, 1999, p. 1). The theory is based on the foundation
of engaged learning, defined by Kearsley and Shneiderman
(1999) as “participant activities that involve active cognitive
processes such as creating, problem-solving, reasoning, making

decisions, and evaluating” (p. 1). HPT practitioners can use
engagement theory to aid in selecting the appropriate tools for
evaluating HPT applications. Practitioners can use the theory to
formulate such critical thought-provoking questions as (1) For
what target audience is engagement theory most and least
effective? (2) What skills, required of participants to be
effectively evaluated, are essential to collaborative activities?
(3) How should individual differences be evaluated in
collaborative work? and (4) What tools would be most
appropriate to evaluate collaborative software tools that might
be used during the intervention (Kearsley & Shneiderman,
1999)?
As mentioned previously, most technology-based evaluation
tools are multi-tasked whereby the tools can collectively gather
data, store and retrieve data, analyze data, and report data. All
features, as well as some prime examples of technology-based
tools, will be discussed in the following sections about stand-
alone evaluation tools, built-in evaluation features, tools to
evaluate one-way communication interventions, and tools to
evaluate two-way communication interventions.
Stand-Alone Evaluation Tools
Some technology-based evaluation tools can be used on their
own or can be blended with performance interventions such as,
for example, Questionmark (see www.questionmark.com) and
One-Touch system (see www.one.touch.com).
Questionmark. Perception Assessment Management System
enables educators and trainers to author, schedule, deliver, and
report on surveys, quizzes, tests, and exams. Questionmark
offers evaluators the ability to author questions and assessments
easily; bank questions by learning objective; deliver to browser,
PDA, CD-ROM, or paper; provide instant feedback to enhance
learning; provide a secure delivery platform for high-stakes
exams; give administrators on-demand results, reports, and item

analysis; and randomize the presentation of questions and
choices (see www.questionmark.com/us/perception/index.aspx).
Idaho Power is a case study in using Questionmark. The
company provides electricity to Idaho and to sections of Oregon
and Nevada. It employs eighteen hundred line service workers
and field engineers. Idaho Power sponsors a rigorous
apprenticeship program of four and half years’ duration and
administers an average of three to four hundred exams per week
as well as a final examination at the program’s end. Idaho
Power began to use Questionmark to maintain all exams in one
database and to have the ability to work continuously on new
exams or improve existing ones.
One-Touch. One-Touch System is the leading provider of
interactive distance communication and learning solutions for
corporate, educational institutions, and governmental offices.
One-Touch integrates video, voice, and data over any broadband
network with measurable and certifiable results, extending the
reach of communication and learning sessions to geographically
dispersed participants.
J.C. Penney became a customer of One-Touch System in 1996
after closing its management training facility in Texas. J.C.
Penney decided to use distance learning to train 149,000 store
associates so as to ensure consistency in its training. Store
associates see the facilitators’ video live on a classroom
television and interact by using the One Touch interactive touch
pad. Facilitators can pose questions and post quizzes, providing
an important means to measure and establish accountability
among associates for their learning. All participant data are
gathered in a central database for further analysis, including
attendance training records, the number of times an associate
calls in, and question and quiz performance. J.C. Penney saved
$58,000 in delivering one training class and reduced travel costs
for training by $12 million.

Built-In Evaluation Features
Sometimes “evaluation” features are built into a technology-
based performance intervention. A good example is a learning
management system (LMS). An LMS is a set of software tools
designed to manage user learning interventions. LMS has
become a popular method of technology-based application in
both the corporate and academic worlds. An LMS allows an
organization to manage users, roles, courses, instructors, and
facilities; post course calendars; post messages to learners;
offer assessment and testing, including administering
participant pre-tests and post-tests; display scores and
transcripts; grade course-work; and host web-based or blended
learning (see
http://en.wikipedia.org/wiki/learning_management_system).
Ilias is an example of an LMS that has a built-in evaluator tool
(see
www.ilias.de/docu/ilias.php?ref_id=392&frame=maincontent&o
bj_id=11674&cmd=layout&cmdClas). Ilias is a powerful web-
based learning management system that supports tools for
collaboration, communication, evaluation, and assessment. The
evaluation and assessment component of Ilias permits
evaluators to create assessments based on multiple choice,
single choice, allocation questions, cloze questions (free text,
select box) arrangement duties, hot spots (search images to
click on), and unsettled questions.
TOOLS PRACTITIONERS USE TO EVALUATE ONE-WAY
COMMUNICATION INTERVENTIONS
Webcasts, podcasts, and web-based surveys rely primarily on
one-way communication. A presenter broadcasts a message, or a
series of messages, that can only be received by participants
over the web, iPod, or other channels. Participants cannot
respond in real time, so they may properly be termed as forms
of one-way communication.

Podcasts and Webcasts
“A podcast is a digital audio program, a multimedia computer
file that can be downloaded to a computer, an iPod, or another
device, and then played or replayed on demand” (Islam, 2007, p.
5). A webcast is used on the Internet to broadcast live or
delayed audio and/or video transmissions, like traditional
television and radio broadcasts (see
www.webopedia,com/term/w/webcast.html). Unlike webcasts,
podcasts are always available, portable, user-friendly,
immediately useable, and inexpensive (Islam, 2007). Podcasts
and webcasts can be used in the evaluation phase of
performance improvement interventions. The intervention is
recorded and that allows multiple reviews to measure the
effectiveness of the intervention and assess its outcomes (Islam,
2007). Facilitators could also evaluate their training skills after
a training session so as to identify possible improvements for
future training sessions.
Web-Based Surveys
Web-based surveys are assessment tools designed by evaluators
and posted on the Internet for participants to evaluate the
technology-based application used to deliver the learning
intervention. Web-based surveys collect data through self-
administered electronic sets of questions on the web (Archer,
2003). Software tools for web-based surveys include, among
others, Survey Monkey (see www.surveymonkey.com) and
Zoomerang (see www.zoomerang.com).
According to Granello and Wheaton (2004) and Archer (2003),
web-based surveys can be administered easily by following
several important steps:
Determine the target population to be assessed and the purpose
of the survey;
Design the layout of the survey and format questions,

remembering to keep the survey short and simple or else risk
skewed answers;
Provide a welcome screen that is user friendly from the start
that explains the easiness of responding to questions with
instructions;
Post questions that have a consistent format and flow;
Test the system before launching it to make certain the glitches
are worked out;
Pilot test the survey with a small group similar to the targeted
audience; and
Strategically plan the survey distribution with only a few email
reminders.
TOOLS PRACTITIONERS USE TO EVALUATE TWO-WAY
Many new technologies facilitate the interaction of people in
real time. These technologies are called social networks.
Examples include group decision support systems, webinars,
web conferences, virtual classrooms, instant messages, chat
rooms, blogs, Facebook, YouTube, Twitter, wikipedias (wikis),
second-life virtual reality applications—and many others.
Social network websites are legion and their range encompasses
every imaginable topic. Examples include Facebook, MySpace,
Windows Live Spaces, Linkedln, and Broadcaster.com. HPT
practitioners may pose questions and then receive real -time—or
near-real-time—responses. The greatest promise for realtime
data collection and analysis probably exists with group decision
support systems in which live participants interact in real
time—or virtual participants interact concurrently—to reach
decisions. Participants can reach some level of agreement
collaboratively on:
What performance problems are important;
How important they are;
What causes them;
What solutions should be applied to solving those problems by

addressing root causes;
How those solutions can be implemented, and
How results may be evaluated.
As a result, those affected by the problems can have a stake in
diagnosing the problem, solving it, and establishing and
tracking relevant metrics (Rothwell, 2005).
HPT practitioners can pose questions, generate responses from
many people, and then facilitate group activities as members
analyze their own results. Data collected from these activities
are useful for evaluating the process and the results of the group
decision support system intervention itself. HPT practitioners
using social networks can get many different views to evaluate
interventions. They can, for instance, benchmark how other
HPT practitioners have done evaluations for similar
interventions and share knowledge on the latest evaluation tools
and techniques.
ARE HPT PRACTITIONERS USING TECHNOLOGY-BASED
EVALUATION TOOLS?
The most common method of evaluation used by HPT
practitioners is a Level 1 on Donald Kirkpatrick’s four-level
evaluation model. Level 1 is the reaction level, and it focuses
on how much people liked the intervention. Participant
evaluations— also called “smile sheets”—remain the most
popular way to evaluate training, and most smile sheets are
administered using paper-and-pencil forms. Some large training
companies have advanced as far as using scannable sheets to
collect participant reactions. Some are even using web-assisted
survey tools.
Web-Assisted Survey Tools
Several organizations have gone the next step to use Survey
Monkey and similar web-assisted survey software packages to
administer participant evaluations. These organizations also
post results of participant evaluations online for participants

and managers to review.
For instance, the leaders of one company experiencing high
turnover asked human resources to explore the root cause of the
turnover. The human resources manager, using Survey Monkey,
surveyed middle managers and production supervisors. The
software made it easy to collect and analyze the data. The
human resources manager then fed the results of the survey back
to company leaders. Online survey-based methods are
particularly popular to use with distance education methods,
since online data collection just seems appropriate to use with
online training delivery.
Another organization has experimented with real-time data
collection using wireless personal digital assistants (PDAs). At
the end of a training program, participants are polled in real
time to determine their perceptions of the training. Their
reactions are compiled in front of them and projected on a
screen. They are then asked by the facilitator to offer immediate
ideas about how to improve the training for subsequent
delivery. This approach, of course, is really an application of a
group decision support system.
Beyond Level 1
Why do most HPT practitioners typically evaluate only
participant reactions and learning? What about evaluating HPT
performance interventions for transfer of skill and impact on the
organization? Few HPT practitioners conduct transfer of
learning studies and impact studies. They typically give several
reasons. Some are that such evaluation studies are more time-
consuming; evaluators lack resources and sometimes an
understanding of how to conduct sophisticated evaluation
studies using robust research designs; and organizational
leaders do not ask for evaluation studies.
But it is critically important that HPT practitioners conduct

evaluation studies at all levels of Kirkpatrick’s pyramid—
including transfer and impact studies— to build support from
organizational leaders (Kirkpatrick & Kirkpatrick, 2006;
Phillips & Phillips, 2005). Technology-based evaluation tools
can help HPT practitioners achieve this goal.
WHY USE TECHNOLOGY-BASED TOOLS IN
EVALUATION?
There are many advantages and disadvantages to HPT
practitioners using technology-based tools for evaluation. This
section will examine those advantages and disadvantages.
Advantages to Using Technology-Based Tools in Evaluation
The advantages of using technology-based tools in evaluation
include decreased cost, immediacy, and ease of use for the
respondee and the evaluator:
Cost. New technology offers the promise of reducing the cost of
collecting and analyzing data. Instead of paying for postage on
mail questionnaires, participants can use survey software at a
per-use cost. And it is less expensive for an organization to
email surveys to hundreds of people than to mail a hard copy of
a survey.
Immediacy. In most cases, the greatest appeal of new
technology when applied to evaluating performance
interventions is its immediacy—that is, how quickly it can be
collected and used. Gone are the days when lengthy analysis
could be painstakingly applied in step-by-step, linear project
approaches and then followed up by deliberate action. Time has
become the only strategic resource. Fast approaches to
addressing performance problems have grown more appealing to
time-pressured, stressed-out managers and workers alike. Many
of these approaches call for concurrent project management
approaches that combine steps or carry them out at the same
time.

Ease of Use. From the participants’ point of view, it is easier to
complete web-based surveys. The respondent does not have to
hassle mailing back a survey form. From the evaluator’s
standpoint, some survey software permits back-end statistical
analysis so that an entire data set can be recomputed every time
one user completes a survey. That can also reduce both the time
and cost to collect and analyze survey data.
Disadvantages to Using Technology-Based Tools in Evaluation
Of course, new technology is no panacea. It does introduce new
challenges. According to Lefever, Dal, and Matthiasdottir
(2007), the disadvantages of using online surveys including the
following:
Target population being surveyed may lack computer skills, not
accept using a computer, or may not have easy access to a
computer;
Technological problems could occur with the equipment or the
Internet service;
Emails sent to participants asking for their voluntary
participation to complete the survey may be routed to the junk
mail box, instead of the inbox, and participants may delete the
email from researchers without opening it;
Challenges may occur when attempting to get a representative
sample of the target population for the study by identifying
email addresses; and
It may be difficult to verify the identity of participants
completing the survey.
Verifying Participants. When using technolo gy-based data
collection tools, evaluators often face problems in determining
whether the right people are actually responding (Do we really
know who logged in?), whether those completing online forms
actually completed instruction or participated in a performance
intervention, and whether data collected are kept secure and
confidential.

Security. Security is a major concern when using the Internet
for data collection. Virtual private networks (VPN) are a
common, popular vehicle used by researchers to ensure the
security of respondents completing data collection instruments,
although VPNs are by no means foolproof or hacker proof.
According to Wikipedia, virtual private networks (VPNs) are
computer networks in which some links between nodes are
carried by open connections or virtual circuits in some larger
network (for example, the Internet) instead of by physical wires.
The link-layer protocols of the virtual network are said to be
“tunneled” through the larger network. One common application
is to secure communications through the public Internet, but a
VPN need not have explicit security features, such as
authentication or content encryption. VPNs, for example, can be
used to separate the traffic of different user communities over
an underlying, secure network (see
http://en.wikipedia.org/wiki/virtual_private_network).
Overcoming the Disadvantages
With these new problems come new opportunities. Here are
some suggestions:
Security. In addition to using a VPN, HPT practitioners may
select programming language based on security needs. Use
JavaScript to validate responses given by participants in online
surveys. JavaScript is a programming language that alerts
respondents participating in online studies if the fields were
completed incorrectly or if fields were left blank during
submission (White, Carey, & Dailey, 2000). Respondents
receive error messages to fix the issue(s) in the survey before
the final survey submission can be made. Perl is another
programming language that can avoid both the cost and the
potential for error incurred by the tedious data processing. Perl
automatically deletes field names and other extraneous
characters; transforms the various data fields into a single

subject record; aggregates the subject records into single files
maintained on the server; and performs any required algorithmic
data re-coding before the researcher sees it (White, Carey, &
Dailey, 2000).
Identifying Participants. One way to make sure that the intended
participants are responding is to ask the participants to pr ovide
information that only would be able to identify—such as a
password, an encrypted code, or an email address).
WHAT DOES THE FUTURE HOLD IN STORE?
The future holds exciting possibilities for applying emerging
technology to evaluating performance interventions. Here are a
few examples for the immediate future and beyond.
Time Travel with Kirkpatrick . . .
It may be necessary to rethink Kirkpatrick’s well-known four
levels of evaluation to include a consideration of time. The
three time points are before the intervention, during the
intervention, and after the intervention. (See Figure 17.1.)
Kirkpatrick’s four levels are, of course, quite well known,
although often criticized (see Holton, 1996). Level 1 focuses on
reactions, addressing how much people liked the training or
other performance interventions in which they participated.
Level 2 focuses on learning, addressing how much people
learned from training or from other performance interventions
in which they participated. Level 3 focuses on behavior,
addressing how much people changed their on-the-job behavior
based on the performance intervention. Level 4 focuses on
results, addressing how much the organization gained in
productivity improvement as a result of the performance
intervention. But each level may also be examined at three
points in time—before, during, and after the performance
intervention.

Adding the time element transforms the “hierarchy” into a
“grid.” Each cell of the grid suggests a range of different
approaches and focal points of concern. For instance, reaction
viewed before the performance intervention focuses on what
expectations the participants targeted for change have about the
intervention; reaction viewed during the performance
intervention focuses on monitoring changing attitudes about the
intervention while the intervention is implemented; and reaction
viewed after the performance intervention measures how much
people liked the change effort upon completion. Each cell of the
grid may prompt HPT practitioners to think of several ways by
which to measure at each point in time. New, emerging
technology may support evaluation in each cell of the grid.
Possible ideas about doing that are summarized in Table 17.1.
Of course, creative HPT practitioners may come up with
additional ideas about ways to apply new technology in
conducting evaluation. Use the worksheet in Exhibit 17.1 to
stimulate thinking about other ways to evaluate performance
interventions on different levels of the hierarchy and at
different points in time.
Getting Help from Avatars
Futurists already predict a time when avatars, representations of
people often used in second-life applications, will be guided by
artificial intelligence and not just by people (see Hill, 2008).
That will mean that second-life avatars may be able to interact
with people in simulated ways, giving them opportunities to
rehearse actions in real life that might not otherwise be possible
due to safety or other concerns. If that happens, then it may be
possible to simulate efforts to collect, analyze, and evaluate
data about instruction—or other performance interventions—
before the change effort is implemented. It would go the step
beyond mere guesswork forecasting to actual, robust, and
simulated pilot-testing of interventions assisted by new
technology.

Getting on the Grid
Futurists predict that the Internet and the web will be
transformed into the grid in which people, objects, and data can
be linked together as “nodes.” That will speed up efforts to
achieve results by seamlessly integrating interactions between
humans and technology. It will mean that assistive technology,
now associated with technology that helps the disabled to
perform like the able-bodied, may become associated with the
means to integrate technology seamlessly with human efforts to
perform work (Foroohar, 2002).
Table 17.1 Possible Ways That Emerging Technologies May Be
Used to Evaluate Performance Technology Interventions
example, eyeglasses that permit users to see while also
permitting them to access real-time Internet and video (see
“Newer design of close-up computer monitors increases ease of
use,” 2002). This could permit real-time, task-specific coaching
on how to perform work. It could also give HPT practitioners
the ability to analyze data even as they collect it, thereby
slashing the time it takes to make data usable.
Another example: cell phones that do everything from access
the Internet to permit instant messaging. Those cell phones are
already with us, but their applications in collecting, analyzing,
and evaluating real-time performance information is only now
being considered. Cell-phone-based training and wristwatch-
based training, perhaps in three dimensions, may be the next
wave after podcasting, and that opens up the potential to give
learners realtime performance assistance and give HPT
practitioners the capability to do realtime data collection,
analysis, and evaluation.

What Ethical and Confidentiality Issues Arise When Using New
Technology to Evaluate Performance Interventions?
With new forms of technology arise new challenges in ethics
and confidentiality. For example, there are issues with almost
every method of online intervention technology—especially
blogs, wikipedias, Facebook, YouTube, Twitter, seminars, and
web conferences. Blogs, wikipedias, Facebook, YouTube, and
Twitter are technological avenues (social networks) where it is
easy to be unethical and violate copyright because the
individuals posting information need not identify themselves
and are not held accountable or mandated to publish the source.
Just Google “media and confidentiality” to find instances in
which clients have been specifically discussed by healthcare
professionals.
Blogs. Some organizations are beginning to encourage employee
groups to create blogs. Yet, blogs can also prompt many ethical,
confidentiality, and copyright dilemmas. Blogs are a highly
popular method to communicate by media. There are a multitude
of blogs that are not controlled and that permit individuals to
display text, images, audio files, and video available on the
Internet without permission from the original creator or
copyright holder (Kuhn, 2007). Currently, no code of ethics
exists for bloggers, and so blogging on the web can be as
lawless as the Old West. However, if a code of ethics were to be
established, accountability and anonymity should be two key
critical components. Bloggers should be accountable for the
information that they post to give proper credit when due and
should preserve anonymity when posting information (Kuhn,
2007). Evaluation of blogging as a performance intervention
would require the evaluator to identify and evaluate the ethical
issues as well as the performance results of the blogging
intervention.
Digital Media. The U. S. government has considered issues of

ethics and digital media. This has prompted the government to
legislate a few federal acts in the hopes of upholding ethical
standards. A few of these acts are as follows:
Audio Home Recording Act of 1992 (Pub.L.No-102-563):
Exempts from copyright infringement the making of copies for
personal use of music files in digital form, provided that those
copies are made with approved equipment.
No Electronic Theft Act of 1997 (Pub. L. No. 105-147):
Establishes penalties for unauthorized sharing and redistribution
of copyrighted material over the Internet.
Digital Millennium Copyright Act of 1998 (Pub. L. No. 105-
304): Creates several protection and enforcement mechanisms
for copyrighted works in digital form.
Copyright. HPT practitioners should be aware of resources
available from the U.S. Copyright Office (see
www.copyright.gov). This link provides information about
copyright issues. It also explains how to copyright a work or
record a document and reviews applicable copyright laws and
policies. If there is uncertainty or questions of issues with
copyright, HPT practitioners should refer to this site for
additional information. They may also consult The Congress of
the United States Congressional Budget Office (2004).
The federal laws listed in this chapter are based on copyri ght
issues to protect individuals’ work that is displayed on the
Internet. It is very important that HPT practitioners follow the
federal laws when conducting technology-based evaluations. A
significant case would be if a practitioner is using an evaluation
assessment tool that he or she found online and the practitioner
feels that, since the assessment tool was found online, there is
no need to obtain copyright permission for subsequent use. This
is a common misconception. Whether evaluation material is
obtained online or off-line, proper permission must be obtained
from the copyright holder.

Use of Information. One dilemma for HPT practitioners is how
to use information on ethics and confidentiality when selecting
the appropriate evaluation tool for their needs. Practitioners
must know how utilizing the information obtained from the data
collection methods may impact the participants in regard to
ethics and confidentiality. For example, as with any data
collection method to be selected for a study, HPT practitioners
have to first gauge the conditions under which the data will be
collected, the population to be sampled, and then the most
appropriate method that will yield the most desirable results—
while also protecting confidentiality and well-being of human
subjects. Internal Review Boards (IRBs) are helpful to HPT
practitioners in the sense that IRBs can help practitioners
remain ethical and preserve confidentiality as they collect data.
The IRBs will also educate practitioners on legal issues that can
arise during data collection. Many federal government
guidelines exist to protect human subjects when data are
collected.
SUMMARY
This chapter listed some examples of technology-based
evaluation tools that could be used to evaluate performance
interventions. Each new technology holds the promise of
providing new, exciting ways to select or create data collection
tools and to collect, store, retrieve, analyze, and communicate
data in faster, more effective ways. Each new technology may
also challenge HPT practitioners to come up with new ways to
think about evaluating performance interventions. Indeed, it
may be necessary to reinvent Kirkpatrick’s famous hierarchy
into a grid that addresses times before, during, and after
performance interventions.
The future is likely to offer even more opportunities to apply
technology to how HPT practitioners evaluate. Just a few
exciting possibilities include avatars that “think”; the
emergence of the grid to replace the web and the Internet; and

human-machine interactions that are seamlessly integrated in
real time.
It is critically important to obey the appropriate copyright laws
when using evaluation tools owned by others. Following the
copyright laws means obtaining permission from copyright
holders to use their work in your evaluation studies. Equally
important, it is the duty of HPT practitioners to abide by all
ethical standards and confidentiality laws of the U.S.
government.
References
Archer, T. M. (2003, August). Web-based surveys. Retrieved
July 7, 2008, from www.joe.org/joe/2003august/tt6.shtml.
Foroohar, R. (2002). Life in the grid. Retrieved July 7, 2008,
from www.newsweek.com/id/65626.
Week 6 Assignment Technological Advancements in Evaluation
Process
Many individuals associated with community agencies, heal th

break the utility and value of the program under considerati on.
Mosele, J. & Dessinger, J. (2009). Handbook of Improving
Performance in the Workplace. (Volume 3). Pfeiffer-Wiley
To prepare for this Discussion, pay particular attention to the
following Learning Resources:
· Review this week’s Learning Resources, especially:
· Read Week 6 Lecture – See Word doc.
· Read Chapter 16-17 – See Word doc
Assignment:
This week you learned about the importance of technology and
the role it plays in the evaluations process. Conduct research
using the internet to determine what emerging technologies can
be used in the evaluation process.
· Consider the differences in non-profit and for-profit business
and the methods they use for evaluation.
· Using your experience and your career choice, analyze the
benefits of either non-profit or for-profit organizations
· Analyze the importance for evaluation within the workplace.
· Construct a plan for using emerging technology based on the
organization choice.
· How it will generate success long-term.
· Write at least 2-3 pages
· Include cover page and reference page.
· Microsoft Word in APA style.

· Use font size 12 and 1” margins.
· Use at least three references from outside the course material
· One reference must be from Text book, lectures
CHAPTER SIXTEEN
Understanding Context: Evaluation and Measurement in Not-
for-Profit Sectors
Dale C. Brandenburg
avoided. A major reason for this perspective is that evaluati on

CONTRAST OF CONTEXTS
Evaluation in not-for-profit settings tends to have different
criteria for the judgment of its worth than is typically found in
corporate and similar settings. Such criteria are likely to include
the following:
How useful is the evaluation?
Is the evaluation feasible and practical?
Does the evaluation hold high ethical principles?
Does the evaluation measure the right things, and is it accurate?
Using criteria such as the above seems a far cry from concepts
of return on investment that are of vital importance in the profit
sector. Even the cause of transfer of training can sometimes be
of secondary importance to assuring that the program is
described accurately. Another difference is the pressure of time.
Programs offered by not-for-profit organizations, such as an
alcohol recovery program, take a long time to see the effects
and, by the time results are viewable, the organization has
moved on to the next program. Instead we often see that
evaluation is relegated to measuring the countable, the numbers
of people who have completed the program, rather than the life-
changing impact that decreased alcohol abuse has on
participants. While the latter is certainly important, the typical
community-based organization (CBO) is limited in its resources
to perform the long-term follow-through needed to answer the
ultimate utility question. Thus, the choice of what is measured
tends to be the result of negotiation with stakeholders. The
broad goals of evaluation tend to be grouped among the

following:
Understanding and sharing what works with other organizations
and communities;
Building sustainability of programs and ensuring funding;
Strengthening the accountability of the programs with various
public constituencies;
Influencing the decisions of relevant policy makers and program
funders;
Building community capacity so that future engagements have
greater community participation; and
Understanding where the program is going so that it results in
reflecting on progress that can improve future programs.
These goals reflect some of the typical objectives for applying
evaluation in not-for-profit settings. The goals embody specific
activities that can be designed to collect evidence on a
program’s effectiveness, be accountable to stakeholders,
identify projects for improvement, clarify program plans, and
improve communication among all groups of stakeholders. The
types of programs or actions that are designed to improve
outcomes for particular individuals, groups, or communities
considered in this chapter include the following:
Direct service interventions: improve the nutrition of pre-school
children;
Research endeavors: determine whether race disparities in
emergency room care can be reduced;
Advocacy efforts: campaign to influence legislation on proper
use of infant car seats; and
Workforce training programs: job training program to reduce
unemployment among economically disadvantaged urban
residents.
The results of evaluation in not-for-profit settings are typically
designed to provide information for future decision making.
Most efforts can be grouped into three categories: process
evaluation, short-term evaluation, or long-term (outcome)

evaluation. In fact, it is rare to see a program evaluation effort
that does not include some process evaluation. Process
evaluation is considered important because it typically yields an
external view of how the program was conducted; in other
words, it provides a detailed description of the objectives,
activities, resources used, management of the program, and
involvement of stakeholders. It would be difficult to judge any
outcomes of a program without understanding the components
of the program in detail. That is why process evaluation is
important. Short-term evaluation deals with the accomplishment
of program objectives or the intermediate links between the
program activities and the long-term outcomes. Outcome
evaluation is associated with the long-term effects such as
health status or system changes that are often beyond the range
of a typical evaluation effort.
STAKEHOLDER ANALYSIS
One of the key tools used to structure the evaluation process in
not-for-profit settings is the stakeholder analysis. It is a key
initial stage to develop the primary issues and evaluation
questions from which other stages of the evaluation process can
be built. The stakeholder analysis is designed to identify the
needs and values of the separate stakeholders and combine the
results so that an adequate plan can be developed. It is rare to
find an evaluation effort in not-for-profit situations for which
there are at least three stakeholder groups with a major interest
in the effort. The stakeholder analysis is a means to organize
the political process and create an evaluation plan, as well as
satisfy the desires of the different perspectives and needs for
information. A first step in the process is to identify all groups
or individuals with a stake in the process, followed by a second
dividing the groups into primary and secondary members.
Primary members are those stakeholders who are likely to be
direct users of the evaluation results; whereas secondary
stakeholders may have an interest in the findings, but the results
of the evaluation are not likely to impact them directly.

For example, in the evaluation of a workforce development
project for urban adults, primary stakeholders would include the
sponsoring agency or funder, the manager of program delivery,
program developers, instructors, a third-party program
organizer, and partner organizations that might include
community-based organizations, a local business association,
and a representative from the city government. Secondary
stakeholders might include parents or relatives of program
participants, local welfare officials, advocacy groups, and the
participants themselves. A list of stakeholders can be
determined by answering the following questions:
Who pays for program staff time?
Who selects the participants?
Who champions the desired change?
Who is responsible for after-program behavior/performance?
Who determines success?
Who provides advice and counsel on program conditions?
Who provides critical guidance?
Who provides critical facts?
Who provides the necessary assistance?
Note that these questions do not identify any of the purposes of
the evaluation, only a means to distinguish who should be
involved in evaluation planning. The amount of stakeholder
involvement in the evaluation can be limited or it can be
substantial enough so that they assist in the design of the
overall process, including the development of data collection
protocols. The involvement of stakeholders should also lead to
increasing the value and usefulness of the evaluation (Greene,
1988) as well increasing the use of the findings. Such a process
increases the ownership in the report and possibly leads to a
common vision of collective goals (Kopczynski & Pritchard,
2004).
The second step of performing a stakeholder analysis involves

the identification of the requirements, or primary evaluation
purposes, and aligning those against the list of primary
stakeholders. The matrix shown in Table 16.1 is an example of
this analysis for a publicly funded workforce development
program for inner city adults.
The “requirements” or purposes of the evaluation can be
developed in a number of ways, but the major sources of the list
usually come from the published description of the program,
supplemented by interviews with the stakeholders. It would be
ideal if the matrix could be completed in a group setting, but it
is more likely that the evaluator develops the chart
independently and then shares the results in a program advisory
meeting. The “X” in a given box represents where a requirement
matches the possible use of that information by the selected
stakeholder. One can note, for example, that program Strengths
and Weaknesses are needed for staff internal to the development
and execution of the program but not of particular interest to
those outside that environment. Another outcome of this process
is at least a general expectation of reporting relationships that
would occur during the communication of the interim and final
results of the evaluation.
While it might seem that the stakeholder analysis might focus
on the program itself, there are other outputs that could be
entered into the analysis. Results of evaluation data are often
used to tell compelling stories that can increase the visibility
and marketing of the organization and increase accountability
with board members and community, as well as attract
additional sources of revenue. These findings were supported by
a national survey of local United Way organizations as reported
by Kopczynski & Pritchard (2004).
EVALUATION PURPOSES
While the stakeholder analysis establishes the initial phase of
understanding evaluation purposes, further definition is

supplied by a list of evaluation issues. These issues are
typically generated during the stakeholder analysis after further
understanding of the program description. Evaluation issues can
range from general questions about program impact to detailed
questions on the selection of service providers. A sample
(Gorman, 2001) set of questions or primary issues from a low -
income workforce development project that leverages resources
from a local housing authority are listed below:
Describe the major partners leading, managing, and staffing
each major activity area.
Describe the major program areas and the target population
service goals by area. What are the major strategies to be
employed to reach these service goals?
alleviating educational barriers to sustained employment? What
were the major instructional and training strategies employed to
attain the stated goals and objectives?
alleviating personal and social barriers to employment: poor
work histories, cultural barriers, substance abuse and
developmental disabilities, and limitations of transportation and
adequate childcare?
What were the participant impacts: wages, self-efficacy, and
employability?
Was there differential effectiveness of the program relative to
categories of the target population: public housing residents,
non-custodial parents, learning-disabled individuals, persons
with limited English proficiency, and other economically
disadvantaged groups?
Was there differential effectiveness relative to the categories of
high-demand sectors targeted by the program:
building/construction trades, health care, and hospitality?
What evidence indicates that the program management and its
partners can actively disseminate/replicate this program in other
regions via its current programs?

What are some key success stories that serve to illustrate
overall program value to participants and community?
How cost-effective was the project in general terms? By
program area? How did costs or efficiency of service change
over the course of the project?
What were the major lessons learned in the project? How do
these relate to self-sufficiency for the target population?
Community economic development? Leadership effectiveness?
These results were obtained from a stakeholder analysis and are
not yet grouped into evaluation components. Combining these
results with the stakeholder analysis assists in defining the
overall evaluation plan. It provides the framework for
developing the data requirements as well as the type of
measurement needed for instrumentation.
LOGIC MODELS
The final phase for planning the evaluation is the development
of a logic model, a description or map of how the major
components of an evaluation are aligned; that is, the connection
between how the program is designed and its intended results.
Logic models can be used in any phase of the program
development cycle (McLaughlin & Jordan, 2004) from initial
design to examining long-term impact. It is a means to make the
underlying theory behind the intervention more explicit and to
discover its underlying assumptions. Even the development of a
model, that is, mapping out all of its components, can be
instructive for program staff and other stakeholders. A logic
model is particularly useful for evaluators both as an advance
organizer as well as a planning tool for assessment development
(McLaughlin & Jordan, 2004). In many situations, such as
community or public health, the specification of a logic model
is a proposal requirement. Whether or not an evaluator builds on
an existing logic model or develops a preliminary version for
the evaluation plan, the map created is a visualization of how
the human and financial investments are intended to satisfy
program goals and lead to program improvements. Logic models

contain the theoretical and practical program concepts in a
sequence from input of resources to the ultimate impact.
Most logic models have a standard nomenclature (see the
Kellogg Foundation guidelines [W.K. Kellogg Foundation,
2007] and McLaughlin & Jordan) that contain the following
elements:
Resources: program inputs like needs assessment data and
capabilities (financial, human, organizational partnerships, and
community relationships) that can be allocated to the project.
Activities: the tasks or actions the program implements with its
resources to include events, uses of tools, processes, or
technology to perform actions to bring about intended changes
or results.
Outputs: the direct products of program activities or services
delivered by the program, even reports of findings that may be
useful to other researchers.
Outcomes: both short-term (one to three years) and longer-term
(four to six years) specific changes in the targeted individuals
or organizations associated with behavior, functioning,
knowledge, skills, or status within the community. Short-term
outcomes are those that are assumed to be “caused” by the
outputs; long-term outcomes are benefits derived from
intermediate outcomes.
Impact: the ultimate consequences or results of change that are
both intended and unintended for the individuals, organizations,
or communities that are part of the system, generally occurring
after program conclusion.
Whatever the process, such as a focus group, used to create a
logic model, another useful outcome is a listing of key
contextual factors not under the control of the program that
might have both positive and negative influences on the
program. These context factors can be divided into two
components—antecedent conditions and mediating variables
(McLaughlin & Jordan, 2004). Geography, economic conditions,

and characteristics of the target group to be served are examples
of the former; whereas staff turnover, new government
incentives, and layoffs for a local major employer are examples
of the latter.
The example shown in Figure 16.1 is a portion of a logic model
from public health and concerns a neighborhood violence
prevention program (Freddolino, 2005). The needs were derived
from data that showed that poverty and violence were
significant threats to the health of residents in Central County.
The general goals of the program were to increase the capacity
of local neighborhood groups to plan and implement local
prevention programs and work to establish links between
existing programs and agencies. Many logic models are likely
more complex than the example shown in that specific actions
are linked to given outputs and outcomes in order to
differentiate the various conceptual links in a program.
Logic models can be constructed in various ways to illustrate
various perspectives of a program. Figure 16.1 represents an
activities-based approach that concentrates on the
implementation of the program, and it is most useful for
program management and monitoring. A second approach based
on outcomes seeks to connect the resources to the desired
results and is specifically geared to subdivide the short-term,
long-term, and ultimate impact of the program. Such models are
most useful for designing evaluation and reporting strategies.
The logic model developed for a program should suggest the
type of measurement required in order to prove or improve the
model specified. Since the model is related to performance
objectives, it can also lead to assist in judging the merit or
worth of the outcomes observed.
A third type, the theory approach emphasizes the theoretical
constructs behind the idea for the program. Such models
concentrate on solution strategies and prior empirical evidence

that connect the selected strategies to potential activities and
assumptions. These are most useful for program planning and
design. Regardless of the type of model selected, each is useful
for the evaluator to provide more comprehensive descriptive
information that can lead to effective evaluation design and
planning.
Collaborative projects can also suffer from lack of
comparability in implementation due to inconsistent allocation
or availability of resources across partner organizations. This
lack of comparability in program resources often thwarts
attempts at gathering data to compare results across sites and
leads to misinterpretation of findings or no consistent findings.
An example evaluation plan (Gorman & Brandenburg, 2002)
from a multiple-site consortia project managed by a systems
integrator and funded by the U.S. Department of Labor is
provided in Table 16.3. In this case, the systems integrator,
American Community Partnerships, is an affiliated organization
associated with a large U.S. labor union whose goals, in part,
are to promote high-wage jobs in union roles for economically
disadvantaged urban residents. This particular program operated
in some of the very poorest neighborhoods in the respective
cities involved.
One can note that in complex arrangements represented in
Exhibit 16.1, that the management of internal and external
partner organizations is crucial to obtaining the needed services
for the target population as well as attempting to meet the long-
range goal of program sustainability. Initial sponsor-allocated
funds cover the start-up of the partnership, but the leveraging of
existing funds and other resources are required for the overall
effort to be successful. Other evaluation can be considerably

more complex when funded by two or more federal agencies.
The report provided by Hamilton and others (2001) is such an
example and provides a description of an evaluation funded
over five years by the U.S. Departments of Labor and
Education.
Foundation Guidelines
Numerous resources are available on outcome measurements
that are designed to assist not-for-profit organizations. A
current major source of information can be found through the
website of the Utica (New York) Public Library, associated with
the Foundation Center, a foundation collaborative. The site is
annotated and contains links to major foundations, professional
associations, government sponsors, technology uses, data
sources, statistical information, best practices, benchmarking,
and assessment tools.
Another set of evaluation models can be derived from an
examination of community change efforts. Two organizations
that have funded a considerable number of these actions include
the W. K. Kellogg Foundation and the Annie E. Casey
Foundation. Both organizations offer assistance in designing
evaluation efforts. These foundations, among others, want their
grantees to be successful in their efforts, so they have
developed evaluation guidelines for potential grantees to build
in evaluation efforts at the time of proposal writing. More
extensive guidelines exist for larger-scale efforts. Especially
instructive in this regard is Kellogg’s perspective on evaluating
social change in that they have constructe d a framework
especially for external evaluators. Based on the dynamics of
social change and the context of the program, they detail four
types of evaluation designs: exploratory, predictive, self-
organizing, and initiative renewal. Given the complexity of the
type of program under review, evaluators are charged to pay
attention primarily to the major aspects of the system change
and disregard those features that are of minor consequence.

agencies and program staff in understanding the major issues to
be considered. These organizations can be quite effective
because they understand the local and regional context.
DATA COLLECTION AND MEASUREMENT
As can be concluded from some of the previous discussion, data
collection and measurement for evaluation in not-for-profit
settings can range from the deceptively simple to the complex.
Most data collection schemes tend to be customized to the
environment of the program being evaluated. There are a
number of reasons for this. The measurement of a program is
most often linked to stakeholder considerations and the needs of
the sponsoring organization, as opposed to creating an elegant
experimental design. The use of both quantitative and
qualitative data is applied to answer the evaluation questions
posed at the outset of the investigation.
Second, there is a strong bias for producing data that are useful
for all stakeholders, and many users, such as directors of
community-based organizations are not used to interpreting
sophisticated statistical analyses. Data on effectiveness of
programs often can be derived from solid descriptions of
program activities and survey data. This is not to say that
measurement is based on the lowest common denominator of
sophistication, but the end-result is to use the findings in a way
that can improve program activities. The availability of
resources within not-for-profits generally implies that program
staff collects data for monitoring purposes, that is, what is
countable from an administrative perspective (Kopczynski &
Pritchard, 2004). Other not-for-profit organizational
shortcomings may enter into the development of a plan for a
comprehensives data collection scheme, namely “the lack of

appreciation of data, lack of training, poorly developed
information systems, and high turnover rates” (Kopczynski &
Pritchard, 2004, p. 652), besides the issue of limited budgets.
A third reason to customize data collection is that they may be
limited by budget considerations. Elegant and rigorous designs
cost more to implement and can have an indirect effect on
program activities if certain target participants cannot
participate in the primary intervention. This is not to say that
rigorous designs are not applied in not-for-profit settings,
especially in the public health domain. One such effort
(Huffman & others, 2002) funded by the Packard Foundation
presents a wealth of information on outcome measurement,
valid evaluation constructs, and a set of principles to apply in
such settings. Subjects of workforce development programs, for
example, are often difficult to recruit, so allocating a portion of
participants to a control setting is not cost-effective in many
cases. Even more challenging would be follow-up data
collection efforts on a homeless population (Kopczynski &
Pritchard, 2004).
It is probably instructive at this point to introduce an example
of the type and range of data needed to satisfy requirements in a
large-scale workforce development effort. Using the same
program (Gorman & Brandenburg, 2002) from the example in
Table 16.3, Exhibit 16.1 represents a listing of the data
elements selected to provide a comprehensive evaluation.
REPORTING RESULTS
If evaluation reports are to be useful for all stakeholders, it is
important to consider the communication of evaluation results
early in the design of the evaluation process. Such
considerations are needed because the demand for clear and
credible information from not-for-profits is high (Kopczynski &

Pritchard, 2004). In general, it is prudent to negotiate what
results are to be presented to whom at the time the stakeholder
analysis is conducted. Final reports of evaluation are typically
reported in two stages: (1) a formal written report that contains
all details of the design, implementation, data collection, data
analysis, findings, and conclusion with possible
recommendations and (2) a second summary version, such as a
PowerPoint presentation that lists the highlights of the
investigation. Other reports, such as interim findings or data
from a single site in a multiple-site investigation, results of a
set of interviews or a survey, or case descriptions, may also be
added, depending on funding and stakeholder needs.
Regardless of the type of report, there is still a need to organize
reports so that stakeholders may be guided to make appropri ate
decisions and plan future actions based on the findings. An
evaluation that contains as many data elements as depicted in
Exhibit 16.1 would need to be organized efficiently if
stakeholders were to be able to comprehend and sift through the
volume of data likely to be generated. Smock & Brandenburg
(1982) suggest a tool to aid in the process of organizing data
and presenting findings as portrayed in Table 16.4.
While the classification shown may be an oversimplification of
available data, it is nonetheless quite useful in presenting data
to unsophisticated users. The concept underlying its structure is
that the entirety of information may be ranked hierarchically
from its continuous mode into three levels. These three levels
represent an artificial trichotomy from very general (overall
success) to success or failure of specific program components to
very specific findings that have meaning to possibly a single
stakeholder. Level I information is the most general, must be
inferred from a variety of findings (rolled up across data
elements), permits a general summary of findings, can be
applied to most settings in which the question asked is limited
to a few statements, and is useful for general management “go

or no go” decisions, the kind of final judgment that a
sponsoring program officer would be interested in. This type of
data would be the primary target to use in a summary
presentation of the findings.
Level II information represents data useful for identifying the
strengths and weaknesses of a program by its representative
components, for example: the materials worked well,
instructional delivery could be improved, stakeholders were
satisfied, recruitment should be strengthened, or case work
referrals worked well. Comparisons across sites can be made
only in cases in which program elements are identical or very
similar and this comparison information, when legitimate, could
be included in the Level I report. The Level II information is
most useful for local program staff to identify areas in which
program development and delivery could be improved. It
provides a starting point to delve into specifics. Both Level I
and Level II information tend to be reliable due to the fact that
judgments made are cross-referenced and double-checked
during the data collection and analysis, whereas Level III
information is more site-specific and detailed.
Level III information is the most detailed, such as open-ended
comments on surveys or specific references to incidents made
during interviews, and while such data tend to be rich in detail,
they are also less reliable in that general judgments about
success or failure of a program component cannot be made. On
the other hand, they can provide very useful information for
improving specific aspects of a program as well as offer
explanations as to why Level II information yielded results in a
particular direction.
CONCLUSIONS
If one wishes to compare how evaluation work is done in not-
for-profit settings to how it is done in corporate training or
organizational development efforts, there are at least three

points of distinction. First, not-for-profit evaluation planning
tends to be more complex in that it requires a formal,
comprehensive stakeholder analysis. The Aspen Institute (2000)
study pointed out stakeholder analysis as the initial best
practice in not-for-profit program evaluation. The complexity is
driven more from the political side because there are likely to
be more stakeholders, with considerable variance in levels of
sophistication regarding evaluation methods, not all of whom
share the desire to even conduct the evaluation. Specific
attempts must be made to, first, identify all relevant
stakeholders, and then perform a process of consensus-building
to assure that all legitimate evaluation questions have been
posed. What happens during this initial stage has considerable
influence on the remainder of the evaluation process.
Second, evaluation methodologies employed tend to be dictated
by the sponsoring organizations, which can be quite varied in
their standards. With minor requirements, an evaluation
conducted under a U.S. Department of Labor or Housing and
Urban Development are quite flexible, whereas projects under
the Centers for Disease Control and Prevention, Health and
Human Services or a major grant from a private foundation are
more prescriptive. In fact, some foundations require a form of
“certification” for many of their external evaluators. Most
evaluation efforts are likely to assign to an external evaluator
due to the fact that most not-for-profit organizations do not
have the requisite capabilities to fulfill that role. This fact adds
to project complexity, but especially adds to the politics in
dealing with the recipient organization.
Third, evaluation efforts in not-for-profits can be as rigorous as
those in the private sector: employing sophisticated
experimental designs, requiring validated data collection tools,
and using complex analysis of qualitative data, but there is an
underlying requirement that the program must be described
adequately. This requirement that evaluation must address the

description of the target activities leads to the use of logic
models not often found in private or corporate evaluation
efforts. This author believes that a similar process might be
useful in those settings.
References
Annie E. Casey Foundation. (1997). Evaluating comprehensive
community change. Baltimore, MD: Author.
Aspen Institute. (2000, February). Current evaluation practice
in the nonprofit sector. Snapshots, Research Highlights for the
Nonprofit Sector Research Fund, 9.
Centers for Disease Control and Prevention. (1999). Framework
for program evaluation in public health (MMWR 48, No. RR–
11). Atlanta, GA: Author.
CHAPTER SEVENTEEN
Using New Technology to Create a User-Friendly Evaluation

Process
William J. Rothwell
Anita Pane Whiteford
Much interest has been focused on emerging technologies in
instructional design and delivery (see, for instance, Rothwell,
Butler, Maldonado, Hunt, Peters, Li, & Stern, 2004). However,
less attention has focused on applying new technology to
evaluating instructional or other performance improvement
interventions (see Rothwell, 2005; Rothwell, Hohne, & King,
2007). After all, emerging technologies do have the potential to
expand the scope of evaluation in terms of number and
geographic distribution of participants; types of data that can be
collected; size and relational capabilities of storage and
retrieval systems; types of analysis conducted; and type of
reports generated. Emerging technologies also offer the promise
of exciting new ways to select or design data collection tools;
collect and analyze data; store and retrieve data and results; and
report evaluation results.
This chapter examines some of the new technology-based
evaluation tools. The chapter will answer the following
questions: (1) What technology-based evaluation tools are
available to human performance technology (HPT)
practitioners? (2) How can HPT practitioners use these tools to
evaluate performance interventions? (3) Are HPT practitioners
conducting evaluations of performance interventions? Why or
why not? (4) Why use technology-based tools? (5) What does
the future hold for HPT practitioners who use technology-based
tools to evaluate performance interventions? In addition, this
chapter will discuss ethical issues that could be posed by new
technology that could take HPT practitioners into uncharted
territory.
WHAT TECHNOLOGY-BASED EVALUATION TOOLS ARE
AVAILABLE TO HPT PRACTITIONERS?

Technology-based evaluation tools can be understood to mean
any electronically enabled applications that have experienced
accelerating growth the last few years in assisting HPT
practitioners and evaluators with ability to perform work faster,
better, and with increased skills. With new technology-based
applications appearing almost daily, it remains difficult to
catalog all the new technologies. Therefore, this section will
discuss only some technology-based tools that are expanding the
scope of evaluation, collecting evaluation data, storing and
retrieving evaluation data, analyzing evaluation data, and
reporting evaluation results.
General Characteristics: Multi-Tasking and Engagement
All evaluation tools that are available to HPT practitioners
expand the scope of evaluation in some way. Most tools have
the ability to allow users to multi-task: collect, store, retrieve,
analyze, and report data. Engagement theory is perhaps the most
well-applied theory to explain the reasons for the successes for
technology-based applications for HPT practitioners.
“Engagement theory is best applied because the theory is based
on three primary means to accomplish engagement: an emphasis
on collaborative effort, project-based assignments, and non-
academic focus” (Kearsley & Shneiderman, 1999, p. 5).
“Engagement theory is intended to be a conceptual framework
for technology-based learning and teaching because technology
can facilitate engagement in different ways that are difficult to
achieve without technology-based applications” (Kearsley &
Shneiderman, 1999, p. 1). The theory is based on the foundation
of engaged learning, defined by Kearsley and Shneiderman
(1999) as “participant activities that involve active cognitive
processes such as creating, problem-solving, reasoning, making
decisions, and evaluating” (p. 1). HPT practitioners can use
engagement theory to aid in selecting the appropriate tools for
evaluating HPT applications. Practitioners can use the theory to
formulate such critical thought-provoking questions as (1) For
what target audience is engagement theory most and least

effective? (2) What skills, required of participants to be
effectively evaluated, are essential to collaborative activities?
(3) How should individual differences be evaluated in
collaborative work? and (4) What tools would be most
appropriate to evaluate collaborative software tools that might
be used during the intervention (Kearsley & Shneiderman,
1999)?
As mentioned previously, most technology-based evaluation
tools are multi-tasked whereby the tools can collectively gather
data, store and retrieve data, analyze data, and report data. All
features, as well as some prime examples of technology-based
tools, will be discussed in the following sections about stand-
alone evaluation tools, built-in evaluation features, tools to
evaluate one-way communication interventions, and tools to
evaluate two-way communication interventions.
Stand-Alone Evaluation Tools
Some technology-based evaluation tools can be used on their
own or can be blended with performance interventions such as,
for example, Questionmark (see www.questionmark.com) and
One-Touch system (see www.one.touch.com).
Questionmark. Perception Assessment Management System
enables educators and trainers to author, schedule, deliver, and
report on surveys, quizzes, tests, and exams. Questionmark
offers evaluators the ability to author questions and assessments
easily; bank questions by learning objective; deliver to browser,
PDA, CD-ROM, or paper; provide instant feedback to enhance
learning; provide a secure delivery platform for high-stakes
exams; give administrators on-demand results, reports, and item
analysis; and randomize the presentation of questions and
choices (see www.questionmark.com/us/perception/index.aspx).
Idaho Power is a case study in using Questionmark. The
company provides electricity to Idaho and to sections of Oregon

and Nevada. It employs eighteen hundred line service workers
and field engineers. Idaho Power sponsors a rigorous
apprenticeship program of four and half years’ duration and
administers an average of three to four hundred exams per week
as well as a final examination at the program’s end. Idaho
Power began to use Questionmark to maintain all exams in one
database and to have the ability to work continuously on new
exams or improve existing ones.
One-Touch. One-Touch System is the leading provider of
interactive distance communication and learning solutions for
corporate, educational institutions, and governmental offices.
One-Touch integrates video, voice, and data over any broadband
network with measurable and certifiable results, extending the
reach of communication and learning sessions to geographically
dispersed participants.
J.C. Penney became a customer of One-Touch System in 1996
after closing its management training facility in Texas. J.C.
Penney decided to use distance learning to train 149,000 store
associates so as to ensure consistency in its training. Store
associates see the facilitators’ video live on a classroom
television and interact by using the One Touch interactive touch
pad. Facilitators can pose questions and post quizzes, providing
an important means to measure and establish accountability
among associates for their learning. All participant data are
gathered in a central database for further analysis, including
attendance training records, the number of times an associate
calls in, and question and quiz performance. J.C. Penney saved
$58,000 in delivering one training class and reduced travel costs
for training by $12 million.
Built-In Evaluation Features
Sometimes “evaluation” features are built into a technology-
based performance intervention. A good example is a learning
management system (LMS). An LMS is a set of software tools

designed to manage user learning interventions. LMS has
become a popular method of technology-based application in
both the corporate and academic worlds. An LMS allows an
organization to manage users, roles, courses, instructors, and
facilities; post course calendars; post messages to learners;
offer assessment and testing, including administering
participant pre-tests and post-tests; display scores and
transcripts; grade course-work; and host web-based or blended
learning (see
http://en.wikipedia.org/wiki/learning_management_system).
Ilias is an example of an LMS that has a built-in evaluator tool
(see
www.ilias.de/docu/ilias.php?ref_id=392&frame=maincontent&o
bj_id=11674&cmd=layout&cmdClas). Ilias is a powerful web-
based learning management system that supports tools for
collaboration, communication, evaluation, and assessment. The
evaluation and assessment component of Ilias permits
evaluators to create assessments based on multiple choice,
single choice, allocation questions, cloze questions (free text,
select box) arrangement duties, hot spots (search images to
click on), and unsettled questions.
TOOLS PRACTITIONERS USE TO EVALUATE ONE-WAY
Webcasts, podcasts, and web-based surveys rely primarily on
one-way communication. A presenter broadcasts a message, or a
series of messages, that can only be received by participants
over the web, iPod, or other channels. Participants cannot
respond in real time, so they may properly be termed as forms
of one-way communication.
Podcasts and Webcasts
“A podcast is a digital audio program, a multimedia computer
file that can be downloaded to a computer, an iPod, or another
device, and then played or replayed on demand” (Islam, 2007, p.

5). A webcast is used on the Internet to broadcast live or
delayed audio and/or video transmissions, like traditional
television and radio broadcasts (see
www.webopedia,com/term/w/webcast.html). Unlike webcasts,
podcasts are always available, portable, user-friendly,
immediately useable, and inexpensive (Islam, 2007). Podcasts
and webcasts can be used in the evaluation phase of
performance improvement interventions. The intervention is
recorded and that allows multiple reviews to measure the
effectiveness of the intervention and assess its outcomes (Islam,
2007). Facilitators could also evaluate their training skills after
a training session so as to identify possible improvements for
future training sessions.
Web-Based Surveys
Web-based surveys are assessment tools designed by evaluators
and posted on the Internet for participants to evaluate the
technology-based application used to deliver the learning
intervention. Web-based surveys collect data through self-
administered electronic sets of questions on the web (Archer,
2003). Software tools for web-based surveys include, among
others, Survey Monkey (see www.surveymonkey.com) and
Zoomerang (see www.zoomerang.com).
According to Granello and Wheaton (2004) and Archer (2003),
web-based surveys can be administered easily by following
several important steps:
Determine the target population to be assessed and the purpose
of the survey;
Design the layout of the survey and format questions,
remembering to keep the survey short and simple or else risk
skewed answers;
Provide a welcome screen that is user friendly from the start
that explains the easiness of responding to questions with
instructions;

Post questions that have a consistent format and flow;
Test the system before launching it to make certain the glitches
are worked out;
Pilot test the survey with a small group similar to the targeted
audience; and
Strategically plan the survey distribution with only a few email
reminders.
TOOLS PRACTITIONERS USE TO EVALUATE TWO-WAY
Many new technologies facilitate the interaction of people in
real time. These technologies are called social networks.
Examples include group decision support systems, webinars,
web conferences, virtual classrooms, instant messages, chat
rooms, blogs, Facebook, YouTube, Twitter, wikipedias (wikis),
second-life virtual reality applications—and many others.
Social network websites are legion and their range encompasses
every imaginable topic. Examples include Facebook, MySpace,
Windows Live Spaces, Linkedln, and Broadcaster.com. HPT
practitioners may pose questions and then receive real -time—or
near-real-time—responses. The greatest promise for realtime
data collection and analysis probably exists with group decision
support systems in which live participants interact in real
time—or virtual participants interact concurrently—to reach
decisions. Participants can reach some level of agreement
collaboratively on:
What performance problems are important;
How important they are;
What causes them;
What solutions should be applied to solving those problems by
addressing root causes;
How those solutions can be implemented, and
How results may be evaluated.
As a result, those affected by the problems can have a stake in
diagnosing the problem, solving it, and establishing and

tracking relevant metrics (Rothwell, 2005).
HPT practitioners can pose questions, generate responses from
many people, and then facilitate group activities as member s
analyze their own results. Data collected from these activities
are useful for evaluating the process and the results of the group
decision support system intervention itself. HPT practitioners
using social networks can get many different views to evaluate
interventions. They can, for instance, benchmark how other
HPT practitioners have done evaluations for similar
interventions and share knowledge on the latest evaluation tools
and techniques.
ARE HPT PRACTITIONERS USING TECHNOLOGY-BASED
EVALUATION TOOLS?
The most common method of evaluation used by HPT
practitioners is a Level 1 on Donald Kirkpatrick’s four-level
evaluation model. Level 1 is the reaction level, and it focuses
on how much people liked the intervention. Participant
evaluations— also called “smile sheets”—remain the most
popular way to evaluate training, and most smile sheets are
administered using paper-and-pencil forms. Some large training
companies have advanced as far as using scannable sheets to
collect participant reactions. Some are even using web-assisted
survey tools.
Web-Assisted Survey Tools
Several organizations have gone the next step to use Survey
Monkey and similar web-assisted survey software packages to
administer participant evaluations. These organizations also
post results of participant evaluations online for participants
and managers to review.
For instance, the leaders of one company experiencing high
turnover asked human resources to explore the root cause of the
turnover. The human resources manager, using Survey Monkey,

surveyed middle managers and production supervisors. The
software made it easy to collect and analyze the data. The
human resources manager then fed the results of the survey back
to company leaders. Online survey-based methods are
particularly popular to use with distance education methods,
since online data collection just seems appropriate to use with
online training delivery.
Another organization has experimented with real-time data
collection using wireless personal digital assistants (PDAs). At
the end of a training program, participants are polled in real
time to determine their perceptions of the training. Their
reactions are compiled in front of them and projected on a
screen. They are then asked by the facilitator to offer immediate
ideas about how to improve the training for subsequent
delivery. This approach, of course, is really an application of a
group decision support system.
Beyond Level 1
Why do most HPT practitioners typically evaluate only
participant reactions and learning? What about evaluating HPT
performance interventions for transfer of skill and impact on the
organization? Few HPT practitioners conduct transfer of
learning studies and impact studies. They typically give several
reasons. Some are that such evaluation studies are more time-
consuming; evaluators lack resources and sometimes an
understanding of how to conduct sophisticated evaluation
studies using robust research designs; and organizational
leaders do not ask for evaluation studies.
But it is critically important that HPT practitioners conduct
evaluation studies at all levels of Kirkpatrick’s pyramid—
including transfer and impact studies— to build support from
organizational leaders (Kirkpatrick & Kirkpatrick, 2006;
Phillips & Phillips, 2005). Technology-based evaluation tools
can help HPT practitioners achieve this goal.

WHY USE TECHNOLOGY-BASED TOOLS IN
EVALUATION?
There are many advantages and disadvantages to HPT
practitioners using technology-based tools for evaluation. This
section will examine those advantages and disadvantages.
Advantages to Using Technology-Based Tools in Evaluation
The advantages of using technology-based tools in evaluation
include decreased cost, immediacy, and ease of use for the
respondee and the evaluator:
Cost. New technology offers the promise of reducing the cost of
collecting and analyzing data. Instead of paying for postage on
mail questionnaires, participants can use survey software at a
per-use cost. And it is less expensive for an organization to
email surveys to hundreds of people than to mail a hard copy of
a survey.
Immediacy. In most cases, the greatest appeal of new
technology when applied to evaluating performance
interventions is its immediacy—that is, how quickly it can be
collected and used. Gone are the days when lengthy analysis
could be painstakingly applied in step-by-step, linear project
approaches and then followed up by deliberate action. Time has
become the only strategic resource. Fast approaches to
addressing performance problems have grown more appealing to
time-pressured, stressed-out managers and workers alike. Many
of these approaches call for concurrent project management
approaches that combine steps or carry them out at the same
time.
Ease of Use. From the participants’ point of view, it is easier to
complete web-based surveys. The respondent does not have to
hassle mailing back a survey form. From the evaluator’s
standpoint, some survey software permits back-end statistical

analysis so that an entire data set can be recomputed every time
one user completes a survey. That can also reduce both the time
and cost to collect and analyze survey data.
Disadvantages to Using Technology-Based Tools in Evaluation
Of course, new technology is no panacea. It does introduce new
challenges. According to Lefever, Dal, and Matthiasdottir
(2007), the disadvantages of using online surveys including the
following:
Target population being surveyed may lack computer skills, not
accept using a computer, or may not have easy access to a
computer;
Technological problems could occur with the equipment or the
Internet service;
Emails sent to participants asking for their voluntary
participation to complete the survey may be routed to the junk
mail box, instead of the inbox, and participants may delete the
email from researchers without opening it;
Challenges may occur when attempting to get a representative
sample of the target population for the study by identifying
email addresses; and
It may be difficult to verify the identity of participants
completing the survey.
Verifying Participants. When using technology-based data
collection tools, evaluators often face problems in determining
whether the right people are actually responding (Do we really
know who logged in?), whether those completing online forms
actually completed instruction or participated in a performance
intervention, and whether data collected are kept secure and
confidential.
Security. Security is a major concern when using the Internet
for data collection. Virtual private networks (VPN) are a
common, popular vehicle used by researchers to ensure the
security of respondents completing data collection instruments,

although VPNs are by no means foolproof or hacker proof.
According to Wikipedia, virtual private networks (VPNs) are
computer networks in which some links between nodes are
carried by open connections or virtual circuits in some larger
network (for example, the Internet) instead of by physical wires.
The link-layer protocols of the virtual network are said to be
“tunneled” through the larger network. One common application
is to secure communications through the public Internet, but a
VPN need not have explicit security features, such as
authentication or content encryption. VPNs, for example, can be
used to separate the traffic of different user communiti es over
an underlying, secure network (see
http://en.wikipedia.org/wiki/virtual_private_network).
Overcoming the Disadvantages
With these new problems come new opportunities. Here are
some suggestions:
Security. In addition to using a VPN, HPT practitioners may
select programming language based on security needs. Use
JavaScript to validate responses given by participants in online
surveys. JavaScript is a programming language that alerts
respondents participating in online studies if the fields were
completed incorrectly or if fields were left blank during
submission (White, Carey, & Dailey, 2000). Respondents
receive error messages to fix the issue(s) in the survey before
the final survey submission can be made. Perl is another
programming language that can avoid both the cost and the
potential for error incurred by the tedious data processing. Perl
automatically deletes field names and other extraneous
characters; transforms the various data fields into a single
subject record; aggregates the subject records into single files
maintained on the server; and performs any required algorithmic
data re-coding before the researcher sees it (White, Carey, &
Dailey, 2000).

Identifying Participants. One way to make sure that the intended
participants are responding is to ask the participants to provide
information that only would be able to identify—such as a
password, an encrypted code, or an email address).
WHAT DOES THE FUTURE HOLD IN STORE?
The future holds exciting possibilities for applying emerging
technology to evaluating performance interventions. Here are a
few examples for the immediate future and beyond.
Time Travel with Kirkpatrick . . .
It may be necessary to rethink Kirkpatrick’s well-known four
levels of evaluation to include a consideration of time. The
three time points are before the intervention, during the
intervention, and after the intervention. (See Figure 17.1.)
Kirkpatrick’s four levels are, of course, quite well known,
although often criticized (see Holton, 1996). Level 1 focuses on
reactions, addressing how much people liked the training or
other performance interventions in which they participated.
Level 2 focuses on learning, addressing how much people
learned from training or from other performance interventions
in which they participated. Level 3 focuses on behavior,
addressing how much people changed their on-the-job behavior
based on the performance intervention. Level 4 focuses on
results, addressing how much the organization gained in
productivity improvement as a result of the performance
intervention. But each level may also be examined at three
points in time—before, during, and after the performance
intervention.
Adding the time element transforms the “hierarchy” into a
“grid.” Each cell of the grid suggests a range of different
approaches and focal points of concern. For instance, reaction
viewed before the performance intervention focuses on what
expectations the participants targeted for change have about the

intervention; reaction viewed during the performance
intervention focuses on monitoring changing attitudes about the
intervention while the intervention is implemented; and reaction
viewed after the performance intervention measures how much
people liked the change effort upon completion. Each cell of the
grid may prompt HPT practitioners to think of several ways by
which to measure at each point in time. New, emerging
technology may support evaluation in each cell of the grid.
Possible ideas about doing that are summarized in Table 17.1.
Of course, creative HPT practitioners may come up with
additional ideas about ways to apply new technology in
conducting evaluation. Use the worksheet in Exhibit 17.1 to
stimulate thinking about other ways to evaluate performance
interventions on different levels of the hierarchy and at
different points in time.
Getting Help from Avatars
Futurists already predict a time when avatars, representations of
people often used in second-life applications, will be guided by
artificial intelligence and not just by people (see Hill, 2008).
That will mean that second-life avatars may be able to interact
with people in simulated ways, giving them opportunities to
rehearse actions in real life that might not otherwise be possible
due to safety or other concerns. If that happens, then it may be
possible to simulate efforts to collect, analyze, and evaluate
data about instruction—or other performance interventions—
before the change effort is implemented. It would go the step
beyond mere guesswork forecasting to actual, robust, and
simulated pilot-testing of interventions assisted by new
technology.
Getting on the Grid
Futurists predict that the Internet and the web will be
transformed into the grid in which people, objects, and data can
be linked together as “nodes.” That will speed up efforts to

achieve results by seamlessly integrating interactions between
humans and technology. It will mean that assistive technology,
now associated with technology that helps the disabled to
perform like the able-bodied, may become associated with the
means to integrate technology seamlessly with human efforts to
perform work (Foroohar, 2002).
Table 17.1 Possible Ways That Emerging Technologies May Be
Used to Evaluate Performance Technology Interventions
example, eyeglasses that permit users to see while also
permitting them to access real-time Internet and video (see
“Newer design of close-up computer monitors increases ease of
use,” 2002). This could permit real-time, task-specific coaching
on how to perform work. It could also give HPT practitioners
the ability to analyze data even as they collect it, thereby
slashing the time it takes to make data usable.
Another example: cell phones that do everything from access
the Internet to permit instant messaging. Those cell phones are
already with us, but their applications in collecting, analyzing,
and evaluating real-time performance information is only now
being considered. Cell-phone-based training and wristwatch-
based training, perhaps in three dimensions, may be the next
wave after podcasting, and that opens up the potential to give
learners realtime performance assistance and give HPT
practitioners the capability to do realtime data collection,
analysis, and evaluation.
What Ethical and Confidentiality Issues Arise When Using New
Technology to Evaluate Performance Interventions?
With new forms of technology arise new challenges in ethics
and confidentiality. For example, there are issues with almost

every method of online intervention technology—especially
blogs, wikipedias, Facebook, YouTube, Twitter, seminars, and
web conferences. Blogs, wikipedias, Facebook, YouTube, and
Twitter are technological avenues (social networks) where it is
easy to be unethical and violate copyright because the
individuals posting information need not identify themselves
and are not held accountable or mandated to publish the source.
Just Google “media and confidentiality” to find instances in
which clients have been specifically discussed by healthcare
professionals.
Blogs. Some organizations are beginning to encourage employee
groups to create blogs. Yet, blogs can also prompt many ethical,
confidentiality, and copyright dilemmas. Blogs are a highly
popular method to communicate by media. There are a multitude
of blogs that are not controlled and that permit individuals to
display text, images, audio files, and video available on the
Internet without permission from the original creator or
copyright holder (Kuhn, 2007). Currently, no code of ethics
exists for bloggers, and so blogging on the web can be as
lawless as the Old West. However, if a code of ethics were to be
established, accountability and anonymity should be two key
critical components. Bloggers should be accountable for the
information that they post to give proper credit when due and
should preserve anonymity when posting information (Kuhn,
2007). Evaluation of blogging as a performance intervention
would require the evaluator to identify and evaluate the ethical
issues as well as the performance results of the blogging
intervention.
Digital Media. The U. S. government has considered issues of
ethics and digital media. This has prompted the government to
legislate a few federal acts in the hopes of upholding ethical
standards. A few of these acts are as follows:
Audio Home Recording Act of 1992 (Pub.L.No-102-563):

Exempts from copyright infringement the making of copies for
personal use of music files in digital form, provided that those
copies are made with approved equipment.
No Electronic Theft Act of 1997 (Pub. L. No. 105-147):
Establishes penalties for unauthorized sharing and redistribution
of copyrighted material over the Internet.
Digital Millennium Copyright Act of 1998 (Pub. L. No. 105-
304): Creates several protection and enforcement mechanisms
for copyrighted works in digital form.
Copyright. HPT practitioners should be aware of resources
available from the U.S. Copyright Office (see
www.copyright.gov). This link provides information about
copyright issues. It also explains how to copyright a work or
record a document and reviews applicable copyright laws and
policies. If there is uncertainty or questions of issues with
copyright, HPT practitioners should refer to this site for
additional information. They may also consult The Congress of
the United States Congressional Budget Office (2004).
The federal laws listed in this chapter are based on copyright
issues to protect individuals’ work that is displayed on the
Internet. It is very important that HPT practitioners follow the
federal laws when conducting technology-based evaluations. A
significant case would be if a practitioner is using an evaluation
assessment tool that he or she found online and the practitioner
feels that, since the assessment tool was found online, there is
no need to obtain copyright permission for subsequent use. This
is a common misconception. Whether evaluation material is
obtained online or off-line, proper permission must be obtained
from the copyright holder.
Use of Information. One dilemma for HPT practitioners is how
to use information on ethics and confidentiality when selecting
the appropriate evaluation tool for their needs. Practitioners
must know how utilizing the information obtained from the data
collection methods may impact the participants in regard to

ethics and confidentiality. For example, as with any data
collection method to be selected for a study, HPT practitioners
have to first gauge the conditions under which the data will be
collected, the population to be sampled, and then the most
appropriate method that will yield the most desirable results—
while also protecting confidentiality and well-being of human
subjects. Internal Review Boards (IRBs) are helpful to HPT
practitioners in the sense that IRBs can help practitioners
remain ethical and preserve confidentiality as they collect data.
The IRBs will also educate practitioners on legal issues that can
arise during data collection. Many federal government
guidelines exist to protect human subjects when data are
collected.
SUMMARY
This chapter listed some examples of technology-based
evaluation tools that could be used to evaluate performance
interventions. Each new technology holds the promise of
providing new, exciting ways to select or create data collection
tools and to collect, store, retrieve, analyze, and communicate
data in faster, more effective ways. Each new technology may
also challenge HPT practitioners to come up with new ways to
think about evaluating performance interventions. Indeed, it
may be necessary to reinvent Kirkpatrick’s famous hierarchy
into a grid that addresses times before, during, and after
performance interventions.
The future is likely to offer even more opportunities to apply
technology to how HPT practitioners evaluate. Just a few
exciting possibilities include avatars that “think”; the
emergence of the grid to replace the web and the Internet ; and
human-machine interactions that are seamlessly integrated in
real time.
It is critically important to obey the appropriate copyright laws
when using evaluation tools owned by others. Following the

copyright laws means obtaining permission from copyright
holders to use their work in your evaluation studies. Equally
important, it is the duty of HPT practitioners to abide by all
ethical standards and confidentiality laws of the U.S.
government.
References
Archer, T. M. (2003, August). Web-based surveys. Retrieved
July 7, 2008, from www.joe.org/joe/2003august/tt6.shtml.
Foroohar, R. (2002). Life in the grid. Retrieved July 7, 2008,
from www.newsweek.com/id/65626.
Week 6 Discussion 1 Evaluation and Measurement in Not-for-
Profit Sectors

Mosele, J. & Dessinger, J. (2009). Handbook of Improving
Performance in the Workplace. (Volume 3). Pfeiffer-Wiley
To prepare for this Discussion, pay particular attention to the
following Learning Resources:
· Review this week’s Learning Resources, especially:
· Read Week 6 Lecture – See Word doc.
· Read Chapter 16-17 – See Word doc
Assignment:
This week you had the opportunity to learn about evaluation
within the workplace at a level where results affect the outcome
of the company’s success.
· Using your current workplace as an example.
· Justify the need for an evaluation process within the
workplace.
· Your discussion should analyze the tools that are currently
being used within the workplace to evaluate processes and
employees.
· Distinguish which tools you have learned about that could be
recommended as a positive addition to current evaluation
process.
· 3 – 4 paragraphs
· No plagiarism
· APA citing

Evaluating Results and Benefits Week #6 Lecture 1The Impact of

Evaluating Results and Benefits Week #6 Lecture 1The Impact of

More Related Content

Similar to Evaluating Results and Benefits Week #6 Lecture 1The Impact of (20)

More from BetseyCalderon89 (20)

Recently uploaded (20)

Evaluating Results and Benefits Week #6 Lecture 1The Impact of