Page 291LEARNING OBJECTIVES· Discuss the issues created by.docx

LEARNING OBJECTIVES
· Discuss the issues created by generalizing research results to
other populations, including potential problems using college
students as research participants.
· Discuss issues to consider regarding generalization of research
results to other cultures and ethnic groups.
· Describe the potential problem of generalizing to other
experimenters and suggest possible solutions.
· Discuss the importance of replications, distinguishing between
exact replications and conceptual replications.
· Distinguish between narrative literature reviews and meta-
analyses.
Page 292IN THIS CHAPTER, WE WILL CONSIDER THE
ISSUE OF GENERALIZATION OF RESEARCH
FINDINGS. When a single study is conducted with a particular
sample and procedure, can the results then be generalized to
other populations of research participants, or to other ways of
manipulating or measuring the variables? Recall from Chapter
4 that internal validity refers to the ability to infer that there is
a causal relationship between variables. External validity is the
extent to which findings may be generalized.
GENERALIZING TO OTHER POPULATIONS
Even though a researcher may randomly assign participants to
experimental conditions, rarely are participants randomly
selected from the general population. As we noted in Chapters
7 and 9, the individuals who participate in psychological
research are usually selected because they are available, and the
most available population consists of college students—or more
specifically, first- and second-year students enrolled in the
introductory psychology course to satisfy a general education
requirement. They may also be from a particular college or

university, may be volunteers, or may be mostly males or
mostly females. So, are our research findings limited to these
types of subjects, or can we generalize our findings to a more
general population? After considering these issues, we will
examine the larger issue of culture and how research findings
can be generalized to different cultural groups.
College Students
Smart (1966) found that college students were studied in over
70% of the articles published between 1962 and 1964 in
the Journal of Experimental Psychology and the Journal of
Abnormal and Social Psychology. Sears (1986) reported similar
percentages in 1980 and 1985 in a variety of social psychology
journals; Arnett (2008) found that 67% of the articles in the
2007 volume of the Journal of Personality and Social
Psychology used college student samples. The potential problem
is that such studies use a highly restricted population. Sears
points out that most of the students are first-year students and
sophomores taking the introductory psychology class. They
therefore tend to be young and to possess the characteristics of
emerging adults: a sense of self-identity that is still developing,
social and political attitudes that are in a state of flux, a high
need for peer approval, and unstable peer relationships. They
are intelligent with high cognitive abilities. Thus, what we
know about “people in general” may actually be limited to a
highly select and unusual group. Indeed, Peterson (2001) found
that students, as a group, are more homogenous than nonstudent
samples. That is, students are more similar to each other than
adults are similar to other adults in the general population.
Research by Henry (2008) illustrates how the use of college
students may affect the external validity of research on
prejudice. In his sample of articles Page 293from 1990 to 2005,
an increasing percentage of studies used college students as
participants. Further, in looking at the actual results of studies
on prejudice that compared college students with adults, he
reported a variety of differences among adults and college
students. For example, college students were less conservative

and rated women and ethnic minorities more favorably.
Volunteers
Researchers usually must ask people to volunteer to participate
in their research. At many colleges, introductory psychology
students are required either to volunteer for research or to
complete an alternative project. If you are studying populations
other than college students, you are even more dependent on
volunteers—for example, asking people at a homeowners'
association meeting to participate in a study of marital
interaction or conducting research on the Internet in which
people must go to your web page and then agree to participate
in the study, or conducting a telephone survey of county
residents to determine health care needs. In all these cases,
external validity of the findings may be limited because the data
from volunteers may be different from what would be obtained
with a more general sample. Some research indicates that
volunteers differ in various ways from nonvolunteers. In their
comprehensive study on the topic, Rosenthal and Rosnow
(1975) reported that volunteers tend to be more highly educated,
of a higher socioeconomic status, more in need of approval, and
more social.
Further, different kinds of people volunteer for different kinds
of experiments. In colleges, there may be a sign-up board with
the titles of many studies listed or a web page that manages
research participants and volunteer opportunities for the
university. Different types of people may be drawn to the study
titled “problem solving” than to the one titled “interaction in
small groups.” Available evidence indicates that the title does
influence who signs up (Hood & Back, 1971; Silverman &
Margulis, 1973).
Online Research
Another important consideration arises when asking participants
to volunteer for online surveys and experiments. Researchers
can find potential participants through online survey design
services. Psychologists are increasingly using Amazon
Mechanical Turk (https://www.mturk.com; Jacquet, 2011), a

website for recruiting people to work on many types of tasks
including participating in research for a specified payment. This
sort of sampling strategy has important implications for external
validity. While the online sample is more diverse than the
typical college student sample, there are still generalization
issues because Internet users represent a unique demographic.
The Pew Research Center's Internet and American Life Project
(Pew Internet, 2010) found that living in an urban/suburban
area, being college educated, being younger, and having a
higher income are all related to reporting more time online.
Thus, by asking Page 294for volunteers for an online survey,
researchers are sampling from a particular demographic that
may not generalize well to the population of interest.
Gender
Sometimes, researchers use only males or only females (or a
very disproportionate ratio of males to females) simply because
this is convenient or the procedures seem better suited to a
particular gender. Given the possible differences between males
and females, however, the results of such studies may not be
generalizable (Denmark, Russo, Frieze, & Sechzer, 1988).
Denmark et al. provide an example of studies on contraception
practices that use only females because of stereotypical
assumptions that only females are responsible for contraception.
They also point out several other ways that gender bias may
arise in psychological research, including confounding gender
with age or job status and selecting response measures that are
gender-stereotyped. The solution is to be aware of possible
gender differences and include both males and females in our
research investigations. Moreover, it is important to recognize
the ways that males and females might differentially interpret
independent variable manipulations or questions asked in a
questionnaire.
Locale
The location that participants are recruited from can also have
an impact on a study's external validity. Participants in one
locale may differ from participants in another locale. For

example, students at UCLA may differ from students at a nearby
state university, who in turn may differ from students at a
community college. People in Iowa may differ from people in
New York City. Thus, a finding obtained with the students in
one type of educational setting or in one geographic region may
not generalize to people in other settings or regions. In fact,
studies have explored how personality traits like extraversion
(the tendency to seek social stimulation) and openness to new
experiences vary across geographic areas. Rentfrow, Gosling,
and Potter (2008) looked at geographic differences in
personality traits among citizens of various U.S. states and
found extraversion to vary by state. People in midwestern states
tended to be more extraverted than people in northeastern
states, and people in western states tended to be more open to
new experiences. Thus, a study conducted in one location may
not generalize well to another, particularly if the variables in
question are related to location in some way.
Culture
Whether theories and research findings generalize across
cultures is a critically important issue. Some observers of
current psychological research have been very critical of the
types of samples employed in behavioral research. Based on
analyses of published research by Arnett (2008) and others,
Henrich, Heine, and Norenzayan (2010) contend that
psychology is built on the study of WEIRD Page 295(Western,
Educated, Industrialized, Rich, Democratic) people. In many
cases, research samples consist primarily of college students
from the United States, other English-speaking countries, and
Europe. Ultimately, researchers wish to discover aspects of
human behavior that have universal applications but in fact
cannot generalize beyond their limited samples. This is, at its
heart, a critique of the external validity of behavioral research:
Does our human behavioral research generalize to all humans,
or is it really a study of the WEIRD?
Clearly, if psychologists want to understand human behavior,
they must understand human behavior across and among

cultures (Henrich et al., 2010; Miller, 1999). Miller described
research on self-concept by Kitayama, Markus, Matsumoto, and
Norasakkunkit (1997) to illustrate the benefits of incorporating
culture into psychological theory. Traditional theories of self-
concept are grounded in the culture of the United States and
Western Europe; the “self” is an individualistic concept where
people are independent from others and self-enhancement comes
from individual achievements. Kitayama and his colleagues take
a broader, cultural perspective: In contrast to the U.S. meaning
of self, in other cultures the “self” is a collective concept in
which self-esteem is derived from relationships with others.
Often, Japanese engage in self-criticism, which can be seen as
relationship-maintaining, whereas Americans work to maintain
and enhance self-esteem—thus, very different activities
contribute to a positive self-concept in the two cultures
(Kitayama et al., 1997). This is a very common theme in
research that incorporates culture in psychological processes:
“The significance of self-esteem, however, may be much more
specific to a culture than has typically been supposed in the
literature” (p. 1262).
Much of this cultural research centers on identifying similarities
and differences that may exist in personality and other
psychological characteristics, as well as ways that individuals
from different cultures respond to the same environments
(Matsumoto, 1994). Research by Kim, Sherman, and Taylor
(2008) provides another example of the limits of external
validity across cultural groups. This research focused on how
people from different cultures use social support to cope with
stress. In reviewing the research on the topic, they concluded
that Asians and Asian Americans might benefit from different
styles of social support as compared with European Americans.
For example, Asian Americans are more likely to benefit from
support that does not involve the sort of intense disclosure of
personal stressful events and feelings that is the hallmark of
support in many European American groups. Rather, they
suggest that Asians and Asian Americans may benefit more

from support that comes with the comforts of proximity (being
with close friends) rather than sharing.
These examples all focused on differences among cultures.
Many studies also find similarities across cultures. Evolutionary
psychologists, for instance, often conduct studies in different
cultural groups because they are looking for similarities across
cultures in order to see if a particular behavior or attitude can
be tied to our evolutionary past. For example, Singh, Dixson,
Jessop, Morgan, and Dixson (2010) wanted to see if a particular
aspect of beauty that is tied to greater reproductive success—
namely waist-to-hip ratio (e.g., the ratio for Page 296a 25-inch
waist and 35-inch hips is .71), which is related to sex hormones
and thus fertility—would be seen as attractive across cultures.
Diverse groups from Africa, Samoa, Indonesia, and New
Zealand evaluated photographs of females with small and large
waist-to-hip ratios. The researchers found that indeed, low
waist-to-hip ratio among females was seen as more attractive
across all these groups. In this example, the results obtained in
one culture do generalize to other cultures.
Nonhuman Animals
We noted in Chapter 3 that about 7% of psychological research
is conducted with nonhuman animals. Almost all of this
research is done with rats, mice, and birds. Most research with
other species is conducted to study the behavior of those
animals directly to gather information that may help with the
survival of endangered species and increase our understanding
of our bonds with nonhuman animals such as dogs, cats, and
horses (http://www.apa-hai.org/human-animal-interaction).
The basic research that psychologists conduct with nonhuman
animals is usually done with the expectation that the findings
can be generalized to humans. This research is important
because the research problems that are addressed require
procedures such as long-term observation that could not be done
with human samples. We do expect that we can generalize as
our underlying biological and behavioral patterns are shared. In
fact, the value of studying nonhuman animals has been

demonstrated by research that does apply to humans. These
applications include the biological bases of memory, food
preferences, sexual behavior, choice behavior, and drug
addictions. The American Psychological Association has
prepared a brochure on animal research:
(http://www.apa.org/research/responsible/research-animals.pdf).
In Defense of College Students
It is easy to criticize research on the basis of subject
characteristics, yet criticism by itself does not mean that results
cannot be generalized. Although we need to be concerned about
the potential problems of generalizing from unique populations
such as college students (cf. Sears, 1986), we should also keep
several things in mind when thinking about this issue. First,
criticisms of the use of any particular type of subject, such as
college students, in a study should be backed with good reasons
that a relationship would not be found with other types of
subjects. College students, after all, are human, and researchers
should not be blamed for not worrying about generalization to a
particular type of subject if there is no good reason to do so.
Moreover, college student bodies are increasingly diverse and
increasingly representative of the society as a whole (although
college students will always be characterized as having the
ability and motivation to pursue a college degree). Second,
replication of research studies provides a safeguard against the
limited external validity of a single study. Studies are replicated
at other colleges using different mixes of students, and Page
297many findings first established with college students are
later applied to other populations, such as children, aging
adults, and people in other countries. It is also worth noting that
Internet samples are increasingly used in many types of studies.
Although such studies raise their own issues of external
validity, they frequently complement studies based on college
student samples.
GENERALIZING ACROSS METHODS
The person who actually conducts the experiment is the source
of another external validity problem. In most research, only one

experimenter is used, and rarely is much attention paid to the
personal characteristics of the experimenter (McGuigan, 1963).
The main goal is to make sure that any influence the
experimenter has on subjects is constant throughout the
experiment. There is always the possibility, however, that the
results are generalizable only to certain types of experimenters.
Some of the important characteristics of experimenters have
been discussed by Kintz and his colleagues (Kintz, Delprato,
Mettee, Persons, & Schappe, 1965). These include the
experimenter's personality and gender and the amount of
practice in the role of experimenter. A warm, friendly
experimenter will almost certainly produce different results
from a cold, unfriendly experimenter. Participants also may
behave differently with male and female experimenters. It has
even been shown that rabbits learn faster when trained by
experienced experimenters (Brogden, 1962)! The influence of
the experimenter may depend as well on the characteristics of
the participants. For example, participants seem to perform
better when tested by an experimenter of the other sex
(Stevenson & Allen, 1964).
One solution to the problem of generalizing to other
experimenters is to use two or more experimenters. A fine
example of the use of multiple experimenters is a study by
Rubin (1975), who sent several male and female experimenters
to the Boston airport to investigate self-disclosure. The
experimenters revealed different kinds of information about
themselves to both male and female travelers and recorded the
passengers' self-disclosures in return. One interesting result was
that women tended to reveal more about themselves to male
experimenters, and men tended to reveal more about themselves
to female experimenters.
Pretests and Generalization
Researchers are often faced with the decision of whether to give
a pretest. Intuitively, pretesting seems to be a good idea. The
researcher can be sure that the groups are equivalent on the
pretest, and it is often more satisfying to see that individuals

changed their scores than it is to look only at group means on a
posttest. A pretest also enables the researcher to assess
mortality (attrition) effects when it is likely that some
participants will withdraw from an experiment. Page 298If you
give a pretest, you can determine whether the people who
withdrew are different from those who completed the study.
Pretesting, however, may limit the ability to generalize to
populations that did not receive a pretest. (cf. Lana, 1969).
Simply taking the pretest may cause subjects to behave
differently than they would without the pretest. Recall
from Chapter 8 that a Solomon four-group design (Solomon,
1949) can be used in situations in which a pretest is desirable
but there is concern over the possible impact of taking the
pretest. In the Solomon four-group design, half of the
participants are given the pretest; the other half receive the
posttest only. That is, the same experiment is conducted with
and without the pretest. Mortality effects can be assessed in the
pretest conditions. Also, the researcher can examine whether
there is an interaction between the independent variable and the
pretest: Are posttest scores on the dependent variable different
depending on whether the pretest was given? Sometimes,
researchers find that it is not feasible to conduct the study with
all four groups in a single experiment. In this case, the first
study can include the pretest; the study can be replicated later
without the pretest.
Generalizing from Laboratory Settings
Research conducted in a laboratory setting has the advantage of
allowing the experimenter to study the impact of independent
variables under highly controlled conditions. The internal
validity of the research is the primary consideration. The
question arises, however, as to whether the artificiality of the
laboratory setting limits the ability to generalize what is
observed in the laboratory to real-life settings.
Mook (1983) articulated one response to the artificiality issue:
Generalization to real-life settings is not relevant when the
purpose of the study was to investigate causal relationships

under carefully controlled conditions. Mook is concerned that a
“knee-jerk” criticism of laboratory research on the basis of
external validity is too common. Good research is what is most
important.
Another response to the laboratory artificiality criticism is to
examine the results of field experiments. Recall from Chapter
4 that in a field experiment, the researcher manipulates the
independent variable in a natural setting—a factory, a school, or
a street corner, for example.
Anderson, Lindsay, and Bushman (1999) asked whether
laboratory and field experiments that examine the same
variables do in fact produce the same results. To answer this
question, they found 38 pairs of studies for which a laboratory
investigation had a field experiment counterpart. The studies
were drawn from a variety of research areas including
aggression, helping, memory, leadership style, and depression.
Results of the laboratory and field experiments were in fact
very similar—the effect size of the independent variable on the
dependent variable was very similar in the two types of studies.
Thus, even though lab and field experiments are conducted in
different settings, the results Page 299are complementary rather
than contradictory. When findings are replicated using multiple
methods, our confidence in the external validity of the findings
increases.
SUPPORTING GOOD EXTERNAL VALIDITY
It may seem as if no research can possibly be generalizable! In
some ways, this is true. Furthermore, it can be very difficult to
understand the extent to which a study is generalizable; external
validity is an aspect of a study that we try to assess, but cannot
truly know. How, then, can we support good external validity?
There are a few ways that external validity can be supported.
The key way that external validity can be supported is related to
a study's methodology. Using a census, or a random sample will
always produce better external validity than using a nonrandom
sample. This, of course, is not always possible. Next, we will
explore a few other ways in which external validity can be

supported.
Generalization as a Statistical Interaction
The problem of generalization can be thought of as an
interaction in a factorial design (see Chapter 10). An interaction
occurs when a relationship between variables exists under one
condition but not another or when the nature of the relationship
is different in one condition than in another. Thus, if you
question the generalizability of a study that used only males,
you are suggesting that there is an interaction between gender
and the independent variable. Suppose, for example, that a
study examines the relationship between crowding and
aggression among males and reports that crowding is associated
with higher levels of aggression. You might then question
whether the results are generalizable to females.
Figure 14.1 shows four potential outcomes of a hypothetical
study on crowding and aggression that tested both males and
females. In each graph, the relationship between crowding and
aggression for males has been maintained. In Graph A, there is
no interaction—the behavior of males and females is virtually
identical. Thus, the results of the original all-male study could
be generalized to females. In Graph B, there is also no
interaction; the effect of crowding is identical for males and
females. However, in this graph, males are more aggressive than
females. Although such a difference is interesting, it is not a
factor in generalization because the overall relationship between
crowding and aggression is present for both males and females.
Graphs C and D do show interactions. In both, the original
results with males cannot be generalized to females. In Graph
C, there is no relationship between crowding and aggression for
females. In Graph D, the interaction tells us that a positive
relationship between crowding and aggression exists for males
but that a negative relationship exists for females. As it turns
out, Graph D describes the results of several studies on this
topic (cf. Freedman, Levy, Buchanan, & Price, 1972).
Page 300

FIGURE 14.1
Outcomes of a hypothetical experiment on crowding and
aggression
Note: The presence of an interaction indicates that the results
for males cannot be generalized to females.
Researchers can address issues of external validity that stem
from the use of different populations by including subject type
as a variable in the study. By including variables such as
gender, age, or ethnic group in the design of the study, the
results may be analyzed to determine whether there are
interaction effects like the ones illustrated in Figure 14.1.
The Importance of Replications
Replication of research is a way of overcoming any problems of
generalization that occur in a single study. There are two types
of replications to consider: exact replications and conceptual
replications.
Exact replications An exact replication is an attempt to replicate
precisely the procedures of a study to see whether the same
results are obtained. A researcher who obtains an unexpected
finding will frequently attempt Page 301a replication to make
sure that the finding is reliable. If you are starting your own
work on a problem, you may try to replicate a crucial study to
make sure that you understand the procedures and can obtain the
same results. Often, exact replications occur when a researcher
builds on the findings of a prior study. For example, suppose
you are intrigued by Singh et al.'s (2010) research on waist-to-
hip ratio that was mentioned previously. Singh reports that
males rate females with a ratio of .70 as most attractive. In your
research, you might replicate the procedures used in the original
study and expand on the original research. For example, you
might study this phenomenon with males similar to those in the
original sample as well as males from different cultures or age
groups. When you replicate the original research findings using
very similar procedures, your confidence in the external validity
of the original findings is increased.
The “Mozart effect” provides us with an interesting example of

the importance of replications. In the original study by
Rauscher, Shaw, and Ky (1993), college students listened to 10
minutes of a Mozart sonata. These students then showed better
performance on a spatial-reasoning measure drawn from the
Stanford-Binet Intelligence Scale than students exposed to a
relaxation tape or simple silence. This finding received a great
deal of attention in the press as people quickly generalized it to
the possibility of increasing children's intelligence with Mozart
sonatas. In fact, one state governor began producing Mozart
CDs to distribute in maternity wards, and entrepreneurs began
selling Mozart kits to parents over the Internet. Over the next
few years, however, there were many failures to replicate the
Mozart effect (see Steele, Bass, & Crook, 1999). We noted
above that failures to replicate may occur because the exact
conditions for producing the effect were not used. In this case,
Rauscher and Shaw (1998) responded to the many replication
failures by precisely describing the conditions necessary to
produce the Mozart effect. However, Steele et al. (1999) and
McCutcheon (2000) were unable to obtain the effect even
though they followed the recommendations of Rauscher and
Shaw. Research on the Mozart effect continues. Some recent
findings suggest that the effect is limited to music that also
increases arousal; it is this arousal that can cause better
performance following exposure to the music (Thompson,
Schellenberg, & Husain, 2001). Bangerter and Heath (2004)
present a detailed analysis of the development of the research
on the Mozart effect.
A single failure to replicate does not reveal much, though; it is
unrealistic to assume, on the basis of a single failure to
replicate, that the previous research is necessarily invalid.
Failures to replicate share the same problems as nonsignificant
results, discussed in Chapter 13. A failure to replicate could
mean that the original results are invalid, but it could also mean
that the replication attempt was flawed. For example, if the
replication is based on the procedure as reported in a journal
article, it is possible that the article omitted an important aspect

of the procedure. For this reason, it is usually a good idea to
write to the researcher to obtain detailed information on all of
the materials that were used in the study.
Page 302Several scientific societies are encouraging systematic
replications of important scientific findings. The
journal Perspectives on Psychological Science (published by the
Association for Psychological Science) is sponsoring the
publication of Registered Research Replications
(http://www.psychologicalscience.org/index.php/replication).
Multiple groups of researchers will undertake replications of
important studies using procedures that are made public before
initiating the research. When completed, all of the replications
will be described in a single report. In addition to the
Psychological Science initiative, the online journal PLOS
ONE (Public Library of Science) has developed the
Reproducibility Initiative to encourage independent replication
of research in the clinical sciences (Pattinson, 2012). Such
developments should lead to greater understanding of the
generalizability of research findings.
Conceptual replications A conceptual replication is the use of
different procedures to replicate a research finding. In a
conceptual replication, researchers attempt to understand the
relationships among abstract conceptual variables by using new,
or different, operational definitions of those variables.
Conceptual replications are even more important than exact
replications in furthering our understanding of behavior.
In most research, a key goal is to discover whether a
relationship between conceptual variables exists. In the original
Mozart effect study, researchers examined the effect
of exposure to classical music on spatial reasoning. These are
conceptual variables; in the actual study, specific operational
definitions of the variables were used. Exposure to classical
music was operationalized as 10 minutes of exposure to the
Mozart Sonata for Two Pianos in D Major. Spatial
reasoning was operationalized as performance on a particular
spatial reasoning measure.

In a conceptual replication, the same independent variable is
operationalized in a different way, and the dependent variable
may be measured in a different way, too. Conceptual
replications are extremely important in the social sciences
because the variables used are complex and can be
operationalized in different ways. Complete understanding of
any variable involves studying the variable using a variety of
operational definitions. A crucial generalization question is
whether the relationship holds when other ways of manipulating
or measuring the variables are studied. Sometimes the
conceptual replication may involve an alternative stimulus (e.g.,
a different Mozart sonata, a selection by a different composer)
or an alternative dependent measure (e.g., a different spatial-
reasoning task). Or as we previously noted, the same variables
are sometimes studied in both laboratory and field settings.
When conceptual replications produce similar results, our
confidence in the generalizability of relationships between
variables is greatly increased.
This discussion should also alert you to an important way of
thinking about research findings. The findings represent
relationships between conceptual variables but are grounded in
specific operations. You may read about the specific methods
employed in a study conducted 20 years ago and question Page
303whether the study could be replicated today. You might also
speculate that the methods used in a study are so unusual that
they could never generalize to other situations. These concerns
are not as serious when placed within the context of conceptual
replications because, although operational definitions can
change over time, the underlying conceptual variable often
remains more consistent. Admittedly, a specific method from a
study conducted at one time might not be effective today, given
changes in today's political and cultural climate. A conceptual
replication of the manipulation, however, would demonstrate
that the relationship between the conceptual theoretical
variables is still present. Similarly, the narrow focus of a
particular study is less problematic if the general finding is

replicated with different procedures.
Evaluating Generalizations via Literature Reviews and Meta-
analyses
Researchers have traditionally drawn conclusions about the
external validity of research findings by conducting literature
reviews. In a literature review, a reviewer reads a number of
studies that address a particular topic and then writes a paper
that summarizes and evaluates the literature. The Publication
Manual of the American Psychological Association provides the
following description: “Literature reviews, including research
syntheses and meta-analyses, are critical evaluations of material
that has already been published.… By organizing, integrating,
and evaluating previously published material, authors of
literature reviews consider the progress of research toward
clarifying a problem” (APA, 2010, p. 10). The literature review
provides information that (1) summarizes what has been found,
(2) tells the reader which findings are strongly supported and
which are only weakly supported in the literature, (3) points out
inconsistent findings and areas in which research is lacking, and
(4) discusses future directions for research.
Sometimes a review will be a narrative in which the author
provides descriptions of research findings and draws
conclusions about the literature. The conclusions in a narrative
literature review are based on the subjective impressions of the
reviewer. Another technique for comparing a large number of
studies in an area is meta-analysis (Borenstein, Hedges,
Higgins, & Rothstein, 2009). In a meta-analysis, the researcher
combines the actual results of a number of studies. The analysis
consists of a set of statistical procedures that employ effect
sizes to compare a given finding across many different studies.
Instead of relying on judgments obtained in a narrative
literature review, you can draw statistical conclusions from this
material. The statistical procedures need not concern you. They
involve examining several features of the results of studies,
including the effect sizes and significance levels obtained. The
important point here is that meta-analysis is a method for

determining the reliability of a finding by examining the results
from many different studies.
Stewart and Chambless (2009) conducted a meta-analysis of
research on the effectiveness of cognitive-behavioral therapy
(CBT) for anxiety disorders. Page 304Both a traditional
literature review and a meta-analysis begin with a body of
previous research on a topic; in this case, Stewart and
Chambless located 56 studies using CBT with adults diagnosed
with an anxiety disorder (including panic disorder, social
anxiety, post traumatic stress disorder, generalized anxiety
disorder, and obsessive-compulsive disorder). Studies that
included an additional medication treatment were excluded. The
researchers performed a statistical analysis of the results of
these studies and concluded that CBT was effective in treating
all of the types of anxiety disorders. In a traditional literature
review, it can be difficult to provide the type of general
conclusion that was reached with the meta-analysis because it is
necessary to integrate information from many studies with
different experimental designs, disorders, and measures of
anxiety.
One of the most important reasons a meta-analysis can lead to
clear conclusions is that meta-analysis studies focus on effect
size (recall that an effect size represents the extent to which two
variables are associated, see page 256). A typical table in a
meta-analysis will show the effect size obtained in a number of
studies along with a summary of the average effect size across
the studies. More important, the analysis allows comparisons of
the effect sizes in different types of studies to allow tests of
hypotheses. For example, Miller and Downey (1999) analyzed
the results of 71 studies that examined the relationship between
weight and self-esteem. Table 14.1 shows a few of the findings.
The effect size r averaged across all studies was −.18: Heavier
weight is associated with lower self-esteem. However, several
variables moderate the relationship between weight and self-
esteem. Thus, the effect size is larger when the weight variable
is a report of self-perceived rather than actual weight, and the

relationship between weight and self-esteem is somewhat larger
for females than for males. Finally, the effect is greater among
individuals with a high socioeconomic background.
TABLE 14.1 Some meta-analysis findings for weight and self-
esteem
Page 305Both narrative reviews and meta-analyses provide
valuable information and in fact are often complementary. A
meta-analysis allows statistical, quantitative conclusions
whereas a narrative review identifies trends in the literature and
directions for future study—a more qualitative approach. A
study by Bushman and Wells (2001) points to an interesting way
in which knowledge of meta-analysis can improve the way that
we interpret information for literature reviews.
The reviewers in their study were undergraduates who were
provided with both titles and information about the findings of
20 studies dealing with the effect of attitude similarity on
attraction. Sometimes the titles were salient with respect to the
findings (“Birds of a Feather Flock Together”) and others were
nonsalient (“Research Studies Who Likes Whom”). Salient titles
are obviously easier to remember. When asked to draw
conclusions about the findings, naive reviewers with no
knowledge of meta-analysis overestimated the size of the
similarity–attraction relationship when provided with salient
titles. Other reviewers were given brief training in meta-
analysis; these reviewers drew accurate conclusions about the
actual findings. That is, they were not influenced by the article
title. Thus, even without conducting a meta-analysis, a
background in meta-analysis can be beneficial when reviewing
research findings.
USING RESEARCH TO IMPROVE LIVES
In a presidential address to the American Psychological
Association, George Miller (1969) discussed “psychology as a
means of promoting human welfare” and spoke of “giving
psychology away.” Miller was addressing the broadest issue of
generalization, taking what we know about human behavior and

allowing it to be applied by many people in all areas of
everyday life. Zimbardo's (2004) presidential address to the
American Psychological Association described many ways in
which Miller's call to give psychology away is being honored.
The impact of psychological research can be seen in areas such
as health (programs to promote health-related behaviors related
to stress, heart disease, and sexually transmitted diseases), law
and criminal justice (providing data on the effects of 6- versus
12-person juries and showing how law enforcement personnel
can improve the accuracy of eyewitness identification),
education (providing methods for encouraging academic
performance or reducing conflict among different ethnic
groups), and work environments (providing workers with more
control and improving the ways that people interact with
computers and other machines in the workplace). In addition,
psychologists are using the Internet to provide the public with
information on parenting, education, mental health, and Page
306many other topics—for example, the websites of the
American Psychological Association and the Association for
Psychological Science
(http://www.apa.org; http://www.psychologicalscience.org),
national mental health resource websites
(http://www.mentalhealth.gov/ and http://www.samhsa.gov/),
and many individual psychologists who are sharing their
expertise with the public.
We have discussed only a few of the ways that basic research
has been applied to improve people's lives. Despite all the
potential problems of generalizing research findings that were
highlighted in this chapter, the evidence suggests that we can
generalize our findings to many aspects of our lives.
ILLUSTRATIVE ARTICLE: GENERALIZING RESULTS
Driving around in a 4,000-pound automobile is a dangerous
thing. Motor vehicle accidents are among the leading
preventable causes of death in the United States every year.
Distraction is one of the most common causes of automobile
accidents, and talking to another person is a very common

distraction.
In an effort to observe the impact of conversation on driving,
Drews, Pasupathi, and Strayer (2008) conducted a study using a
driving simulator that tracks errors committed by drivers. The
researchers varied the type of conversation. In one condition,
participants had a conversation with a passenger; in another
condition, participants talked on a cell phone. There was also a
no conversation, control condition. As you would expect,
having any conversation resulted in more driving errors.
However, the number of driving errors was highest in the cell
phone condition.
For this exercise, acquire and read the article:
Drews, F., Pasupathi, M., & Strayer, D. (2008). Passenger and
cell phone conversations in simulated driving. Journal of
Experimental Psychology: Applied, 14, 392–400.
doi:10.1037/a0013119
After reading the article, consider the following:
1. Describe how well you think the sample of participants in
this study generalizes to other groups of people. What about
age? What about sex?
2. In this study, participants were told to have a conversation
about a time when “their lives were threatened.” Do you think
that the results of this study would be different if the
conversation were about something else? How so? Why?
3. Do you think that the findings from this study would
generalize to other cultures? Do you think that a sample of
college students in Mexico, Italy, and Germany would generate
similar results? Why or why not?
4. How well do you think the driving simulator generalizes to
real-world driving? What would you change to improve the
generalizability of the simulator?Page 307
5. Evaluate the internal validity of this study. Explain your
answer.
6. Evaluate the external validity of this study. Explain your
answer.
Study Terms

Conceptual replication (p. 302)
Exact replication (p. 300)
External validity (p. 292)
Literature review (p. 303)
Meta-analysis (p. 303)
Replication (p. 300)
Solomon four-group design (p. 298)
Review Questions
1. What is external validity?
2. Why should a researcher be concerned about generalizing to
other populations?
3. How can the fact that most studies are conducted with
college students, volunteers, and individuals from a limited
location and culture potentially impact external validity?
4. How does the use of the Internet to recruit subjects and
collect data impact external validity?
5. What is the source of the problem of generalizing to other
experimenters? How can this problem be solved?
6. Why is it important to pretest a problem for generalization?
Discuss the reasons why including a pretest may affect the
ability to generalize results.
7. Distinguish between an exact replication and a conceptual
replication. What is the value of a conceptual replication?
8. What is a meta-analysis?
Activities
1. It is easy to collect data for experiments and surveys on the
Internet. Anyone in the world who is connected to the Internet
can participate in an online experiment or survey. Use a search
term such as “psychological research on the Internet” to find
some studies that are being conducted. Page 308What issues of
generalization might arise when interpreting the results of such
studies? Does the computer aspect of the research make this
research less generalizable than traditional research, or does the
fact that people throughout the world can participate make it
more generalizable? Could you empirically answer this
question?

2. Use PsycINFO to find abstracts of articles that included
race, ethnicity, gender, or nationality as a key variable.
Consider topics such as body image, rumination, academic
achievement, or identity development. What conclusions do the
authors of these studies draw about generalization?
3. Find a meta-analysis published in a journal; two good
sources are the Review of Educational
Research and Psychological Bulletin. What conclusions were
drawn from the meta-analysis? How were studies selected for
the analysis? How was the concept of effect size discussed in
the meta-analysis?
ChapterResearch Training: Joys and Challenges
It takes many years to become truly skilled at a complex set of
tasks, such as becoming a skilled potter. For example, in Japan,
it is common for an apprentice potter to make the same vase
form for many months just to acquire the specific skills to
consistently make 15–20 identical vases in a row. Expertise
develops over time within an environment that fosters
development.
In this chapter we discuss the process of becoming a skilled
researcher and the type of training environments that seem to be
most helpful for students to acquire the necessary skills and
attitudes. For many students in counseling, graduate school is
their first introduction to research and research design, and it
evokes many emotional reactions that come with any novel
experience, from joy and excitement to anxiety and
disenchantment. We seek to prepare students to approach
research with enthusiasm for the creativity involved and with a
willingness to learn the intricacies of the craft and the joy of
creating new knowledge for the profession. In addition, we want
to promote an awareness of the anxiety that may be created by
learning a technical skill that may not be central to one's
interests but is required to accomplish a goal (that is, obtaining
a graduate degree and becoming a professional in counseling).

We address four topics in this chapter. First, we provide a brief
overview of how science intersects with counseling and
psychology training through the scientist-practitioner model and
the topic of evidence-based practice. Second, we identify and
discuss some issues related to the developmental process of
acquiring research competencies, specifically the joy as well as
the challenges and fears. Third, we discuss ways in which
counselors and counseling psychologists can train others to
become competent, eager, and productive researchers. In this
part of the chapter, we discuss what is known about research
training, and how training environments can be structured to
create the opportunity for students both to learn about research,
and to consume and produce quality research products. Fourth,
we discuss ways to broaden the concept of scientific training to
include scientific thinking skills as well as research application
skills. Most important, throughout this chapter we emphasize
the developmental process of acquiring skills step by step to
become a skilled researcher and scientist.SCIENCE AND
PRACTICE
The Scientist-Practitioner Model
Bridging the gap between research and practice in counseling is
an important issue (Murray, 2009). As we mention in Chapter 1,
many graduate training programs in counseling espouse
the scientist-practitioner model of training. Basically, this
model consists of training in both scientific and practitioner
activities; the basic assumption is that students trained in both
science and practice will be better prepared for the multitude of
employment demands in the counseling profession. The
scientific activities include courses that focus on the philosophy
of science, qualitative and quantitative designs and methods,
statistics, evaluation, counseling research literature, and often
involvement in research projects (Larson & Besett-Alesch,
2000). The practice-oriented side includes courses such as
counseling methods, counseling theories, personality,
assessment, and involvement in a variety of practicum

experiences. When students enter a graduate training program in
counseling, they typically have a wide range of interests along
with the scientist-practitioner continuum. These interests often
change over time, not only during graduate training but also
throughout their career; thus, students need to prepare
themselves broadly to allow for career changes over time.
The scientist-practitioner model goes back to over 60 years. The
clinical psychologists developed what they dubbed the scientist-
practitioner model of training, which is also referred to as the
Boulder model (Raimy, 1950).
Meara et al. (1988) succinctly captured the scientist-practitioner
model:
Psychologists, whatever their work, are professionals and their
attitude toward their work is scientific. The scientist-
professional model is an integrated approach to knowledge that
recognizes the interdependence of theory, research, and
practice. The model emphasizes systematic and thoughtful
analyses of human experiences and judicious application of the
knowledge and attitudes gained from such analyses. An attitude
of scholarly inquiry is critical to all the activities of those
educated in the scientist-professional model. The model
encompasses a variety of research methods, assessment
techniques, and intervention strategies. The counseling
psychologist is engaged in the pursuit and application of
psychological knowledge to promote optimal development for
individuals, groups, and systems (including families), and to
provide remedies for the psychological difficulties that
encumber them. To implement these goals, the scientist-
professional psychologist adopts a scientific approach based on
observation of psychological phenomena. This approach
generates theoretical constructs and propositions, which are in
turn tested as hypotheses (Claiborn, 1987; Pepinsky &
Pepinsky, 1954). (p. 368)
The scientist-practitioner model has been the predominant
model of most counseling psychology programs with 73% of the
programs specifically referring to the scientist-practitioner

model in their program description (Horn et al., 2007).
However, over the years there have been some critiques,
especially in the earlier years about the utility of the scientist-
practitioner model, which students may also have concerns
about today. These included questions such as whether most
practitioners use research findings in their practice. Moreover,
it could also be that graduate training in practitioner activities
does not adequately incorporate scientific activities, and vice
versa (e.g., Stoltenberg et al., 2000). Some people might think
that perhaps the type of research being conducted is too distant
from the reality of the practitioner (see, e.g., Howard, 1984), or
perhaps our research methods reduce counseling phenomena to
meaningless numbers (see, e.g., Goldman, 1976). Yet another
factor could be that students admitted to graduate programs
have predominantly social or interpersonal interests (see, e.g.,
Magoon & Holland, 1984).
Therefore, other training models have emerged. These models
share the core goals of integrating science and practice, but with
different emphasis. For example, the practitioner-scholar
model places more emphasis on the training of practitioners and
was closely associated with the emergence of professional
schools of psychology and Psy.D. degrees. This model also
parallels the idea of local clinical scientists (Stricker &
Trierweiler, 1995) and practice-based evidence (Barkham &
Mellor-Clark, 2000) that emphasize using clinical work and
observations of clients to collect data and test clinical
hypotheses. On the other end of the science-practice training
spectrum, the clinical science model (Academy of Psychological
Clinical Science, 2014) is similar to the scientist-practitioner
model in theory, but places even more emphasis on science and
uses empirical approaches to advance clinical science.
The importance of science in the psychology field has not
diminished, but rather has grown over the past few decades.
There has been increased emphasis on utilizing scientific
findings in mental health practices. Thus, it is important for
students to have the scientific skills to critically evaluate

research findings rather than simply accepting knowledge from
research as authoritative prescriptions (Stoltenberg et al., 2000).
Integrating the scientist-practitioner model with current trends
in the mental health field is important, and there are
recommendations on how the scientist-practitioner model can be
better implemented (see Stoltenberg et al., 2000). Chwalisz
(2003) suggested that evidence-based practice is the framework
for training scientist-practitioners in the 21st century. The
evidence-based practice movement in psychology has become a
main focus over the past decade.
Evidence-Based Practice
The increased demands from health care systems and policies to
integrate science and psychology practice have made evidence-
based practice an important movement in the psychology and
mental health field (Baker, McFall, & Shoham, 2008).
Evidence-based practice in psychology (EBPP) is “the
integration of the best available research with clinical expertise
in the context of patient characteristics, culture, and
preferences.” (APA Presidential Task Force on Evidence-Based
Practice, 2006, p. 273). The field of medicine is an example of
how EBPP has shaped the improvement of patient outcomes in
utilizing research in clinical practice (Woolf & Atkins, 2001).
EBPP is a more comprehensive concept than a closely related
movement of establishing empirically supported
treatments (ESTs), which refers to the specific treatment that
works for certain disorders or problems. Whereas ESTs focus on
the treatment approach, EBPP starts with the client and with the
attempt to utilize research evidence to inform treatment to
achieve the best possible outcome for the client. Moreover,
EBPP includes a broader scope of clinical/counseling aspects,
such as assessment, case formulation, counseling relationships,
etc.
In APA's 2006 Presidential Task Force on Evidence-Based
Practice, research evidence, clinical expertise, and patient
characteristics were all emphasized as critical components

leading to best practices. In terms of this research evidence, a
broad array of scientific results related to intervention
strategies, assessment, clinical problems, and client populations
are all included. Multiple types of research evidence such as
efficacy, effectiveness, cost effectiveness, cost benefit,
epidemiological, and treatment utilization are all recognized as
contributors to informing best psychological practices. EBPP
also requires mental health practitioners to recognize that
treatment method (Nathan & Gorman, 2007),
counselors/therapists (Wampold, 2001), therapy relationship
(Norcross, 2002), and the client (Bohart & Tallman, 1999) are
all critical components to achieve best outcomes in
psychological practices, and therefore research evidence on any
of these aspects should not be overlooked.
Clinical expertise is also a critical component of EBPP (APA
Presidential Task Force on Evidence-Based Practice, 2006).
Counselors and psychologists are recognized as essential
contributors in identifying the best research evidence based on
the client's characteristics, background, presenting issue, and
preferences. Thus, it is important that counselors and
psychologists are trained not only as practitioners but also as
scientists so that their practice is informed by scientific
literature. Client characteristics, culture, and preferences are
also important parts of EBPP. Counselors and psychologists
should take into account what works for whom when providing
guidelines for effective practice. EBPP requires attention to
client characteristics such as gender, sexual orientation, culture,
ethnicity, race, age, religion, nationality, disability status, etc.
In addition to client characteristics, it is also important to
consider the client's values, religious beliefs, worldviews,
goals, and preferences for treatment when determining and
utilizing the best available treatment for the client.
Although APA and many training programs have supported
EBPP, there has also been resistance toward EBPP. Similar to
the critiques of the scientist-practitioner model, there have been
several reasons that EBPP raises concerns. Lilienfeld and

colleagues (2013) examined the roots of resistance to EBPP by
clinical psychologists and summarized six sources:
(a) naïve realism, which can lead clinicians to conclude
erroneously that client change is due to an intervention itself
rather than to a host of competing explanations; (b) deep-seated
misconceptions regarding human nature (e.g., mistaken beliefs
regarding the causal primacy of early experiences) that can
hinder the adoption of evidence-based treatments; (c) statistical
misunderstandings regarding the application of group
probabilities to individuals; (d) erroneous apportioning of the
burden of proof on skeptics rather than proponents of untested
therapies; (e) widespread mischaracterizations of what EBP
entails; and (f) pragmatic, educational, and attitudinal obstacles,
such as the discomfort of many practitioners with evaluating the
increasingly technical psychotherapy outcome literature.
(Lilienfeld, Ritschel, Lynn, Cautin, & Latzman, 2013, p. 883)
In addition to highlighting the roots of resistance, Lilienfeld et
al. (2013) also offered suggestions on how to address these
resistances among students in training. They suggested exposing
students to a better awareness that they are not immune from
cognitive biases, such as confirmation bias, hindsight bias, and
illusory correlation. They also suggested graduate instructors to
not only convey accurate information, but also disabuse
students of inaccurate information, especially around
misunderstandings of the human nature. They also argue that
students should not just be taught the “hows” of implementing
therapy techniques, but more so the “whys” of how these
approaches have been established as effective. In sum,
Lilienfeld et al. stressed the importance of critical thinking and
scientific training of students in the mental health field. For
more detailed information on EBPP, as well as the resistance
toward EBPP, please see APA Presidential Task Force on
Evidence-Based Practice (2006) and Lilienfeld et al. (2013).
Science and Practice Training
The focus between research and practice has been an issue for

training programs that emphasize both. Sometimes the model
has been interpreted as a 50–50 split of performing science and
practice activities. The type of model in its ideal form (i.e.,
implicitly 50% practitioner and 50% scientist/researcher) may
be just that—an ideal that is rarely found in reality. Gelso
(1979) proposed that it may be more realistic to train students
in both domains (in varying degrees depending on their
interests) with the expectation that students will find a suitable
place for themselves in performing relevant activities on the
scientist-practitioner continuum. Thus, one student might prefer
a 20–80 split while another might choose a 75–25 split.
Sometimes there have been implicit values attached to either
science or practice; that is, some educators might value science
more than practice, and thus feel more satisfaction when a new
graduate obtains the “best” job, which to them is an academic
position complete with a myriad of scientific pursuits. We
strongly believe that science and practice are both highly valued
activities in the counseling profession, and that as a profession
we are stronger (and only can survive) when we train students
to be competent in both science and practice. In short, it is
important for the profession to equally value various points
along this performance continuum.
We prefer to conceptualize the core of the scientist-practitioner
model in terms of scientific or critical thinking. In short, we are
suggesting that the role of scientific or critical thinking is a
central outcome of a wide range of science and practice
activities. The choice of whether a graduate engages in science
or practice activities is not the most important outcome, but
rather whether the graduate can utilize scientific or critical
thinking in whatever professional activities she or he chooses.
For example, a counseling center staff member might be
engaged primarily in direct client service and, say, one program
evaluation project (see Chapter 22); this might be a 5%–95%
science-practice split of professional activities. A faculty
member (three-quarters time) with a quarter-time direct service
appointment in the counseling center might have a 75%–25%

split of professional activities. In addition to the quantitative
difference in how research and practice is split, the role of
research can also be qualitatively different. Gelso and Fretz
(2001) proposed a three-level model of how research functions
for counselors. First, the minimum level involves counselors
being able to consume and apply research in their work with
clients. The second level involves counselors being able to use
critical thinking to develop and test hypotheses during their
work with clients. The third level involves being able to
conduct empirical studies and to present the research findings.
Regardless of the type of professional activities a person selects
along the scientist-practitioner continuum, we maintain that
counselors utilize scientific thinking within their practice and
science activities.
Baker, McFall, and Shoham (2008) suggested that the lack of
adequate scientific training and utilization of scientific
knowledge has been a major problem in the development of the
clinical psychology field, which is also a reflection of the
mental health profession:
Clinical psychologists' failure to achieve a more significant
impact on clinical and public health may be traced to their deep
ambivalence about the role of science and their lack of adequate
science training, which leads them to value personal clinical
experience over research evidence, use assessment practices
that have dubious psychometric support, and not use the
interventions for which there is the strongest evidence of
efficacy. Clinical psychology resembles medicine at a point in
its history when practitioners were operating in a largely
prescientific manner. Prior to the scientific reform of medicine
in the early 1900s, physicians typically shared the attitudes of
many of today's clinical psychologists, such as valuing personal
experience over scientific research. (p. 67)
In the end, there is no doubt that research has enhanced the
practice of counseling. Even though a practitioner may not be
able to cite a specific reference, his or her graduate school
training was likely based on a tremendous amount of research

data, all the way from personality theory to intervention
strategies. The accumulation may be slow, but the data
eventually advance our working knowledge of the field. The
scientific method and research have advanced the field, but they
could be even more helpful. In addition, training in both science
and practice has become more sophisticated with not only more
integration but also more emphasis on critical thinking, which is
the core of all science and practice activities of the counseling
profession. For more information on a broad array of
recommendations for enhancing training in the scientist-
practitioner model and how EBPP is a framework for training
scientist-practitioners in the 21st century, see Chwalisz
(2003).JOYS AND CHALLENGES IN ACQUIRING
RESEARCH COMPETENCIES
Students will often experience a number of different reactions
to research training, and these emotional reactions can change
throughout the course of their graduate training and actually
many years beyond (see Table 2.1). Sometimes students will be
elated with the joy of discovering new and useful information
from their research. Other times students will become restless
and disgruntled with the minutiae associated with research. In
addition, some students will experience obstacles such as
negative self-talk, procrastination, perfectionist tendencies, and
a range of “shoulds” (e.g., I should be better at math; see
Heppner & Heppner, 2004).
TABLE 2.1Common Student Reactions toward Research and
Our Suggestions
Student Feelings
Our Suggestions
I don't know enough to conduct any meaningful research.
Volunteer to help on a research project; slowly work your way
up to learn the more advanced skills.
I am a fraud and really don't know how to do research.
Feelings of anxiety and doubt associated with research are
normal; they are often due to having limited experiences.
What if I make a mistake and report inaccurate findings?

It is a learning process; allow yourself time to learn and develop
more advanced skills.
I'm scared of running into debilitating writing blocks in my
research.
Acquiring research and thinking skills occurs over a long period
of time; sometimes it is a sign that you may need to acquire
more knowledge of the literature.
I am afraid that my results will be statistically nonsignificant.
Programmatic research often results in the accrual of knowledge
over time; nonsignificant results are also helpful information.
The whole thesis process seems overwhelming, tedious, long,
anxiety producing, confusing, and complicated.
The more you get involved in research, the more self-
efficacious you will feel.
I just want to practice; I am not interested in science/research.
Be open to learning more about the research process as well as
the research literature.
science = statistics
Science and research are broader than quantitative studies that
utilize statistics. Critical thinking is the core of science.
I don't believe that research in counseling can make a difference
in the profession.
Read literature review and meta-analysis articles to get a better
understanding of cumulative knowledge base.
Science is too intellectual, too cognitive, and unrelated to the
experiences of real people.
As you obtain more experience with research activities, your
research interests may increase and you might see a higher
value on the role of research.
The developmental process of acquiring research competencies
takes considerable time, years in fact, and often differs across
students. Throughout this chapter are reflections from
experienced and skillful researchers who provide some useful
glimpses into this developmental process of acquiring an array
of research competencies. A great deal of wisdom is
communicated by these prolific and creative scholars, and we

appreciate their candid perspectives on their own developmental
journey.
These reflections on their journeys throughout the chapter
depict various aspects of the developmental process of
acquiring research competencies. From our collective
experiences as authors of this text, we have observed that many
students question whether they can adequately learn the skills
associated with research and science: How can I, a mere
graduate student, come up with a good research idea? What if I
get bogged down in my research and never finish my degree?
Statistics scare me—what if I make a mistake in analyzing my
data? We contend that these and other fears are common and
developmentally normal. We believe it is essential to discuss
the negative affective and cognitive components associated with
research, and thus the first section focuses on such common
student reactions, namely (a) performance anxiety and efficacy;
(b) the false dichotomous conceptualization as research versus
practice; and (c) belief in oneself as a researcher, and science in
general.
Performance Anxiety and Efficacy From our experience working
with students, we have found that performance anxiety is a
central and crucial affective issue. For example, common
student disclosures include: “I don't know enough to conduct
any meaningful research on my own. How can I do a thesis?”
“Any study that I could do would be so basic and bland as to
make it worthless.” “I feel completely inadequate in even
conceiving a study that examines counseling from a unique
perspective.” “Although I made it through the statistics and
research design courses, once I start to do some research, I'll be
‘found out’; I am a fraud, and really don't know how to do
research.” “What if I make a mistake and report inaccurate
findings?” “I've had trouble writing before, and I'm scared to
death that I will run into debilitating writing blocks again in my
research.” “I am afraid that my results will be statistically
nonsignificant, and all of my research efforts and time will just
be one big waste.” “I've seen other students in the throes of a

dissertation. The whole process seems overwhelming, tedious,
long, anxiety producing, confusing, and complicated.”
It is important to put these and similar feelings into the context
of the typical entering graduate student. Most incoming
graduate students have had some experience (or even a lot) with
helping other people, and consequently the benefits of
counseling and other practitioner activities are abundantly clear.
Moreover, for most of us, helping people is a personally
rewarding experience, and students have an understanding of
those rewards. Conversely, it has been our experience that most
incoming students have had considerably less experience with
science. Typically, students do not imagine themselves making
contributions to the profession by publishing in our professional
journals. Since students have had little, if any, research
experience, they often have legitimate reasons to question and
doubt their skills and abilities—these feelings, concerns, and
fears are thus developmentally normal. In fact, it is not
uncommon even for faculty to experience performance anxiety
and question their efficacy (e.g., see Research Journey
2.1). This suggests to us that acquiring the necessary research
and thinking skills occurs over a long period of time; it does not
happen all at once. In fact, expert performance is found
associated with active engagement in deliberate practice, which
involves training on particular tasks, receiving immediate
feedback, having time for evaluation, problem solving, and the
opportunity to refine behaviors (Ericsson, 2008).
RESEARCH JOURNEY 2.1
Developing competencies in research is not an event. Instead, it
is a slow journey; as some markers are achieved, new
competency goals quickly surface, even as a senior faculty
member. I am committed to life-long learning! This applies to
life in general and also to my life as a researcher. I did not
always hear encouraging words from others about my ability to
carry out a project or the importance of my work. Early in my
career, I focused on the negative messages that I received. It

took a while for me to tune out the naysayers and find ways to
identify growth areas to strengthen my research so that it could
contribute to the literature in the ways that I found important. I
have had to learn how to believe in what I do and in my ideas. I
was able to do this by seeking feedback from those who
conducted similar research—from people who “got” my work
and thus were able to identify strengths and limitations of the
work in a constructive way. I now recognize that to be a good
researcher requires feedback and input from a number of
sources. To help develop a research project, it is absolutely
essential to share your work with others and get feedback at all
stages. Over the years, I have learned that it is okay to have a
project be underdeveloped in its initial conceptualization—
others do not judge me on where I start, but rather on where I
end up. That is, how I have incorporated the ideas and
suggestions from others and how much the project has
developed over time. Knowing this has provided me the space to
make mistakes, learn, and grow.
—Helen Neville, Ph.D.
As students acquire and begin to apply their knowledge about
research design, we have been struck by many students' surprise
and actual delight about research as they get past their initial
fears. We often have heard comments like, “I'm
actually enjoying this research course”; “I never thought I
would be a researcher; now I'm beginning to reconsider my
career goals”; “This research stuff is contagious; I'm starting to
think about research studies a lot and I want to collect data
now!” (e.g., see Research Journey 2.2). In sum, the major point
is that beginning graduate students often have considerably less
information about research and science than about practitioner
activities. Moreover, there is evidence that as students obtain
more experience with research activities (especially if it is early
in their training), their research interests increase and they
place a higher value on the role of research (e.g., Gelso,
Baumann, Chui, & Savela, 2013).

When I started graduate school, my dream was to become a
director of a counseling center. Research was not part of my
career dreams, and I had no sense that I could ever publish in a
professional journal. However, subsequent research experiences
in my coursework, especially with my doctoral advisor, David
N. Dixon, shifted my career aspirations toward academia.
Several years later after having a modicum of success in
publishing, I realized that the most important goal was the
accumulation of knowledge from the systematic study of a
particular topic, and subsequent theory development. Several
more years later, I more fully realized the importance of the
context surrounding my research topic, and in particular, the
many variables (age, SES, race) within participants' cultural
context that greatly affected our research findings. I also
realized our research efforts alone were insufficient to create
the knowledge bases needed for tomorrow, and that the
worldviews and power of journal editors, editorial boards, and
others in leadership positions in our profession played a critical
role in the creation of our future knowledge bases. This
realization led me to engage in a number of professional roles
(e.g., editorial boards, assuming editorships, involvement and
leadership positions in professional organizations), all aimed at
improving knowledge bases and promoting social justice in
counseling psychology. In the end, one of the things I greatly
value in our profession is the many ways that we can join hands
with diverse partners to create knowledge bases and theories to
promote the development of a broad array of people.
—Puncky Heppner, Ph.D.
Thus, our advice to students regarding performance anxiety is
first of all to recognize that feelings of anxiety and doubt
associated with research are normal, and often reflect where one
is in the developmental process of acquiring new competencies,
in this case research competencies. Second, we strongly
recommend that students get involved with research projects to
slowly develop various research competencies. We believe the

best way to create a sense of research efficacy is to learn and
perform those skills. For example, students might offer to
volunteer to help on a research project to learn the basic
research skills, and slowly work their way up to learn the more
advanced skills (e.g., see Research Journey 2.3). In essence,
both our experience and our research have suggested that the
more students get involved in research, the more self-efficacy
they tend to feel; the more efficacious students feel about
conducting research, the more positive they are about the
training environment and interest in research (Gelso et al.,
2013; Kahn, 2001).
I had the good fortune of becoming Professor Donald Atkinson's
doctoral advisee in counseling psychology at the University of
California, Santa Barbara, back in 1996. I learned firsthand via
Don's work the power of research—that is, good research work
can lead to significant changes in our profession and society
(e.g., increased attention to multiculturalism). It was also
during that time that my research training became more
solidified through several courses on research methods and
statistics (by the way, I used the first edition of this book in my
main research methods course and it couldn't have offered me a
better start!). Most importantly, however, it was the day-to-day
work on several research studies with Don and other students
that taught me the intricacies of research, starting from problem
conceptualization and moving into research design, data
collection, data analyses, and manuscript publication. So in
hindsight, I mostly “learned by doing.” And I've come to
understand that there is no magic formula to becoming a good
researcher but to have good people around you who also are
enthusiastic about research. At this point in my career, as a full
professor with tenure and the editor of Asian American Journal
of Psychology, I continue to believe that research success is a
function of inquisitive and informed researchers working
together for a common goal. So my recommendation continues

to be: “Get involved, join a research team, study the topic of
your passion, and don't forget to enjoy the process! The
outcome will then take care of itself.”
—Bryan S. K. Kim, Ph.D.
Threats, Unrewarding Training, and Disenchantment For any of
a number of reasons, students' research training is often less
than rewarding, and actually often rather threatening, all of
which can result in student disenchantment. It is not uncommon
for students to enter graduate school with ambivalence and
anxiety about the academic and research requirements that they
have to meet over the next several years. Many students are
intimidated by their instructors and professors; in particular
those that teach courses in which the students feel less
competent in (such as statistics and research methodology,
which are less directly associated with counseling practice).
Moreover, the idea of coming up with an innovative research
topic for their thesis is quite daunting. It is hard enough to
generate adequate research questions for a study, let alone
finding measures, collecting data, statistical analysis, and
writing all these up into a lengthy thesis! With the anxiety
experienced around research, the program and training can feel
quite threatening. Unfortunately, when students have little
experience with research and science, one bad experience can
create tremendous attitudinal and motivational damage and
move a student a long way from future scientific activities.
Some training programs may actually decrease students' interest
in research.
The profession has come a long way in enhancing research
training over the last two decades. Consideration of students'
affective reactions, values, and individual differences, and of
the training environment will most likely continue to enhance
the development of students' research efficacy. In addition,
consideration of cultural values and social connections are
especially important issues for some racial/ethnic minority and
international students whose cultural values make it more
difficult to seek and participate in the research groups without

others' invitation or help.
The False Dichotomous Conceptualization of Science versus
Practice Another developmental issue is how students
conceptualize science and research, and their relationship to
practice. A common statement is, “I just want to practice; I am
not interested in science/research.” Sometimes the scientific
enterprise is erroneously conceptualized as totally separated
from applied work in counseling. Thus, a false dichotomy is
established, science versus practice. Actually there is interplay
between science and practice. Our belief is that those students
who develop skills in both arenas will be better equipped to do
the work of the counseling profession (for example,
see Research Journey 2.4).
When I started my undergraduate studies, I was sure that I
wanted to be a police detective. Although I realized that path
was not in the cards for me, the drive to find solutions to
mysteries has never left. I quickly realized when it comes to
research that the world is far more complex than I originally
thought. Additionally, with complex questions come complex
answers that frequently result in multiple equally valid
solutions. This process is what draws me into doing research
and at the same time the multitude of possibilities and enormous
amounts of information can be paralyzing—a common theme I
hear from my students. Yet I have found two sources of
inspiration: (a) working with others (counselors, researchers,
and clients) to be sure I am expanding the way I am thinking
about any given topic and to stay connected to the lived
experience of those who matter the most, and (b) tackling
specific questions so that each study is manageable. In this way,
each study is akin to adding to an ongoing conversation, and
developing a research agenda or doing multiple studies in the
same area allow for the conversation to continue. I find this
process very gratifying and rewarding!
—Jesse Owen, Ph.D..

Moreover, one of the hallmarks of graduate work in counseling
is to acquire critical thinking skills, the ability to identify and
process information with fewer biases, stereotypes, and
assumptions; formulate hypotheses; gather data; and make
informed decisions. These critical thinking skills are essential
in both the practice and science arenas. Higher critical thinking
ability has been found as a key characteristic of master
therapists (identified as having exceptional mastery of therapy
by their peers; Jennings & Skovholt, 2015; Ronnestad &
Skovholt, 2013; Skovholt, 2012). A qualitative meta-analysis of
six studies that examined characteristics of master therapists in
different countries found that master therapists have
developed more sophisticated and complex conceptualization of
clients' presenting issues (Jennings & Skovholt, 2015). One of
the key themes identified across master therapists in all the
studies was the capacity for cognitive complexity and intricate
conceptualization. This capacity was developed through
ongoing learning through reflection, feedback, and teaching
others.
Our advice to students is to avoid a narrow conceptualization of
research and science and to not prematurely conclude that
research and science have little or nothing to offer the
practitioner (Skovholt, 2012). Rather, we encourage practice-
oriented students to be open to learning more about the research
process as well as the research literature, and how these
experiences can inform practice and improve their counseling
competence. The scientific method is simply a set of tools to
establish a controlled mode of inquiry about all of our
professional activities, research and practice.
Disbelief in Oneself as a Researcher and Science in General A
critically important belief in the process of becoming a
competent researcher is believing in one's ability to conduct
meaningful research as well as belief in the utility of research,
that research does contribute to the profession's knowledge
bases, and that our research can and does make a difference in
clients' lives (e.g., see Research Journey 2.5).

Being involved in research that has the potential to make a
difference is a meaningful aspect of my work and motivates me
to persist when encountering inherent obstacles in the research
process. After graduating from college with degrees in Biology
and Sociology (and a minor in Women's Studies), I worked in an
inner city battered women's shelter. At the shelter, I became
interested in what differentiated women who returned to the
abuser from women who struggled to create nonviolent lives for
themselves and their children. In part, this desire to understand
critical social problems motivated me to purse a master's
degree, where I focused my thesis on shelter workers'
perceptions of battered women. I loved how my work in the
shelter informed and was informed by my research. I also began
to see that while helping people individually was rewarding to
me, I might be able to make a difference through sharing my
research findings with others. My doctoral program solidified
my passion for research as I was given the opportunity to study
topics of interest that had real-world applications, including
helping women to achieve economic self-sufficiency. I
sometimes struggle with a component of research (e.g.,
collecting data from difficult to obtain samples), but as soon as
this task becomes very painful, the next step in research brings
new and exciting challenges (e.g., analyzing the data!). Also,
working with students and colleagues on research teams makes
the research process a wonderful, fun, and memorable
adventure.
—Karen O'Brien, Ph.D.
It is common for students to not initially feel efficacious as a
researcher nor to believe that our research in counseling can
make a difference in the profession. Most beginning students do
not have much of an understanding of the role of science in our
profession, and since they do not have a long history in the
profession they are unable to identify how previous research has
contributed to the profession. Similarly, students do not have a

clear understanding of the utility of the scientific method as a
way of thinking about either research or clients. Finally, the
student typically has had relatively little experience of
expanding his or her understanding about clients and applied
problems by engaging in research, reading the professional
literature, and contemplating the implication of empirical
findings. One unfortunate outcome is that students sometimes
fill this void with misinformation, like “science = statistics.” Or
students believe that “science is too intellectual, too cognitive,
and unrelated to the experiences of real people.” Or “the
scientific method is so mechanical and reductionistic that it
cannot be used adequately to study the complexities of real
human behavior.” Conversely, seasoned researchers understand
both the challenges of conducting meaningful research and the
gradual but steady accumulation of knowledge from our
research. Even though it is rare that any one study has a major
impact on the profession, programmatic research often does
result in the accrual of knowledge over time, and often achieves
significant conclusions. For example, many of the chapters in
the APA Handbook of Counseling Psychology (Fouad, Carter, &
Subich, 2012) are based on extensive literature reviews and
nicely reflect the utility of our research in creating knowledge
as well as enhancing the sophistication of our conceptualization
of relevant constructs in counseling.
Clara Hill, a creative and prolific counseling researcher, was
invited to write a personal account of her evolution as a
researcher. Her account illustrates how she made decisions, how
complex the process of research can be, and how she coped with
the ups and downs along the way and, most important, her
“crisis of faith in research”:
After this series of studies I underwent a crisis in terms of
research. I had achieved tenure, had my first baby, and turned
30. I started questioning everything. Because I could see so
many imperfections in my research, I was not very proud of my
work. It had taken an incredible amount of time for little “hard
information.” I could not even remember the results of many of

my studies. Further, I seemed woefully far from even describing
what happens in the counseling process let alone understanding
what counselor behaviors were useful in effecting change. I
despaired that counseling was far too complex to be studied. I
had previously had lots of doubts about my research, but it was
mostly related to self-confidence in my research abilities. This
crisis seemed to have more to do with whether I felt that
research was a viable means to answer questions about the
counseling process. (Hill, 1984, p. 105)
Hill's behind-the-scenes disclosures are illuminating as well as
often relieving for many students and faculty. We strongly
encourage students to read this article, perhaps during their first
course on research methods. We have found that Hill's
frustrations, jubilations, feelings of inadequacy, and doubts
about the role of research resonate strongly with students. The
evolution of Hill's confidence in the utility of conducting
research in her own way not only sends a useful message, but
also illuminates the developmental processes of students and
faculty in becoming skilled and competent researchers.
From our experience, it is critical not only that researchers feel
efficacious in conducting relevant and important research, but
also that they begin to perceive that their research is important,
that it contributes useful information to our professional
knowledge bases, and that it can make a difference in the
profession as well as the lives of our clients. Such beliefs often
are the result of identifying implications of your research
findings to practice, being cited in the professional literature in
meaningful ways, or seeing either that others are using your
research findings in applied settings or that your research is
helping other researchers subsequently ask other important
questions.
Belief in the utility of research can also result from reading the
professional literature to learn the impact of others' research.
Our advice is for students to not only read and reflect on the
literature, but also to read literature reviews in particular (see
Major Contributions in The Counseling Psychologist or review

papers in Psychological Bulletin). In addition, we advise
students to get involved in conducting research on topics that
are really important to them, either topics that they are very
interested in or ones that reflect their personal values in some
way (e.g., see Research Journey 2.6).
I firmly believe that the best researchers usually have a
connection and passion about their research. I developed my
research program, in part, to help me understand and make
sense of my own experiences as an African American man in a
predominantly White college environment struggling to do well
academically, while at the same time trying to learn more about
my history and culture. This desire to better understand myself
and the experiences of other people like me led to a
dissertation, my first publication, several other articles, and a
book. Everyone has experienced challenges in writing and
conducting research at some point, and everybody has
experienced rejection when trying to publish something. I have
learned that it is not always the “smartest” people who can
successfully carry out a research project and publish it, but
often it is those individuals who are the most resilient.
—Kevin Cokley, Ph.D.A MODEL FOR RESEARCH TRAINING
The purpose of this section is to discuss a number of factors
that are important in enhancing research training. Our goal is
not only to enhance students' awareness of what appears to be
some of the major constructs in research training, but also to
promote additional attention and research on this important
developmental process. Based on the culminated research on
effective research training, Kahn (2001) proposed a model that
incorporated various factors, which is presented in Figure 2.1.
We will discuss this model and the empirical evidence that
supports parts of it, as well as add a few other important aspects
of research training not included in the model. We begin by
discussing each of the major constructs in the model: (a)
investigative interests, (b) research training environment, (c)

relationship with mentor, (d) research outcome expectations, (e)
research self-efficacy, (f) research interest, (g) scholarly
activity, and (h) year in program.
FIGURE 2.1Parameter Estimates for the Modified Structural
Model of Kahn (2001)
Constructs of the Model
Investigative Interests Most students in counseling and
counseling psychology can be categorized into one of Holland's
(1992) three personality types: social, artistic, or investigative.
Research has shown that the personality type of counseling
students is related to their level of interest in research and in
research courses, choice of graduate program types with varying
levels of research focus, and to research productivity
(Mallinckrodt et al. 1990; Tinsley, Tinsley, Boone, & Shim-Li,
1993). Specifically, it appears that students who are
investigative types have more interest in research activities and
ultimately produce more research (e.g., see Research Journey
2.7). Accordingly, Kahn (2001) found that a student's
investigative interest would be directly related to expectations
of research outcome and research interest, which in turn would
be related to research productivity. In particular, investigative
types are hypothesized to be more interested in research, and
social types less interested.
My passion for science originated at a very early age, when I
found myself setting up mini “experiments” to test various
questions. I was curious about the world and about human
interactions in the world. At this young age I also was acutely
aware of individuals whose experiences in the world were
different from my own. I viewed humans as equal with unequal
opportunities. After multiple undergraduate transfers, I was
fortunate to have ended up in a college that provided me access
to mentoring and advising that enabled my interests, skills, and
efficacy in science to flourish. Were it not for these individuals

and my experiences, I would have never realized that a doctoral
degree or a career as a scientist-practitioner was a viable
option. Throughout my graduate training and career, my passion
to share my excitement for the scientific process in a way that is
meaningful for others (e.g., via knowledge in research design
and analysis, consultation with organizations, developing and
testing hypotheses with clients in psychotherapy) has continued
to blossom. As I reflect upon my research journey, I appreciate
the combination of factors that have contributed. These personal
characteristics (I am deeply curious about the world), beliefs (I
believe that science has the potential to impact practices and
prompt positive change in the world), and values (opportunities,
and access to them, matter) continue to shape the work in which
I choose to engage and the questions that I choose to ask.
Mindi Thompson, Ph.D.
Research Training Environment Gelso, Baumann, Chui, and
Savela (2013) discussed 10 ingredients of graduate training
programs that foster positive attitudes toward research. These
components have been identified as important features of the
research training environment: (a) faculty modeling of scientific
behavior; (b) the positive reinforcement of scientific activity,
both formally and informally; (c) early and minimally
threatening involvement in research; (d) emphasizing science as
a partly social-interpersonal experience; (e) emphasizing that all
studies are limited and flawed; (f) teaching and valuing varied
approaches to research; (g) teaching students to look inward for
research ideas when they are developmentally ready to do so;
(h) the wedding of science and practice; (i) relevant statistics
and the logic of design; and (j) teaching how research can be
done in practice settings.
An illustration of a positive training environment that included
the 10 components would be having an environment that
involved students in research as early as possible and in ways
that are interesting, involving, and consistent with their skill
level. Many counseling students are socially oriented,
increasing the social aspects of research through the forms of

advising, and research teams will increase the attractiveness of
the endeavor for them. In research groups, students of various
levels can participate to the extent that they feel comfortable
and have something to contribute. Research advising and
modeling is beyond transferring skills and involves having
research mentors exhibit healthy attitudes toward research
including both the excitement and disappointment in response to
its trials and tribulations. Honest expressions of reinforcement
to students all along the way in the research process are an
important part of building interest. Providing a nonintimidating
atmosphere for learning is important. Students should not feel
pressured to design the perfect study; rather, they should feel
motivated to create a study that can address their research
questions. Also, many counseling students have little fondness
for statistics and are turned off to research because they equate
doing research with studying for statistics examinations. Some
students will be attracted to quantitative methods and others to
qualitative methods, and these preferences should be respected
and honored. Not only do students benefit by using a method
that they enjoy, but the field benefits from this methodological
pluralism (Gelso et al., 1988). A true integration of science and
practice will make research more attractive, especially to the
majority of students whose primary interest is in practice.
Looking inward to rely on clinical experience is an example of
wedding science and practice. Students who have a deep interest
in their topic finish their dissertations faster than those who are
motivated to finish their programs but who pick a convenient
topic of study. Not all research environments are ideal, but it is
important to make the most out of them (e.g., see Research
Journey 2.8).
I applied to graduate school with the goal of conducting
community-engaged research on Asian immigrant families and
disseminating this research through community organizations
and churches. However, I ended up at a graduate program that

did not have such research opportunities at the time. Rather than
get disheartened, I had to make the most of what was available
to me, so I pursued more mainstream research on individual
differences. But I kept my interests alive by independently
reading ethnic minority psychology research, writing papers on
ethnic minority issues, and finding support and inspiration
through my membership with the Asian American Psychological
Association. It was just a dream deferred, but not like a raisin in
the sun (h/t Langston Hughes). Ideas percolated; studies were
imagined and reimagined over and over again. And then when I
finally had the chance to pursue my original research interests, I
felt so relieved, happy, and completely scared. Was I really
ready? Could I do it? Like most things in life, you have to just
do it and don't stop. Keep pushing yourself to do it better. Learn
from mistakes. Collaborate with others. Dare to be different, but
be sure you are ready to defend your decisions. My program of
research today is consistent with my original research interests,
but also ever changing—almost always for the better. You need
to have passion and commitment to pursue a research career, but
don't get stubborn or set in your ways. Be adaptive and open to
new ideas and opportunities.
—Richard Lee, Ph.D.
Gelso and colleagues (Gelso et al., 1996; Kahn & Gelso, 1997;
Royalty et al., 1986) developed an instrument (the Research
Training Environment Scale, RTES, and its revision, RTES-R)
that operationalized these nine components of the training
environment. Because it was unlikely that these aspects are
independent, Kahn and Miller (2000) later developed a short
version of the RTES-R (RTES-R-S), which included two higher
order factors that underlie the nine aspects operationalized by
the RTES-R. The first factor, labeled instructional dimension,
contains aspects of the environment that are present because of
curricular components of graduate training, which included
teaching statistics and the logic of design, teaching students to
look inward, teaching that experiments are flawed, teaching
varied approaches, and wedding science and practice. The

second factor was labeled interpersonal dimension because the
components involved the interaction of trainer and trainee,
which includes modeling, reinforcing, involving students early,
and conducting science as a social experience. Although the
nine components of the research environment can be reduced to
two factors statistically, the examination of the individual
components that follow reveals how research trainers can
construct curricula and experiences to foster the development of
skilled and motivated consumers and producers of research.
Thus, we recommend that professionals involved in research
training read Gelso's (1979) seminal article on research in
counseling and Gelso et al's (2013) review of empirical
evidence over the past three decades supporting the importance
of the research training environment.
Relationship with Mentor Similar to the research training
environment, a graduate student's relationship with his or her
mentor is an important aspect associated with research outcomes
(e.g., see Research Journey 2.9). The quality of research
mentoring that a student receives is a critical aspect of the
environment, which has been found to be related to students'
attitude toward research and their research self-efficacy
(Hollingsworth & Fassinger, 2002; Morrison & Lent, 2014). A
qualitative study identified four main factors related to students'
satisfaction of the mentoring relationship: (a) the ability to
choose their own advisors, (b) the frequency of individual and
group meetings with their advisors, (c) how conflict was
handled in the advisory relationship, and (d) the benefits and
costs associated with the advisory relationship (Schlosser,
Knox, Moskovitz, & Hill, 2003). Schlosser et al. categorized
each of the 16 participants in their study based on whether they
were satisfied (n = 10) or unsatisfied (n = 6) with her or his
relationship with her or his advisor. Most of the satisfied
students were found to have had the option to choose their
advisors, whereas all the unsatisfied students were assigned to
their advisors. As for meetings with advisors, satisfied students
were found typically to have frequent individual or group

meetings, whereas unsatisfied students generally had infrequent
individual meetings (e.g., once or twice a semester). Conflict in
advising relationships was dealt more openly and difficult
issues were addressed among the satisfied students, whereas
conflict was avoided for those that were unsatisfied. All
unsatisfied students reported that their needs were being met
outside of their advisory relationship, whereas only some of the
satisfied students sought out external resources when their
advisors were unable to meet their needs. These findings
provide some important insights for developing a positive
advisory relationship.
As I was approaching the end of my doctoral training and about
to start my dissertation study, I vividly remember a meeting
with Puncky Heppner, one of my mentors in graduate school.
This meeting stands out clearly to me because his advice on
what I could do as a graduate student to start laying a
foundation for a career in research was so critical at this stage
of my career. He suggested that I consider doing my dissertation
study in an area that could develop into a programmatic line of
research—and that this area be one that sustains my interest
over the years and fills a void in the current literature. This
incident and many other teaching moments from valued mentors
have taught me that the world of research operates on a unique
time frame that requires patience, perseverance, planning, and
good advice from research mentors because we often don't see
immediate outcomes for the behaviors we are engaged in as
early career researchers.
—Lisa Flores, Ph.D.
Research Self-Efficacy Social cognitive models of academic and
career interest and choice hypothesize that self-efficacy
mediates the relationship between direct and vicarious
experiences and attainment of goals (Lent, Brown, & Hackett,
1994). Research self-efficacy is defined as the confidence in
one's ability to successfully complete various research-related

tasks (Kahn & Schlosser, 2014). Research self-efficacy is an
integral part of building interest and shaping one's career goals
(e.g., see Research Journey 2.10). There have been a few
measures developed to assess the research self-efficacy of
counseling students. Among these scales, Self-Efficacy in
Research Measure (SERM; Phillips & Russell, 1994) is one
widely used to measure research self-efficacy (Gelso & Lent,
2000). The SERM includes four subscales measuring the design,
practical, writing, and quantitative aspects of research skills
(see Table 2.2 for a list of the scale items), and can be a tool
used to assess your own research self-efficacy. Kahn (2001),
based on social cognitive models and past research in this area
(see, e.g., Brown et al., 1996; Phillips & Russell, 1994), found
that research training environment and mentorship predict
research self-efficacy, which in turn would predict scholarly
activity.
TABLE 2.2Self-Efficacy in Research Measure (Phillips &
Russell, 1994)
The following items are tasks related to research. Please
indicate your degree of confidence in your ability to
successfully accomplish each of the following tasks on a scale
of 0–9 with 0 representing no confidence and 9 representing
total confidence.
1
2
3
4
5
6
7
8
9
No Confidence

Total Confidence
1.Selecting a suitable topic for study
2.Knowing which statistics to use
3.Getting an adequate number of subjects
4.Writing a research presentation for a conference
5.Writing the method and results section for a research paper
for publication
6.Manipulating data to get it onto a computer system
7.Writing a discussion section for a thesis or dissertation
8.Keeping records during a research project
9.Collecting data
10.Designing an experiment using nontraditional methods (e.g.,
ethnographic, cybernetic, phenomenological approaches)
11.Designing an experiment using traditional methods (e.g.,
experimental, quasi-experimental designs)
12.Making time for research
13.Writing the introduction and literature review for a
dissertation
14.Reviewing the literature in an area of research interest
15.Writing the introduction and discussion sections for a
research paper for publication
16.Contacting researchers currently working in an area of
research interest
17.Avoiding the violation of statistical assumptions
18.Writing the method and results sections of a dissertation
19.Using simple statistics (e.g., t test, ANOVA, correlation)
20.Writing the introduction and literature review for a thesis
21.Controlling for threats to validity
22.Formulating hypotheses
23.Writing the method and results sections of a thesis
24.Utilizing resources for needed help
25.Understanding computer printouts
26.Defending a thesis or dissertation
27.Using multivariate statistics (e.g., multiple regression, factor

analysis)
28.Using statistical packages (e.g., SPSS-X, SAS)
29.Selecting a sample of subjects from a given population
30.Selecting reliable and valid instruments
31.Writing statistical computer programs
32.Getting money to help pay for research
33.Operationalizing variables of interest
Scoring:
Sum items within each subscale for subscale scores, or sum all
33 items for a total score.
Research Design Skills = 1, 10, 11, 21, 22, 29, 30, 33
Practical Research Skills = 3, 8, 9, 12, 16, 24, 26, 32
Quantitative and Computer Skills = 2, 6, 17, 19, 25, 27, 28, 31
Writing Skills = 4, 5, 7, 13, 14, 15, 18, 20, 23
Having been a Taiwanese international student and now a
scholar in the United States, I encountered additional challenges
in my research journey. My research interests started when
working as an assistant to Professor Kuo-Shu Yang in
conducting Chinese indigenous psychology research. I was
intrigued by how much I learned about myself within my
cultural context through this approach as opposed to simply
utilizing findings from Western studies. A first challenge was
how to incorporate the Chinese cultural context into my
research of Chinese populations while conducting the studies in
the United States. I was privileged to have an advisor, Dr.
Robert B. Slaney, who supported me to study perfectionism
across cultures. A second challenge was academic writing with
English as my second language. Writing has not been easy, but I
have learned and improved through the process of simply
writing, revising, rewriting, etc. It also takes reminding myself
that research design, conceptualization, and data analyses are
more essential to conducting quality research than writing
skills. My research efficacy grew through utilizing my
statistical analyses skills, which facilitated a greater interest in

research, and eventually led me to pursue an academic career.
One of the important things I've learned as a developing
researcher is that most scholars have relative strengths and
weaknesses. It is important for me to utilize my strengths and
also have patience with my weaker areas. It has been helpful
when I refocus my attention to growth and development as
opposed to only productivity. Being mentored and normalizing
my learning curve have helped me develop. Also, collaborating
with others whose skill set complement mine has made research
much easier and more fun.
—Kenneth Wang, Ph.D.
Research Outcome Expectations Students' expectation of the
consequences and outcome of conducting research is also a
factor related to their research experience. These expectations
can include how much they perceived conducting research and
impact their future career or job opportunities. Negative
expectations might be related to how involvement in research
may take away from opportunities to enhance their counseling
skills or time for leisure activities. Research outcome
expectations have been found associated with one's level of
interest in research and research self-efficacy (Bishop &
Bieschke, 1998).
Research Interest Kahn (2001) hypothesized that research
interest serves as a mediator between the relationships of
various research-related predictors mentioned previously and
scholarly activity. Put another way, the established relationship
between personality type and research productivity is due to
interest. Investigative types have more interest in research and
therefore have goals related to research and produce more
research products. A multilevel study has also shown that
counseling programs with positive research training
environments had students with stronger research interests
(Kahn & Schlosser, 2010).
Scholarly Activity The outcome construct in Kahn's model
(2001) is scholarly activity, which was defined broadly to
include past accomplishments related to published journal

articles, unpublished empirical manuscripts, manuscript
submission and research in process, presentations presented or
in process, and conferences attended, as well as current
involvement in data collection or data analyses. Research
productivity has a long history as an outcome variable due to
many observations that the research productivity of graduates
from counseling programs is low.
Research productivity as a goal is somewhat controversial.
Increasing research productivity has often been advocated by
leaders in the field. However, we feel that the quantity of
research published by students, before and after they graduate,
should not be the primary indicator of the quality of a training
program. We think students should be skilled researchers who
can conduct quality research (and who do so in their
dissertations), and also intelligently consume research, cogently
think about scientific issues, balance research and practice from
an informed standpoint, appreciate and value the scientific
method, and most important apply critical thinking skills to a
broad range of research and applied contexts. Hill (1997) noted
that even though few of her students had taken academic
positions or produced much research, she should not as a result
be judged as an inadequate research mentor.
Year in Program Kahn (2001) also found that the amount of
research training that a student has received, which was
approximated by the year they were in their program, was
directly associated with their scholarly activity. This is an
important factor to consider as we mentioned about how
research skills take time to develop, and students should take
into account the developmental process as they assess their
research abilities and potential.
RESEARCH IN ACTION 2.1
Complete the Self-Efficacy in Research Measure (SERM)
presented in Table 2.2 to assess your self-efficacy levels in the
four different aspects of research skills. What did you learn
about yourself? How do your results from the SERM relate to

the concerns, obstacle, and fears that you have about scientific
training? What steps can you take now to enhance the
development of your research skills?
Testing the Model
Kahn (2001) tested the path model and the results are presented
in Figure 2.1. Several aspects of the model are informative. The
critical mediating construct was research interests along with
research self-efficacy and research outcome expectations. All
four predictive factors (investigative interests, research training
environment, relationship with mentor, and year in program)
were directly or indirectly associated with predicting scholarly
activity. Among these factors, the research training environment
appeared to be most predictive of scholarly activity through
research outcome expectations and self-efficacy. The
relationship with the mentor was significantly associated with
scholarly activity through research self-efficacy. As for year in
program, it was directly associated with scholarly output, which
points to the importance of multiple research experiences. This
study highlights the importance of students having research
interests and self-efficacy, and then provides the kind of
research environment and mentorship that helps to foster those
interests and self-efficacy. See Kahn (2001) and Kahn and Scott
(1997) for more details.
Research Competence: The Missing Construct
From our perspective, the missing construct in the Kahn (2001)
model is research competence. As counselors and counseling
psychologists we appropriately attend to how we structure
environments to foster positive attitudes and self-efficacy.
However, having competencies to conduct and consume
research is also critical to producing quality research. Wampold
(1986b) found that training programs did not teach their
students various statistical procedures that were used in typical
counseling research. In addition, Royalty and Reising (1986)
surveyed professionals and found that respondents felt confident

of their research skills, with the exception of statistical and
computer skills, two skills that were highly correlated with
productivity. Although the respondents indicated that graduate
programs adequately contributed to their skill level, they
became less confident of their skills later in their careers, and
they indicated that it is unlikely that they would augment their
research skills after graduation. Self-efficacy is due in part to a
research environment that fosters positive attitudes toward
research, but it is also built upon competencies.
Based on the empirical evidence for competence in statistics
and design, Wampold (1986b, p. 44) listed the following design
and statistics competencies: (a) knowledge of designs including,
but not limited to, traditional experimental designs, quasi-
experimental designs appropriate for field settings, single-
subject designs, survey designs, and qualitative designs; (b) an
understanding of design issues, such as validity, methods of
sampling, and power; (c) knowledge of the statistical analyses
commonly used in counseling research; (d) an understanding of
statistical issues, such as the role of assumptions, hypothesis-
testing strategies, and confirmatory versus exploratory analyses;
and (e) the ability to perform analyses with computer
assistance, when appropriate. It is hoped that this book,
statistics classes, and other experiences will provide readers
these competencies. In addition, professional writing skills,
which are covered in the last chapter of this book and in a more
detailed way in Heppner and Heppner (2004), are critically
important skills for students to develop.THE NEED TO
BROADEN SCIENTIFIC TRAINING
Most counselor training programs have a number of training
components aimed at promoting the development of an array of
research competencies (e.g., research design, statistics).
However, we want to discuss two additional competencies that
merit attention: basic scientific or critical thinking skills and
basic research application skills.
Training in Basic Scientific Thinking Skills

All counselors, regardless of where their professional activities
fall on the scientist-practitioner continuum, need basic
scientific or critical thinking skills. Scientific thinking refers to
a controlled method of inquiry and reasoning, typically to
collect data of some kind for the purpose of testing a
hypothesis. A crucial characteristic of a professional counselor
is the integration of scientific thinking into the daily activities
of professional practice (e.g., Gambrill, 2005; Pepinsky &
Pepinsky, 1954). Scientific thinking can be instrumental in how
counselors process information about a specific client during
counseling as well as evaluate the counseling process.
How a person thinks is an exceedingly complex process. It is
becoming increasingly clear over time that human beings often
think or process information in both rational and irrational
ways, in systematic and unsystematic ways, and in linear and
nonlinear ways. Moreover, there is reason to believe that one's
cognitive processes (that is, how one thinks) interact with one's
affective processes and behaviors, thereby creating a complex
triadic process (Heppner & Krauskopf, 1987). Research clearly
indicates that people do not think as “objective computers,” but
rather are selective or biased in the type of information to which
they attend (Garb, 1998). In particular, people often attend to
information that confirms their existing beliefs or discount
information that is contrary to their existing beliefs. Moreover,
most therapists and counselors give more weight to their
personal experiences than to science when making decisions
while working with clients (Stewart & Chambless, 2007). Such
biases can lead to problems for professional counselors as they
process information about clients and evaluate the effectiveness
of their work.
Carl Rogers was very aware of this danger of counselor bias; in
1955 he observed that he could “deceive himself in regard to my
creatively formed subjective hunches” about a client (p. 275).
He believed that the scientific method, as a way of thinking, led
him to check “the subjective feelings or hunch or hypothesis of
a person with the objective fact” (p. 275). Rogers would often

check his hunches very directly by asking the client, “Do you
mean this?” or “Could it be this?” Rogers would sometimes go a
step further and develop a written transcript of an interview to
analyze the relationships between counselor and client
statements. Years later his face would still light up with
excitement about what he would learn about a particular client,
or the counseling process, by stepping back from the immediacy
of an interview to analyze those transcripts (Rogers, personal
communication with P. P. Heppner and L. A. Lee, January
1984).
Training in scientific thinking may be particularly important in
evaluating counseling outcomes. Lilienfeld et al. (2014) provide
an illustration of how mental health providers can be biased:
A clinically depressed client obtains psychotherapy; 2 months
later, she is free of serious symptoms. Was her improvement
due to the treatment?
The correct answer is “We don't know.” On the one hand, ample
data demonstrate that scientifically supported psychotherapies
can alleviate many mental health difficulties (Barlow, 2004), so
the client's improvement may well stem at least partly from the
intervention. On the other hand, as most mental health
professionals know, we cannot draw valid conclusions regarding
a treatment's effectiveness in the absence of methodological
safeguards against errors in inference, such as well validated
outcome measures, randomized control groups, and blinded
observations (Gambrill, 2012). Yet even seasoned clinicians and
researchers can easily fall prey to the error of concluding that a
treatment worked when the evidence for this inference is
insufficient. They can commit this mistake when evaluating the
effectiveness of treatment for a given client, the effectiveness
of a specific school or modality of psychotherapy, or both. (p.
355)
Thus, training in scientific thinking and methodologies can help
counselors evaluate the effectiveness of counseling
interventions more objectively and with less personal or
subjective bias.

More recently, there has been a strong movement for
empirically validated treatments or evidence-based practice to
enrich the training and practice of counselors (McHugh &
Barlow, 2010; Norcross, 2011; Waehler, Kalodner, Wampold, &
Lichtenberg, 2000). Scientific thinking is a crucial feature that
distinguishes the counselor from a nonprofessional layperson, in
large part because of its self-corrective characteristics. That is,
whereas laypeople tend to fit what they see or hear into
preconceived notions of their world view, well-trained
counselors should view their assumptions or observations as
subject to change based on scientific evidence (Lilienfeld et al.,
2014).
Thus, the difference between the counselor-as-scientist and the
scientist is more a matter of goals than of procedure. We
believe that it would be worthwhile to pursue more thorough
training in basic scientific thinking skills to reflect training
outcomes associated with a more integrated scientist-
practitioner model. To facilitate the development of scientific
thinking, particularly with practice-oriented students,
Hoshmand (1994) developed a practice-oriented approach to
research supervision; her experiences led her to conclude that a
practice-oriented approach engages students' interest and
prepares them for contributing scholarly research relevant to
practice. At this point, more attention is needed on the efficacy
of different approaches of scientific thinking for different types
of students. Thus, critical thinking, hypothesis testing, and
other scientific methodology should be integrated into all
aspects of training.
Training in Basic Research Application Skills
The counseling research literature contains a great deal of
information about client populations, counseling processes and
outcomes, assessment, interventions, crisis intervention,
professional issues, human behavior, and a variety of special
topics (see APA Handbook of Counseling Psychology edited by
Fouad et al., 2012, for various topics related to the counseling

profession). This information is obviously essential for any
person engaging in counseling research and practice. This
information also can be an extremely useful database or tool in
helping a practitioner solve particular client problems. In a
world that is constantly changing, counselors' knowledge bases
can quickly become out of date. Thus, counselors in the field
need to train themselves continually to work effectively with
new client problems.
The professional research literature is a resource that can both
provide a wealth of useful information about specific client
problems and suggest topics for additional research. Moreover,
reading the literature may affect the counselor's thinking and
refine his or her conceptualizations of the counseling process.
Thus, a basic activity for all graduate students in counseling is
to become active consumers of research. Learning how to use
the research literature (e.g., becoming familiar with the types of
articles in different journals, learning how to access information
most efficiently) and making the literature a part of one's
problem-solving repertoire does not simply happen, but rather
typically involves specific training.SUMMARY AND
CONCLUSIONS
The trainers of future researchers are the students of today. In
this chapter we discuss some basic issues related to research
training, which we hope will help to create positive attitudes
toward research as well as increase research competence not
only for future researchers but also for future research trainers.
We strongly believe that there can be a much greater integration
between the science and practice domains, not only within
counselor training, but also within the professional activities of
all counselors. Carl Rogers's hope for the future was that
researchers would practice in some field and practitioners
would be engaged in some research. This is one type of
integration; we have also suggested that scientific research
training be broadened to emphasize scientific thinking and basic
research application skills. In our personal experience, our
research has clearly been facilitated by observations made

during many hours of counseling practice. Moreover, our
experience has been that our thinking and counseling skills have
been sharpened, extended, and clearly enhanced by subjecting
our thinking to empirical tests as well as contemplating and
applying new ideas from the research literature. Integration
might also happen on another level: between those who are
primarily research oriented and those who are practice oriented.
In some ways, the people who have the most well developed set
of observations of the counseling process are those who are
heavily engaged in counseling practice; it behooves the typical
researcher to develop collaborative relationships with such
practitioners. In short, we suspect that both scientific and
practice-oriented professional activities could be enhanced by a
more complete integration of the two domains.
We have also maintained that a basic issue in scientific training
is scientific thinking. Such an outcome requires not only
scientific skills, but also scientific values. The latter are also
important goals and need specific attention in training (perhaps
by way of discussions about the philosophy of science, research
methods, the slow but steady accumulation of knowledge, and
students' own research experiences).
An attitude of scholarly inquiry goes beyond scientific thinking
and involves curiosity, inquisitiveness, healthy skepticism,
exploration, and a desire to learn (e.g., see Research Journey
2.11). In a way, all counseling professionals are pioneers in
extending the boundaries of their own knowledge throughout
their careers, and possibly extending the knowledge bases of the
profession as well. In this way, scholarly inquiry involves
discovery, excitement, and even a sense of adventure. Not
surprisingly, pioneers in the field of counseling and
development report that they were motivated to achieve, in part,
by the joys of intellectual discovery, “the thirst to know
something” (Pepinsky, cited in Claiborn, 1985, p. 7).
Looking back at my career as a counseling researcher, I find

gratification in observing that foremost my research has been
driven by curiosity. Curiosity is the desire to know something—
to probe, to inquire, to design studies, and to examine data until
a coherent story about the phenomenon under scrutiny appears.
The psychotherapy story is not exactly what I expected but that
is what science is about—discovering something unexpected.
For many years, I believed and taught my students that some
forms of psychotherapy for some problems were more
scientifically sound and more effective than others. Yet, when I
examined the process and outcome evidence about
psychotherapy, the purported scientific bases of psychotherapy
appeared to be fragile. So I set out to conduct meta-analyses
and analyses of large naturalistic psychotherapy data to examine
these issues more fully. Increasingly, evidence from many
different types of designs, in different areas, some from my
research group but much from others, including advocates of
particular treatments, was telling a different story—but what
story was it? It was not enough to observe that the evidence was
not consistent with existing psychotherapy theory. The evidence
demanded an alternative explanation, a new theory that would
parsimoniously explain the current research evidence and
anticipate evidence produced by future research. This led me to
read more widely—evolutionary psychology, anthropology,
social psychology, neuropsychology, placebo studies,
religiosity, and epidemiology. This quest, which was immensely
fascinating, led to the development of the contextual model of
psychotherapy. One has to be humble in such an endeavor
because scientific progress dictates that theories will be
abandoned and replaced by superior ones, not something I look
forward to but inevitable nevertheless.
—Bruce Wampold, Ph.D.
Maintaining a pioneering attitude in professional scholarly
inquiry can also be a way of life. Enos Mills was an early
pioneer in the mountainous area of Colorado that is now Rocky
Mountain National Park. In the following quote, Mills (1924)
aptly describes the qualities of a pioneer's life. Even though he

was referring to pioneering in the American West, the qualities
of life he describes also apply to the pioneering attitude
involved in discovering professional knowledge through
scholarly inquiry:
Those who live pioneer lives are usually the most fortunate of
people. They suffer from no dull existence. Each hour is full of
progressive thought, and occasions which call for action
accompanied by the charm of exploration—action that makes
their lives strong, sincere, and sweet. Their days are full of
eagerness and repose, they work with happy hands. The lives of
pioneers are rich with hope and their future has all the promise
of spring. (p. 9)
We hope that your life in the counseling profession will be a
good place for you to be a pioneer and that scientific thinking
will enrich your life with discoveries that are exciting to you
and beneficial to the various clienteles we serve.
RESEARCH IN ACTION 2.2
After reading about the research joys and challenges of various
researchers, are there particular journeys that stood out for you?
Which aspects of their journey were more helpful and in what
ways?STIMULUS QUESTIONSReflections on Research
Training
1.In what ways do you think scientific thinking might be useful
for both practice and science activities?
2.Identify five activities you can do to incorporate scientific
thinking into your counseling practice.
3.What do you see as the most important advantages of
integrating science and practice in the counseling profession?
4.How do you see yourself in terms of being a researcher and/or
practitioner?
5.What do you feel very passionate about in your future as a
counselor? Can you see any way that research could help you
enhance the work you will do within the areas that excite you
the most?
6.What concerns, obstacles, and fears do you have about

scientific training?
7.Examine the reactions you listed in question 6; why do you
think you have these reactions?
Page 2
Wk 8 Team - Program Evaluation Paper and Presentation
Assignment Content
Develop a program evaluation study based on the following
information about Stoneybrook Middle School:
· Stoneybrook Middle School is currently implementing a new
peer tutoring program.
· The program pays eligible eighth-grade students $5 per hour
to tutor sixth-grade students who are struggling in mathematics.
· Student tutors must have a B+ average or better in math
courses and no history of conduct problems in school.
· Parental permission for tutors and students is required.
· Thus far, 60 eighth-grade tutors have been employed to tutor
75 sixth-grade students. The tutors meet with their sixth-grade
pupils at least one hour per day. They work with the sixth
graders on understanding math concepts and building math
skills.
· The tutors have been trained to build confidence, provide
positive reinforcement, and allow their pupils plenty of time to
think. The tutors act as role models socially and academically.
Tutors also model good study habits such as keeping notebooks
organized, completing homework on time, and preparing
effectively for quizzes and tests.
· The program has been running for 3 months, and it is time for
the first evaluation. Continued funding of the peer tutoring
program is contingent upon demonstration of its effectiveness.
Design an outcome study focusing on the effectiveness of
Stoneybrook Middle School's peer tutoring program. In your

program evaluation design, you must include the following
information:
· Statement of the problem: Introduce the reader to the problem
to be studied. Provide sufficient background information so that
the reader has a grasp of the situation.
· Review of literature: Provide the reader with a review of
salient literature, beginning with general information, and
narrowing the focus to the specific issues under consideration in
the study. Use subheadings to organize your information. This
section should be two to three pages in length. Provide brief
reviews of four or five similar studies.
· Purpose of the study: Clearly state the purpose of the study.
Identify why the study is needed.
· Hypotheses research questions: List these as simple
statements. Make sure they are measurable.
· Definition of terms: Operationally define terms the average
reader may not know or that have a specific meaning in your
study.
· Research methods and procedures: word count 233
· Population: Describe the population sample under study.
· Procedure: Discuss how the study will be carried out.
· Instruments: Describe the specific measurements used to test
each hypothesis research question.
· Data analysis: Describe the statistical tests or procedures that
would be most appropriate to analyze the data produced from
your instruments. Word count 233
· Conclusion: Summarize the problem and the proposed
evaluation strategy. Discuss the advantages of the strategy that
you have developed. Discuss any known limitations of your
program evaluation plan.
Write a 2,450- to 2,800-word program evaluation paper. It must
be formatted like a professional research article according to
appropriate course-level APA guidelines. Include a references
section. Proofread your paper carefully before you submit it.

Create a 10-minute oral presentation of your program evaluation
strategy. You may nominate a spokesperson to deliver the
presentation, or each Learning Team member may present a
section of the overall strategy. The in-class presentation must
be a straightforward discussion of your program evaluation
strategy. You are not to read your written report; think of your
audience as the people who will consider funding your study.
Thus, your presentation should be engaging and compelling.
Complete your oral presentation.
Format your paper according to appropriate course-level APA
guidelines.
Page 2

Page 291LEARNING OBJECTIVES· Discuss the issues created by.docx

More Related Content

Similar to Page 291LEARNING OBJECTIVES· Discuss the issues created by.docx (20)

More from karlhennesey (20)

Recently uploaded (20)

Page 291LEARNING OBJECTIVES· Discuss the issues created by.docx