Rationale and Standards of Evidence in Evidence-Based Practice.docx

Rationale and Standards of Evidence in Evidence-Based
Practice
OLIVER C. MUDFORD, ROB MCNEILL, LISA WALTON
AND KATRINA J. PHILLIPS
What is the purpose of collecting evidence to inform clinical
practice in psychology concerning the effects of psychological
or other interventions? To quote Paul’s (1967) article that has
been cited 330 times before November 4, 2008, it is to
determine the answer to the question: “What treatment, by
whom, is most effective for this individual with that specific
problem, under which set of circumstances?” (p. 111). Another
answer is pitched at a systemic level, rather than concerning
individuals. That is, research evidence can inform health-care
professionals and consumers about psychological and behavioral
interventions that are more effective than pharmacological
treatments, and to improve the overall quality and cost-
effectiveness of psychological health service provision
(American Psychological Association [APA] Presidential Task
Force on Evidence-Based Practice, 2006). The most general
answer is that research evidence can be used to improve
outcomes for clients, service providers, and society in general.
The debate about what counts as evidence of effectiveness in
answering this question has attracted considerable controversy
(Goodheart, Kazdin, & Sternberg, 2006; Norcross, Beutler, &
Levant, 2005). At one end of a spectrum, evidence from
research on psychological treatments can be emphasized.
Research-oriented psychologists have promoted the importance
of scientific evidence in the concept of empirically supported
treatment. Empirically supported treatments (ESTs) are those
that have been sufficiently subjected to scientific research and
have been shown to produce beneficial effects in wellcontrolled
studies (i.e., efficacious), in more natural clinical environments
(i.e., effective), and are the most cost-effective (i.e., efficient)

(Chambless & Hollon, 1998). The effective and efficient criteria
of Chambless and Hollon (1998) have been amalgamated under
the term “clinical utility” (APA Presidential Task Force on
Evidence-Based Practice, 2006; Barlow, Levitt, & Bufka, 1999).
At the other end of the spectrum are psychologists who value
clinical expertise as the source of evidence more highly, and
they can rate subjective impressions and skills acquired in
practice as providing personal evidence for guiding treatment
(Hunsberger, 2007). Kazdin (2008) has asserted that the schism
between clinical researchers and practitioners on the issue of
evidence is deepening. Part of the problem, which suggests at
least part of the solution, is that research had concentrated on
empirical evidence of treatment efficacy, but more needs c01 20
April 2012; 12:43:29 3 Hersen, Michel, and Peter Sturmey.
Handbook of Evidence-Based Practice in Clinical Psychology,
Child and Adolescent Disorders, John Wiley & Sons,
Incorporated, 2012. ProQuest Ebook Central,
http://ebookcentral.proquest.com/lib/ashford-
ebooks/detail.action?docID=817355. Created from ashford-
ebooks on 2017-11-07 11:30:44. Copyright © 2012. John Wiley
& Sons, Incorporated. All rights reserved. to be conducted to
elucidate the relevant parameters of clinical experience. In a
separate dimension from the evidence– experience spectrum
have been concerns about designing interventions that take into
account the uniqueness of the individual client. Each of us can
be seen as a unique mix of levels of variables such as sex, age,
socioeconomic and social status, race, nationality, language,
spiritual beliefs or personal philosophies, values, preferences,
level of education, as well as number of symptoms, diagnoses
(comorbidities), or problem behavior excesses and deficits that
may bring us into professional contact with clinical
psychologists. The extent to which there can be prior evidence
from research or clinical experience to guide individual’s
interventions when these variables are taken into account is
questionable, and so these individual differences add to the mix
of factors when psychologists deliberate on treatment

recommendations with an individual client. Recognizing each of
these three factors as necessary considerations in intervention
selection, the APA Presidential Task Force on Evidence-Based
Practice (2006, p. 273) provided its definition: “Evidence-based
practice in psychology (EBPP) is the integration of the best
available research with clinical expertise in the context of
patient characteristics, culture, and preferences.” The task force
acknowledged the similarity of its definition to that of Sackett,
Straus, Richardson, Rosenberg, and Haynes (2000) when they
defined evidence-based practice in medicine as “the integration
of best research evidence with clinical expertise and patient
values” (p. 1). Available research evidence is the base or
starting point for EBPP. So, in recommending a particular
intervention from a number of available ESTs, the psychologist,
using clinical expertise with collaboration from the client,
weighs up the options so that the best treatment for that client
can be selected. As we understand it, clinical expertise is not to
be considered as an equal consideration to research evidence, as
some psychologists have implied (Hunsberger, 2007). Like
client preferences, the psychologist’s expertise plays an
essential part in sifting among ESTs the clinician has located
from searching the evidence. Best research evidence is
operationalized as ESTs. Treatment guidelines can be developed
following review of ESTs for particular populations and
diagnoses or problem behaviors. According to the APA (APA,
2002; Reed, McLaughlin, & Newman, 2002), treatment
guidelines should be developed to educate consumers (e.g.,
clients and health-care systems) and professionals (e.g., clinical
psychologists) about the existence and benefits of choosing
ESTs for specific disorders over alternative interventions with
unknown or adverse effects. Treatment guidelines are intended
to recommend ESTs, but not make their use mandatory as
enforceable professional standards (Reed et al., 2002). The
declared status, implications, misunderstanding, and misuse of
treatment guidelines based on ESTs continue to be sources of
controversy (Reed et al., 2002; Reed & Eisman, 2006). Our

chapter examines the issues just raised in more detail. We start
with a review of the history and methods of determining
evidencebased practice in medicine because the evidence-based
practice movement started in that discipline, and has led the
way for other human services. Psychologists, especially those
working for children and young people, tend to work
collaboratively with other professionals and paraprofessionals.
Many of these groups of colleagues subscribe to the evidence-
based practice movement through their professional
organizations. We sample those organizations’ views. The
generalizability to psychology of methods for evidence-based
decision making in medicine is questionable, and questioned.
Next we examine criteria for determining the strength of
evidence for interventions in psychology. These criteria are
notably different to those employed in medicine, particularly
concerning the relative value to the evidence base of 4
Overview and Foundational Issues c01 20 April 2012; 12:43:29
Hersen, Michel, and Peter Sturmey. Handbook of Evidence-
Based Practice in Clinical Psychology, Child and Adolescent
Disorders, John Wiley & Sons, Incorporated, 2012. ProQuest
Ebook Central, http://ebookcentral.proquest.com/lib/ashford-
& Sons, Incorporated. All rights reserved. research in
psychology that has employed methods distinct from medicine
(e.g., small-N design research). The controversies concerning
treatment guidelines derived from ESTs are outlined briefly.
The extent to which special considerations exist regarding
treatment selection for children and adolescents is then
discussed. Finally, we highlight some of the aspects of EBPP
that require further work by researchers and clinicians.
EVIDENCE-BASED MEDICINE Evidence-based practice can
be perceived as both a philosophy and a set of problem-solving
steps, using current best research evidence to make clinical
decisions. In medicine, the rationale for clinicians to search for
best evidence when making diagnostic and treatment decisions

is the desire or duty, through the Hippocratic Oath, to use the
optimal method to prevent or cure physical, mental, or social
ailments and promote optimal health in individuals and
populations (Jenicek, 2003). This section of the chapter
provides an overview of the history of evidence-based practice
in medicine (EBM), as well as a description of the current
methodology of EBM. A critical reflection on the process and
evidence used for EBM is also provided, leading to an
introduction of the relevance of evidence-based practice (EBP)
to other disciplines, including psychological interventions.
History of Evidence-Based Practice in Medicine The earliest
origins of EBM can be found in 19th-century Paris, with Pierre
Louis (1787– 1872). Louis sought truth and medical certainty
through systematic observation of patients and the statistical
analysis of observed phenomena; however, its modern origins
and popularity are found much more recently in the advances in
epidemiological methods in the 1970s and 1980s (Jenicek,
2003). One of the key people responsible for the emergence and
growth of EBM, the epidemiologist Archie Cochrane, proposed
that nothing should be introduced into clinical practice until it
was proven effective by research centers, and preferably
through double-blind randomized controlled trials (RCTs).
Cochrane criticized the medical profession for its lack of
rigorous reviews of the evidence to guide decision making. In
1972, Cochrane reported the results of the first systematic
review, his landmark method for systematically evaluating the
quality and quantity of RCT evidence for treatment approaches
in clinical practice. In an early demonstration of the value of
this methodology, Cochrane demonstrated that corticosteroid
therapy, given to halt premature labor in high-risk women,
could substantially reduce the risk of infant death (Reynolds,
2000). Over the past three decades, the methods and evidence
used for EBM have been extended, refined, and reformulated
many times. From the mid-1980s, a proliferation of articles has
instructed clinicians about the process of accessing, evaluating,
and interpreting medical evidence; however, it was not until

1992 that the term evidence-based medicine was formally
coined by Gordon Guyatt and the Evidence-Based Working
Group at McMaster University in Canada. Secondary
publication clinical journals also started to emerge in the early
to mid-1990s, with the aim of summarizing original articles
deemed to be of high clinical relevance and methodological
rigor (e.g., Evidence-Based Medicine, ACP Journal Club,
Evidence-Based Nursing, EvidenceBased Mental Health).
Various guidelines for EBM have been published, including
those from EvidenceBased Working Group (Guyatt et al., 2000),
the Cochrane Collaboration, the National Institute for Clinical
Excellence (NICE), and the British Medical Journal Clinical
Evidence group, to name but a few. David Sackett, one of the
strong proponents and authors in EBM, describes it as an active
Practice 5 c01 20 April 2012; 12:43:29 Hersen, Michel, and
Peter Sturmey. Handbook of Evidence-Based Practice in
Clinical Psychology, Child and Adolescent Disorders, John
Wiley & Sons, Incorporated, 2012. ProQuest Ebook Central,
& Sons, Incorporated. All rights reserved. clinical decision-
making process involving five sequential steps: (1) convert the
patient’s problem into an answerable clinical question; (2) track
down the best evidence to answer that question; (3) critically
appraise the evidence for its validity—closeness to truth;
impact—size of the effect; and applicability— usefulness in
clinical practice; (4) integrate the appraisal with the
practitioner’s clinical expertise and the patient’s unique
characteristics, values, and circumstances; and (5) evaluate the
change resulting from implementing the evidence in practice,
and seek ways to improve (Sackett et al., 2000, pp. 3–4). The
guidelines for EBM are generally characterized by this sort of
systematic process for determining the level of evidence for
treatment choices available to clinicians, while at the same time

recognizing the unique characteristics of the individual’s
characteristics, situation, and context. It is not difficult to see
the inherent benefits of EBM, and health professionals have
been quick to recognize the potential benefit of adopting it as
standard practice. Citing a 1998 survey of UK general
practitioners (GPs), Sackett and colleagues (2000) wrote that
most reported using search techniques, with 74% accessing
evidence-based summaries generated by others, and 84%
seeking evidence-based practice guidelines or protocols. The
process of engaging in EBM requires some considerable
understanding of research and research methods, and there is
evidence that health professionals struggle to use EBM in their
actual practice. For example, Sackett et al. (2000) found that
GPs had trouble using the rules of evidence to interpret the
literature, with only 20% to 35% reporting that they understood
appraising tools described in the guidelines. Clinicians’ ability
to practice EBM also may be limited by lack of time to master
new skills and inadequate access to instant technologies
(Sackett et al., 2000). In addition to these practical difficulties,
there have also been some criticisms of EBM’s dominant
methodology. An early criticism of EBM, and nearly all EBP
approaches, is that it appears to give greatest weight to science
with little attention to the “art” that also underlies the practice
of medicine, nursing, and other allied health professions. For
example, Guyatt and colleagues cited attention to patients’
humanistic needs as a requirement for EBM (EvidenceBased
Medicine Working Group, 1992). Nursing and other health-care
disciplines note that EBP must be delivered within a context of
caring to achieve safe, effective, and holistic care that meets the
needs of patients (DiCenso, Cullum, Ciliska, & Guyatt, 2004).
Evidence-based practice is also criticized as being “cookbook
care” that does not take the individual into account. Yet a
requirement of EBP is that incorporating research evidence into
practice should consistently take account of the patient’s unique
circumstances, preferences, and values. As noted by Sackett et
al. (2000), when these three elements are integrated to inform

clinical decision making, “clinicians and patients form a
diagnostic and therapeutic alliance which optimizes clinical
outcomes and quality of life” (p. 1). One of the key issues in the
use of EBM is the debate around what constitutes best evidence
from the findings of previous studies and how these various
types of evidence are weighted, or even excluded, in the
decisionmaking process. The following section first outlines the
way in which evidence currently tends to be judged and then
provides a more indepth analysis of each type of evidence.
Levels and Types of Evidence Partly through the work of Archie
Cochrane, the quality of health care has come to be judged in
relation to a number of criteria: efficacy (especially in relation
to effectiveness), efficiency, and equity. Along with
acceptability, access, and relevance, these criteria have been
called the “Maxwell Six” and have formed the foundation of
decision making around service provision and funding in the
United Kingdom’s National Health Service (Maxwell, 1992).
Other health systems 6 Overview and Foundational Issues c01
20 April 2012; 12:43:29 Hersen, Michel, and Peter Sturmey.
& Sons, Incorporated. All rights reserved. around the world
have also adopted these criteria to aid in policy decision
making, in both publicly and privately funded settings. Despite
this there has been a tendency for evidence relating to
effectiveness to dominate the decision-making process in EBM
and, in particular, effectiveness demonstrated through RCTs.
There has been considerable debate about the ethical problems
arising from clinicians focusing too much on efficacy and not
enough on efficiency when making decisions about treatment
(Maynard, 1997). In any health system with limited resources,
the decision to use the most effective treatment rather than the

most efficient one has the potential to impact on the likelihood
of having sufficient resources to deliver that treatment, or any
other, in the future. By treating one person with the most
effective treatment the clinician may be taking away the
opportunity to treat others, especially if that treatment is very
expensive in relation to its effectiveness compared to less
expensive treatment options. According to current EBM
methods, the weight that a piece of evidence brings to the
balance of information used to make a decision about whether a
treatment is supported by evidence can be summarized in Table
1.1. Although there are subtle variations in this hierarchy
between different EBM guidelines and other publications, they
all typically start with systematic reviews of RCTs at the top
and end with expert opinion at the bottom. Before discussing
systematic reviews, metaanalyses, and clinical guidelines, it is
important to understand the type of study that these sources of
evidence are typically based on: the RCT. RCTs are a research
design in which the participants are randomly assigned to
treatment or control groups. RCTs are really a family of
designs, with different components such as blinding
(participants and experimenter), different randomization
methods, and other differences in design. Analysis of RCTs is
quantitative, involving estimation of the statistical significance
or probability of the difference in outcomes observed between
the treatment and control groups in the study. The probabilities
obtained are an estimate of the likelihood of that size
difference, or something larger, existing in the population.
There are a number of international databases that store and
provide information from RCTs, including the Cochrane Central
Register of Controlled Trials (CENTRAL;
mrw.interscience.wiley.com/cochrane/cochrane
_clcentral_articles_fs.html), OTseeker (www .otseeker.com),
PEDro (www.pedro.fhs.usyd .edu.au), and the Turning Research
Into Practice (TRIP; www.tripdatabase.com) database. Another
effort to increase the ease with which clinicians can access and
use evidence from RCTs has come through efforts to

standardize the way in which they are reported, such as the
CONSORT Statement and other efforts from the CONSORT
Group (www.consortstatement.org). The major strength of well-
designed RCTs and other experimental designs is that the
researcher has control over the treatment given to the
experimental groups, and also has control over or the ability to
control for any confounding factors. The result of this control
TABLE 1.1 Typical Hierarchy of Evidence for EBM Level of
Evidence Type of Evidence 1 (High) Systematic reviews or
meta-analysis of randomized controlled trials (RCTs) OR
Evidence-based clinical practice guidelines based on systematic
reviews of RCTs 2 At least one well-designed RCT 3 Well-
designed quasi-experimental studies 4 Well-designed case
control and cohort studies 5 Systematic reviews of descriptive
and qualitative studies 6 A single descriptive or qualitative
study 7 (Low) The opinion of authorities and/or expert
committees Source: Adapted from Melnyk and Fineout-Overholt
(2005, p. 10) Rationale and Standards of Evidence in Evidence-
Based Practice 7 c01 20 April 2012; 12:43:29 Hersen, Michel,
and Peter Sturmey. Handbook of Evidence-Based Practice in
& Sons, Incorporated. All rights reserved. is the ability,
arguably above all other designs, to infer causation—and
therefore efficacy— from the differences in outcomes between
the treatment and control groups. RCTs, and other purely
experimental designs, are often criticized for having poor
ecological validity, where the conditions of the experiment do
not match the conditions or populations in which the treatment
might be delivered in the real world. Low ecological validity
can, although does not necessarily, lead to low external validity
where the findings of the study are not generalizable to other
situations, including the real world. Another issue with RCTs is

that they are relatively resource intensive and this leads to
increased pressure to get desired results or to suppress any
results that do not fit the desired outcome (Gluud, 2006; Simes,
1986). In social sciences, RCTs are not always possible or
advisable, so many systematic reviews and meta-analyses in
these areas have less restrictive inclusion criteria. Commonly,
these include studies that compare a treated clinical group with
an untreated or attention control group (Kazdin & Weisz, 1998;
Rosenthal, 1984). In these situations, a quasiexperimental
design is often adopted. There is also evidence that RCTs are
not necessarily any more accurate at estimating the effect of a
given treatment than quasi-experimental and observational
designs, such as cohort and case control studies (Concato, Shah,
& Horwitz, 2008). Systematic reviews provide a summary of
evidence from studies on a particular topic. For EBM this
typically involves the formation of an expert committee or
panel, followed by the systematic identification, appraisal, and
synthesis of evidence from relevant RCTs relating to the topic
(Melnyk & Fineout-Overholt, 2005). The result of the review is
usually some recommendation around the level of empirical
support from RCTs for a given diagnostic tool or treatment. The
RCT, therefore, is seen as the gold standard of evidence in
EBM. Systematic review and meta-analysis are closely related
EBP methods to evaluate evidence from a body of sometimes
contradictory research to assess treatment quality. There is a
great deal of overlap between methods, stemming in part from
their different origins. The systematic review arises from EBM
and the early work of Cochrane and more recently Sackett and
colleagues (2000). Metaanalysis originated in the 1970s, with
the contributions of Glass (1976) in education and Rosenthal
(1984) in psychology being central. Meta-analyses are a
particular type of systematic review. In a meta-analysis,
measures of the size of treatment effects are obtained from
individual studies. The effect sizes from multiple studies are
then combined using a variety of techniques to provide a
measure of the overall effect of the treatment across all of the

participants in all of the studies included in the analysis. Both
approaches use explicit methods to systematically search,
critically appraise for quality and validity, and synthesize the
literature on a given issue. Thus, a key aspect is the quality of
the individual studies and the journals in which they appear.
Searches ideally include unpublished reports as well as
published reports to counteract the “file drawer” phenomenon:
Published findings as a group may be less reliable than they
seem because studies with statistically nonsignificant findings
are less likely to be published (Rosenthal, 1984; Sackett et al.,
2000). The principal difference between systematic review and
meta-analysis is the latter includes a statistical method for
combining results of individual studies that produces a larger
sample size, reduces random error, and has greater power to
determine the true size of the intervention effect. Not only do
these methods compensate for the limited power of individual
studies resulting in a Type I error, the failure to detect an actual
effect when one is present, they can also reconcile conflicting
results (Rosenthal, 1984; Sackett et al., 2000). The criticisms of
meta-analysis mostly relate to the methodological decisions
made during the process of conducting a meta-analysis, 8
& Sons, Incorporated. All rights reserved. often reducing the
reliability of findings. This can result in meta-analyses of the
same topic yielding different effect sizes (i.e., summary
statistic), with these differences stemming from differences in
the meta-analytic method and not just differences in study
findings (Flather, Farkouh, Pogue, & Yusuf, 1997).
Methodological decisions that can influence the reliability of
the summary statistic produced from a meta-analysis include the

coding system used to analyze studies, how inclusive the study
selection process was, the outcome measures used or accepted,
and the use of raw effect sizes or adjusted sample sizes (Flather
et al., 1997). Some meta-analyses are conducted without using a
rigorous systematic review approach, and this is much more
likely to produce a mathematically valid but clinically invalid
conclusion (Kazdin & Weisz, 1998; Rosenthal, 1984). There are
also some criticisms of the meta-analysis approach, specifically
from child and adolescent psychology literature. Meta-analyses
can obscure qualitative differences in treatment execution such
as investigator allegiance (Chambless & Hollon, 1998; Kazdin
& Weisz, 1998) and are limited by confounding among
independent variables such as target problems, which tend to be
more evident with certain treatment methods (Kazdin & Weisz,
1998). Evidence-based clinical practice guidelines are also
included in this top level of evidence, with the caveat that they
must be primarily based on systematic reviews of RCTs. The
purpose of these practice guidelines is to provide an easy-to-
follow tool to assist clinicians in making decisions about the
treatment that is most appropriate for their patients (Straus,
Richardson, Glasziou, & Haynes, 2005). There are some issues
with the publication and use of evidence-based clinical practice
guidelines. One problem is that different groups of experts can
review the same data and arrive at different conclusions and
recommendations (Hadorn, Baker, Hodges, & Hicks, 1996).
Some guidelines are also criticized for not being translated into
tools for everyday use. One of the major drawbacks of clinical
guidelines is that they are often not updated frequently to
consider new evidence (Lohr, Eleazer, & Mauskopf, 1998). In
addition to individual researchers and groups of researchers,
there are a large number of organizations that conduct
systematic reviews, publish clinical practice guidelines, and
publish their findings in journals and in databases on the
Internet, including the Cochrane Collaboration
(www.cochrane.org/), the National Institute for Clinical
Evidence (NICE; www.nice.org.uk/), the Joanna Briggs Institute

(www.joannabriggs.edu.au/about /home.php), and the TRIP
database (www .tripdatabase.com/index.html). For example, the
Cochrane Collaboration, an organization named after Archie
Cochrane, is an international network of researchers who
conduct systematic reviews and meta-analyses and provide the
results of these to the research, practice, and policy community
via the Cochrane Library. The overall strengths of systematic
reviews, meta-analyses, and clinical practice guidelines relate
partly to the nature of the methods used and partly to the nature
of RCTs, which were discussed earlier. The ultimate goal of
scientific research is to contribute toward a body of knowledge
about a topic (Bowling, 1997). Systematic reviews and clinical
practice guidelines are essentially trying to summarize the body
of knowledge around a particular treatment, which is something
that would otherwise take a clinician a very long time to do on
their own, so it is easy to see the value in a process that does
this in a rigorous and systematic way. There are a number of
overall weaknesses for this top level of evidence for EBM. For
example, the reliability of findings from systematic reviews has
been found to be sensitive to many factors, including the
intercoder reliability procedures adopted by the reviewers
(Yeaton & Wortman, 1993). It is possible for different people
using the same coding system Rationale and Standards of
Evidence in Evidence-Based Practice 9 c01 20 April 2012;
12:43:29 Hersen, Michel, and Peter Sturmey. Handbook of
Evidence-Based Practice in Clinical Psychology, Child and
Adolescent Disorders, John Wiley & Sons, Incorporated, 2012.
ProQuest Ebook Central,
& Sons, Incorporated. All rights reserved. to come up with
different conclusions about the research evidence for a
particular treatment. The selection of comparable studies is also
a major issue, and especially in matching comparison groups
and populations. The control and treatment groups in RCTs and

other studies are usually not identical and this raises issues,
particularly in the use of meta-analyses, where the treatment
effects are being combined (Eysenck, 1994). The mixing of
populations can also cause problems, where the treatment may
only be effective in one or more specific populations but the
review has included studies from other populations where the
treatment is not effective. The overall effect of mixing these
populations is to diminish the apparent effectiveness of the
treatment under review (Eysenck, 1994). Other issues with this
top level of evidence relate to the information not included.
Once a review or guideline is completed it must be constantly
updated to check for new evidence that may change the
conclusions. There are also issues around the information not
included in the review process, including nonRCT studies, gray
literature, such as technical reports and commissioned reports,
and unpublished studies. Studies that do not get into peer-
reviewed journals, and therefore usually get excluded from the
review process, are often those that did not have significant
results. The effect of this is that reviews will often exaggerate
the effectiveness of a treatment through the exclusion of studies
where the treatment in question was found to be ineffective. For
this reason there has been a recent move to introduce what is
called the “failsafe N” statistic. This is the hypothetical number
of unpublished (or hidden) studies showing, on average, no
effect that would be required to overturn the statistically
significant effects found from review of the published (or
located) studies results (Becker, 2006). Quasi-experimental
designs are the same as RCTs but the groups are not randomly
assigned, there is no control group, or they lack one or more of
the other characteristics of an RCT. In many situations
randomization of participants or having a control group is not
practically and/or ethically possible. The strengths of quasi-
experimental designs come from the degree of control that the
research has over the groups in the study, and over possible
confounding variables. A well-designed quasiexperimental
design has the important characteristic of being able to infer

some degree of causation from differences in outcomes between
study groups. Case control studies identify a population with
the outcome of interest (cases) and a population without that
outcome (controls), then collects retrospective data to try to
determine their relative exposure to factors of interest (Grimes
& Schulz, 2002). There are numerous strengths of case control
studies. They have more ecological validity than experimental
studies and they are good for health conditions that are very
uncommon (Grimes & Schulz, 2002). There is a relatively clear
temporal sequence, compared to lower level evidence, which
allows some degree of causality to be inferred (Grimes &
Schulz, 2002). They are also relatively quick to do, relatively
inexpensive, and can look at multiple potential causes at one
time (Grimes & Schulz, 2002). The weaknesses of case control
studies include the inability to control potentially confounding
variables except through statistical manipulations and the
reliance on participants to recall information or retrospectively
collating information from existing data; the choice of control
participants is also difficult (Grimes & Schulz, 2002;
Wacholder, McLaughlin, Silverman, & Mandel, 1992). All of
this often leads to a lot of difficulty in correctly interpreting the
results of case control studies (Grimes & Schulz, 2002). Cohort
studies are longitudinal studies where groups are divided in
terms of whether they receive or do not receive a treatment or
exposure of interest, and are followed over time to assess the
outcomes of interest (Roberts & Yeager, 2006). The strengths of
cohort studies include the relatively clear 10 Overview and
Foundational Issues c01 20 April 2012; 12:43:29 Hersen,
Michel, and Peter Sturmey. Handbook of Evidence-Based
Practice in Clinical Psychology, Child and Adolescent
& Sons, Incorporated. All rights reserved. temporal sequence
that can be established between the introduction of an

intervention and any subsequent changes in outcome variables,
making the establishment of causation possible, at least to some
degree. The limitations of cohort studies include the difficulty
in controlling extraneous variables, leading to a relatively
limited ability to infer causality. They are also extremely
resource intensive and are not good where there is a large gap
between treatment and outcome (Grimes & Schulz, 2002).
Descriptive studies involve the description of data obtained
relating to the characteristics of phenomena or variables of
interest in a particular sample from a population of interest
(Melnyk & Fineout-Overholt, 2005). Correlational studies are
descriptive studies where the relationship between variables is
explored. Qualitative studies collect nonnumeric data, such as
interviews and focus groups, with the analysis usually involving
some attempt at describing or summarizing certain phenomena
in a sample from a population of interest (Melnyk & Fineout-
Overholt, 2005). Descriptive studies in general sit low down on
the ladder of evidence quality due to their lack of control by the
researcher and therefore their inability to infer causation
between treatment and effect. On the other hand, much research
is only feasible to conduct using this methodology. Qualitative
research in health services mostly arose from a desire to get a
deeper understanding of what the quantitative research finding
meant for the patient and provider (Mays & Pope, 1996). It asks
questions such as “How do patients perceive . . . ?” and “How
do patients value the options that are offered?” Expert opinion
is material written by recognized authorities on a particular
topic. Evidence from these sources has the least weight in EBM,
although to many clinicians the views of experts in the field
may hold more weight than the higher level evidence outlined
above. Despite its criticisms and methodological complexity,
EBM is seen by most clinicians as a valid and novel way of
reasoning and decision making. Its methods are widely
disseminated through current clinical education programs at all
levels throughout the world. As proposed by Gordon Guyatt and
colleagues in 1992, EBM has led to a paradigm shift in clinical

practice (Sackett et al., 2000). CURRENT STATUS OF EBP
MOVEMENTS ACROSS HEALTH AND EDUCATION
PROFESSIONS Evidence-based practice was initially in the
domain of medicine, but now most human service disciplines
subscribe to the principles of EBP. For example, EBP or the use
of empirically supported treatment is recommended by the APA,
Behavior Analyst Certi- fication Board (BACB), American
Psychiatric Association, National Association of Social
Workers, and General Teaching Council for England. For U.S.
education professionals, EBP has been mandated by law. The
No Child Left Behind Act of 2001 (NCLB; U.S. Department of
Education, 2002) was designed to make the states, school
districts, principals, and teachers more answerable for the
performances shown by the students for whom they were
providing education services. Along with an increase in
accountability, the NCLB requires “schoolwide reform and
ensuring the access of children to effective, scientifically based
instructional strategies and challenging academic content” (p.
1440). The advent of the NCLB and the resulting move toward
EBP occurred because, despite there being research conducted
on effective and efficient teaching techniques (e.g., Project
Follow Through: Bereiter & Kurland, 1981; Gersten, 1984) and
a growing push for accountability from the public (Hess, 2006),
this seldom was translated into actual practice. Education
appears to have been particularly susceptible to implementing
programs that were fads, based on little more than personal
ideologies and good marketing. Much money and time has been
lost by school districts adopting programs that have no

& Sons, Incorporated. All rights reserved. empirical support for
their effectiveness, such as the adoption of known ineffective
substance abuse prevention programs (Ringwalt et al., 2002) or
programs that are harmful for some students and their families,
such as facilitated communication (Jacobson, Mulick, &
Schwartz, 1995). This leads not only to resource wastage, but an
even greater cost in lost opportunities for the students involved.
It seems like common sense for EBP to be adopted by other
disciplines, as it is difficult to understand why reputable
practitioners, service providers, and organizations would not
want to provide interventions that have been shown to be the
most effective and efficient. Yet, despite this commonsense
feeling, the codes of conduct and ethical requirements, and
mandated laws, many disciplines (including medicine) have
found it difficult to bridge the gap between traditional
knowledge-based practice and the EBP framework (Greenhalgh,
2001; Stout & Hayes, 2004). Professions such as social work
(Roberts & Yeager, 2006; Zlotnick & Solt, 2006), speech
language therapy (Enderby & Emerson, 1995, 1996),
occupational therapy (Bennett & Bennett, 2000), and education
(Shavelson & Towne, 2002; Thomas & Pring, 2004) have all
attempted to overcome the difficulties that have arisen as they
try to move EBP from a theoretical concept to an actual usable
tool for everyday practitioners. Although some of these
disciplines have unique challenges to face, many are common to
all. Many human services disciplines have reported that one of
the major barriers to the implementation of EBP is the lack of
sound research. For example, speech language therapy reports
difficulties with regard to the quality and dissemination of
research. A review of the status of speech language therapy
literature by Enderby and Emerson (1995, 1996) found that
there was insufficient quality research available in most areas of
speech language therapy; however, they did find that those
areas of speech therapy that were associated with the medical
profession (e.g., dysphasia and cleft palate) were more likely to
have research conducted than those associated with education

(e.g., children with speech and language disorders and
populations with learning disabilities). There continues to be a
lack of agreement on speech language therapies effectiveness
(Almost & Rosenbaum, 1998; Glogowska, Roulstone, Enderby,
& Peters, 2000; Robertson & Weismer, 1999). In addition to the
lack of research, Enderby and Emerson were also concerned
about the manner in which health resources were being
allocated for speech language therapists. They found that of the
resources allocated to speech language therapy approximately
70% were being used with children with language impairments,
despite there being limited quality evidence of the effectiveness
of speech language therapy with the population at that time.
They also identified that dysarthria was the most commonly
acquired disorder, but had very little research outside of the
Parkinson’s disease population. So, again, many resources have
been allocated to programs that have no evidence of
effectiveness. This questionable allocation of resources is seen
in other fields also. For example, a number of studies have
found that people seeking mental health treatments are unlikely
to receive an intervention that would be classified as EBP and
many will receive interventions that are ineffective (Addis &
Krasnow, 2000; Goisman, Warshaw, & Keller, 1999). A number
of organizations and government agencies within a variety of
fields try to facilitate the much-needed research. Examples in
education include the United States Department for Children,
Schools, and Families (previously Department for Education
and Skills), the National Research Council (United Kingdom),
and National Teacher Research Panel (United Kingdom).
Similarly, in social work the National Association of Social
Workers (United States) and the Society for Social Work and
Research (United States) both facilitate the gathering of
evidence to support the use of EBP within their field; 12

& Sons, Incorporated. All rights reserved. however, even if
these organizations generate and gather research demonstrating
the effectiveness of interventions, this does not always translate
into the implementation of EBP. As mentioned previously,
resource allocation does not necessarily go to those treatments
with evidence to support them due to a lack of effective
dissemination on the relevant topics. Despite medicine being the
profession with the longest history of EBP, Guyatt et al. (2000)
stated that many clinicians did not want to use original research
or they fail to do so because of time constraints and/or lack of
understanding on how to interpret the information. Rosen,
Proctor, Morrow-Howell, and Staudt (1995) found that social
workers also fail to consider empirical research when making
research decisions. They found that less than 1% of the practice
decisions were justified by research. Although the lack of
research-based decision making is concerning, many professions
are attempting to disseminate information in a manner that is
more user friendly to its practitioners. Practice or treatment
guidelines have been created to disseminate research in a
manner that will facilitate EBP. These guidelines draw on the
empirical evidence and expert opinion to provide specific best-
practice recommendations on what interventions or practices are
the most effective/efficient for specific populations (Stout &
Hayes, 2004). There are a number of guideline clearinghouses
(National Guidelines Clearinghouse [www.guideline.gov/] and
What Works Clearinghouse [ies.ed.gov/ncee/wwc/]).
Associations and organizations may also offer practice
guidelines (e.g., American Psychiatric Association). Although
there appears to have been emphasis placed on EBP, there is
still much work to be done in most human service fields,
including medicine, to ensure that there is appropriate research
being conducted, that this research is disseminated to the
appropriate people, and that the practitioners then put it into

practice. Reilly, Oates, and Douglas (2004) outlined a number
of areas that the National Health Service Research and
Development Center for Evidence-Based Medicine had
identified for future development. They suggest that there is a
need for a better understanding of how practitioners seek
information to inform their decisions, what factors influence the
inclusion of this evidence into their practice, and the value
placed, both by patients and practitioners, on EBP. In addition,
there is a need to develop information systems that facilitate the
integration of evidence into the decision-making processes for
practitioners and patients. They also suggest that there is a need
to provide effective and efficient training for frontline
professionals in evidence-based patient care. Finally, they
suggest that there is simply a need for more research.
EVIDENCE IN PSYCHOLOGY Although RCT research is still
held up as the gold standard of evidence in medical science, in
other clinical sciences, especially psychology, best research
evidence has been less focused on RCTs as being the only
scientifically valid approach. Randomized controlled trials are
still held in high regard in clinical psychology, but it has long
been recognized that alternative designs may be preferred and
still provide strong evidence, depending on the type of
intervention, population studied, and patient characteristics
(APA Presidential Task Force on Evidence-Based Practice,
2006; Chambless & Hollon, 1998). Research Methods
Contributing to Evidence-Based Practice in Psychology We
have described and discussed RCTs in the previous section on
EBM. The issues concerning RCTs and their contribution to the
clinical psychology evidence base are similar. The inclusion of
evidence of treatment efficacy and utility from research
paradigms other than RCT methods and approximations thereto
has Rationale and Standards of Evidence in Evidence-Based

& Sons, Incorporated. All rights reserved. been recommended
for clinical psychology since the EBPP movement commenced
(Chambless & Hollon, 1998). The APA Presidential Task Force
on Evidence-Based Practice (2006) findings support the use of
qualitative studies, single-case or small-N experimental designs,
and process-outcome studies or evaluations, in addition to the
RCT and systematic review techniques employed in EBM. A
more in-depth discussion of some of these methods will be
provided. Psychologists and members of other professions that
are more influenced by social science research methods than
medicine may consider that findings from qualitative research
add to their EBP database. Although qualitative research is not
accepted as legitimate scientific study by many psychologists,
and not usually taught to clinical psychologists in training, it
may well have its place in assessing variables regarding clinical
expertise and client preferences (Kazdin, 2008). Research
questions addressable through qualitative research relevant to
EBPP are similar to those mentioned for EBM. Another
important source of scientific evidence for psychologists,
educators, and other nonmedical professionals is that derived
from small-N research designs, also known as single-case,
single-subject, or N ¼ 1 designs. These alternative labels can
confuse, especially since some single-case designs include more
than one subject (e.g., multiple baseline across subjects usually
include three or more participants). The most familiar small-N
designs are ABAB, multiple baseline, alternating treatments,
and changing criterion (Barlow & Hersen, 1984; Hayes, Barlow,
& Nelson-Gray, 1999; Kazdin, 1982). Data from small-N
studies have been included as sources of evidence, sometimes
apparently equivalent to RCTs (Chambless & Hollon, 1998), or
at a somewhat lower level of strength than RCTs (APA, 2002).
Strength of evidence from small-N designs. Small-N designs
can be robust in terms of controlling threats to internal validity;

however, external validity has often been viewed as
problematic. This is because participants in small-N studies are
not a randomly selected sample from the whole population of
interest. The generality of findings from smallN studies is
established by replication across more and more members of the
population of interest in further small-N studies. A hypothetical
example of the process of determining the generality of an
intervention goes like this: (a) A researcher shows that a
treatment works for a single individual with a particular type of
diagnosis or problem; (b) Either the same or another researcher
finds the same beneficial effects with three further individuals;
(c) Another researcher reports the same findings with another
small set of individuals; and so on. At some point, sufficient
numbers of individuals have been successfully treated using the
intervention that generality can be claimed. Within a field such
as a particular treatment for a particular disorder, small-N
studies can be designed so results can be pooled to contribute to
an evidence base larger than N ¼ 1 to 3 or 4 (Lord et al., 2005).
Even if there is an evidence base for an intervention from a
series of small-N studies, every time another individual receives
the same treatment, the clinician in scientistpractitioner role
evaluates the effects of the intervention using small-N design
methods. Thus, every new case is clinical research to determine
the efficacy and effectiveness of this treatment for that
individual. Clinical psychologists can be cautioned that
physicians and medical researchers have a similar-sounding
name for an experimental design with their “N of 1 trials.” The
definition of N of 1 trials can vary somewhat but typically they
are described as “randomised, double blind multiple crossover
comparisons of an active drug against placebo in a single
patient” (Mahon, Laupacis, Donner, & Wood, 1996, p. 1069).
These N of 1 trials are continued until the intervention, usually
a drug, in question shows consistently better effects than its
comparison treatment or control condition. 14 Overview and

Practice in Clinical Psychology, Child and Adolescent
& Sons, Incorporated. All rights reserved. Then the intervention
is continued, or discontinued if there were insufficient
beneficial effects. The N of 1 trials can be considered top of the
hierarchy of evidence in judging strength of evidence for
individual clinical decisions, higher even than systematic
reviews of RCTs (Guyatt et al., 2000). This is because
clinicians do not have to generalize findings of beneficial
effects of the treatment to this patient from previous researched
patients. Consistent beneficial findings from many N of 1 trials
add up to an “N of many” RCT, thus providing evidence of the
generality of findings. An N of 1 trial is quite similar to the
alternating treatments small-N design employed in clinical
psychology, especially applied behavior analysis; however, they
are not identical and the differences may be overcome for some
psychosocial interventions only with careful planning, but many
interventions could not be assessed by fitting them into an N of
1 trial format. Further details are beyond the scope of this
chapter; however, close study of the methodological
requirements of both N of 1 designs and small-N designs can
show the similarities and differences (Barlow & Hersen, 1984;
Guyatt et al., 2000; Hayes et al., 1999; Kazdin, 1982; Mahon et
al., 1996). Some state that small-N designs may be most
suitable for evaluating new treatments (Lord et al., 2005),
others that single-case studies are most suitable for determining
treatment effects for individuals (APA Presidential Task Force
on Evidence-Based Practice, 2006). We do not disagree with
either, except to point out that it has been argued that small-N
studies can contribute much more to EBPP than these two
advantages. In the following section, we review how ESTs can
be determined from an evidence base consisting entirely of
small-N studies. Criteria for Assessing Efficacy Lonigan,

Elbert, and Johnson (1998) tabulated criteria for determining
whether an intervention for childhood disorders could be
considered well-established (i.e., efficacious) or probably
efficacious (i.e., promising). For the former, they recommended
at least two well-conducted RCT standard studies by
independent research teams or a series of independent well-
designed small-N studies with at least nine participants
carefully classified to the diagnostic category of interest
showing that the intervention was better than alternative
interventions. The availability of treatment manuals was
recommended. For “promising” treatments, the criteria were
relaxed to allow nonindependent RCTs, comparing treatment to
notreatment, or a minimum of three small-N studies. These
criteria followed those established at the time for psychological
therapies in general (Chambless & Hollon, 1998). We have
discussed the determination of empirical support from RCTs
already. The next sections will examine how evidence is derived
from small-N studies: First, how evidence is assessed from
individual research reports; and second, how the evidence can
be combined from a group of research articles addressing the
same topic of interest. Evaluating Evidence From Small-N
Research Designs How do those assessing the efficacy of a
treatment from small-N designs measure the strength of the
design for the research purpose and whether, or to what extent,
a beneficial effect has been demonstrated? Chambless and
Hollon (1998) recommended reviewers to rate single-case
studies on the stability of their baselines, use of acceptable
experimental designs, such as ABAB or multiple baseline
designs, and visually estimated effects. Baselines need not
always be stable to provide an adequate control phase; there are
other valid designs (e.g., changing criterion, multielement
experimental designs); and visual estimates of effects are not
necessarily reliable (Cooper, Heron, & Heward, 2007). Validity

& Sons, Incorporated. All rights reserved. and, probably,
reliability of reviews using only Chambless and Hollon’s (1998)
criteria could not be assured. Attempts to improve reliability
and validity of judgments about the value of small-N design
studies have included establishing more welldefined criteria.
Quite detailed methods for evaluating small-N studies have been
published. For example, Kratochwill and Stoiber (2002)
describe the basis of a method endorsed by the National
Association of School Psychologists (NASP) designed for
evaluating multiple research reports that used single-case
designs for establishing evidence-based recommendations for
interventions in educational settings. The system is more
inclusive of design variations beyond those recommended by
Chambless and Hollon (1998), and reviewers are instructed to
code multiple variables, including calculating effects sizes from
graphical data. The coding form extended over 28 pages.
Shernoff, Kratochwill, and Stoiber (2002) illustrated the
assessment procedure and reported that, following extensive
training and familiarity with the coding manual, they achieved
acceptable agreement among themselves. They stated that the
process took 2 hours for a single study, although, so our own
graduate students report, it takes much longer if multiple
dependent and independent variables or hundreds of data points
had been features of the research article with which they chose
to illustrate the NASP procedure. The NASP method was
constructively criticized by Levin (2002), and is apparently
being revised and expanded further (Kratochwill, 2005).
Meanwhile, others have developed what appear to be even more
labor intensive methods for attempting to evaluate objectively
the strength of evidence from a series of smallN designs. As an
example of a more detailed method, Campbell (2003) measured

every data point shown on published graphs from 117 research
articles on procedures to reduce problem behaviors among
persons with autism. Each point was measured by using dividers
to determine the distance between the point and zero on the
vertical axis. Campbell calculated effect sizes for three
variables: mean baseline reduction, percentage of zero data
points, and percentage of nonoverlapping data points. Use of
these statistical methods may have been appropriate considering
the types of data Campbell examined; nonzero baselines of
levels of problem behavior followed by intervention phases in
which the researchers’ goal was to produce reduction to zero
(see Jensen, Clark, Kircher, & Kristjansson, 2007, for critical
review of meta-analytic tools for small-N designs).
Nevertheless, the seemingly arduous nature of the task and the
lack of generalizability of the computational methods to
reviewing interventions designed to increase behaviors are
likely to mitigate wider acceptance of Campbell’s (2003)
method. A final example of methods to evaluate the strength of
an evidence base for interventions is that outlined by
Wilczynski and Christian (2008). They describe the National
Standards Project (NSP), which was designed to determine the
benefits or lack thereof of a wide range of approaches for
changing the behaviors of people with autism spectrum
disorders (ASD) aged up to 21 years. Their methods of
quantitative review enabled the evidence from group and small-
N studies to be integrated. Briefly, and to describe their method
for evaluating small-N studies only, their review rated research
articles based on their scientific merit first. Articles were
assessed to determine whether they were sufficiently
welldesigned in terms of experimental design, measurement of
the dependent variables, assessment of treatment fidelity, the
ability to detect generalization and maintenance effects, and the
quality of the ASD classifi- cation of participants. If the article
exceeded minimum criteria on scientific merit, the treatment
effects were assessed as being beneficial, ineffective, adverse,
or that the data were not sufficiently interpretable to decide on

effects. Experienced trained reviewers were able to complete a
review 16 Overview and Foundational Issues c01 20 April 2012;
Evidence-Based Practice in Clinical Psychology, Child and
& Sons, Incorporated. All rights reserved. for a research article
in 1 hour or less with interreviewer agreement of 80% or more.
Inevitable problems for all these systems for review of single
case designs arise from the necessity of creating “one size fits
all” rules. For example, the evaluative methods of both the
NASP (Kratochwill & Stoiber, 2002) and NSP projects
(Wilczynski & Christian, 2008) include consideration of level
of interobserver agreement for determining partly the quality of
measurement of the dependent variables. The minimum
acceptable level of agreement is specified at, say, 70% or 80%,
which makes it relatively convenient for reviewers to determine
from reading the published research article that they are rating;
however, it has been known for more than 30 years that an
agreement percentage is rather meaningless without examination
of how behaviors were measured, what interobserver agreement
algorithm was employed, and the relative frequency or duration
of the behaviors measured (Hawkins & Dotson, 1975). Another
example concerns coding rules for evaluating the adequacy of
baseline measures of behavior. Whether the criterion for a
baseline phase of highest scientific merit is a minimum of 3
points (NASP) or 5 points (NSP), there will be occasions when
the rule should be inapplicable. For example, in Najdowski,
Wallace, Ellsworth, MacAleese, and Cleveland (2008), after
more than 20 observational sessions during intervention
showing zero severe problem behavior, a return to baseline in
an ABAB design raised the rate of problem behavior to more
than four per minute, which was higher than any points in the

first A-phase. To continue with the baseline to meet NASP or
NSP evaluative criteria would have been unnecessary to show
experimental control and dangerous for the participant. These
examples indicate why more detailed and codified methods for
reviewing small-N studies quantitatively are, as yet, not firmly
established. Although there may be satisfactory methods for
fitting RCTs’ scientific merit and size of effects into databases
for metaanalyses, that appears to be more problematic with
small-N designs, given their flexibility in use (Hayes et al.,
1999). Volume of Evidence From Small-N Studies Required to
Claim That an Intervention Is Evidence Based Having rated the
strength of evidence from individual research articles, the next
step is to determine whether a group of studies on the same
topic between them constitute sufficient evidence to declare that
an intervention is an empirically supported treatment or a
promising or emerging intervention. We discuss only
empirically supported treatment criteria here. Consensus on
what minimum criterion should apply has yet to be reached.
Chambless and Hollon (1998) originally recommended a
minimum of two independent studies with three or more
participants (N $ 3) showing good effects for a total of N $ 6
participants. Lonigan et al. (1998) required three studies with N
$ 3; that is, beneficial effects shown for N $ 9 participants.
Since then, the bar has been raised. For instance, Horner et al.
(2005) proposed that the criteria for determining that an
intervention is evidence-based included a minimum of five
small-N studies, from three or more separate research groups,
with at least 20 participants in total. Wilczynski and Christian
(2008) used similar criteria with $ 6 studies of strongest
scientific merit totaling N $ 18 participants with no conflicting
results from other studies of adequate design. Others have
recommended similar standards: Reichow, Volkmar, and
Cicchetti (2008) described a method for evaluating research
evidence from both group and small-N designs, as had
Wilczynski and Christian (2008). Finally, Reichow et al. (2008)
set the criterion for an established EBP at $ 10 small-N studies

of at least “adequate report strength” across three different
locations and three different research teams with a total of 30
participants, or, if at least five studies of “strong report
strength” with a total of 15 or more participants existed, that
could substitute for the 10-study criterion. Rationale and
Standards of Evidence in Evidence-Based Practice 17 c01 20
April 2012; 12:43:30 Hersen, Michel, and Peter Sturmey.
& Sons, Incorporated. All rights reserved. The rationale for
selecting the numerical criteria is usually not stated. Thus, all
we can state is that some systems (e.g., Reichow et al., 2008)
are more conservative than others (e.g., Lonigan et al., 1998).
Sometimes conservative criteria may be appropriate, for
example, when the costs of treatment are high, and/or the
intervention is exceedingly complex requiring highly skilled
intervention agents, and/or the benefits for the intervention are
less than ideal (i.e., it reduces problems to a more manageable
level, but does not eliminate them), and/or some negative side
effects have been observed. On the other hand, if relatively few
resources are required to implement an effective and rapid
intervention without unwanted side effects, fewer well-
conducted studies may be needed to persuade consumers that
the intervention is empirically supported, and therefore worth
evaluating with the individual patient. This brief discussion
should advise readers to examine and consider criteria carefully
when reviewers claim that a particular intervention is an
empirically supported treatment for a particular disorder or
problem for a particular population. The discussions on
evidence from small-N designs have left much out. For instance,
Reichow et al. (2008) and Wilczynski and Christian (2008)
developed algorithms for assessing the strength of evidence at

different levels, although we have outlined only the highest
levels. Both groups have also reported algorithms for
determining ESTs from mixed methods (e.g., RCTs and small-
Ns). Wilczynski and Christian (2008) report rules for
incorporating conflicting results into the decision-making
process about overall strength of evidence (De Los Reyes &
Kazdin, 2008). Level of Specificity of Empirically Supported
Treatments The issue to be discussed next concerns the unit of
analysis of the research evidence. We illustrate by examining an
example provided by Horner et al. (2005) in which they
assessed the level of support for functional communication
training (FCT; Carr & Durand, 1985). Functional
communication training is an approach to reducing problem
behaviors that teaches an appropriate nonproblematic way for
individuals to access reinforcers for the problem behavior that
have been identified through functional assessment. Horner and
colleagues cited eight published research reports that included
42 participants who had benefited in FCT studies across five
research groups. The evidence was sufficient in quantity for it
to be concluded that FCT is an empirically supported treatment,
exceeding all criteria reviewed earlier except that two more
studies would have been needed to reach the number of studies
$ 10 by the criteria of Reichow et al. (2008). It might
reasonably be asked: “For which population is FCT beneficial?”
Perusal of the original papers cited by Horner et al. (2005)
shows that 21/42 participants’ data were reported in one of the
eight cited studies (Hagopian, Fisher, Sullivan, Acquisto, &
LeBlanc, 1998), with the oldest participant being 16 years old,
and none reported to have a diagnosis of autism. Thus, applying
Horner et al.’s criteria, it cannot be concluded from the studies
cited that FCT is an empirically supported treatment for
participants with autism or for participants older than 16,
regardless of diagnosis. As an aside, eight participants across
the other seven studies were reported to have autism, and three
participants in total were aged over 20 years; however, the
literature on FCT that Horner et al. (2005) included did not

appear to have been obtained from a systematic search, so it is
possible that there has been sufficient research to show that
FCT is an empirically supported treatment for subgroups, and
perhaps autism and adults are two of those. TREATMENT
GUIDELINES Treatment guidelines specifically recommend
ESTs to practitioners and consumers. Alternative descriptors are
clinical practice and 18 Overview and Foundational Issues c01
& Sons, Incorporated. All rights reserved. best practices
guidelines (Barlow, Levitt, & Bufka, 1999). The view of APA
was that guidelines are not intended to be mandatory,
exhaustive, or definitive . . . and are not intended to take
precedence over the judgment of psychologists. APA’s official
approach to guidelines strongly emphasizes professional
judgment in individual patient encounters and is therefore at
variance with that of more ardent adherents to evidence-based
practice. (Reed et al., 2002, p. 1042) It is apparent that many
health-care organizations, insurance companies, and states in
the United States interpret the purpose of lists of ESTs and
treatment guidelines differently (Gotham, 2006; Reed &
Eisman, 2006). They can interpret guidelines as defining what
treatments can be offered to patients and, via manualization,
exactly how treatment is to be administered, by whom, and for
how long. The requirement for manualization allows funders of
treatment to specify a standard reimbursement for the treatment
provider. Thus, the empirically supported treatment movement
was embraced by governments and health-care companies as it
was anticipated to be a major contributor to controlling
escalating health-care costs. Many practicing psychologists
were less enthusiastic about the empirically supported treatment

and EBPP movements (see contributions by clinicians in
Goodheart et al., 2006; Norcross et al., 2005). General concerns
included that requirements to use only ESTs restrict
professionalism by reframing psychologists as technicians going
by the book mechanically; restricting client choice to effective
interventions that have been granted empirically supported
treatment status higher than others, only because they, like
drugs, are relatively easy to evaluate in the RCT format.
Prescription of one-size-fits-all ESTs may further disadvantage
minorities, and people with severe and multiple disorders for
whom there is scant evidence available. There were also
concerns that the acknowledged importance of clinical
expertise, such as interpersonal skills to engage the client
(child) and significant others (family) in a therapeutic
relationship, would be ignored. Contrary to the pronouncements
from the APA (2002, 2006), guidelines have been interpreted or
developed that “assume the force of law” in prescribing some
interventions and proscribing others (Barlow et al., 1999, p.
155). Compulsion of psychologists in practice to follow
treatment guidelines has been reported to occur in the United
States by giving immunity from malpractice lawsuits to those
who use only ESTs, and increasing the vulnerability of those
who do not to litigation and increased professional indemnity
insurance (Barlow et al., 1999; Reed et al., 2002). Some
guidelines, especially those produced by agencies or companies
employing psychologists, have been viewed as thinly veiled
cost-cutting devices justified with a scientistic gloss. Ethical
Requirements For many human service professional
organizations, EBP and ESTs have become an ethical
requirement. The APA’s Ethical Principles of Psychologists and
Code of Conduct document mentions the obligation to use some
elements of EBPP; for example, “Psychologists’ work is based
upon established scientific and professional knowledge of the
discipline” (American Psychological Association, 2010, p. 5).
Other professional groups appear to be more prescriptive with
regard to EBP. The BACB’s Code for Responsible Conduct, for

example, recommends EBP with statements such as, “Behavior
analysts rely on scientifically and professionally derived
knowledge when making scientific or professional judgments in
human service provision” (Behavior Analyst Certification
Board, 2004, p. 1). The BACB also require the use of ESTs: a.
The behavior analyst always has the responsibility to
recommend scientifically Rationale and Standards of Evidence
in Evidence-Based Practice 19 c01 20 April 2012; 12:43:30
& Sons, Incorporated. All rights reserved. supported most
effective treatment procedures. Effective treatment procedures
have been validated as having both long-term and short-term
benefits to clients and society. b. Clients have a right to
effective treatment (i.e., based on the research literature and
adapted to the individual client). c. Behavior analysts are
responsible for review and appraisal of likely effects of all
alternative treatments, including those provided by other
disciplines and no intervention. (Behavior Analyst Certification
Board, 2004, p. 4) As shown by the previous examples, each of
these organizations have variations in how they have included
EBP and empirically supported treatment into their codes of
conduct and ethical statements; however, they both include the
basic tenets of EBP: combining the best research information
with clinical knowledge and the preferences of the individuals
involved. CHILDREN, ADOLESCENTS, AND EVIDENCE-
BASED PRACTICE IN PSYCHOLOGY Despite there being
ESTs for a number of disorders and problem behaviors
manifesting in children and adolescents, there are more than
500 treatments in use with children (Kazdin, 2008), most of
which are unresearched. Chambless and Ollendick (2001) listed
108 ESTs for adults, compared with 37 for children, suggesting

that research on therapies for young people is lacking relative to
adults. Further research with child and adolescent populations
has been prioritized by APA (APA Presidential Task Force on
Evidence-Based Practice, 2006). Researching the effects of
treatments for children brings special difficulties (Kazdin &
Weisz, 1998). Regarding practice, children do not typically
selfrefer for mental, psychological, or behavioral disorders, nor
are they active seekers of ESTs or preferred interventions.
Parents or guardians tend to take those roles, either
independently or following recommendations from family, or
health or education professionals. Children and youth cannot
legally provide informed consent for treatments or for
participation in research studies. These are among the
developmental, ethical, and legal factors that affect
consideration of EBPP with children. Lord et al. (2005)
discussed the challenges of acquiring rigorous evidence
regarding efficacy of treatments for children with complex,
potentially chronic behavioral/psychological disorders.
Contributors to the article were researchers from multiple
disciplines assembled by the National Institutes of Health in
2002. They wrote about autism spectrum disorders specifically,
but acknowledged that the issues may have relevance to other
child and youth problems. Parents may be unwilling to consent
to randomization studies in case their child is assigned to what
parents perceive to be a less preferred treatment alternative,
particularly when the intervention is long term and early
intervention is, or is perceived to be, critical, such as early
intensive behavioral intervention for pervasive developmental
disorders. Lord et al. (2005) noted that ethical concerns may
prohibit RCTs of promising interventions when randomization
to no treatment or treatment of unknown effects is required by
the evaluation protocol. Additional factors that reduce the
internal validity of group comparison studies of psychosocial
interventions include that parental blindness to the intervention
allocated to their children is nigh on impossible, diffusion of
treatment through parent support groups is likely, parents may

choose to withdraw their children from no treatment or
treatment as usual groups and obtain the experimental
intervention or an approximation to it from outside the study,
and children with severe disorders will often be subjected to
multiple interventions of unknown benefit, provided with
varying 20 Overview and Foundational Issues c01 20 April
2012; 12:43:30 Hersen, Michel, and Peter Sturmey. Handbook
of Evidence-Based Practice in Clinical Psychology, Child and
& Sons, Incorporated. All rights reserved. fidelity that may
interact with one another to produce uninterpretable beneficial,
neutral, or adverse effects (Smith & Antolovich, 2000). Parents
seek out information on their child’s disorder and intervention
recommendations through Internet or parent organizations,
sometimes accepting advice from professionals. Mackintosh,
Meyers, and Goin-Kochel (2006) received 498 responses to a
Web-based survey of parents with children with autism
spectrum disorders and found that the most oft-cited sources of
information were books (88%), Web pages (86%), other parents
(72%), and autism newsletters (69%). Lagging somewhat as
sources of advice were professionals other than educators or
physicians (57%). Physicians, education professionals, and
family members were cited as sources of information by fewer
than half the parents who responded to the survey. Multiple fad
treatments have been recommended for childhood onset
disorders, and their adoption by families and some
professionals, including psychologists, wastes resources and
time that could have been spent profitably by employing ESTs
(Jacobson, Foxx, & Mulick, 2005). Although some of our
examples of supported and unsupported treatments have related
to children with autism spectrum disorders (Romanczyk,
Arnstein, Soorya, & Gillis, 2003), the problem of treatment

selection uninformed by research affects children with other
difficulties and their families. Some interventions for children
with developmental and other disabilities have been found
ineffective or harmful (Jacobson et al., 2005), and the same
occurs for children with ADHD (Waschbusch & Hill, 2003).
(See also Chapter 2 of this volume for a further discussion of
this point by Waschbush, Fabiano, and Pelham.) We believe that
clinical psychologists working with young people ought to have
a professional ethical obligation to inform themselves and
others about empirically unsupportable treatments as well as
ESTs for their clients. LIMITATIONS OF THE EVIDENCE
BASE REGARDING EVIDENCE-BASED PRACTICE IN
PSYCHOLOGY There is a relatively large body of evidence
concerning efficacy of treatments (Kazdin & Weisz, 1998,
2003), but far less on treatment utility, effectiveness, and
efficiency. There is evidence that the utility of some efficacious
treatments has been demonstrated, but further study is needed
before the general statements can be made about the similarity
or difference between outcomes from controlled research and
clinical practice (Barlow et al., 1999; Hunsley, 2007). There is
less evidence concerning dimensions of clinical expertise and
client characteristics, culture, and preferences that are relevant
to beneficial treatment outcomes (Kazdin, 2008). Employment
of qualitative research methods may help us to understand
clients’ experiences of psychological treatments. The APA Task
Force has suggested that clinical expertise is made up of at least
eight components, including assessment and treatment planning,
delivery, interpersonal skills, self-reflection, scientific skills in
evaluating research, awareness of individual and social factors,
the ability to seek additional resources where necessary, and
having a convincing rationale for treatment strategies (APA
Presidential Task Force on Evidence-Based Practice, 2006).
Qualitative methods may provide evidence regarding clinical
expertise also. Improvement of two-way communication
between psychologists who are primarily researchers and those
who identify more as practitioners would assist dissemination of

ESTs, collaboration in clinical utility studies of efficacious
treatments, and facilitate research into clinical expertise and
barriers to the adoption of ESTs by clinicians (Kazdin, 2008).
Lilienfeld (2005) and McLennan, Wathen, MacMillan, and
Lavis (2006) recommended further research on interventions in
child Rationale and Standards of Evidence in Evidence-Based
& Sons, Incorporated. All rights reserved. psychopathology on
two fronts: (1) increased research on promising or new but
plausible interventions; and (2) on research to combat
questionable, but potentially harmful, interventions. Attention
is needed to increase training of clinical psychologists in ESTs
and cessation of training to use treatments that have been shown
to be harmful, ineffective, or less effective. Decreasing the
demand and use of treatments that have not been evaluated
scientifically or have been found ineffective or harmful may be
another strategy for helping clinical psychologists orient more
to treatments that do work (i.e., ESTs). Woody, Weisz, and
McLean (2005) report that APA accredited clinical psychologist
training programs taught and supervised interns in fewer ESTs
in 2003 than they had in 1993. The training of clinical
psychologists should include sufficient study of research
methodology so that career-long learning, and contribution to
research in practice concerning ESTs can be enhanced (Bauer,
2007; Kazdin, 2008). Considering the APA definition of EBPP,
trainees need supervised practice in incorporating clients’
preferences, values, and cultural considerations as well as to
develop clinical expertise (Collins, Leffingwell, & Belar, 2007).
Further training for universitybased psychologist training
faculty should be studied. Psychologists in practice may wish or

feel forced to adapt to employing EBPP, but willingness or
compulsion to do so is not the same as becoming immediately
competent to use an empirically supported treatment effectively.
The typical workshop format for introducing new techniques is
as ineffective for professionals (Gotham, 2006), as it is for
direct care staff (Reid, 2004). Skill-based training can be
effective when trainees practice an intervention method in the
natural clinical environment with differential feedback from the
trainer on their performance of the skill. This should occur after
workshops that introduce the rationale and method, and include
in vivo or videotaped demonstration of the skill by the trainer
(i.e., modeling). Frequent follow-up observations by the trainer,
again with objective feedback to the trainee, can facilitate
maintenance of the newly acquired skills (Reid, 2004). Gotham
(2006) identified barriers to implementation of EBPP, and
provided an example of how to implement an empirically
supported treatment statewide despite obstacles. McCabe (2004)
wrote quite optimistically for clinicians about the challenges of
EBPP, offering advice in a step-by-step form to psychologists.
Reorienting and training clinicians is an area for further
clinicianresearcher collaboration that requires emphasis. We
have not included discussion of an area of EBPP that has, to
date, received less attention than evidence-based treatment,
which is “evidence-based assessment” (Kazdin, 2005; Mash &
Hunsley, 2005). An initial assessment with clinical utility will
identify what is the disorder or problem behavior so that it
points the psychologist in the right direction for identifying the
range of available ESTs for this client. Evidence-based
assessment includes identifying reliable and valid ongoing
measures that show the effects of an intervention with the
individual client (Kazdin, 2005). Typically, empirically
supported treatment reviews identify treatments for DSM-type
classifications, the nosological approach; however, an
idiographic functional approach to assessment may lead to
better problem-EST match (Sturmey, 2007). It was mentioned
earlier that some professional groups are concerned at the lack

of research on the outcomes of their assessment and treatment
methods. Because clinical and other psychologists have
extensive training in research methods, we can assist other
professions to assess the evidence for their intervention
methods. Interdisciplinary collaborations also may help
elucidate the interaction effects of behavioral or psychological
ESTs with interventions with presently unknown effects
delivered by other professions. 22 Overview and Foundational
Issues c01 20 April 2012; 12:43:31 Hersen, Michel, and Peter
Sturmey. Handbook of Evidence-Based Practice in Clinical
Psychology, Child and Adolescent Disorders, John Wiley &
Sons, Incorporated, 2012. ProQuest Ebook Central,
& Sons, Incorporated. All rights reserved. CONCLUDING
REMARKS The issues concerning evidence-based clinical
practice of psychology for children and adolescents are
arguably even more complex than for adult clients. All readers
would welcome the day when, for any referred psychological,
behavioral, or mental problem, psychologists with clinical
expertise were able to offer a wide range of effective and safe
interventions. Then the young person, with more or less help
from those who care for them, could select the effective
treatment that suited their culture, preferences, and resources.
The literature reviewed for this chapter suggests that, generally
speaking, the health care, including clinical psychology, and
education professions are starting on the road to addressing the
aspirational targets of the EBP movement. The lack of
unanimous agreement within professions that the evidence-
based practice movement is desirable and what constitutes
evidence for beneficial interventions is unsurprising; however,
with more evidence for safe, acceptable, and effective
interventions such as contained in the present volume and more
education for consumers, professionals, and the public,
eventually the naysayers and peddlers of unsupportable

treatments may find they have no raison d’eˆtre (and also no
income!).
Limitations to Evidence Based Practice
THOMAS MAIER
The promotion of evidence-based medicine (EBM) or, more
generally, of evidence-based practice (EBP) has strongly
characterized most medical disciplines over the past 15 to 20
years. Evidence-based medicine has become a highly influential
concept in clinical practice, medical education, research, and
health policy. Although the evidence-based approach has also
been increasingly applied in related fields such as psychology,
education, social work, or economics, it was and still is
predominantly used in medicine and nursing. Evidence-based
practice is a general and nonspecific concept that aims to
improve and specify the way decision makers should make
decisions. For this purpose it delineates methods of how
professionals should retrieve, summarize, and evaluate the
available empirical evidence in order to identify the best
possible decision to be taken in a specific situation. So EBP is,
in a broader perspective, a method to analyze and evaluate large
amounts of statistical and empirical information to understand a
particular case. It is therefore not limited to specific areas of
science and is potentially applicable in any field of science
using statistical and empirical data. Many authors often cite
Sackett, Rosenberg, Muir Gray, Haynes, and Richardson’s
(1996) article entitled “Evidence-based medicine:What it is and
what it isn’t” as the founding deed of evidence-based practice.
David L. Sackett (born 1934), an American-born Canadian
clinical epidemiologist, was professor at the Department of
Clinical Epidemiology and Biostatistics of McMaster University
Medical School of Hamilton, Ontario, from 1967 to 1994.
During that time, he and his team developed and propagated

modern concepts of clinical epidemiology. Sackett later moved
to England, and from 1994 to 1999, he headed the National
Health Services’ newly founded Centre for Evidence-Based
Medicine at Oxford University. During that time, he largely
promoted EBM in Europe by publishing articles and textbooks
as well as by giving numerous lectures and training courses.
David Sackett is seen by many as the founding father of EBM as
a proper discipline, although he would not at all claim this
position for himself. In fact, Sackett promoted and elaborated
concepts that have been described and used by others before;
the origins of EBM are rooted back in much earlier times. The
foundations of clinical epidemiology were already laid in the
19th century mainly by French, German, and English physicians
systematically studying the prevalence and course of diseases
and the effects of therapies. As important foundations of the
EBMmovement, certainly the works and insights of the Scottish
epidemiologist Archibald (Archie) L. Cochrane (1909–1988)
have to be c04 18 April 2012; 19:44:27 55 Hersen, Michel, and
Clinical Psychology : Adult Disorders, John Wiley & Sons,
& Sons, Incorporated. All rights reserved. mentioned. Cochrane,
probably the true founding father of modern clinical
epidemiology, had long before insisted on sound
epidemiological data, especially from RCTs, as the gold
standard to improve medical practice (Cochrane, 1972). In fact,
the evaluation of epidemiological data has always been one of
the main sources of information in modern academic medicine,
and many of the most spectacular advances of medicine are
direct consequences of the application of basic epidemiological
principles such as hygiene, aseptic surgery, vaccination,
antibiotics, and the identification of cardiovasular and
carcinogenic risk factors. One of the most frequent objections

against the propagation of EBM is, “It’s nothing new, doctors
have done it all the time.” Rangachari, for example,
apostrophized EBM as “old French wine with a new Canadian
label” (Rangachari, 1997, p. 280) alluding to the French 19th
century epidemiology pioneer Pierre Louis, who was an
influencing medical teacher in Europe and North America, and
to David L. Sackett, the Canadian epidemiologist. Even though
the “conscientious, explicit and judicious use of the current best
evidence in making decisions about the care of individual
patients” (Sackett et al., 1996, p. 71) seemsto be a perfectly
reasonable and unassailable goal, EBM has been harshly
criticized from the very beginning of its promotion (Berk
&Miles Leigh, 1999; B. Cooper, 2003; Miles, Bentley,
Polychronis, Grey, and Price, 1999; Norman, 1999; Williams &
Garner, 2002). In 1995, for example, the editors of The Lancet
chose to publish a rebuking editorial against EBM entitled
“Evidence-based medicine, in its place” (The Lancet, 1995):
The voice of evidence-based medicine has grown over the past
25 years or so from a subversive whisper to a strident insistence
that it is improper to practise medicine of any other kind.
Revolutionaries notoriously exaggerate their claims;
nonetheless, demands to have evidence-based medicine
hallowed as the new orthodoxy have sometimes lacked finesse
and balance, and risked antagonising doctors who would
otherwise have taken many of its principles to heart. The Lancet
applauds practice based on the best available evidence–bringing
critically appraised news of such advances to the attention of
clinicians is part of what peer-reviewed medical journals do–but
we deplore attempts to foist evidencebased medicine on the
profession as a discipline in itself. (p. 785) This editorial
elicited a fervid debate carried on for months in the letter
columns of The Lancet. Indeed, there was a certain doggedness
on both sides at that time, astonishing neutral observers and
rendering the numerous critics even more suspicious. The
advocates of EBM on their part acted with great self-confidence
and claimed no less than to establish a new discipline and to put

clinical medicine on new fundaments; journals, societies,
conferences, and EBM training courses sprang up like
mushrooms; soon academic lectures and chairs emerged;
however, this clamorous and pert appearance of EBM repelled
many. A somehow dogmatic, almost sectarian, tendency of the
movement was noticed with discontent, and even the deceased
patron saint of EBM, Archie Cochrane, had to be invoked in
order to push the zealots back: How would Archie Cochrane
view the emerging scene? His contributions are impressive,
particularly to the development of epidemiology as a medical
science, but would he be happy about all the activities linked
with his name? He was a freethinking, iconoclastic individual
with a healthy cynicism, who would not accept dogma. He
brought an open sceptical approach to medical problems and we
think that he would be saddened to find that his name now
embodies a new rigid medical orthodoxy while the real impact
of his many achievments might be overlooked. (Williams &
Garner 2002, p. 10) THE DEMOCRATIZATION OF
KNOWLEDGE How could such an emotional controversy arise
about the introduction of a scientific 56 Overview and
Practice in Clinical Psychology : Adult Disorders, John Wiley &
& Sons, Incorporated. All rights reserved. method (Ghali, Saitz,
Sargious, & Hershman, 1999)? Obviously, the propagation and
refusal of EBM have to be seen not only from a rational
scientific standpoint but also from a sociological perspective
(Miettinen, 1999; Norman, 1999): The rise of the EBM
movement fundamentally reflects current developments in
contemporary health care concerning the allocation of
information, knowledge, authority, power, and finance (Berk &
Miles Leigh, 1999), a process becoming more and more critical

during the late 1980s and the 1990s. Medicine has, for quite
some time, been losing its prestige as an intangible, moral
institution. Its cost-value ratio is questioned more and more and
doctors are no longer infallible authorities. We do not trust
doctors anymore to know the solution for any problem; they are
supposed to prove and to justify what they do and why they do
it. These developments in medicine parallel similar tendencies
in other social domains and indicate general changes in Western
societies’ self-conception. Today we are living in a knowledge
society, where knowledge and information is democratized,
available and accessible to all. There is no retreat anymore for
secret expert knowledge and for hidden esoteric wisdom. The
hallmarks of our time are free encyclopedic databases, open
access, the World Wide Web, and Google©. In the age of
information, there are no limitations for filing, storage,
browsing, and scanning of huge amounts of data; however, this
requires more and more expert knowledge to handle it. So,
paradoxically, EBM represents a new specialized expertise that
aims to democratize or even to abolish detached expert
knowledge. The democratization of knowledge increasingly
questions the authority and selfsufficiency of medical experts
and has deeply unsettled many doctors and medical scientists.
Of course, this struggle is not simply about authority and truth;
it is also about influence, power, and money. For all the
unsettled doctors, EBM must have appeared like a guide for the
perplexed leading them out of insecurity and doubt. Owing to
its paradoxical nature, EBM offers them a new spiritual home of
secluded expertise allowing doctors to regain control over the
debate and to reclaim authority of interpretation from bold
laymen. For this purpose, EBM features and emphasizes the
most valuable label of our time that is so believable in science:
science- or evidencebased. In many areas of contention, terms
like evidence-based or scientifically proven are used for the
purpose of putting opponents on the defensive. Nobody is
entitled to question a fact, which is declared evidence-based or
scientifically proven. By definition, these labels are supposed to

convey unquestioned and axiomatic truth. It requires rather
complex and elaborate epistemological reasoning to demonstrate
how even true evidence-based findings can at the same time be
wrong, misleading, and/or useless. All these accounts and
arguments apply in particular to the disciplines of psychiatry
and clinical psychology, which have always had a marginal
position among the apparently respectable disciplines of
academic medicine. Psychiatrists and psychologists always felt
particularly pressured to justify their actions and are constantly
suspected to practice quackery rather than rational science. It is
therefore not surprising that among other marginalized
professionals, such as the general practitioners, psychiatrists
and psychotherapists made particularly great efforts over the
last years to establish their disciplines as serious matters of
scholarly medicine by diligently adopting the methods of EBM
(Geddes & Harrison, 1997; Gray & Pinson, 2003;
OakleyBrowne, 2001; Sharpe, Gill, Strain, & Mayou, 1996).
Yet, there are also specific problems limiting the applicability
of EBP in these disciplines.
EMPIRICISM AND REDUCTIONISM In order to understand
the role and function of EBP within the scientific context, it
may be Limitations to Evidence-Based Practice 57 c04 18 April
of Evidence-Based Practice in Clinical Psychology : Adult
& Sons, Incorporated. All rights reserved. helpful to give a
brief overview of the theoretical backgrounds of science in
general. What is science and how does it proceed? Science can
be seen as a potentially endless human endeavour that aims to
understand and determine reality. Not only are physical objects
matters of science, but also immaterial phenomena like
language, history, society, politics, economics, human behavior,
thoughts, or emotions. Starting with the Greek scientists in the

ancient world, but progressing more rapidly with the
philosophers of the Enlightenment, modern science adopted
defined rules of action and standards of reasoning that delineate
science from nonscientific knowledge such as pragmatics, art,
or religion. Unfortunately, notions like science, scientific, or
evidence are often wrongly used in basically nonscientific
contexts causing unnecessary confusion. The heart and the
starting point of any positive science is empiricism, meaning
the systematic observation of phenomena. Scientists of any kind
must start their reasoning with observations, possibly refined
through supportive devices or experimental arrangements.
Although positive science fundamentally believes in the
possibility of objective perception, it also knows the inherent
weaknesses of reliability and potential sources of errors. Rather
than have confidence in single observations, science trusts
repeated and numerous observations and statistical data. This
approach rules out idiosyncratic particularities of single cases
to gain the benefit of identifying the common characteristics of
general phenomena (i.e., reductionism). This approach of
comprehending phenomena by analytically observing and
describing them has in fact produced enormous advancements in
many fields of science, especially in technical disciplines;
however, contrasting and confusing gaps of knowledge prevail
in other areas such as causes of human behavior, mind–body
problems, or genome–environment interaction. Some areas of
science are apparently happier and more successful using the
classical approach of positive science, while other disciplines
feel less comfortable with the reductionist way of analyzing
problems. The less successful areas of science are those
studying complex phenomena where idiosyncratic features of
single cases can make a difference, in spite of perfect empirical
evidence. This applies clearly to medicine, but even more to
psychology, sociology, or economics. Medicine, at least in its
academic version, usually places itself among respectable
sciences, meeting with and observing rules of scientific
reasoning; however, this claim may be wishful thinking and

medicine is in fact a classical example of a basically
atheoretical, mainly pragmatic undertaking pretending to be
based on sound science. Inevitably, it leads to contradictions
when trying to bring together common medical practice and
pure science. COMPLEXITY Maybe the deeper reasons for
these contradictions are not understood well enough. Maybe
they still give reason for unrealistic ideas to some scientists. A
major source of misconception appears to be the confused
ontological perception of some objects of scientific
investigation. What is a disease, a disorder, a diagnosis? What
is human behavior? What are emotions? Answering these
questions in a manner to provide a basis for scientific reasoning
in a Popperian sense (see later) is far from trivial. Complex
objects of science, like human behavior, medical diseases, or
emotions, are in fact not concrete, tangible things easily
accessible to experimental investigation. They are emergent
phenomena, hence they are not stable material objects, but exist
only as transitory, nonlocal appearances fluctuating in time.
They continuously emerge out of indeterminable complexity
through repeated self-referencing operations in complex systems
(i.e., autopoietic systems). Indeterminable complexity or
deterministic chaos means that a huge number of mutually
interacting parameters autopoietically 58 Overview and
& Sons, Incorporated. All rights reserved. form a system,
rendering any precise calculation of the system’s future
conditions impossible. Each single element of the system
perfectly follows the physical rules of causality; however, the
system as a whole is nevertheless unpredictable. Its fluctuations
and oscillations can be described only probabilistically. In order

to obtain reasonable and useful information about a system,
many scientific disciplines have elaborated probabilistic
methods of approaching their objects of interest.
Thermodynamics, meteorology, electroencephalography,
epidemiology, and macroeconomics are only a few such
examples. Most structures in biological, social, and
psychological reality can be conceived as emergent phenomena
in this sense. Just as the temperature of an object is not a
quality of the single molecules forming the object—a single
molecule has no temperature— but a statistic description of a
huge number of molecules, human behavior cannot be
determined through the description of composing elements
producing the phenomenon—for example, neurons—even if
these elements are necessary and indispensable preconditions
for the emergence of the phenomenon. The characteristics of the
whole cannot be determined by the description of its parts.
When the precise conditions of complex systems turn out to be
incalculable, the traditional reaction of positive science is to
intensify analytical efforts and to compile more information
about the components forming the system. This approach allows
scientists to constantly increase their knowledge about the
system in question without ever reaching a final understanding
and a complete determination ofthe function ofthe system. This
is exactly what happens currently in neurosciences.
Reductionist approaches have their inherent limitations when it
comes to the understanding of complex systems. A similar
problem linked to complexity that is particularly important is
the assumed comparability of similar cases. In order to
understand an individual situation, science routinely compares
defined situations to similar situations or, even better, to a large
number of similar situations. Through the pooling of large
numbers of comparable cases, interfering individual differences
are statistically eliminated, and only the common ground
appears. The conceptual assumption behind this procedure is
that similar—but still not identical— cases will evolve similarly
under identical conditions. One of the most important insights

from the study of complex phenomena is that in complex
systems very small differences in initial conditions may lead to
completely different outcomes after a short time—the socalled
butterfly effect. This insight is well known to natural scientists;
however, clinical epidemiologists do not seem to be completely
aware of the consequences of the butterfly effect to their area of
research. FROM KARL POPPER TO THOMAS S. KUHN Based
on epistemological considerations, the Anglo-Austrian
philosopher Karl Popper (1902–1994) demonstrated in the
1930s the limitations of logical empiricism. He reasoned that
general theories drawn from empirical observations can never
be proven to be true. So, all theories must remain tentative
knowledge, waiting to be falsified by contrary observations. In
fact, Popper conceived the project of science as a succession of
theories to be falsified sooner or later and to be replaced by new
theories. This continuous succession of new scientific theories
is the result of natural selection of ideas through the
advancement of science. According to Popper, any scientific
theory must be formulated in a way to render it potentially
falsifiable through empirical testing. Otherwise, the theory is
not scientific: It may be metaphysical, religious, or spiritual
instead. This requires that a theory must be formulated in terms
of clearly defined notions and measurable elements. Popper’s
assertions were later qualified as being less absolute by the
American philosopher of science Thomas S. Kuhn (1922–1996).
Limitations to Evidence-Based Practice 59 c04 18 April 2012;
Evidence-Based Practice in Clinical Psychology : Adult
& Sons, Incorporated. All rights reserved. Kuhn, originally a
physicist, pointed out that in real science any propagated theory
could be falsified immediately by contrary observations because
contradicting observations are always present; however, science

usually ignores or even suppresses observations dissenting with
the prevailing theory in order to maintain the accepted theory.
Kuhn calls the dissenting observations anomalies, which are—
according to him—always obvious and visible to all, but
nevertheless blinded out of perception in order to maintain the
ruling paradigm. In Kuhn’s view, science will never come to an
end and there will never be a final understanding of nature. No
theory will ever be able to integrate and explain consistently all
the observations drawn from nature. At this point, even the
fundamental limitations to logical scientific reasoning
demonstrated by Go¨del’s incompleteness theorems become
recognizable (cf. also Sleigh, 1995). Based on his
considerations, Kuhn clear-sightedly identified science to be a
social system, rather than a strictly logical and rational
undertaking. Science, as a social phenomenon, functions
according to principles of Gestalt psychology. It sees the things
it wants to see and overlooks the things that do not fit. In his
chief work The Structure of Scientific Revolutions, Kuhn
(1962) gives several examples from the history of science
supporting this interpretation. It is in fact amazing to see how
difficult it was for most important scientific breakthroughs to
become acknowledged by the contemporary academic
establishment. Kuhn uses the notion normal science to
characterize the established academic science and emphasizes
the self-referencing nature of its operating mode. Academic
teachers teach students what the teachers believe is true.
Students have to learn what they are taught by their teachers if
they want to pass their exams and get their degrees. Research is
mainly repeating and retesting what is already known and
accepted. Journals, edited and peerreviewed by academic
teachers, publish what conforms with academic teachers’ ideas.
Societies and associations—headed by the same academic
teachers—ensure the purity of doctrine by sponsoring those who
confirm the prevailing paradigms. Dissenting opinions are
unwelcome. Based on Kuhn’s view of normal science, EBP and
EBM can be identified as classical manifestations of normal

science. The EBP helps to ensure the implementation of
mainstream knowledge by declaring to be most valid what is
best evaluated. Usually the currently established practices are
endorsed by the best and most complete empirical evidence;
dissenting ideas will hardly be supported by good evidence,
even if these ideas are right. Since EBP instructs its adherers to
evaluate the available evidence on the basis of numerical rules
of epidemiology, arguments like plausibility, logic consistency,
or novelty are of little relevance. AN EXAMPLE FROM
RECENT HISTORY OF CLINICAL MEDICINE When in 1982
the Australian physicians Barry Marshall and Robin Warren
discovered Helicobacter pylori in the stomachs of patients with
peptic ulcers, their findings were completely ignored and
neglected by the medical establishment of that time. The idea
that peptic ulcers are provoked by an infectious agent conflicted
with the prevailing paradigm of academic gastroenterology,
which conceptualized peptic ulcers as a consequence of stress
and lifestyle. Although there had been numerous previous
reports of helicobacteria in gastric mucosa, all these findings
were completely ignored because they conflicted with the
prevailing paradigm. As a consequence Marshall and Warren’s
discovery was ignored for years because it fundamentally
challenged current scientific opinion. They were outcast by the
scientific community, and only 10 years later their ideas slowly
started to convince more and more clinicians. Now, 25 years
later, it is common basic clinical knowledge that 60 Overview
and Foundational Issues c04 18 April 2012; 19:44:28 Hersen,
& Sons, Incorporated. All rights reserved. Helicobacter pylori is
one of the major causes of peptic ulcers, and eradication therapy
is the accepted and rational therapy for gastric ulcers. Finally,

in 2005 Barry Marshall and Robin Warren gained the Nobel
Price for their discovery (Parsonnet, 2005). BENEFITS AND
RISKS OF EVIDENCE-BASED PRACTICE The true benefits of
EBP for patients and society in terms of outcomes and costs
have not been proven yet—at least not through sound empirical
evidence (B. Cooper, 2003; Geddes & Harrison, 1997).
Nevertheless, there is no doubt that the method has a beneficial
and useful potential. Many achievements of EBP are
undisputable and undisputed, hence they are evident. Owing to
the spread of methodical skills in retrieving and evaluating the
available epidemiological evidence, it has become much harder
to apply any kind of obscure or idiosyncratic practices. The
experts’ community, as well as the customers and the general
public, are much more critical toward pretended effects of
treatments and ask for sound empirical evidence of
effectiveness and safety. It is increasingly important not only to
know the best available treatment, but also to prove it. The EBP
is therefore a helpful instrument for doctors and therapists to
justify and legitimate their practices to insurance, judiciary,
politics, and society. Furthermore, individual patients might be
less at risk to wrong or harmful treatment due to scientific
misapprehension. Of course, common malpractice owing to
inanity, negligence, or viciousness will never be eliminated, not
even by the total implementation of EBP; however, treatment
errors committed by diligent and virtuous doctors are minimized
through careful adherence to rational guidelines. In general,
clinical decision-making paths have become more
comprehensible and rational, probably also due to the spread of
EBP. As medicine is in fact not a thoroughly scientific matter
(Ghali et al. 1999), continuous efforts are needed to enhance
and renew rationality. The EBP contributes to this task and
helps clinicians to maintain rationality in a job where
inscrutable complexity is daily business. In current medical
education, the algorithms of EBP are now instilled into students
as a matter of course. Seen from that perspective, EBP is also
an instrument of discipline and education, for it compels

medical students and doctors to reflect continuously all their
opinions and decisions scientifically (Norman, 1999). Today
EBP has a great impact on the education and training of future
doctors, and it thereby enhances the uniformity and
transparency of medical doctrine. This international alignment
of medical education with the principles of EBP will, in the
long run, allow for better comparability of medical practice all
over the world. This is an important precondition for the
planning and coordination of research activities. Thus, the
circle of normal science is perfectly closed through the
widespread implementation of EBP. GENERAL LIMITATIONS
TO EVIDENCE-BASED PRACTICE It has been remarked, not
without reason, that the EBP movement itself has adopted
features of dogmatic authority (B. Cooper, 2003; Geddes et al.,
1996; Miles et al., 1999). This appears particularly ironic,
because EBP explicitly aims to fight any kind of orthodox
doctrine. The ferocity of some EBP adherents may not
necessarily hint at conceptual weaknesses of the method; rather,
it is more likely a sign of an iconoclastic or even patricidal
tendency inherent to EBP. Young, diligent scholars, even
students, possibly without any practical experience, are now
entitled to criticize and rectify clinical authorities (Norman,
1999). This kind of insurgence must evoke resistance from
authorities. If the acceptance of EBP among clinicians should
be enhanced, Limitations to Evidence-Based Practice 61 c04 18
April 2012; 19:44:28 Hersen, Michel, and Peter Sturmey.
Handbook of Evidence-Based Practice in Clinical Psychology :
Adult Disorders, John Wiley & Sons, Incorporated, 2011.
& Sons, Incorporated. All rights reserved. it is advisable that
the method is not only propagated by diligent theoreticians, but
mainly by experienced practitioners. One of the first and most
important arguments against EBP is reductionism (see earlier,

Welsby, 1999). Complex and maybe fundamentally diverse
clinical situations of individual patients have to be condensed
and aggregated to generalized questions in order to retrieve
empirical statistical evidence. Important specific information
about the individual cases is inevitably lost owing to this
generalization. The usefulness of the retrieved evidence is
therefore inevitably diluted to a very general and dim level. Of
course, there are some frequently used standard interventions,
which are really based upon good empirical evidence (Geddes et
al., 1996). EXAMPLES FROM CLINICAL MEDICINE Scabies,
a parasitic infection of the skin, is an important public health
problem, mainly in resource-poor countries. For the treatment
of the disease, two treatment options are recommended: topical
permethrin and oral ivermectin. Both treatments are known to
be effective and are usually well tolerated. The Cochrane
Review concluded from the available empirical evidence that
topical permethrin appears to be the most effective treatment of
scabies (Strong & Johnstone, 2007). This recommendation can
be found in up-to-date medical textbooks and is familiar to any
well-trained doctor. Acute otitis media in children is one of the
most common diseases, one ofthe main causes for parents to
consult a pediatrician, and a frequent motive for the
prescription of antibiotics, even though spontaneous recovery is
the usual outcome. Systematic reviews have shown that the role
of antibiotic drugs for the course of the disease is marginal, and
there is no consensus among experts about the identification of
subgroups who would potentially profit from antibiotics. In
clinical practice, in spite of lacking evidence of its benefit, the
frequent prescription of antibiotic drugs is mainly the
consequence of parents’ pressure and doctors’ insecurity. A
recent meta-analysis (Rovers et al., 2006) found that children
youngerthan 2 years of age with bilateral acute otitis media and
those with otorrhea benefited to some extent from antibiotic
treatment; however, even for these two particular conditions,
differences were moderate: After 3–7 days, 30% of the children
treated with antibiotics still had pain, fever, or both, while in

the control group the corresponding proportion was 55%. So,
the available evidence to guide a clinician when treating a child
with acute otitis media is not really significant and the decision
will mostly depend on soft factors like parents’ preferences or
practical and economical considerations. Evidently, clinicians
choosing these interventions do not really need to apply the
algorithms of EBP to make their decisions. They simply
administer what they had learned in their regular clinical
training. The opponents of EBP rightly argue that the real
problems in clinical practice arise from complex, multimorbid
patients presenting with several illnesses and other factors that
have to be taken into account by the treating clinician. In order
to manage such cases successfully there is usually no specific
statistical evidence available to rely on. Instead, clinicians have
to put together evidence covering some aspects of the actual
case and hope that the resulting treatment will still work even if
it is not really designed and tested for that particular situation.
Good statistical evidence meeting the highest standards of EBP
is almost exclusively derived from ideal monomorbid patients,
who are rarely seen in real, everyday practice (Williams &
Garner, 2002). It is not clear at all—and far from evidence-
based—whether evidence from ideal cases can be transferred to
62 Overview and Foundational Issues c04 18 April 2012;
& Sons, Incorporated. All rights reserved. more complex cases
without substantial loss of validity. Another argument
criticizing EBP points at an epistemological problem. Because
the EBP operates retrospectively by evaluating what was done
in the past, it cannot directly contribute to developing new
strategies and to finding new therapies. The EBP helps to
consolidate well-known therapies, but cannot guide researchers

toward scientific innovations. No scientific breakthrough will
ever be made owing to EBP. On the contrary, if all clinicians
strictly followed recommendations drawn from available
retrospective evidence and never dared to try something
different, science would stagnate in fruitless selfreference.
There is a basically conservative and backward tendency
inherent to the method. Although it cannot exactly be called
antiscientific on that account (B. Cooper, 2003; Miles et al.,
1999), EBP is a classical phenomenon of normal science (Kuhn,
1962). It will not itself be the source of fundamental new
insights. Finally, there is an external problem with EBP, which
is probably most disturbing of all: Production and compilation
of evidence available to clinicians is highly critical and exposed
to different nonscientific influences (Miettinen, 1999).
Selection of areas of research is based more and more on
economic interests. Large, sound, and therefore scientifically
significant epidemiologic studies are extremely complex and
expensive. They can be accomplished only with the support of
financially potent sponsors. Compared with public bodies or
institutions, private companies are usually faster and more
flexible in investing important amounts of money into medical
research. So, for many ambitious scientists keen on collecting
publishable findings, it is highly appealing to collaborate with
commercial sponsors. This has a significant influence on the
selection of diseases and treatments being evaluated. The
resulting body of evidence is necessarily highly unbalanced
because mainly diseases and interventions promising important
profits are well evaluated. For this reason, more money is
probably put into trials on erectile dysfunction, baldness, or
dysmenorrhea than on malaria or on typhoid fever. So, even
guidelines based on empirical evidence—considered to be the
ultimate gold standard of clinical medicine—turn out to be
arbitrary and susceptible to economical, political, and dogmatic
arguments (Berk & Miles Leigh, 1999). So, EBP’s goals to
replace opinion and tendency by knowledge are in danger of
being missed, if the relativity of available evidence is

unrecognized. The uncritical promotion of EBP opens a
clandestine gateway to those who have interests in controlling
the contents of medical debates and have the financial means to
do so. Biasing clinical decisions in times of EBP is probably no
longer possible by false or absent evidence; however, the
selection of what is researched in an EBP-compatible manner
and what is published may result in biased clinical decisions
(Miettinen, 1999). One of the most effective treatment options
in many clinical situations—watchful waiting—is notoriously
under-researched because there is no commercial or academic
interest linked to that treatment option. Unfortunately, there
will never be enough time, money, and workforce to produce
perfect statistical evidence for all useful clinical procedures.
So, even in the very distant future, clinicians will still apply
many of their probably effective interventions without having
evidence about their efficacy and effectiveness; thus, EBP is a
technique of significant but limited utility (Green & Britten,
1998; The Lancet, 1995; Sackett et al., 1996). EXAMPLE
FROM CLINICAL MEDICINE Lumbar back pain is one of the
most frequent health problems in Western countries. About 5%
of all low back problems are caused by prolapsed lumbar discs.
The treatment is Limitations to Evidence-Based Practice 63 c04
& Sons, Incorporated. All rights reserved. mainly nonsurgical
and 90% of acute attacks of nerve root pain (sciatica) settle
without surgical intervention; however, different forms of
surgical treatments have been developed and disseminated.
Usually these methods are considered for more rapid relief in
patients whose recovery is unacceptably slow. The Cochrane
reviewers criticize that “despite the critical importance of

knowing whether surgery is beneficial for disc prolapse, only
four trials have directly compared discectomy with conservative
management and these give suggestive rather than conclusive
results” (Gibson & Waddell, 2007, p. 1). They concluded:
Surgical discectomy for carefully selected patients with sciatica
due to lumbar disc prolapse provides faster relief from the acute
attack than conservative management, although any positive or
negative effects on the lifetime natural history of the underlying
disc disease are still unclear. (p. 2) Surgical treatments of low
back pain hold an enormous commercial potential due to the
worldwide frequency of the problem. It appears obvious that
there are only a few trials comparing conservative treatment
with surgery. SPECIFIC LIMITATIONS TO EBP IN
PSYCHIATRY, PSYCHOTHERAPY, AND CLINICAL
PSYCHOLOGY In psychiatry and psychotherapy, there is an
ambivalent attitude toward EBP. Attempting to increase their
scientific respectability, some psychiatrists and clinical
psychologists zealously adopted EBP algorithms (Geddes &
Harrison, 1997; Gray & Pinson, 2003; OakleyBrowne, 2001;
Sharpe et al., 1996) and started evidence-based psychiatry.
Others remain hesitant or doubtful about the usefulness of EBP
in their field, and several authors have addressed different
critical aspects of evidence-based psychiatry (Berk & Miles
Leigh, 1999; Bilsker, 1996; Brendel, 2003; Geddes & Harrison,
1997; Goldner & Bilsker, 1995; Harari, 2001; Hotopf,
Churchill, & Lewis, 1999; Lawrie, Scott, & Sharpe, 2000;
Seeman, 2001; Welsby, 1999; Williams & Garner, 2002) with
all of them fundamentally concerning practical and scientific
particularities of psychiatry and clinical psychology. Next, we
shall try to clarify these arguments. The evidence-based
approach to individual cases is critically dependent on the
validity of diagnoses. This is an axiomatic assumption of EBP,
which is rarely analysed or scrutinized in detail. If in a concrete
case no diagnosis could be attributed, the case would not be
amenable to EBP, and no evidence could support decisions in
such a case. If the diagnosis is wrong, or—even more

intricate—if cases labeled with a specific diagnosis are still not
homogenous enough to be comparable in relevant aspects, EBP
will provide useless results. EXAMPLE FROM PSYCHIATRY
According to DSM-IV, eating disorders are classified in
different categories: anorexia nervosa (AN), bulimia nervosa
(BN), binge eating disorder (BED), and eating disorder not
otherwise specified (EDNOS). These categories are clinically
quite distinct and diagnostic criteria are clear and easily
applicable. In spite of the phenomenological diversity of the
disease patterns, there is a close relationship between the
different forms of eating disorders. In clinical practice, switches
between different diagnoses and temporary remissions and
relapses are frequent. In the course of time, patients may change
their disease pattern several times: At times they may not meet
the criteria for a diagnosis anymore, although they are not
completely symptom free, and later they may relapse to a full-
blown eating disorder again or may be classified as having
EDNOS. 64 Overview and Foundational Issues c04 18 April
of Evidence-Based Practice in Clinical Psychology : Adult
& Sons, Incorporated. All rights reserved. Corresponding to
these clinical impressions, longitudinal studies demonstrate that
the stability of eating disorder diagnoses over time is low (
Fichter & Quadflieg, 2007; Grilo et al., 2007; Milos, Spindler,
Schnyder, & Fairburn, 2005). Based on systematic evaluation of
the available evidence, however, treatment guidelines give
specific recommendations for the different conditions (National
Institute for Clinical Excellence [NICE], 2004). For patients
with AN, psychological treatment on an outpatient basis is
recommended. The treatment should be offered by “a service
that is competent in giving that treatment and in assessing the
physical risk of people with eating disorders” (p. 60). For

patients with BN, the NICE guideline proposes as a possible
first step to follow an evidence-based self-help program. As an
alternative, a trial with an antidepressant drug is recommended,
followed by cognitive behavior therapy for bulimia nervosa. In
the absence of evidence to guide the treatment of EDNOS, the
NICE guideline recommends pragmatically that “the clinician
considers following the guidance on the treatment of the eating
problem that most closely resembles the individual patient’s
eating disorder” (p. 60). So even though specific diagnoses of
eating disorders are not stable and a patient with AN might be
diagnosed with BN a few months later, treatment
recommendations vary considerably for the two conditions. It
becomes obvious that different treatment recommendations for
seemingly different conditions reflect rather accidental
differences in the availability of empirical evidence than real
differences in the response of certain conditions to specific
treatments. Hence, the guidance offered by the guideline is
basically a rather unstable crutch, and of course, cognitive
behavior therapy or an evidence-based self-help program might
be just as beneficial in AN or in EDNOS than it is in BN, even
though nobody has yet compiled the statistical evidence to
prove this. What does the validity of a diagnosis mean? The
question concerns epistemological issues and requires a closer
look to the nature of medical diagnoses with special regard to
psychiatric diagnoses. R. Cooper (2004) questioned if mental
disorders as defined in diagnostic manuals are natural kinds. In
her thoughtful paper, the author concluded that diagnostic
entities are in fact theoretical conceptions, describing complex
cognitive, behavioral, and emotional processes (R. Cooper,
2004; Harari, 2001). Diagnostic categories are based upon
observations, still they are strongly influenced by theoretical,
social, and even economical factors. The ontological structure
of psychiatric diagnoses is therefore not one of natural kinds.
They are not something absolutely existing that can be observed
independently. Rather they are comprehensive theoretical
definitions serving as tools for communication and scientific

observation. Kendell and Jablensky (2003) have also recently
addressed the issue of diagnostic entities and concluded that the
validity of psychiatric diagnoses is limited. They analysed
whether diagnostic entities are sufficiently separable from each
other and from normality by zones of rarity. They concluded
that this was not the case; rather, they concluded that
psychiatric diagnoses often overlap (R. Cooper, 2004; Welsby,
1999), shift over time within the same patient, and several
similar diagnoses can be present in the same patient at the same
time (comorbidity). Not surprisingly, diagnosis alone is a poor
predictor of outcome (Williams & Garner, 2002).
Acknowledging this haziness of diagnoses, one realizes these
problems when trying to match individual cases to empirical
evidence. When even the presence of a correctly assessed
diagnosis does not assure comparability to other cases with the
same diagnosis, empirical evidence about mental disorders is
highly questionable (Harari, 2001). Of course, limited validity
does not imply complete absence of validity, and empirical
evidence on mental disorders is still useful to some extent;
however, insight Limitations to Evidence-Based Practice 65 c04
& Sons, Incorporated. All rights reserved. into the limitations is
important and that insight points out that psychiatric diagnoses
represent phenomenological descriptions rather than natural
kinds. Several authors have treated the same issue when writing
about the complexity of cases, the problem of subsyndromal
cases, and of single cases versus statistical evidence (Harari,
2001; Welsby, 1999; Williams & Garner, 2002). NONLINEAR
DYNAMICS IN THE COURSE OF DISEASES It might be
fruitful to look at evidence-based psychiatry from another

perspective and to address the issues of complexity and
nonlinear dynamics. With regard to their physical and mental
functioning, humans can be conceptualized as systems of high
complexity (Luhmann, 1995). This means that they cannot be
determined precisely, but only in a probabilistic manner;
however, probabilistic determination is sufficient for most
purposes in observable reality. Human life consists
fundamentally in dealing with probabilities. Social systems and
human communication are naturally designed to manage
complexity more or less successfully. Medicine itself is a social
system (Luhmann, 1995) trying to handle the effects of
complexity (Harari, 2001), for example, by providing
probabilistic algorithms for treatments of diseases. In most
situations, medicine can ignore the particular effects emerging
from the complex nonlinear structure of its objects, although
such effects are always present. Only sometimes do these
effects become obvious and irritating, as for example in
fluctuations of symptoms in chronic diseases, variations in
response to treatment, unexpected courses in chronic diseases,
and so on. Such phenomena can be seen as manifestations of the
butterfly effect (see earlier). This insight questions deeply the
core principle of EBP that assumes that it is rational to treat
similar cases in the same manner because similarity in the
initial conditions will predict similar outcomes under the
identical treatment. The uncertainty of this assumption is
particularly critical in psychiatry and psychotherapy. In these
fields similar appearance is just a palliation for untraceable
difference, and this exact difference may crucially influence the
outcome. Addressing such problems is daily business for
psychiatrists and psychotherapists, so their disciplines have
developed special approaches. Diagnostic and therapeutic
procedures in these disciplines are much less focused on critical
momentary decisions, but more on gradual, iterative procedures.
Psychiatric treatments and even more psychotherapy are self-
referencing processes, where assessments and decisions are
constantly reevaluated. Instead, EBP focuses primarily on

decision making as the crucial moment of good medical
practice. One gets the impression that EBM clinicians are
constantly making critical decisions, and after having made the
right decision, the case is solved. Maybe it is because of this
misfit between the proposals of the method and real daily
practice that many psychiatrists are not too attracted by EBP.
EXAMPLE FROM PSYCHIATRY The diagnosis of
posttraumatic stress disorder (PTSD) was first introduced in the
third edition of the Diagnostic and Statistical Manual of Mental
Disorders (DSM-III) in 1980. Before that time, traumatized
individuals were either diagnosed with different nonspecific
diagnoses (e.g., anxiety disorders, depression, neurasthenia) or
not declared ill at all. Astonishingly, the newly discovered
entity appeared to be a clinically distinct disorder and the
corresponding symptoms (re-experiencing, avoidance,
hyperarousal) were quite characteristic and easily identifiable.
Within a short time after its invention (Summerfield, 2001),
PTSD became a very popular disorder; 66 Overview and
& Sons, Incorporated. All rights reserved. clinicians and even
patients loved the new diagnosis (Andreasen, 1995). The key
point for the success of the new diagnosis was that it is
explicitly based on the assumption of an external etiology; that
is, the traumatic experience. This conception makes PTSD so
appealing for the attribution of cause, responsibility, and guilt
is neatly separated from the affected individual. PTSD allows
for the exculpation of the victim, a feature that was particularly
important when caring for Holocaust survivors and Vietnam
War veterans. But what was almost proscribed for some time
after the introduction of PTSD is now evidence-based:

Preexisting individual factors play an important role in the
shaping of posttraumatic response. Whether or not an individual
develops PTSD after a traumatic experience is not only
determined by the nature and the intensity of the traumatic
impact, but also by various pretraumatic characteristics of the
affected individual. Furthermore, PTSD is not the only
posttraumatic mental disorder. A whole spectrum of mental
disorders is closely linked to traumatic experiences, although
they lack the monocausal appearance of PTSD. Anyway, the
most frequent outcome after traumatic experiences is recovery.
In the second rank of frequency comes major depression.
Borderline personality disorder is fully recognized now as a
disorder provoked by traumatic experiences in early childhood.
Dissociative disorders, chronic somatoform pain, anxiety
disorders, substance abuse, and eating disorders are equally
related to traumatic experiences. Not surprisingly, PTSD is
often occurring as a comorbid condition with one or more
additional disorder or vice versa. In clinical practice,
traumatized patients usually present more complex than
expected. This may explain to some extent why PTSD was
virtually overlooked by clinicians for many decades before its
introduction, a fact that is sometimes hard to understand by
younger therapists who are so familiar with the PTSD diagnosis.
At any rate, the high-functioning, intelligent, monomorbid
PTSD patient is indeed best evaluated in clinical trials, but
rarely seen in everyday practice. PTSD was right in the focus of
research since its introduction. Also from a scientific point of
view, the disorder is appealing because it is provoked by an
external event. PTSD allows ideally for the investigation of the
human-environmentinteraction, whichis a crucial issue for
psychiatry and psychology in general. The number of trials on
diagnosis and treatment of PTSD is huge, and the disorder is
now probably the best evaluated mental disorder. What is the
benefit of the accumulated large body of evidence on PTSD for
clinicians? There are several soundly elaborated guidelines on
the treatment of PTSD (American Psychiatric Association,

2004; Australian Centre for PosttraumaticMental Health, 2007;
NICE, 2005), meta-analyses, and Cochrane Reviews providing
guidance for the assessment and treatment of the disorder. When
we look at the existing conclusions and recommendations, we
learn that: Debriefing is not recommended as routine practice
for individuals who have experienced a traumatic event. When
symptoms are mild and have been present for less than 4 weeks
after the trauma, watchful waiting should be considered.
Trauma-focused cognitive behavior therapy on an individual
outpatient basis should be offered to people with severe
posttraumatic symptoms. Eye movement desensitization and
reprocessing is an alternative treatment option. Drug treatment
should not be used as a routine first-line treatment in preference
to a trauma-focused psychological therapy. Drug treatment
(Specific Serotonin Reuptake Inhibitors) should be considered
Limitations to Evidence-Based Practice 67 c04 18 April 2012;
& Sons, Incorporated. All rights reserved. for the treatment of
PTSD in adults who express a preference not to engage in
trauma-focused psychological treatment. In the context of
comorbid PTSD and depression, PTSD should be treated first.
In the context of comorbid PTSD and substance abuse, both
conditions should be treated simultaneously. These
recommendations are obviously clear, useful, and practical.
They give real guidance to therapists and do not leave much
room for doubts or insecurity. On the other hand, they are
basically very simple, almost trivial. For trauma therapists,
these recommendations are commonplace and serve mainly to
endorse what they are practicing anyway. The main points of
the guidelines for the treatment of PTSD could be taught in a 1-
hour workshop. The key messages of the guidelines represent

basic clinical knowledge on a specific disorder as it has been
instructed in times before EBP. Through their standardizing
impact on the therapeutic community, guidelines may in fact
align and improve the general service quality offered to
traumatized individuals, although this effect has not yet been
demonstrated by empirical evidence. The treatment of an
individual patient remains a unique endeavor where
interpersonal relationship, flexibility, openness, and cleverness
are crucial factors. This challenge is not lessened by evidence
or guidelines.

Rationale and Standards of Evidence in Evidence-Based Practice.docx

More Related Content

Similar to Rationale and Standards of Evidence in Evidence-Based Practice.docx (20)

More from makdul (20)

Recently uploaded (20)

Rationale and Standards of Evidence in Evidence-Based Practice.docx