Integrating Science and Practice (Mller & Bargmann)

INTEGRATING

ScienceandPractice
Volume 2 number 2 noVember 2012

10 Tools
for Progress Monitoring
in Psychotherapy

CelestHealth ORS and SRS

CORE-OM Polaris-MH

BASIS-24 PSYCHLOPS

Integra/COMPASS SOS-10

OQ-45 TOP

MISSION OF INTEGRATING SCIENCE AND PRACTICE

Integrating Science and Practice is published twice yearly by the Ordre des psychologues du Québec.
The goal of the journal is to provide syntheses of scientific knowledge in the area of psychology
and to facilitate the transfer of scientific knowledge to the field of practice. The journal aims to
give practitioners in psychology, from all areas and fields of practice, the tools they need
by providing them with critical reviews of the literature and brief syntheses of knowledge on
specific themes. The journal is further intended to inform the public and professionals who work
in collaboration with psychologists about recent scientific and clinical developments in psy-
chology and about the contribution of psychologists towards improving people’s quality of life.
The journal publishes articles by invitation only, following a call for proposals.
Independent submissions are neither considered nor accepted. However, the editorial board
may receive suggestions for themes. The choice of themes is made on the basis of their clinical
relevance and their scientific, social and political relevance. Preference is given to articles that
propose best practices in a specific field or context, or that question existing practices or poli-
cies based on available research findings. In every instance, the value of an article is assessed on
the basis of its scientific merit and its potential for improving practices. All articles undergo
anonymous peer review before being accepted and published.
www.ordrepsy.qc.ca/scienceandpractice

Integrating Science and Practice is published by the Communications Department of
Vol. 2 no. 2 noVember 2012

the Ordre des psychologues du Québec. The reproduction and distribution of texts are
authorized for non-profit purposes with reference to the source.

Editor: Martin Drapeau, Psychologist, Professor of Counselling Psychology and
Psychiatry, McGill University
Publication Coordinator: Diane Côté, Director of Communications,
Ordre des psychologues du Québec Legal deposit
Editorial Board: Bibliothèque nationale du Québec
Rose-Marie Charest, President, Ordre des psychologues du Québec ISBN 978-2-923164-49-6
Diane Côté, Director of Communications, Ordre des psychologues du Québec (Original version: 978-2-923164-48-9)
Pierre Desjardins, Psychologist, Director of Quality and Development of ISSN 1923-189X
Integrating Science and Practice

Professional Practice, Ordre des psychologues du Québec
Martin Drapeau, Psychologist, member of the Ordre des psychologues du Québec Ordre des psychologues du Québec
board of directors and Professor at McGill University 1100 Beaumont avenue, suite 510
Nathalie Girouard, Psychologist, Advisor on Quality and Development of Mont-Royal, QC H3P 3H5
Professional Practice, Ordre des psychologues du Québec www.ordrepsy.qc.ca/en

Special thanks to: Dr. Marilyn Fitzpatrick, Louise Overington, and Gabriela Ionita
from McGill University, for their contribution to this issue.

2

TABLE OF CONTENTS INTEGRATING SCIENCE AND PRACTICE noVember 2012

The Value of Progress Tracking in Psychotherapy 5
martin Drapeau

The CelestHealth System 7
Craig J. bryan, Stephen mark Kopta, bryan D. lowes

The Core-om (Clinical outcomes in routine evaluation) and Its Derivatives 12
Chris evans

The bASIS-24 behavior and Symptom Identification Scale 16
Thomaskutty b. Idiculla, Susan V. eisen

The Integra /ComPASS Tracking Assessment System 20
robert J. lueger

The outcome Questionnaire-45 24
michael J. lambert

The outcome and Session rating Scales 28
Scott D. miller, Susanne bargmann

The Polaris-mH 32
Grant Grissom,Tina Harralson, Jesse nankin

The PSYCHloPS (Psychological outcome Profiles) 36
mark Ashworth, maria Kordowicz, Peter Schofield

The Schwartz outcome Scale - 10 (SoS-10) 40
mark A. blais

The Treatment outcome Package (ToP) 43
David r. Kraus

3

Every English and French issue of Integrating Science and Practice is available
on the Ordre des psychologues du Québec website:
www.ordrepsy.qc.ca/scienceandpratice

Integrating Science and Practice Cahier recherche et pratique
_ Documenting the effectiveness _ Documenter l’efficacité des
of Psychotherapeutic Interventions interventions en psychothérapie

_ Depression: Considerations _ La dépression : considérations
Surrounding Treatment Choices autour des choix de traitements

_ Addressing Childhood _ Reconnaître et traiter la

Suffering souffrance des enfants

4

EDITORIAL

The Value of Progress Tracking
in Psychotherapy

A previous issue of Science and Practice (March 2010) was expanded upon by the APA Council of Representatives in 20122.
dedicated to the assessment of one’s clinical practice, more specif- Likewise, in 2012, the Canadian Psychological Association (CPA)
ically to the value of systematically documenting the effects of the struck a Presidential Taskforce on Evidence Based Practice3 that
treatment one offers. The value of progress tracking is unquestion- defined evidence-based practice as “the conscientious, explicit and
able, as it addresses the question that ultimately matters most: is judicious use of the best available research evidence to inform each
this treatment, as I am delivering it now, helping this patient sitting stage of clinical decision making and service delivery (… and the
in front of me? application of this knowledge…) in the context of specific client
Asking this question at an individual level, for each characteristics, cultural backgrounds, and treatment preferences”.
patient, and not only at a populational level, is certainly something However, the Taskforce also innovated by adding to this that
that distinguishes psychologists from most policy makers. Indeed, “following the initiation of treatment, data obtained from the on-
psychologists often move well beyond a patient’s diagnosis and going monitoring of clients’ reactions, symptoms, and functioning
treatment recommendations that are based exclusively on this di- should be used to modify or discontinue the selected treatment”.
agnosis to also take into consideration a number of other impor- Hence, the very definition of evidence-based practice includes on-
tant factors that can affect treatment outcome. There are of course going treatment monitoring and progress tracking. Good practice

good reasons for this. First, the DSM does not aim to explicitly and is not only evidence based; it is also practice based.
systematically account for extraneous and personal factors, such as This is an important step forward, which is perfectly
social support available, marital status, or other factors that may af- congruent with what science has taught us to date. A decade of re-
fect a patient’s prognosis, which in and of itself does raise some search has shown the value of progress tracking, and its added ben-
questions about psychotherapy recommendations that are based efit to the practice of psychotherapy. It can help clinicians who do
solely on diagnosis. The need to move beyond diagnosis and diag- not notice when a patient is deteriorating and even predict poor
nosis-based treatment recommendations is also supported by re- outcome before it happens (e.g., see Hannan et al., 2005). Second,
search. A number of studies have shown that numerous patient progress tracking can improve retention and adherence to treat-
characteristics account for variance in treatment outcome well be- ment and even improve treatment outcome (e.g., Anker et al., 2009;

yond the effects related to fit between treatment type and diagno- Bickman et al., 2011; Reese et al., 2010; Shimokawa et al., 2010;
sis (e.g., Beutler et al., 2011; Joyce et al., 2007). see also the special issue of Canadian Psychology on progress track-
It is no surprise, then, that the conclusions of the ing4). It can also help to direct the clinicians’ attention to areas and
American Psychological Association (APA) Presidential Task Force domains where they may require additional training or supervision,
on Evidence Based Practice in Psychology (20061) emphasized the or help them identify those patients with whom they are most
integration of patient, relational and treatment variables. Likewise, effective. Unlike symptom measures, progress tracking methods in-
the APA and the Order of psychologists of Quebec statements dicate areas that are problematic in the patient’s life (e.g. family,

on evidence based practice define evidence-based practice in etc.), by such providing the clinician with clinically useful informa-
psychology as the integration of the best available research with tion that can translate into specific techniques or lead to the dis-
clinical expertise in the context of patient characteristics, culture, cussion of specific topics within therapy. Furthermore, patients
and preferences. This statement was reiterated and in many ways respond favourably to progress tracking (e.g., Anker et al., 2011),

1 See http://www.apa.org/practice/resources/evidence/evidence-based-statement.pdf
2 See http://www.apa.org/about/governance/council/index.aspx
3 See http://www.cpa.ca/researchers/sciencedirectorate/defebp/
4 See Canadian Psychology (2012), vol. 53, issue 2.
5

and, finally, a number of progress tracking methods provide tools measure and the technical considerations tied to using the meas-
and advice to help the clinician adjust the treatment on an ongoing ure, and provides a brief overview of the settings and institutions
basis. that have chosen that measure for their clinicians to use. Each
With this issue, our goal is to provide readers with an of these measures is short, both clinician and patient friendly, pan-
easy to use introduction to the most popular progress tracking theoretical, and can be used in private practice as well as in other
measures. Each paper presents one measure and is written by the clinical settings.
author of, or important contributors to that measure. All authors We had received very positive feedback about the
kindly agreed to structure their paper in the same manner. Hence, previous issue of Science and Practice on progress tracking. Along
each paper succinctly presents the measure, the populations it can with this feedback, readers also asked where they can find progress
be used with, the languages it is available in, the domains it as- tracking measures and what they actually look like. This issue, which
sesses, how it is used, and how it can be useful in treatment plan- presents 10 of the most widely used methods, is our response to
ning. It also briefly presents the psychometric properties of the those questions.

martin Drapeau, Ph.D., Psychologist
editor,
Professor of Counselling Psychology
and Psychiatry,

mcGill university

REFERENCES

Anker, M. G., Duncan, B. L., & Sparks, J. A. (2009). Using client feedback to improve Hannan, C., Lambert, M. J., Harmon, C., Nielsen, S. L., Smart, D. W., Shimokawa,
couple therapy outcomes: A randomized clinical trial in a naturalistic setting. K., & Sutton, S. W. (2005). A lab test and algorithms for identifying clients at risk
Journal of Consulting and Clinical Psychology, 77, 693–704. for treatment failure. Journal of Clinical Psychology, 61, 155–163.
Anker, M., Sparks, J., Duncan, B., & Stapnes, A. (2011). Footprints of couple therapy: Joyce, A. S., Piper, W. E., Ogrodniczuk, J. S., & Klein, R. H. (2007). Patient characteristics
Client reflections at follow up using a mixed method design in routine care. and variations in termination processes and outcomes. In A. S. Joyce et coll. (éd.),
Journal of Family Psychotherapy, 22, 22-45. Termination in psychotherapy: A psychodynamic model of processes and
outcomes, pp. 109-131, Washington, DC: American Psychological Association.

Beutler, L. E., Harwood, T. M., Kimpara, S., Verdirame, D., Blau, K. (2011). Coping style.
Journal of Clinical Psychology, 67(2),176-183. Reese, R. J., Toland, M. D., Slone, N. C., & Norsworthy, L. A. (2010). Effect of client
Beutler, L. E., Harwood, T. M., Bertoni, M., & Thomann, J. (2006). Systematic treatment feedback on couple psychotherapy outcomes. Psychotherapy, 47(4), 616-630.
selection and prescriptive therapy. In G. Stricker et coll. (éd.), A casebook of Shimokawa, K., Lambert, M. J., & Smart, D. W. (2010). Enhancing treatment outcome
psychotherapy integration, p. 29-41, Washington, DC: American Psychological of patients at risk of treatment failure: Meta-analytic and mega-analytic review of
Association. a psychotherapy quality assurance system. Journal of Consulting and Clinical
Bickman, L., Kelley, S. D., Breda, C., de Andrade, A. R., & Riemer, M. (2011). Effects Psychology, 78, 298–311.
of routine feedback to clinicians on mental health outcomes of youths: results
of a randomized trial. Psychiatric Services, 62(12),1423-1429.
6

The The CelestHealth System (CHS-MH) is the evolved tool
of over 30 years of mental health treatment outcomes assessment.
In the early days, the belief was more items meant better assess-
CelestHealth ment. In the late 1970’s, paper and pencil questionnaires became
popular with research studies investigating self-reported outcomes.
System In the early 1980‘s, clinical tracking began with paper and pencil
questionnaires given at every session. This tracking approach was
incorporated into studies investigating dose-effect relationships
Prof. Craig J. bryan, PsyD, AbPP across sessions, where dose is the number of sessions and effect
The university of utah & was the probability of improvement (e.g., Howard, Kopta, Krause &
The national Center for Veterans Studies
Orlinsky, 1986; Kopta, Howard, Lowry, and Beutler, 1994). Offered
by Integra Incorporated, COMPASS (Howard, Brill, Lueger,
craig.bryan@utah.edu O’Mahoney & Grissom, 1995; Sperry, Brill, Howard & Grissom, 1996)
arose in the 1990’s as the most comprehensive outcomes assess-
Prof. Stephen mark Kopta, Ph.D., H.S.P.P., ment system featuring measures for mental health, psychothera-
university of evansville peutic bond, patient satisfaction, treatment need/expectations,
bryan D. lowes, nSCI Group, Washington Crossing, and presenting problems. It totaled 123 items for the patient to
Pennsylvania complete and 7 life functioning items that the clinician completed.
For a fee, COMPASS responses were faxed to an Integra office and
a report was returned by fax to the clinician. At each session, the
clinician reviewed the new clinical report with the patient. However,
in the late 1990’s, the demand for briefer questionnaires and more

immediately available outcomes increased.
To answer this demand, the CHS-MH was created in
part from COMPASS, with updated technology using computerized
The authors present the CelestHealth System, which can systems to provide a quicker clinical report to the clinician. The
be used to monitor patient progress during psychotherapy. System features four instruments: (a) the 20-item Behavioral Health
The System includes four instruments: the 43-item
behavioral Health measure-43 (bHm-43), which assesses
Measure-20 (BHM-20) assesses mental health and takes 90 seconds
overall mental health functioning; the 20-item behavioral to complete; (b) the 43-item Behavioral Health Measure-43 also as-
Health measure-20 (bHm-20), a shorter version of the sesses mental health with additional subscales, and takes 3 minutes
bHm-43; the the 5-item Psychotherapy readiness Scale,
which can be used to predict risks of poor treatment to complete; (c) the 5-item Psychotherapy Readiness Scale predicts

outcomes; and the 6-item Therapeutic bond Scale, which risk to do poorly in psychotherapy and takes 30 seconds to com-
evaluates the relationship between the psychotherapist plete; and (d) the 6-item Therapeutic Bond Scale, evaluates the rela-
and patient. Clinicians can select within the system which
instruments to use and how frequently they are adminis- tionship between the psychotherapist and patient in 30 seconds.
tered. The system is appropriate for adults 18 years and All four instruments are optional, with the clinician selecting within
older of normal or greater intelligence, and can be used in
the system which instruments to use and how frequently they are
outpatient mental health settings, primary care medicine,
and college counseling center settings. This paper presents administered (e.g. every session, only pre-post treatment).The
the system, reports on its psychometric properties, and CHS-MH is appropriate for adults 18 years and older of normal or
describes how the system can assist in treatment

planning and delivery.
greater intelligence. It has been used in outpatient mental health,
primary care medicine, and college counseling center settings. It is
Keywords: CelestHealth System; CHS-mH; behavioral
Health measure; bHQ; Psychotherapy readiness Scale; available in English, Spanish, and Vietnamese.
Therapeutic bond Scale; treatment outcome; progress
monitoring; psychotherapy

7

Domains Assessed health, self-management, and sexual func- All client raw responses and
The BHM-20 measures the most fre- tioning concerns. The CHS-MH is compati- the calculated scale scores are stored in
quently endorsed symptoms in outpatient ble with most clinical theories and highly secure CHS-MH servers and can be
psychotherapy consistent with the three practices. used for research (with approval from an
phases of mental health change. This appropriate institutional review board)
phase model proposes that improvement Use and Procedures and administrative purposes. An impor-
in behavioral health occurs in three pro- The CHS-MH input and output informa- tant new feature is the CelestHealth Treat-
gressive, sequential stages across therapy tion is securely communicated across com- ment Outcomes Profile, which uses
sessions, with improvement at each phase puter networks to a centralized processing collected data to provide an overview of
being contingent upon improvement in system. Before the session, the patient en- the clinical status and progress for all
the previous stage. First, the client obtains ters his/her responses using a computer clients within the center. This profile shows
a greater sense of well-being with in- (e.g., Netbook, iPad, or desktop computer) a variety of outcome variables including
creased optimism and hope. Next, specific that is available in the waiting room. Using percentage of patients recovered, im-
symptoms such as panic attacks, depres- simple and familiar browser-based inter- proved, unchanged, and deteriorated for
sive thinking, episodes of binge eating, faces, patients respond to a maximum of the most recent session, as well as distress
and sleep disturbance diminish. Finally, 31 multiple choice items; the typical time levels at intake and at most recent session.
life functioning improves across areas such for completion of the entire CHS-MH is 2.5 These variables are provided for all scales
as work, as a parent and partner, and in life minutes. The patient’s responses are ana- and subscales of the BHM-20, and the sys-
enjoyment. The phase model of psy- lyzed and scored in secure CHS-MH tem is updated frequently throughout the
chotherapy has been validated in several servers; a complete, formatted report is im- day to provide a “real-time” snapshot of the
research studies (e.g., Howard, Lueger, mediately available to the clinician as soon center’s client population. Date ranges can
Maling & Martinovich, 1993; Leon, Kopta, as the patient finishes the assessment. The also be applied to look at the center’s sta-

Howard & Lutz, 1999; Lutz, Lowry, Kopta, report produces several color-coded tus over historical timeframes.
Einstein & Howard, 2001; Stulz & Lutz, graphs and tables. Dose-Outcome graphs The annual license fee for
2007). for the subscales show the patient’s centers that do not charge for services (e,g,
Within the phase domains, progress across sessions, the Behavioral college counseling centers) is $US110 per
the BHM-20 assesses several mental health Health Profile displays color-coded sub- full time equivalent clinician who uses in-
problems: (a) well-being (distress, life scale scores based on normative data, and formation from the system, with a mini-
satisfaction, motivation), (b) psychological the Suicide Monitoring Scale helps clini- mum cost of $US550. For a private practice
symptoms (depression, anxiety, panic dis- cians to consider current suicide risk level. or center who charges for services, then
order, mood swings associated with bipo- The clinician discusses with the patient the license fee is based on number of in-

lar disorder, eating disorder, alcohol/drug his/her distress levels (i.e., severe, moder- takes because use of the system can be
abuse, suicidality, risk of violence), and (c) ate, mild, normal) and level of suicide risk billed to insurance companies and
life functioning (work/school performance, (high, moderate, low, no risk) as indicated Medicare/Medicaid. There are no set-up
intimate relationships, social relationships, by the Behavioral Health Profile. If chosen costs; consulting with CelestHealth staff is
life enjoyment). Within each phase do- as an option, the clinician also discusses re- free by phone or email.
main, the longer BHM-43 has more sub- sults from the Bond Scale and the Psy-
scales comprised of less frequently chotherapy Readiness Scale. With this Assessment and Treatment
Planning

endorsed problems in outpatient psy- method, both patient and clinician can see
chotherapy. For example, the symptoms which problems need to be targeted and The CHS-MH streamlines the assessment
scale measures hostility, sleep disorder, followed as treatment moves forward in of clients’ symptoms and functioning by
obsessive-compulsive symptoms, and psy- time. Based on this feedback, technical helping clinicians to more rapidly identify
chotic symptoms, and the life functioning adjustments within therapy can be made problem areas and symptom domains of
scale additionally measures physical to further reduce the patient’s psy- interest. From a clinical perspective, clini-
chopathology and increase motivation. cians and clients can quickly determine
8

current clinical severity and historical outpatients). Internal consistency coeffi- as a method for quickly and reliably as-
trends over the course of treatment be- cients ranged from .89 to .90 for the Global sessing clients’ “vital signs” of mental
cause of the CHS-MH’s ability to immedi- Mental Health score. For the three phases, health. Clients additionally report positive
ately calculate and display scores using a the ranges were as follows: Well-Being, feelings about the use of the CHS-MH
color-coded scheme. Clinicians can use the .65 to .74; Symptoms, .85 to .86; and Life when clinicians review and refer to the
CHS-MH’s output to ask more targeted as- Functioning.72 to .77. Construct validity clients’ responses during the appointment.
sessment questions and to focus conver- analyses using the discriminant validity Clinicians have reported that the system’s
sations on specific factors that are more method showed significant differences brevity and reliability/validity, as well as its
proximally related to treatment outcomes (p < .001) between the samples for all four separate measurement of life functioning
(e.g., interventions, life events). The color- scales, with each scale distinguishing from more generalized symptom clusters,
coded feedback system also can be used clinical from nonclinical groups. Sensitiv- are particularly attractive features. Primary
to facilitate interventions themselves. For ity to change using college counseling and care providers have similarly reported pos-
example, a client’s responses or scores can the psychotherapy outpatient samples itive feedback because of the CHS-MH’s
be used to disconfirm their beliefs that showed improved outcomes when com- separate measurement of daily function-
treatment is not working, or to reinforce paring intake scores to session 3 scores for ing. The separation of a functional domain
adherence and motivation for treatment all scales. Concurrent validity is supported is not common among outcomes meas-
(e.g., demonstrating improvement from based on very high correlations of BHM-20 urement tools, and is especially useful for
one session to the next despite the pa- scales with the other well-known meas- clinicians’ case conceptualization, diag-
tient’s assertion that “things aren’t getting ures of mental health functioning includ- nostic impressions, and treatment plan-
any better”). The CHS-MH can therefore ing the BASIS-32 (-.83), COMPASS (-.76), ning.
enhance clinical accuracy and efficiency, OQ-45 (-.81), and the SCL-R-90 (-.85). More
and can facilitate successful treatment recently, the BHM-20’s psychometric prop- Unique Features of the Measure

outcomes. erties were investigated across seven sep- Perhaps the CHS-MH’s most distinguishing
arate samples (Blount et al., 2010): four feature is its electronic, web-based format
Technical Support primary care samples, two clinical samples and immediate feedback system that can
Support includes three electronic manu- of deployed military personnel, and one be implemented across computing plat-
als: (a) the CelestHealth Getting Started nonclinical sample of deployed military forms (e.g., PC, Apple, smart phone,
Manual describes the procedures for set- personnel. Internal reliability estimates tablets, etc.). This capability makes the
ting up administrators, clinicians, and were consistent with Kopta and Lowry’s CHS-MH system especially flexible and
clients on the CHS-MH; (b) the CelestHealth (2002) findings. The four scales also practical to use, and has contributed di-
Clinical Report Manual explains to the demonstrated medium to large correla- rectly to its easy implementation across a

clinician how to understand the CHS-MH tions in the expected directions with more wide range of clinical settings. For general
output; and (c) the CelestHealth specific mood and symptom scales (e.g., clinical settings such as an outpatient psy-
Psychotherapist Manual instructs the psy- happiness, fatigue, anxiety, depression, chotherapy practice, primary care, or
chotherapist how to use the system with PTSD). Further psychometric evaluation of emergency departments, the CHS-MH’s
the client. Free online and telephone the BHM-20 is ongoing. design for measuring generalized mental
support is also available. health functioning, as opposed to diagno-
Patient /Client/Clinician Feedback sis- or condition-specific symptoms, is es-
Psychometric Properties

Positive feedback regarding the simplicity pecially beneficial. Within primary care, for
The BHM-20 has demonstrated good reli- and practicality of the CHS-MH has been example, the BHM-20 can be a better indi-
ability and validity. An initial psychometric received from both clients and clinicians, cator of mental health functioning than
evaluation was conducted by Kopta and especially regarding the color-coded feed- more limited or restricted symptom meas-
Lowry (2002) using four samples (i.e., com- back feature. Acceptability of the CHS-MH ures of depression or anxiety given the
munity adults, college students, college system is especially enhanced within med- wide spectrum of clinical issues that
counseling clients, and psychotherapy ical settings when presented or described typically present in this setting (e.g.,
9

depression, anxiety, weight management, Administered at intake, the client responds Within primary care, the
diabetes management, insomnia, chronic to five items in about 30 seconds that re- CHS-MH has been used to track outcomes
pain, etc). late to the duration of presenting prob- for both general patient populations
The BHM-20 additionally in- lems, previous psychotherapy experience, (Bryan, Morrow & Appolonio, 2009; Corso,
cludes screening items for suicidal ideation and motivation for treatment. Using Bryan, Corso, Morrow & Kanzler, 2010;
and impulses that have been shown to im- Global Mental Health as the outcome vari- Ray-Sannerud et al., 2011) and for specific
prove the detection of suicidal patients six- able, the scale has been shown to distin- subpopulations (e.g., PTSD; Cigrang et al.,
fold in primary care relative to standard guish patients demonstrating poor 2011; Corso et al., 2009). The CHS-MH’s
interviewing and assessment approaches psychotherapy outcomes from those BHM-20 has additionally been used to
by primary care providers (Bryan et al., showing good psychotherapy outcomes improve the detection of suicidal patients
2008). The CHS-MH’s Suicide Monitoring (Kopta, 2010). in primary care clinics (Bryan, Corso,
System (SMS) was additionally developed Rudd & Cordero, 2008), and is currently
in collaboration with suicide experts to aid Institutional Implementation undergoing pilot testing as a suicide as-
clinicians in tracking and managing sui- CHS-MH is used in college counseling set- sessment aid in emergency departments.
cide risk over the course of treatment in a tings including Harvard University, Johns The CHS-MH’s recent addition of an option
more reliable manner (Kopta, Mond, Hopkins University, University of Min- for tracking psychotropic medication
David, Potruzski & Doll, 2010), thereby nesota, Indiana University, and the Univer- along with clinical outcomes provides an
helping clinicians to meet standards of sity of Florida. Other settings include especially useful tool for health care
care for suicide risk assessment and man- primary care medical clinics at several U.S. providers of all disciplines, but most
agement. Air Force Bases and university medical cen- notably psychiatrists and nonpsychiatric
The Psychotherapy Readi- ters, as well as and private mental health prescribers (e.g., primary care physicians).
ness Scale of the CHS-MH is unique as a clinics. The CHS-MH has been imple-

reliable, valid alert for distinguishing pa- mented by mental health professionals
tients who do poorly in psychotherapy. deployed to Iraq to track clinical outcomes.

10

REFERENCES

Beck, A.T., Ward, C., & Mendelson, M. (1961). Beck Depression Inventory (BDI). Howard, K.I., Lueger, R.J., Maling, M.S., and Martinovich, Z. (1993). A phase model of
Archives of General Psychiatry 4, 561–571. psychotherapy: Causal mediation of outcome. Journal of Consulting and Clinical
Blount, T.H., Bryan, C.J., Kanzler, K.E., Morrow, C.E., Corso, K.A., Corso, M.L. (2010, Psychology, 38, 139-149.
November). Psychometric properties of the BHM-20 in military samples. Poster Kopta, S.M. (2010, October). The future is here for college counseling centers: Counseling
presented at the annual meeting of the Association for Behavioral and Cognitive works, preventing suicide, and improving outcomes. Workshop presented at the
Therapy, San Francisco, CA. Annual University and College Counseling Center Directors (AUCCCD) Conference,
Bryan, C.J., Corso, K.A., Rudd, M.D., & Cordero, L. (2008). Improving identification of Portland, Oregon.
suicidal patients in primary care through routine screening. Primary Care and Kopta, S.M., & Lowry, J.L. (2002). Psychometric evaluation of the Behavioral Health
Community Psychiatry, 13, 143-147. Questionnaire-20: A brief instrument for assessing global mental health and the
Bryan, C.J., Morrow, C.E., & Appolonio, K.A.K. (2009). Impact of behavioral health three phases of psychotherapy outcome. Psychotherapy Research, 12, 413-426.
consultant interventions on patient symptoms and functioning in an integrated Kopta, S.M., Howard, K.I., Lowry, J.L., & Beutler, L.E. (1994). Patterns of symptomatic
family medicine clinic. Journal of Clinical Psychology, 65, 281-293. recovery in psychotherapy. Journal of Consulting and Clinical Psychology, 62,
Cigrang, J. A., Rauch, S. A. M., Avila, A. L., Bryan, C. J., Goodie, J. L., Hryshko-Mullen, A., 1009-1016.
Peterson, A. L., & the STRONG STAR Consortium. (2011). Treatment of active-duty Kopta, S.M., Mond, M., David, L., Potruzski, N., & Doll, L. (2010, June). Assessing
military with PTSD in primary care: early findings. Psychological Services, 8, 104-113. suicidality: Validation of the Suicide Monitoring Scale of the Behavioral Health
Corso, K.A., Bryan, C.J., Morrow, C.E., Appolonio, K.K., Dodendorf, D.M., & Baker, M.T. Measure-20 and implications for psychotherapeutic strategies. Paper presented at
(2009). Managing posttraumatic stress disorder symptoms in active duty military the annual meeting of the International Society for Psychotherapy Research,
personnel in primary care settings. Journal of Mental Health Counseling, 31, Monterey.
119-136. Leon, S.C., Kopta, S.M., Howard, K.I., & Lutz, W. (1999). Predicting patients’ responses
Corso, M., Bryan, C.J., Corso, K.A., Morrow, C.E., & Kanzler, K.E. (2010, April). Mental to psychotherapy: Are some more predictable than others? Journal of Consulting
health functioning recovery curves associated with behavioral health consultant and Clinical Psychology, 67, 698-704.
services in integrated primary care. Poster presented at the annual meeting of Lutz, W., Lowry, J.L., Kopta, S.M., Einstein, D., & Howard, K.I. (2001). Prediction of
the Society of Behavioral Medicine, Seattle, WA. dose-response relations based patient characteristics. Journal of Clinical Psychology,
Derogatis, L.R. (1983). Administration, scoring, and procedures manual—II. Baltimore, 57, 889-900.
MD: Clinical Psychometric Research. Ray-Sannerud, B., Bryan, C.J., Dolan, D., Morrow, C.E., Corso, K.A., Kanzler, K.E., &

Horowitz, L. M., Rosenberg, S. E., Baer, B. A., Ureño, G., & Villaseñor, V. S. (1988). Corso, M.L. (April 2011). Preliminary evidence for long-term outcomes following brief
The Inventory of Interpersonal Problems: Psychometric properties and clinical behavioral health intervention in primary care clinics. Poster presented at the annual
applications. Journal of Consulting and Clinical Psychology, 56, 885-895. meeting of the Society of Behavioral Medicine, Washington, DC.
Howard, K.I., Brill, P.L., Lueger, R.J., O’Mahoney, M.T., & Grissom, G.R. (1995). Integra Sperry, L., Brill, P.I., Howard, K.I., and Grissom, G.R. (1996). Treatment outcomes in
outpatient tracking assessment. Philadelphia: Compass Information Services, Inc. psychotherapy and psychiatric interventions. New York: Brunner/Mazel.
Howard, K.I., Kopta, S.M., Krause, M.S., & Orlinsky, D.E. (1986). The dose-effect Stultz, N. & Lutz, W. (2007). Multidimensional patterns of change in outpatient
relationship in psychotherapy. American Psychologist, 41, 159 164. psychotherapy: The phase model revisited. Journal of Clinical Psychology, 63,
817-833.


11

The CORE-OM The CORE-OM (Chris Evans et al., 2000) is a 34 item
self-report questionnaire designed to measure change in mental
health of adults, particularly change brought about by psycholog-
(Clinical Outcomes ical therapies. It was launched in 1998 as a central part of the “CORE
system” complemented by a practitioner completed instrument
in Routine (CORE-A) comprising the CORE-TAF (Therapy Assessment Form) and
CORE-EoT (End of Therapy) and by two 18 item shortened forms for

Evaluation) repeated use (CORE-SF/A and SF/B; Michael Barkham, Margison, et
al., 2001). Subsequent work produced another shortened version
for general population surveys (GP-CORE; Sinclair, Barkham, Evans,
and its derivatives Connell & Audin, 2005), and two more short forms for sessional and
screening use: CORE-10 (in prep.) and CORE-5 (Wright, Bewick,
Barkham, House & Hill, 2009). It has been translated into 22 lan-
guages to date, including a French version currently in progress,
and adaptations exist for adolescents (YP-CORE; Twigg et al., 2010),
Dr. Chris evans1, mrCPsych., m.Sc.,
Institute of Group Analysis, for people with learning difficulties (CORE-LD Brooks & Davies, 2008;
nottinghamshire Personality Disorder and in prep.) and for description of families (SCORE-15; Stratton,
and Development network,
Bland, Janes & Lask, 2010 and SCORE-28; Cahill, O’Reilly, Carr,
mandala Centre
Dooley & Stratton, 2010). A health economic Quality of Life (QoL)
chris@psyctc.org scoring is emerging (Mavranezouli, Brazier, Young & Barkham, 2010)
as are algorithms or lookup tables to map CORE-OM scores to and
from scores on other questionnaires such as the BDI-II (Leach et al.,

2006).
The CORE-OM assesses change in as wide a group of
clients as possible, from those with no problems to those with very
serious thoughts of suicide, self-harm or other severe distress. It was
This paper presents the Clinical outcomes in routine
evaluation (Core-om), a 34 item self-report questionnaire not designed for forensic services nor for people with current para-
designed to measure change in the mental health of noid disorders who might mistrust its use, though it is as un-
adults in the context of psychotherapy service delivery.
provocative for such clients as possible. The measure was also not
The questionnaire includes items with a primarily intra-
personal focus and others with a primarily interpersonal intended for use by adolescents and a 10 item YP-CORE (Young Per-
focus. It assesses a number of different domains, including son’s CORE) has been derived by selection and simplification from

client well-being, problems and symptoms, functioning
and risk. All Core-om derived measures can be scored
the CORE-OM for ages 11-16 (Twigg et al., 2010). A 14 item related
easily by hand with scoring boxes on the instruments. measure, LD-CORE for use with adults with mild to moderate learn-
Particular strengths of the Core-om include its broad ing difficulties (LD) has also been developed though this includes
domain coverage with risk "flag" items; the existence of
well tested short forms; adaptations for families, young items not in the CORE-OM as the problems faced by people with LD
people and people with learning difficulties; the mapping are not the same as those without LD.
of some items to health economic valuation; and
the availability of the system in 22 languages. The
psychometric properties of the questionnaire are

discussed along with its use in treatment planning and
delivery and the procedures, costs and training it requires.

Keywords: Clinical outcomes in routine evaluation;
Core-om; treatment outcome; progress monitoring;
psychotherapy
1 Dr. Evans is a Trustee of CORE System Trust (CST), the not-for-profit company that holds the copyright
on the CORE instruments. Like the other CST Trustees, Dr. Evans receives no income from the CORE-OM
nor from CORE-IMS. This paper is a personal view, not the view of CST or CORE-IMS.

12

Domains Assessed only one company, CORE Informa- CORE-IMS have supported
The design ensured some items of prima- tion Management Systems (CORE-IMS, session-by-session presentation of indi-
rily intrapersonal focus and others prima- www.coreims. co.uk) has permission to put vidual client data since 1999 and two
rily interpersonal and to cover well-being, CORE instruments into software for non- shortened, 18 items forms (CORE-SF-A/B)
problems/symptoms, functioning and risk. research use. were provided from the CORE-OM launch.
An early challenge was to CORE-IMS provide two com- The SF-A and SF-B have four well-being
obtain a measure not too dominated by puter solutions originally for CORE meas- items in common and 14 other items that
psychiatric diagnosis hence few items map ures alone but they now support over 30 are different, minimising memory effects.
to DSM or ICD though anxiety and depres- other measures, though sometimes with Over the last decade, expected length of
sion are well covered. No self-report meas- additional licence costs, and could be ex- routine measures has fallen and two other
ure can directly measure unconscious tended to cover any typical measure. Their short forms, the CORE-5 and CORE-10 are
functioning but many items fit with systems are CORE-PC and CORE-Net, the now available. In therapies appropriate to
Freud’s aims that a successful therapy re- former a standalone PC system and the lat- this, sessional use of the CORE-OM or
place neurotic misery with normal human ter a networked client-server system. shorter forms, supports score guided
unhappiness and his equally famous CORE-Net uses a routine web browser to adaptive therapy as championed by Lam-
aim that a good therapeutic outcome en- access a server, usually a CORE-IMS inter- bert (Lambert, 2010) and others.
hances the ability to love and to work. net server, though dedicated local servers
Objections from CBT therapists were can be used. Costs depend on usage with Assessment and Treatment
mainly that they preferred measures spe- CORE-PC available by annual license with Planning
cific to presenting problems. For all modal- a minimum license for 125 clients costing As for sessional use, our “bottom up”
ities it has been important to explain that £250 ($CA396) and additional clients philosophy means that how the CORE-OM
no measure should replace normal clinical charged from £2 ($CA3) downwards as vol- relates to assessment must be determined

information channels and the CORE-OM ume increases. CORE-Net is also provided locally. Some services position it as an ap-
no more replaces specific measures in CBT by annual license with license costs aver- praisal of the service and not part of the
than it replaces counter-transference in an- aging £3 ($CA4.75) per client. Neither CST therapy, thus minimising the clients’ use of
alytic work. Counsellors and humanistic nor CORE-IMS dictate what services do the measure to communicate to their ther-
therapists surprisingly proved the most with their data. The original CORE system apists and removing it from assessment.
rapid adopters perhaps reflecting that handbook from 1998 had a chapter on ap- Others make it part of assessment and find
item wording which was kept in lay lan- propriate informed consent to use identi- that it provides a structure with broad
guage. fiable data, congruent with European data coverage that often leads smoothly into
protection law. CORE system advice has al- discussion of risk and of particular prob-

Use and Procedures ways been that data will be useful, and lems.
All CORE-OM derived measures can be sometimes only legal, if the service/thera-
scored easily by hand with scoring and pist have thought clearly about planned Technical Support
scoring boxes on the instruments. All uses of the data so informed consent for CORE-IMS provide high quality support
measures have been formatted to be use can be obtained. All data held centrally both for the software but more impor-
scanned for optical character reading has always been scrupulously anonymised tantly to help services choose how best to
(OCR) and most modern OCR systems preventing central analysis from identify- use CORE (or other) scores. This support is

should read scores without any illegal ing services, therapists or clients. CORE- widely used in the UK, the Netherlands,
modification of the forms. All the CORE IMS and all the server companies used by Norway and Denmark with CORE-IMS
instruments are free for use on paper CORE-Net are accredited to ISO/IEC27001, support appropriate to different needs in
(www.coreims.co.uk/copyright.pdf). We an international standard for information different countries developing as demand
have always given permission for the item security. emerges. Extensive information about
text to be used in software for research but the entire CORE system is available at
www.coreims.co.uk.
13

Psychometric Properties Unique Features of the Measure using the CORE-OM range from primary
Psychometric properties are sometimes The CORE-OM is not unique: a number of care medical practitioners and psycholog-
reported as if they were as fixed properties good general measures exist, though few ical therapists, through a plethora of sec-
of measures but are, of course, statistical are free to reproduce on paper as the CORE ondary care psychological therapy services
parameters, i.e. empirical findings from measures are. The empirical literature on including the Tavistock Clinic, to prisons
samples generalised to populations. Trans- general and specific measures shows them and high secure hospitals. It is used in Clin-
lations will not show exactly the same all to have high covariance with only small ical Psychology trainings in Norway and by
psychometric properties as English UK variance specific to diagnosis or problem large provider organisations in the Nether-
samples (e.g. Chris Evans et al., 2002). area. In this convergent field, particular but lands and Norway and international use
So far, reliability and validity in different not unique strengths of the CORE-OM are: is growing steadily. YP-CORE is one of
samples using the English version and its broad domain coverage with risk “flag” the measures recommended by CORC
translations have been good. Overall in- items; the existence of well tested short (www.corc.uk.net) and the SCORE meas-
ternal reliability has been excellent, in the forms, adaptations for families, young peo- ures are being fostered by AFT
range of .92 to .94. Internal reliability for ple and people with learning difficulties; (www.aft.org.uk).
domain scores varies more across samples the mapping of six items to health eco- Service comparison was a
but is always acceptable. Test-retest relia- nomic valuation; and the existence of a primary driver; Michael Barkham, Margi-
bility is good but not excessive, as appro- standard translation protocol with 22 son, et al. (2001) and Evans, Connell,
priate to a change measure: typical good translations. Barkham, Marshall & Mellor-Clark (2003)
values are from .64 (for the risk domain) to are early examples of such work. This work
.91 (overall score). Discrimination between Institutional Implementation has been extended to specific areas, e.g.
clinical and non-clinical samples is always As the measure is copyleft there is no sin- university student counselling services
strong and sensitivity to change good. gle register of uptake. However, over 500 (Janice Connell, Barkham & Mellor-

Correlations with other instruments show services and over 5000 practitioners use Clark,2007, 2008). Further examples can be
strong convergent validity and often slight CORE-IMS software. In the UK, services found in the list of CORE related publica-
evidence of within-domain correlations tions (see next page).
being higher than cross-domain correla-
tions (Evans et al., 2002)2.

Patient /Client/Clinician Feedback
An excellent collection of accounts of
using the CORE-OM and other compo-

nents of the CORE system (Gray, Penny &
Mellor-Clark, John, 2007) is freely available
at http://www.coreims.co.uk/ site_
downloads/CORE-A-Decade -of-
Development.pdf. The CORE-OM emerged 2 The main criticisms of the psychometric properties of the CORE-OM come from our decision to facilitate reporting of domain scores.
well from user feedback in the NHS com- Publications and presentations by the CORE team have always recommended using domain scores only where there might be a specific
clinical or research interest in the domain and we never expected the domains to emerge as cross-sectional factors of variance. Initial analy-
pendium of approved mental health ses (Evans et al., 2002) using principal component analysis showed a large first factor, a second mainly involving the positively keyed items
and a third involving mainly the risk items. Criticism that the domains do not fall out as neat factors recurs. We believed we had laid these

outcome measures (National Institute for issues to rest with a hierarchical factor analysis showing second-order general factor and first-order factors of the domains and positively
and negatively keying methods factors and we noted that scale quality was satisfactory where the non-risk items are treated as a single scale
Mental Health England, 2008) and a recent and risk items as a second is satisfactory (Lyne, Barrett, Evans & Barkham, 2006). However, the fantasy that a 34 item measure could show
user-led review of measures (Crawford et a neat four factor structure in which well-being, problems, functioning and risk were factorially simple and distinct persists (e.g. Bedford
et al., 2010). We are aware of no psychological theory, no psychotherapy theory nor any empirical data from any measure, that has ever
al., 2011). suggested that this is the case. A recent development may help though. This work extracts a small set of items from measures to provide
a direct translation of scores to QALY (Quality of Life Year) valuations used for health economic (HE) life valuation through time trading
tests widely used for HE scales. This work, (Mavranezouli et al., 2010) which the same team have applied to other problem specific meas-
ures with equal success, neatly inverts the usual complaint noting that the CORE-OM’s complex factor structure and broad domain cover-
age gives a good set of anchor items for HE evaluation. Exploration of the CORE-OM in clinical samples led to selection of six items,
five covering psychological state and one physical state, that provide HE evaluation. Perhaps this will finally dent the fantasy that domain
scores would or could show a clean factor structure.
14

REFERENCES

An extensive list, currently of 123 publications many with abstracts and some with full Gray, Penny, & Mellor-Clark, John. (2007). CORE: a decade of development. Rugby,
text is at www.coreims.co.uk/Downloads_References.html and updated regularly. England: CORE IMS. Retrieved from
http://www.coreims.co.uk/site_downloads/CORE-A-Decade-of-Development.pdf
Barkham, Michael, Margison, F., Leach, C., Lucock, M., Mellor-Clark, J., Evans, C., Lambert, M. (2010). Prevention of treatment failure : the use of measuring,
Benson, L., et al. (2001). Service profiling and outcomes benchmarking using the monitoring, and feedback in clinical practice (1st ed.). Washington D.C.: American
CORE-OM: toward practice-based evidence in the psychological therapies. Psychological Association.
Journal of Consulting and Clinical Psychology, 69, 184-196.
Leach, C., Lucock, M., Barkham, M., Stiles, W. B., Noble, R., & Iveson, S. (2006).
Bedford, A., Watson, R., Lyne, J., Tibbles, J., Davies, F., & Deary, I. J. (2010). Mokken Transforming between Beck Depression Inventory and CORE-OM scores in routine
scaling and principal components analyses of the CORE-OM in a large clinical clinical practice. British journal of clinical psychology, 45, 153-166.
sample. Clinical Psychology & Psychotherapy, 17, 51-62. doi:10.1002/cpp.649
Lyne, K. D., Barrett, P., Evans, C., & Barkham, M. (2006). Dimensions of variation on the
Brooks, M., & Davies, S. (2008). Pathways to participatory research in developing a CORE-OM. British journal of clinical psychology, 45, 185-203.
tool to measure feelings. British Journal of Learning Disabilities, 36, 128-133.
doi:10.1111/j.1468-3156.2007.00476.x Mavranezouli, I., Brazier, J. E., Young, T. A., & Barkham, M. (2010). Using Rasch analysis
to form plausible health states amenable to valuation: the development of
Cahill, P., O’Reilly, K., Carr, A., Dooley, B., & Stratton, P. (2010). Validation of a 28-item CORE-6D from a measure of common mental health problems (CORE-OM).
version of the Systemic Clinical Outcome and Routine Evaluation in an Irish Quality of Life Research, 20, 321-333. doi:10.1007/s11136-010-9768-4
context: the SCORE-28. Journal of Family Therapy, 32, 210-231. doi:10.1111/j.
1467-6427.2010.00506.x National Institute for Mental Health England. (2008). Mental Health Outcomes
Compendium (p. 84). London: National Institute for Mental Health England.
Connell, Janice, Barkham, M., & Mellor-Clark, J. (2007). CORE-OM mental health norms Retrieved from http://www.dh.gov.uk/prod_consum_dh/groups/dh_digitalassets/
of students attending university counselling services benchmarked against an documents/digitalasset/dh_093677.pdf
age-matched primary care sample. British Journal of Guidance and Counselling,
35(1), 41-57. doi:10.1080/03069880601106781 Sinclair, A., Barkham, M., Evans, C., Connell, J., & Audin, K. (2005). Rationale and
development of a general population well-being measure: Psychometric status
Connell, Janice, Barkham, M., & Mellor-Clark, J. (2008). The effectiveness of UK student of the GP-CORE in a student sample. British journal of guidance and counselling, 33,
counselling services: an analysis using the CORE System. British Journal of 153-173.
Guidance & Counselling, 36, 1-18. doi:10.1080/03069880701715655
Stratton, P., Bland, J., Janes, E., & Lask, J. (2010). Developing an indicator of family
Crawford, M. J., Robotham, D., Thana, L., Patterson, S., Weaver, T., Barber, R., Wykes, T., function and a practicable outcome measure for systemic family and couple
et al. (2011). Selecting outcome measures in mental health: the views of service

therapy: The SCORE. Journal of Family Therapy, 32(3), 232-258.
users. Journal of Mental Health, 20, 336-346. doi:10.3109/09638237.2011.577114
Twigg, E., Barkham, M., Bewick, B. M., Mulhern, B., Connell, J., & Cooper, M. (2010).
Evans, C., Connell, J., Barkham, M., Marshall, C., & Mellor-Clark, J. (2003). Practice-based The Young Person’s CORE: Development of a brief outcome measure for young
evidence: Benchmarking NHS primary care counselling services at national and people. Counselling and Psychotherapy Research: Linking research with practice, 9,
local levels. Clinical Psychology and Psychotherapy, 10(6), 374-388. 160 - 168.
Evans, Chris, Connell, J., Barkham, M., Margison, F., McGrath, G., Mellor-Clark, J., &
Audin, K. (2002). Towards a standardised brief outcome measure: Psychometric Wright, F., Bewick, B. M., Barkham, M., House, A. O., & Hill, A. J. (2009). Co-occurrence of
properties and utility of the CORE-OM. British Journal of Psychiatry, 180(JAN.), 51-60. self-reported disordered eating and self-harm in UK university students. British
Evans, Chris, Mellor-Clark, J., Margison, F., Barkham, M., Audin, K., Connell, J., & Journal of Clinical Psychology, 48, 397-410. doi:10.1348/014466509X410343
McGrath, G. (2000). CORE: Clinical Outcomes in Routine Evaluation. Journal of
Mental Health, 9, 247-255.


15

The BASIS-24 The Behavior and Symptom Identification Scale
24 (BASIS-24), copyrighted by McLean Hospital, is a 24 item
patient self-report questionnaire designed to assess treatment out-
Behavior and comes by measuring symptoms and functional difficulties experi-
enced by individuals seeking mental health services. The original
Symptom tool, Behavior and Symptom Identification Scale 32 (BASIS–32) was
developed in the early 1980s to meet the need for a brief but com-

Identification Scale prehensive mental health status measure that would be useful in
assessing the outcomes of mental health treatment from the
consumer’s point of view. It is a measure of self-reported difficulty
Thomaskutty b. Idiculla, Ph.D., in the major symptom and functioning domains that lead to the
mclean Hospital,
Harvard medical School need for mental health services (Eisen, Dill & Grob, 1994).
BASIS-24 sought to improve upon BASIS-32 by in-
creasing applicability across diverse populations and improving the
tidiculla@mclean.harvard.edu reliability and validity of the instrument. Revision of the instrument
included (a) review of literature; (b) input from 75 researchers,
Susan V. eisen, Ph.D., Center for Health Quality, outcomes & administrators, clinical providers, and consumers; (c) readability
economic research (CHQoer), bedford, mA, VA Hospital, analysis; (d) review of survey question design principles and meth-
boston university School of Public Health
ods; (e) meeting of the research team to review progress and make
suggestions for the revision; (f) drafting of a revised instrument;
(g) cognitive testing of the revised instrument; (h) analysis of cog-
nitive test data; (i) further revisions of the instrument; (j) a second

round of cognitive testing; (k) analysis of the second round of cog-
nitive testing; and (l) further revisions and construction of the in-
strument for field-testing. The revised instrument was then tested
The behavior and Symptom Identification Scale 24 on over 6,000 individuals who were receiving inpatient or outpa-
(bASIS-24) is a 24 item patient self-report questionnaire
designed to assess treatment outcomes by measuring
tient treatment for mental health or substance abuse (Eisen et al.,
symptoms and functional difficulties experienced by 2004a).
individuals seeking mental health services. The survey is BASIS-24 is intended for adults, ages 18 and older,
intended for adults of ages 18 and older who present a
broad range of symptoms and problems at all levels of and is appropriate for individuals who present a broad spectrum of
care including in inpatient, residential, partial and symptoms and problems at all levels of care including inpatient, res-

outpatient settings. The bASIS-24, which has been idential, partial and outpatient. Furthermore, the instrument can be
translated into five languages, can be used across a wide
range of therapeutic treatment models. Scores are used across a wide range of therapeutic treatment models. It was
computed for the overall scale, as well as for six subscales designed to assess improvement over time regardless of type of
that assess depression and functioning; interpersonal
mental health treatment. A BASIS-24 Adolescent Pilot version is also
relationships; psychosis, substance abuse; emotional
lability, and self-harm. Studies using the questionnaire available. Both instruments can be used with multiple administra-
have shown that its use is associated with an increase tions throughout treatment and post-treatment follow-up (Eisen et
in patient satisfaction with care after the domain scores
were discussed with the patient as part of developing
al., 2004b). The adult version of BASIS-24 is available in English,

a treatment plan. In this paper, the authors describe the Spanish, Portuguese, French, and Russian.
measure, the procedures related to its use, its psychomet-
ric properties and the contexts in which it can be used.
Domains Assessed
Keywords: behavior and Symptom Identification Scale;
The BASIS-24 survey cuts across diagnoses, recognizing the wide
bASIS-24; treatment outcome; progress monitoring;
psychotherapy range of symptoms and problems that occur across the diagnostic
spectrum. BASIS-24 is designed to measure outcomes for a broad
range of treatments and services encompassing many theoretical
16

orientations. Scores can be computed for “response cards” on which the rating scale unlimited number of BASIS-24 administra-
the overall BASIS-24, as well as for six sub- is printed in large letters. Telephone inter- tions. Along with BASIS-24, clients may
scales: Depression and Functioning, Inter- views and mailed self-report question- also purchase WebScore, an optional on-
personal Relationships, Psychosis, naires can be used at discharge or line scoring and reporting application.
Substance Abuse, Emotional Lability, and termination and at follow-up time points The cost of WebScore is based on the esti-
Self-Harm. (Eisen et al., 2004a; McLean Hospital, 2006). mated number of surveys entered into
The first of these domains, Among the 24 items, each the online application for each year
Depression and Functioning, seeks to assess has five ordered response options report- (McLean Hospital, 2011).
daily/role functioning and depression ing either the level of difficulty experi-
and anxiety symptoms. The Interpersonal enced (no difficulty to extreme difficulty), Assessment and Treatment
Relationships domain evaluates the client’s or the frequency with which a symptom or Planning
perception of the quality of their interper- problem has occurred (none of the time to Evidence-based practice has been trend-
sonal experiences with family and others. all of the time). Respondents answer each ing to include analysis of individual patient
The Psychosis domain assesses four symp- question describing behavior and symp- responses for real-time intervention and
toms of psychotic disorders, such as hallu- toms during the past week. For example, treatment planning in addition to the
cinations and delusions. The Substance “During the past week, how much of the older models of aggregate benchmarking.
Abuse domain seeks information regarding time did you feel sad or depressed?” BASIS- Self-report measures are especially impor-
the client’s urge to drink or use drugs as 24 is administered at the beginning of a tant in this new trend of measurement
well as possible problems resulting from treatment episode, with repeat assess- tools as an adjunct to treatment planning
alcohol and/or drug use. The Emotional ments obtained at desired intervals to as- as they systematically inform providers
Lability domain, which measures what the sess change during or following treatment. about difficulties the patient may not oth-
Royal College of Psychiatrists describes as The 24 questions are scored on a 5-point erwise express in other clinical measures

an “excessive emotional response to a scale (from 0 to 4) and each subscale and (Newnham & Page, 2010). The use of a self-
minor stimulus,” includes three items overall mean scores also range from 0 to 4, report tool also allows the patient to take a
reflecting mood swings, racing thoughts with 0 being the lowest severity of symp- more active role in treatment planning.
and feeling short-tempered. Finally, toms and 4 being the highest severity of Studies using the BASIS-32 have shown an
the Self-Harm domain measures clients’ symptoms. The overall BASIS-24 score is a increase in patient satisfaction with care
thoughts about hurting themselves weighted sum that is computed by multi- after the domain scores were discussed
and/or ending their lives. plying the rating for each question by its with the patient as part of developing the
weight and totaling the weighted ratings treatment plan. The patients specifically
Use and Procedures for each question. The score for each sub- reported a greater feeling of involvement

BASIS-24 is typically self-administered scale is a weighted sum that is computed in care decisions and respect from the cli-
by the patient and takes between 5 and by multiplying the rating for each question nicians (Eisen, Dickey & Sederer, 2000). The
15 minutes to complete. The 24 items are in the subscale by its weight and totaling use of self-report measures in treatment
written at a fifth- to sixth-grade reading the weighted ratings for all questions in planning can be useful in more focused
level, which maximizes the number of in- the subscale (McLean Hospital, 2006). treatment as well as a better patient-clini-
dividuals who are able to complete the In order to use BASIS-24, a cian rapport.
questionnaire by themselves. However, it single-site client must purchase an annual Although the BASIS-24 is

can also be administered via computer, on site license for $US395. Organizations with not designed to replace a comprehensive
the telephone, or in an interview. When multiple sites can purchase additional site clinical evaluation, the tool documents the
the BASIS-24 is administered through a licenses for $US95 each. Included with the consumers’ perspectives on the symptoms
structured interview, a clinician, researcher, license fee are a BASIS Instruction Guide, a and problems that bring them to treat-
support staff member, or volunteer reads survey form for photocopying and the ment. It is also simple to incorporate into
the items to the respondent and elicits scoring procedures and algorithm for the a clinical evaluation process when already
ratings with the help of 8 × 11 laminated instrument. Each site license allows for an part of a Quality Improvement or outcome
17

assessment program, allowing it to fulfill did not improve in areas they had not https://secure.ebasis.org/basisdemo/login.php.
both roles in the same administration identified, as indicated by both clinician WebScore provides both pa-
(Eisen et al., 2004a). Using the same stan- and patient reports. Patient perception is tient-level and aggregate population-level
dardized tool for both individual planning shown to be a predictor of outcomes, and reporting capabilities, which can be sorted
and aggregate outcomes assessment also should thus be addressed at the planning by time point, level of care, gender, or age.
provides consistent metrics between the stage in order to maximize the effective- Reports can also be produced by patient
care objectives and its results. The con- ness of treatment. As evidence-based admission or discharge date. Results can
structs at the individual assessment level practice becomes more integrative of all be downloaded into a CSV/Excel file that
thus match those used to measure overall facets of care, it is important to use each can easily be imported into SPSS, SAS or
outcomes and assessment, leading to metric in one’s toolbox to its maximum other statistical software applications.
greater consistency. potential. For those clients who choose to utilize
The BASIS-24 can be used to WebScore for online scoring, eBASIS
identify primary and secondary problems Technical Support Systems ensures the highest standards of
from the individual consumer’s perspec- The process to use BASIS-24 begins with confidentiality and security, including
tive (Eisen et. al, 2004a). Where BASIS-24 preregistration on www.ebasis.org, where compliance with all HIPAA guidelines and
subscales overlap with diagnosis, there is the client can create an account and agree requirements. Data is stored in a secure
usually a consistency between subscale to the terms of a general service agree- server and individual cases are not identi-
scores and clinical diagnosis in that con- ment. Once payment and completed fied in any report or aggregate results.
sumers diagnosed with depression or anx- paperwork, including signed end-user li- All hard-copy patient information is stored
iety tend to report more difficulty with cense agreement, has been received, the in confidential, locked areas and paper sur-
depression and anxiety than do con- account will be approved by an eBASIS veys are shredded after 3 years, once all
sumers with other diagnoses (Eisen, Dill & staff member and clients will be able to data is verified, cleaned, and backed up

Grob, 1994). Frequently, however, con- begin using BASIS-24. Additionally, large (McLean Hospital, 2011).
sumers tend to report high levels of diffi- volume users can utilize BASIS-24’s optical The Performance Measure-
culty in areas that do not correspond to scanning forms. Clients can complete the ment System offers custom reports, in-
their diagnosis as well. Problems in inter- survey on these forms and the eBASIS staff cluding change scores for a given quarter,
personal relationships, managing day-to- will provide scanning services at a current control charts showing month-by-month
day life, and depression often tend to be charge of $US1 per form so that data does outcomes, and comparison charts com-
identified as more difficult than psychotic not need to be manually entered. paring the client’s data with national
symptoms for consumers diagnosed with Clients who wish to use benchmarks. Consultation is available re-
schizophrenia (Eisen et. al, 2004a). In these BASIS-24 have four levels of service avail- garding design of an outcome assessment

cases, whereas a clinician may see psy- able to them: (a) BASIS-24 license; (b) li- system, data collection, data manage-
chosis as the main focus of treatment, the cense and access to WebScore; (c) license, ment, and reporting, and can be cus-
consumer may identify other priority areas WebScore, and Performance Measurement tomized to meet the organization’s specific
for treatment. The BASIS-24 can thus high- System Reporting; or (d) consultation. needs.
light possible high levels of distress not di- WebScore is an internet-based scoring and
rectly symptomatic of primary diagnosis as reporting tool for the BASIS-24 survey. It is Psychometric Properties
well as areas in which the diagnosis has an easy-to-use data entry and reporting Confirmatory factor analysis conducted

had a negative impact on day-to-day func- application that lets users automatically with BASIS-24 items confirmed the six
tioning. In addition, Eisen and Grob (1982) score the BASIS-24 from a personal com- factors described above under BASIS-24
found that psychiatric outpatients in a re- puter, download and print survey results, domains. The Adjusted Goodness of
habilitation program improved signifi- and maintain data for future analysis and Fit Index (0.81), root mean square error of
cantly in the areas they themselves had reporting. Clients may try a free demo of approximation (0.08), standardized root
identified as goals for treatment, but WebScore for 30 days by signing up at mean squared residual (0.06), Comparative

18

Fit Index (0.95), and Non-Normed Fix Index related with other measures of mental
(0.95) all indicate adequate to excellent fit health, but not with measures of physical
For more information about
(Eisen et al., 2004b). Internal consistency health. bASIS-24, please visit
reliability (Cronbach’s alpha) coefficients http://ebasis.org/
for the 6 domains ranged from 0.75 to 0.89 Institutional Implementation or contact by mail at
for inpatients and from 0.77 to 0.91 for out- BASIS-24 is currently in use in 5 countries mclean bASIS Plus/ebASIS,
patients (Eisen et al., 2004b). When broken in over 200 hospitals, mental health cen- mS-112,
down by race-White, African-American, ters, community-based outpatient clinics, mclean Hospital,
115 mill Street, belmont,
and Latino-Cronbach’s alpha coefficients schools and managed care organizations.
mA 02478-9106;
exceeded 0.70 for all domains and for all BASIS-24 was previously used for accredi-
race/ethnicity groups for both inpatients tation purposes by The Joint Commission by phone at
and outpatients, with one exception: for and is approved by the Massachusetts (617) 855-2424;
Latino inpatients, the alpha was 0.66 for Behavioral Health Partnership for use in
the emotional lability domain (Eisen et al., clinical outcomes measurement. or by email at
2006). When broken down by gender, the
basisadmin@mclean.harvard.edu.
internal consistencies ranged from 0.73 to
0.89 for males and 0.77 to 0.89 for females
(Idiculla, 2008). Test-retest reliability coef-
ficients ranged from 0.81 to 0.96 for inpa-
tients, and 0.89 to 0.96 for outpatients
(Eisen et al., 2006).
For both inpatients and out-

patients, correlations of the BASIS-24 do-
main and summary scores with the Mental
Component Score of the Short Form (SF)- REFERENCES
12 (Ware et al., 1996) ranged from 0.15 to
0.77, and correlations with global ratings Eisen, S. V., Dickey, B., & Sederer, L. I. (2000). A self-report symptom and problem rating scale to increase inpatients'
involvement in treatment. Psychiatric Services, 349-353.
of mental health ranged from 0.12 to 0.75.
Eisen, S.V., Dill D.L., & Grob M.C. (1994). Reliability and validity of a brief patient-report instrument for psychiatric
Correlations of the BASIS-24 scores with
outcome evaluation. Hospital and Community Psychiatry, 45(3), 242-247.
the Physical Component Score (PCS) of the
Eisen, S.V., Gerena, M., Ranganathan. G., Esch. D., & Idiculla. T. (2006). Reliability and Validity of the BASIS-24 Mental
SF-12 ranged from 0.01 to 0.15 for inpa- Health Survey for Whites, African-Americans, and Latinos. The Journal of Behavioral Health Services & Research, 33,

tients, and from 0.06 to 0.28 for outpa- 304-323.
tients (Eisen et al., 2004b). In a later study Eisen, S. V., & Grob, M. C. (1982). Clients’ rehabilitation goals and outcome. Psychological Reports, 50, 763-767.
comparing racial/ethnic groups, correla- Eisen, S.V., Normand, S.L.T., Belanger, A.J., Gevorkian, S., & Irvin, E.A. (2004a). BASIS-32® and the Revised Behavior
and Symptom Identification Scale BASIS-R. In M. Maruish (Ed.) The Use of Psychological Testing for Treatment
tions of the BASIS-24 summary score with Planning and Outcome Assessment, Third Edition (79-113). Mahwah, NJ: Lawrence Erlbaum.
other self-reported measures of mental Eisen, S.V., Normand, S.L.T., Belanger, A.J., Spiro, A., & Esch, D. (2004b). The Revised Behavior and Symptom
health status (MCS, global mental health, Identification Scale (BASIS-R): Reliability and Validity. Medical Care, 42, 1230–1241.
and satisfaction with life) ranged from 0.59 Eisen, S.V., Youngman, D., Grob, M.C., & Dill, D.L. (1992). Alcohol, drugs and psychiatric disorders: A current view of
hospitalized adolescents. Journal of Adolescent Research, 7(2), 250-265.

to 0.82, for both inpatients and outpatients
in each group. Additionally, correlations Idiculla, T. (2008). Gender invariance of Behavior and Symptoms Identification Scale factor structure, International
Journal of Diversity (in progress).
between the summary score with PCS
McLean Hospital. (2006). BASIS-24 Instruction Guide. Belmont, MA: Author.
were consistently lower, ranging from 0.07
McLean Hospital. (2011). McLean Hospital BASISplus/eBASIS. Retrieved from http://ebasis.org/
to 0.45 (Eisen et al., 2006), indicating that,
Newnham, E. A., & Page, A. C. (2010). Bridging the gap between best evidence and best practice in mental health.
as expected, BASIS-24 is substantially cor- Clinical Psychology Review, 30(1), 127-142.Ware JE, Kosinksi M, Keller S. A 12-item short-form health survey
(SF-12): construction of scales and preliminary tests of reliability and validity. Med Care. 1996;24:220-233.
19

The Integra/ The COMPASS Tracking Assessment System origi-
nated in 1991 as an outcome measure for assessing whether
psychotherapy interventions produced measurable change. The
COMPASS first version of the COMPASS was developed using data and meas-
ures from the Northwestern Study of Long-term Psychotherapy that
Tracking was funded by the U.S. National Institutes of Health and conducted
by Kenneth Howard, Ph.D. of Northwestern University and David

Assessment System Orlinsky, Ph.D. of the University of Chicago among others. In 1992
Dr. Howard teamed with Peter Brill, M.D., founder of the Integra, Inc.,
a managed behavioral health care network that operated primarily
on the U.S. eastern coast, to publish a slightly revised measure
under the title, the Integra Outpatient Tracking System (IOTA). Basic
robert J. lueger, Ph.D., psychometric data were published in a manual (Howard et al.,
Creighton university 1992), and data collection began with the Integra, Inc., service de-
livery system in 1992. By 1996 approximately 16,000 service partic-
ipants had contributed a measurement on at least one occasion.
The IOTA was renamed the COMPASS Tracking Assessment System
when the corporate entity, COMPASS Information Services, was
formed in 1993.
In 1997, the 84-item COMPASS for Primary Care
(COMPASS-PC) was developed in collaboration with Bristol-Myers
Squibb for assessment of patient response to medication and be-

havioral treatments in primary care settings. Within a year, however,
Bristol-Myers Squibb reorganized its behavioral health division, and
This paper presents the Integra/ComPASS Tracking
support for network-wide implementation was shelved. In 1999, the
Assessment System, which can be used for the ongoing remnants of the COMPASS service delivery system were sold to a
monitoring of patient response to mental health treat- new management group using the older name of Integra, Inc., and
ment. The system, which is available in four languages,
includes 68 items, with 4 items devoted to assessing the rights to the outcomes measurement system were included in
subjective well-being; 40 items assessing symptoms the purchase. The outpatient tracking system continued to be used
of anxiety, depression, obsessive-compulsive disorder, in the Integra, Inc., network under the title COMPASS(R) for at least
physical symptoms, adjustment problems, and post-
traumatic stress; and 24 items that assess disabilities another five years, and was converted to web-based delivery in

in daily living, social, work, intimacy, and development 2000. A scan of the professional literature reveals few or no publi-
of self. Two shorter forms of the 68-item measure have
cations on the COMPASS-PC or the COMPASS(R) after 2003, but the
also been developed. The system can be used to define a
probable course of response to treatment which becomes Integra/COMPASS, which contains 68 items, continues to be used
the standard for defining whether the treatment is going widely. It is available in English, Spanish, German, and Italian lan-
as expected, which in turn can enable clinicians to make
adjustments given evidence of the patient’s response or
guages.
non-response to intervention. This paper describes the
Integra/ComPASS Tracking Assessment System and Domains Assessed
presents its psychometric properties, as well as the

procedures related to its use in diverse professional The Integra/COMPASS outpatient tracking system was developed
contexts. from an a priori conceptual theory, the Phase Model, which charac-
Keywords: Integra/ComPASS Tracking Assessment terizes change during psychotherapy as consisting of three distinct
System; ComPASS; Integra; treatment outcome; and sequential phases. The Phase Model of psychotherapeutic
progress monitoring; psychotherapy
change (Howard, Lueger, et al. 1993) postulates that patients first

20

are remoralized to become hopeful, symp- norms on the Integra/COMPASS were ob- vent deterioration, for others the goal
toms then go into remission, and finally tained from approximately 7,000 patients might be to manage chronic problems
problems in functioning are rehabilitated seeking psychotherapy in the first session more effectively (but not to a symptom
or more adaptive skills are learned. The of treatment. Two non-patient samples free state), and for still others, the goal may
Phase Model was introduced to provide a totaling approximately 700 adults were be a return to a normal or non-patient
guide to what changes when in the course used to obtain comparison norms. The state. By using the known courses of treat-
of a psychotherapeutic treatment. Empiri- responses from first-session patients were ment for similar patients, there is a nor-
cal support for the Phase Model generally normalized and converted to T-scores with malized, empirical reference or benchmark
has been positive (Callahan et al., 2006; a mean of 50 and a standard deviation of for an individual patient in treatment.
Hilsenroth et al., 2001; Lueger, 1995; 10. Higher scores indicated greater behav- Empirical studies with large
Lueger, 2010; Stulz & Lutz, 2007). Approxi- ioral/psychological health. data sets of former patients have identified
mately 63% of psychotherapy patients/ Short forms of the 68-item clinical variables for predicting individual
clients show a Phase Model characteristic measure have been developed. A 12-item responses to treatment (Lutz et al., 1999;
pattern of change during psychotherapy short form of the Integra/COMPASS meas- Lueger et al, 2001; Stulz & Lutz, 2007) uses
(Stulz & Lutz, 2007). For patients who re- ure was constructed and psychometrically random regression or hierarchical linear
spond positively during psychotherapy, validated to more economically track pa- modeling, and Nearest Neighbor grouping
about 42% show the pattern of remoral- tient outcomes, although this brief form techniques. Knowing the initial scores on
ization preceding remission of symptoms, has been seldom used in clinical practice. the components of the phase model and
and remission of symptoms preceding re- Like most brief forms, the 12-item short the status on the clinical characteristics,
habilitation of functioning (Lueger, 2010) form seems to measure a single construct. the pattern of change can be represented
as predicted by the Phase Model, and only A 35-item version of the full 68-item Inte- as a slope of change (modeled with either
12-16% of those who manifest reliable gra/COMPASS measure was developed by linear or log-linear assumptions). Patients

change are not consistent with the predic- focusing on items that had demonstrated can be categorized or grouped as most
tions. Cumulative patient improvement sensitivity to change in psychotherapy probably belonging to a set of patients
curves indicate that approximately 50% of (Lueger, 2010). who had similar clinical characteristics
all patients who begin psychotherapy re- prior to treatment. The patients of a group
moralize by session 6, 50% show symp- Assessment and Treatment will have similar patterns or slopes of
toms improvement by session 12, and 50% Planning change. Most of the work to date has used
show improvement in functioning by ses- A distinctive feature of the Integra/ the Mental Health Index (MHI) as the out-
sion 24-27 (Lueger, 2002). COMPASS system is its use of clinical come variable to be modeled in the ex-
The Integra/COMPASS meas- characteristics (distress level, severity, pected course. Confidence boundaries can

ure consists of 68 items, with 4 items chronicity, previous treatment history, ex- be built around the expected course of
devoted to assessing Subjective Well- pectation of improvement) to define a response by identifying the percentage
Being (Remoralization), forty items assess- probable course of response to treatment. of individuals (top and bottom 25th per-
ing symptoms of anxiety, depression, This expected course, or “Expected Treat- centiles) at a particular session of treat-
obsessive-compulsive disorder, physical ment Response,” becomes the standard for ment. A score outside the confidence
symptoms, adjustment problems, and defining whether the treatment is going as boundaries is either a better or a worse
post-traumatic stress. The Life Functioning well as, less than, or better than expected. than expected outcome.

portion of the measure consists of 24 items This approach to outcomes standards Early in its development, the
that assess disabilities in daily living, social, takes into account the wide range of dif- potential of using Integra/COMPASS data
work, intimacy, and self development. The ferences, response styles, and levels of as feedback during the treatment (“smart
Mental Health Index has served as the change that characterize individuals expe- system”) was recognized. Feedback about
focal variable for most of the outcomes riencing psychotherapeutic interventions. patient progress enables clinicians to
work related to the COMPASS. Intake For some patients, the goal may be to pre- make adjustments given evidence of the

21

patient’s response or non-response. Feed- Use and procedures/Technical Institutional implementation
back also can enhance the confidence of a Support The first ten years of using the COMPASS
therapist that the treatment is on track. The Integra/COMPASS measure was origi- Tracking Assessment System included
Given that approximately 14% of com- nally within the private domain and sub- inevitable resistance to incorporating out-
pleted treatments end as failures, the ject to costs per administration. Since the comes measures into a naturalistic treat-
greatest payoff in using feedback seems to demise of the supporting organization, In- ment process. The demands of evidence of
be that of preventing these failure treat- tegra Incorporated, the measure has been outcome as a condition of reimbursement
ment outcomes. Research (Lueger et al., used within private mental health service brought by managed care processes have
2001) has focused on predictors of treat- clinics as part of treatment outcome as- changed the nature of that resistance.
ment failure, and has identified several sessment. The technical support originally Nonetheless, a premium is placed on the
potential indicators using the overall made available through Integra Incorpo- most time efficient method of obtaining
measure of functioning, the MHI. When the rated a decade ago no longer is available, that evidence. Thus shorter forms have
self-reported overall functioning is very and clinics generally have developed their emerged from the longer forms of most
positive and the clinician-rated function- own web-based technical support. The outcomes measures. Also, significant ad-
ing is low, there is a high probability of extent of the use of the original 68-item vances in electronic personal devices in
later treatment failure. About two-thirds measure is unknown, but several behav- the past ten years have provided multiple
to three-quarters of patients who have ioral health clinics in Minnesota—Center platforms (cell phones, smart phones,
two successive tracking measurements in for Life Counseling, Midwest Center for iPads, and computers) for completing
the less-than-expected category later be- Personal and Family Development— these briefer forms. This has eliminated the
come failures. Non-changes on Phase use the measure. The original measure, need for paper management, and has en-
Model components also predict treatment scales for scoring, and original norms, as abled immediate scoring of the completed
failure. Half of the patients who do not re- well as the shorter 35-item scale are avail- forms. These platforms also permit form

moralize by the fourth session, and 80% able from this author. completion away from the site of the serv-
who do not improve in symptoms by the ice delivery. For intermittent sessions,
twelfth session will not improve by the end Psychometric Properties weekly readings can be taken using ab-
of treatment. For the Integra/COMPASS measure, inter- breviated forms even when the patient
Feedback to the therapist nal consistency measures range from .79 does not come to the clinic for treatment.
follows the phase model of change. Lack- for the four-item Subjective Well-Being However, issues of confidentiality are
ing evidence of improvement in remoral- subscale, to .95 for the Symptoms sub- raised in these contexts.
ization, the therapist is encouraged to scale, .93 for the Life Functioning subscale, The science and technology
attend to instilling hope, building the and .87 for the global scale (Mental Health of outpatient tracking and assessment

relationship, normalizing the experience of Index) using normalized scores on each of have greatly advanced over the past
the patient, and increasing the confidence the three domains. For the 35-item short 20 years. The Integra/COMPASS Tracking
of the patient in treatment. Lacking evi- form of the measure, factor analyses reveal Assessment System was an early pioneer
dence of improvement in symptoms, the that this shorter form retains the compo- of these efforts to identify a suitable
therapist is encouraged to refocus on at- nents of the Phase Model, and has high outcome measure and to develop rules of
tainable, specific, short-term treatment intra-measure reliability (alpha = .94). use that related to treatment goals. The
goals related to elements of the symp- clinically adjusted expected treatment

toms, or possibly to add adjunctive treat- response has been a relatively unique fea-
ments (e.g., medication management) or ture of the COMPASS system. The accepted
to increase the frequency of the treatment use of tracking progress feedback has al-
sessions. tered the way that psychotherapeutic in-
terventions are delivered, and has
increased the acceptance of benchmarked
outcome standards.
22

REFERENCES

Barkham, M., Margison, F., Leach, C., Lucock, M., Mellor-Clark, J., Evans, C., Benson, L., Lueger, R.J. (1995). A phase model of psychotherapy outcome. Psychotherapeut, 40,
Connell, J., Audin, K., & McGrath, G. (2001). Service profiling and outcomes 267-278.
benchmarking using the CORE-OM: Toward practice-based evidence in the
psychological therapies. Journal of Consulting and Clinical Psychology, 69, 2, Lueger, R.J. (2002, June). The phase model of psychotherapy outcomes: Memorial to
184-196. Ken Howard. Paper presented at the international meeting of the Society for
Psychotherapy Research, Santa Barbara, CA.
Callahan, J.L., Swift, J.K., & Hynan, M.T. (2006). Test of the phase model of
psychotherapy in a training clinic. Psychological Services, 3, 129-136. Lueger, R.J. (2010, June). Testing predictions of the Phase Model: a comparison of
therapist and patient report. Paper presented at the international meeting of the
Grissom, G.R., Lyon, J.S., & Lutz, W. (2002). Standing on the shoulders of a giant: Society for Psychotherapy Research, Asilomar, CA.
Development of an outcome management system based on the dose model and
phase model of psychotherapy. Psychotherapy Research, 12 (4), 397-412. Lueger, R.J., Howard, K.I., Martinovich, Z., Lutz, W., Anderson, E.A., & Grissom, G.
(2001). Assessing treatment progress of individual patients with expected
Hilsenroth, M.J., Ackerman, S.J., & Blagys, M.D. (2001). Evaluating the phase model of treatment response models. Journal of Consulting and Clinical Psychology, 69,
change during short-term psychodynamic psychotherapy. Psychotherapy Research, 150-158.
11 (1), 29-47.
Lutz, W., Martinovich, Z., & Howard, K.I. (1999). Patient profiling: An application of
Howard, K.I., Brill, P., Lueger, R.J., & O'Mahoney, M. (1992, 1993, 1995). The Integra random coefficient regression models to depicting the response of a patient to
Outpatient Tracking Assessment. Radnor, PA: Integra, Inc. outpatient psychotherapy. Journal of Consulting and Clinical Psychology, 67,
Howard, K.I., Lueger, R.J., Maling, M., & Martinovich, A. (1992, June). A phase model of 571-577.
psychotherapy outcomes. Paper presented at the annual meeting of the Society Stulz, N., & Lutz, W. (2007). Multidimensional patterns of change in outpatient
for Psychotherapy Research, Berkeley, CA. psychotherapy: The phase model revisited. Journal of Clinical Psychology, 63 (9),
Kopta, S., & Lowrey, J. (2002). Psychometric evaluation of the Behavioral Health 817-833.
Questionnaire-20: A brief instrument for assessing global mental health and the
three phases of psychotherapy. Psychotherapy Research, 12, 413-426.


23

The Outcome The Outcome Questionnaire-45 (OQ-45) is a 45 item
self-report scale used to estimate client degree of disturbance at
the outset and over the course of treatment. It provides an index of
Questionnaire-45 mental health functioning for adults 18 years of age or older. It was
originally developed for use in managed care as a means of meas-
uring the outcomes of treatment and enhancing them. A person
who takes the measure is compared to inpatient, community men-
tal health, outpatient, employee assistance program, college coun-
michael J. lambert, Ph.D.,
brigham Young university seling center, and normal populations. Scores on the measure are
referenced against expected treatment responses based on the
progress of 12,000 treated individuals across the United States.
michael_lambert@byu.edu These data provide a bench mark of success on a session-by-
session basis in order to identify treatment non-responders, and
clients at-risk for negative outcomes. It also provides cut-scores for
reliable change and recovery as markers for gauging treatment
success and possible termination of services. It has been translated
into more than 30 languages other than English, including French.
It requires reading ability at the 6th grade level. Copyright
to the measure is held by OQ Measures, LLC; 2171 Lake Street,
Salt Lake City, UT 84106; www.oqmeasures.com; email:
office@oqmeasures.com.

Domains Assessed
Because almost all adults who enter treatment experience symp-
toms of anxiety and depression, half of the items of the OQ-45
measure core aspects of these disorders, or what may be called
symptomatic distress or subjective discomfort. Because a satisfac-
The outcome Questionnaire-45 (oQ-45) is a 45 item
self-report scale which can be used to estimate client
tory quality of life and the wellbeing depends on positive interper-
disturbance at the outset and over the course of treat- sonal functioning, a quarter of the items assess disturbance in
ment. The scale provides an index of mental health interpersonal relationships with intimate others. The final quarter
functioning for adults 18 years of age or older, which
can be compared to inpatient, community mental of the items assess functioning in social roles such as work, school,

health, outpatient, employee assistance program, homemaking and leisure activities. Effective functioning in social
college counseling center, and normal population data.
roles has important consequences for society as well as individuals.
Scores on the measure can also be referenced against
expected treatment responses thus serving as a Nine items are worded in a positive manner in an attempt to tap
benchmark of treatment success on a session-by-session into wellbeing in addition to psychopathology. These items are a
basis. Furthermore, the oQ, which is available in over
30 languages, provides cut-scores for reliable change
part of the other three domains, as are substance abuse and suicide
and recovery as markers for gauging treatment success screening items. The clinician report highlights substance abuse,
and possible termination of services. In this paper, suicidal ideation, and anger when scores reach a critical level. The
the author presents the scale, including how it is

administered, the domains it assesses and its psycho- OQ-45 can be used regardless of the type of psychotherapy, mode
metric properties, and describes how it can be used of psychotherapy, or medication intervention. It is atheoretical in
to improve treatment delivery in clinical practice. nature and serves as a mental health vital sign or lab test to be used
Keywords: outcome Questionnaire; oQ-45; by clinicians to manage illness by quantifying the patient’s current
treatment outcome; progress monitoring; psychotherapy
mental health functioning. There is also a brief version of the OQ-45,
the OQ-30 which does not include subscales.

24

Use and Procedures scoring, and creating clinical reports make predict treatment failure and provide
Ideally, the OQ-45 is administered online, the work load for clinicians and support alerts to clinicians if the patient is pre-
via handheld devices, or personal com- staff minimal. If a therapist, instead of a dicted to have a negative treatment out-
puter (it can be administered and scored third party such as a receptionist is in come. Because clinicians are confident in
via hard copy as well). It takes about 5-10 charge of handing out a handheld device, their ability to recognize and predict treat-
minutes of patient time to rate all of the this takes an additional 30 to 60 seconds ment failure, but fail to do so, a psycho-
45 items, typically prior to the treatment of time. logical test that can perform this task is an
session. Each item is answered on a five The OQ-Analyst software invaluable addition to routine decision
point scale according to the patient’s system is available for $US200 per clinician making and care. Between 85-100% of
recollection of the preceding week—on a per year. This is based on an average of 200 treatment failures can be identified by the
scale from “almost always” to “never”. This clients per year per clinician, at $1 per OQ-45 before they leave treatment and
allows the clinician to get a quick overview client per year. The cost includes unlimited often within 3-5 sessions after entering
of functioning that would take an exces- administrations for each client along with treatment. When clinicians are alerted to
sive amount of time if based on a clinical scoring, alerts, and progress profiling. A potential treatment failure they tend to
interview. Software (OQ-Analyst) scores fully hosted Web-based system is available retain patients in treatment longer. Dete-
the measure, graphs the results in relation using a prorated cost model for organiza- rioration rates are reduced by 1/3 to 2/3,
to earlier administrations, and in relation tions that serve more than 1,000 patients while improvement rates double for these
to normative functioning and expected per year and employ more than 50 clini- difficult patients. Tracking treatment re-
treatment response based on other indi- cians. The initial start-up costs for the sponse and feedback to clinicians reduces
viduals who have the same initial level of hosted system average about $3 per pa- the number of sessions used by clients
disturbance. All this is accomplished and tient per year, which includes software and who never go off-track, and improves their
available on the therapist’s computer in hardware, and yearly costs thereafter are final outcome.

about one second after completion of the under $1 per patient per year. Both cost The OQ-Analyst can be set
45th item. models deliver an unlimited number of up to provide clinical information to inter-
It takes therapists about administrations per patient providing an disciplinary teams who are also working
18 seconds to access the client’s report on incentive to repeatedly track patients at no with each specific patient. The clinical ad-
his or her computer or a clinician report additional cost (i.e., there are no per- ministrator enters all providers within a
and client report can be printed and deliv- administration charges). system of care, a primary, and secondary
ered to individuals as a hard copy. Cumu- providers are then selected from the list.
lative data from multiple administration of Assessment and Treatment Thus all members of a team who are given
the measure can be housed on a self-sup- Planning access by administrators through a pass-

ported server (or personal computer) or Although the OQ-45 is designed to track word, including clinical managers and su-
through OQMeasures. The OQ-45 is part of changes in mental health functioning over pervisors, can be given access to progress
a larger Outcome Measurement System– time, scores on the subscales can be used reports and alert notifications for specific
the OQ-Analyst, which includes measures to determine which areas of functioning patients who are being treated by team
of child functioning, the Brief Psychiatric are most problematic for the patient and members. This shared information allows
Rating Scale, and the Assessment for these can be imbedded in a treatment for all the team to be aware of no, or neg-
Signal Clients, a clinical support tool used plan and tracked over time. Patient ative treatment response and settle on

to guide problem solving with failing progress graphs can be cut and pasted coordinated efforts to turn the course
cases. Users typically prefer to keep the into clinical records. In addition, OQ-re- of treatment in a positive direction. The
data collected in house, but data stored ports estimate the number of sessions OQ-Analyst also produces reports,
outside a clinic is encrypted and HIPPA needed for a patient to return to a state of summed across all patients or patient sub-
compliant methods are used to protect normal functioning or at least achieve groups (such as substance-abuse patients,
confidentiality of the data. As noted above, reliable change. A major advantage of the males vs. females, program A versus
automated methods of administrating, OQ-45 is underlying valid algorithms that program B) for the purpose of comparing
25

benchmarks across similar services or for users. This is very helpful to insure that the outpatient substance abuse, to inpatient
particular clinicians. system becomes adapted to the emerging eating disorder treatment. The OQ-45 has
needs of agencies or if they want OQ- been judged by the National Registry of
Technical Support Measures to host their system. Evidence-based Programs & Practices in
Technical support is available for $150 the United States (NREPP) as an evidence-
per year. Technical manuals for the instru- Psychometric Properties based practice based on the weight of ev-
ments and user guides on implementation Since its development in the early 1990’s idence derived from experimental studies
of the OQ-Analyst are included at no addi- dozens of studies have been published on in routine care settings.
tional charge and are regularly updated as the psychometric properties of the OQ-45. Qualitative research has
new research supports modifications. It has high internal consistency (.90), test been completed indicating that clients do
NREPP rated the degree to which the re-test reliability (.84 over 3-weeks), and not mind taking the OQ-45 on a weekly
OQ-Analyst is ready for dissemination at concurrent validity with scales such as the basis. This is particularly true if staff have a
3.9 on a 4-point scale. Users choose to Symptom Checklist-90 and BDI, with coef- positive opinion about the measure, the
host the OQ-Analyst themselves or have ficients hovering in the mid .80s. Factor patient is provided with a rationale for tak-
OQMeasures host it. In order to use the analytic studies support the presence of an ing the measure (such as it is like monitor-
OQ-Analyst IT, support is sometimes overall distress factor with three subordi- ing blood pressure in order to manage it,
required. And on occasion clinical ques- nate factors consistent with the subscales. or, it is a way of finding out how you are
tions arise. This support comes through Most items, the subscales, and the Total doing), and clinicians provide some feed-
www. oqmeasures.com or office@ Score are sensitive to the effects of inter- back, indicating they are aware that the
oqmeasures.com or toll free at ventions while remaining stable in un- patients has completed it. Patients are very
1-888-647-2673 or specifically from indi- treated individuals. This is the most pleased with the clinical reports that they
viduals who are in contact with users: important psychometric characteristic of receive (if clinicians choose to share them).

Individuals include: Tameisha Hastings, the OQ-45 since it is used to monitor Providers are less positive initially than
Marketing and Sales, Tameisha. change in patients in treatment. The items clients, especially if use of the measure is
Hastings@OQMeasures.com; Sue A. in the OQ-45 have been examined over forced on them by administrators. After a
Jenkins, Executive Officer, Sue.Jenkins@ time in both patients and individuals who time even the most resistant clinicians
OQMeasures.com, Amy, IT Manager, are disturbed, but not in psychological come to enjoy the feedback graphs of pa-
Amy@OQMeasures.com. Clinical inquires treatments or using psychoactive medica- tient progress and find ways to use the in-
and research questions are directed tions. formation provided in treatment planning
to: Michael J. Lambert, Ph.D., Mike. and risk assessment. From the clinicians
Lambert@OQMeasures.com; Gary M. Patient/Client/Clinician Feedback point of view there is no efficient way to

Burlingame, Ph.D., Gary.Burlingame@ The strength of the OQ-45 is the extensive gather information about patient func-
OQMeasures.com published evidence on the degree to tioning across the range of factors that
Training can be provided which providing feedback to clinicians and make up their mental health. Thus they
upon request and ranges from trainings patients based on the OQ-45 alert system come to see it as a quick check on areas of
lasting from two hours to all day work- maximizes patient outcome and reduces functioning that may not be the focus of a
shops. Training can be accomplished on treatment failure. Seven RCT’s have been particular treatment session.
site or via phone conference with internet published and two more have been com-

connection. Developers of the OQ-Analyst pleted showing that the feedback and
& Lanark Systems work closely with problem-solving tools delivered to thera-
OQMeasures and have made modifica- pists work in a variety of routine care set-
tions of the software at the request of large tings from university counseling centers,

26

Institutional Implementation
There are hundreds of clinics, institutions,
REFERENCES
and individuals across North America and
Hannan, C., Lambert, M. J., Harmon, C., Nielsen, S. L., Smart, D. M., Shimokawa, K., & Sutton,S. W. (2005). A lab test
the world who are licensed users of the
and algorithms for identifying patients at risk for treatment failure. Journal of Clinical Psychology: In Session,61(2),
OQ-45 and related instruments for child 155-163.
assessment contained within the OQ-Ana- Lambert, M. J. (2010). Prevention of treatment failure: The use of measuring, monitoring, & feedback in clinical practice.
lyst. It is widely used and clearly the dom- Washington, DC: APA Press.
inant outcome instrument in training Lambert, M. J., Hansen, N. B., & Harmon, S. C. (2010). The OQ-45 system: Development and practical applications in
health care settings. In M. Barkham, G. Hardy, & J. Mellor-Clark (Eds.) Developing and delivering practice-based
clinics serving clinical psychology gradu- evidence: A guide for the psychological therapies (pp. 141-154). New York: Wiley-Blackwell.
ate trainees in the USA and Australia. Use Lambert, M. J., Kahler, M., Harmon, C., Burlingame, G. M., & Shimokawa, K. (2011).Administration & scoring manual for
in Training clinics allows for evaluation of the Outcome Questionnaire-45.2. Salt Lake City: OQMeasures.
patient progress as well as trainee per- Okiishi, J. Lambert, M. J., Eggett, D., Nielsen, S. L. Dayton, D. D.& Vermeersch, D. A. (2006). An analysis of therapist
formance over time. treatment effects: Toward providing feedback to individual therapists on their clients’ psychotherapy outcome;
Journal of Clinical Psychology, 62, 1157-1172.
Shimokawa, K., Lambert, M.J., & Smart, D. (2010). Enhancing treatment outcome of patients at risk of treatment
failure: Meta-analytic & mega-analytic review of a psychotherapy quality assurance program. Journal of
Consulting & Clinical Psychology, 78, 298-311.
Vermeersch, D A., Whipple, J. L., Lambert, M. J., Hawkins, E. J., Burchfield, C. M., & Okiishi, J. C. (2004). Outcome
Questionnaire: Item sensitivity to changes in counseling center clients. Journal of Counseling Psychology, 51,
38-49.


27

The Outcome The Outcome and Session Rating Scales (ORS and
SRS are brief measures for tracking client functioning and the
quality of the therapeutic alliance. Each instrument takes less than
Rating Scale (ORS) a minute for consumers to complete and for clinicians to score and
interpret. Both scales were developed in clinical settings where
and the Session longer, research-oriented measures had been in use and deemed
impractical for routine use. Versions of the ORS and SRS are avail-

Rating Scale (SRS) able for adults, children, adolescents and groups in 18 different lan-
guages, including French. Individual clinicians may download the
scales free-of-charge after registering online at: http://www.scottd-
miller.com/?q=node/6. A significant and growing body of research
shows the scales to be valid, reliable, and feasible for assessing
Scott D. miller, Ph.D., progress and the alliance across a wide range of consumers and
International Center for Clinical excellence, presenting concerns.
Chicago

Domains Assessed
scottdmiller@talkingcure.com The ORS is designed to assess the individual, interpersonal, and
social functioning of the consumer. On the other hand, the SRS as-
sesses three elements of the alliance, including: (1) the quality of
Susanne bargmann, International Center
for Clinical excellence, Chicago the relational bond; (2) the degree of agreement between con-
sumer and clinician regarding goals; and (3) consumer and clinician
agreement regarding the methods and approach employed in care.

The tools neither assume nor require that practitioners adhere to a
particular model or approach. Instead, clinicians from any back-
The outcome and Session rating Scales are brief measures ground or discipline may solicit feedback from consumers regard-
which can be used to track client functioning and the ing the working relationship and outcome of care and use the
quality of the therapeutic alliance over the course of
resulting information to inform and tailor service delivery. Routinely
psychotherapy. Versions of the scales are available for
adults, children, adolescents and groups in 18 different monitoring of progress and the quality of the relationship is not
languages. The outcome rating Scale (orS) is designed only consistent with but also operationalizes the American Psycho-
to assess the individual, interpersonal, and social
functioning of the client, whereas the Session rating logical Association’s definition of evidence-based practice, which
Scale (SrS) assesses the quality of the relational bond includes, “the integration of the best available research… and mon-

between the client and therapist, the degree of agreement itoring of patient progress (and of changes in the patient’s circum-
between the client and clinician regarding treatment
goals, and their agreement regarding the methods and stances–e.g., job loss, major illness) that may suggest the need to
approach employed in care. The tools do not require that adjust the treatment… e.g., problems in the therapeutic relation-
practitioners adhere to a particular model or approach
ship or in the implementation of the goals of the treatment)” (APA,
and administering and scoring the measures is simple
and straightforward. In this paper, the authors present 2006, p. 273, 276-277).
the two measures, discuss the domains they assess,
and describe how they can be used in routine clinical
practice to aid in service plan development. In addition,
Use and Procedures

the authors discuss the psychometric properties of the Administering and scoring the measures is simple and straightfor-
scales and describe the resources available to clinicians ward. The ORS is given at the beginning of the session. The scale
who wish to use them.
asks consumers of therapeutic services to think back over the prior
Keywords: outcome rating Scale; orS; Session rating
week (or since the last visit) and place a hash mark (or “x”) on four
Scale; SrS; treatment outcome; progress monitoring;
psychotherapy different lines, each representing a different area of functioning
(e.g., individual, interpersonal, social, and overall wellbeing). The

28

SRS, by contrast, is completed at the end Assessment and Treatment a very specific problem—one that does
of each visit. Here again, the consumer Planning not impact the overall quality of life or
places a hash mark on four different lines, Soliciting clinically meaningful feedback functioning, but is troubling nonetheless.
each corresponding to a different and im- requires more than administering two Less frequent causes for a high initial ORS
portant quality of the therapeutic alliance scales, the ORS and SRS or otherwise. Cli- include: (1) high functioning people who
(e.g., relationship, goals and tasks, ap- nicians must work at creating an atmos- want therapy for growth, self-actualiza-
proach and method, and overall). On both phere where consumers feel free to rate tion, and optimizing performance; and
measures, the lines are ten centimeters in their experience of the process and out- (2) people who may have difficulties read-
length. Scoring is a simple matter of de- come of services: (1) without fear of retri- ing and writing or who have not under-
termining the distance in centimeters (to bution; and (2) with a hope of having stood the meaning or purpose of the tool.
the nearest millimeter) between the left an impact on the nature and quality of With regard to the latter, it should be
pole and the client’s hash mark on each in- services delivered. Beyond displaying an noted that a validated oral version of the
dividual item and then adding the four attitude of openness and receptivity, ORS is available and can be administered.
numbers together to obtain the total. creating a “culture of feedback” involves Research and experience document that
In addition to hand scoring, taking time to introduce the measures in consumers scoring above 25 at intake are
several computer-based applications are a thoughtful and thorough manner. Pro- at a heightened risk for deterioration.
also available which can simplify and ex- viding a rationale for using the tools is Therefore, care should be taken to clarify
pedite the process of administering, critical, as is including a description of how the wishes of the person in treatment. In
scoring, and aggregating data from the the feedback will be used to guide service order to maintain engagement, the best
ORS and SRS. As just one example, con- delivery (e.g., enabling the therapist to approach is a cautious one. In particular,
sider the web-based application, www. catch and repair alliance breaches, prevent using the least invasive and intensive
fit-outcomes.com. Briefly, the system or- dropout, correct deviations from optimal methods needed to resolve the problem at

ganizes treatment outcome and thera- treatment experiences, etc). With regard hand.
peutic alliance data, and compares the to interpreting the ORS, low scores corre- With regard to interpreting
scores to the expected treatment response spond to a poor sense of well-being (or the SRS, research to date shows that the
(ETR) of the client. Importantly, the client high level of distress). Note that the aver- majority of clients score relatively high.
and therapist receive feedback in real time, age ORS intake score in outpatient mental Thus, the cutoff on the measure is 36. It is
indicating whether treatment is on or off health settings is between 18 and 19. Over important to keep in mind that a high
track. Additionally, the system aggregates time, whatever the initial score, the num- score (36+) does not necessarily confirm
outcome and alliance data across episodes ber should increase in response to services the presence of a strong alliance. The best
of care, thereby providing clinicians and offered. A lack of movement, deterioration, response to a high score is thanking the

agencies with an overall measure of effec- or seemingly random pattern of scores is consumer and remaining open to the pos-
tiveness as well as the ability to compare cause for concern and should be discussed sibility of feedback in the future. Scores
the outcomes of individual clinicians with the client at the time of service that fall at or below 36 are considered
and programs. With regard to privacy delivery. Between 25-33% of people com- “cause for concern” and should be dis-
and security, all data entered into fit- pleting the measure will fall above a total cussed prior to ending the visit. Single-
outcomes.com is first anonymized and score of 25 at intake—a number known as point decreases in SRS scores from session
then encrypted according to current the cutoff, or the dividing line between a to session have also been found to be

international standards. clinical and non-clinical population (Miller associated with poorer outcomes at termi-
& Duncan, 2000, 2004). The most common nation—even when the total score consis-
reason for such a score is that the con- tently falls above 36—and should
sumer has been mandated into treatment. therefore be addressed in the session
Another is that the person desires help for

29

(Miller, Hubble & Duncan, 2007). Interest- Manuals Development Team, 2011a,b, Reese, Norsworthy & Rowlands, 2009).
ingly, there is growing evidence that the c, d,e,f [http://www.scottdmiller.com/?q= The SRS has been shown to assess the
process of responding to a client’s nega- node/5]). The manuals are written in clear, qualities of the alliance as first defined by
tive feedback, even about an aspect of practical, step-by-step, and easy-to-under- Bordin (1976). Numerous studies have
therapy that may seem relatively trivial, stand language and cover: documented the concurrent validity, test-
can contribute to the strength of the (1) the empirical foundation; retest reliability, and internal consistency
therapeutic alliance and set in place a (2) basics of administration, scoring, of the SRS (e.g., Duncan et al. 2003, Miller,
strong foundation for future work. There is and interpretation; Duncan, Brown et al. 2003). Several ran-
also evidence that the most effective ther- (3) use of the measures in supervi- domized clinical trials have documented
apists elicit more negative feedback from sion; the significant impact that both measures
their clients. Whatever the circumstance, (4) aggregation and interpretation have on the outcome of and retention in
openness and transparency are central to of data generated by the ORS treatment (e.g., Anker et al., 2009; Miller et
successfully eliciting meaningful feedback and SRS; al., 2006; Reese et al., 2009).
on the SRS. (5) application of the ORS and SRS
with special populations; and Institutional Implementation
Technical Support (6) implementing the measures in Worldwide, there are currently 30,000+
An international, online community is agencies and systems of care. registered individual practitioners, and
available to support the use of the scales As mentioned previously, 100’s of licensed agencies and treatment
for informing, evaluating, and improving several computer and web-based applica- settings using the scales. Since 2009, the
the quality of behavioral healthcare. Mem- tions are available for administering, membership of the International Center of
bership in the International Center for scoring, interpreting, and aggregating Clinical Excellence (ICCE) has grown expo-
Clinical Excellence (ICCE) is free-of-charge, data from the ORS and SRS. The most nentially. The ICCE community is where

open to clinicians from all disciplines and current information about such applica- most users receive training and support
approaches, and no selling or promotion tions can be found online at: http:// in the use of the measures. Each year, the
of products or particular treatment ap- www.scottd miller.com/?q=node/6. ICCE conducts two intensive training
proaches is allowed. The site features events: (1) the “Advanced Intensive”; and
hundreds of discussion groups, articles, Psychometric Properties (2) the “Training of Trainers” course. Atten-
and how-to videos in many different lan- The ORS has been shown to be sensitive dance at both trainings, submission of a
guages. Members also have access to the to change among those receiving behav- sample training video, and passing the
“Get Answers” feature to obtain specific ioral health services. Numerous studies “core competency” exam enable partici-
help quickly from community members. have documented concurrent, discrim- pants to become ICCE Certified Trainers.

Certified trainers and associates are avail- inative, criterion-related, and predictive Currently, the ICCE has “Certified Trainers”
able for consultation and training. validity, test-retest reliability, and internal- available for consultation in the USA,
To register, go to: www.centerforclinical consistency reliability for the ORS (e.g., Canada, Australia, New Zealand, Western
excellence.com. Anker, Duncan & Sparks, 2009; Bringhurst, and Eastern Europe.
A series of six manuals are available Watson, Miller & Duncan, 2006; Campbell
that cover the most important information & Hemsley, 2009; Duncan, Miller, Reynolds,
for practitioners and agencies implement- Brown & Johnson, 2003; Duncan, Sparks,

ing the ORS and SRS are available (Inter- Miller, Bohanske & Claud, 2006; Miller,
national Center for Clinical Excellence FIT Duncan, Brown, Sparks & Claud, 2003;

30

REFERENCES

Anker, M., Duncan, B., Sparks, J. (2009). Using client feedback to improve couple International Center for Clinical Excellence FIT Manuals Development Team. (2011c).
therapy outcomes: an RCT in a naturalistic setting. Journal of Consulting and Manual 3: Feedback Informed Supervision. Chicago, IL: ICCE Press.
Clinical Psychology, 77, 693-704. International Center for Clinical Excellence FIT Manuals Development Team. (2011d).
APA Presidential Task Force on Evidence-Based Practice. (2006). Evidence-based Manual 4: Documenting Change: A Primer on Measurement, Analysis, and Report-
practice in psychology. American Psychologist, 61(4), 271–285. ing. Chicago, IL: ICCE Press.
Bordin, E.S. (1976). The generalizability of the psychoanalytic concept of the working International Center for Clinical Excellence FIT Manuals Development Team. (2011e).
alliance. Psychotherapy, 16, 252-260. Manual 5: Feedback Informed Clinical work; Advanced Applications. Chicago, IL:
Bringhurst, D. L., Watson, C. S., Miller, S. D., & Duncan, B. L. (2006). The reliability and ICCE Press.
validity of the outcome rating scale: A replication study of a brief clinical measure. International Center for Clinical Excellence FIT Manuals Development Team. (2011f).
Journal of Brief Therapy, 5(1), 23-29. Manual 6: Implementing Feedback-Informed Work in Agencies and Systems of
Campbell, A., & Hemsley, S. (2009). Outcome rating scale and session rating scale Care. Chicago, IL: ICCE Press.
in psychologicalpractice: Clinical utility of ultra-brief measures. Clinical Psychologist, Miller, S. D., & Duncan, B. L. (2000, 2004). The Outcome and Session Rating Scales:
13, 1-9. Administration and Scoring Manual. Chicago, IL: ISTC.
Duncan, B. L., Miller, S. D., Sparks, J. A., Reynolds, L. R., Brown, J., Johnson, L. D. (2003). Miller, S. D., Duncan, B. L., Brown, J., Sparks, J. A., & Claud, D. A. (2003). The outcome
The session rating scale: Preliminary psychometric properties of a “working rating scale: A preliminary study of the reliability, validity, and feasibility of a brief
alliance” inventory. Journal of Brief Therapy, 3(1), 3-11. visual analog measure. Journal of Brief Therapy, 2(2), 91-100.
Duncan, B. L., Sparks, J. S., Miller, S. D., Bohanske, R., Claud, D. (2006). Giving youth a Miller, S.D., Duncan, B.L., Sorrell, R., Brown, G.S., & Chalk, M.B. (2006). Using outcome
voice: A preliminary study of the reliability and validity of a brief outcome measure to inform therapy practice. Journal of Brief Therapy, 5(1), 5-22.
for children, adolescents, and caretakers. Journal of Brief Therapy, 5, 71-87. Miller, S.D., Hubble, M.A., & Duncan, B.L. (2007). Supershrinks: Learning from the
International Center for Clinical Excellence FIT Manuals Development Team. (2011a). Field’s Most Effective Practitioners. Psychotherapy Networker, 31(6), 26-35, 56.
Manual 1: What works in therapy: A primer. Chicago, IL: ICCE Press. Reese, R.J., Norsworthy, L.A., & Rowlands, S.R. (2009). Does a continuous feedback
International Center for Clinical Excellence FIT Manuals Development Team. (2011b). system improve psychotherapy outcome. Psychotherapy: Theory, Research,
Manual 2: Feedback Informed Clinical Work: The Basics. Chicago, IL: ICCE Press. Practice, Training, 46, 418-431.


31

The Polaris-MH is a Web-based system designed to plan,
manage, and improve adult outpatient mental health treatment. It
is the product of more than a decade of research and development
Polaris-MH by internationally respected researchers working in collaboration
with clinicians, patients, IT professionals, utilization review profes-
sionals, provider system administrators, and managed care execu-
tives (3,12). Polaris-MH was developed with funding from the
National Institutes of Health. It combines a strong scientific foun-
Grant Grissom, Ph.D.,
Polaris Health Directions dation with state-of-the-art technology to help providers and
health care organizations improve the quality and cost-effective-
ness of their services, and document treatment outcomes for
grant@polarishealth.com payers and accreditation organizations. It is designed for multiple
administrations during treatment and provides a uniquely com-
prehensive assessment of clinical problems, patient strengths, and
Tina Harralson, Ph.D., Polaris Health Directions progress. It addresses the needs of various stakeholders: patients,
Jesse nankin, mA, Polaris Health Directions clinicians, clinical managers/administrators, case managers/utiliza-
tion reviewers and payers. For clinicians and case managers, it pro-
vides an evidence-based answer to the question, “Is this treatment
working for this patient?” For administrators, quality improvement
professionals and payers, the system provides multivariate severity
adjustment of program outcomes for identification of best prac-
tices, and “apples-to-apples” evaluation of program effectiveness,

controlling for initial severity and other patient characteristics.
The system can be used with adults 18 years of age or
older in outpatient mental health treatment, including those with
Polaris-mH is a Web-based system designed to plan, co-occurring substance use disorders. Aside from English, it is avail-
manage, and improve adult outpatient mental health able in a number of languages, including French. Literacy at the
treatments. The system, which is available in a number
of languages, can be used with adults 18 years of age 6th grade reading level is required.
or older in outpatient mental health settings, including
those with co-occurring substance use disorders. The Domains Assessed
system measures domains, which are pan-theoretical,
that correspond to the three phases of the therapeutic Polaris-MH is grounded in basic research on psychotherapy process

progress established through Phase Theory: Subjective and outcome: Phase, Dose-Response and Expected Treatment
Well-being (remoralization), Symptoms (remediation)
Response (ETR) models. The Dose-Response Model (4) of psy-
and Functional Disability (rehabilitation). It includes
five major components: (1) a web-based patient chotherapeutic impact describes a positive relationship between
self-report assessment (initial and update assessments); therapeutic dose and rate of clinical improvement, and a pattern of
(2) real-time reports; (3) a data management module
that allows clinical staff to securely log in and access data
relatively rapid early improvement with more and more sessions
and reports; (4) an aggregate reporting module that needed to achieve incremental improvement later in treatment
provides online access to customizable aggregate reports; (a pattern of diminishing returns). The Phase Model (5) extended
and (5) documentation and training materials. In this

paper, the authors describe the measure, the procedures and interpreted the dose-response model by proposing three
related to its use, its psychometric properties and the progressive sequential phases of the psychotherapeutic recovery
clinical contexts in which it can be used. process: (a) remoralization–the enhancement of well-being;
Keywords: Polaris-mH; Polaris; treatment outcome; (b) remediation–the achievement of symptomatic relief; and
progress monitoring; psychotherapy
(c) rehabilitation–the reduction of troublesome, maladaptive
behaviors that interfere with life functioning. The phase model
suggests that the decelerating curve of improvement for a patient
32

can be attributed to the increasing diffi- scales extend clinical and prognostic computers in a provider’s facility for
culty of treatment goals as they change usefulness. In keeping with the basic administering assessments, and a printer
(e.g., from symptom remediation to design principle, the scales have pan- in a secure location for printing assess-
improved functioning) over the course theoretical utility and assess alcohol and ment reports. A broadband Internet con-
of treatment. The Expected Treatment drug severity and resilience. Screens in- nection is required, but no additional
Response (ETR) Model (6,7,8) uses pretreat- clude medical health problems, use of psy- equipment (servers, etc.) is needed. The
ment clinical characteristics (e.g., severity, choactive medications, psychosis and patient-accessible computers may be con-
chronicity, previous treatment, treatment bipolar disorder. Strengths (resilience, figured as “kiosks” on which only the
expectation) to predict the patient's ex- meaning or purpose in life) and items re- Polaris-MH application is accessible, or
pected response during the course of lating to treatment motivation, treatment they may be multi-purpose machines with
treatment. Using individualized growth satisfaction and the therapeutic bond are Polaris-MH as one available option.
curve analysis for a large sample of outpa- also assessed. Customization for the Cana- Development of Polaris MH
tients in psychotherapy, a single patient's dian Forces included addition of the involved extensive interviews with clini-
course of treatment can be predicted as PCL-C (PTSD Checklist, Civilian Version) cians and support staff. As a result, it
soon as his or her intake information is developed by the U.S. Department of includes numerous features that facilitate
available. Ongoing therapeutic effective- Veterans Affairs (10). its integration into routine clinic proce-
ness can be assessed for a single patient dures. The patient completes an initial
by tracking the patient’s actual progress Use and Procedures assessment, usually prior to their first
in comparison to his or her expected Polaris-MH consists of five major compo- appointment. Remote Access enables pa-
progress. nents: (1) Web-based patient self-report tients to complete assessments from any
The Polaris-MH measure- assessments (initial and update assess- computer with Internet access, including
ment domains, which are pan-theoretical, ments); (2) Real-time reports; (3) Data their home computer. Polaris-MH does not

correspond to the three phases of the ther- management module that allows clinical require prior computer experience; pa-
apeutic progress established through staff to securely log in and access data and tients use only the number keys and
Phase Theory: Subjective Well-Being (re- reports; (4) Aggregate reporting module “Enter,” and do not need the mouse. Pa-
moralization), Symptoms (remediation) that provides online access to customiz- tients can “pause” the assessment at any
and Functional Disability (rehabilitation). able aggregate reports; and (5) Documen- point. If they log on within 48 hours the as-
These domains are readily accepted by tation and training materials. Polaris-MH is sessment will resume where they left off.
clinicians as being central to clinical deci- typically delivered as a service through the This enables a patient to complete the
sions and outcomes assessment, irrespec- Web from Polaris’s secure data facilities, assessment after their session with the
tive of the therapeutic model used. These though some customers choose to install clinician. Mean completion time is 14 min-

domains provide the framework for con- the Web-delivery system on their own utes (see next page for a briefer version,
structive dialogue between clinicians and networks and manage the entire system the Emotional Vital Signs). The unassisted
utilization review staff. Subjective Well- themselves. completion rate in outpatient mental
Being is a single scale. The Symptoms The system is designed for health settings is more than 95%. Patients
scale is a composite of seven subscale use throughout treatment. It assesses fac- are typically asked to complete assess-
scores; each subscale corresponds to a tors known to predict engagement in ments prior to meeting with their clinician.
disorder commonly treated in outpatient treatment and clinical outcomes. The data This ensures that the time required to

settings (Depression, Anxiety, PTSD, Panic, synthesized on the system’s intake, update complete the assessment does not reduce
OCD, Phobia, Somatization). Functional and severity-adjusted, aggregate-level the clinician’s time with the patient, and
Disability is a composite of three subscales reports can guide decision-making from it makes the results of the assessment
(Social, Vocational, Personal). A measure of the individual to the organizational level. available at the start of the session.
overall clinical status, Behavioral Health A typical implementation, which can be As soon as the assessment is
Status (BHS), is a composite of the three adapted to specific provider requirements, completed, all scores are automatically
core measures. Additional screens and consists of one or more patient-accessible calculated and the clinical report is gener-
33

ated and available for review with the upon the patient’s assets and identify score that falls below the Failure Boundary
patient. The system may be configured so specific areas of concern when making de- suggests, with 75% certainty, that the
the clinical report prints automatically, or cisions about the course of a patient’s treatment outcome will not be favorable
it can be easily accessed online. The report treatment. The clinic may designate when (8).
presents all the measures and clinical to give the update assessments, and
items described above in an easy-to-use the type of update to be administered. A Technical Support
“lab report” format. All scores are reported Full Update provides scores for all the Complete documentation of the system,
as percentiles, based upon norms for men- scales assessed at Intake, requiring about assessment and reports are provided.
tal health outpatients. Not all patients eight minutes to complete. A Brief Update A User Manual and Clinical Reference Guide
respond authentically to all questions on provides scores for all core scales, requir- are also provided. Polaris can conduct
an assessment, due to literacy limitations, ing about four minutes to complete. training sessions (either in-person or
cognitive impairment, deliberate “faking” The system keeps track of each patient, through Web meetings) and supports
and other factors. Polaris-MH automati- and provides a number of tools to manage train-the-trainer approaches. Polaris pro-
cally conducts a series of checks and alerts when patients are due for update assess- vides end-user technical support during
clinicians with a note on the report if there ments, and which type they should com- normal business hours.
is evidence of inauthentic responding. plete. Update intervals of three or four
Licenses for the use of Polaris-MH start weeks are often employed: this period is Psychometric Properties
at $US750 per year. It is available for free long enough so that measurable improve- Polaris-MH is normed for adults in outpa-
for researchers. Please contact Polaris sales ment is likely to occur, and the program is tient mental health treatment. Psychome-
for additional information: sales@polaris assured of a final assessment within a few tric properties have been documented
health.com. weeks of termination (necessary to the across a broad range of behavioral and
An abbreviated version of evaluation of treatment outcomes). The medical patient populations. They include

Polaris-MH (Polaris-EVS – “Emotional Vital update report indicates the patient’s satis- internal consistency reliability; face, con-
Signs”) is available in both computer and faction with treatment, therapeutic bond, struct, criterion, concurrent and predictive
paper/fax formats. The EVS was developed and compliance with medication (when validity; and sensitivity to change. Findings
to accommodate programs where it is applicable); a trend line of BHS and De- are published elsewhere (2,3) and summa-
impractical for all patients to complete an pression scale scores show the changes rized here. The internal consistency (Coef-
assessment using a computer so Polaris that have occurred since admission; and ficient Alpha) of all scales is acceptable
developed a one-page fax form as an al- a table of change scores for scales and sub- (r>.70) for use in individual patient moni-
ternative. EVS preserves as much of the scales of the symptoms and functioning toring. Reliabilities of the core scales are
clinical content and functionality of the domains indicate areas of improvement or all in the Good-Excellent range (.80-1.0):

Polaris-MH system as possible while of continuing difficulty. Subjective Well-Being (SWB) - .86; Symp-
substantially reducing its length; the EVS The update report also in- toms/S - .93; and Functional Disability/FD -
requires six minutes to complete. Addi- cludes an ETR curve (described above). It .81; BHS - .83. The majority of subscales
tional information on EVS can be obtained is displayed on the update report, together have internal consistency in the good-
from the Polaris website (www.polaris with the patient’s actual BHS score and a excellent range as well, ranging from .75 to
health.com). Failure Boundary. The ETR curve indicates .91. Internal consistency for the strength
the rate and amount of improvement that subscale (Resilience) is .80.
Assessment and Treatment

would normally be achieved by patients All Polaris-MH items are face
Planning with similar characteristics and initial valid, and directly relevant to the evalua-
The broad scope of the clinical report severity score. By comparing the patient’s tion of a patient’s condition and treatment
supports treatment planning and joint pa- progress with the ETR the clinician, (or clin- monitoring. Staff report strong patient
tient-clinician identification of treatment ical supervisor, care manager, or patient) acceptance; patients view the compre-
goals. Presentation of both problem areas can readily determine whether treatment hensive assessment as indicative of the
and strengths enables the clinician to draw is “working” as well as expected. A patient clinician’s commitment to providing
34

the best possible care. In more than 60,000 total symptoms, .61 for behavioral health support reflect the holistic approach often
patient assessments there have been no status, and .75 for subjective wellbeing. used by social workers. Review of the pa-
reports of concern about the appropriate- tient’s progress in relation to ETR is useful
ness of the questions. Construct validity of Institutional Implementation to all involved in a case by enabling clini-
all scales is strong. Items for the symptom Polaris-MH is uniquely suited to the re- cians, clinical supervisors and case man-
scales were constructed by re-casting quirements of interdisciplinary case managers to identify and conduct a detailed
symptoms from the Diagnostic and Statis- agement. Its pan-theoretical construction review of cases that are at risk for poor out-
tical Manual, Fourth Edition (1) for self- avoids the limitations inherent in meas- comes or are candidates for (successful)
report. Items of the Functional Disability ures that are based upon a specific “school” termination.
subscales were constructed from the U.S. of therapy. It tracks changes in patients’ Polaris MH is used in diverse
Social Security disability guidelines. feelings of well-being, symptoms and mental health treatment settings, includ-
Concurrent validity (with the Social functioning–issues of primary concern to ing Kaiser Permanente Psychiatric Clinics
Adjustment Scale, Global Severity Index, all mental health providers. A Summary in southern California, the Canadian Forces
OQ-45, and General Well Being Scale) of Screen enables supervisors, case man- behavioral health treatment facility at
the core scales is also strong. Sensitivity to agers and clinicians to quickly review the Stadacona (N.S.), the Gosnold treatment
change for 792 adults in outpatient mental clinician’s caseload to identify cases for network in Massachusetts (specializes in
health treatment for the core scales, after intensive review, e.g. due to suicidal risk, dually-diagnosed patients), the University
an average of 37 days (s.d.=28 days) in drug abuse or poor progress. In considera- of Wisconsin (Madison) Clinical Training
treatment was also assessed. Findings are tion of psychiatrists, Polaris-MH contains Program and by individual practitioners.
consistent with the Phase Model, with items regarding medication compliance.
effect sizes of .32 for functioning, .51 for The measures for strengths and social

REFERENCES

1. American Psychiatric Association. (1994). Diagnostic and Statistical Manual of
Mental Disorders (4th ed.). Washington, DC. 8. Lutz, W., Martinovich, Z., & Howard, K. I. (1999). Patient profiling: An application

2. Grissom, G. and Lyons, J. (2006). Pragmatic Case Studies and Evidence-Based of random coefficient regression models to depicting the response of a patient
Treatment: Research and Clinical Applications of a Computerized Outcomes to outpatient psychotherapy. Journal of Consulting and Clinical Psychology, 67,
Management System. Pragmatic Case Studies in Psychotherapy Vol 2 (3), 1-28. 571-577.

3. Grissom, G., Lyons, J. & Lutz, W. (2002): Standing on the shoulders of a giant: 9. McLellan, A. T., Kushner, H., Metzger, D., Peters, R., Grissom, G., Pettinati, H., and
development of an outcome management system based on the dose model and Argeriou, M. (1992). Fifth Edition of the Addiction Severity Index. Journal of
phase model of psychotherapy. Journal of Psychotherapy Research, 12, 397-412. Substance Abuse Treatment, Vol.9.

4. Howard, K.I., Kopta, S.M., Krause, M.S. & Orlinsky, D.E. (1986). The dose-effect 10. Ruggiero, K.J., Del Ben, K., Scotti, J.R. & Rabalais, A.E. (2003). Psychometric
relationship in psychotherapy. American Psychologist, 41, 159-164. Properties of the PTSD Checklist—Civilian Version. Journal of Traumatic Stress,
Volume 16, Number 5, 495-502.
5. Howard, K., Lueger, R., Maling, M. & Martinovich, Z. (1993). A phase model of

psychotherapy: Causal mediation of outcome. Journal of Consulting and Clinical 11. Sangsland, Scott B. (2001) Outcomes Management: A Case Study of
Psychology, 61, 678-685. Implementation-Strategy and Experience. Behavioral Outcomes and Guidelines
Sourcebook, New York: Faulkner & Gray, 312-320.
6. Howard, K.I., Moras, K., Brill, P.L, & Martinovich, Z. (1996). Evaluation of
psychotherapy: Efficacy, effectiveness, and patient progress. American 12. Sperry, L., Brill, P., Howard, K. I., & Grissom, G. (1996). Treatment outcomes in
Psychologist. 51, 1059-1064. psychotherapy and psychiatric interventions. New York: Bruner/ Mazel.

7. Lueger, R., Martinovich, Z., Anderson, E., Howard, K., Lutz, W. & Grissom, G. (2001).
Assessing treatment progress of individual patients using expected treatment
response models. Journal of Consulting and Clinical Psychology, 69, 150-158.
35

The PSYCHLOPS PSYCHLOPS, or Psychological Outcome Profiles,
has been designed as a mental health outcome measure. Originally
intended as a before and after measure, to be used for determining
(Psychological outcomes following therapy, it has now developed into a repeated
measures instrument used to track progress throughout a course
Outcome Profiles) of therapy. PSYCHLOPS aims to refocus outcome measurement
away from professionally determined domains and towards a
patient centred definition of outcomes. This emphasis on the pa-
tient perspective is intended to capture items of greatest personal
significance rather than imposing an external frame of reference to
mark Ashworth, Ph.D., Dm, FrCGP, mrCP, interpret psychological distress.
Department of Primary Care and Public Health
Sciences, King’s College london The development of PSYCHLOPS started in 1999 with
the search by primary care therapists for an outcome instrument
that captured aspects of recovery which appeared to be missed by
mark.ashworth@kcl.ac.uk conventional instruments. They reported patients who, during the
course of therapy, appeared to have resolved many of the issues for
maria Kordowicz, Department of Primary Care and Public Health which they were originally referred and yet outcome measurement
Sciences, King’s College london had failed to capture this recovery. In response, an idiographic in-
Peter Schofield, Department of Primary Care and Public Health strument was designed which contained questions asking patients
Sciences, King’s College london to describe their problems in their own words using freetext
response boxes. Idiographic instruments are well described in the
literature (Donnelly & Carswell, 2002) but are generally used in

secondary care and require assistance to complete either from the
therapist or by someone trained in the specific requirements of
the instrument. In contrast, PSYCHLOPS was developed with the
The PSYCHloPS, or Psychological outcome Profiles, is intention of being the first, easy-to-use, self-administered idio-
a mental health outcome measure which can be used graphic measure. The end product was a one-page questionnaire.
as a before and after measure for determining outcomes
following therapy, and as a repeated measures instrument Two principles guided the development of PSY-
used to track patient progress over the course of therapy. CHLOPS. User-involvement was a key feature in its development
PSYCHloPS aims to refocus outcome measurement away and was provided by the UK patient organisation, Depression
from professionally determined domains and towards a
patient centered definition of outcomes. This emphasis Alliance. Since the instrument was intended to be patient centred,

on the patient perspective is intended to capture items it also had to be patient friendly. The wording was scrutinised by
of greatest personal significance rather than imposing
the Plain English Campaign and, after various revisions, the instru-
an external frame of reference to interpret psychological
distress. The measure, which is available in 6 languages, ment was awarded the ‘Crystal Mark’ in recognition of the clarity of
is designed for use in the context of primary care psy- its language. An attractive design was also a feature of the patient-
chotherapy; with the development of a version for
children, the PSYCHloPS is suitable for anyone from the
friendly approach to development and a simple format was devised
age of five years and upwards. The scale is self-adminis- with coloured banding used to highlight each question and colour
tered and self-completed, and assesses three domains, differences to distinguish each version.
including client problems, function and wellbeing.

In this paper, the authors describe the scale, including PSYCHLOPS was piloted then launched in 2004 as a
how it is administered and scored, and the domains it pre-therapy and post-therapy mental health outcome instrument.
assesses. Its use in clinical practice is also discussed. Following validation studies, the instrument went through several
Keywords: PSYCHloPS; Psychological outcome Profiles; stages of refinement to both wording and scoring, and a new dur-
ing-therapy version was introduced in 2010 (Version 5). The inten-
tion of this version was that, after a period of change derived

36

through the iterative process of validation, functional capacity which in turn triggers change, and the score reflects its purpose.
it should be the definitive version of diminished wellbeing. In parallel with its The actual change is simply the during-
PSYCHLOPS, remaining unchanged for a applicability to a broad range of mental therapy or post-therapy total score
minimum of five years. Stability over a health problems, the measure is applica- subtracted from the pre-therapy score.
longer period was intended to promote in- ble to a breadth of talking therapies and Interpreting this change requires calcula-
ternational collaborations and longer term may be used before, during and after any tion of the Effect Size for a sample. The
studies. type of psychological intervention. Effect Size is calculated by dividing the
The measure is designed for change score by the standard deviation of
use in the context of primary care psy- Use and Procedures the pre-therapy score. By using this
chotherapy. Validation studies have ex- The Problem and Function domains of method, change is ‘standardised’ and the
cluded those with literacy problems, PSYCHLOPS elicit freetext responses which greater the pre-therapy score variability,
although therapist assisted completion are then scored by the therapist on an the greater the pre-therapy standard
would be possible in this situation. In 2011, ordinal 6-point scale (ranging from a score deviation and the less the overall Effect
a children’s version of PSYCHLOPS was of zero to five). If the patient only reports Size. Effect Size values greater than 0.8
launched: PSYCHLOPS Kids. This is a short- one Problem, rather than two, then the are generally considered large in health
ened version of PSYCHLOPS and uses score is pro-rated (doubled) such that the service research (Kazis et al., 1989). In com-
emoticon faces rather than tickboxes to maximum possible score for the Problem mon with all idiographic instruments,
elicit scores. With the development of domain remains at ten. The Wellbeing do- there is no population norm since the
‘PSYCHLOPS Kids’, the instrument is suit- main is a nomothetic measure (omitting a baseline score is a measure of items which
able for anyone from the age of five years freetext component), again scored zero to differ between each person and is not
and upwards. five. Thus the score range, derived from the strictly comparable between individuals.
PSYCHLOPS is not intended sum of each domain, is from zero to 20. All data are stored with the

for use as a diagnostic instrument and can In the pre-therapy version of therapist and instrument completion is
therefore be used with patients experi- PSYCHLOPS, patients are asked to describe based on hard copies of the questionnaire,
encing a wide variety of mental health their main Problem (in a freetext box) and not on-line copies. There is no on-line ver-
problems without being confined to those to score it. In subsequent during-therapy sion. Data collection is not centralised.
fulfilling single disease-based diagnostic and post-therapy versions, the therapist The simplicity of score calculation means
criteria. This broad spectrum of distress transcribes the freetext description of the that a score can be calculated immediately
is typical of the sort of mental health original Problem, and the patient is asked upon completion of the instrument.
problems encountered in primary care. to re-score the original Problem (the orig- Progress, or otherwise, can be charted on a
PSYCHLOPS has been developed in English inal score is not disclosed). The same zero to twenty scale. Ideally, this score

but is also available in French, Spanish, process is followed for the other freetext would be available at the start of each
Dutch, Polish and Arabic. questions: the second Problem and Func- talking therapy session.
tion. The Wellbeing score is simply scored There is a charge for use of
Domains Assessed on the scale numbered zero to five, each PSYCHLOPS. Specimen copies may be
Three domains are included in PSYCH- time the instrument is administered. viewed on the website. Actual copies are
LOPS: Problems (2 questions); Function PSYCHLOPS is self-adminis- available on CD-ROM and cost £40 ($CA65)
(1 question); Wellbeing (1 question). The tered and self-completed with the proviso for individual therapists, £100 ($CA160)

underlying Problem-Function-Wellbeing that the therapist transcribes the freetext for small organizations and £250 ($CA400)
domains are derived from a pan- sections from the pre-therapy version to all for larger health service organizations
theoretical model which describes an em- subsequent during-therapy versions and (employing over 100 people). There is no
pirical sequence of causality; psychologi- the post-therapy version. PSYCHLOPS is an annual fee and there is no limit to usage.
cal problems which then trigger deficits in outcome measure, designed to measure

37

Assessment and Treatment Psychometric Properties Patient/Client/Clinician Feedback
Planning Internal reliability has been tested by Based on the findings of a qualitative
Therapists have described the usefulness calculating Cronbach’s alpha for the three study of the views of therapists (Ashworth,
of pre-therapy freetext information domain scores in PSYCHLOPS. Three et al., 2005a), PSYCHLOPS was perceived as
reported in the Problem and Function studies have so far reported internal relia- complementing the information derived
domains of PSYCHLOPS (Ashworth et al., bility data based on alpha scores: 0.79 from conventional quantitative instru-
2005a). This information can be triangu- pre-therapy and 0.87 post therapy (Ash- ments, with its qualitative information
lated with referral information to the ther- worth et al., 2005b); 0.75 pre-therapy and being of particular interest to therapists,
apist from other health professionals such 0.83 post therapy (Ashworth et al., 2008); contributing to the therapist-patient inter-
as general practitioners, and provides a and 0.81 pre-therapy, 0.85 during therapy action. Therapists reported that it was a
focus for therapy from the outset, acting as and 0.88 post therapy (Czachowski et al., ‘therapist friendly’ instrument and likely
a tool to instigate therapeutic work. The 2011). One study has reported test-retest to increase acceptance and uptake of
during-therapy versions elicit information reliability, based on a survey of students outcome measures.
on new problems arising during the in higher education. The test-retest intra-
course of therapy, adding to the informa- class correlation coefficient was 0.70 Institutional Implementation
tion available to the therapist. Although (Evans et al., 2010). Sensitivity to change PSYCHLOPS is a generic instrument, de-
research evidence on the importance of has been reported as Effect Size in three signed within the context of primary care
this information is not yet available, it studies: 1.53 (95% CI 1.30 to 1.76) (Ash- but not aligned to any one primary care
would seem intuitive to suggest that new worth et al., 2005b); 1.61 (95% CI 1.41, 1.80) discipline nor psychotherapy discipline. Its
issues described by patients on PSYCH- (Ashworth et al., 2008); 3.1 (95% CI 2.7, 3.4) findings are of relevance to all primary care
LOPS would need to be addressed by ther- (Czachowski et al., 2011) health professionals involved in the care of
apists during the talking therapy process. The first two studies were patients with mental health problems. The

There are no population norm data for conducted in the setting of psychotherapy UK Department of Health (DH) has pro-
PSYCHLOPS, in common with all idio- offered within the context of primary care duced a list of DH ‘approved’ mental health
graphic instruments (Lacasse et al., 1999; in the UK. The latter study was conducted outcome measures which was published
Donnelly & Carswell, 2002). in a Polish setting and the brevity of the in an ‘Outcomes Compendium’ in 2009.
psychotherapy programme (three sessions PSYCHLOPS is included in the list of
Technical Support of CBT conducted by GPs with a special in- approved measures: www.dh.gov.uk/en/
Background information about PSYCH- terest in CBT) may have contributed to the Publicationsandstatistics/Publications/Pub
LOPS is available from the website: www. high observed Effect Size. licationsPolicyAndGuidance/DH_093316
psychlops.org.uk. The website provides Convergent validity of the PSYCHLOPS is also included

links to background literature, validation measure has been reported in two studies: in the international Quality of Life Instru-
studies, the scoring system, latest devel- comparison with CORE-OM (Clinical Out- ments database: www.proqolid.org.
opments and an email address for further comes in Routine Evaluation – Outcome
information. Upon purchase, an informa- Measure) revealed a Spearman’s rho of
tion pack is mailed out, providing further 0.61, pre- and post therapy data combined
in-depth information on instrument usage. (Ashworth et al., 2005b); comparison with
HADS (Hospital Anxiety Depression Scale)

showed a rho of 0.47 pre-therapy and 0.63
post-therapy (Ashworth et al., 2008).

38

REFERENCES
PSYCHloPS combines both quantita-
tive and qualitative information. Ashworth M, Robinson S, Godfrey E, Parmentier H, Shepherd M, Christey J, Wright K, Matthews V. The experiences
on-going studies will report on both of therapists using a new client-centred psychometric instrument, ‘PSYCHLOPS’ (‘Psychological Outcome
qualitative analysis of patient reported Profiles’). Counselling and Psychotherapy Res 2005;5:37-41.
data and more detailed quantitative Ashworth M, Robinson S, Godfrey E, Shepherd M, Evans C, Seed P, Parmentier H, Tylee A. Measuring mental
analyses in a variety of international health outcomes in primary care: the psychometric properties of a new patient-generated outcome measure,
PSYCHLOPS (Psychological Outcome Profiles). Primary Care Mental Health 2005;3:261-270.
contexts.
Ashworth M, Evans C, Clement S. Measuring psychological outcomes after cognitive behaviour therapy in primary
care: a comparison between a new patient-generated measure, ‘PSYCHLOPS’ (Psychological Outcome Profiles)
If you would like to know more about and ‘HADS’ (Hospital Anxiety Depression Scale). Journal of Mental Health 2008;1-9 iFirst article.
PSYCHloPS or would like to purchase Czachowski S, Seed P, Schofield P, Ashworth M. Measuring psychological change during cognitive behaviour
a copy, please go to the website: therapy in primary care: a Polish study using ‘PSYCHLOPS’ (Psychological Outcome Profiles). PLoS ONE 2011;
www.psychlops.org.uk 6(12): e27378 doi:10.1371/journal.pone.0027378
or contact the Donnelly, C. & Carswell, A. (2002) Individualized outcome measures: a review of the literature. Canadian Journal
of Occupational Therapy, 69, 84-94.
PSYCHloPS research manager,
marilyn Peters: Evans C, Ashworth M, Peters M. Are problems prevalent and stable in non-clinical populations? Problems and
test-retest stability of a patient-generated measure, ‘PSYCHLOPS’ (Psychological Outcome Profiles), in a
marilyn.peters@kcl.ac.uk non-clinical student sample. British Journal of Guidance and Counselling 2010;38:431-439.
Kazis, L., Anderson, J. & Meenan, R. (1989) Effect sizes for interpreting changes in health status. Medical Care, 27,
S178-189.
Lacasse, Y., Wong, E, & Guyatt, G. (1999) Individualising questionnaires. In: Joyce CRB, O’Boyle CA, McGee H, eds.
Individual quality of life. Approaches to conceptualisation and assessment. Harwood Academic Publishers:
Amsterdam, 87-103.


39

The Schwartz In the late 1990’s leadership in the Department of
Psychiatry at Massachusetts General Hospital identified the need
for “an outcome measure suitable for all patients, all treatments, and
Outcome Scale – 10 all levels of care”. The Schwartz Outcome Scale (SOS-10; Blais et al.,
1999) was created to fill that need. The SOS-10 is a unique broad-
(SOS)-10 band low burden measure developed to monitor outcomes, at both
the individual and aggregate level, across a wide range of adult
mental health services. The distinctiveness of the SOS-10 derives
from its method of development. Rather than relying on theory,
symptoms or existing instruments, construction of the SOS-10 was
mark A. blais, Psy.D., guided by insights obtained from a diverse group of senior clini-
Department of Psychiatry,
massachusetts General Hospital cians and patients. Specifically, interviews conducted with senior
and Harvard medical School psychologists, psychiatrists and a neurosurgeon along with patient
focus groups were used to discover the changes that occurred (ex-
cluding symptoms) with successful treatment. The interviews and
mblais@partners.org
focus group discussions were transcribed and reviewed for com-
mon themes. Common themes were used to generate an initial
item pool. Empirical evaluation and refinement identified 20 well
performing items and Rasch analysis was employed to reduce the
scale to its final 10-item version (see Blais et al., 1999 for a detailed
description of the development process).
The SOS-10 is suitable for individuals ages 17 and up.

It has been formally translated into French, Czech, and Spanish.
Chinese and Italian translations are also available. Recent promis-
ing efforts to extend the use of the scale downward into adolescent
populations are also in process.

The Schwartz outcome Scale – 10 (SoS-10) was developed Domains Assessed
to fill the need for an outcome measure suitable for all The SOS-10 is a measure of psychological health and well-being.
patients, all treatments, and all levels of care. more Psychological health is conceived of as an overarching construct
specifically, it was designed to monitor outcomes, at both
the individual and aggregate level, across a wide range that encompasses life satisfaction, interpersonal effectiveness,

of adult mental health services. The SoS-10 is suitable for positive self-appraisal, optimism, and the absence of psychiatric
individuals ages 17 and up, and is available in a number
symptoms.
of languages. It assesses a number of key areas, including
life satisfaction, interpersonal effectiveness, positive
self-appraisal, optimism, and the absence of psychiatric Use and Procedures
symptoms. SoS-10 scores can be used to rapidly identify
a patient’s level of emotional distress or psychological
Patients are asked to rate how they have been doing over the last
dysfunction, which in turn is valuable in clarifying the week on 10 items using a 0 (Never) to 6 (All or nearly all the time)
intensity of services needed and in treatment planning. scale. The SOS-10 is scored by summing the numerical ratings for
This paper describes the development of the scale,

how it is administered and scored, its psychometric each item. This process creates a total score ranging from 0 to
properties, and the domains it assesses. Its use in clinical 60 with higher scores representing greater psychological health
practice is also discussed. well-being and lower scores indicating emotional distress and
Keywords: Schwartz outcome Scale – 10; SoS-10; poorer psychological health. While the SOS-10 has no validity scale,
scores at the extreme ends of the range (0 or 60) are rare (occurring

40

less than 2 percent of the time) and are greatly enhances the information ob- treatments). The accumulated research
therefore considered invalid. The scale can tained from TAU outcome measurement also supports the construct validity of the
be scored with up to two missing items by programs and increases the comparability SOS-10 as a broad measure of psychologi-
using a mean score imputation to gener- of findings across studies (see Blais et al., cal functioning (Blais et al., 1999; Haggerty
ate a total score. The SOS-10 can be ad- 2011). SOS-10 scores can also be used to et al., 2009; Young et al., 2004). The
ministered in traditional paper-and-pencil rapidly identify a patient’s level of emo- SOS-10 correlates significantly and in
format or electronically (score equivalence tional distress or psychological dysfunc- the predicted direction with measures of
has been demonstrated for web based ad- tion. Drawing on data from over 8,000 psychiatric symptom severity (-0.67),
ministration). It is recommended that pa- outpatients the following distress ranges alexithymia (-0.58), hopelessness (-0.66),
tients complete the scale prior to a may prove helpful markers: Minimal negative affect (-0.72), self-esteem (0.81),
treatment appointment. This way the cli- (59-40), Mild (39-33), Moderate (32-23) satisfaction with life (0.78), positive affect
nician can determine whether the SOS-10 and Severe (22-1). Accurately identifying a (0.67) and physical functioning (0.36).
was completed and is valid, and review the patient’s level of distress at the outset of SOS-10 is also significantly related to
total score for clinical implications prior to treatment can help clarify the intensity of measures of the normal personality (Big
the session. services needed, i.e. weekly individual psy- Five Traits). The SOS-10 correlates signifi-
The SOS-10 has been widely chotherapy, multiple sessions per week or cantly with the Outcomes Questionnaire-
adapted as a program level treatment multiple forms of treatments. In this way 45 (OQ-45; Lambert et al., 1996). The
evaluation tool and many programs have routine use of the SOS-10 can aid treat- SOS-10 is strongly correlated with OQ-45
contributed data to the SOS-10 interpre- ment planning. Furthermore, as clinicians total score (-0.84), and with its subscales.
tive database. Presently, the database become familiar with the tool, the use Together these findings demonstrate the
contains intake SOS-10 scores for 8,056 of severity ranges can provide easily breadth of the SOS-10 and offer solid evi-
outpatients and 5,541 inpatients. As a recognized reference points for multi- dence of its construct validity.

measure of psychological health and well- disciplinary communication. Lastly, be- The SOS-10 has also demon-
being, the SOS-10 is also attractive to non- cause SOS-10 items are not directly related strated sensitivity to change for a wide
clinical researchers. As a result our data to psychiatric symptoms, reviewing variety of treatment modalities and may
base also contains SOS-10 scores for unique responses to individual items with be especially sensitive to detecting early
2,000 non-patients. patients can afford a non-threatening treatment change (Hilsenroth et al., 2001).
Although the SOS-10 is a avenue for discussing personal strengths The SOS-10 has been employed as an out-
proprietary instrument, the scale is made and weaknesses. comes measure in studies of Psychody-
available free of charge for practitioners, namic Psychotherapy, Dialectical Behavior
researchers and non-profit healthcare Psychometric Properties Therapy, residential treatment for refrac-

organizations. The SOS-10 has outstanding psychomet- tory Obsessive Compulsive Disorder, Inpa-
ric properties. Its internal consistency in tient psychiatric treatment as usual and
Assessment and Treatment published studies has ranged from 0.84 to inpatient substance abuse treatment as
Planning 0.96. The test-retest reliability for the scale usual. A study by Blais et al. (2010) demon-
Owen and Imel (2010) outline a rationale is also strong, with studies reporting retest strated the utility of the SOS-10 as a com-
and a practice friendly procedure for in- correlations of 0.86 and 0.87. In addition, mon outcome measure for evaluating
corporating the SOS-10 into ongoing clin- no meaningful age or gender effects have treatment as usual across a large diverse

ical care. The availability of non-patient been reported. Multiple studies both in outpatient psychiatric practice.
reference data is valuable as it allows for the original English and in translations
calculation of both a Reliable Change have found the SOS-10 to be uni-factorial. Institutional Implementation
Index and Clinically Significant Improve- Factor invariance has also been shown The SOS-10 is currently used as a common
ment. The ability to apply more sophisti- across samples (patients & non-patients), outcome measure for all adult psychiatry
cated treatment effectiveness analyses and measurement points (pre & post services provided within the Partners

41

Healthcare System. Partners Healthcare
includes the majority of hospitals and REFERENCES
community heath clinics associated with
Harvard Medical School. Many other psy- Blais, M. A., Lenderking, W. R. Baer, L., deLorell, A., Peets, K., Leahy, L., & Burns, C. (1999). Development and initial
validation of a brief mental health outcome measure. Journal of Personality Assessment, 73, 359-373.
chiatric hospitals, community mental
Blais, M. A., Sinclair, S., Baity, M., Worth, J., Weiss, A., Ball, L., et al. (2011). Measuring outcomes in adult outpatient
health centers and college counseling cen- psychiatry Clinical Psychology & Psychotherapy published online 17 Mar 2011 (DOI: 10.1002/cpp.749).
ters across the United States have been Haggerty, G., Blake, M., Naraine, M., Siefert, C. & Blais, M. (2010). Construct validity of the Schwartz Outcome
granted permission to use the SOS-10, as scale-10: Comparisons to interpersonal distress, adult attachment, alexithymia, the five factor model, romantic
relationship length and ratings of childhood memories. Clinical Psychology and Psychotherapy, 17(1) 44-50.
have a number of treatment facilities in the
Hilsenroth, M., Ackerman, S., & Blagys, M. (2001). Evaluating the Phase Model of Change During Short-Term
United Kingdom. It has also been licensed
Psychodynamic Psychotherapy. Psychotherapy Research, 11, 29-47.
for use by managed care organizations.
Lambert, M., Hansen, N., Umphress, V., Lunnen, K., Okiishi, J., Burlinggame, G., & Reinsinger, C. (1996). Administration
and Scoring Manual for the Outcome Questionnaire (OQ-45.2). Stevenson, MD. American Professional Credentialing
Services.
Owens, J., and Imel, Z. (2010). Rating scales in psychotherapy practice. In L. Baer & M Blais (Eds.), Handbook of
Clinical Rating Scales and Assessment in Psychiatry and Mental Health. New York, NY: Humana Press,
pages 257-270.
Young, J. L., Waehler, C. A., Laux, J. M., McDaniel, P. S., & Hilsenroth, M. J. (2003). Four studies extending the utility of
the Schwartz Outcome Scale (SOS-10). Journal of Personality Assessment, 80, 130-138.

42

The Treatment The Treatment Outcome Package (TOP) was designed
to meet the objectives of the Core Battery Conference (Horowitz,
Lambert & Strupp, 1997) and developed to serve as a clinically use-
Outcome Package ful assessment and outcome battery for all levels of behavioral
healthcare (Kraus, Seligman & Jordan, 2005). Initial versions of the
(TOP) tool began with 250 items derived from DSM-IV symptoms and re-
fined through extensive exploratory and confirmatory factor ana-
lytic work for adult, adolescent and child populations (e.g., Kraus,
Boswell, Wright, Castonguay & Pincus, 2010). TOP is available in Eng-
lish, Spanish, German, Dutch, Portuguese, Chinese, Vietnamese,
David r. Kraus, Ph.D., Haitian, and Cape Verdean. In addition, a French version is in
outcome referrals, Inc.
process.

Domains Assessed
TOP is a-theoretical and assesses twelve clinical and functional
dkraus@outcomereferrals.com
domains that include the following (depending on the age version
chosen):
• Quality of Life
• Substance Abuse
• Depression
• Panic/Anxiety
• Psychosis

• Mania
• Suicidality
• Violence
• Work/School
This paper presents the Treatment outcome Package • Social Dysfunction
(ToP), which is designed to serve as a clinically useful as-
sessment and outcome battery for all levels of behavioral • Sexual Functioning
healthcare, including traditional psychotherapy. The ToP, • Sleep Disorders
which is available in nine languages, is a-theoretical and
• Eating Disorders
assesses twelve clinical and functional domains. Depend-
ing on the age of the identified patient these domains can • Conduct Disorder

include: quality of life, substance abuse, depression; • Sexual Aggression
panic/anxiety; psychosis, mania, suicidality, violence, work
and school, social dysfunction, sexual functioning, sleep
• Separation Anxiety
disorders, • Attention Deficits (ADHD)
eating disorders, conduct disorders, sexual aggression, • Assertiveness
separation anxiety, attention deficits, assertiveness,
bladder control and psychological strengths. The ToP • Bladder Control
provides clinicians with real-time clinical reports through • Psychological Strengths
the use of alerts to off-track treatment that is likely
to end in deteriorated outcomes. Scoring is compared

to non-clinical norms so that the level of pathology can Use and Procedures
easily be tracked. This paper describes the ToP and TOP is provided as a free service through WellnessCheck.net,
presents its psychometric properties, as well as the
including free scoring and real-time clinical reporting. Online man-
procedures related to its use in diverse clinical contexts.
uals and videos provide help with administration and scoring.
Keywords: Treatment outcome Package, ToP; treatment
outcome; progress monitoring; psychotherapy Methods are also available that send customized links to patients so
that clinicians do not need to administer questionnaires in the

43

office and patients can complete ques- world (or local regions) based on a grow- The discriminant validity of the TOP is
tionnaires (ideally once per month) at ing database of over a million patients excellent, with 92% of consumers showing
home or work. Daily reminders are pro- (Kraus & Castonguay, 2010). TOP domains pathological scores and TOP scores
vided via email until the patient completes have been linked to evidence-based prac- demonstrating an ability to discriminate
the agreed upon questionnaire or decides tices and principles and reporting struc- between subjects who are in treatment
to withdraw. Privacy is protected by the tures facilitate rapid improvement of and those who are not. Convergent valid-
use of identifiers that only the provider substandard treatment benchmarks ity has been demonstrated with a number
knows how to link back to an individual (Kraus, Wolfe & Castonguay, 2006; Adel- of well established measures, including
and all direct identifiers to an individual man, 2005, 2006, 2007, 2008). These sys- the Beck Depression Inventory (BDI), the
(name, address, etc.) are not collected or tems are used to identify the inherent Minnesota Multiphasic Personality Inven-
stored. Potentially identifiable information strengths in any given provider population tory (MMPI), the Brief Symptom Inventory
is encrypted and kept in non-linkable with 96% of all providers demonstrating (BSI), the BASIS-32, the SF-36, the Child Be-
systems so that patient anonymity is reliable proficiency in treating at least one havioral Checklist (CBCL), and UNCOPE. For
protected (Kraus & Horan, 1998). major symptom cluster (Kraus, Caston- example, the depression scale of the TOP
guay, Boswell, Nordberg & Hayes, 2011). was highly correlated (r=.91) with the BDI.
Assessment and Treatment Norms for the TOP were established using
Planning Technical Support large samples of over 1 million participants
Real-time clinical reports are designed by Customer support is provided through for the clinical norms and of 2,000 partici-
clinicians to maximize clinical relevance toll-free telephone numbers and on-line pants for the general population norms.
(Kraus, Wolfe & Castonguay, 2006) through resources, manuals and videos. Since
the use of alerts to off-track treatment that WellnessCheck.net scores, reports, stores, Patient/Client/Clinician Feedback
is likely to end in deteriorated outcomes and benchmarks all data, there is no need Each completed TOP is centrally

and/or expensive psychiatric hospitaliza- for scoring manuals or procedures. processed, scored and returned with a
tion. Scoring is compared to general pop- two- or three-page feedback report. This
ulation (non-clinical) norms so that the Psychometrics Properties feedback provides alerts as to whether
level of pathology can easily be tracked Numerous studies (Kraus, Seligman & treatment is on track and delivers a check-
graphically for the past 20 TOP administra- Jordan, 2005; Kraus & Castonguay, 2010; list of evidence-based tasks that should be
tions. Reviewing the detailed clinical re- Kraus, Boswell, Wright, Castonguay & completed or considered in order to in-
ports throughout treatment enhance the Pincus, 2010) using confirmatory factor crease the chances that treatment will end
therapeutic alliance, helps patients to re- analysis with data collected from 19,801 successfully. In addition, clinicians are
veal important clinical information, aids in participants in 383 facilities (5 split sam- alerted if the patient is at high risk of being

treatment plan reviews and setting goals ples) have shown the construct validity of hospitalized within the next six months for
and priorities. the TOP, with a Goodness-of-Fit Index expensive psychiatric or substance abuse
Monthly aggregate reports above .95, a Comparative Fit Index of .95, treatment (McAleavey, Nordberg, Kraus &
are provided for each clinician, service a Non-Normed Fit Index of .94, and a Root Castonguay, in press). The second page of
(e.g., a partial hospitalization program) and Mean Square Error of Approximation of the TOP feedback report is designed to be
each agency. These reports highlight areas .035. Studies have also shown that the copied and given to the client as patient-
of strength and weakness and are bench- TOP has good internal consistency, level feedback (Youn, Kraus & Castonguay,

marked, risk adjusted, and mapped ranging from .53 to .93, and strong test- in press).
against similar professionals across the retest reliability ranging from .76 to .94.

44

Institutional Implementation
TOP has been used by more than 40,000
REFERENCES
clinicians, and thousands of organizations
Adelman, R. et. al, (2005). Reducing adolescent clients’ anger in a residential substance abuse treatment facility.
including self-insured employers, health
Journal on Quality and Patient Safety, 31, (6), 325-327.
plans, hospitals, community mental health
Adelman, R. (2006). The Angry Adolescent & Constructivist REBT. In Cummins, P. (Ed.), Working with Anger:
centers and provider networks to docu- a Practical Constructivist Approach, London: John Wiley & Sons.
ment and improve the quality of care. Each Adelman. R. (2007). Reducing Anger in Adolescents: An REBT Approach, Center City, Minnesota: Hazelden Press.
TOP domain is tied to libraries of evidence- Adelman, R. (2008). Methods of Reconstruction with Adolescent Substance Abusers: Combining REBT and
based practices and principles that include Constructivism. In Raskin, J.D. & Bridges, S. (Ed.), Studies in Meaning 3: Constructivist Psychotherapy in the Real
World, New York: Pace University Press.
scorecards, risk-adjusted benchmarking
Horowitz, L.M., Lambert, M.J., & Strupp, H.H. (Eds.). (1997). Measuring patient change in mood, anxiety, and
and improvement reports all designed to personality disorders: Toward a core battery. Washington, DC: American Psycho- logical Association Press.
deliver roadmaps for innovative quality Kraus, D., Boswell, J., Wright, A. Castonguay, L., & Pincus, A., (2010). Factor Structure of the Treatment Outcome
improvement strategies that have won Package for Children. Journal of Clinical Psychology, 66, 627-640.
TOP customers the highest awards for Kraus, D. R., & Castonguay, L. G. (2010). TOP: Development & use In naturalistic settings. In M. Barkham, G. Hardy,
quality improvement (Adelman, 2005, & J. Mellor-Clark (Eds.), A CORE approach to delivering practice-based evidence in counseling and the psychological
therapies. London: Wiley Press.
2006, 2007, 2008).
Kraus, D.R., Castonguay, L.G., Boswell, J.F., Nordberg, S.S., & Hayes, J.A. (2011). Therapist effectiveness: Implications
for accountability and patient care. Psychotherapy Research, 21, 267-276.
Kraus, D.R., Castonguay, L.G., Hayes, J.A. & Barber, J.P. (2010) The Empirically Supported Therapist: All Clinicians
Have Strengths and Weaknesses. Psychologie Quebec. 1, 12-15.
Kraus, D. R., & Horan, F. P. (1998). Protecting Client Confidentiality and Improving Provider Relations Through a
New Form of Managed Care: Collaborative Care Management. The 1998 Behavioral Outcomes and Guidelines;
Faulkner and Gray: New York.

Kraus, D.R., Seligman, D., & Jordan, J.R., (2005). Validation of a behavioral health treatment outcome and assessment
tool designed for naturalistic settings: The treatment outcome package. Journal of Clinical Psychology, 61,
285–314.
Kraus, D., & Wolfe, A. & Castonguay, L.G., (2006). The outcome assistant: A kinder philosophy to the management of
outcome. Psychotherapy Bulletin, 41, 23-31.
McAleavey, A.A., Nordberg, S.S., Kraus, D.R., Castonguay, L.G., (in press) Errors in treatment outcome monitoring:
Implications of multidimensional and general measurements for real-world psychotherapy. Canadian Psychology.
Youn, S.J., Kraus, D.R., Castonguay, L.G. (in press) The Treatment Outcome Package: Facilitating practice and clinically
relevant research. Psychotherapy.


45

Integrating Science and Practice (Mller & Bargmann)

More Related Content

Viewers also liked (20)

Similar to Integrating Science and Practice (Mller & Bargmann) (20)

More from Scott Miller (20)

Integrating Science and Practice (Mller & Bargmann)