![variety of illicit drugs used, or self-esteem, between the kids who received Project
DARE versus the comparison health education training.
Suppose you were a school counselor and assigned the task of choosing a drug
abuse prevention program for your school to provide. Given the much higher costs
of the DARE program, relative to the less expensive (in terms of time and money)
health education training; the failure by Lynam et al. (1999) to find a superior long-
term effect of DARE; and prior evaluation studies indicating that DARE has no
long-term effect on drug use (e.g., Dukes, Ullman, & Stein, 1996), would you try
to obtain Project DARE for your school? Common sense suggests no, until such
time as DARE develops a stronger empirical research base. But should you make a
decision without looking at the studies yourself? Probably not.
Suppose you are asked to help provide mental health counseling services to
women who experience post-traumatic stress disorder (PTSD) following a sexual
assault. What should you do to help these clients? An evaluation study of therapy
BOX 1.1
Opinions on the Importance of
Evaluation Research
±
±
±
±
±
±
±
±
±
±
±
±
I appeal to you…. Measure, evaluate, estimate, appraise your results, in some form, in
any terms that rest upon something beyond faith, assertion, and “illustrative case.”
State your objectives and how far you have reached them…. Out of such evaluations
will come, I believe, better service to the client. (Cabot, 1931)
The third type of research, evaluative studies of welfare programs and the activities of
practitioners, are the most important of all. (Angell, 1954, p. 169)
[S]ocial work is not a science whose aim is to derive knowledge; it is a technology
whose aim is to apply knowledge for the purpose of control. Therefore, on the re-
search continuum social work research falls nearer to the applied end, because of its
purpose of practical knowledge. (Greenwood, 1957, p. 315)
Evaluation and client feedback are not only necessary for effective service delivery, but
are an ethical requirement of the profession. Systematic methods must be developed to
assess whether social workers are helping, harming, or doing nothing for the people
they serve. (Rosenberg & Brody, 1974, p. 349)
Social work has no more important use of research methods than assessment of the
consequences of practice and policy choices…. [S]mall scale, agency-based studies are
worthwhile if they succeed in placing interest in effectiveness at the center of agency
practice and when they create a critical alliance between practitioners and researchers.
(Mullen, 1995, pp. 282–283)
Studies are needed on the effectiveness of psychosocial intervention, including inter-
ventions previously tested under ideal controlled conditions, in real-world health care
systems. (Ell, 1996, p. 589)
Research on actual service interventions is the critical element in connecting research
to the knowledge base used by professional practitioners…. [T]he issue now is one of
developing investigations of social work intervention initiatives, studies that go be-
yond descriptions and explanatory research. (Austin, 1998, pp. 17, 43)
We need to establish a research agenda for social work…. And intervention studies
must be high in priority to such an agenda. (Rosen, Proctor, & Staudt, 1999, p. 9).
introduction 3](https://image.slidesharecdn.com/2226304-250619003056-8b473278/85/Program-Evaluation-An-Introduction-5th-Edition-David-Royse-Bruce-A-Thyer-20-320.jpg)
![Keep in mind that from its inception, professional social work accepted the doctrine
of positivism, established in the early part of the 1800s by the French philosopher
and scientist Auguste Comte. Comte is said to have established the discipline of soci-
ology, which he originally called social physics, based on his contention that human
behavior could be studied using the same tools and principles that science used to
study natural phenomena:
The word positive came from ponere and had been employed since the fourteenth
century to mean laid down. In the sixteenth century, it began to refer to knowledge
that was based on facts and was thus reasonably certain. Eighteenth century thinkers
used the word positive to oppose the metaphysical. (Pickering, 1993, p. 65)
According to Comte, “Unlike theological beliefs, scientific truths could be
proved if necessary…. [S]cientific truths were always provisional because they could
be proved wrong at any time” (Pickering, 1993, p. 171). Despite its name, positiv-
ism took a provisional approach to the development of knowledge, with Comte
noting in 1825 that “Scientific laws were only hypotheses constructed by man
with external materials and confirmed by observation; they amounted to no more
than approximations of a reality that could never be rigorously understood”
(Pickering, 1993, p. 294).
From its inception, positivism was concerned not just with studying human
phenomena, but also with improving the human condition. “Social science was
not … just an intellectual mixture of history, the physical sciences, physiology, and
political economy. It had a practical vocation: to regenerate society…. Although
Comte admitted that we could never fully know external reality, he assumed that
scientific theories were getting closer to representing it exactly” (Pickering, 1993,
pp. 294, 296, italics added). Positivism came to exert an enormous influence on sci-
ence, on the emerging social sciences, and in social welfare in particular. In the
United States, the American Social Science Association (ASSA) was established in
1865, and set forth as its mandate:
Social science was understood by (ASSA) members to refer to the whole realm of
problematical relationships in human affairs. One became a social scientist “by
contributing to the store of esoteric knowledge and practical expertises … a new way to
care for the insane or to administer charity—all of these were equally valuable
contributions to social science.” (Haskell, 1997, pp. 97–98, italics added)
This Association proposes to afford to all persons interested in human improvement,
an opportunity to consider social economics as a whole…. They are to collect all
facts, diffuse all knowledge, and stimulate all inquiry, which have a bearing on social
welfare. (Haskell, 1997, p. 102, italics added)
From the ASSA emerged, in 1879, the Conference on Charities, transformed in
1884 into the National Conference of Charities and Corrections (NCCC). In 1917
the NCCC became the National Conference on Social Work, transformed again
into the National Conference on Social Welfare in 1957, which in turn dissolved
in the mid-1980s. A paper presented at the 1889 meeting of the NCCC was titled
“Scientific Charity” and an article appearing in an 1894 issue of the influential
journal The Charities Review was titled “A Scientific Basis for Charity.” Such early
works were a part of the movement called scientific charity (or scientific philan-
thropy), which had its own origins in the 1870s, again based on the fundamental
assumptions of positivism.
introduction 27](https://image.slidesharecdn.com/2226304-250619003056-8b473278/85/Program-Evaluation-An-Introduction-5th-Edition-David-Royse-Bruce-A-Thyer-44-320.jpg)
![This ebook is for the use of anyone anywhere in the United States
and most other parts of the world at no cost and with almost no
restrictions whatsoever. You may copy it, give it away or re-use it
under the terms of the Project Gutenberg License included with this
ebook or online at www.gutenberg.org. If you are not located in the
United States, you will have to check the laws of the country where
you are located before using this eBook.
Title: Miss Santa Claus of the Pullman
Author: Annie F. Johnston
Illustrator: Reginald Bathurst Birch
Release date: December 11, 2012 [eBook #41604]
Most recently updated: October 23, 2024
Language: English
Credits: E-text prepared by Suzanne Shell, Emmy, and the Online
Distributed Proofreading Team (http://www.pgdp.net) from
page images generously made available by Internet
Archive/American Libraries
(http://archive.org/details/americana)
*** START OF THE PROJECT GUTENBERG EBOOK MISS SANTA
CLAUS OF THE PULLMAN ***](https://image.slidesharecdn.com/2226304-250619003056-8b473278/85/Program-Evaluation-An-Introduction-5th-Edition-David-Royse-Bruce-A-Thyer-60-320.jpg)
Program Evaluation An Introduction 5th Edition David Royse Bruce A Thyer
- 1. Program Evaluation An Introduction 5th Edition David Royse Bruce A Thyer download https://ebookbell.com/product/program-evaluation-an- introduction-5th-edition-david-royse-bruce-a-thyer-4452608 Explore and download more ebooks at ebookbell.com
- 2. Here are some recommended products that we believe you will be interested in. You can click the link to download. Program Evaluation An Introduction To An Evidencebased Approach 6th David Royse https://ebookbell.com/product/program-evaluation-an-introduction-to- an-evidencebased-approach-6th-david-royse-22485132 Program Evaluation And Performance Measurement An Introduction To Practice Paperback James C Mcdavid Irene Huse Laura R L Ingleson https://ebookbell.com/product/program-evaluation-and-performance- measurement-an-introduction-to-practice-paperback-james-c-mcdavid- irene-huse-laura-r-l-ingleson-10010244 Prevention Program Development And Evaluation An Incidence Reduction Culturally Relevant Approach Robert K Conyne https://ebookbell.com/product/prevention-program-development-and- evaluation-an-incidence-reduction-culturally-relevant-approach-robert- k-conyne-49441398 Achieving Nutrient And Sediment Reduction Goals In The Chesapeake Bay An Evaluation Of Program Strategies And Implementation Committee On The Evaluation Of Chesapeake Bay Program Implementation For Nutrient Reduction To Improve Water Quality https://ebookbell.com/product/achieving-nutrient-and-sediment- reduction-goals-in-the-chesapeake-bay-an-evaluation-of-program- strategies-and-implementation-committee-on-the-evaluation-of- chesapeake-bay-program-implementation-for-nutrient-reduction-to- improve-water-quality-2323604
- 3. An Outcome Evaluation Of The Success For Kids Program Nicole Maestas Sarah J Gaillot https://ebookbell.com/product/an-outcome-evaluation-of-the-success- for-kids-program-nicole-maestas-sarah-j-gaillot-47122048 Program Development In The 21st Century An Evidencebased Approach To Design Implementation And Evaluation Nancy G Calley https://ebookbell.com/product/program-development-in-the-21st-century- an-evidencebased-approach-to-design-implementation-and-evaluation- nancy-g-calley-43191282 An Online Doctorate For Researching Professionals Program Design Implementation And Evaluation 1st Edition Swapna Kumar https://ebookbell.com/product/an-online-doctorate-for-researching- professionals-program-design-implementation-and-evaluation-1st- edition-swapna-kumar-36877652 Evaluating School Programs An Educators Guide 3rd Edition James R Sanders https://ebookbell.com/product/evaluating-school-programs-an-educators- guide-3rd-edition-james-r-sanders-44523888 Automatic Programming In An Arbitrary Language Evolving Programs With Grammatical Evolution Phd Thesis Michael Oneill https://ebookbell.com/product/automatic-programming-in-an-arbitrary- language-evolving-programs-with-grammatical-evolution-phd-thesis- michael-oneill-6817066
- 7. This page intentionally left blank
- 8. Program Evaluation An Introduction F I F T H E D I T I O N David Royse University of Kentucky Bruce A. Thyer Florida State University Deborah K. Padgett New York University ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± Australia • Canada • Mexico • Singapore • Spain • United Kingdom • United States
- 9. Program Evaluation: An Introduction, Fifth Edition David Royse, Bruce A. Thyer, and Deborah K. Padgett Acquisitions Editor: Seth Dobrin Assistant Editor: Diane Mars Editorial Assistant: Rachel McDonald Media Editor: Andrew Keay Senior Marketing Manager: Trent Whatcott Marketing Assistant: Darlene Macanan Senior Marketing Communications Manager: Tami Strang Editorial Production: Matt Ballantyne Creative Director: Rob Hugel Art Director: Caryl Gorska Print Buyer: Paula Vang Permissions Editor, Text: Roberta Broyer Production Service: Pre-PressPMG Copy Editor: Wendy Catalano Cover Designer: Lisa Buckley Compositor: Pre-PressPMG © 2010, 2006 Wadsworth, Cengage Learning ALL RIGHTS RESERVED. No part of this work covered by the copyright herein may be reproduced, transmitted, stored, or used in any form or by any means graphic, electronic, or mechanical, including but not limited to photocopying, recording, scanning, digitizing, taping, Web distribution, information networks, or information storage and retrieval systems, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without the prior written permission of the publisher. For product information and technology assistance contact us at Cengage Learning Academic Resource Center 1-800-354-9706 For permission to use material from this text or product, submit allrequests online at www.cengage.com/ permissions Further permissions questions can be emailed to permissionrequest@cengage.com Library of Congress Control Number: 2008943608 ISBN-13: 978-0-495-60166-1 ISBN-10: 0-495-60166-7 Wadsworth 10 Davis Drive Belmont, CA 94002-3098 USA Cengage learning is a leading provider of customized learning solutions with office locations around the globe, including Singapore,theUnitedkingdom,Australia,Mexico,Brazil,andJapan. Locate your local office at: www.cengage.com/international. Cengage Learning products are represented in Canada by Nelson Education, Ltd. For your course and learning solutions, visit academic.cengage.com Purchase any of our products at your local college store or at our preferred online store www.ichapters.com Printed in Canada 1 2 3 4 5 6 7 13 12 11 10 09
- 10. Contents PREFACE xi Chapter 1 Introduction 1 The Importance of Program Evaluation 1 What Is a Program? 5 Characteristics of “Good” Programs 5 Program Evaluation Defined 12 Reasons Why Programs Are Evaluated 13 Motivations for Program Evaluation 14 Overcoming the Subjective Perspective 18 Philosophical Assumptions of Program Evaluation 22 More on Positivism 26 Chapter Recap 29 Chapter 2 Ethical Issues in Program Evaluation 34 Historical Background: The Origin and Growth of Institutional Review Boards (IRBS) 34 Ethical Guidelines 35 How Ethical Guidelines Get Operationalized 40 Research with Special Populations 43 v
- 11. Ethical Issues Pertaining to Diverse Cultures and Groups 44 The Practitioner-Evaluator’s Ethical Responsibilities 47 Concluding Thoughts on Ethics and Evaluation 52 Chapter 3 Needs Assessment 55 What Is Needs Assessment? 55 Definitions of Need 57 Planning a Needs Assessment 59 Selecting a Needs Assessment Approach 61 Convergent Analysis and Multimethod Approaches 69 Thinking Creatively about Needs Assessment 72 Community Readiness 76 Chapter 4 Qualitative and Mixed Methods in Evaluation 83 Introduction 83 What Is “Qualitative Evaluation”? 83 When Is Qualitative Evaluation Useful? 84 Qualitative Methods and Evaluation of Practitioners In Situ 86 Methods of Qualitative Evaluation 86 Managing and Organization Data 93 Data Analysis 93 Quality Control 94 Writing the Report 96 Dissemination and Advocacy 98 Examples of Qualitative Evaluations 102 Chapter 5 Formative and Process Evaluation 108 Developing a Logic Model 108 Formative Evaluation 112 Conducting a Formative Evaluation 113 Process Evaluation 116 Program Description 117 Program Monitoring 121 Becoming a Program Monitor 122 vi contents
- 12. Mission Statements, Goals, and Objectives 123 Writing Program Objectives 127 What Should Be Monitored? 129 Quality Assurance 132 Total Quality Management 134 Chapter Recap 137 Chapter 6 Single System Research Designs 141 What Are Single System Research Designs? 141 Selecting Outcome Measures 143 Assessing Measures Over Time 145 Needs Assessments 145 Notation and General Principles 147 Formative Evaluations 148 Quality Assurance Studies 152 Summative Evaluation Designs 154 Experimental Designs 157 External Validity 162 Inferential Statistics 165 How To Prepare Graphs 167 Ethics of Single System Research Designs 168 Chapter Recap 170 Chapter 7 Client Satisfaction 175 The Importance of Monitoring Consumer Satisfaction 175 The Problem with Client Satisfaction Studies 177 A Sampling of Recent Client Satisfaction Studies 180 Annotations about Client Satisfaction 181 Explanations for High Ratings 183 Recommendations for Client Satisfaction Studies 184 Chapter 8 Sampling 194 Nonprobability Sampling Designs 194 Probability (Scientific) Sampling 198 Considerations in Selecting a Sample 200 contents vii
- 13. How Big Should the Sample Be? 201 Chapter Recap 204 Chapter 9 Group Research Designs 207 What Are Group Research Designs? 207 Starting an Outcome Evaluation 208 Outcome Evaluation Designs 210 General Principles of Group Research Designs 212 Pre-experimental Research Designs 212 Quasi-Experimental Research Designs 223 Some Threats to Internal Validity 229 Protection Against Alternative Explanations 233 Experimental Designs 234 Efficacy and Effectiveness Studies 241 What About Negative Outcome Studies? 242 A Note About the Term Experiment 245 Chapter Recap 247 Chapter 10 Cost-Effectiveness and Cost Analysis 255 Cost as an Evaluative Criterion 255 Example of a Cost-Effectiveness Evaluation 257 How to Do a Cost-Effectiveness Study 258 Whose Point of View? 260 Cost-Benefit Analysis 262 Chapter Recap 265 Chapter 11 Measurement Tools and Strategies 271 Importance of Measurement 271 Deciding What to Measure 273 Reliability 275 Locating Appropriate Instruments 285 Constructing “Good” Evaluation Instruments— Questionnaire Design 288 Troubleshooting Questions—An Opportunity to Apply What You’ve Learned 291 viii contents
- 14. Chapter 12 Illustrations of Instruments 301 Criteria for Selecting Evaluation Instruments 302 Clinical Anxiety Scale 303 CES-D Scale 305 Rosenberg Self-Esteem Scale 307 Evaluation Self-Efficacy Scale 309 Policy Advocacy Behavior Scale 311 The Community Attitudes Toward Sex Offenders Scale (CATSO) 314 Chapter 13 Pragmatic Issues 317 Treatment Fidelity 317 Fidelity Nightmares 321 Program Drift and the Transfer of Programs 324 Political Nature of Evaluation 325 The “Threat” of Evaluation 328 Guidelines for Evaluation in Politically Charged Arenas 329 Culturally Sensitive Evaluation Practice 332 Final Thoughts 336 Chapter 14 Data Analysis 341 What Does It Mean to Analyze Data? 341 Data Analysis and the Computer 342 Levels of Measurement 343 Univariate Analysis 345 Bivariate Analysis 351 Multivariate Analysis 360 Myths About Statistical Significance 362 Understanding Trends 364 Using Statistics in Reports 365 Type I and Type II Errors 367 Final Thoughts 370 Chapter 15 Writing Evaluation Proposals, Reports, and Journal Articles 373 Components of the Evaluation Proposal and Report 373 Considerations in Planning and Writing Evaluation Reports 383 contents ix
- 15. Common Mistakes Made by Students in Writing Evaluation Reports 384 Checklist for Writing and Assessing Evaluation Reports 387 The Utilization of Evaluation Reports 387 Writing for Professional Publication 389 Index 393 x contents
- 16. Preface A textbook must provide, first and foremost, information to assist the reader in bet- ter understanding the topic. Second, it ought to provide the information in a way that can be easily accessed and digested, and it needs to be credible. Textbooks that have gone through multiple editions continue to improve as a result of re- viewers’ comments and readers’ feedback, and this one is no exception. Looking back over the efforts associated with this Fifth Edition, the old wedding custom of “something old, something new, something borrowed, something blue” comes to mind. We have built upon the solid foundation of previous editions, but then added “something new.” It almost goes without saying that we have “borrowed” from others in that we both cite and quote examples of program evaluation studies from the literature. “Something blue” . . . well, we’re not sure about that. Those who have used the Fourth Edition might be interested in knowing what has changed in this new edition. Based on reviewers’ comments we have: • Created a new chapter to explain sampling. • Incorporated new material on designing questionnaires. • Overhauled the chapter on qualitative evaluation. It is now “Qualitative and Mixed Methods in Evaluation.” • Reworked the “Formative and Process Evaluation” chapter with expanded coverage on developing logic models. • Added new studies and references; new Internet sources of information. • Included new examples of measurement instruments (scales) with a macro focus. • Inserted new checklists and guides (such as ways to minimize and monitor for potential fidelity problems—Chapter 13). • Revised the chapter “Writing Evaluation Proposals, Reports, and Journal Articles” to give it less of an academic slant. There’s new material on writing xi
- 17. executive summaries and considerations in planning and writing evaluation reports for agencies. • Deleted the chapter on Goal Attainment Scaling. Sometimes it is hard to know how much to “redo” and how much to leave alone. There are always new ideas and recent studies that ought to be highlighted as exemplars. So, some content must change to keep a text current. However, edu- cators choose books that they have used in the past because they know and like that content. If a book changes too much, then it may not always have the same comfortable fit as when it was previously chosen. It is somewhat of a challenge, then, to know what to keep and what to toss. And that is why, dear reader, you are invited to let us know what works and what doesn’t. If there is some aspect of program evaluation that we have missed or haven’t explained very well, then send an e-mail and tell us what you think. For those of you who are just discovering this book, Program Evaluation is designed to be a primary textbook for graduate students in social work, sociology, psychology, public administration, counseling, education, nursing, and other re- lated disciplines. It may also be used as a supplemental textbook in research meth- odology classes. We hope that we have written clearly enough that practitioners, administrators, and other persons who have just acquired the responsibility for overseeing or evaluating specific programs may find it to be a valuable resource. Our focus is on communicating the essentials, that is, the basic tools and knowledge necessary to conceptualize a program evaluation and carry out the tasks associated with examining and appraising program performance. This book is designed to help students and practitioners understand and contribute to evidence- based practice. It is vitally important that we professionals and professionals- in-training continually examine the effectiveness of our programs and their impact upon our clients. This is the way improvement comes about—with benefits for both clients and society. We hope in some small way that our book is able to pre- pare and assist you for this important responsibility. David Royse (droyse@uky.edu) Bruce Thyer (bthyer@fsu.edu) Deborah Padgett (deborah.padgett@nyu.edu) xii preface
- 18. CHAPTER ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± 1 Introduction THE IMPORTANCE OF PROGRAM EVALUATION Welcome to the field of program evaluation, that aspect of professional training aimed at helping you to integrate research and practice skills, using the former to enhance the latter. We recognize that relatively few practitioners in the human ser- vices of social work, psychology, counseling, education, public administration, and nursing will have careers devoted to scientific research. However, everyone in those fields will be concerned with providing services to clients at some level (individuals, families, couples, small groups, organizations, or communities), and every human services practitioner will be intensely interested in learning whether the services pro- vided really do help the clients they are intended to serve. Thus, even if research may not be your cup of tea, evaluating services and programs using scientifically credible research tools is a professional skill that you will find valuable. And apart from possibly being involved in the design and conduct of program evaluations yourself, you will read and critically analyze evaluation studies conducted by others and published in disciplinary journals and books. Your ability to judge the value of published and unpublished program evaluations is another valuable skill to en- hance. Otherwise, how will you be able to decide what types of services are worth providing to clients? In the human services field, scientific research can be broadly classified into having three main (often interrelated) purposes: to objectively describe things; to empirically evaluate the effectiveness of services; or to validly explain things. Descriptive research can be undertaken to better understand the characteristics or needs of clients of a particular agency. Evaluative research helps determine whether these needs are being met or clients’ goals attained, while explanatory studies aim at uncovering the causes of psychosocial problems or the processes by which 1
- 19. interventions work (thus contributing to what is known respectively as etiological or interventive theory). Program evaluation can be seen as a subset of those activities labeled research, which itself has been simply defined as “systematic procedures used in seeking facts or principles” (Barker, 2003, p. 368). Evaluation research refers to “system- atic investigation to determine the success of a specific program” (Barker, 2003, p. 149). Program evaluation is a practical endeavor, not an academic exercise, and is not primarily an attempt to build theory or necessarily to develop social sci- ence knowledge (although it is wonderful when that happens). Tripodi (1987) has noted, in the Encyclopedia of Social Work, that “the mission of program evalua- tion in social work is to provide information that can be used to improve social programs” (p. 366). Curiously, although for many decades the singularly crucial importance of hu- man services professionals designing and conducting program evaluations has been widely recognized, program evaluation research remains rarely undertaken. Rosen, Proctor, and Staudt (1999) reviewed all articles (N = 1,849) published in 13 major social work journals from 1993 to mid-1997. Of these, only 863 (47 percent) were research articles. Of these 863 research articles, 314 were descriptive studies, 423 were explanatory ones, and only 126 involved evaluation (e.g., outcome) studies of some type. Overall, program evaluation articles with replicable interventions re- presented only 3 percent of all published social work articles! Consider this fact in light of the quotations contained in Box 1.1. Currently then, we have the very odd situation that although many authorities consider program evaluation to be perhaps the most valuable type of research con- tribution one can make, such studies seem to be very rarely undertaken and pub- lished. Instead, most empirical studies have a focus on descriptive and explanatory research work, which, at best, may have potential implications for practice in the human services, not the potential for direct applications that a well-crafted outcome study would possess. Why is it necessary to evaluate established services? Because there are always alternative, and sometimes better, ways to solve problems. For instance, consider an article entitled “Time-Limited Therapy in University Counseling Centers: Do Time-Limited and Time-Unlimited Centers Differ?” (Gyorky, Royalty, & Johnson, 1994). The authors found that counseling centers with time limits had longer wait- ing lists and served a smaller percentage of the student body than did centers with no limits on counseling duration. Now, if you were in charge of a university counseling center, would you want to read this article? Would it be important to understand the authors’ methodology and sample? As another example, take a study authored by several behavioral scientists and social workers (Lynam et al., 1999). These authors evaluated a widely used drug abuse prevention program (known as Project DARE) by following up (10 years later) with over 1,000 fifth graders who had received either Project DARE (in 23 different schools) or a less structured drug education program (provided in 8 differ- ent schools). The schools were randomly assigned to provide DARE (17 one-hour sessions over 17 weeks, taught by police officers) or health education training (weekly 30 to 45 minute sessions over 2 to 4 weeks, taught by health educators). Ten years after training, there were no differences, as assessed by marijuana use, 2 chapter 1
- 20. variety of illicit drugs used, or self-esteem, between the kids who received Project DARE versus the comparison health education training. Suppose you were a school counselor and assigned the task of choosing a drug abuse prevention program for your school to provide. Given the much higher costs of the DARE program, relative to the less expensive (in terms of time and money) health education training; the failure by Lynam et al. (1999) to find a superior long- term effect of DARE; and prior evaluation studies indicating that DARE has no long-term effect on drug use (e.g., Dukes, Ullman, & Stein, 1996), would you try to obtain Project DARE for your school? Common sense suggests no, until such time as DARE develops a stronger empirical research base. But should you make a decision without looking at the studies yourself? Probably not. Suppose you are asked to help provide mental health counseling services to women who experience post-traumatic stress disorder (PTSD) following a sexual assault. What should you do to help these clients? An evaluation study of therapy BOX 1.1 Opinions on the Importance of Evaluation Research ± ± ± ± ± ± ± ± ± ± ± ± I appeal to you…. Measure, evaluate, estimate, appraise your results, in some form, in any terms that rest upon something beyond faith, assertion, and “illustrative case.” State your objectives and how far you have reached them…. Out of such evaluations will come, I believe, better service to the client. (Cabot, 1931) The third type of research, evaluative studies of welfare programs and the activities of practitioners, are the most important of all. (Angell, 1954, p. 169) [S]ocial work is not a science whose aim is to derive knowledge; it is a technology whose aim is to apply knowledge for the purpose of control. Therefore, on the re- search continuum social work research falls nearer to the applied end, because of its purpose of practical knowledge. (Greenwood, 1957, p. 315) Evaluation and client feedback are not only necessary for effective service delivery, but are an ethical requirement of the profession. Systematic methods must be developed to assess whether social workers are helping, harming, or doing nothing for the people they serve. (Rosenberg & Brody, 1974, p. 349) Social work has no more important use of research methods than assessment of the consequences of practice and policy choices…. [S]mall scale, agency-based studies are worthwhile if they succeed in placing interest in effectiveness at the center of agency practice and when they create a critical alliance between practitioners and researchers. (Mullen, 1995, pp. 282–283) Studies are needed on the effectiveness of psychosocial intervention, including inter- ventions previously tested under ideal controlled conditions, in real-world health care systems. (Ell, 1996, p. 589) Research on actual service interventions is the critical element in connecting research to the knowledge base used by professional practitioners…. [T]he issue now is one of developing investigations of social work intervention initiatives, studies that go be- yond descriptions and explanatory research. (Austin, 1998, pp. 17, 43) We need to establish a research agenda for social work…. And intervention studies must be high in priority to such an agenda. (Rosen, Proctor, & Staudt, 1999, p. 9). introduction 3
- 21. provided by social workers and psychologists conducted by Foa, Rothbaum, Riggs, and Murdock (1992) would be relevant in helping you decide what services to offer. Forty-five female rape victims who suffered from PTSD as a consequence of being sexually assaulted were randomly assigned to receive cognitive-behavioral therapy, supportive counseling, or to a wait-list condition. At three and one-half months after treatment, the cognitive-behavioral therapies were considerably more beneficial in helping the women overcome PTSD symptoms than supportive counseling or the passage of time alone (wait-list condition). Moreover, a second study with a much larger sample size (N = 96) recently replicated the finding that cognitive-behavioral therapy was superior to no treatment (Foa et al., 1999). Evaluation studies such as these should have a very important bearing on your decision about choosing therapeutic services for clients. Empirical research is cer- tainly not the only consideration in adopting treatment models, but we do view it as an essential one. Here is one other compelling study (Weiss, Catron, Harris, & Phung, 1999). One hundred and sixty children referred for mental health treatment were randomly assigned to conventional child psychotherapy as provided in commu- nity mental health clinics by clinical social workers, psychologists, or nurses or to ac- ademic tutoring by graduate students or elementary school teachers. Psychotherapy lasted an average of 60 sessions, while the tutoring lasted about 53 sessions, each over about 2 years. A number of reliable and valid outcome measures assessed psy- chopathology, adaptive functioning, and behavioral problems. The children receiving professional psychotherapy significantly improved over the 2-year period. This is a desirable outcome, of course, but so did the kids receiving academic tutoring, at a much lower cost intervention. In effect, the authors concluded that traditional child psychotherapy had no positive effects beyond those obtainable by nonprofessional academic tutoring not focused on mental health issues. Now, suppose you are em- ployed as an administrator at a local mental health clinic, and you are asked to choose interventions for children referred to your program. You can spend a lot of money on licensed mental health providers, or less money on graduate student tutors, and expect similar results. What will you do? Well, it would take a pretty confident administrator to announce the abandonment of agency efforts to conduct psychother- apy for children (illustrating the potential influence of nonscientific issues into the decision-making process). And, in fact, a well-informed administrator might wait to see if the findings are replicated by other studies—that they were not just a fluke. Nonetheless, studies like this one should serve as a stimulus to search the men- tal health literature very carefully for effective alternatives to traditional child psy- chotherapy (e.g., Christophersen & Mortweet, 2001; Hibbs & Jensen, 2005) or perhaps to see whether other more positive studies better support the use of tradi- tional interventions. Again, the implications are clear. There are some examples of evaluations with positive results that can be used to develop effective policies and programs by human service practitioners seeking guid- ance as to what types of interventions work best. For example, a group of Scandinavian researchers completed a systematic review of high-quality evaluation studies examining the effects of welfare-to-work programs. Their 122-page analysis was grounded in a review of 46 programs involving over 412,000 participants worldwide, although most such studies were conducted in North America. These welfare-to-work programs “… results in a slight increase in the likelihood that the 4 chapter 1
- 22. unemployed person will find employment, that earnings will increase and that welfare payments will be reduced. They also reduce the likelihood that the unemployed person will remain on long-term welfare” (see http://www.sfi.dk/sw45769.aspn, downloaded on 28 August 2008). If you are working in the field of helping the poor find jobs, you would be well-advised to obtain this free systematic review and to become familiar with those approaches shown to be more effective than others. Another systematic review on the Campbell Collaboration website found that cogni- tive behavioral treatment is helpful to children suffering from post-traumatic stress disorder. Another review supported the positive effects of after-school parent involve- ment at improving children’s academic performance. The websites of the Campbell Collaboration (www.campbellcollaboration.org) and the Cochrane Collaborations (www.cochrane.org) contain numerous sytematic reviews which represent some of the most credible research projects ever undertaken in a wide array of domains in the human services, including social welfare, criminal justice, education, health, and medicine. These are exceptional resources for program evaluators to be familiar with. We believe that previously published and credible evaluation studies should be a major source of input into the design of an agency’s programs. Our hope is that through reading this book, you will acquire greater skills in locating and critically evaluating such studies, and in designing and conducting empirical evaluations of your own practice and of the outcomes of the agency where you may be employed. When we improve our programs and interventions by making them more effective and efficient, all those involved with or touched by the social service delivery system are affected. Consumers and their families may recover faster when we discover that one approach works better than another. Armed with information from the program evaluation, workers and managers can better treat and advocate for their clients—pos- sibly making their own jobs more enjoyable and less frustrating. Ultimately, even tax- payers benefit. But let us back up a bit and discuss what constitutes a program. WHAT IS A PROGRAM? A program is an organized collection of activities designed to reach certain objectives. Let’s consider the two main elements of this definition in depth. Organized activities— programs—are not a random set of actions but a series of planned actions designed to solve some problem. If there is no problem, then there is no need for programmatic intervention. So, programs are interventions or services that are expected to have some kind of an impact on the program participants. Could a bereavement support group for school-aged children be considered a program? What about a telephone hot- line for parents? Would it be stretching things too much to describe a residential drug treatment facility’s efforts aimed at reducing client attrition as a program? CHARACTERISTICS OF “GOOD” PROGRAMS Programs tend to have certain characteristics that help us identify them. First of all, programs tend to require staffing. A residential drug treatment facility, for instance, is going to need a lot of staff. It may even have a separate staff who run an after- care or outpatient drug treatment program. The personnel of both programs may occasionally be asked to speak to high school students and groups in the introduction 5
- 23. community as part of the facility’s drug education program. Staff may have their time allocated among several programs or dedicated to only one. Second, programs usually have their own budgets. Because employing staff re- quires financial resources, programs sometimes can be identified by their budgets. However, some fine programs have minimal budgets because of heavy reliance on volunteers. Stable funding is important to the success of most programs. Morale and performance fall when employees do not get paid on a regular basis, or when they are asked to put aside normal duties and engage in last minute fund- raising or grant writing to get the program through several more months. Programs started with “soft money” (grants or nonrecurring funds) often experi- ence high rates of staff turnover until the programs secure some continuity in funding. Another characteristic of programs is that they have their own identity. In short, they are visible or recognizable by the public. Big Brothers/Big Sisters is an example of an organization with a national reputation for a single program. In some communities, a program may be recognized by the location where it has been housed for a number of years, or by its unique slogan, sign, letterhead, spokes- person, or public service announcements. When an organization has multiple programs, differences are sometimes found in philosophy, policies or procedures, and mission, and perhaps even in the way their corresponding staffs dress and how they account for their time. Such contrasts make it easy to differentiate one program from another. Within an agency, one outpatient counseling program may have the service phi- losophy that “no one is turned away,” while another outpatient counseling pro- gram may have a different philosophy—providing service only for those who meet certain eligibility guidelines, such as having private insurance or being able to afford to pay. A service philosophy may also clearly communicate how the clientele is to be treated, for example, “We respect the dignity and worth of all those we serve in caring for their physical, spiritual, psychological, and social well-being” or “The customer is always right.” Unfortunately for program evaluators, programs can be vague and hard to dis- tinguish and define. A former governor once made a public announcement that he was unveiling “a new program” to put state social workers in public schools. The program, he said, should help prevent dropouts and poor achievement among stu- dents who faced serious personal and family problems. However, the newspaper ac- count said the program would require no additional staff or funding. In essence, some social services employees would be placed in schools that could supply them with office space and phones. Did the governor’s announcement create a program? Not in this instance. It never got off the ground. Why not? It had no name, no staff, no funding, no slogan, no visibility. Most schools did not have surplus office space. Further, the governor made no suggestion of any new activities or ways of tackling the problems children and their families faced. On the other hand, starting a bereavement support group in an elementary school, even if volunteers contribute the leadership and the group has no budget to speak of, could be considered a program if it has an ongoing presence and a pre- sumed impact that could be measured. For evaluation purposes, speaking to an 6 chapter 1
- 24. assembly of high school students once or twice a year about drugs and alcoholism might also be considered a program. In the best of all possible worlds, every human services program would be solidly established on the basis of scientifically credible evidence that had been pre- viously published in peer-reviewed professional journals. That is, before the practi- tioners jumped into a social problem and started “helping,” someone did serious review and appraisal of the relevant evaluation studies that tested the usefulness of various methods of potentially helping. If a careful search of the literature and critical review of the existing outcome studies found that one or more models of intervention had credible evidence of effectiveness, and these approaches were “teachable” to the existing service providers, cost effective, and ethical, contempo- rary standards of ethical practice would suggest that the service program be focused around these empirically supported services as opposed to interventions lacking a sufficient foundation in empirical research. Now, this poses a dilemma for practitioners and administrators, namely, “What if no empirically-based services are known to exist for a particular problem?” In that case, one would be justified in primarily relying on the more traditional sources of practice knowledge, namely, theory, practice wisdom, common sense, tradition, and authority. But (and this is a big but), practitioners should only claim that no evidence-based interventions exist after having made a thorough and up-to-date search of the relevant practice research literature. Fortunately, evidence-based inter- ventions are now well established for a majority of the serious conditions described in the Diagnostic and Statistical Manual of Mental Disorders (DSM; American Psychiatric Association, 2000), and increasingly for conditions that do not lend themselves to the DSM system—problems such as unemployment, domestic violence, child abuse and neglect, and troubled youth. We believe that every human services agency should keep abreast of these developments by subscribing to relevant journals (e.g., Journal of Consulting and Clinical Psychology, Archives of General Psychiatry, Research on Social Work Practice) and acquiring the latest edition of professional books that summarize the latest studies on evidence-based practice (e.g., Brownson, Baker, Leet, & Gillespie, 2003; Drake, Herrens, & Lynde, 2005; Hibbs & Jensen, 2005; LeCroy, 2008; Nathan & Gorman, 2007; Roth & Fonagy, 2005; Seligman & Reichenberg, 2007; Thyer & Wodarski, 2007). The question may legitimately arise, “How much evidence is enough evidence in order for a given intervention to be considered to have an adequate empirical foundation?” The Task Force on Promotion and Dissemination of Psychological Procedures (Chambless et al., 1996, p. 16) of Division 12 (Clinical Psychology) of the American Psychological Association has promulgated one set of minimal recom- mendations. In order for a psychosocial intervention to be considered for inclusion in their list of empirically supported treatments, it had to meet the following criteria: The Treatment Must Be Supported by 1. At least two good between-group design experiments demonstrating efficacy in one or more of the following ways: a. Superior to pill or psychological placebo or to another treatment b. Equivalent to an already established treatment in experiments with adequate statistical power, or introduction 7
- 25. 2. A large series of single case designs (N > 9) demonstrating efficacy. These experiments must have: a. Used good experimental designs, and b. Compared the intervention to another treatment, as in 1a Among the other criteria to be applied are that experiments must be conducted using treatment manuals (this enhances the ability to replicate interventions), the characteristics of the clients must be clearly specified, and effects must have been demonstrated by at least two different investigators or treatment teams. Although these standards may seem a bit stringent to students and practitioners unaccus- tomed to rigorous research, they are not unwarranted; and they serve as an initial starting place to begin classifying particular interventions as evidence-based or not. Human services professionals can employ this set of standards when selecting types of treatment programs to provide. Over time, it is likely that these standards will be augmented by additional criteria (e.g., “The treatment has been evaluated in real- life clinical settings”) that will enhance their usefulness. The human services professions are slowly moving in the direction of evidence- based practice, which has been defined as “the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individuals” (Sackett, Richardson, Rosenberg, & Haynes, 1997, p. 2). Further, it is “the integra- tion of best research evidence with clinical expertise and patient values” (Sackett, Strauss, Richardson, Rosenberg, & Haynes, 2000, p. 1). Gambrill (1999) concurs: “It involves integrating individual practice expertise with the best available external evidence from systematic research as well as considering the values and expectations of clients” (p. 346). Although this statement may seem like common sense, the fact is that at present no clear ethical or legal mandates require that human services profes- sionals deliver evidence-based practices, where evidence-based practices are known to be established. This could change, however, in the very near future. Clearly, it is the direction in which the human service professions are heading (see Norcross, Beutler, & Levant, 2006; Norcross, Hogan, & Koocher, 2008; Straus, Richardson, Glasziou, & Haynes, 2005; Thyer, 2004, 2008b; Thyer & Wodarski, 2007). The Role of Theory in Program Evaluation In the best of all possible worlds, every program would also be based on a sound the- oretical model. That is, before the helpers jumped into a social problem and started helping, they would develop a model that would have examined the problem—how and why it originated and what would work best to remedy the situation. A psycho- social or other theory can be an organizing principle for each program, one that pro- vides a consistency of effort by suggesting a standard approach derived from some well-articulated and comprehensive social or behavioral science theory. Such a theory may serve as a guide in conceptualizing the causes of problems and in proposed mechanisms of action for interventions. Take the problem of hyperactive behavior (HB). If one approaches the under- standing and treatment of children with HB from a biological orientation, one would focus on possible underlying disturbances in brain chemistry to explain HB, and on the use of medications to regulate these chemical irregularities. If one sub- scribed to a learning theory orientation, one might examine the possible role of an 8 chapter 1
- 26. overstimulating environment, or of peer/parental/teacher inadvertent reinforcement for HB, as tentative causes of HB, and of environmental manipulation and positive behavior management programs to reinforce on-task activities as an intervention. One trained in a biopsychosocial approach might employ both approaches. There are many theories on the causes of HB and theoretically based interventions. Another example might be that of drinking too much alcohol (often called alco- holism). There are also a wide array of theories as to the etiology of abusive drink- ing, and of ways to help people drink less or to abstain from drinking. According to the theory of Alcoholics Anonymous (AA), alcoholism is caused by a biologically based allergic reaction to ethanol, in that the smallest sip triggers an irresistable craving to drink more, leading to a loss of control. This biological theory leads, nat- urally, to an interventive theory, namely to abstain completely from exposure to alcohol. Just as a person with a peanut allergy is told to avoid all consumption of peanuts, the alcoholic is provided a program of education, interpersonal support, and spiritual development intended to reinforce their attempts to abstain completely. No one is ever said to “recover” from alcoholism, anymore than one is likely to cease having a peanut allergy, according to the AA theory—people are simply in var- ious stages of “recovery.” AA is a program of recovery clearly derived from both an etiological and interventive theory, in this case a biological one. However, there are certainly other theories of abusive drinking based on learning theory, genetic, psychodynamic, and familial influences, and so forth. Each theory has its own deriv- ative programs (e.g., antabuse medication to produce nausea upon drinking, anti- craving drugs, behavior modification, the community-reinforcement approach to abstinence, psychotherapy, rational recovery, family therapy, meditation, etc.). Consider a different problem. Suppose you are hired to run a treatment program for men who batter. Do these men fit a single profile? Is one interventive strategy all that is needed? Saunders (1992) argues that there are three theoretically distinct types of men who batter: those who were severely abused as children; emotionally volatile men with rigid sex-role attitudes, who fear losing their partners and are de- pressed, suicidal, and angry; and family-only aggressors who tend to have relatively liberal attitudes about sex roles, the lowest rate of abuse in childhood, the most mar- ital satisfaction, and who are generally nonassertive. Is there a possibility that some interventions may work better with one type of abuser than with another? Theoretical models can be important to understanding how a program should work and where one should look for indications that a program is successful (see Box 1.2). But all too often what passes for theory in many social service agencies is a blend of past experience and tradition. Evaluators would have no problem with that if the program was often successful in rehabilitating, helping, fixing, or curing clients. But when a program is not successful much of the time, the possibil- ity exists that even though the program was implemented as designed, the underlying theory is flawed. Such a situation calls for new thinking and a new approach to the problem. Theories may be able to tell us how to accomplish our goals (Conrad & Miller, 1987), but not in every case. Program evaluation studies should generally be considered as relatively poor tests of the validity of theories. Take, for example, a program with a theoretically derived intervention that is found to produce positive results with the clients—they appear to be really benefitting from the program’s services. You might be tempted introduction 9
- 27. to conclude that this means that the program’s underlying theory is thus proven to be valid. Doing so would be making a mistake. An intervention may work quite well for reasons completely unrelated to its underlying theoretical rationale. For example, the early (circa 1950s) behavioral psychotherapy called systematic desensitization (SD) was said to work because of an underlying theoretical process called reciprocal inhi- bition (RI). Careful tests in the late 1960s and 1970s found out pretty conclusively that while, yes, SD was moderately helpful in overcoming certain anxiety disorders it did not work because of any processes of RI. More recently the psychotherapy called eye movement desensitization and reprocessing (EMDR, circa 1990s) was said to work because of underlying physiological changes in the brain created by ask- ing the client to move their eyes back and forth in a certain pattern established by the therapist, while thinking of traumatic events. It turns out that these so-called saccadic eye movements are completely irrelevant to the modest improvements garnered by EMDR, which are more plausibly attributed to mild exposure to anxiety-evoking im- agery and to placebo factors (see Lohr, Hooke, Gist, & Tolin, 2003). Generally speaking, the theoretical propositions that therapies and programs are based on require evidence of their validity independent of their successful outcomes. Negative outcomes also are problematic in terms of making inferences about the validity of a program’s underlying theory. Say you conduct a program evalua- tion of a theoretically based service and the outcome is negative, for example, the clients do not improve, or, heaven forbid, get worse. Doesn’t this “prove” that the theory the program is based on is incorrect? Not so fast. Advocates of the theory may claim that the outcomes were negative because the treatment or service was not really implemented correctly, or that the therapists were insufficiently trained or supervised, the duration of treatment was too brief, the outcome measures were insufficiently sensitive to detect the delicate but far-reaching beneficial effects of therapy, that the follow-up period was not long enough, etc. There can be a myriad of reasons (some genuine and some spurious) why a program may not work, and invalidity of underlying theory is but one. We do not encourage program evaluators to make inferences about the truth or falsity of theories based on the program’s out- comes. In very rare cases, a really stringently designed and executed outcome study with negative results could be seen as reflective of the underlying theory’s invalidity, but it is much less credible to claim that a positive outcome means that the theory is correct. Programs may work for many reasons apart from those hypothesized by the program’s developers or conceptual grandparents. To a certain extent, program designers and implementers are protected from many of the problems that accompany misguided and erroneous theories when BOX 1.2 Characteristics of “Good” Social Service Programs ± ± ± ± ± ± ± ± ± ± ± ± • Staffing • Conceptual or theoretical foundation • Budgets • A service philosophy • Stable funding • Systematic efforts at empirical evaluation of services • Recognized identity • Evidence-based research foundation 10 chapter 1
- 28. they base programs not solely on theories alone but also on methods of practice that have been empirically supported by research and evaluation. The best pro- grams, those on the “cutting edge,” will be built on a firm foundation of the latest empirical research. These programs can provide a level of confidence and knowl- edge about what to expect in terms of success rates. These would be the programs we would choose for ourselves if we or our families needed intervention. Theory and programs are linked in complex ways that we will discuss at vari- ous times throughout the book. In this chapter, we will consider the ways theory and empirical research can influence and shape a program. If you want to see how evaluators and researchers present an intervention in terms of a theoretical model, review the study by Telch, Agras, Rossiter, Wilfley, and Kenardy (1990) that examines the use of group cognitive-behavioral treatment for nonpurging bulimia. Similarly, Jemmott and Jemmott (1991) have applied the theory of reasoned action in order to increase the use of condoms and prevent the spread of AIDS. Zambelli and Derosa (1992) have looked at specific group inter- vention techniques for bereaved school-aged children that were theoretically based and derived from four protective mechanisms identified by Rutter (1987). You should realize, however, that many (perhaps most) human services pro- grams are not based on any explicit theory of human behavior or any etiological social or behavioral social science theory explaining how particular problems arise, or even any particular interventive theory. Such “atheoretical” programs may be based on common sense, authority, or tradition. For example, we know of one gov- ernor who arranged for the state to provide the mother of every newborn baby a compact disc (CD) of classical music. These CDs were duly distributed to thousands of new mothers, costing the state hundreds of thousands of dollars in taxpayer money. This authoritative initiative was instigated by the governor’s reading about something supposedly called the “Mozart effect,” wherein listening to a particular Mozart sonata resulted in college students earning higher test scores. The (vain) hope was that mothers who were given CDs of classical music (which failed to con- tain a recording of the particular Mozart sonata in question!) would play these within hearing of their infants, who would grow up to be smarter. This human ser- vices program was not based on any etiological theory about intelligence, or even an interventive theory on how listening to classical music was supposed to improve intelligence. It was based on authority, the governor’s mandate. Unfortunately, sub- sequent studies failed to demonstrate the Mozart effect in test taking, and none demonstrated any improvement in infant intelligence. Numerous examples of such atheoretical programs can be found from policies removing the driver’s licenses of high school dropouts to policies within child pro- tective service programs, homeless shelters, soup kitchens, and so forth. Ask the practitioners providing such services about their program’s social or behavioral the- oretical orientation, and often you will be met with puzzled looks. To be sure, these programs can be prospectively designed in accord with some theory, but in many instances they are not. And it can be misleading to retroactively attempt to fit a the- oretical approach to explain a particular program, because the same program could no doubt be similarly explained by perhaps dozens of rival theories, leaving you no closer to a valid understanding of how the program may be exerting its effects, if any. The relationship between formal theory and research on practice outcomes is introduction 11
- 29. a complex one, and Thyer (1994, 2001) addresses it more completely than is possi- ble here. Not to be ignored are the possible negative effects of theory in human service programs, especially as they relate to programs and their evaluation (Thyer, 2008c). Many social programs are likely based on false theory, theories based on invalid etiological and therapeutic propositions. So-called boot camps for juvenile delinquents are one example. Here the theory (loosely used) is that youth who get into trouble with the law have some sort of defect in their “character.” Certain poli- ticians’ military experiences led them to believe that a Marine-style boot camp regi- men could somehow correct the character of misbehaving young men (and women), and then, upon their release from the boot camp, be restored to their home commu- nities having become somehow “stronger” in character and better prepared to resist the influence of evil peers. Well, the evidence is pretty clear that boot camps do not work well at preventing recidivism in youth (see systematic review on the topic avail- able at www.campbellcollaboration.org and MacKenzie, Bierie, & Mitchell, 2007). Those states which support boot camps for juveniles can be seen as providing a dis- service to kids, squandering taxpayers’ funds, and employing hundreds of staff en- gaged in delivering useless services, not to mention diverting youth from potentially more effective programs. The efforts expended by evaluators to appraise the out- comes of ineffective programs can also be seen as diverted from more useful activi- ties. These are all harmful effects of grounding social services in invalid theories. Programs vary greatly. Some are sophisticated and others simplistic—even com- posed of a single activity. In scale they range from a small cooperative babysitting program for young mothers to the federal food stamp program that touches mil- lions of lives. It is not always easy to determine whether certain activities should be considered a program or part of the collection of activities that comprise a larger single program. Either way, program evaluation can be undertaken as long as a de- sired objective or outcome can be stated. Although in some agencies programs are referred to as services, in this book the terms will be used interchangeably. PROGRAM EVALUATION DEFINED Program evaluation is applied research used as part of the managerial process. Evaluations are conducted to aid those who must make administrative decisions about human services programs. Unlike theoretical research, where scientists engage in science for its own sake, program evaluation systematically examines human ser- vices programs for pragmatic reasons. Decision makers may need to know if a pro- gram accomplished its objectives, if it is worth funding again next year, or if a less expensive program can accomplish the same results. We like the following defini- tion provided by Grinnell and Unrau (2008, p. 553): A form of appraisal, using valid and reliable research methods, that examines the processes or outcomes of an organization that exists to fulfill some social need. Program evaluation is like basic research in that both follow a logical, orderly sequence of investigation. Both begin with a problem, a question, or a hypothesis. Normally, there is some review of what is known about the problem, including prior efforts and theoretical approaches (this is known as reviewing the literature). 12 chapter 1
- 30. A research or evaluation design (a blueprint to guide the data collection efforts) is developed, and data are gathered and then analyzed. When thought of this way, both research and evaluation are similar to the task-centered or problem-solving process known to many human services professionals. Research and evaluation differ with regard to the expected use or utility of the data. There may be no anticipated need or demand for “pure” research, whereas an assemblage of individuals may anxiously await the results of a program evaluation. Also, the goal of research is to produce generalizable knowledge, while information from a program evaluation may be applicable to only a specific program. However, both are approached with some degree of rigor. Think of program evaluation as a tool—a management tool that you can use to make (and to help others make) bet- ter decisions about social and human services programs. Program evaluation helps us to make the best use of our resources as we labor to improve the quality of life of our clients. Program evaluation involves making comparisons. In fact, Schalock and Thornton (1988) have defined program evaluation as “structured comparison.” Few programs can be evaluated without comparing them to something. Programs in one agency may be compared to similar programs in other agencies, to past or prior efforts, or against a stated objective; but without some form of comparison, there can be no evaluation. A major thrust of this book is to help you find or create (to conceptualize) bases of comparison for your own program evaluation efforts. REASONS WHY PROGRAMS ARE EVALUATED Quite often social and human services programs are evaluated because of a need to be accountable to a sponsoring or funding agency, or because competition for scarce funds requires that only one program (normally, the most effective or efficient pro- gram) can be funded. Program evaluation is needed whenever new interventions are being tried and it is not known whether they will be as successful as former methods, or when there is a perception that a program could be improved—that it could be- come more productive or better in some way. We evaluate on those occasions when it is important to have some objective assessment or feedback about the worth of our social and human services programs. The following scenarios illustrate some of the occasions when program evaluations are encountered. Scenario 1: The Required Evaluation Your agency is applying for funding from the United Way in your community to begin a new program designed to pro- vide counseling to men who have been prosecuted for domestic violence. You have been asked to prepare the program proposal. As you read the instructions for pre- paring the proposal, you notice that besides describing the project, listing its objec- tives, pointing out its uniqueness, and stating the amount of funding that will be required, the proposal also requires a project evaluation. At the end of the project year, data must be presented to show that the project had a successful outcome and an impact on the problem of domestic violence. Scenario 2: Competition for Scarce Funds Your innovative program for men who batter has been operating for a year. You have been able to obtain introduction 13
- 31. some data that you hope will favorably influence the committee deciding the con- tinuation of funding for your program. As you prepare your presentation, you dis- cover that a second domestic violence project from another agency will also be making a request to be funded. You further learn that there is only enough money to fund one program. Scenario 3: Evaluation of New Interventions Many more clients desire the services of your outpatient counseling agency than you have staff to serve. At a planning session, one of the newer staff members suggests that the agency move from a one-on-one counseling model to a group services model. The benefits are clear—instead of limiting each practitioner to seven or eight scheduled clients a day, each therapist could conduct three or four group sessions a day and have con- tact with 25 to 30 clients. In spite of being able to serve more clients, the staff is not very supportive of this proposal, because they believe that individual counseling is much more effective than group counseling. Scenario 4: Evaluation for Accountability You work in a large residential agency serving young children. Unfortunately, a child care aide was recently discov- ered molesting one of the children. The public is in an uproar. Community leaders are calling for the agency director and all key staff to resign. You feel that the agency is a good one—better than other residential programs within the commu- nity. Because the agency director knows that you are enrolled in a program evalua- tion course at the nearby university, she calls you into her office and asks you to find some way of objectively documenting the strengths of the agency. “Can you show,” she asks, “that the great majority of our young people have a favorable ex- perience here, a good impression of the agency, and that they go on to do well in school and in life after they leave the agency?” MOTIVATIONS FOR PROGRAM EVALUATION Why do we evaluate human services programs? Programs are evaluated basically because administrative decisions have to be made, and it is important to know (or to show) that our programs are “good” programs. Individual policy or deci- sion makers may have a hypothesis about a program (e.g., the free clinic’s counsel- ing program is highly effective). At other times, questions may be raised (e.g., is the free clinic’s counseling program effective?). Hypotheses or questions provide the motivation for a program evaluation. It makes no real difference whether a ques- tion or a hypothesis serves as the catalyst for an evaluation. This can be seen in Box 1.3. The list in Box 1.3 could easily be made much longer. An interest in exploring one question may lead to other areas. The evaluator may start off wanting to know whether clients were being helped, but in the process of designing a methodology the initial question or problem becomes modified somewhat. The evaluator may want to know not only whether clients were helped but also whether one approach was cheaper (more cost-effective) than another. Other questions may concern whether improvement has been made in a certain staff’s productivity since last year or whether the program has reached its intended target population. On some 14 chapter 1
- 32. occasions, administrators may want to use evaluation data to help garner public support for human services programs. (The public is much more likely to support tax increases for those programs perceived to be “good” than those thought to be ineffective or poorly run.) Program evaluation can also be used in terms of market- ing programs to the public. (As a program manager or agency director, having data showing that 92 percent of your clientele say that they would refer their friends or family members to your agency could be very useful information to have on hand.) Social and human services programs have evolved to combat such social prob- lems as drug abuse. Think for a moment of other social problems in this country. We could begin listing such problems as: Poverty Substance abuse Homelessness Adolescent pregnancies Unemployment Mental illness Child abuse Illiteracy Domestic violence High infant mortality rates Crime Hunger AIDS BOX 1.3 Motivations for Program Evaluation ± ± ± ± ± ± ± ± ± ± ± ± We want to show: We want to know: 1. That clients are being helped. Are clients being helped? 2. That clients are satisfied with our services. Are clients satisfied with the services received? 3. That the program has an impact on some social problem. Has the program made any real difference? 4. That a program has worth. Does the program deserve the amount of money spent on it? 5. That one program or approach is better than another. Is the new intervention better than the old? 6. That the program needs additional staff or resources. How do we improve this program? 7. That staff are well utilized. Do staff make efficient use of their time? introduction 15
- 33. For each social problem, there are hundreds if not thousands of programs. Some of these programs work and need to be continued; others are ineffective. If it cannot be demonstrated that certain programs have any impact on these problems, then further evaluative research should be undertaken to discover why the programs were not successful. There may be very logical reasons; for example, the programs could be poorly managed, underfunded, or poorly conceptualized or designed. There are many other reasons. As human services professionals, we need to be just as interested in the outcomes of national programs as we are in our local programs. Program evaluation is not to be understood as having application only to the agency that employs us. Although the examples used thus far have helped us to understand the need for program evaluation primarily at the local level, an immense need remains for pro- gram evaluation of national expenditures and programs. For instance, an article in Brandweek (April 1998) entitled “Drug Money” has noted that the Partnership for a Drug-Free America and the White House Office of National Drug Control Policy have embarked on an antidrug campaign costing almost $2 billion. The author of the article, Daniel Hill, says that this enormous expenditure of money is backed only by “flimsy” research. Two of the three studies supposedly showing the effec- tiveness of media antidrug messages had yet to be published, and the author of the third acknowledged that her respondents might have been saying what they thought the researchers wanted to hear. Several experts have pointed out that no well-controlled studies show that me- dia campaigns are effective in changing behavior. The deputy director of the Office of National Drug Control Policy was, according to the Hill article, unable to cite any research supporting the contention that antidrug advertising works. Although there is no doubt that a serious drug abuse problem exists in this country, should billions of dollars be spent on untested interventions? Shouldn’t research support the effectiveness of interventions (even those perceived to be harmless) before vast sums of money are spent on them? The example of the antidrug advertising campaign is not an out-of-the ordinary one. According to an article in the New York Times, in the 1960s and 1970s, the federal government invested billions of dollars on job training “without a clue about what worked and what did not” (Passell, 1993). As a society we need to test new ideas to combat old problems. For example, do monetary incentives to mothers on welfare for using Norplant contraceptives significantly affect the number of children they have? Is offering full college scholarships to low-income students who remain in school, pass their courses, remain drug-free, and do not become pregnant or get in trouble with the law a realistic way to combat poverty? Evaluators, through carefully controlled studies, can determine whether spend- ing money “up front”—for example, paying low-income, pregnant women to attend prenatal education and care classes—saves money in the long run. There is some indication that giving pregnant Medicaid recipients a $10 bill for each ap- pointment they keep results in a considerable reduction in the amount of time new- born infants stay in intensive care units (Kolata, 1994). Whether at the local, state, or national level, program evaluation often begins by identifying a problem. Decision makers want to distinguish programs that work from those that do not and to know if their money is well spent. They may have developed questions about a program because of some incident or problems 16 chapter 1
- 34. brought to their attention. These problems can be visible and well-recognized or those known only to a handful of staff, administrators, or trustees. A problem is any undesirable situation or condition. Sometimes program evalua- tions are undertaken in order to determine the extent or magnitude of a problem or to confirm a suspected problem. As you think about the agency where you are working or interning, what problems come to mind? (If you do not initially think of any problems, have you seen any recent data suggesting that the program is effective or efficient?) There are probably as many reasons for conducting program evaluation as there are different programs. In addition to the reasons already given, those in the helping professions also conduct program evaluations because they have a responsibility to improve programs. For instance, the National Association of Social Workers’ Code of Ethics (1999) states, under ethical standard 5.02(a), “Social workers should monitor and evaluate policies, the implementation of programs, and practice interventions.” And social workers’ ethical responsibility to the profession is also seen in standard 5.02(b), which states: “Social workers should promote and facilitate evaluation and research to contribute to the development of knowledge.” We all have an ethical obligation to evaluate our practice (see Box 1.4). All too often, we get caught up in service delivery as measured by billable hours, home visits, numbers of phone calls, and internal audits of agency and accredita- tion forms without systematically appraising whether all this effort produces ben- eficial outcomes for clients. We have an ethical mandate to determine whether our clients are being helped, whether they are any better off as a result of our inter- ventions. Program evaluation is a major means by which we can fulfill this ethical responsibility. BOX 1.4 Selected Professional Association Guidelines on the Ethical Mandate to Evaluate Programs and Services ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± Social workers should monitor and evaluate policies, the implementation of programs, and practice interventions. Social workers should promote and facilitate evaluation and research to contribute to the development of knowledge. Social workers should … fully use evaluation and research evidence in their professional practice. (National Association of Social Workers, Code of Ethics, 1999, p. 20) All school social work programs, new or long-standing, should be evaluated on an on- going basis to determine their relevance, effectiveness, efficiency, and contributions to the process of educating children. (NASW, 1992, p. 16) Clinical social workers shall have … knowledge about and skills in using research to evaluate the effectiveness of a service. (NASW, 1989, p. 7) There are periodic, systematic, and effective evaluations of psychological services…. When the psychological service unit is a component of a larger organization, regular assessment of progress in achieving goals is provided in the service delivery plan. Such evaluation could include consideration of the effectiveness of psychological ser- vices…. (Board of Professional Affairs, 1987, p. 8) Monitor effectiveness. Counselors continually monitor their effectiveness as profes- sionals and take steps to improve when necessary. (American Counseling Association, Code of Ethics, 1999, p. 6) introduction 17
- 35. OVERCOMING THE SUBJECTIVE PERSPECTIVE Anytime we have a choice, we find ourselves in a position where a decision must be made between two or more alternatives. Often, informal (and perhaps even uncon- scious) criteria guide us in making choices. Although these criteria may be more the product of visceral reactions than of contemplation, they aid us in the making of choices. They help us to determine such things as “good” restaurants and “good” movies, and to rate the services of care providers (e.g., a “good” physician). In each of these instances, good is defined subjectively and somewhat arbitrarily. For exam- ple, my notion of the best restaurant in town may be one that specializes in Italian food. You, on the other hand, may intensely dislike Italian cooking. My notion of a good movie may be Texas Chainsaw Massacre, whereas your taste may run to less violence. My notion of a good physician may be one who, although known for a disheveled appearance, answers my every question, while your opinion of a good physician requires that the physician dress appropriately and look distinguished. Because appearance is important to you, you may have no confidence in a physician who does not look the role (whether or not your questions get answered). What does this have to do with program evaluation? Just this: every day (some- times many times a day) human services professionals must direct people to their programs or refer them to other programs based on their subjective impressions. When we make referrals, we want clients to go, not to programs that are not effec- tive, but to the “good” programs. We want them to have the best possible chance of succeeding or doing well in that program. We have a professional responsibility to avoid making referrals to ineffective or deficient programs. We also want the programs we direct or that employ us to benefit our clients. But, how do we recog- nize a good program, or a poor program? How do we know when our programs are effective? We like to believe that we help our clients, but what actual evidence do we have that the majority of our clients are helped by our programs? Most helping professionals have had clients who have made giant strides as a result of skilled intervention. We feel rewarded by these suc- cessful clients. They help us feel that we are competent and that we have chosen the right career. Unfortunately, there are also those clients with whom we are unsuccess- ful. These clients, despite our best efforts, drop out of programs, make a mess of their lives, or seem to have gained nothing from our interventions. Think of all the clients who have made their exits from your programs. What is the proportion of “successful” clients to “unsuccessful” clients? Are you more successful than unsuc- cessful with your clients? What evidence could you present of your success? We have raised these questions to help you understand that program evaluation involves a different perspective than you may normally employ when thinking about your clients. Clinicians and practitioners tend to evaluate their practice subjectively and in terms of selected individual cases. They think of Mrs. Smith (with whom they were successful), Mr. Arthur (who was a model client and who now comes back to volunteer his services), or perhaps Kathy M., with whom they were not a success. However, this “case focus” does not facilitate the aggregation of data at a program level so that an overall determination can be made about the effectiveness of the pro- gram as a whole. Although one bad apple may spoil an entire bushel, one client who does not succeed does not mean that a whole program needs to be overhauled. 18 chapter 1
- 36. The difficulties with attempting evaluation using a “case focus” with a single client can be demonstrated easily. Consider Mrs. Smith. Although you felt that you were successful in helping Mrs. Smith to quit drinking, others may not be so quick to shower accolades on you. Those who are skeptical of your abilities as a clinician may point out that while Mrs. Smith may no longer drink, the rest of her family is in turmoil. Her husband left home; a teenage daughter ran away. Mrs. Smith is now living with another recovering alcoholic and working for minimum wage as a waitress, although she was previously employed as a registered nurse. You reply to these critics, “She’s not drinking. She feels good about herself. I think she’s shown great improvement.” Although it may be possible to argue that any given case was or was not a success, a manager needs to look at the program as a whole. Are the clients (as an aggregate) better or worse off as a result of participating in the program? Consider the case of Mr. Arthur. Everyone in the agency agrees that he has made significant changes in his life since becoming a client of your program. However, on closer inspection, it is revealed that you spent twice as much time with Mr. Arthur as you did with the average client. Was he a success because he got twice as much attention? Would he have been a success if he had received only as much time as the “average client” receives? (Did he get so much time because he was an “easy” client to work with?) We have already admitted that the program was not successful with Kathy M. However, is Kathy the typical client or the unusual client? Perhaps Kathy was the most severely disturbed client that your program has ever admitted. Given her pre- vious history of multiple hospitalizations, perhaps no one really expected her to make any significant gains. We can see from these examples that our perspective as practitioners often in- volves subjective evaluations. That is, we believe that a client has improved or not improved. The problem with subjective evaluations is that others may not share them. While you think of Mrs. Smith as an example of a successful client, perhaps your best friend and coworker thinks of Mrs. Smith as something less than a suc- cess. Although you are quite pleased that Mr. Arthur has overcome a great many of his problems, perhaps your program director has sent you a strongly worded mem- orandum suggesting that the program’s waiting list is such that you are not to spend as much time with the rest of your clients. Although Kathy M. made no progress in treatment, the same program director is not disappointed. “We learned something,” she says. “We learned what won’t work with clients like this. Next time, we’ll try something a little different.” In conversation we can get away with saying things like “I did a good job with that family,” “She’s a good therapist,” or “It’s a good program, you’ll like it there.” However, a thesaurus lists nine different meanings for the word good as an adjec- tive (see Box 1.5). Seldom does anyone ask how we define good. What we are allowed to do as conversationalists we cannot do as program evaluators. Evaluators are concerned with specificity and measurements. We want verifi- able evidence, not someone’s opinion. It matters whether a program produces changes in behavior, attitudes, or knowledge. Further, we might want to know how much change was experienced by the average client, how long it was sus- tained, and at what cost. introduction 19
- 37. Subjective evaluations about the success of individual clients are very much like the initial examples of a “good” movie and a “good” restaurant. We can expect differences in opinion. Within most groups, if someone says, “That is not a good restaurant!” there are sure to be others within the crowd who will disagree. Someone else may say, “Well, it is my favorite restaurant!” or “That’s interesting. We were just there on Wednesday and had a wonderful meal.” The problem with subjective evaluations is that everyone is usually right. The person who had a bad ex- perience with a restaurant probably got poor service or an improperly prepared meal. The person who ate there on Wednesday could have just as easily not had a wonder- ful meal. The individual who boldly proclaimed the restaurant to be his favorite res- taurant might be quite willing to forget an occasional bad meal because he goes there for the atmosphere, he is personal friends with the proprietor, or his girlfriend works there. Another possibility is that he just does not have a discriminating palate. To become evaluators, we need to adjust our perspectives so that we are able to see beyond a single meal or a single client. We need to see the larger picture. We need to go from a micro focus to a macro focus. What are the experiences that most of the restaurant patrons or clients have? In a sense, we need to forget the in- dividual and broaden our perspective to focus on the most common or frequent ex- perience. What percent of the patrons would not return? With what percent of our caseload are we successful? We need to look for corroborative evidence that might convince neutral observers. (For instance, counting the number of patrons leaving meals unfinished or leaving in the middle of a movie might substantiate rather pow- erfully one’s own subjective experience.) As evaluators, we want to be able to objectively conclude that this program is a good one and that another is not—based not on personal opinion but on factual evidence. When we go beyond our own personal experience or opinions and collect information about the experiences that others have had, we have begun to develop an evaluative stance—we have moved from subjectivity to objectivity. BOX 1.5 Denotations of the Word GOOD ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± Usage Example 1. Pleasant, fine I had a good meal. 2. Moral, virtuous Mother Theresa was a good person. 3. Competent, skilled She is a good worker. 4. Useful, adequate It was good that I read the book before the quiz. 5. Reliable Pat is a good source of information. 6. Kind, giving My grandmother is so good. 7. Authentic, real He makes a good point. 8. Well-behaved Rachel is such a good child. 9. Considerable There is a good deal more poverty now than 5 years ago. 20 chapter 1
- 38. An objective stance tends to place faith in numbers and counting. As a rough rule of thumb, the more individuals we are able to interview, survey, or contact, the more confidence we can place in our evaluative findings. Numbers constitute objective data. When, for instance, 97 out of 100 clients indicate that they would recommend our services to their friends, this constitutes objective data. Anyone examining the responses of the 100 clients and sorting them into piles of “would recommend” and “would not recommend” services ought to arrive at the same conclusion. Evaluators are, in some respects, applied scientists. Scientists seek to understand and explain the world around them. However, it is not just explanations that scientists seek, but correct explanations. Whether we think of ourselves as program evaluators or as applied scientists, our findings must stand independently, apart from our claims or per- suasive oratory. Our findings should be replicable (reproducible); others must be able to independently arrive at the same conclusions. If someone did not like or agree with the findings from a particular program evaluation, then this person could repeat the evalua- tion using the same methodology. Assuming that no major changes occurred within the agency in the interim and that the original evaluation methodology was sound, findings from the second study should be the same or very similar to those of the first study. Objectivity demands precision. Evaluators must be precise about the program they are evaluating, what they will be measuring, how they will collect and analyze their data, and who they will be interviewing or observing during a given time pe- riod. Such matters require specificity. Vagueness is rarely tolerated in research or evaluation. Note the lack of specificity in the following: “This evaluation will deter- mine if specialized inservice training on the use of empathy helps nurses perform their jobs better.” Do you find it too vague? The statement is unclear because we are left wondering: What nurses are being discussed? Has it been established that empathy is necessary to perform their jobs? What jobs are under consideration? What does it mean to perform better? How is empathy to be measured? One way that evaluators become more specific and precise is by using operational definitions. An operational definition is the way a variable or concept (such as empathy) is to be defined and measured for the purposes of the evaluation. The evaluator may use a standardized scale to measure level of empathy. Or the evaluator may use some sort of behavioral measures, such as the number of times during a therapeutic session the counselor nods affirmatively or makes supportive statements such as “I understand.” Counselors may be operationally defined as those holding certain degrees or as all persons who work in a certain program regardless of their educational background. As one begins to operationally define the key concepts for a proposed evaluation or study, often the vagueness disappears. In the case of the vague statement, “This evalua- tion will determine if specialized inservice training on the use of empathy helps social workers perform their jobs better,” operationally defining important concepts might change it to: “Do social workers with higher levels of empathy place more children in adoptive homes per year than social workers with lower levels of empathy?” The effort to become more precise does not rule out the subjective experience in program evaluation. Although a single “bad” subjective experience cannot consti- tute a program evaluation, it may lead to a formal evaluation as a program man- ager, agency director, or members of the board of directors become concerned about whether an incident or experience reflects what is “really going on” with the program. The program evaluator seeks to understand the “reality” or “truth” introduction 21
- 39. about a program. In the process, the evaluator may collect a large number of sub- jective opinions about the program. Objective evaluations do not rely on the opin- ions of any one person (no matter how influential), but instead attempt to gain a comprehensive view from the opinions of the aggregate or group. Because the reality about a program’s performance can sometimes be painful and have far-reaching implications (e.g., loss of funds and the corresponding laying off of a number of an agency’s employees), program evaluators often seek the best possible objective evidence that they can obtain (given such pragmatic constraints as budget, time, access to clients or their records, and cooperation of the staff). Having objective or “hard data” to guide decisions about programs is superior to decision making without program evaluation data. By way of analogy, if you were on trial for an offense that you did not commit, you would want your lawyer to present as much objective evidence on your behalf as possible to assist the jury in realizing your innocence. You probably would not feel comfortable in allowing your attorney to hinge the entire case on the subjective testimony of a single charac- ter witness who would testify that you were a “good” student or a “good” friend. This emphasis on overcoming the subjective perspective is aspirational, not nec- essary perfectly accomplished. We are all human beings with our own biases, some of which we may be unaware of, and the introduction of this human element is something we strive to reduce to the irreducible minimum so that our conclusions are not significantly compromised. Good science is not a question of conducting a “purely” objective piece of research, which, lacking perfect objectivity, is fatally flawed and its results no better than “other ways of knowing.” A more realistic per- spective is addressed by Gorenstein (1986, p. 589): It makes no sense to reject the potential scientific import of a construct simply because social values may have played some role in its formulation. The question of whether a construct has any scientific importance is an empirical one. It has to do with whether the construct exhibits lawful properties. PHILOSOPHICAL ASSUMPTIONS OF PROGRAM EVALUATION As a specialized form of social and behavioral science research, program evaluation is predicated on certain philosophical assumptions pertaining to the nature of reality, and of the design and conduct of research inquiry. Although these philosophical assump- tions are often confused with theory, you can differentiate them by recalling that social and behavioral science theories are explanations of problems or interventions, explana- tions amenable to investigation via scientific research, and that theories themselves are based on certain philosophical assumptions that are fundamentally untestable. Some of these philosophical foundations of program evaluation are listed in Box 1.6. Each of these points of view has occupied central controversies within philoso- phy for hundreds of years, and in some cases, millennia. In offering them as charac- terizing the philosophy of science undergirding program evaluation, we make no pretense that any one of them can be considered irrefutably justified either by logic or by empirical data. Consider them, like Euclid’s axioms, as undemonstrated prop- ositions (although common sense would suggest that they are self-evident) that serve as intellectual pivot points around which program evaluation efforts revolve. 22 chapter 1
- 40. BOX 1.6 Selected Philosophical Foundations of Program Evaluation ± ± ± ± ± ± ± ± ± ± ± ± Acceptance of: Realism—The point of view that the world has an independent or objective existence apart from the perceptions of the observer Determinism—The assumption that all phenomena have physical causes that are potentially amenable to scientific investigation Positivism—The belief that valid knowledge about the objective world can be arrived at through scientific research Rationalism—The belief that reason and logic are useful tools for scientific inquiry, and that, ultimately, truthful explanations of human behavior will be rational Empiricism—A preference to rely on evidence gathered systematically through obser- vation or experiment, and capable of being replicated (that is, verified) by others, using satisfactory standards of evidence Operationalism—The assertion that it is important to develop measures of phenomena (e.g., client problems, interventive procedures) that can be reliably replicated by others Parsimony—A preference for the simpler of the available adequate explanations for behavioral phenomena Pragmatism—The view that the meaning or truth of anything resides in its conse- quences in action Scientific skepticism—The point of view that all claims should be considered of doubt- ful validity, until substantiated by credible scientific data Rejection of: Nihilism—A doctrine that all values are baseless and that nothing is knowable or can be communicated Anecdotism—The belief that anecdotes prove something empirically Metaphysics—Explanation involving supernatural, incorporeal or immaterial entities or factors Dualism—The view that the world consists of the two fundamental entities of mind and matter Reification—Attributing reality status to an abstract or hypothetical construct (e.g., the superego) in the absence of adequate evidence supporting the existence of that construct Circular Reasoning—An explanation for human behavior in which causes and effects cannot be distinguished from each other Scientism—The theory that the investigational methods used in the natural sciences should be applied to answer all questions of concern to people (e.g., ethical questions, moral questions, etc.) Solipsism—The view that there is no proof that anything exists outside the mind introduction 23
- 41. No one, for example—certainly not the authors—is capable of providing a philo- sophically irrefutable proof that the world has a physical reality (as opposed to a subjective construction in one’s mind). But in order to take action, some assump- tions are necessary. Thus, for example, the distinguished social work educator Bertha Capen Reynolds claimed “A second characteristic of scientifically oriented social work is that it accepts the objective reality of forces outside itself with which it must cooperate” (Reynolds, 1942, p. 24). Can this assumption of an objective reality, known in philosophy as realism, be irrefutably shown to be true? No, but for the purposes of attempting to improve the condition of the world we live in, some such practical assumptions must be accepted. We urge the reader to understand that the assumption of the validity of some of these philosophical beliefs does not preclude accepting others, even those appar- ently at odds with the former. For example, the assumption of realism does not dis- avow the importance of individual clients’ perceptions of their world, or of the meaning they ascribe to their lives and relationships. Nor does it deny that to some extent individuals construct their own reality. But realism asserts that there is an element to the client’s world that objectively exists, and that this is often a very important one, in fact often the most important to program evaluators. For example, evaluations of domestic violence interventions are more likely to concen- trate on ascertaining whether or not acts of violence really declined or ceased, as opposed to trying to understand a spouse’s “meaning” of what it means to be beaten. The evaluation of a neighborhood watch crime reduction program may look at community members’ perceptions of crime rates, but the actual rates of var- ious crimes before and after the implementation of the neighborhood watch pro- gram are usually seen as more important. Program evaluations usually focus on determining objective changes in the lives of clients, not on understanding subjec- tive processes. Of course, it is a legitimate part of science to study such subjective processes, such as the meaning to clients of changes brought about by a program, but such is not usually the primary focus of program evaluation. Sometimes evalu- ation studies encompass both aspects of the impact of programs—objective and subjective—and that can be worthwhile. To accept determinism does not deny that many other phenomena (e.g., psycho- social problems) exist and possess such a complex constellation of interlocking etio- logical factors that creates an appearance of chaos or randomness at times. And it may be that only a portion of a particular psychosocial phenomena is “determined” in a scientific sense. If so, then it is the task of scientific inquiry and of program evaluation to fully explore the limits of this determinism, even if it yields an under- standing less than 100 percent complete. A “Program of Assertive Community Treatment” that helps reduce psychiatric rehospitalization by 30 percent among the chronically mentally ill is not a failure because it is not 100 percent effective. Clearly, a 30 percent reduction leaves much to be understood and plenty of room for addi- tional improvements. But 30 percent is 30 percent better than where the clients were previously. To be a positivist is not to assert that certain knowledge in a particular area currently exists, but rather to claim that it is a good idea to strive for such knowledge through scientific methods because this will result in ever closer approx- imations to truth. Although one may be an empiricist, this need not deny the role of common sense, intuition, authority, and tradition as potentially valuable sources of knowledge. 24 chapter 1
- 42. Rationalism contends that factual explanations for psychosocial phenomena can be arrived at via logic—armchair reasoning if you will. Unfortunately, rationalism can also lead to errors. Thus, rationalism and logical thinking are seen as components of ultimately truthful explanations (e.g., being able to explain logically why a program works), but these conclusions must be evaluated using empirical tests, as these are the best means of sorting out truth from fiction. Parsimony in theory is nothing new. Clinicians are taught, for example, to exclude organic factors as possible causes of a client’s depression prior to implementing an in- tervention based on a presumptive psychodynamic etiology. Similarly, a theoretical ex- planation that uses fewer unverified factors is generally preferred over one that invokes more complex accounts. Consider, for instance, that for many years homosexuality was associated with severe psychopathology. Why? Well, many homosexuals sought treatment from psychiatrists and other mental health professionals. Thus, they ap- peared to have greater pathology than the general population. Was the assumption correct? One possibility was that homosexuality was associated with mental disorder. The second possibility was that this conclusion was only a distortion caused by a selec- tion bias (those seeking treatment for mental and emotional problems were not repre- sentative of all gays and lesbians). Parsimony suggested that careful attention be given to the latter explanation prior to accepting the former; and indeed, studies have found that gays and lesbians are no more or less mentally disordered than heterosexuals. Certainly not all parsimonious explanations possess greater validity than more complex reasons. After all, many psychosocial problems are really very complex. Parsimony simply cautions us to rule out or entertain potentially simpler accounts prior to accepting more involved ones. Pragmatism, although commonly seen as synonymous with practicality, has a more in-depth meaning involving the consequences of some program. According to pragmatism, the “meaning” of a child abuse prevention program resides in the numbers of children who are not exposed to abuse as a result of that program. A job-finding program’s real meaning is based on the number of clients successfully placed in good jobs. A program or intervention that fails to produce any changes can be said, in a pragmatic sense, to be a meaningless program. The meaning of a program is not how clients or service providers view the service, but stems from the practical, positive results obtained because of involvement with it. Pragmatism does not exclude other (perhaps more subjectivist) interpretations of what a program may mean to individuals; but for the purposes of program evaluation and practice in the human services, it refers to the practical outcomes of an intervention. And that, after all, is what program evaluation is aimed at finding out. Scientific skepticism is the profession’s shield against the premature adoption of unsupported claims. The burden of proof lays at the feet of the person making an unusual claim. “Facilitated communication helps kids with an autistic disorder communicate.” “Neurolinguistic programming is an effective treatment for phobias.” “Primal scream therapy helps people who are depressed.” It is not difficult for the reader to encounter such claims every day. Scientific skepticism is the modest request that such assertions be supported with credible evidence prior to being accepted or widely adopted in practice. The burden of proof is not on the skeptic to show that these things do not work—it rests with the proponents to demonstrate that they do. Unlike the doctrine of nihilism, a point of view fatal to the spirit of scientific inquiry, skepticism is a tempered perspective. It does not deny the possibility that genuinely introduction 25
- 43. truthful explanations can be obtained; skepticism requires only that appropriate proofs be provided for positive assertions. Whenever a human services professional en- counters a proposal to fund a new program, it is always a good idea (scientifically, if not politically) to ask “Where are the data to support this approach?” We like the idea of using the national census to illustrate these principles. At a given point in time, one number represents how many persons live in the United States (realism). The census is designed to try and capture this mythic number as closely as possible (positivism). The scientific methods used by the Bureau of the Census attempt to use empirical and operationalized methods to obtain data from U.S. residents. This is certainly not a perfect approach to trying to gain an accurate census—minorities of color, undocumented aliens, and other marginalized persons (e.g., the homeless) are undercounted. However, no reasonable alternative method- ologies outside of conventional scientific inquiry can be employed for this purpose. We cannot simply ask some noted authority, “Excuse me, Mr. President, how many people are there in the United States at this moment?” Our intuition or even per- sonal practice experience cannot help, and it is extremely unlikely that divine reve- lation will lend a hand. Similarly, it can be assumed that certain outcomes follow clients’ experiences with a particular human services program, and it is the task of the program evaluator to best ascertain what these outcomes may be. Cumbersome and imperfect though they may sometimes be, there is no substitute for systematic, empirically oriented efforts at evaluation guided by the methods and philosophy of mainstream science, which, parenthetically, can embrace both quantitative and qualitative research approaches. Lastly, the notion of what might be called anecdotism—the belief that anec- dotes prove something empirically—needs to be addressed. Personal experiences and accounts can create very powerful belief systems that may yield true conclu- sions in a situation or for a particular individual. Someone who is mugged by a bearded bandit may justifiably be afraid of this fellow the next time he is encoun- tered; however, to generalize that fear to all bearded individuals would be unfortu- nate. The problem with anecdotes is that their “truth” may not generalize. At the same time, the personal experiences of clients and staff with a program are important to the program evaluator. We want to know about the problems en- countered and the good things that resulted. These experiences may become part of a qualitative evaluation or aggregated into more of a quantitative evaluation. However, we should not assume that an individual anecdote (or even several of them) constitutes a philosophical foundation for program evaluation. Although a single event (e.g., death of a client) may trigger an evaluation, that event does not provide a worldview guiding all inquiry as do the positions previously discussed (sci- entific skepticism, pragmatism, and so on). Science does not reject anecdotes as false, or claim that all are false, but merely asserts that as evidence they provide a low level of proof and that they are most meaningful when augmented by more robust stan- dards of evidence. It has been justifiably said that the plural of anecdote is not data. MORE ON POSITIVISM Much discussion has been expended in the program evaluation literatures over the meaning of the term positivism, and of its value as a foundation for inquiry in the human services (Bolland & Atherton, 2002; Smith, 2002; Thyer, 1993, 2008b, 2008c). 26 chapter 1
- 44. Keep in mind that from its inception, professional social work accepted the doctrine of positivism, established in the early part of the 1800s by the French philosopher and scientist Auguste Comte. Comte is said to have established the discipline of soci- ology, which he originally called social physics, based on his contention that human behavior could be studied using the same tools and principles that science used to study natural phenomena: The word positive came from ponere and had been employed since the fourteenth century to mean laid down. In the sixteenth century, it began to refer to knowledge that was based on facts and was thus reasonably certain. Eighteenth century thinkers used the word positive to oppose the metaphysical. (Pickering, 1993, p. 65) According to Comte, “Unlike theological beliefs, scientific truths could be proved if necessary…. [S]cientific truths were always provisional because they could be proved wrong at any time” (Pickering, 1993, p. 171). Despite its name, positiv- ism took a provisional approach to the development of knowledge, with Comte noting in 1825 that “Scientific laws were only hypotheses constructed by man with external materials and confirmed by observation; they amounted to no more than approximations of a reality that could never be rigorously understood” (Pickering, 1993, p. 294). From its inception, positivism was concerned not just with studying human phenomena, but also with improving the human condition. “Social science was not … just an intellectual mixture of history, the physical sciences, physiology, and political economy. It had a practical vocation: to regenerate society…. Although Comte admitted that we could never fully know external reality, he assumed that scientific theories were getting closer to representing it exactly” (Pickering, 1993, pp. 294, 296, italics added). Positivism came to exert an enormous influence on sci- ence, on the emerging social sciences, and in social welfare in particular. In the United States, the American Social Science Association (ASSA) was established in 1865, and set forth as its mandate: Social science was understood by (ASSA) members to refer to the whole realm of problematical relationships in human affairs. One became a social scientist “by contributing to the store of esoteric knowledge and practical expertises … a new way to care for the insane or to administer charity—all of these were equally valuable contributions to social science.” (Haskell, 1997, pp. 97–98, italics added) This Association proposes to afford to all persons interested in human improvement, an opportunity to consider social economics as a whole…. They are to collect all facts, diffuse all knowledge, and stimulate all inquiry, which have a bearing on social welfare. (Haskell, 1997, p. 102, italics added) From the ASSA emerged, in 1879, the Conference on Charities, transformed in 1884 into the National Conference of Charities and Corrections (NCCC). In 1917 the NCCC became the National Conference on Social Work, transformed again into the National Conference on Social Welfare in 1957, which in turn dissolved in the mid-1980s. A paper presented at the 1889 meeting of the NCCC was titled “Scientific Charity” and an article appearing in an 1894 issue of the influential journal The Charities Review was titled “A Scientific Basis for Charity.” Such early works were a part of the movement called scientific charity (or scientific philan- thropy), which had its own origins in the 1870s, again based on the fundamental assumptions of positivism. introduction 27
- 45. The distinguished social work educator Frank Bruno (1964) provided this overview of a National Conference on Charities meeting of the late 1800s: Most of the leaders of the Conference accepted the implications of a scientific approach to social work problems. They acted on the tacit assumption that human ills—sickness, insanity, crime, poverty—could be subjected to study and methods of treatment, and that a theory of prevention could be formulated as well…. This attitude raised these problems out of the realm of mysticism into that of a science…. As a result of the adoption of this scientific attitude, Conference speakers and programs looked toward progress, not backward toward a golden age. They believed in the future; that it was possible, by patient, careful study and experimentation, to create a society much better than the one they lived in. (pp. 26–27) And about 30 years earlier Bruno (1936) had provided his own congruent perspec- tive on the integration of science and social work: Social work holds as its primary axiom that knowledge of human behavior can be acquired and interpreted by the senses and that inferences drawn from such knowledge can be tested by the principles of logic. The difference between the social work of the present and all of the preceding ages is the assumption that human behavior can be understood and is determined by causes which can be explained. We may not at present have a mastery of the methods of understanding behavior, but any scientific approach to behavior presupposed that it is not in its nature incomprehensible by sensory per- ceptions and inference therefrom. It follows from such a theory that understanding is the first step in the direction of control and that the various forms of human misery are susceptible not only of amelioration, which our predecessors attempted, but also of prevention or even of elimination, when once thir nature is understood. (pp. 192–193) Both of the above statements, dated though they may be, could be said to charac- terize the mainstream contemporary perspective adhered to by science in general, and by program evaluation in particular. A more recent synopsis of positivism was put forth by Glashow (1989, p. 24E): We believe that the world is knowable, that there are simple rules governing the be- havior of matter and the evolution of the universe. We affirm that there are eternal, objective, extra historical, socially neural, external and universal truths and that the assemblage of these truths is what we call … science. Natural laws can be discovered that are universal, invariable, inviolate, genderless and verifiable. They may be found by men or women … You will encounter the term positivism in your academic studies, and in the program evaluation literature. Keep in mind the relatively simple premise behind it: “A paradigm introduced by Auguste Comte, which held that social behavior could be studied and understood in a rational, scientific manner—in contrast to ex- planations based on religion or superstition” (Rubin & Babbie, 2008, p. 642). Positivism is an approach, not intended to represent accomplished facts about hu- man behavior. As Todd noted back in 1920 “Science does not claim to have com- plete knowledge for the world or to have established perfect order out of chaos in this world. It is less an accomplished fact than an attitude” (p. 71). Nor does positiv- ism deny the potential utility of other nonscientific approaches to knowledge develop- ment in social work—tradition, values and ethics, religion, mysticism; all may have a valuable contributing role to our field. Asserting that one approach (called A) is 28 chapter 1
- 46. useful, is not to say that B is valueless. Positivism does make strong claims regarding the value of scientific approaches to knowledge development in general and program evaluation in particular. Some may even claim it makes the strongest claims. What is important is the development of reliable and valid information, and this can be approached through many methods, including positivistic ones. CHAPTER RECAP Whether you are a direct service worker, program director, or an agency adminis- trator, you want the agency that employs you to be well managed and responsive to the needs of clients and community. How does an agency become a well-managed agency? One essential way is the evaluation of its efforts, where problems are iden- tified and corrective action taken. What is essential to learn about program evaluation? Besides understanding the purpose of program evaluations and some of the various reasons why they are con- ducted, you need to know the difference between a subjectively held opinion and one that is derived from objective data. This book will help you develop ways of identifying, collecting, and using data that will allow you to be as objective as pos- sible when evaluating programs in the social and human services. Objective data are seen as having greater credibility and as providing better information for the decisions that face program managers. Evaluators use operational definitions to obtain objective data that can be replicated if necessary. Theoretical models suggest not only what interventions might work, but also where to find the changes that have resulted. Questions for Class Discussion 1. Make a list of five or six human services programs with which you or members of the class are familiar. In another column, list what is known about how well each program does its job. For example, what is its success rate? Other than subjective feelings about these programs, what is known about how “good” these programs are? In a third column, make a list of questions that you would like to have answered about each program. 2. Evaluators must operationally define what will be recognized as “success” or a “successful outcome.” Try your hand at operationally defining “success” for several of the programs you listed in question 1. 3. For the human services programs you listed in question 1, discuss your ideas about theoretical orientations on which the interventions might be based. 4. Discuss your experiences with program evaluation in your job or field practicum. 5. What are the characteristics of a “good” television program? Make a list of all the subjective opinions held by the class members about a “good” television program. How could you objectively determine if a television program is “good”? 6. Why is it necessary to develop operational definitions about such things as what constitutes recidivism or a successful client outcome? Use specific examples. 7. Pick one of the philosophical assumptions undergirding program evaluation described in Box 1.6 and defend it as a useful orientation for empirical research. introduction 29
- 47. Pick another assumption and present some reasoned arguments as to why you do not think it to be a valid perspective. 8. Assume that the ‘perfect’ program evaluation has been conducted (sadly it has not) and it found that participation in Alcoholic’s Anonymous leads to sobriety. Discuss what implications this has for the theory underlying AA. Mini-Projects: Experiencing Evaluation Firsthand 1. Choose a product (e.g., coffeemakers, DVD players, televisions, microwave ovens), and develop a set of objective standards that could help consumers select a model of superior performance and avoid the inferior models. Once you have finished, consult back issues of Consumer Reports to see how the standards you used compare with those used by the Consumer Products Testing Union. 2. What would you request in the way of an evaluation if you were in a position to require evaluation of a national program? Select a national program, and iden- tify what information would be needed in order for an unbiased panel of experts to conclude that the program was successful. 3. Find an example of a program evaluation study published in a professional journal. We suggest that you learn to use the online database called PsycINFO, which is likely available via your local university library, to locate such studies. Briefly describe how key variables were operationally defined in this study, how the program’s success was measured, and whether or not the program explicitly made use of an explanatory theory—etiological or interventive, or both. References and Resources American Counseling Association. (1999). Code of ethics. Alexandria, VA: Author. www. counseling.org/resources/codeofethics.htm. American Psychiatric Association. (2000). Diagnostic and statistical manual for mental Disorders IV-TR. Washington, DC: Author . Angell, R. C. (1954). A research basis for welfare practice. Social Work Journal, 35, 145–148, 169–171. Austin, D. M. (1998). A report on progress in the development of research resources in social work. Austin, TX: University of Texas School of Social Work. Barker, R. (Ed.). (2003). The social work dictionary (5th ed.). Washington, DC: NASW Press. Board of Professional Affairs, Committee on Professional Standards. (1987). General guide- lines for providers of psychological services. American Psychologist, 42, 1–12. Board of Professional Affairs, Committee Professional Standards. (1987). General guidelines for providers of psychological services. American Psychologist, 42, 1–12. Bolland, K., & Atherton, C. (2002). Heuristics versus logical positivism: Solving the wrong problem. Families in Society, 83, 7–13. Brownson, R. C., Baker, E. A., Lett, T. L., & Gillespie, K. N. (2003). Evidence-based public health. New York: Oxford University Press. Bruno, F. (1936). The theory of social work. New York: Health. Bruno, F. (1964). Trends in social work: 1874–1956. New York: Columbia University Press. Cabot, R. (1931). Treatment in social casework and the need of criteria and of tests of its success or failure. Proceedings of the National Conference of Social Work. 30 chapter 1
- 48. Chambless, D., Sanderson, W., Shoham, V., Johnson, S., Pope, K., Crits-Cristoph, P., et al. (1996). An update on empirically validated therapies. The Clinical Psychologist, 49(2), 5–18. Christophersen, E. R., & Mortweet, S. L. (2001). Treatments that work with children: Empirically supported strategies for managing childhood problems. Washington, DC: American Psychological Association. Conrad, K. J., & Miller, T. Q. (1987). Measuring and testing program philosophy. New Directions for Program Evaluation, 33, 19–42. Drake, R. E., Merrens, M. R., & Lynde, D. W. (2005). Evidence-based mental health practice: A textbook. New York: Norton. Dukes, P. L., Ullman, J. B., & Stein, J. A. (1996). A three-year follow-up of Drug Abuse Resistance Education. Evaluation Review, 20, 49–66. Ell, K. (1996). Social work research and health care practice and policy: A psychosocial research agenda. Social Work, 41, 583–592. Foa, E. B., Dancu, C. V., Hembree, E. A., Jaycox, L. H., Meadows, E. A., & Street, G. P. (1999). A comparison of exposure therapy, stress inoculation training, and their combi- nation for reducing posttraumatic stress disorder in female assault victims. Journal of Consulting and Clinical Psychology, 67, 194–200. Foa, E. B., Rothbaum, B. O., Riggs, D. S., & Murdock, T. B. (1992). Treatment of posttraumatic stress disorder in rape victims: A comparison between cognitive- behavioral procedures and counseling. Journal of Consulting and Clinical Psychology, 59, 715–723. Gambrill, E. (1999). Evidence-based practice: An alternative to authority-based practice. Families in Society, 80, 341–350. Glashow, S. (1989, October 22). Positivism lives: We believe that the world is knowable. New York Times, p. 24E. Gorenstein, E. E. (1986). On the distinction between science and valuation in the mental health field (letter). American Psychologist, 41, 588–590. Greenwood, E. (1957). Social work research: A decade of reappraisal. Social Service Review, 31, 311–320. Grinnell, R. M., & Unrau, Y. A. (Eds.). (2008). Social work research and evaluation (8th ed.). New York: Oxford University Press. Gyorky, Z., Royalty, G. M., & Johnson, D. H. (1994). Time-limited therapy in university counseling centers: Do time-limited and time-unlimited centers differ? Professional Psychology: Research and Practice, 25(1), 50–54. Haskell, T. L. (1997). The emergence of professional social science: The American Social Science Association. Urbana, IL: The University of Illinois Press. Hibbs, E. D., & Jensen, P. S. (Eds.). (2005). Psychosocial treatments for child and adolescent disorders: Empirically based strategies for clinical practice (2nd ed.). Washington, DC: American Psychological Association Press. Hill, D. (1998). Drug money. Brandweek, 39(17), 20–26. Jemmott, L. S., & Jemmott, J. B. (1991). Applying the theory of reasoned action to AIDS risk behavior: Condom use among black women. Nursing Research, 40(4), 228–234. Kolata, G. (1994, May 4). Clinic entices patients by paying them $10 a visit. New York Times, p. B8. LeCroy, C. W. (2008). Handbook of evidence-based treatment manuals for children and adolescents. New York: Oxford University Press. Lohr, J. M., Hooke, J., Gist, R., & Tolin, D. F. (2003). Novel and controversial treatments for trauma-related stress disorders. In S. D. Lilienfeld, S. J. Lynn, & J. M. Lohr (Eds.)., Science and pseudoscience in clinical psychology (pp. 243–272). New York: Guilford Press. introduction 31
- 49. Lynam, D., Milich, R., Zimmerman, R., Novak, S., Logan, T. K., Martin, C., et al. (1999). Project DARE: No effects at 10-year follow-up. Journal of Consulting and Clinical Psychology, 67, 590–593. MacKenzie, D. L., Biere, D., & Mitchell, O. (2007). An experimental study of a therapeutic boot camp: Impact on impulses, attitudes and recidivism. Journal of Experimental Criminology, 3, 221–241. Mullen, E. (1995). A review of research utilization in the social services. Social Work, 40, 282–283. Nathan, P. E., & Gorman, J. M. (Eds.). (2007). A guide to treatments that work (3rd ed.). New York: Oxford University Press. National Association of Social Workers. (1989). NASW standards for the practice of clinical social work. Silver Spring, MD: Author. National Association of Social Workers. (1992). NASW standards for school social work services. Washington, DC: NASW Press. National Association of Social Workers. (1999). Code of ethics. Washington, DC: NASW Press. Norcross, J. C., Beutler, L. E., & Levant, R. F. (Eds.). (2006). Evidence-based practices in mental health. Washington, DC: American Psychological Association. Norcross, J. C., Hogan, T. P., & Koocher, G. P. (2008). Clinician’s guide to evidence-based practices: Mental health and the addictions. New York: Oxford University Press. Passell, P. (1993, March 9). Like a new drug, social programs are put to the test. New York Times, p. B5. Pickering, M. (1993). Auguste Comte: An intellectual biography, vol. I. New York: Cambridge University Press. Reynolds, B. C. (1942). Learning and teaching in the practice of social work. New York: Farrar & Rinehart. Rosen, A., Proctor, E. K., & Staudt, M. M. (1999). Social work research and the quest for effective practice. Social Work Research, 23, 4–14. Rosenberg, M. L., & Brody, R. (1974). The threat or challenge of accountability. Social Work, 19, 344–350. Roth, A., & Fonagy, R. (2005). What works for whom: A critical review of psychotherapy research (2nd ed.). New York: Guilford. Rubin, A., & Babbie, E. (2008). Research Methods for Social Work (6th ed.). Belmont, CA: Thomson. Rutter, M. (1987). Psychosocial resilience and protective mechanisms. American Journal of Orthopsychiatry, 57, 316–331. Sackett, D. L., Richardson, W. S., Rosenberg, W., & Haynes, R. R. (1997). Evidence-based medicine: How to practice and teach E.M. New York: Churchill-Livingston. Sackett, D. L., Strauss, S. E., Richardson, W. S., Rosenberg, W., & Haynes, R. R. (2000). Evidence-based medicine: How to practice and teach E.M. (2nd ed.). New York: Churchill-Livingston. Saunders, D. G. (1992). A typology of men who batter: Three types derived from cluster analysis. American Journal of Orthopsychiatry, 62, 264–275. Schalock, R. L., & Thornton, C. V. D. (1988). Program evaluation: A field guide for administrators. New York: Plenum. Seligman, L., & Reichenberg, L. W. (2007). Selecting effective treatments: A comprehensive guide to treating mental disorders (3rd ed.). New York: Wiley. Smith, D. (2002). The limits of positivism revisited. Social Work and Social Sciences Review, 10, 27–37. Strauss, S. E., Richardson, W. S., Glasziou, P., & Haynes, B. (2005). Evidence-based medicine: How to practice and teach EBM (3rd ed.). New York: Elsevier. 32 chapter 1
- 50. Sugarman, B. (1988). The well-managed human service organization: Criteria for a manage- ment audit. Administration in Social Work, 12(4), 12–27. Telch, C. F., Agras, W. S., Rossiter, E. M., Wilfley, D., & Kenardy, J. (1990). Group cognitive- behavioral treatment for the nonpurging bulimic: An initial evaluation. Journal of Consulting and Clinical Psychology, 58, 629–635. Thyer, B. A. (1993). Social work theory and practice research: The approach of logical pos- itivism. Social Work and Social Sciences Review, 4(1), 5–26. Thyer, B. A. (1994). Are theories for practice necessary? Journal of Social Work Education, 30, 147–151. Thyer, B. A. (2001). What is the role of theory in research on social work practice? Journal of Social Work Education, 37, 9–25. Thyer, B. A. (2004). What is evidence-based practice? Brief Treatment and Crisis Intervention, 4, 167–176. Thyer, B. A. (2008a). Evidence-based practice, science, and social work. In A. Roberts (Ed.). Social worker’s desk reference (2nd ed.). New York: Oxford University Press. Thyer, B. A. (2008b). The potentially harmful effects of theory in social work. In B. A. Thyer (Ed.), Comprehensive handbook of social work and social welfare: Vol. 2. Human Behavior in the Social Environment (pp. 519–541). New York: Wiley. Thyer, B. A. (2008c). The quest for evidence-based practice?: We are all positivists! Research on Social Work Practice, 18, 339–345. Thyer, B. A., & Wodarski, J. S. (Eds.). (2007). Social work in mental health: An evidence- based approach. New York: Wiley. Todd, J. (1920). The scientific spirit and social work. New York: Macmillan. Tripodi, T. (1987). Program evaluation. In A. Minahan (Ed.), Encyclopedia of social work. Silver Spring, MD: National Association of Social Workers. Weiss, B., Catron, T., Harris, V., & Phung, T. M. (1999). The effectiveness of traditional child psychotherapy. Journal of Consulting and Clinical Psychology, 67, 82–94. Zambelli, G. C., & Derosa, A. P. (1992). Bereavement support groups for school-aged children: Theory, intervention, and case example. American Journal of Orthopsychiatry, 62(4), 484–493. introduction 33
- 51. CHAPTER ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± 2 Ethical Issues in Program Evaluation Ethical issues in program evaluation encompass a broad realm of concerns that be- come heightened when the participants of a study are vulnerable in some way. Although most often associated with medical experiments, harm can result from participation in evaluation research, even if not life threatening. In keeping with up- dated (and more accurate) terminology, the phrase “human subjects” will be re- placed whenever possible with “participants” in this chapter. HISTORICAL BACKGROUND: THE ORIGIN AND GROWTH OF INSTITUTIONAL REVIEW BOARDS (IRBS) By the 1970s, widespread concern about unethical research experiments led to a federal mandate requiring that institutional review boards (IRBs) be convened to oversee research in organizations receiving federal funds (primarily universities but also hospitals, and other entities). As such, IRBs were empowered to review the inves- tigator’s research objectives, methodology, and protocols with special emphasis on plans for recruiting participants and gaining their consent. IRBs were created to protect human participants from research that is risky, harmful, or does not respect the rights and dignity of human beings. These boards are usually comprised of professionals knowledgeable about research methods as well as lay citizens representing the community. It is the IRB’s task to consider pro- posed research and to determine whether it follows ethical guidelines set forth by the federal government’s Office of Human Research Protections (OHRP). Do we need IRBs? Are they a waste of time for the knowledgeable and ethical researcher? Despite public awareness of the horrific medical experiments carried out by Nazi physicians, serious and flagrant abuses of human subjects occurred in this country, the most notorious of these was the infamous Tuskegee study conducted by 34
- 52. the U.S. Public Health Service. Beginning in 1932, 400 black males, mostly poor and illiterate, with tertiary-stage syphilis were informed that they would receive free treat- ment for their “bad blood.” In actuality, these men received no treatment for syphilis even after penicillin became available. Instead, they received free physical exams, peri- odic blood testing, hot meals on examination days, free treatment for minor disorders, and a modest burial fee. The public outcry over the Tuskegee study led Congress to pass the National Research Act (Public Law 93-348) in 1974 mandating the forma- tion of IRBs. The legacy of Tuskegee, and the distrust it engendered regarding research ethics, continues today in African American communities and elsewhere. According to Oakes (2002), the literature on evaluation has largely neglected discussion of IRBs despite their rapid growth in numbers and in power. By so often straddling the threshold between “research” and “non-research” (the latter in the form of internal evaluation for program improvement), program evaluations may or may not fall under IRB jurisdiction. Evaluations considered to be “research,” that is, their findings are intended for dissemination and knowledge development, may be exempt from IRB review if there are no federally funded programs involved. Increasingly, however, evaluation researchers are seeking institutional IRB approval so that they may publish and share their findings with a wider audience. Many pro- grams and agencies have begun to establish their own internal IRBs as a result. ETHICAL GUIDELINES The National Commission for the Protection of Human Subjects in Biomedical and Behavioral Research (the Belmont Report) in 1979 identified three ethical principles for research on humans: beneficence—maximizing good outcomes for humanity and research subjects while minimizing or avoiding risk or harm; respect—protecting the autonomy of all persons and treating them with courtesy and respect, including those who are most vulnerable; and justice—ensuring reasonable, nonexploitative, and well-considered procedures are administered fairly (the fair distribution of costs and benefits). These principles have been fashioned into the following ethical guidelines for researchers and evaluators to follow. GUIDELINE 1: RESEARCH PARTICIPANTS MUST BE VOLUNTEERS All of those partici- pating in a research or evaluation effort should freely decide to do so without coercion of any kind. They must also be competent to understand their choices. If eligible partici- pants are under age 18 or not considered able to fully comprehend the consent process due to mental impairment, their legal caretaker must give permission, and the participant still must give assent. Although very young children are generally asked to orally assent to research procedures in language they can understand, the older the minor, the more the informed consent should conform to a written document like adults would receive. Study participants’ right to self-determination must be respected, and their free- dom to withdraw from the study at any time carefully protected. In most instances, IRBs require that consent forms be in writing and signed permission be obtained, but exceptions can be made for vulnerable populations such as undocumented immigrants. Consent forms provide general but brief information on the nature of the project and indicate that the participant is free to withdraw consent and to dis- continue participation in the project at any time without any penalty or loss of bene- fits (see Figure 2.1). When evaluating a social service program, it is vitally important ethical issues in program evaluation 35
- 53. Outpatient Drug Treatment Program Aftercare Study Investigator: Ellen Samovar, MSW, Principal Investigator, (231) 555–5760 I __________ have been asked to participate in an evaluation of the Outpatient Drug Treatment Program (ODTP) being conducted by Ms. Samovar. Purpose: I understand that the purpose of this study is to examine the success of the Outpatient Drug Treatment Program in which I am participating—to learn how approximately 300 clients have stopped using drugs and what factors may influence these decisions. Duration and Location: I understand the study will take place at the ODTP offices on 717 South First Street. Further, I understand that the study will take about 60 minutes of my time on two different occasions. Procedures: I will be asked to answer questions about my social and psychological well-being, relationships, employment, drug use, and illegal activities. In addition, I will be asked to provide a urine sample to test for evidence of drugs in my system and will be given a Breathalyzer to test for alcohol. Risks/Discomforts: It has been explained to me that some of the interview questions are very personal, involving drug and criminal behavior, and may cause some discomfort in answering them. Benefits: I understand that the benefits from participating in this study may be to help researchers and those involved in public policy better understand the factors that lead to the starting and stopping of drug use. Confidentiality: I understand that a research code number will be used to identify my responses from those of other clients and that my name, address, and other identifying information will not be directly associated with any information obtained from me. A master listing of persons participating in the study and their identifying information will be kept in a secure location under lock and key except when being used by select staff. Further, I understand that a certificate of confidentiality has been obtained from the Department of Health and Human Services (DHHS) that protects investigators from being forced to release any of my data, even under a court order or a subpoena. When results of this study are published, my name or other identifying information will not be used. Payments: I will be paid $25 for my time and cooperation for each scheduled testing episode. If I stop early, I understand that I will be paid an amount appropriate to the time I have spent. Right to Withdraw: I understand that I do not have to take part in this study, and my refusal to participate will involve no penalty or loss of rights to which I am entitled. I may withdraw from the study at any time without fear of losing any services or benefits to which I am entitled. Signatures: I have read this entire consent form and completely understand my rights as a potential research subject. I voluntarily consent to participate in this research. I have been informed that I will receive a copy of this consent should questions arise and I wish to contact Ms. Samovar or the University of Somewhere’s Institutional Review Board (231-555-4949) to discuss my rights as a research subject. _________________________ Date _________________________ Date _________________________ Date __________________________________________________ Signature of Research Subject __________________________________________________ Signature of Witness __________________________________________________ Signature of Investigator Figure 2.1 | Consent to Participate in a Research Study 36 chapter 2
- 54. that recipients of services fully understand these assurances. Evaluators can use a checklist to ensure that all the necessary informational items are contained in the con- sent forms they use (see Box 2.1). Consent forms are not typically employed with mail, Internet, or telephone surveys unless the data being gathered are in some way sensitive (e.g., involve informa- tion about past or present drug use or illegal activities). In such cases, consent forms may need to be mailed in advance and consent obtained before the participant is contacted for data collection. When questionnaires or surveys are used with adults who are not part of a vulnerable population, the principle of implied consent may be used—that is, the act of participation is seen as giving informed consent. In these in- stances, IRBs may not require written consent. A problem arises when potential participants feel that they cannot refuse con- sent. If they are on probation or parole, or receiving some form of public assistance, they may not feel free to refuse without putting themselves in some jeopardy. Clients in treatment for mental health problems, if approached by a clinician, may fear being penalized if they refuse. A homeless person may be “coerced” into partic- ipation by being offered a substantial financial incentive. This is when consultation with an IRB can come in handy—extra steps may be needed to allay suspicions and inform potential participants of their rights. Human services professionals must be alert to the possibility that even good- naturedly encouraging clients to participate in research can be perceived as coer- cion. Because professionals are “gatekeepers” of services, clients can feel pressured into participating in order to gain access to or continue receiving services. GUIDELINE 2: POTENTIAL PARTICIPANTS SHOULD BE GIVEN SUFFICIENT INFORMATION ABOUT THE STUDY TO DETERMINE ANY POSSIBLE RISKS OR DISCOMFORTS AS WELL AS BENEFITS Sufficient information includes an explanation of the purpose of the research, the expected BOX 2.1 Checklist for Informed Consent Forms ± ± ± ± ± ± ± ± ± ± ± ± Each informed consent form should contain explanation of the following points: ___ How the research subject was chosen/invited to participate and how many will be involved ___ Who is conducting the study ___ The purpose of the study ___ Where the study will take place and how long it will last ___ What is required of the research subject ___ Possible risks or discomforts ___ Possible benefits of participation ___ Alternatives associated with nonparticipation ___ Any costs of participation ___ Payment or incentives for taking part in the study ___ Stopping or ending participation ___ What happens if medical attention is needed ___ Who can answer questions about the project ethical issues in program evaluation 37
- 55. Another Random Document on Scribd Without Any Related Topics
- 59. The Project Gutenberg eBook of Miss Santa Claus of the Pullman
- 60. This ebook is for the use of anyone anywhere in the United States and most other parts of the world at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this ebook or online at www.gutenberg.org. If you are not located in the United States, you will have to check the laws of the country where you are located before using this eBook. Title: Miss Santa Claus of the Pullman Author: Annie F. Johnston Illustrator: Reginald Bathurst Birch Release date: December 11, 2012 [eBook #41604] Most recently updated: October 23, 2024 Language: English Credits: E-text prepared by Suzanne Shell, Emmy, and the Online Distributed Proofreading Team (http://www.pgdp.net) from page images generously made available by Internet Archive/American Libraries (http://archive.org/details/americana) *** START OF THE PROJECT GUTENBERG EBOOK MISS SANTA CLAUS OF THE PULLMAN ***
- 61. The Project Gutenberg eBook, Miss Santa Claus of the Pullman, by Annie Fellows Johnston, Illustrated by Reginald B. Birch Note: Images of the original pages are available through Internet Archive/American Libraries. See http://archive.org/details/misssantaclausof00john MISS SANTA CLAUS OF THE PULLMAN
- 62. Miss Santa Claus MISS SANTA CLAUS OF THE PULLMAN
- 63. BY ANNIE FELLOWS JOHNSON Author of "The Little Colonel Series," etc. With illustrations by REGINALD B. BIRCH NEW YORK THE CENTURY CO. 1913 Copyright, 1913, by The Century Co. Published, October, 1913 TO MY SISTERS LURA AND ALBION
- 64. LIST OF ILLUSTRATIONS Miss Santa Claus Frontispiece PAGE "Oh, dear Santa Claus" 19 "Here!" he said 29 "Oh, rabbit dravy!" he cried 57 He pushed aside the red plush curtain and looked in 69 And ran after the boy as hard as she could go 77 It was about the Princess Ina 99 The shower of stars falling on the blanket made her think of the star-flower 121 "Take it back!" 165
- 65. MISS SANTA CLAUS OF THE PULLMAN
- 66. T CHAPTER I HE last half hour had seemed endless to Will'm, almost as long as the whole four years of his life. With his stubby little shoes drawn up under him, and his soft bobbed hair flapping over his ears every time the rockers tilted forward, he sat all alone in the sitting- room behind the shop, waiting and rocking. It seemed as if everybody at the Junction wanted something that afternoon; thread or buttons or yarn, or the home-made doughnuts which helped out the slim stock of goods in the little notion store which had once been the parlor. And it seemed as if Grandma Neal never would finish waiting on the customers and come back to tell the rest of the story about the Camels and the Star; for no sooner did one person go out than another one came in. He knew by the tinkling of the bell over the front door, every time it opened or shut. The door between the shop and sitting-room being closed, Will'm could not hear much that was said, but several times he caught the word "Christmas," and once somebody said "Santa Claus," in such a loud happy-sounding voice that he slipped down from the chair and ran across the room to open the door a crack. It was only lately that he had begun to hear much about Santa Claus. Not until Libby started to school that fall did they know that there is such a wonderful person in the world. Of course they had heard his name, as they had heard Jack Frost's, and had seen his picture in story- books and advertisements, but they hadn't known that he is really true till the other children told Libby. Now nearly every day she came home with something new she had learned about him. Will'm must have known always about Christmas though, for he still had a piece of a rubber dog which his father had sent him on his first one, and—a Teddy Bear on his second. And while he couldn't
- 67. recall anything about those first two festivals except what Libby told him, he could remember the last one perfectly. There had been a sled, and a fire-engine that wound up with a key, and Grandma Neal had made him some cooky soldiers with red cinnamon-drop buttons on their coats. She wasn't his own grandmother, but she had taken the place of one to Libby and him, all the years he had been in the world. Their father paid their board, to be sure, and sent them presents and came to see them at long intervals when he could get away from his work, but that was so seldom that Will'm did not feel very well acquainted with him; not so well as Libby did. She was three years older, and could even remember a little bit about their mother before she went off to heaven to get well. Mrs. Neal wasn't like a real grandmother in many ways. She was almost too young, for one thing. She was always very brisk and very busy, and, as she frequently remarked, she meant what she said and she would be minded. That is why Will'm turned the knob so softly that no one noticed for a moment that the door was ajar. A black-bearded man in a rough overcoat was examining a row of dolls which dangled by their necks from a line above the show case. He was saying jokingly: "Well, Mrs. Neal, I'll have to be buying some of these jimcracks before long. If this mud keeps up, no reindeer living could get out to my place, and it wouldn't do for the young'uns to be disappointed Christmas morning." Then he caught sight of a section of a small boy peeping through the door, for all that showed of Will'm through the crack was a narrow strip of blue overalls which covered him from neck to ankles, a round pink cheek and one solemn eye peering out from under his thatch of straight flaxen hair like a little Skye terrier's. When the man saw that eye he hurried to say: "Of course mud oughtn't to make any difference to Santy's reindeer. They take the Sky Road, right over the house tops and all."
- 68. The crack widened till two eyes peeped in, shining with interest, and both stubby shoes ventured over the threshold. A familiar sniffle made Grandma Neal turn around. "Go back to the fire, William," she said briskly. "It isn't warm enough in here for you with that cold of yours." The order was obeyed as promptly as it was given, but with a bang of the door so rebellious and unexpected that the man laughed. There was an amused expression on the woman's face, too, as she glanced up from the package she was tying, to explain with an indulgent smile. "That wasn't all temper, Mr. Woods. It was part embarrassment that made him slam the door. Usually he doesn't mind strangers, but he takes spells like that sometimes." "That's only natural," was the drawling answer. "But it isn't everybody who knows how to manage children, Mrs. Neal. I hope now that his stepmother when he gets her, will understand him as well as you do. My wife tells me that the poor little kids are going to have one soon. How do they take to the notion?" Mrs. Neal stiffened a little at the question, although he was an old friend, and his interest was natural under the circumstances. There was a slight pause, then she said: "I haven't mentioned the subject to them yet. No use to make them cross their bridge before they get to it. I've no doubt Molly will be good to them. She was a nice little thing when she used to go to school here at the Junction." "It's queer," mused the man, "how she and Bill Branfield used to think so much of each other, from their First Reader days till both families moved away from here, and then that they should come across each other after all these years, from different states, too." Instinctively they had lowered their voices, but Will'm on the other side of the closed door was making too much noise of his own
- 69. to hear anything they were saying. Lying full length on the rug in front of the fire, he battered his heels up and down on the floor and pouted. His cold made him miserable, and being sent out of the shop made him cross. If he had been allowed to stay there's no telling what he might have heard about those reindeer to repeat to Libby when she came home from school. Suddenly Will'm remembered the last bit of information which she had brought home to him, and, scrambling hastily up from the floor, he climbed into the rocking chair as if something were after him: "Santa Claus is apt to be looking down the chimney any minute to see how you're behaving. And no matter if your lips don't show it outside, he knows when you're all puckered up with crossness and pouting on the inside!" At that terrible thought Will'm began to rock violently back and forth and sing. It was a choky, sniffling little tune that he sang. His voice sounded thin and far away even to his own ears, because his cold was so bad. But the thought that Santa might be listening, and would write him down as a good little boy, kept him valiantly at it for several minutes. Then because he had a way of chanting his thoughts out loud sometimes, instead of thinking them to himself, he went on, half chanting, half talking the story of the Camels and the Star, which he was waiting for Grandma Neal to come back and finish. He knew it as well as she did, because she had told it to him so often in the last week. "An' the wise men rode through the night, an' they rode an' they rode, an' the bells on the bridles went ting-a-ling! just like the bell on Dranma's shop door. An' the drate big Star shined down on 'em and went ahead to show 'em the way. An' the drate big reindeer runned along the Sky Road"—he was mixing Grandma Neal's story now with what he had heard through the crack in the door, and he found the mixture much more thrilling than the original recital. "An' they runned an' they runned an' the sleighbells went ting-a-ling! just like the bell on Dranma's shop door. An' after a long time they all
- 70. comed to the house where the baby king was at. Nen the wise men jumped off their camels and knelt down and opened all their boxes of pretty things for Him to play with. An' the reindeer knelt down on the roof where the drate big shining star stood still, so Santy could empty all his pack down the baby king's chimney." It was a queer procession which wandered through Will'm's sniffling, sing-song account. To the camels, sages and herald angels, to the shepherds and the little woolly white lambs of the Judean hills, were added not only Bo Peep and her flock, but Baa the black sheep, and the reindeer team of an unscriptural Saint Nicholas. But it was all Holy Writ to Will'm. Presently the mere thought of angels and stars and silver bells gave him such a big warm feeling inside, that he was brimming over with good-will to everybody. When Libby came home from school a few minutes later, he was in the midst of his favorite game, one which he played at intervals all through the day. The game was Railroad Train, suggested naturally enough by the constant switching of cars and snorting of engines which went on all day and night at this busy Junction. It was one in which he could be a star performer in each part, as he personated fireman, engineer, conductor and passenger in turn. At the moment Libby came in he was the engine itself, backing, puffing and whistling, his arms going like piston-rods, and his pursed up little mouth giving a very fair imitation of "letting off steam." "Look out!" he called warningly. "You'll get runned over." But instead of heeding his warning, Libby planted herself directly in the path of the oncoming engine, ignoring so completely the part he was playing that he stopped short in surprise. Ordinarily she would have fallen in with the game, but now she seemed blind and deaf to the fact that he was playing anything at all. Usually, coming in the back way, she left her muddy overshoes on the latticed porch, her lunch basket on the kitchen table, her wraps on their particular hook in the entry. She was an orderly little soul. But to-day she came in, her coat half off, her hood trailing down her back by its strings,
- 71. and her thin little tails of tightly braided hair fuzzy and untied, from running bare-headed all the way home to tell the exciting news. She told it in gasps. "You can write letters to Santa Claus—for whatever you want— and put them up the chimney—and he gets them—and whatever you ask for he'll bring you—if you're good!" Instantly the engine was a little boy again all a-tingle with this new delicious mystery of Christmastide. He climbed up into the rocking chair and listened, the rapt look on his face deepening. In proof of what she told, Libby had a letter all written and addressed, ready to send. One of the older girls had helped her with it at noon, and she had spent the entire afternoon recess copying it. Because she was just learning to write, she made so many mistakes that it had to be copied several times. She read it aloud to Will'm. "Dear Santa Claus:—Please bring me a little shiny gold ring like the one that Maudie Peters wears. Yours truly, Libby Branfield." "Now you watch, and you'll see me send it up the chimney when I get my muddy overshoes off and my hands washed. This might be one of the times when he'd be looking down, and it'd be better for me to be all clean and tidy." Breathlessly Will'm waited till she came back from the kitchen, her hands and face shining from the scrubbing she had given them with yellow laundry soap, her hair brushed primly back on each side of its parting and her hair ribbons freshly tied. Then she knelt on the rug, the fateful missive in her hand. "Maudie is going to ask for 'most a dozen presents," she said. "But as long as this will be Santy's first visit to this house I'm not going to ask for more than one thing, and you mustn't either. It wouldn't be polite."
- 72. "But we can ask him to bring a ring to Dranma," Will'm suggested, his face beaming at the thought. The answer was positive and terrible out of her wisdom newly gained at both church and school. "No, we can't! He only brings things to people who bleeve in him. It's the same way it is about going to Heaven. Only those who bleeve will be saved and get in." "Dranma and Uncle Neal will go to Heaven," insisted Will'm loyally, and in a tone which suggested his willingness to hurt her if she contradicted him. Uncle Neal was "Dranma's" husband. "Oh, of course, they'll go to Heaven all right," was Libby's impatient answer. "They've got faith in the Bible and the minister and the heathen and such things. But they won't get anything in their stockings because they aren't sure about there even being a Santa Claus! So there!" "Well, if Santa Claus won't put anything in my Dranma Neal's stocking, he's a mean old thing, and I don't want him to put anything in mine," began Will'm defiantly, but was silenced by the sight of Libby's horrified face. "Oh, brother! Hush!" she cried, darting a frightened glance over her shoulder towards the chimney. Then in a shocked whisper which scared Will'm worse than a loud yell would have done, she said impressively, "Oh, I hope he hasn't heard you! He never would come to this house as long as he lives! And I couldn't bear for us to find just empty stockings Christmas morning." There was a tense silence. And then, still on her knees, her hands still clasped over the letter, she moved a few inches nearer the fireplace. The next instant Will'm heard her call imploringly up the chimney, "Oh, dear Santa Claus, if you're up there looking down, please don't mind what Will'm said. He's so little he doesn't know any better. Please forgive him and send us what we ask for, for Jesus' sake, Amen!"
- 73. "Oh, dear Santa Claus" Fascinated, Will'm watched the letter flutter up past the flames, drawn by the strong draught of the flue. Then suddenly shamed by the thought that he had been publicly prayed for, out loud and in the daytime, he ran to cast himself on the old lounge, face downward among the cushions. Libby herself felt a trifle constrained after her unusual performance, and to cover her embarrassment seized the hearth broom and vigorously swept up the scraps of half-dried mud which she had tracked in a little while before. Then she stood and drummed on the window pane a long time, looking out into the dusk which always came so surprisingly fast these short winter days, almost the very moment after the sun dropped down behind the cedar trees. It was a relief to both children when Grandma Neal came in with a lighted lamp. Her cheerful call to know who was going to help her set the supper table, gave Will'm
- 74. an excuse to spring up from the lounge cushions and face his little world once more in a natural and matter-of-course way. He felt safer out in the bright warm kitchen. No stern displeased eye could possibly peer at him around the bend of that black shining stove- pipe. There was comfort in the savory steam puffing out from under the lid of the stew-pan on the stove. There was reassurance in the clatter of the knives and forks and dishes which he and Libby put noisily in place on the table. But when Grandma Neal started where she had left off, to finish the story of the Camels and the Star, he interrupted quickly to ask instead for the tale of Goldilocks and the Three Bears. The Christmas Spirit had gone out of him. He could not listen to the story of the Star. It lighted the way not only of the camel caravan, but of the Sky Road too, and he didn't want to be reminded of that Sky Road now. He was fearful that a cold displeasure might be filling the throat of the sitting-room chimney. If Santa Claus had happened to be listening when he called him a mean old thing, then had he ruined not only his own chances, but Libby's too. That fear followed him all evening. It made him vaguely uncomfortable. Even when they sat down to supper it did something to his appetite, for the dumpling stew did not taste as good as usual.
- 75. I CHAPTER II T was several days before Will'm lost that haunting fear of having displeased the great power up the chimney past all forgiveness. It began to leave him gradually as Libby grew more and more sure of her own state of favor. She was so good in school now that even the teacher said nobody could be better, no matter how hard he tried. She stayed every day to help clean the blackboards and collect the pencils. She never missed a syllable nor stepped off the line in spelling class, nor asked for a drink in lesson time. And she and Maudie Peters had made it up between them not to whisper a single word until after Christmas. She was sure now that even if Santa Claus had overheard Will'm, her explanation that he was too little to know any better had made it all right. It is probable, too, that Will'm's state of body helped his state of mind, for about this time his cold was well enough for him to play out of doors, and the thought of stars and angels and silver bells began to be agreeable again. They gave him that big, warm feeling inside again; the Christmas feeling of good-will to everybody. One morning he was sitting up on a post of the side yard fence, when the passenger train Number Four came rushing in to the station, and was switched back on a side track right across the road from him. It was behind time and had to wait there for orders or till the Western Flyer passed it, or for some such reason. It was a happy morning for Will'm. There was nothing he enjoyed so much as having one of these long Pullman trains stop where he could watch it. Night after night he and Libby had flattened their faces against the sitting-room window to watch the seven o'clock limited pass by. Through its brilliantly lighted windows they loved to see the passengers at dinner. The white tables with their gleam of glass and shine of silver and glow of shaded lights seemed wonderful to them.
- 76. More wonderful still was it to be eating as unconcernedly as if one were at home, with the train jiggling the tables while it leaped across the country at its highest speed. The people who could do such things must be wonderful too. There were times when passengers flattening their faces against the glass to see why the train had stopped, caught the gleam of a cheerful home window across the road, and holding shielding hands at either side of their eyes, as they peered through the darkness, smiled to discover those two eager little watchers, who counted the stopping of the Pullman at this Junction as the greatest event of the day. Will'm and Libby knew nearly every engineer and conductor on the road by sight, and had their own names for them. The engineer on this morning train they called Mr. Smiley, because he always had a cheerful grin for them, and sometimes a wave of his big grimy hand. This time Mr. Smiley was too busy and too provoked by the delay to pay any attention to the small boy perched on the fence post. Some of the passengers finding that they might have to wait half an hour or more began to climb out and walk up and down the road past him. Several of them attracted by the wares in the window of the little notion shop which had once been a parlor, sauntered in and came out again, eating some of Grandma Neal's doughnuts. Presently Will'm noticed that everybody who passed a certain sleeping coach, stooped down and looked under it. He felt impelled to look under it himself and discover why. So he climbed down from the post and trudged along the road, kicking the rocks out of his way with stubby little shoes already scuffed from much previous kicking. At the same moment the steward of the dining-car stepped down from the vestibuled platform, and strolled towards him, with his hands in his trousers' pockets. "Hullo, son!" he remarked good-humoredly in passing, giving an amused glance at the solemn child stuffed into a gray sweater and blue mittens, with a toboggan cap pulled down over his soft bobbed hair. Usually Will'm responded to such greetings. So many people
- 77. came into the shop that he was not often abashed by strangers. But this time he was so busy looking at something that dangled from the steward's vest pocket that he failed to say "Hullo" back at him. It was what seemed to be the smallest gold watch he had ever seen, and it impressed him as very queer that the man should wear it on the outside of his pocket instead of the inside. He stopped still in the road and stared at it until the man passed him, then he turned and followed him slowly at a distance. A few rods further on, the steward stooped and looked under the coach, and spoke to a man who was out of sight, but who was hammering on the other side. A voice called back something about a hot-box and cutting out that coach, and reminded of his original purpose, Will'm followed on and looked, likewise. Although he squatted down and looked for a long time he couldn't see a single box, only the legs of the man who was hammering on the other side. But just as he straightened up again he caught the gleam of something round and shiningly golden, something no bigger than a quarter, lying almost between his feet. It was a tiny baby watch like the one that swung from the steward's vest pocket. Thrilled by the discovery, Will'm picked it up and fondled it with both little blue mittens. It didn't tick when he held it to his ear, and he couldn't open it, but he was sure that Uncle Neal could open it and start it to going, and he was sure that it was the littlest watch in the world. It never occurred to him that finding it hadn't made it his own to have and to carry home, just like the rainbow-lined mussel shells that he sometimes picked up on the creek bank, or the silver dime he had once found in a wagon rut. Then he looked up to see the steward strolling back towards him again, his hands still in his trousers' pockets. But this time no fascinating baby watch bobbed back and forth against his vest as he walked, and Will'm knew with a sudden stab of disappointment that was as bad as earache, that the watch he was fondling could never be his to carry home and show proudly to Uncle Neal. It belonged to the man.
- 78. "Here!" he said "Here!" he said, holding it out in the blue mitten. "Well, I vow!" exclaimed the steward, looking down at his watchfob, and then snatching the little disk of gold from the outstretched hand. "I wouldn't have lost that for hardly anything. It must have come loose when I stooped to look under the car. I think more of that than almost anything I've got. See?" And then Will'm saw that it was not a watch, but a little locket made to hang from a bar that was fastened to a wide black ribbon fob. The man pulled out the fob, and there on the other end, where it had been in his pocket all the time, was a big watch, as big as Will'm's fist. The locket flew open when he touched a spring, and there were two pictures inside. One of a lady and one of a jolly, fat-cheeked baby. "Well, little man!" exclaimed the steward, with a hearty clap on the shoulder that nearly upset him. "You don't know how big a favor you've done me by finding that locket. You're just about the nicest
- 79. Welcome to our website – the perfect destination for book lovers and knowledge seekers. We believe that every book holds a new world, offering opportunities for learning, discovery, and personal growth. That’s why we are dedicated to bringing you a diverse collection of books, ranging from classic literature and specialized publications to self-development guides and children's books. More than just a book-buying platform, we strive to be a bridge connecting you with timeless cultural and intellectual values. With an elegant, user-friendly interface and a smart search system, you can quickly find the books that best suit your interests. Additionally, our special promotions and home delivery services help you save time and fully enjoy the joy of reading. Join us on a journey of knowledge exploration, passion nurturing, and personal growth every day! ebookbell.com