Creating effective internal processes for quality assessment,managent anddevelopment
- 1. 6 Assessment Update • November–December 1999 • Volume 11, Number 6 M ANY COUNTRIES HAVE IN- troduced or revised their proce- dures for external scrutiny of institutional arrangements for quality assurance in higher education, and for assessment of the quality of specific edu- cational programs. Reactions within the academy range from frustration verging on anger through reluctant compliance to acceptance that some improvements may be necessary and beneficial. Nevertheless, the worldwide trend toward higher rates of participation, especially when aligned with the knowledge and information rev- olutions, is transforming the higher edu- cation scene. Increasingly, external scrutiny is being justified on these grounds, that is, on the grounds of in- creased complexity and diversity. Many institutions, at least initially, re- sponded to the external forces by focus- ing on the immediate task, that is, on the impending accreditation, quality assess- ment, or quality audit. The strategic re- sponse has shifted progressively toward reviewing, revising, and strengthening in- ternal policies, processes, and procedures. Not only should this focus reduce the strain of external scrutiny, but it is also perceived as a means of redressing the balance and enabling institutions to play a more positive and active role in the de- bate and in the development of practices. Creating Effective Institutional Cultures Creating effective institutional policies, processes, and procedures for quality as- sessment, assurance, and enhancement will be facilitated if attention is paid to the culture of the institution. The thesis is that institutions with a strong tradition of de- volved responsibility, and with shared vi- sions of agreed-upon goals and means of achieving them, differ from those where the ethos is either strongly centralist or determinedly individualistic. Yet the ex- pectation, in Britain and elsewhere, is that whatever the institutional ethos, prevail- ing culture, or preferred style of manage- ment is, robust internal systems for setting, monitoring, reviewing, and en- hancing arrangements will operate effi- ciently, effectively, and expediently. More- over, these systems should address a broad, common agenda to satisfy exter- nal reviewers. Regardless of which approach it fa- vors, the institution needs to secure ad- herence. Adherence, in turn, depends on factors such as involvement, commitment at various levels, clarity of communica- tion, allocation of responsibilities, rein- forcement, and support. The dual, even triple, allegiance of faculty to their acad- emic guild, department, and institution introduces further complexity and adds subtle nuances to some of the determin- ing factors. Yet the challenge revolves around a small number of key issues, such as the following: • Do faculty accept the need for the policies, procedures, and processes? • Do they believe that the policies, procedures, and processes are work- able and that they produce worthwhile benefits? • Are the policies, procedures, and processes framed in an acceptable manner, including being in an accept- able language? • Do they fit with the way we do things here? If not, how is any culture change determined, defined, and progressed? Illustration The foregoing discussion indicates that there are several possible ways of creat- ing effective institutional approaches to quality assessment, management, and en- Creating Effective Internal Processes for Quality Assessment, Management, and Development George Gordon This focus on reviewing, revising, and strengthening internal policies, processes, and procedures not only should reduce the strain of external scrutiny, but it is also perceived as a means of redressing the balance and enabling institutions to play a more positive and active role in the debate and in the development of practices. ᪓
- 2. Assessment Update • November–December 1999 • Volume 11, Number 6 7 (continued on page 13) hancement. I now focus on one illustra- tion and approach, namely the case of the University of Strathclyde in Glasgow. This example is offered partly because I know it well, having worked at the Uni- versity of Strathclyde for more than thirty years, and partly because in recent years I have been closely involved in the devel- opment of the university’s quality assur- ance system. Strathclyde is also one of the five European universities that Burton Clark used as case studies for his recent book, Creating Entrepreneurial Universi- ties (1998). Designated as a university in 1964, the University of Strathclyde’s origins can be traced back to the end of the eighteenth century when a professor of the Univer- sity of Glasgow expressed in his will that a new university be established in the city as a “place of useful learning.” That motto still guides the University of Strathclyde. There are now five faculties: arts and so- cial sciences, business, education, engi- neering, and science. The university has more than 16,000 full-time students. An- other 40,000 participate, both on campus and at a distance, in programs of lifelong learning. In the late 1980s, Strathclyde adopted a devolved management structure for fi- nancial and academic affairs, with the five faculties being given powerful roles. Cen- tral steering came from the group of se- nior officers, from the senior forums of the institution (from the senate as the aca- demic body and from the court as the governing body), and from a small num- ber of new committees. Foremost among these committees was the university man- agement group (the senior officers of the university, the senior lay officers of the court, the five deans, and the student pres- ident). This body handles the day-to-day management of the university and formu- lates policy papers for consideration by the senate and the court. Of more specific relevance to quality assurance was the establishment in the early 1990s of the Academic Quality As- surance Group. This small committee, chaired by the director of academic prac- pected to act on major items for im- provement within agreed-upon (and usu- ally short) time lines. In every institution of higher educa- tion in Britain, these experiences have influenced the evolution of effective in- ternal processes and policies for quality assurance, management, and develop- ment. At Strathclyde, quality assessment harmonized broadly with the culture of devolved responsibility, although atten- tion had to be paid to items such as good communication, consistency of practice, monitoring and evaluating policies and practices, and learning from experience. Departments led and owned the process, with the relevant academic vice dean, faculty subcommittees, and the Academic Quality Assurance Group providing ad- vice and support. In general, the subtle interplay of cen- tral steering and devolved responsibility is heightened when the focus of attention switches to institutional policies, proce- dures, and practices, that is, the province of quality audit. The academic vice deans have provided the vital conduit that en- ables a two-way flow of ideas, issues, and developments. Through this process of it- eration, the university developed a series of guidelines and codes of practice cov- ering a wide range of aspects of the aca- demic endeavor. These were assembled into a single document, Teaching and Learning: A Guide to University Policy and Procedures, which was approved by the senate in December 1997. This docu- ment and the detailed analytical account (also approved by the Senate in Decem- ber 1997) that constituted the formal self- assessment for the quality audit in 1998 drew together the evolutionary work that tice, consists of the academic vice dean of each faculty, the academic registrar, the student president, and the vice princi- pal. The senate’s response to the first quality audit report on Strathclyde had suggested a reexamination of the justifi- cation for interfaculty diversity of policy and practice. That view reflected the prevailing ethos at Strathclyde, that is, an ethos of sub- stantial responsibility devolved to de- partments and individual academics. An- other historic feature was the sizeable proportion of the university’s qualifi- cations recognized by professional in- stitutions and subject to their recurrent accreditation procedures. When external quality assessment and audit were intro- duced, Strathclyde responded by building on existing traditions and practices and deliberately adopting approaches that matched the evolving institutional cul- ture. In Scotland, the initial cycle of qual- ity assessments has been completed. At Strathclyde, between 1993 and 1998 vir- tually every academic program was ex- ternally assessed. Additionally, the whole institution has been quality audited on two occasions, most recently in March 1998. Audit and assessment reports are published. Both processes are guided by clear frameworks. Institutions or subject areas, as appropriate, are required to submit a critical self-assessment. There is then a visit by a group of assessors or auditors to test practice and discuss policies, pro- cedures, and practices with samples of staff and students. Assessors observe teaching and look at student work. Re- ports are drafted by the auditor-assessors and published, and institutions are ex- Creating effective institutional policies, processes, and procedures for quality assessment, assurance, and enhancement will be facilitated if attention is paid to the culture of the institution. ᪓
- 3. Assessment Update • November–December 1999 • Volume 11, Number 6 13 least one hundred students are re- quired for testing. When one hun- dred students are tested, at least ten students are assigned to each task. This helps to ensure adequate cover- age of skills and subskills and en- hances the reliability of group scores. ETS notes that if separate scores are to be reported for subgroups (such as for females and males or for busi- ness and education majors) there must be at least one hundred stu- dents in each subgroup. Although assessments that rely on constructed-response formats generally produce lower-reliability coefficients than traditional multiple choice or recognition measures, ex- perience with Tasks indicates that acceptable levels of agreement can be achieved when scoring proce- dures carefully follow the scoring rubrics developed by ETS. The Core Scoring Manual provides detailed instructions for planning and con- ducting local scoring sessions. Other scoring options include local scoring with an ETS consultant and scoring by ETS. Additional information about the effective use of Tasks is presented in Peter Ewell’s book A Policy Guide for Assessment: Mak- ing Good Use of the Tasks in Criti- cal Thinking. This book is available from ETS. s Gary R. Pike is assistant vice chancellor for student affairs and director of student life studies at the University of Missouri– Columbia. had taken place in the 1990s and collated it in a form that could be communicated widely within the university community. As a consequence of the developmen- tal process, changes have happened at all levels. In departments, long-standing practices and processes of monitoring, evaluation, and review have been strength- ened and made more explicit. Objectives have been specified, and innovation and experimentation have been encouraged. Regular reports are made to the appropri- ate committee at the faculty level on a wide variety of quality-related issues, and the dean of each faculty reports annually on quality assurance to the university management group. Faculties have chosen to strengthen the roles of key committees and have added new ones, such as subcommittees for teaching and learning. Likewise, the roles and remits of central university commit- tees and groups have been amended and extended to sharpen their effectiveness and capacity to demonstrate connectivity across the institution’s system for quality assurance. All of this has occurred without diluting the aim of a people-based system that is effective, insightful, and lean. The 1998 quality report praised the effectiveness of communication and the involvement of the whole academic com- munity in the university’s quality assur- ance system. Next, attention is turning to a fairly substantial agenda of improve- ments and enhancements that will need to be moved forward to maintain that posi- Internal Processes (continued from page 7) tion. The journey will continue to involve widespread participation and two-way di- alogues that use the organizational struc- ture of the institution and match the shared culture. Of course, it is not just a matter of dealing with the agenda set by the latest scrutiny. Internal quality sys- tems also have to be responsive to the nu- merous changes that are affecting higher education, and sufficiently robust to sat- isfy the expectations associated with new developments, such as those that have re- cently emanated from the QAA. Com- mitment, communication, involvement, reflection, and experimentation are likely to be key ingredients of effective devel- opmental strategies. s References Clark, B. R. Creating Entrepreneurial Universities: Organisational Pathways of Transformation. Oxford, England: IAU Press, Pergamon, 1998. Quality Assurance Agency. “Quality Assurance: A New Approach.” Higher Quality 1998, 4, 2–12. This article draws on a paper the author presented at the AAHE Assessment Conference in Cincinnati, June 1998. George Gordon is professor and director of academic practice at the University of Strathclyde, Glasgow, Scotland. It is not just a matter of dealing with the agenda set by the latest scrutiny; internal quality systems also have to be responsive to the numerous changes that are affecting higher education, and sufficiently robust to satisfy the expectations associated with new developments. ᪓
- 5. Copyright of Assessment Update is the property of John Wiley & Sons, Inc. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.