2.peer evaluation

The current issue and full text archive of this journal is available at
www.emeraldinsight.com/1463-5771.htm

BIJ
18,4 Peer evaluation to develop
benchmarking in the public sector
Lucio Cappelli
490 ` `
Facolta di Economia, Universita di Cassino, Cassino, Italy
Roberta Guglielmetti
`
Universita di Roma, Rome, Italy, and
Giovanni Mattia, Roberto Merli and Maria Francesca Renzi
`
Universita degli Studi Roma Tre, Rome, Italy

Abstract
Purpose – The urgency to strengthen the effectiveness and efficiency of public administrations has
led to the adoption in the public sector – in Italy as well as in other countries – of tools and models
inspired by the total quality management (TQM) approach, such as the common assessment
framework (CAF). A parallel need was felt within public structures to train people, the “peers”, who
aside from being able to implement the self-evaluation activities of their own administration, also
needed to be equipped with the necessary skills to complete external evaluation activities, namely
“peer evaluations”, with the ultimate aim of spreading quality management culture and best practices
in the field through an approach based on benchmarking. The purpose of this paper is to present the
results of a survey designed to determine the training requirements that a “peer” should acquire in
order to perform “evaluation” activities.
Design/methodology/approach – The paper has strong empirical connotations and is essentially
based on, first, a questionnaire given to evaluators/self-evaluators to identify the problems emerging
with the application of the CAF model in administrations that have adopted it. In this sphere a great deal
of attention has been paid to – in addition to the preparation of the questionnaire – the delicate task of
codifying the answers to open questions proposed to interviewees; and second, the investigation
inherent in the current educational choices in Italy in the TQM arena, specifically addressing the public
sector.
Findings – Apart from the analysis of data obtained from the investigation, presented with
descriptive statistics, the paper identifies the training content necessary: first, to place “peers” in a
position to be able to autonomously and fully carry out their evaluation work on the basis of the CAF
model; and second, to render the evaluation activities systematically comparable among the various
administrations.
Research limitations/implications – At the present time the research deals with the Italian
context. However, future investigations involving different European countries could be carried out,
taking the present results as a starting point for a benchmarking activity.
Practical implications – The paper puts forward the creation of a “network” of actors who manage
all the “peer evaluation” activities, from the provision of the training pack and monitoring the “peer”
exchange process, up to benchmarking initiatives and the dissemination of best practices.
Originality/value – The paper presents original data and information in order to identify the
training needs of individuals that actually use CAF, starting from the assumption that the training of
evaluators is the primary condition for promoting the adoption and diffusion of CAF in Italian public
administrations and thus maximizing its benefits.
Benchmarking: An International
Journal Keywords Peer evaluation, Benchmarking, Public administration, Common assessment framework,
Vol. 18 No. 4, 2011 Total quality management
pp. 490-509
q Emerald Group Publishing Limited Paper type Research paper
1463-5771
DOI 10.1108/14635771111147605

1. Introduction Peer evaluation
Modernization of the public administration process, which today is still a fundamental
requirement for many countries, is framed within the intense debate that started at the
end of the 1970s, and developed during the 1980s and 1990s, around the problem of
changing government systems and public sector management. A key reference can be
found in the scientific management philosophy known as New Public Management, that
has certainly influenced – in different ways, to both acclaim and criticism – the Public 491
Administration change processes of several countries, notably Anglo-Saxon countries
and first amongst these the UK, in turn drawing research inspiration from the practical
experiences of these same countries (Boston et al., 1996, Denhardt, 1981, Dunleavy et al.,
2006, Osborne and Gaebler, 1993, Simon, 1976). In Italy, the start of the debate on Public
Administration reforms was greatly delayed with respect to the precursory countries:
for example, we recall that precisely in the 1980s – while other countries were
undertaking harsh structural reforms of their public sectors – the Italian public debt
explosion was talking place. Reinforcing the urgency of a radical change of the complex
Italian Public Administration was brought about the European unification process and
especially participation to the single currency, with the resulting constraints related to
debt and public deficit. These problems, which a good part of European countries share
with Italy, have been brought to the fore by some recent events (think of the Greek crisis
and the difficult situations in Portugal and Spain). In this context, numerous instruments
have been identified as useful references for the innovative management of public sector
organizations. Amongst these several models inspired by total quality management
(TQM), such as common assessment framework (CAF, to be discussed extensively
below), have become widespread.
This present work, which has as its central theme several issues specifically linked to
the use of CAF, draws its inspiration from two surveys developed by the authors: it thus
has a strong empirical connotation and the aim of identifying the key factors that can
promote the diffusion of the CAF instrument – such as internal and external
benchmarking activities aimed at improving public organizations – largely but not
exclusively referring to Italy. Both these objectives, as will be ascertained, can be
attained by means of the “peer evaluation” instrument.
In wanting to find an initial and concrete public act to motivate the process of change
in the Italian public sector we should probably start with Law no. 59 of March 15, 1997,
the so-called “Bassanini Law” that urges institutions to be more responsible towards
citizens in terms of transparency, efficiency and effectiveness. The aim of the process is
to strengthen awareness that the competitiveness of a country depends on the quality of
its public administration. In particular, through the concept of quality, the public utility
process supplied to citizens is linked to the organizational and managerial performance
of institutions. This is demonstrated on one side by the creation of initiatives leading to
the improvement of the service provided, to the transparency of management, to
rewarding the best performing administrations and, on the other side, by the
development of benchmarking (Camp, 1991) as a tool to compare best practices among
institutions.
The December 19, 2006 Directive “for quality-oriented public administrations” is
linked to this context of transformation and change. It aims to produce a substantial
increase in the number of administrations engaged in quality improvement, both on a
central and local level, with a focus on areas and contexts where citizen satisfaction

BIJ and trust in public administrations is quite low. The Directive promotes the introduction
18,4 of the TQM approach in the self-evaluation process as a standard managerial procedure
in public administrations. Self-evaluation can be defined as a systematic analysis of the
results of the organizations, leading to the identification of their strengths and
weaknesses in order to implement specific improvement actions.
The spread of self-evaluation, considered as the essential requirement for continuous
492 performance improvement and benchmarking, is supported by the adoption of
internationally defined models of excellence such as the EFQM model thoroughly tested
in the public and private sector. Within the public sector, the CAF (2006) is used more
and more often. This is a common European framework leading to the spread of quality
culture in the public sector and originates from the EFQM model. It is composed of nine
criteria, five Enablers and four Results, which can be used by the organization to initiate
the self-evaluation process. CAF is the reference point used by the Department of Public
Function to develop competition in Italy, entitled “Quality in Public Administration
Award”, and is dedicated to all public administrations that apply the CAF model. The
winner is the organization that achieves the highest scores in CAF-based assessments.
CAF is also the reference point used by the Department of Public Function to implement
peer evaluation in the public sector as a natural systematic process. It provides
European institutions with a common system of references and language to carry out
comparisons as well as promoting the definition of plans to improve critical processes in
administrations. The CAF model can therefore be regarded as a powerful tool at the
disposal of public administrations in the context of systematic benchmarking activities.
To activate a successful benchmarking process, institutions need to submit the
results of their self-evaluations for external validation, adopting the most suitable
options according to the needs of the organization, such as certification, participation in
international and national awards and peer-to-peer evaluations.
Peer evaluation is one of the tools available. This kind of activity, expressly indicated
in the Department of Public Function’s Action Plan, promotes the spread of quality
culture through the adoption of appropriate benchmarking policies, benchlearning and
the identification of best practices to be diffused in the public sector. Peer evaluation is
carried out by a taskforce of experts, the peers, who are in charge of evaluating quality
management performance in the organization. In order to be considered external
auditors, peers must have adequate knowledge, skills and preparation on TQM
principles and on the CAF model. Training is therefore a key factor to activate peer
evaluation in the public sector.
This paper presents some results of a research project carried out in cooperation
with the Department of Public Function. The main objectives of the research are:
(1) identification of the training needs of those “peers” that actually have to
implement evaluation activities; and
(2) designing a network of different actors able to assume the management
function – on a strategic, coordination and operational level – of the set of “peer
evaluation” activities.

2. Methodology
As we already mentioned, the paper has empirical bases. In fact, in order to achieve the
research objectives indicated above, two parallel field-investigations were carried out:

(1) The first relies on a questionnaire-based survey; such a tool was administered to Peer evaluation
evaluators/self-evaluators belonging to organizations where CAF was already
adopted, with the aim of defining the sectors where training requirements are
most needed.
(2) The second intended to identify the training offer based on TQM content
existing today and targeted at the public sector.
493
The research endeavours to define a training format aimed at:
.
guaranteeing a common and shared package of knowledge and skills;
.
facilitating the creation of a stakeholder network;
.
consolidating the peer evaluation system; and
.
speeding up the process of change indicated above.
The present paper highlights some of the most interesting results and proposals
deriving from the entire research project, based on the assumption that CAF is
considered the reference model to implement “peer evaluation”.
It is worth recalling that literature on CAF has produced numerous and significant
contributions since the earliest years of the model’s dissemination (2000). However, this
is almost entirely about works dealing with issues such as:
.
research inherent to the spread of CAF in European countries (Staes and Thijs,
2005; Engel and Fitzpatrick, 2003);
.
research on the dissemination of CAF in specific public administration sectors
(Dearing et al., 2006; Dipartimento della Funzione Pubblica, 2006); and
.
interpretation – and opinions – of the model in its concrete application in public
administration context (Heino et al., 2007; Hellenic Ministry of the Interior, 2006;
Conti, 2004).

There are practically no contributions focusing research on actors that carry out
evaluations and self-evaluations with the CAF model with the specific objective of
identifying the knowledge and skills necessary for the implementation of this activity.
Up to today – with specific reference to the situation in Italy, but probably common to
other European countries – training of evaluators has been based on “spot” initiatives,
neither systematically planned nor managed. Understanding the training needs of those
who practice concrete evaluation activities seems to be the first step in bridging this gap
and to establish a systematic process of “creating” evaluators in the context of a more
comprehensive approach based on “peer evaluation”. This paper – through the two
empirical surveys on which it is based – has the objective of defining both an
appropriate training course for “peers” and putting forward the constitution of a
network of public subjects for the management of all “peer evaluation” activities.
The carrying out and some significant results obtained in both investigations –
A and B – are described below.

3. First investigation (A): functioning and results
The first investigation has been developed on a sample of evaluators/self-evaluators,
in order to define the areas where training and support interventions are necessary.
The analysis of training needs is the first step of the entire training process to highlight

BIJ the gaps that a professional/operator needs to bridge to continuously adjust his knowledge
18,4 and skills. This is a basic step in defining each training objective and is pivotal in achieving
effective interventions. In Italy, ISFOL is a point of reference for concepts, models, schemes
and skill exemplification. The proposed model refers to expertise as a dynamic system of
knowledge, skills, attitudes and capabilities: it is achieved not only through knowledge, but
also through “know-how” and “know-how-to-be”. The training objective is to fill the gap
494 between actual skills, knowledge and capabilities and the optimum working standards as
established by the organization. In this research, in order to evaluate this gap, an
investigation among civil servants with direct experience of the CAF model was carried
out. The objectives of the investigation required drafting a questionnaire as the survey tool
to analyze peer evaluators in terms of knowledge, know-how and know-how-to-be.
The questionnaire was structured on the “gap model” and aimed to establish the
gap between the ideal evaluator/self-evaluator and actual knowledge, skills and
relational aspects.
The structure of the questionnaire was based on the following criteria:
. to refer to the knowledge and skills (Boyatzis, 1992; Hay Group, 1993) required of
evaluators/self-evaluators and to criticalities in the use of the CAF model (CAF,
2006);
.
to carry out a critical analysis with appropriate systematic sorting of the
knowledge and skills held by evaluators/self-evaluators; as a result of this
analysis, an initial set of knowledge and skills of reference was proposed to the
interviewees on each of the CAF criteria;
.
to define a picture of the evaluation/self-evaluation experiences acquired by
interviewees;
.
to also duly consider the “immaterial” aspects of the evaluation, such as the
relational aspects that somewhat impact on the outcome of the process;
.
to obtain direct indications on training programmes for evaluators/
self-evaluators, investigating the balance between theory and application in
room and on the job training, live or distance training; and
.
to maximize personal and free contributions of interviewees through the ample
use of open questions.

The questionnaire was composed of six sections with a total of 31 questions: seven of
which were open questions and the remaining were multiple choice and scale questions
(intensity, ratios, semantic differentials). The questionnaire was submitted as a pre-test
to 15 target individuals and some minor optimization adjustments were made as a
consequence.
It should be pointed out that the open questions in reality were more than seven:
a particular question requires in fact an answer (open) for each of the nine criteria of the
CAF model. When taking this into consideration, open questions total 15.
As known, open questions pose significant additional problems for the elaborations
of answers. The main issues arising from open questions are:
.
for people filling in the questionnaire: the difficulty in summarizing answers; and
.
for people analyzing the questionnaire: the difficulties in respect of codification,
risking loosing relevant information.

Nonetheless, on one side the interpretation and codification of open questions requires Peer evaluation
additional work in terms of time and commitment but on the other they are rather
useful to investigate the subject in-depth, thus acquiring ulterior information. In fact,
a large quantity and variety of comments and observations on criticalities/difficulties
encountered by evaluators/self-evaluators was collected. In order to organize the
informative content a number of steps were followed:
(1) differentiate the multiple concepts contained in the same answer in order not to 495
lose the informative content;
(2) homogenize the language used by different interviewees to express similar
concepts;
(3) interpret the answers through synthetic yet representative expressions;
(4) codify answers via a homogeneous classification system, suitable to implement
the statistical descriptive analysis;
(5) eliminate redundancies (overlapping codification); and
(6) group the codified answers into tailor-made macro-categories. Each macro-area
contains those expressions codified with the same meaning.

With this approach it was possible to maximize the amount of information and to
proceed with the statistical analysis of the results in order to establish the critical
difficulties to focus on.
It should be specified that the choice to use descriptive statistical tools is coherent
with the type of information and data that existed. Indeed, it should be stressed that the
main difficulty – which required an intense work of interpretation, homogenization
and classification of answers – was in achieving a simple and instant representation of
a complex situation of a mass of highly heterogeneous information.
The questionnaire was made up of 27 closed questions investigating 6 themes:
knowledge and skills; evaluation experiences; use of the model; improvement plans and
designs; rational aspects; evaluation training programmes.
Opinions were requested on the importance, utility and difficulty of each of the
above-mentioned themes, depending on the specific investigation area. It should be
stressed that the analysis of the results of open and closed questions was performed
collectively in order to maximize the survey’s informative value. The population to
whom the questionnaire was submitted was identified through a Formez database and
was made up of 160 CAF managers and evaluators trained during the two editions of the
“Quality in the Public Administration Award” that took place in Italy in 2005 and 2007.
The survey was submitted via e-mail. A total of 78 questionnaires were compiled and
returned, representing 49 percent of the population. The sample was mainly constituted
of individuals who worked as report writers (41 percent), followed by self-evaluators
(32 percent) and evaluators (26 percent). The interviewees took part in the following
quality training programmes: CAF Evaluator Course (30 percent); ISO 9000 Audit
Course (25 percent); “The paths of Quality” Laboratory projects (15 percent). In the
evaluation team, 71 percent of interviewees worked as team member, while 29 percent
worked as “team leaders”. In accordance with the research aims, simple statistical tools
were adopted with the purpose of defining the spread of evaluation and self-evaluation
approaches within public administrations as much as possible.

BIJ The following statistical instruments were used to analyse the results of open
18,4 questions, following their codification:
.
bar chart, to identify the frequencies of distribution for each codified expression;
.
Pareto chart, to identify priority criticalities/difficulties according to the standard
80-20 rule; and
496 .
cause-and-effect chart (or Ishikawa chart (Ishikawa, 1985)), to represent the
primary and secondary causes of a given effect of the study.

Since in this context it is not possible to provide an overview of the processing of
answers, selection criteria based on a double principle were adopted:
(1) the “object” of the survey principle, the CAF model; and
(2) the principle based on the “role” of interviewees: evaluators and self-evaluators.

It is important to point out that the data analysis is based on interviewee responses.
From the “object” principle point of view, the most relevant element is the particularity
of the CAF model, which can be divided into two parts: as mentioned above, the first
area includes the criteria defined as Enablers, the second criteria are defined as Results.
The Enablers examine the managerial approaches used by the organizations and how
they are applied in order to achieve their planned objectives, whereas the Results focus on
indicators and measurements to demonstrate what the organization is achieving.
Analyzing the Results and comparing them with the planned objectives allows
implementing continuous improvements through innovation and benchlearning processes.
The hypothesis to be verified is whether the model’s dichotomy is confirmed in the
criticalities/difficulties indicated in the survey, so that they are differentiated when
referring to Enablers rather than Results in the CAF model.
From the “role” principle point of view, the key element lies in the different
experiences of interviewees: as evaluators and self-evaluators. Evaluators work in
institutions that they do not belong to, carrying out evaluations starting out from
documentation (the so-called “application”) of the organization to be assessed, and
ending with the so-called “site visit”. Self-evaluators carry out assessment activities in
the institutions they belong to, producing the respective documentation. Of interest is
understanding whether different criticalities/difficulties emerged in the survey and to
what extent they are represented by the two categories of interviewees.
Figures 1 and 2 analyze the data gathered according to the CAF model.
Through a comparison of the two models, it is possible to note that in the case
of Results the criticalities/difficulties are clustered around a few micro-categories.
In particular, the macro-category “measurement indicators” represents 50 percent of the
total, followed by “data analysis” and by “model use” with 15 percent. In the case of
Enablers, the most important macro-categories are “model use” with 25 percent,
followed by “evaluation documents” and “evaluations criteria” with 20 percent and
“quality approach” with 15 percent. It seems clear that in the evaluation of Results the
most relevant aspect in terms of difficulties is linked to the availability of measurements
and/or identification of suitable indicators, followed by the interpretation of data to
analyze. In the evaluation of Enablers, the most relevant factor is the difficulty to apply
the model in the specific context of the organization, followed by the difficulties
regarding evaluation documents, such as drafting and interpreting the application,

100.00 Peer evaluation
240 90.00
80.00

Cumulative percentage
200
70.00
160 60.00
Counts

50.00
120
40.00 497
80 30.00
20.00
40
10.00
0 0.00
se

s

ria

ch

ps

s

e

n

ge

e

t

am

g
si
t

or

m

m
io

in
en
u

hi
oa

ka

vi
ite

at

he

Ti

te
at

n
el

m

ns

ai
pr

lin
ic

te
in
cr

sc

n
od

cu

Tr
io

nd

Si

io
ap

rd
n

el
n
M

do

at

at
io

oo
tI

tio
y

od
el

Figure 1.

u
at

lit

en
n

lc

al
lr

ua

M
io

u

ua

Ev
m
al

na
na

al
at

Main difficulties
Q

re
Ev

Ev

io
u

so

su
al

at
er
Ev

ea

iz

encountered in “Enablers”
rp

an
M
te

rg
In

O

100.00
280 90.00

240 80.00
70.00
200
60.00
Counts

160 50.00
120 40.00
30.00
80
20.00
40 10.00
0 0.00
s

is

e

e

ge

ria

ts

am
or

us

m
ys

en
ka

ite
at

he

te
al

el

m

Figure 2.
lin
ic

cr
sc

n
an

od

cu
nd

tio
n
el
n
M

do
a

tio
ti

tio

ua
at

od

Main difficulties
en

n
D

ua

al
ua

M

tio
m

Ev
al
al

ua
re

encountered in “Results”
Ev
Ev
su

al
Ev
ea
M

the evaluation of the single criteria and the general approach to quality. We should point
out that some participants worked as both evaluators and self-evaluators but were not
taken into account in this context.
The data regarding knowledge (Table I) is consistent with the results of Enablers in
the model, underlining the need for training processes focused on TQM content and
CAF model principles.
Intersecting the data on open questions with some data on closed questions, shown
in Figures 1 and 2, the necessity to provide adequate training on quality, especially
through the creation of appropriate competencies (Table II), is confirmed and
consistent with the above-mentioned CAF Results.
A further in-depth examination of CAF in Table III shows the average difficulties
encountered in each of the nine CAF criteria.
The most relevant information is:

BIJ
List of knowledge Average usefulness (1-10)
18,4
TQM principles and tools 9.13
CAF/EFQM models 9.12
Audit techniques 8.22
Interpersonal relationship management 8.19
498 P.A. Organization 8.18
Communication (internal and external) 8.03
Benchmarking 7.77
Laws on P.A. modernization and simplification 7.64
Management control 7.54
Regulations ISO 9001, 9004, 14001, 19011, SA8000 7.36
IT 6.95
Table I. Statistics 6.88
Average usefulness Marketing 6.14
of knowledge Economic theory 5.92

List of skills Average usefulness (1-10)

Setting-up a system of indicators 8.94
Representation and reading of processes 8.87
Improvement tools and methods 8.81
Audit interview management 8.58
Gathering, reading and quantitative data processing 8.35
Strategic planning 8.18
Evaluation tools for personnel / executives 8.12
Planning and analysis of people satisfaction surveys 7.96
Table II. Project management 7.96
Average usefulness Mapping of personal skills 7.64
of skills Identification of training needs 7.52

List of criteria Average difficulty (1-10)

Company results 6.05
Results on key performances 5.68
Processes 5.12
Citizen/client oriented results 4.96
Personnel results 4.68
Policies and strategies 4.63
Table III. Partnership and resources 4.59
Difficulty of CAF criteria Leadership 4.36
(closed question) Personnel 4.07

.
Criteria 3 “Personnel” is the least difficult to evaluate. This result may be linked
to the fact that some personnel management methods are without doubt used in
all organizations, thus making their evaluation easier.
. Among the five easiest criteria to be evaluated, four belong to factors. This
evidence further strengthens the hypothesis of a clear distinction between

Enablers and Results in the CAF model, consistent with several Peer evaluation
criticalities/difficulties highlighted by the investigation.

The disaggregated data of interviewees, evaluators or self-evaluators is analyzed below.
It is important also in this case to understand whether criticalities/difficulties are
different with reference to the two groups or whether, to the contrary, the “role” is
irrelevant. 499
Figures 3 and 4 report the data on evaluators and self-evaluators.
It is evident that criticalities/difficulties are clearly different in the two groups. The first
three micro-areas for evaluators are “measurement indicators” with 30 percent,
“evaluation documents” with 15 percent and “model use” with 10 percent. On the other
hand the first three macro-areas for self-evaluators are “interpersonal relationships” with
20 percent, “evaluation scheme” and “measurement indicators” with 15 percent. Therefore,
the main difficulties in the evaluation process for evaluators lie in the availability of data
160 100.00
150
90.00
140
130 80.00
120

110 70.00
100
60.00
90
Counts

80 50.00
70
40.00
60
50 30.00
40
30 20.00
20
10.00
10
0 0.00
s

n

ts

e

e

e

a

ch

am

ps

s

is

t
si
or

ge
us

im

m

ri
io

en

ys
hi
oa

vi
ite
at

he

te
at

ka
T

al
el
um

ns
pr
ic

te
lu

cr
sc

n

lin

an
od

tio
nd

io

Si
ap
va

oc

n
n
M

at

a
io

el
la
ti

E

io
D

y

at
lu

od
re
at

lit
en

at

D
va
lu

Figure 3.
lu

ua

M
al
m

va

E
va

on
re

Q
E
su

E

rs

Difficulties for evaluators
pe
ea
M

er
t
In

100.00
180 90.00

160 80.00
140 70.00
120 60.00
Counts

100 50.00
80 40.00
60 30.00
40 20.00
20 10.00
0 0.00
ps e rs ts se h t s s e on ng ffer visit
hi m to en
m
tea el u ac men lysi ge
Ti
m
ati ini
ns he ca m ro e a ka o te
o ns
c
nd
i u
tio
n od pp rov a an l lin alu Tra ning Si
ati o ti a oc M a t ev i
lr
el ati en ion
d
alu al ity Imp Da o de r ia T ra Figure 4.
a lu em luat Ev ite
on va r Qu M
Cr Difficulties for
rs E su va
pe ea E self-evaluators
ter M
In

BIJ and measurements, and in the interpretation of the documents provided by the institution
18,4 to be assessed. Instead, for self-evaluators the main difficulty lies in internal relationships
resulting from the evaluation activity: management of the relationship with executives
and colleagues is quite sensitive and critical for those in charge of collecting proof,
information and data to draft evaluation documents according to the CAF model.
The second macro-category for evaluators is the model’s “evaluation scheme”. It is
500 interesting to note that the macro-category “time” is in fourth position for evaluators,
while it is clearly less relevant for self-evaluators. Therefore, overall, the data
disaggregated into evaluators and self-evaluators confirms the initial hypothesis of a
clear distinction between the criticalities/difficulties of the investigation. This
distinction, as well as the dual structure of the model, needs to be duly considered in
order to identify the basic format of the training content.
From this stratification, we were able to obtain useful information in aid of
designing a customized training format.

4. Second investigation (B): functioning and results
The second investigation focused on the mapping and analysis of the training content
on TQM principles and models for civil servants, leading to defining a scenario to
verify the state of the art in the training offer.
The aim was to verify the current situation on TQM training in public
administrations through the identification of the main training agencies in Italy and
the courses available on this subject.
In detail, the analysis focused on:
.
Identification of the training agencies that thus far offer training courses on these
subjects.
.
Analysis of the training courses offered by universities and Higher Learning
Public Administration Institutions with an in-depth examination of contents
(modules, educational units, hours dedicated to TQM issues).

Attention was focused on two types of offers considered significant for P.A.: 1st and
2nd Level Masters and courses provided by Higher Learning P.A. Institutions.
The research activity was carried out in September 2008 and was designed as follows:
.
setting-up the work methodology and defining the survey’s scope;
.
research of existing TQM training offers in Italian Universities; and
.
research of the existing TQM training offers in P.A. Schools and Colleges.

The starting point was to verify the availability of a database on Italian universities.
Through consulting the Ministry’s web site, and in particular the Statistics Office’s, it
was possible to draft a list of educational offers in Italian universities in the academic
year 2006/2007. However, after a more in-depth analysis, this database was quite
limited due to fact that the Masters are activated on a yearly basis.
A new exercise was carried out to identify the Masters offered by universities and
Higher Learning P.A. Institutions for the academic year 2007-2008. The starting point was
Cineca’s list of Italian universities (www.cineca.it). According to this source, in June 2008,
Italian Universities numbered 84. A census survey was conducted on the universities’ web
sites, intersecting university, faculty and department data available online.

Through this survey of web sites, it was possible to collect basic information on Peer evaluation
Masters and to classify them into a single dataset according to the following
parameters: University, Faculty, Number of 1st Level Masters offered, Number of
2nd Level Masters offered, Masters not available online, Public Administration targets,
Quality contents, Web site, Notes.
With the aim of verifying how quality culture is spread among trainers, a detailed
desk analysis of University Masters was carried out, selecting Masters presenting both 501
of the following characteristics:
.
targeting Public Administration internal personnel (graduates/executives/civil
servants); and
.
examining issues related to TQM.

For these Masters a schema was created with the following characteristics (Table IV).
Up to 4th July 2008, the offer provided by Universities in Italy (public and private)
was 1,166 1st Level Masters and 933 2nd Level Masters, for a total of 2,099 Masters.
In particular, University Masters (1st and 2nd Level) targeted at public administration
and offering contents on quality are 85. It is worth noting that six among those are
MIMAPs (Master in Public Administration Innovation and Management) offered by the
Universities of Cassino, Catania, Foggia, Roma Tre, Roma Tor Vergata, Salerno.
A total of 49 percent of the 85 Masters identified are based in central Italy, while
North and South Italy share the remaining quota.
Figure 5 shows the distribution of the 85 Masters with reference to the Faculties
they belong to; in 14 out of 85 Masters it was not possible to identify the Faculty.
TQM issues are mainly dealt with and offered to Health Sector Public Adminstrations
through the Faculty of Medicine, namely 28 percent of the total, and the Faculty of
Economics with 24 percent. The other faculties show lower percentages.
Out of the 85 Masters, 33 are 1st Level Masters and 50 are 2nd Level Masters (it was
not possible to identify the type of level for 2 of these). Therefore, the 1st and 2nd Level
Masters directed at Public Adminstrations with contents on quality are, respectively,
3 percent and 5 percent of all Masters offered in Italy.
With reference to the 85 Masters, the average cost for the 1st Level is e 3,180 versus e
4,100 for the 2nd Level. With reference to the target, Figure 6 shows that the 85 Masters
are mainly directed at the graduate/executive/civil servant category. About 23 out of
85 Masters specifically target Health Sector professionals: this could be due to the fact
that it is mandatory for health workers to acquire 5 credits per year. Only five Masters
target managers from the School Sector.

University Credits Hours on quality issues

Faculty Duration (in months) Didactic units for each module
Name of Master Places available/total hours Cost
Web address Total modules Target
Master headmaster Details of modules on quality Contacts (mail, telephone) Table IV.
Level Academic year Master classification

BIJ 25
18,4 20

Count 15

10
502
5

0
e

y

es

g

es

y

er

y

w
in

om

in

ar

og

La
th
nc

nc
ic

er

lin

ol
O
Figure 5.
on

ie

ie
ed

ne

ci
ip
sc

sc
Ec
M

gi

So
isc
Faculty offering masters al

n
En

io

rd
ic

at
lit

te
with contents on

uc

In
Po

Ed
quality in P.A.
Faculty

60
50
Counts

40
30
20
10
0
Graduates/
executives/
civil servants

Health workers

School
executives

Graduates and
public sector
professionals
Figure 6.
Target
Target

Strong diversity emerged in the language used to define the specific training contents.
Only for 74 out of 85 Masters was it possible to extract information on the modules
dedicated to quality.
Through interpretation and synthesis, the content diversity was brought back to
12 key categories: Figure 7 shows how contents are spread in the different educational
proposals. It should be pointed out that information on module content was not
available for all 85 Masters and that modules focusing on EFQM/CAF excellence
models are almost entirely non-existent.
The educational offer proposed by P.A. Schools and Colleges (henceforth referred to
as Schools) was simultaneously taken into account with the Masters provided by
Italian Universities.
Owing to the fact that an official list of these Schools is not available, a survey was
carried out using a list of central institutions (Constitutional Bodies, Ministries and
Agencies). Overall, 35 institutions were taken into account and 68 P.A. Schools and
Colleges were identified.
The Schools forming part of the survey can be divided into two different types:
(1) general Schools, providing training for all civil servants; and
(2) specialized Schools, providing training for employees of specific public
institutions.

30 Peer evaluation
25
20
Counts

15
10
5
0
503
s

.

ce

e

ly

m

n

t

..

ip

t

s
t..

en

en
se

el
ic

io

r.
pp

te

sh
ur
ke

od
rv
es

em

m
ct

fo
ys

er
so

Su
ar

se

fa

rn
oc

em
ts

ds
ov

ad
re
m

tis

ve
pr

in

en

ho

Le
pr

nc
an

sa
d

go
d

y

em
an

im

et

lle
an

um

lit

er

em

y
ua
n

ag

lit

ce
m

y
n

-h
io

lit
io

stu

ua
Q

an

Ex
tiv
ng
at

ua
at

Eq
m
ic

Co

ita
ni
iz

Q
un

y
an
an

nt
lit

Figure 7.
m

ua
Pl
rg

ua
m
O

Q
Q

Key contents on quality
Co

Contents

From the analysis emerged that the Schools (general and/or specific) offer 319 courses,
which can be classified as follows:
.
training courses: training activity aimed at strengthening the knowledge of
general or specific content, not directly linked to the attainment of qualifications;
.
qualification courses: training activity aimed at the achievement of a specific
professional qualification;
.
refresher and retraining courses: training activities aimed at the adjustment of
professional knowledge liked to a specific qualification;
.
degree course: training activity recognized by institutions (set up in cooperation
with universities) and granting the attainment of 1st or 2nd Level Degrees;
.
Master: training activity recognized by institutions granting the attainment of
higher education degrees; and
.
conferences, seminars, other activities: spot training activities aimed at
popularizing and spreading specific issues.

It should be highlighted that the survey on courses offered by Schools, and carried out
following the criteria listed above, presents some gaps. This is largely due to the fact
that the courses offered adopt non-uniform presentation criteria.
Figure 8 shows the training offer of Schools according to the above-mentioned
classification. According to the data, 79 percent are training courses while 6 percent are
Masters. The Master category is composed of university masters and training courses
referred to as Masters by the Schools. The latter are different from the former in terms
of duration and attainable qualifications.
A further area of investigation concerned the distribution of courses offered in
comparison with their target. The courses target:
.
top levels (executives, directors, managers, high levels);
.
operative levels (technicians, employees, medium-low levels, officers, doctors,
secretaries, trainers); and
.
top levels and operative levels.

BIJ This information was obtained in 106 out of 319 courses. The results are shown in
18,4 Table V.
As expected, the stratification analysis demonstrated that General Schools focus on
top level training while Specialized Schools target operative levels.
An in-depth analysis was conducted on the contents of courses focusing on quality.
The information available was limited due to the fact that the Schools do not freely
504 disclose their programme contents. Thus, a qualitative approach was adopted, listing
the contents of modules and didactic units. The information collected was arranged
according to key contents in line with the research carried out on university Masters.
The most widespread key contents are marketing and communications, leadership,
human resource management, customer satisfaction, quality improvement,
e-government, organization and processes, strategic planning, quality management
system, quantitative decision-making methods.
Through a comparison of the mapping of Masters offered by Italian Universities
and P.A. Schools and Colleges, the similarity in subjects dealt with became evident.
Furthermore, the absence of initiatives leading to the presentation of topics on
excellence models was confirmed.

5. Research implication
In accordance with the research aims, building a taskforce of people working in the
public sector capable of implementing peer evaluations requires integrating the
information collected in both the explorative and descriptive phases.
Research on training proposals addressed to Public Administrations in Italy
emphasizes the segmentation of the educational offer due to the absence of a common
vision and shared objectives. This results in the impossibility of creating a taskforce of
public employees able to independently spread quality culture in the public sector,
according to a common vision and disengaged from consultancies.
In particular, the first analysis of the needs expressed by public employees taking
part in the survey and with past experience in evaluation and self-evaluations,

Refresher and retraining
2%
2% Degree course
5% 6%
6% Training
Figure 8. Master
Classification of the Qualification
training offered by P.A.
Institution of Seminars, congresses
Higher Learning and other activities
79%

Profile Frequency (%)
Table V.
Targeted attendees Operative 51
of P.A. Schools and Top level 41
Colleges courses Top and operative levels 8

clearly indicates the necessity to outline a basic training programme leading to the Peer evaluation
dissemination of quality culture.
The technical and cultural elements should be combined with the skills required for
teamwork, for developing internal communications and adopting relational models
consistent with the problem solving approach. Generally, management of relationships
was considered the main difficulty in the achieving evaluation and self-evaluation
projects, aimed at the improvement of efficiency and effectiveness in the public sector. 505
The results of the survey allowed us to design a standard training format
customized for peers.
The first characteristic of the training course is to organize it according to a modular
approach, which offers many advantages:
.
From the point of view of those benefiting from training, a modular training course
allows learners to easily select “what” to attend and “when” to do so; it is not
necessary to participate in the entire training proposal, instead selecting the
training modules on the basis of personal ability and the needs of administrations.
.
From the point of view of those providing training, a modular course is essential
to develop and implement the “network”, i.e. the network of different training
agencies appointed to effectively carry out training. If an integrated training
program were devised – indivisible and thus administered by a single entity –
then the network concept would fail. Furthermore, when also taking into account
the desk survey, few would be in a position to provide a global training course.
The modular approach, however, allows different training institutions to
implement the offer and externally provide only a part – one or several models –
of the basic format, without distorting or excessively complicating its specific
nature. Naturally, the different entities would have to activate the training
modules in conformance with their offer – within the limits of the aspects
examined here – to the contents of each module. Only in this way will the
network of training agencies be able to adequately achieve its objectives and to
foster full use and deployment of peer evaluations.

The training format is composed of a general module and various specific modules.
The general module contains training elements that are deemed essential and
indispensable for the implementation of evaluation/self-evaluation activities. Any
training project – intended in the double sense of those who benefit from it and those
who provide it – must foresee the use of the general module. Specific modules are those
devoted to certain and limited aspects relating to evaluation/self-evaluation activities.
They are organized according to the criterion of homogeneity of content and can be
independently provided and used.
In particular, whoever wishes to be a peer evaluator must follow the general basic
module whereas peers who want to prepare themselves in a specialized way could chose to
follow a variety of specific advanced modules. Training would be provided both in-room
and via e-learning. This method allows to guarantee the standardisation of contents and
didactic material. The general module contains topics such as: TQM principles, the CAF
model, the evaluation process and the management of operative and relational aspects.
The specific modules would be concentrated on particular topics in such areas as: process
management, statistical tools and data analysis, leadership and team building, citizen

BIJ and people satisfaction, internal and external communications, benchmarking and
18,4 benchlearning, project management, site visits and audits.

6. Conclusion
The design of a training course organised according to general and specific modules is
the first objective of the research presented here. It has been achieved with the help of
506 two field and desk surveys providing several results: the first allowed identifying the
shortcomings of those who have applied CAF in public administrations in practice and
defining the training content to be offered; the second revealed that currently in Italy
there are no training provisions that satisfy the needs identified.
However, as previously affirmed, the research has as its second objective to
hypothesize the establishment of a “network” able to manage the entire “peer
evaluation” process.
The network, which would have the function of reducing and managing the
complexity and diversity found on the level of training choices, can be conceived on
three different levels to which as many types of actors should correspond:
(1) the strategic level, which would serve as the initiative’s mainstay, called to set
the lines of direction, the levels of involvement and interaction methods of
entities comprising the network;
(2) that of coordination, to define the training format and nurture the links between
the various components of the network to maximise the organic and systematic
dissemination of quality culture from a peer evaluation perspective, also
through appropriate monitoring of training effectiveness; and
(3) the operational level, responsible for physically providing the training but in
close connection with the strategic and coordination roles in defining the
training content and provision methods.

In particular, among the subjects that in Italy could usefully contribute to the
achievement of an objective of this type, the following should be considered:
(1) The Department of Public Administration (DFP) in the strategic role, with the
task of setting plans and the direction of activities linked to peer evaluation;
the DFP, which would also act as the initiative’s political engine, would also
have the critical role of assuring weight and authority to the network function,
thus acting as the body responsible for the accreditation of those subjects – in
particular. universities and Public Administration Schools – that could be a
part of it. The Formez, a technical Agency of the Department of Public
Function, would have didactic coordination responsibility and manage all
training aspects.
(2) A university department (i.e. with a business management orientation) in the
coordination role, inserted in the network, for its scientific and didactic
coordination privilege.
(3) Operational and optimization functions in training content:
.
university Masters that have public sector training as their target offer, who
could be called on to insert a training model on the issues mentioned above
among proposed contents;

.
public administration schools, which could have a targeted training Peer evaluation
role aimed at spreading quality culture and peer evaluations within speciﬁc
public realities; and
.
public employees concerned with addressing a personal growth path and
wanting to be active protagonists in the sector’s processes of change.

The system of subjects and relations thus formed, typical of a network, 507
is characterized by:
. non-causal and limited organizational links, but rather systematic and intense;
and
.
presence of weak or totally absent hierarchical relationships.

In this way, all players would interact on a joint plan and indeed their contribution to
improvement would be called on, naturally in compliance with a deliberated and
shared mission and strategy.
The network is a key challenge for Italy, but would also be a signiﬁcant opportunity
to carry out benchmarking and benchlearning processes among the experiences of
different public administrations all over Europe.

References
Boston, J., Martin, J., Pallot, J. and Walsh, P. (1996), Public Management: The New Zealand Model,
Oxford University Press, Auckland.
CAF (2006), Common Assessment Framework, Stampa XPress, Roma.
Camp, R. (1991), Benchmarking. Come analizzare le prassi delle aziende migliori per diventare
i primi, Itaca, Milano.
`
Conti, T. (2004), Qualita: un’occasione perduta?, Etas, Milano.
Dearing, E., Staes, P. and Prorok, T. (2006), Austrian Federal Changellery, Berger & Sohne, ¨
Ges.m.b.H., Horn.
Denhardt, R.B. (1981), “Toward a critical theory of public organization”, Public Administration
Review, p. 41.
`
Dipartimento della Funzione Pubblica – Formez (2006), Percorsi di Qualita, Stampa XPress, Roma.
Dunleavy, P., Margetts, H., Bastow, S. and Tinkler, J. (2006), “new public management is dead –
long live digital-era governance”, Journal of Public Administration Research and Theory,
Vol. 16 No. 3, pp. 467-94.
Engel, C. and Fitzpatrick, S. (2003), “Study on the use of the common assessment framework in
European Public Administration (Editors)”, Synthesis Paper prepared for the 41st Meeting
of the Directors-General of Public Administration, Roma.
Hay Group (1993), Risorse umane e logiche di impresa, Calimero Autore, Milano.
Heino, J., Leva, K. and Tuominen, K. (2007), Common Assessment Framework – CAF 2006,
ChangeManager-Pro, Turku.
Hellenic Ministry of the Interior (2006), Guide on how to implement the Common Assessment
Framework, Hellenic Ministry of the Interior, Athens.
Ishikawa, K. (1985), What is total quality control?, Prentice-Hall, New York, NY.
Osborne, D. and Gaebler, T. (1993), Reinventing Government: How the Entrepreneurial Spirit Is
Transforming the Public Sector, Penguin Books, New York, NY.

BIJ Simon, H.A. (1976), Administrative Behavior – A Study of Decision-making in Administrative
Organization, The Free Press, New York.
18,4 Staes, P. and Thijs, N. (2005), Report on the state of affairs of Common Assessment Framework
(CAF) after five years, Maastricht, Eipascope.

Further reading
508 Bartam, S. and Gibson, B. (1994), Training Needs Analysis: A Resource for Identifying Training
Needs, Selecting Training Strategies and Developing Training Plans, Ashgate Publishing,
Surrey.
Conti, T. (1997), Autodiagnosi Organizzativa, Sperling & Kupfer Editori S.p.A., Milano.
Conti, T. (2007), “A history and review of the European Quality Award Model”, The TQM
Magazine, Vol. 19 No. 2.
Dale, B.G. (1994), Managing Quality, Prentice-Hall Europe, Herthfordshire.
EIPA (2005), “Study on the Use of the Common Assessment Framework in European Public Services”,
Synthesis Paper prepared for the 41st Meeting of the Directors-General of Public
Administration, Roma, European Institute of Public Administration, Maastricht.
Engel, C. (2002), Common Assessment Framework: The State of Affairs, European Institute of
Public Administration, Maastricht.
Formez (1993), Politiche per la formazione nelle PP.AA. locali: indagine sui fabbisogni formativi,
Formez, Napoli.
`
Galgano, A. (2002), I sette strumenti manageriali della qualita totale, Gestione d’impresa e inno il s,
Milano.
ISFOL (2001), La certificazione delle competenze e il riconoscimento dei crediti: una panoramica
nazionale su riforme e innovazioni, ISFOL, Roma.
ISFOL (2006), Fabbisogni professionali e formativi dei lavoratori in somministrazione: modello per
la rilevazione, ISFOL, Roma.
Menzioni, F. (2001), La via dell’eccellenza. Come perseguirla con i metodi e gli strumenti della
`
qualita totale, Franco Angeli, Milano.
`
Montgomery, D.C. (2006), Controllo statistico della qualita, Milano.
Presidenza del Consiglio dei Ministri (1997), Legge Bassanini, n. 59 Delega al Governo per il
conferimento di funzioni e compiti alle regioni ed enti locali, per la riforma della Pubblica
Amministrazione e per la semplificazione amministrativa, 15 March.
Presidenza del Consiglio dei Ministri (2006), Dipartimento della Funzione Pubblica, Direttiva del
Ministro per le riforme e le innovazioni nella pubblica amministrazione “Per una pubblica
`
amministrazione di qualita”, 16 December.
Presidenza del Consiglio dei Ministri (2006), Dipartimento della Funzione Pubblica, Formez,
Common Assessment Framework, Migliorare unorganizzazione tramite lautovalutazione,
available at: www.eipa.nl
Presidenza del Consiglio dei Ministri (2007), Dipartimento della Funzione Pubblica, Piano di
azione 2007-2010, December 2007.
UNI ISO/TR 10017 (2007), Guida alle tecniche statistiche per la Iso 9001:2000, UNI, Milano.

Web sites
www.cantieri.it
www.cineca.it
www.eipa.nl

www.formez.it Peer evaluation
www.innovazione.gov.it
www.innovazionepa.gov.it
www.italia.gov.it
www.miur.it
www.qualitapa.gov.it
www.sspa.it
www.sspal.it 509

About the authors
Lucio Cappelli is a Full Professor at the University of Cassino, Italy, Department of Economic
Science, where he teaches theory and methods of quality and he is liable for the master innovation
and management in Public Administration. He also teaches TQM in a common Master between
the Faculty of Economics at the University Sapienza of Rome and the FINEC University in
St Petersburg, Russia. He is involved in several European Tempus programmes. His research area
concerns the topic of TQM, in its various facets, with particular interest in the quality aspect
regarding the public sector. He is a member of the Italian Academy of Commodity Science.
Roberta Guglielmetti is a PhD student of Commodity Science. She is interested in quality
management in the private and public sector, quality of the food sector, quality and environment
management system, and corporate social responsibility. In particular, she focuses on citizens’
satisfaction. She is a part of many research projects related to the TQM diffusion. She is a
member of the Italian Academy of Commodity Science and of the Italian Association for Quality
culture.
Giovanni Mattia is a Professor in the Faculty of Economics at the University of Roma Tre.
Currently in charge of the Marketing Research chair, he formerly taught Marketing, Consumer
behavior, and Market Strategies. His research interests concern consumer purchase motivations
and attitudes, customer satisfaction, branding, products and services innovation process. He is a
member of several research projects, at present mainly focused on customer satisfaction
management models for public services.
Roberto Merli obtained a degree in Business Science and was awarded a PhD in Commodity
Science. He now actually works as a researcher at the Faculty of Economics in the University of
Roma Tre. His main research fields are environmental quality management, corporate social
responsibility and quality systems certification. He has been a member of the Italian Academy of
Commodity Science since 1998, and he is author and co-author of more than 45 publications in its
research fields and participates in many private and public research projects. Roberto Merli is
the corresponding author and can be contacted at: r.merli@uniroma3.it
Maria Francesca Renzi is a Full Professor in the Faculty of Economics at the University of
Rome Tre. She teaches Quality management, Corporate social responsibility and Quality
systems: the ISO 9000. She is Director of the teaching staff and Coordinator of the Master Quality
Management in Public Administration. She is interested in the scientific activity concerning
various aspects of quality: quality management in public and private sector, quality of the food
sector, quality and environment management system, and corporate social responsibility. She is
director and co-ordinator of many research projects related to the TQM diffusion. She is a
member of the Italian Academy of Commodity Science.

To purchase reprints of this article please e-mail: reprints@emeraldinsight.com
Or visit our web site for further details: www.emeraldinsight.com/reprints

2.peer evaluation

More Related Content

Viewers also liked (12)

Similar to 2.peer evaluation (20)

More from libfsb (20)

2.peer evaluation