Pfm Measure 2008

Measuring Performance
July 2008

EFFICIENCY UNIT
i
VISION AND MISSION

Vision stateMent

To be the preferred consulting partner for all government bureaux and departments and
to advance the delivery of world-class public services to the people of Hong Kong.

Mission stateMent

To provide strategic and implementable solutions to all our clients as they seek to
deliver people-based government services. We do this by combining our extensive
understanding of policies, our specialised knowledge and our broad contacts and
linkages throughout the Government and the private sector. In doing this, we join our
clients in contributing to the advancement of the community while also providing a
fulfilling career for all members of our team.

This brief was researched and authored by the Research Division, Institute of Public
Administration, Ireland (www.ipa.ie/research). The Research Division provides applied
research services for policy makers in a wide range of public service organisations,
drawing on an extensive network of contacts and experience gained over more than
thirty years.

other efficiency Unit DocUMents
The Efficiency Unit has produced a number of guides on good practice on a wide range
of areas, including outsourcing and contract management. These may be found on the
Efficiency Unit website at www.eu.gov.hk.


Foreword

We have all heard of the consultant jargon which states that ‘what gets measured gets
done’. But quite often, establishing meaningful performance targets and measures
ii
is not a straightforward task. As public managers we believe we are diligently
measuring the performance of our organisations every day. Oversight agencies also
keep us on our toes.

But how good are we at this task?

Performance measurement is an integral part of the process of delivering services to
the community. We have been practising the target-based performance measurement
system for over a decade and we hold ourselves accountable over the use of financial
resources through the Controlling Officers’ Report.

But overseas’ literature and experience show that performance measurement is an
ongoing pursuit not only by the private sector but also by governments. To ensure
that we are in tune with the public at all times, it is of paramount importance that
we continue to adapt and perfect our performance measures so as to reflect the
desired social outcomes that we pledge to achieve.

We know that performance measurement in the public sector is a much more complex
task than that in the private sector. This report resonates with many of our beliefs. It also
recognises the fact that it takes time and resources to compile the necessary indicators
to construe whether a social outcome is achieved or not.

Nevertheless, governments in different parts of the world are fine-tuning their systems
to make them more outcome/output-based than input-based. Setting appropriate
performance measures, establishing challenging and yet achievable targets, and
defining results – these are effective means of focusing our combined efforts to
adhere to social objectives when delivering public services.

We should always remind ourselves to measure the right things, and not just the easy
ones.

Increasingly, governments worldwide are using performance measures and targets,
following consultation with departmental stakeholders. Experience overseas has yielded
mixed results. I hope that this report will assist departments in identifying the most
appropriate opportunities for improving their own regimes.

Comments, feedback and sharing are most welcome.

Head, Efficiency Unit
July 2008


contents 1

execUtiVe sUMMary 2

1. PerforMance MeasUreMent: Key challenges 6

2. iMProVing PerforMance MeasUreMent in BUsiness Planning 7

3. enhancing the MeasUreMent of Policy oUtcoMes 19

4. MeasUring PUBlic sector ProDUctiVity 27

5. ensUring that PerforMance MeasUres are UseD anD UsefUl 31

enDnotes 42

references 44

Executive Summary

A key challenge nowadays is to focus and interests of clients, customers and
2
measurement systems on results. citizens. Business plans should not be
Government policies and programmes internal-looking documents that focus
deliver two main types of results: solely on what managers and staff see as
outputs (the direct products and services important. Measures should be developed
produced) and outcomes (the that address the critical issues surfaced by
consequences of those outputs for citizens as part of the planning process.
society). In terms of this overall focus
on results, four main measurement Having determined a set of measures for
challenges are addressed in this report. inclusion in the business planning process,
These are: improving performance it is useful to check the quality of the
measurement in business planning; proposed performance measures. Good
enhancing the measurement of quality measures are needed if they are
policy outcomes; measuring public to be useful. Various sets of criteria have
sector productivity; and ensuring that been developed for assessing the quality
performance measures are used and of performance measures.
useful.
Business plans and associated performance
iMProVing PerforMance measures are an important part of the
MeasUreMent in BUsiness public accountability process for the
Planning use of public funds. In this accountability
context, some measures may be used
Nowadays, business planning is a properly for individual or programme
vital element in most public service accountability purposes, while other
organisations. An important first step measures may be misleading if used
in developing performance measures for such accountability purposes but
for business plans is to get a clear nevertheless be important for the
understanding of the range of overall management of the programme
performance issues associated with the or activity. Both intermediate and final
area under scrutiny. Public organisations outcome measures, for example, are
in many countries are now using beyond the direct control of staff working
the logic model approach to help in the area. Nevertheless, these outcomes
structure thinking about developing should be reported as part of the business
a range of performance measures planning process. But they should be seen
covering inputs, outputs and outcomes. as contributing to giving an account on
performance rather than being used to
By focusing particular attention on hold staff to account.
outcomes, measures are more likely to
be developed that address the needs


EXECUTIVE SUMMARY

enhancing the MeasUreMent addressing time-lag and attribution
of Policy oUtcoMes issues 3

Whole-of-government and cross- Time-lag and attribution problems are
cutting policy outcome measures significant when developing outcome-
oriented performance measures. Time-lag
Some governments, at both national and issues refer to the fact that it can often be
local levels, have developed measures many years before the full effects of a policy
that have a high-level focus on the final are seen in practice. Attribution issues refer
outcomes of policies concerning social, to the difficulty in attributing causation
economic and environmental issues. with regard to policy outcomes to particular
These measures do not enable tracking of interventions. Changes in infant mortality
individual agency performance, but they rates, to take one example, are influenced
do provide greater clarity on desired high- by a myriad of factors, some associated
level outcomes, and set a context within with health-care practices, some with social
which individual agency performance and economic programmes, and some
measures are developed. with educational programmes. Identifying
the precise contribution of a particular
At both the whole-of-government and programme or agency to such final
sectoral levels, there is increasing attention outcomes is often, in practice, impossible.
on the need to develop measures that cut There are no easy answers to these
across organisational boundaries and focus challenges. But the separation of outcomes
on the social and economic outcomes into intermediate and final outcomes, and
desired by citizens. The interest here is the measurement of each, offers one way
on measures that track and encourage forward for managers. Another approach
joined-up thinking and practice on the part to dealing with attribution is to use impact
of government organisations. For example, evaluation.
policy units in government departments
often have responsibility for developing
and outlining the intermediate and final
outcome objectives for the areas where
they have policy responsibility. While it may
be the task of other delivery agencies to
develop measures and collect information
on performance against the objectives, the
department should report on performance
against these measures as part of its overall
policy responsibility.


EXECUTIVE SUMMARY

MeasUring PUBlic sector the performance of employees and are
4 ProDUctiVity associated, in many instances, with higher
levels of performance than either no, or
The evidence from studies of public general ‘do your best’, targets. But how do
sector productivity measurement is that we know when a ‘good’ target has been
productivity measurement is still in its set? What might a challenging yet realistic
early stages. Despite efforts going back target look like? The identification and use
to the 1980s, the productivity measures of baseline data can be very helpful in this
being produced need to be interpreted context when setting targets.
cautiously. There is also the danger that
over-simplistic use of the measures can engaging citizens in the process of
lead to perverse consequences. It is clear performance measurement
that no single figure of productivity can
be used for public sector activities, unless It is vitally important to develop measures
there is clear and widespread agreement of issues that citizens want measured in
that it is an appropriate measure. A range performance measurement systems. The
of supporting information is needed to question is not whether the customer
measure productivity change. should be engaged in performance
measurement, but how best to engage the
ensUring that PerforMance public. Two issues of central importance
MeasUres are UseD anD are: means of engaging the public and the
UsefUl reporting of performance measures to the
public.
Performance measures need to be seen in
a wider management context if they are Methods of engaging the public include:
to be used and useful. Among the main focus groups; neighbourhood meetings;
issues that need to be addressed when citizen satisfaction surveys; report cards for
placing performance measures in this programme users; web-based discussion
wider context are: linking measures and forums; and web-based surveys. Each
targets; engaging citizens in the process method of engagement has advantages
of performance measurement; and using and disadvantages, and may vary in terms
incentives and sanctions to encourage the of cost. The use of a diversity of approaches
use of performance measures. offers a good way forward.

linking measures and targets Also, the way in which performance
measures are reported back to citizens
Targets are an important element in making affects the degree of engagement of
performance measures useful. Good the public and ultimately the usefulness
targets can lead to enhanced performance. of many performance measures. Issues
Challenging and specific targets improve such as the provision of measures on


EXECUTIVE SUMMARY

a neighbourhood basis and the use of
comparative performance benchmarks 5
can be helpful here.

Using incentives and sanctions to
encourage the use of performance
measures

Performance measures are rarely used
simply on the basis that they have been
produced. Performance measurement is
not just a technical exercise; it has strong
cultural and political components. In such
an environment, incentives and sanctions
(carrots and sticks) can play an important
role in encouraging the use of performance
measures. The commonly used incentives
and sanctions are: legislation; review;
award/recognition; assurance; and
involvement/feedback.


1. Performance Measurement: Key Challenges

A key challenge nowadays for managers is To overcome such difficulties, a results
6
to focus measurement systems on results. and measurement-oriented culture is
The public and politicians are increasingly needed, involving managers and staff
concerned with what is being delivered throughout government. In a supportive
for the public money being used to fund culture, performance measures1 can help
government programmes. They want tell the performance story of a programme
to know that the money is being spent or organisation.
to achieve the purposes for which it was
allocated, but also and equally importantly In terms of this overall focus on results,
that the money is being spent wisely. Hence and to address the issues raised above,
the concern with results. Government four main measurement challenges are
policies and programmes deliver two addressed in this report. These are:
main types of results: outputs (the direct � Improving performance
products and services produced) and measurement in business planning
outcomes (the consequences of those • Enhancing the measurement of
outputs for society). policy outcomes
� Measuring public sector productivity
But a focus on outcomes, while very � Ensuring that performance measures
important, can also be very difficult. It is are used and useful
often hard to connect the performance
of programmes and managers with A particular emphasis is placed on
distant and complex outcomes. There addressing the needs of the citizens.
is a risk of measurement becoming a Ultimately, we need to be sure that we
paper-based exercise that does not really are measuring the results that the citizens
change practice. People may fall back on want to have measured.
measuring what is easiest to measure,
which may not be the most important issue
from the viewpoint of the citizens. Goal
displacement may occur, where measures
themselves become the objectives, taking
attention away from what the programme
should be delivering. There is also often a
tendency to fall back on input measures, as
these are familiar, even though they tell us
only about efficiency, not effectiveness.


2. Improving Performance Measurement in Business
Planning
Public sector organisations are increasingly as the results chain. At the start of the chain
using performance measures as part of are the outputs, results that managers can 7
their business planning process. Divisional largely control, e.g. drafting of legislation or
and work unit business plans are now a giving of grants. At the end of the chain are
common feature in many government the final outcomes, the end results sought,
agencies. In this section, issues that are of e.g. improvements to health, the economy
particular relevance to the development or the environment. And in-between is a
of performance measures for business sequence of intermediate outcomes that
planning are outlined. First, steps to are intended to lead to the final outcomes,
be taken and issues to consider in the e.g. changes in people’s attitudes and
development of performance measures behaviours.
for business plans are presented. Second,
methods of assuring the quality of The United Way of America (1996) notes:
performance measures are reviewed. Third, “A programme logic model is a description
the use of measures for management of how the programme theoretically
and accountability purposes is discussed. works to achieve benefits for participants.
And finally, a specific current issue for the It is the ‘if-then’ sequence of changes that
day-to-day business of many organisations the programme intends to set in motion
– assessment of the performance of through its inputs, activities and outputs.
government websites – is examined. Logic models are useful frameworks for
examining outcomes. They help you
DeVeloPing PerforMance think through the steps of participants’
MeasUres for BUsiness Plans progress and develop a realistic picture
of what your programme can expect to
Using the logic model approach to accomplish for participants. They also help
identify performance measures you in identifying the key programme
components that must be tracked to assess
An important first step in developing the programme’s effectiveness."
performance measures is to get a
structured understanding of the range
of performance issues associated with
the programme2 under scrutiny. Public
organisations in many countries are now
using the logic model approach to help
structure thinking about developing a
range of performance measures covering
inputs, outputs and outcomes. Figure 1
shows the results spectrum produced from
a logic model. This is sometimes referred to


IMPROVING PERFORMANCE MEASUREMENT IN BUSINESS PLANNING

figUre 1 logic MoDel resUlts chain
8

Outputs
(goods and services produced by programme)

Intermediate outcomes
(immediate effects of the outputs)

Intermediate outcomes
(medium-term effects of the outputs)

Final outcomes
(the final or long-term consequences)

An illustrative example of measures derived and management of the organisation and
from using a logic model approach is given service users all have their own information
in Figure 2 – evidential breath testing of needs. The incentive to develop a balanced
drivers to improve road safety. set of performance measures, incorporating
both financial and non-financial measures,
choosing measures that meet stems from trying to meet the needs of
stakeholder needs, particularly these different stakeholders. Initiatives
citizens’ needs such as the logic model and the balanced
scorecard (Kaplan and Norton, 1993) have
An important issue when developing spread through many private and public
performance measures for business organisations in an effort to ensure that
planning is to consider the needs of a performance measures meet the needs of
range of stakeholders when selecting various stakeholders.
measures for business plans. For example,
politicians, central finance officials, staff



figUre 2 Using the logic MoDel to DeVeloP PerforMance
MeasUres: eViDential Breath testing 9

input objective
To ensure that programme and administrative expenditure levels are
contained as agreed

input measures
� Programme expenditure
• Recurrent
• Capital
� Administrative expenditure
• Pay related
• Non-pay related

output objective
To test drivers for evidence of alcohol levels

output measures
� Number of drivers tested

intermediate outcome objectives:
To change the attitudes and behaviour of drivers towards alcohol consumption
To reduce numbers tested who are over the legal limit
To reduce the number of convictions for drink driving offences

intermediate outcome measures
� Survey findings of attitude and behaviour change
� Numbers tested over the legal limit
� Number of convictions for drink driving offences

final outcome objectives:
To reduce alcohol-related road accidents
To improve road safety

final outcome measures
� Number of alcohol-related road accidents
� Number of deaths and injuries on the roads linked to alcohol



A key stakeholder challenge for managers What is needed is some means of involving
is to select performance measures for their citizens in the selection of the measures
10
business plans that measure the issues used in business plans. For example, the
and results that service users and citizens Oregon Progress Board established an
want to see measured. Ho (2007, http:// eight-member advisory committee to give
www.businessofgovernment.org/pdfs/ state agencies a better sense of how citizens
HoReport.pdf ) suggests that this citizen view the state’s current performance
focus is often missing in the traditional measurement system. To do so, four
approach to performance measurement state agencies offered their most recent
development. In the traditional approach, annual performance measure reports for
managers start by asking themselves and members of the advisory committee to
their staff what their vision and objectives read and respond to. This led to initiatives
are. They then develop performance such as more use of comparative data to
measures on the basis of this work. Ho benchmark performance measures.
identifies four main problems with this (See http://www.oregon.gov/DAS/OPB/
approach: docs/kpm/NCCI/NCCIfinal.doc for more
� It assumes managers are capable of details.)
establishing programme vision and
objectives that reflect public concerns A more detailed case of good practice in
and priorities. citizen involvement is given in the case
� Managers may be trapped by their study of the Des Moines (USA) citizen-
own blind spots and fail to see beyond initiated performance assessment project.
what they routinely do in programme
delivery and planning. And the public The Des Moines project was considered
may view the measures as a tool to a success as it led to the development
showcase the achievements of the of performance measures that were
administration. actively used by officials but also were of
� The traditional approach assumes that direct interest to citizens. The measures
performance measures by their own addressed issues that were of concern to
weight can influence how elected citizens and were identified by them as
officials think about effectiveness and very important. But there are limitations
resource allocation. But in practice to this partnership approach. In the Des
politicians may not pay much Moines case, the president of Des Moines
attention to measures if they do not Neighbors (an umbrella organisation for
believe that the measures reflect their neighbourhood groups) and the city
constituencies’ concerns and priorities. manager both stepped down within
� Managers may focus on their own a short period of each other. With the
priorities rather than on how they can loss of two key supporters the project
best collaborate with other agencies lost momentum in 2005. An annual
and the public to deliver results. This performance report continues to be issued,
can emphasise a ‘silo’ mentality rather but the civic engagement process has
than a ‘joined up’ perspective. not been maintained. This illustrates that
maintaining such a partnership approach
requires continuity of commitment, and
that the loss of key individuals who drive
the process may break the continuity.



case stUDy
11
Des Moines citizen-initiateD PerforMance
assessMent Project

Des Moines, Iowa, has a population of about 200,000 people. Between 2001 and 2004
the city implemented a citizen-initiated performance assessment (CIPA) project. Public
officials and neighbourhood representatives jointly developed performance measures.
The process went through several stages:

Topic selection – Participants wanted to measure the quality of life in the city and what
the city had done to improve the well-being of residents. Under the broad heading of
nuisance control, a small number of key issues emerged from the process, e.g. odour
control, noise pollution, traffic law enforcement and clearance of abandoned housing.

Measure selection – For each topic, critical issues were identified and measures
developed to track these critical issues, as illustrated below:

toPic critical issUe PerforMance MeasUres

odour Where are the problems? Number of complaints received
control - by neighbourhood

Are citizens satisfied with the air Satisfaction level of surveyed
quality and the current level of citizens - by neighbourhood
odour control?

Satisfaction of city response Level of satisfaction with the
after a complaint is filed service received after a complaint
was made a month previously

Data collection – Public officials tracked nuisance complaints over time and by
location. They also conducted sample surveys of residents’ satisfaction with city services
and citizens’ perception of quality of life in their residential area. Citizens also directly
participated in digital surveys in selected neighbourhoods, using handheld electronic
devices to document problems such as potholes and graffiti.

Engagement of citizens and other stakeholders in reporting – Performance
measurement results were actively disseminated to the city council and directly to the
public.
Source: Ho, 2007, p.18-23. Performance reports can be accessed at http://www.ci.des-moines.ia.us performancereport.htm



More generally, while having significant assUring the QUality of
12
benefits, involving citizens in the PerforMance MeasUres UseD
development of performance measures in BUsiness Planning
can be a challenging activity, and a number
of issues need to be borne in mind if taking Having determined a set of measures for
this approach: inclusion in the business planning process,
� Managers must be committed to the it is useful to check the quality of the
process and provide leadership to work proposed performance measures. Good
with citizen groups. If this leadership is quality measures are needed if they are to
not there, the process will not be taken be useful. Various sets of criteria have been
seriously. developed for assessing the quality of
� There are risks involved in engaging performance measures. Perhaps the most
citizens in performance measurement. common is to check measures against
Measures may reveal problems that the SMART criteria (specific, measurable,
present challenges to managers and achievable, relevant and time-bound).
elected officials. An illustrative example of this process is
� There is a risk that citizen groups given in Table 1. In the Irish Department
consulted may not be representative of of Agriculture and Food, performance
the wider citizenry. Steps must be taken measures proposed by divisions for their
to ensure that citizen engagement is business plans have been subject to a
comprehensive and inclusive. quality analysis using SMART criteria from
� Citizen engagement is resource- within the economic and planning unit.
intensive, particularly in terms of The benefits of this process in terms of
the time commitment required of improving the specificity of the measures
management and staff. It is also and ensuring there is a link between
demanding of citizens themselves. measures and targets can clearly be seen
� There is no single citizen view of from this example.
performance. More likely there will
be a variety of views expressed as to
the importance of measures selected
and what they mean. Balancing and
interpreting these divergent views
presents challenges to managers.

None of this is to suggest that
citizen engagement in performance
measurement is not important. But
an awareness of the challenges and
limitations can facilitate a more effective
approach to such engagement.



taBle 1 exaMPles of PerforMance MeasUres
analyseD Using the sMart criteria 13

Measure Specific Measurable Achievable Relevant Time-
bound

Level of Relates to In terms What is Relevant Not
animal a specific of number identified to several identified,
diseases item (animal of disease here is an objectives but
disease levels indicator. in the perhaps
levels) Target not statement not
identified, of strategy applicable
therefore
achievability
cannot be
commented
on

Percentage Relates to a In terms of As above As above Time
of specific item percentage criteria
payments (payment within identified
within deadline deadline (protocol
protocol protocols) deadlines)
deadlines

Using MeasUres in BUsiness business plans, are an important part of
Planning: DistingUishing the public accountability process for the
BetWeen ManageMent anD use of public funds. In this accountability
accoUntaBility PUrPoses context, it is important to recognise
that measures may play different roles,
Business planning is intended as part of a depending on the aspect of performance
process of improved management within being measured. Some measures may be
public service organisations. Business used properly for individual or programme
plans are meant to ‘step down’ to the accountability purposes. Other measures
divisional level the high-level goals and may be misleading if used for such
objectives contained in strategic plans. As accountability purposes but nevertheless
such, business plans, and consequently be important for the overall management
the performance measures contained in of the programme or activity. This issue is



best illustrated by way of example, as set changes in these measures directly to the
14
out in Figure 3. programme.

Figure 3 sets out performance objectives Both intermediate and final outcome
and measures used to assess an indicators are beyond the direct control of
anti-smoking television advertisement staff working in the area. But as Schacter
campaign. Input, activity and output (2002) notes: ‘They are nevertheless things
measures in this example can be used that you must monitor, because they have
to help assess individual and team profound relevance to the design and the
performance of staff working in this area. implementation of your program. These
It is the responsibility of the staff working are the results that you are managing
in the area to ensure that the campaign for, even if you can’t control them.’ These
material is designed, tested and runs on outcomes should be reported on as part
the television, within agreed resource of the business planning process. But they
allocations. Staff can be held to account for should be seen as contributing to giving an
this work. account on performance rather than being
used to hold staff to account. Including
The intermediate outcome objectives and outcome indicators such as these is
measures – assessing whether people see important for the successful management
the advertisements and if their attitudes of the programme.
and behaviour are affected – are clearly
beyond the direct responsibility of people
working in the area. But these measures
provide information about the results of
the programme that can readily be directly
attributed to the programme. It is possible
to judge if the advertising campaign has
been used and if it has had an effect on
people’s attitudes and behaviour. As such,
these measures can be used to assess
programme performance and hence
programme accountability.

The final outcome measures – less smoking
and lower incidence of smoking-related
diseases – are affected by many other
factors apart from the campaign itself. These
measures are clearly relevant to judgements
about programme performance, but
it is not possible to directly attribute



figUre 3 Using MeasUres for Different
accoUntaBility PUrPoses 15

inputs $$$
Full Time Equivalent Staff

activities � Design and test campaign material

Fair reflection
performance
of individual
� Produce campaign material
(Measure: material produced by dd/mm/yyyy)
outputs � Advertisements run on television
(Measure: number of schools receiving campaign material)

intermediate � People see the advertisements
outcomes (Measure: number of viewers)

of programme
Fair reflection
performance
� Peoples’ attitudes affected
(Measure: degree of attitude change)

� Behaviour affected
(Measure: degree of behaviour change)

final � Less smoking
performance

outcomes
programme
Relevant to

(Measure: level of smoking)
� Lower incidence of
smoking-related diseases
(Measure: incidence of smoking-related diseases)

Source: adapted from Schacter (2002)



assessing the PerforMance What then is to be done to allow a more
16 of goVernMent WeBsites sophisticated measurement of website
performance? A first step is to be clear
A specific measurement issue, yet about the purposes and objectives of the
one of growing prominence for many website. Is it there to enable people to fill
organisations, is the measurement of the out a survey, use interactive applications
performance of government websites. built into the site, download important
This issue is often dealt with in the context information and so on? In the language
of business planning for an organisation, of the logic model, what are the main
when looking ahead to challenges to be inputs, outputs and outcomes required
addressed in the coming year. Websites have of the website? Measures can then
become a key channel of communication be developed to address these issues
between governments and citizens. As (see Table 2, taken from Stowers, 2004,
such they must address the needs of very http://www.businessofgovernment.
wide-ranging and differing audiences. org/pdfs/8493_Stowers_Repor t.pdf
Usability is crucial. But simply tracking
measures such as the number of hits can The case study, on page 18, highlights
be misleading. A site may get a lot of hits two initiatives to measure website
yet visitors may stay on the site for only a performance, one from Australia and
few seconds, suggesting they did not find one from the USA. Each develops a wide
it useful once there. A site with a smaller variety of measures against which website
number of hits may have people staying performance is judged. Comparative
on for longer and returning frequently, assessment of the websites of different
suggesting that it is meeting a need. government agencies is encouraged.



TABLE 2 POTENTIAL WEBSITE PERFORMANCE MEASURES
17

Input Measures Output Measures Outcome Measures

Application development Number of hits Accessibility of services
and hardware set-up or user contact Number of site pages
Staff costs sessions meeting accessibility
Other development Number of criteria
costs downloads of Accuracy of the assistance
Other vendor costs or information as
documents
measured by percentage
Staff time for Time users spend accuracy rates in random
application development on a site fact checking
Other development time Number of times Adequacy of information
Vendor time for transactions as measured by staff and
development purposes completed, or citizen surveys
the times online Ease of use as measured by
forms have been pop-up or other surveys
accessed and Citizen satisfaction with
completed site
Monetary amounts Service quality
processed through Percentage of time when
website is down and not
each site
available
Webpage errors
Efﬁciency
Cost per transaction
Total cost per user session
Final outcomes
Cost savings from
e-government
Staff time savings from
e-government



case stUDy
18
MethoDs for MeasUring WeBsite PerforMance

assessing federal government websites – australia
The Australian federal government commissioned a private company, UsabilityOne, to
conduct an assessment of ten federal government websites. The company developed a
usability compliance audit for evaluating websites. The audit involves 151 criteria, based
on acknowledged usability principles and extensive user-testing experience. The main
areas covered in the audit include navigation, content and content writing, design and
graphics, search, error prevention and recovery, trust, internationalisation, window titles,
news and press releases plus branding and company information.

Two analysts independently reviewed each website against the criteria, and arrived at an
overall compliance score for each website. Full details of the project can be accessed at
http://www.usabilityone.com/.

the american customer satisfaction index (acsi) methodology
The ACSI was developed by researchers at the University of Michigan to evaluate customer
satisfaction generally, and in 2001 ForeSee Results began to use the methodology to
assess customer satisfaction with federal agency websites. ACSI, which uses a set of well-
researched and benchmarked questions and research processes, uses a causal statistical
model to predict customer behaviour, such as the probability of return visits to the
website. Satisfaction with websites is examined using pop-up surveys.

These results are then analysed using the model, which includes measures both
of customer satisfaction and of customer loyalty. Results for individual websites
can be compared to a nationally developed index based on more than 200
organisations. For more details see http://www.foreseeresults.com/ and Stowers, 2004,
http://www.businessofgovernment.org/pdfs/8493_Stowers_Report.pdf, p.26.


3. Enhancing the Measurement of Policy Outcomes

The use of performance measures to These examples are given, not to suggest
facilitate judgements about the efficiency that measures have no role in policy 19
and effectiveness of policy work presents work, but rather to caution about raising
significant challenges. As Perrin (1998) expectations too highly in terms of the
notes, a requirement to produce more role that measures may play. As part of
policy-focused measures can lead to broader efforts to improve evidence on
situations where there is the widespread performance, measures can be helpful
development of easy-to-count measures for managers and staff engaged in policy
which have little or no relationship to what work.
the policy is supposed to be achieving.
Peripheral activities that are easy to quantify Whole-of-goVernMent
become the focus of reports that nobody anD cross-cUtting Policy
uses. oUtcoMe MeasUres

Such actions as described above arise Some governments, at both national and
in part because of cultural and ‘political’ local levels, have developed measures
reasons, but there are also inherent that have a high-level focus on the final
technical difficulties in ‘measuring’ policy outcomes of policies concerning social,
work. There are a number of particular economic and environmental issues.
issues: These measures do not enable tracking of
� Whole-of-government challenges, individual agency performance, but they
such as multiple high-level outcomes, do provide greater clarity on desired high-
which must be pursued simultaneously level outcomes, and set a context within
and that sometimes are in conflict with which individual agency performance
each other; or where policy is initiated measures are developed. Such high-
in one organisation but responsibility level measurement systems have been
for delivering on outcomes rests with developed because of a recognition of
other agencies. the limitation of relying on economic
� Time-lag problems, associated with measures:
the fact that many years may elapse
between the initiation of a policy and “Economic indicators have traditionally
its implementation. been used to assess the economic ‘state
� Attribution problems, where it of the state’. Strong economic growth,
is impossible to disentangle the low inflation and unemployment were
impact of a particular policy on final regarded as indicative of a healthy
outcomes because these outcomes economic climate and believed to result
are also affected by other policies and in prosperity for citizens. However, citizens
influences. have become increasingly concerned
about their relative quality of life, expressed
in terms such as quality of education and


ENHANCING THE MEASUREMENT OF POLICY OUTCOMES

health care, availability of recreational/ Two longstanding and relatively successful
20
cultural opportunities, clean environment, whole-of-government measurement
and safety from crime. Accounting and systems are Oregon Shines and Alberta’s
economic based measurement systems Measuring Up, examined in the case study
were not designed to address these issues; below.
thus, governments have introduced new
systems for measuring progress, including
policy outcome based performance
measurement (Ogata and Goodkey, 1998).”

case stUDy

Whole-of-goVernMent oUtcoMe MeasUreMent

oregon shines
The state of Oregon, USA, adopted a development strategy in 1989 called Oregon Shines.
The strategy addressed quality-of-life issues as well as economic diversification. Oregon
Shines II was adopted in 1997 as a successor strategy, and Oregon Shines III is planned for
2009. Progress against the strategy is tracked through Oregon Benchmarks, an annual
report which tracks over ninety measures against benchmark targets in seven categories:
economy, education, civic engagement, social support, public safety, community
development and environment. State agencies link their key performance measures to
these benchmarks where it makes sense to do so. For details go to http://www.oregon.
gov/DAS/OPB/docs/obm/New_Benchmark_Numbers.doc.

Measuring Up, alberta
In 1993 the province of Alberta, Canada, started a three-year strategic planning and
performance measurement system. In 1995 the first annual Measuring Up report was
published. Measuring Up contains sixty-five core performance measures related to
fourteen government goals. These measures focus on issues such as infrastructure
capacity, literacy and numeracy rates, crime rate and water quality. The intention is to
develop outcome-based measures to provide information on progress towards long-
term targets. The targets are selected and driven by political leadership, and through
public consultation. State agencies link their business plan objectives to relevant priorities.
For details go to http://www.treasuryboard.gov.ab.ca/1089.cfm



At both the whole-of-government and Additionally, in a critique of earlier PSAs,
sectoral levels, there is increasing attention Neely and Micheli (2004) note that there 21
on the need to develop measures that was little use in the public sector of what has
cut across organisational boundaries and become best practice in the private sector
that focus on the social and economic – developing success or strategy maps
outcomes desired by citizens. The interest when designing measurement systems.
here is on measures that track and Success/strategy maps are intended to
encourage joined-up thinking and practice help people identify which are the critical
on the part of government organisations. measures, and can be helpful when
rationalising the number of measures in use.
The UK government is acknowledged as a
leader in the development of cross-cutting Another common cross-cutting situation
targets and measures, in part through the is that of policy-oriented units operating
development of cross-cutting Performance in government departments who have
Service Agreements (PSAs) (see case study overall responsibility for policy but where
below). It is too early as yet to judge the implementation of that policy is through
success of this initiative. But it has been agencies operating under their control. In
welcomed by the Select Committee on this instance, policy units may sometimes
Treasury, though the Committee cautions not develop measures themselves, saying
that the cross-departmental nature of the that it is the responsibility of the agencies
new PSAs poses a challenge for a system to develop performance measures. The
of accountability based on departmental argument used here, following on from the
reporting and the work of departmental logic model approach, is that policy units
select committees. They also stress that should be responsible for developing and
the new agreements, to be effective, must outlining the programme theory including
genuinely reduce the number of measures the intermediate and final outcome
in use, and not simply bring together objectives. While it may then be the task of
diverse topics within a PSA, and that there the relevant agencies to develop measures
must be corresponding reductions in and collect information on performance
measures down the delivery chain (see against the objectives, the department
http://www.parliament.the-stationery- should report on performance against
office.co.uk/pa/cm200607/cmselect/ these measures as part of its overall policy
cmtreasy/279/27909.htm and http://www. responsibility.
parliament.the-stationery-office.co.uk/pa/
cm200708/cmselect/cmtreasy/55/5507.
htm).



case stUDy
22
UK cross-DePartMental PUBlic serVice agreeMents

In 1998, in the first Comprehensive Spending Review, the UK government introduced
a new performance measurement framework based on Public Service Agreements
(PSAs). While generally welcomed, these PSAs suffered from a number of drawbacks.
The system was seen as top-heavy and unwieldy, with too many measures created at
local level to feed into the system. Particular difficulties arose from targets held jointly
between departments, and some issues that involved more than one department were
not adequately captured.

In the 2007 Comprehensive Spending review, the government announced a smaller
suite of thirty new PSAs that outline the government’s highest priorities and span
departmental boundaries. The agreements are cross-departmental, although there is
a lead department for each agreement. Departments are required to produce a cross-
departmental Delivery Agreement for each PSA, informed by consultation with the
delivery chain. These Delivery Agreements set out how each PSA is to be delivered and
the measures used to track performance against PSAs and their associated delivery.
Agreements can be accessed via http://www.hm-treasury.gov.uk/pbr_csr/psa/pbr_
csr07_psaindex.cfm

An example of this approach is set out number of plans developed, the number
in Table 3. Here, in an Irish example, the of actions implemented from the plans
focus is on the Local Development Social and the quality of the plans (for example
Inclusion Programme (LDSIP), overseen the percentage of plans inspected that fall
by the Department of Community, Rural below a determined quality standard). In
and Gaeltacht Affairs. Implementation this case, the programme logic is that as a
and administration of the programme result of developing high-quality plans, and
is overseen by an agency ADM Ltd. on the subsequent implementation of these
behalf of the Department. An intermediate plans, local groups contribute to sustained
outcome objective for the LDSIP is that social and economic development in their
local groups develop innovative plans communities, which is one of the final
and programmes to improve the local outcome objectives of the community and
social and economic infrastructure. local development programme.
Readily applied measures here are the



With regard to this final outcome objective overall responsibility for the programme, it
of sustained social and economic is important that the Department use these 23
development in communities, Table 3 measures in its own reporting procedures.
gives examples of measures gathered In this way, an overview is provided of
by ADM Ltd. that can be used to track the outcomes of the community and
progress. While ADM Ltd. has responsibility local development programme for public
for developing and gathering the data on accountability purposes.
these measures, as the policy body with

taBle 3 coMMUnity anD local DeVeloPMent MeasUres

interMeDiate oUtcoMes

objective Development by local groups of innovative plans and
programmes to improve the local social and economic
infrastructure

Measures � Number of plans developed
� Quality of plans produced
� Number of actions implemented

final oUtcoMes

objective Sustained social and economic development in
communities, with a special focus on areas of
disadvantage

Measures � Number of people placed into jobs
� Number of people supported into self-employment



aDDressing tiMe-lag anD success of the programme solely on the
24 attriBUtion issUes basis of final outcome measures, though
clearly they provide vital information to
Time-lag and attribution problems are inform an overall judgement. Similarly, the
significant when developing outcome- outputs of the programme – the number
oriented performance measures. Time-lag and value of grants issued – do not on
issues refer to the fact that it can often their own give a satisfactory picture of
be many years before the full effects of performance.
a policy are seen in practice. Attribution
issues refer to the difficulty in attributing This is where intermediate outcome
causation with regard to policy outcomes measures can help. The programme logic
to particular interventions. Changes in is that as a result of receiving supports,
infant mortality rates, to take one example, firms develop their research capability
are influenced by a myriad of factors, some and improve their links with third-level
associated with health-care practices, some research institutes, thus contributing to
with social and economic programmes, the final outcome objectives. Measures
and some with educational programmes. such as level of research skills before and
To identify the precise contribution of after receiving the grant (assessed by
a particular programme or agency to survey) and type and quality of links with
such final outcomes is often, in practice, third-level research institutes can track
impossible. these intermediate outcome objectives.
Measuring the intermediate outcomes
There are no easy answers to these can give a better sense of programme
challenges. But the separation of outcomes performance. And if the programme logic
into intermediate and final outcomes, is demonstrated to be faulty, a new theory
and the measurement of each, offers one can be developed to address the issue.
way forward for managers. Take as an The intermediate outcome measures help
illustrative example the case of a research give a picture of the performance of the
and development (R&D) grant scheme R&D grant scheme. Neither the output nor
aimed at encouraging innovation and the final outcome measures alone could
the development of new products. Final give such a picture. Intermediate outcome
outcome measures, such as number and measures help flesh out the performance
type of new products on the market, and story.
turnover attributed to new products and
services, may be influenced by factors
other than grants given to firms to develop
their R&D capabilities. The national and
international economic situation, local
educational initiatives and so on also have
an impact. It is not possible to examine the



Using impact evaluation (see case study below on increasing tax
revenue in Pennsylvania). By knowing the 25
Another systematic means of developing actual impact of intervention towards the
performance measures that address final outcomes, appropriate performance
attribution and time-lag issues and that may measures can be designed.
be useful in certain circumstances is impact
evaluation. Impact evaluation is defined by
the World Bank (2006) as an assessment
of the impact of an intervention on final
outcomes. It assesses the changes arising
from an intervention that can be attributed
to a particular project, programme or policy

case stUDy

increasing tax reVenUe in PennsylVania

The Pennsylvania state government commissioned a large, randomised control trial
to evaluate the effectiveness of various approaches to improving tax compliance by
businesses that were late in paying their sales taxes. The trial randomly assigned 7,000
such businesses to receive one of seven letters, ranging from threatening to pleading,
and made use of outcome data that the state had already collected for other purposes –
namely, whether the businesses paid their taxes.

The trial found that a letter containing a short (one-third page) statement that tax is due
and that the business is liable produced significantly more tax revenue than the state’s
existing letter (full-page, detailed letter with boxes that the businesses check to indicate
why they have not paid the tax). The trial results indicated that the state’s use of the short
letter for all late-paying businesses could generate US$6 million annually in increased
revenue.

Source: http://www.whitehouse.gov/omb/part/2004_program_eval.pdf, p.12



A key element in impact evaluation is Quasi-Experiments – Like RCTs, these
26
identifying the ‘counterfactual’: What would evaluations assess the differences that
have happened had the intervention not result from an intervention and the result
taken place? This involves the identification that would have occurred without the
of a comparison group who are as alike intervention. However, the control group
as possible to the group who receive the is not randomly assigned. Instead, it is
intervention but who were not subject to designed on the basis that the evaluator
the intervention. The most frequently used judges how to minimise any differences
methods of identifying the counterfactual between the two groups, or it may be a
are the following: pre-existing group. Use of comparison
group studies does increase the risk of
Randomised control trials (RCTs) misleading results because of the difficulty
– An RCT is a study that measures an in eliminating bias in the selection of the
intervention’s effect by randomly assigning control group.
individuals (or other units, such as schools
or hospitals) into an intervention group, The benefits and challenges of impact
which receives the intervention, and into a evaluation are well described by the World
control group, which does not. Following Bank Independent Evaluation Group
the intervention, measurements are taken (http://www.worldbank.org/ieg/ecd/).
to establish the difference between the
intervention group and the control group.
Because the control group simulates what
would have happened if there were no
intervention, the difference in outcomes
between the groups demonstrates the final
outcome or impact one would expect for
the intervention. There are, however, many
programmes for which it would not be
possible to conduct an RCT. To carry out an
RCT, there must be a possibility of creating
a control group who will not receive the
intervention. For practical, legal, and ethical
reasons, this may not always be possible.


4. Measuring Public Sector Productivity

A number of initiatives have taken place papers on public sector productivity
in several countries in recent years aimed measurement, e.g. education and health 27
specifically at improving the measurement sector productivity measurement (see
of public sector productivity. The http://www.statistics.gov.uk/about/data/
experience of a small number of countries methodology/specific/PublicSector/
at the forefront of attempts to improve output/default.asp for details).
the measurement of productivity is
outlined here: the UK, Finland, Sweden and MeasUring PUBlic sector
Australia. ProDUctiVity in finlanD

MeasUring PUBlic sector Finland, along with the UK, is widely
ProDUctiVity in the UK regarded as a world leader in public sector
productivity measurement. In 1995 a
Since 1988, the Office for National project was established to measure public
Statistics has been progressively moving sector productivity in Finland. It was located
away from the output=input approach in Statistics Finland, the national statistics
to productivity, and incorporating direct office. The aim of the project was: ‘to
measures of the volume of government develop a measurement and monitoring
output in the national accounts. By 2005, system for government sector production
these direct output estimates accounted and productivity by using an output
for two-thirds of general government indicator method to measure the volume
final consumption. In the context of of output’ (Niemi, 1998). In 1997 the scope
this focus on output measurement, the of the project was expanded to include
UK government commissioned Sir Tony the measurement of the productivity of
Atkinson to undertake a review of the local government services.
measurement of government output in the
national accounts. This review (Atkinson, Under the terms of the project, for central
2005) provides a comprehensive overview government services the final output
of developments and recommendations and the output measures are specified
for future progress. by the agencies themselves. Examples of
output measures are given in Table 4. The
The UK government accepted the findings agencies for which input and output data
and recommendations of the Atkinson are gathered cover about 80 per cent of
review, and the Office for National Statistics the compensation of employees in central
(ONS) has the lead role in taking forward government. Initial results show growth
the recommendations. To this end, the rates of output and productivity varying
ONS has set up the UK Centre for the extensively.
Measurement of Government Activity
(UKCeMGA). The UKCeMGA has issued a
number of interesting reports and research


MEASURING PUBLIC SECTOR PRODUCTIVITY

TABLE 4
EXAMPLES OF OUTPUT INDICATORS IN THE CENTRAL GOVERNMENT IN FINLAND
28

Consumer Ombudsman’s
Ofﬁce
number of petitions to market court
marketing instructions
(number of ) contractual terms negotiated
statements on legislative initiatives
cases solved individually
replies to written enquiries

Courts
number of cases settled

National Food Administration
number of letters guiding supervision
number of administrative decisions and memos
number of publications
number of statements
number of training events
new instruction materials

State Audit Ofﬁce
number of annual audits
supplementary audits
international audits
expertise activities, statements

Universities
number of degrees completed (generally separated into
graduate and postgraduate degrees)
adult education and continuing education measured,
for example, in days or number of courses (depending
on the university)
number of publications (research)

Source: Niemi, 1998



MeasUring PUBlic sector a 20 per cent increase in output.
ProDUctiVity in sWeDen � To measure the productivity of an 29
agency as a whole means overheads
Sweden has been measuring public sector must be taken into account.
productivity since the mid-1980s. An Expert � Results differ depending on the kind of
Group on Public Finances (a subcommittee price index chosen (Ministry of Finance,
under the Ministry of Finance) established 1997).
a steering group to conduct the work. The
steering group was supported by Statistics MeasUring PUBlic sector
Sweden, the national statistics office. ProDUctiVity in aUstralia

Particular focus is given to the In 1993 the Council of Australian
development of output measures for Governments established the Review of
services. These include items such as the Government Service Provision to provide
number of admitted patients for in-patient information on the effectiveness and
medical care, the number of learning hours efficiency of government services
for education, traffic volume as measured (Australian Productivity Commission,
by vehicle kilometres for public roads, 2006). The review is conducted annually,
number of flying hours for the air force. overseen by a steering committee of
Attempts are made to adjust the quantity senior representatives from the central
of outputs for quality variations where agencies of all the state governments, with
data are available. A number of lessons the assistance of a secretariat provided by
are drawn based on the experience of the Productivity Commission. Performance
producing annual productivity measures information is provided on fourteen service
over a period of time: areas covering six main government
� Productivity varies greatly from year to functions: education; justice; emergency
year. To assess any given year, a time management; health; community services;
series of several years’ information is and housing.
needed.
� Most agencies have several categories The report includes performance
of output. Attributing the same weight comparisons across jurisdictions for the
to all categories may yield deceptive services, using a common method. Both
results. In the enforcement service, for outputs and outcomes are measured, as
example, counting cases dealt with well as efficiency, effectiveness and equity.
regardless of category gives a 2 per Government funding per unit of output
cent decrease in output from 1981 to delivered is typically used as a measure of
1992. If, however, different weights technical efficiency, e.g. recurrent funding
are assigned to different categories of per annual curriculum hour for vocational
output, to take into account differences education and training. Where there are
in composition of the cases, the result is shortcomings in the data, other measures



of efficiency are used (including partial There is some evidence of output measures
30
productivity ratios such as staff level per produced under the various initiatives
student in government schools, staff being used in a limited manner in resource
per prisoner in corrective services and allocation decisions. The OECD (2007) has
administrative costs as a proportion of reviewed the use of output measures in
total expenditure in services for people the budgetary process. It finds that in most
with a disability) (Australian Productivity sectors and cases, performance measures
Commission, 2006). are loosely connected to decisions in the
budgetary process. Output measures
learning the lessons froM are often used by ministries of finance
PUBlic sector ProDUctiVity in the budgetary process along with
MeasUreMent initiatiVes other information on performance and
on political priorities to inform budget
The evidence from studies of public sector allocations. But the OECD warns that a
productivity measurement is that the latter direct linkage, where results determine
is still in its early stages. Despite efforts going funding, creates incentives for gaming
back to the 1980s, the productivity measures the system, such as manipulating the
being produced need to be interpreted data. Ministries of finance have taken a
cautiously. There is also the danger that an cautious approach to using performance
over-simplistic use of the measures could information to financially punish or reward
lead to perverse consequences. It is clear agencies. When programmes show poor
that no single figure of productivity can performance against outputs, the most
be used for public sector activities, unless common course of action is that resources
there is clear and widespread agreement are held constant and the programme
that it is an appropriate measure. The reviewed during the course of the year.
Atkinson (2005) recommendation that a
range of supporting information is needed
to measure productivity change, and not to
rely solely on a single productivity measure,
is one that should be applied generally.


5. Ensuring that Performance Measures are Used
and Useful
Performance measures are not always “A jurisdiction or agency is employing a
used, or used in ways that their designers PerformanceStat performance strategy if it 31
intended. Van Thiel and Leeuw (2002) coined holds an ongoing series of regular, frequent,
the phrase the performance paradox. The periodic, integrated meetings during which
performance paradox refers to the situation the chief executive and/or the principal
where there is a weak correlation between members of the chief executive’s leadership
performance measures and performance team plus the individual director (and the
itself. Measures lose their value over time top managers) of different sub-units use
and can no longer discriminate between data to analyse the unit’s past performance,
good and bad performance. As a result, the to follow-up on previous decisions and
relationship between actual and reported commitments to improve performance, to
performance declines. establish its next performance objectives,
and to examine the effectiveness of its
Behn (2008) too makes the point that overall performance strategies (Behn, 2008,
performance measures do not always of p.2).”
themselves guarantee good performance.
He examines the use of performance Behn identifies seven big mistakes that
measures in structured settings, organisations often make in applying
which he refers to as PerformanceStat. PerformanceStat approaches. These are
PerformanceStat is Behn’s label for the outlined in Table 5.
approach initiated in 1994 by the New York
Police Department when they created The key message emerging from both Van
CompStat, a strategy designed to reduce Thiel and Leeuw and Behn’s analyses is that
the city’s crime rate. The approach was performance measures need to be seen in
adopted by Baltimore and applied to a wider management context if they are
the whole of city government, where it to be used and useful. Some of the main
was labelled CitiStat.3 The approach has issues that need to be addressed when
subsequently been applied in a number of placing performance measures in this
other US cities and in Scotland. Behn uses wider context are: linking measures and
the term PerformanceStat as a generic title targets; performance measurement and
for such an approach, which he defines as citizen engagement; and using incentives
follows: and sanctions.


ENSURING THAT PERFORMANCE MEASURES ARE USED AND USEFUL

taBle 5
32 seVen Big errors of the PerforMancestat aPProach to the
aPPlication of PerforMance MeasUreMent

Error No. 1:
No clear purpose. In this case, the approach is applied as the latest ‘fad’ with little
or no thought as to what results the organisation is trying to achieve or what
better performance might look like.

Error No. 2:
No one has specific responsibilities. You need to be able to answer the question of
who will be responsible for what.

Error No. 3:
The meetings are held irregularly, infrequently, or randomly. If meetings are not held
regularly and frequently, they cannot provide good feedback on successes and
failures.

Error No. 4:
No one person authorised to run the meetings. One person, with clear authority,
needs to consistently run the meetings if they are to have an impact. Ideally, this
should be the chief executive.

Error No. 5:
No dedicated analytic staff. PerformanceStat requires data for the measures used.
For PerformanceStat to provide meaningful results, it needs a few analytical
people working on it full-time.

Error No. 6:
No follow-up. Issues identified by the measures should be followed up from one
meeting to the next, and not start over each time.

Error No. 7:
No balance between the brutal and the bland. Baltimore’s application of CitiStat is
known for being tough and uncompromising with poor performers. Sometimes
overly so. Others have tried to avoid this but in the process have achieved little
more than presenting a picture of accomplishments. Balance is needed.

Source: Behn, 2008, http://www.hks.harvard.edu/thebehnreport/Behn,%207PerformanceStatErrors.pdf



linking Measures and targets 2. Setting learning targets, as opposed to
merely difficult-to-attain targets, may be most 33
Targets are an important element in making effective when tasks are complex. When tasks
performance measures useful. Good are complex, challenging targets may be
targets can lead to enhanced performance less helpful. In such situations, targets that
(see local education authorities case encourage employees to explore strategies
study below). In an extensive review of to tackle the task may improve performance.
the literature on motivating employees,
Perry, Mesch and Paarlberg (2006) 3. The target-performance relationship is
identify three important propositions strongest when employees are committed to
supported by evidence provided by their targets and receive incentives (monetary
the literature with regard to targets: or otherwise), input and feedback related to
the achievement of targets. A manager who
1. Challenging and specific targets improve wants to ensure that challenging targets
the performance of employees. Target setting are met should try to improve the ability
increases individual, group and work unit of employees to meet these targets and
performance. In many instances, specific provide feedback on the results of their
and challenging targets are associated efforts.
with higher levels of performance than
either no, or general ‘do your best’, targets.

case stUDy

target setting for local eDUcation aUthorities

Theories suggest that organisational performance improves if targets for future
achievement are used. Boyne and Chen (2006) explored this theory using panel data for
147 English local education authorities between 1998 and 2003. The dependent variables
in the analysis are exam results for school pupils. The authors found that, controlling for
other variables, the extent of performance improvement is influenced positively by the
presence of a target. The results are consistent with the view that clear and quantified
strategic priority targets lead to better organisational outcomes.



setting good targets – using baselines Sometimes a target range rather than
34
a specific figure may be appropriate,
Knowing that linking performance especially where the measures are new
measures to targets can enhance and untested. However, there is a need
performance is useful, but not enough to guard against the games people
on its own. How do we know when a sometimes play when setting targets.
‘good’ target has been set? What might a Organisations may set such modest
challenging yet realistic target look like? targets that these are achieved quite easily.
The World Bank suggests that the Conversely, unachievable targets may be
identification and use of baseline data set that just end up demoralising staff. Use
can be very helpful when setting targets. of the baseline measures can be important
Kusek and Rist (2004) note that targets are here, especially if trend data are available,
the quantifiable level of a performance to show what a realistic but challenging
measure that a country or organisation target might be. Similarly, external scrutiny
wants to achieve by a given time. They of targets, by key stakeholders or audit
suggest that one method to establish institutions, can help in ensuring that over-
targets is to start with a baseline indicator comfortable targets are not set. Some
level, and include the desired level of examples of good and bad targets are
improvement (taking into consideration outlined in Table 7.
available resources over a specific time
period) to arrive at the performance target
(see Table 6 for a worked example).

taBle 6
DeVeloPing targets Using Baseline Data
Outcomes Measures Baselines Targets

Nation’s children 1. Percentage of 1. In 2003, 75 per 1. By 2010, 85 per
have better access eligible urban cent of children cent of children
to preschool children enrolled in ages 3-5 ages 3-5
programmes preschool education
2. Percentage 2. In 2004, 40 per 2. By 2010, 60 per
of eligible rural cent of children cent of children
children enrolled in ages 3-5 ages 3-5
preschool education
Primary school Percentage of Grade In 2005, 75 per cent By 2010, 80 per cent
learning outcomes 6 students scoring scored 70 per cent scoring 70 per cent
for children are 70 per cent or better or better in maths, or better in maths,
improved on standardised and 61 per cent and 67 per cent
maths and science scored 70 per cent scoring 70 per cent
tests or better in science or better in science
Source: adapted from Kusek and Rist, 2004, p.95



taBle 7
35
exaMPles of gooD anD BaD targets

Examples of good targets
• We will reduce the number of missed household collections by 10 per cent by
next year.
• We will aim to collect 95 per cent of the council tax that is due next year.
• We will increase the number of visits to local libraries by 20 per cent before
the end of 2009.
• We will cut the number of unfilled places in primary schools by 10 per cent by
31 December 2010.

Examples of poor targets
• We will improve the way we handle complaints.
• We will buy as many books for the school as possible.
• We aim to have the best bus service in the region.
• We aim to increase co-operation between school and police authorities.
• We will answer 75 per cent of all letters within 5 days.
(a poor target if the remaining 25 per cent take 3 months to answer)
Source: adapted from UK Audit Commission, 2000, p.24

PerforMance MeasUreMent � Performance measures developed with
anD citizen engageMent citizen input are likely to have more
support from politicians overseeing
The importance of including measures that the work of public organisations.
citizens want measured in performance � Citizens’ views may provide the
measurement systems was discussed in opportunity and stimulus for
Section 2. Generally, there are a number of innovation, suggesting new ways of
reasons why the public should be engaged doing business.
in performance measurement and the
reporting of performance: The question is not whether the customer
� As funders of the service, citizens’ views should be engaged in performance
on how well services are provided are measurement, but how best to engage the
an important direct consideration for public. Two issues of central importance
the staff of public organisations. here are: means of engaging the public and
� The direct involvement of the public the reporting of performance measures to
may help legitimise the decisions the public.
and priorities of public service
organisations.



Means of engaging the public in provided by the World Bank). The use of
36 performance measurement citizen report cards began in Bangalore,4
India, and the practice has since spread
Ho (2007) suggests a portfolio approach to many countries including Canada,
to engaging the public in performance Denmark, the Philippines, Sweden, Ukraine
measurement. Among the methods of and the UK. The case study below gives
engaging the public are the following: details of the Filipino report card.
� Focus groups
� Neighbourhood meetings Three main types of institutional
� Citizen satisfaction surveys arrangements for report cards are
� Report cards for programme users possible:
� Web-based discussion forums � Report card by civil society organisation.
� Web-based surveys This was the model adopted in Bangalore,
where the Public Affairs Centre developed
Each method of engagement has the report card in response to anecdotal
advantages and disadvantages, and may evidence of customer dissatisfaction with
vary in terms of cost. Ho argues for a municipal services. The report card gives
diversified portfolio approach that contains citizen feedback on both quantitative
multiple strategies to receive diverse and and qualitative dimensions of the
balanced input from citizens from all walks selected public services. The strength of
of life. this approach is the independence and
Two of the strategies noted above have credibility attached to the performance
received particular attention in recent measures. The limitations of the approach
years – report cards and citizen surveys. relate to the fact that not many civil society
Experience in using each is discussed briefly organisations have the technical capacity
below, to illustrate some of the benefits and willingness to take on such a job.
and challenges of involving the public in � Report card by government service
performance measurement. provider agency. This approach has been
used in Canada and the UK. Here, a
Citizen report cards government agency takes responsibility for
the report card production, although the
Citizen report cards are participatory actual survey and draft report preparation
services that provide quantitative feedback is often contracted out to a commercial
on user perception of the quality, adequacy organisation. A strength of this approach is
and efficiency of public services (see http:// the ownership of the exercise by the public
web.worldbank.org/WBSITE/EXTERNAL/ agency. This is also a potential limitation,
TOPICS/EXTSOCIALDEVELOPMENT/EXTPC as the public and politicians may question
ENG/0,,contentMDK:20507680~pagePK:1 the independence and objectivity of the
48956~piPK:216618~theSitePK:410306,00. findings.
html for details of the report card approach �



case stUDy
37
institUtionalising the filiPino rePort carD
The Department of Budget and Management (DBM) in the Philippines initiated a
programme on the development of performance measures for public agencies so as to
assess their effectiveness and guide future budget allocations. As part of this process,
they decided to pilot the use of the report card as a means of getting regular user
feedback on key public services. The incentive for service providers to respond to
customer feedback is enhanced, knowing that they will be regularly monitored by DBM.
Analysis and report preparation is contracted out to a credible and independent civil
society organisation. An advisory panel, comprising representatives of service providers,
other government agencies, the private sector, sectoral groups and prominent experts
has been established to provide advice and guidance on the report card process and its
integration with the budgetary process.
Some specific lessons emerged from piloting the report card:

� Focus on key performance measures. The first report card spread the net wide and
tried to cover many aspects of service delivery. Subsequent rounds
are to be limited to a few key performance measures. The intention is
that the measures selected will have a significant overlap with those
used by DBM in monitoring outputs and processes, facilitating the
comparison of results to provide a comprehensive picture of performance.

� Revising the questionnaire. Overlaps in survey questions were identified. Greater
clarity of wording was identified as needed in some questions. Focusing of the
questionnaire on the key performance measures common with those used for
monitoring outputs and processes by DBM was identified as a significant issue.

� Improving cost-effectiveness. Limiting questions and refining the questionnaire
should bring about considerable savings. Also, limiting the scope to a few principal
and common performance measures is intended to focus the attention of public
service providers and policy makers and result in concrete actions and follow-up.

Source: http://siteresources.worldbank.org/INTPCENG/1143333-1116505690049/
20509283/Filipino+Report+Card+on+Pro-chapter7-report-cards.pdf



� Report card by government performance can correspond with
38 oversight agency. This approach has been more ‘objectively’ measured outcomes,
used by the US and the Philippines. It supporting the use of citizen surveys as a
involves a government co-ordinating measure of government performance.
agency engaging an independent civil
society organisation to undertake the However, the researchers urge caution
design and preparation of the report card in interpreting their findings. Street
in consultation with the public service cleanliness may be an issue that lends itself
provider agencies. This approach is the to measurement through a citizen survey,
most comprehensive both in terms of compared to other more complex or less
process and product. Similar limitations visible services. But the findings suggest
exist to the civil society organisation that, in the right circumstances, citizen
approach, in terms of the availability of the surveys can be used to judge, and report
technical capacity to undertake the work. on, the quality of government services, and
may be particularly useful where it is too
Citizen surveys complex or costly to gather data in other
Report cards can be seen as a specific ways.
type of citizen survey. Citizen surveys are
generally and widely used to develop
measures to assess customer satisfaction
with services. But there are concerns about
the representativeness and validity of
citizen surveys. How do we know if the right
questions are being asked? Are they an
accurate indication of performance? Is the
cost of obtaining a sample large enough to
be representative really justified?

In terms of the benefits of citizen surveys,
work at the City University of New York
provides some interesting pointers (Van
Ryzin and Immerwahr, 2007 and Van Ryzin,
Immerwahr and Altman, 2008). Using data
from New York City’s street cleanliness
scorecard, combined with responses to
a citizen survey, the researchers found a
clear and consistent correlation between
the scorecard results and citizen rating of
street cleanliness. The results suggest that
citizen judgements about government



The researchers also note that in analysing derived importance because, although
citizen surveys, it can sometimes be difficult more analytically complex, it has a better 39
to make sense of the ratings citizens give predictive ability with regard to consumer
to various services and to use these ratings behaviour.
to meaningfully compare performance.
They advocate the use of importance- reporting of performance measures
performance analysis (see Table 8). to the public
This charts the relationship between
service performance ratings and service The way in which performance measures
importance as seen by the customer. Efforts are reported back to citizens affects the
should be focused on improving the high degree of engagement of the public
importance-low performance sector. and ultimately the usefulness of many
performance measures. Ho (2007), after
Two alternative means of measuring the examining a wide variety of performance
importance of a service are possible: reporting practice, identifies nine lessons
stated importance, in response to a survey for reporting performance measurement
question that explicitly asks respondents results to citizens:
to rate importance, and statistically derived
importance, based on a regression model 1. Focus on outcomes and intermediate
of overall customer satisfaction in which outcome measures in public reporting.
the services are the independent variables. Citizens are primarily interested in the
On the basis of comparative analysis, results delivered by programmes.
the researchers recommend the use of

taBle 8
the iMPortance-PerforMance griD

Low importance High importance

High Potential overkill in terms of Doing a good job, providing
performance effort and possibility of slack services that citizens value
resources

Low From a citizen satisfaction Services that citizens value
performance perspective, services that could but which suffer from poor
be given a lower priority performance ratings

Source: adapted from Ryzin and Immerwahr, 2007, p.217



2. Make the design of public Citizens’ views of services and
40
performance reports citizen friendly. programmes cut across organisational
The use of figures, graphs and charts and other boundaries. They want to
to replace/complement/supplement know about the issues that concern
tables of numbers and text can make them, not the part of an issue that
a big difference. falls under one particular level of
jurisdiction. Reporting of performance
3. Provide geographic segregation measures should be organised by
of data. Many citizens want topics that citizens are concerned
performance measures to be broken about.
down by neighbourhood rather
than be presented for the whole 7. Use web-based reporting to keep citizens
geographical area covered by more informed. Web-based reporting
the programme. For example, the can be updated regularly and cost-
Boston Indicators project website effectively. It also facilitates user
has a feature known as ‘Geography engagement. For example, visitors
at a Glance’ which highlights to the online version of Oregon’s
some of the key socio-economic 2007 Benchmark Report (http://
measures by neighbourhoods benchmarks.oregon.gov/) can select
(see http://www.bostonindicators. the benchmarks that interest them
org/IndicatorsProject/). and generate a customised report.

4. Provide comparable performance 8. Blend e-reporting and paper copies.
benchmarks. Citizens like to compare Paper copies of reports are still
their neighbourhood with that important. It is not an either/or
of other equivalent communities. situation with regard to e-reporting.
This helps citizens contextualise Paper reports, for example, can be
performance measures. structured to lead people to the
website later.
5. Provide stories to explain and elaborate
the data. Qualitative descriptions 9. Guarantee data accuracy and reliability.
of specific events can complement Public trust and credibility with
quantitative measures. Stories of regard to the performance measures
successes and failures illustrate reported is vital. Actions such as
measures and make them more conducting performance audits to
meaningful. verify the reliability and validity of the
data behind the measures can help
6. Organise performance information secure public trust. So can balanced
by community concerns, not by reporting of both successes and
departmental or agency structure. failures.



Using incentiVes anD Annual Reports Award. Such initiatives
sanctions to encoUrage recognise excellence in the reporting of 41
the Use of PerforMance performance measures. They can be used
MeasUres to highlight and spread good practice.

Performance measures are rarely used Assurance. Four levels of assurance can be
simply on the basis that they have been identified (CCAF-FCVI, 2007, p.38):
produced. Performance measurement is • Assurance is inferred if the user must
not just a technical exercise; it has strong assess the report content to determine
cultural and political components. In such how much confidence to place in the
an environment, incentives and sanctions performance measures.
(carrots and sticks) can play an important • Assurance is affirmed if management
role in encouraging the use of performance affirms its legal responsibility for
measures. Amongst frequently used reporting.
incentives and sanctions are the following: • Assurance is described if management
describes the basis for its judgements,
Legislation. Making performance the steps it has taken to validate
measurement and reporting a legislative measures, and the limitations of the
requirement can help create a performance data.
culture in an organisation. Measuring • Assurance is corroborated if a third
performance is more likely to be seen as party (such as an auditor) examines the
an important activity worthy of senior report and adds assurance (or delivers
management attention. Canada’s Revenue cautions).
Agency, for example, has a legislative
requirement to produce a performance Involvement/feedback. Many studies
report and has stated that this has had a show that if measures are to take root
positive impact. in organisations, staff must have an
opportunity at some stage to influence the
Review. Internal and external reviews choice of measures used. Staff, as well as
of performance measures and reports customers, should also be given feedback
can help ensure that measurement is on performance rather than simply feeding
taken seriously. The Office of the Auditor information up the line. Examples of ways
General in Canada has regularly reviewed of encouraging participation include
departmental reporting of performance developing mixed task teams to develop
measures. measures and regular meetings between
managers and staff to discuss results.
Award/Recognition. In 2006 the New South
Wales Public Bodies Review Committee,
in conjunction with the Public Accounts
Committee, hosted the first Premier’s


Endnotes

1. The term performance measure as used here includes indicators. In the literature,
42
measures and indicators are sometimes differentiated, with measures referring
to quantitative assessment of normative performance and indicators giving a
quantitative or qualitative assessment of relative performance.

2. The term programme as used here is shorthand for a variety of objects of performance
measurement including programmes, projects, operations, administrative systems,
etc.

3. CompStat originated in the NewYork City Police Department in 1994, under leadership
of Police Commissioner William Bratton and his Deputy Commissioner Jack Maple.
On a weekly basis, personnel from each of the Department’s 76 Precincts, 9 Police
Service Areas and 12 Transit Districts compile a statistical summary of the week’s
crime complaint, arrest and summons activity, as well as a written recapitulation of
significant cases, crime patterns and police activities. These data are forwarded to
the Chief of Department’s CompStat Unit where they are collated and loaded into
a citywide database. The data are analysed by computer and a weekly CompStat
Report is generated. The CompStat Report captures crime complaint and arrest
activity at the precinct, patrol borough, and citywide levels, and presents a concise
summary of these and other important performance measures. Comparisons
to previous years’ activity are shown. Precinct commanders and members of the
agency’s top management can see emerging and established crime trends as well
as deviations and anomalies, and can make comparisons between commands.

Also, several American cities, inspired by its success in Baltimore, are engaging with
a technology-enabled management approach called CitiStat. The mayor of the
city has a database of indicators established that tracks performance against his or
her political priorities, such as reducing lead levels in children’s blood from traffic
fumes, and also internal management issues such as overtime and absenteeism.
The mayor then holds regular meetings with city managers to discuss these data
and their trends. The mayor persistently questions managers about problems and
strategies to deal with these problems. Specific targets, assignments and deadlines
are given to address issues raised. The data are actively used to try to improve
performance (see Henderson, 2003, http://www.businessofgovernment.org/pdfs/
HendersonReport.pdf ).


ENDNOTES

4. Inspired by a private sector practice of conducting client satisfaction surveys, a group
of people in Bangalore, India, concerned about the city’s worsening standards of public 43
services, started an exercise in 1993 to collect feedback from public service users. User
views of the quality, efficiency and adequacy of the various services were aggregated
to produce a ‘report card’ that rated the performance of public service providers in
the city. The findings received media attention, which helped mobilise citizen and
government support for reform, and encouraged the rated agencies to improve their
standards. The exercise was repeated in 1999. Report cards provide a useful means
by which citizens can signal to public service agencies their level of satisfaction or
dissatisfaction with services and provide pressure for change. For further details see
http://siteresources.worldbank.org/INTPCENG/1143333-1116505690049/2050927
5/making.pdf


References

Atkinson A.B. (2005),
44
Atkinson Review: Final Report – Measurement of Government Output and Productivity for
the National Accounts,
Final report, Basingstoke: Palgrave Macmillan

UK Audit Commission (2000),
On Target: The Practice of Performance Indicators,
London: Audit Commission

Australian Productivity Commission (2006),
Report on Government Services 2006,
Report of the Steering Committee for the Review of Government Service Provision,
Canberra: Australian Productivity Commission (http://www.pc.gov.au/)

Behn, B. (2008),
The Seven Big Errors of PerformanceStat,
Policy Briefs, John F. Kennedy School of Government, Harvard University
(http://www.hks.harvard.edu/thebehnreport/Behn,%207PerformanceStatErrors.pdf )

Boyne, G.A. and A.A. Chen (2006),
‘Performance targets and public service improvement’, Journal of Public Administration
Research and Theory,
Vol.17, No.3, pp.455-477

Boyle, R. (2005),
Civil Service Performance Indicators,
Committee for Public Management Research Discussion Paper No.29, Dublin: Institute
of Public Administration
(http://www.cpmr.gov.ie/publications/discussion-papers/)

CCAF-FCVI Inc. (2007),
What Can We Learn from Effective Public Performance Reporting?
Ottawa: CCAF-FCVI
(http://www.performancereporting.ca/documents/PPR-Good_Practices.pdf )


REFERENCES

Efficiency Unit (2000),
Step-by-Step Guide to Performance Measurement, 45
Efficiency Unit, The Government of the Hong Kong Special Administrative Region
(http://www.eu.gov.hk/english/publication/files/step_by_step_guide_to_pm.pdf )

Henderson, L.J. (2003),
The Baltimore CitiStat Program,
Washington D.C.: IBM Center for the Business of Government
(http://www.businessofgovernment.org/pdfs/HendersonReport.pdf )

Ho, A.T. (2007),
Engaging Citizens in Measuring and Reporting Community Conditions: A Manager’s Guide,
Washington D.C.: IBM Center for the Business of Government
(http://www.businessofgovernment.org/pdfs/HoReport.pdf )

Kaplan, R.S. and D.P. Norton (1993),
‘Putting the balanced scorecard to work’, Harvard Business Review,
September-October, pp.134-147

Kusek, J.Z. and R.C. Rist (2004),
Ten Steps to a Results-Based Monitoring and Evaluation System,
Washington, D.C.: The World Bank

Ministry of Finance (1997),
Public Sector Productivity in Sweden,
Stockholm: Ministry of Finance

Neely, A. and P. Micheli (2004),
Performance measurement in the UK’s public sector: linking the national to the local agenda,
CIMA New Public Sector Seminar, Edinburgh, 28-29 October 2004
(http://www.som.cranfield.ac.uk/som/research/centres/cbp/downloads/2004BAM_
PMpaper.pdf )


REFERENCES

Niemi, M. (1998),
46 ‘Measuring government sector output and productivity in Finland – application of the
output indicator method’,
Agenda item 1, OECD meeting of national accounts experts, Statistics Directorate, STD/
NA(98)4, 22-25 September, Paris: OECD

OECD (2007),
‘Towards Better Measurement of Government’, OECD Working Papers on Public Governance,
2007/1, Paris: OECD Publishing. doi:10.1787/301575636734
(http://www.oecd.org/dataoecd/11/61/3813 4037.pdf )

Ogata, K. and R. Goodkey (1998),
‘Redefining government performance’, Alberta Finance, Canada
(http://www.finance.gov.ab.ca/publications/measuring/cambridge_paper.html)

Perrin, B. (1998),
‘Effective use and misuse of performance measurement’, American Journal of Evaluation,
Vol.19, No.3, pp. 367-379

Perry, J.L., D. Mesch and L. Paarlberg (2006),
‘Motivating employees in a new governance era: the performance paradigm revisited’, Public
Administration Review,
Vol. 66, No. 4, pp.505-513

Pollitt, C., S. Harrison, R. Bal, G. Doswell and S. Jerak (2007),
‘Conceptualising the development of performance measurement systems’, paper presented
at European Study Group on Performance and Quality conference,
Madrid, 19-22 September,
(http://soc.kuleuven.be/io/egpa/qual/madrid/papers/paper%20Pollitt%20et%20al.doc)


REFERENCES

Schacter, M. (2002),
Not a Tool Kit: Practitioners Guide to Measuring the Performance of Public Programs, 47
Ottawa: Institute on Governance

Stowers, G. (2004),
Measuring the Performance of E-Government,
Washington D.C.: IBM Center for the Business of Government,
(http://www.businessofgovernment.org/pdfs/8493_Stowers_Report.pdf )

United Way of America (1996),
Measuring Program Outcomes: A Practical Approach,
Alexandra, Virginia: United Way of America

Van Ryzin, G.G. and S. Immerwahr (2007),
‘Importance-performance analysis of citizen satisfaction surveys’, Public Administration,
Vol. 85, No. 1, pp.215-226

Van Ryzin, G.G., S. Immerwahr and S. Altman (2008),
‘Measuring street cleanliness: a comparison of New York City’s scorecard and results from a
citizen survey’, Public Administration Review,
Vol. 68, No. 2, pp.295-303

Van Thiel, S. and F.L. Leeuw (2002),
‘The performance paradox in the public sector’, Public Performance and Management
Review,
Vol. 25, No. 3, pp.267-281

World Bank (2006),
Impact Evaluation – The Experience of the Independent Evaluation Group of the World Bank,
Washington D.C.: The World Bank Independent Evaluation Group
(http://lnweb18.worldbank.org/oed/oeddoclib.nsf/DocUNIDViewForJavaSearch/35BC4
20995BF58F8852571E00068C6BD/$file/impact_evaluation.pdf )


EfficiEncy unit
13/f., West Wing
central Government Offices
11 ice House Street
central
Hong Kong

Email: euwm@eu.gov.hk
tel: 2165 7255
fax: 2524 7267
Website: www.eu.gov.hk

Pfm Measure 2008

More Related Content

What's hot (19)

Viewers also liked (20)

Similar to Pfm Measure 2008 (20)

More from euweben01 (20)

Recently uploaded (20)

Pfm Measure 2008