Implementation for performance auditing

IMPLEMENTATION GUIDELINES
FOR PERFORMANCE AUDITING
Standards and guidelines for performance auditing
based on INTOSAI’s Auditing Standards
and practical experience

Preamble
At the International Congress of Supreme Audit Institutions
(INCOSAI) conference in Montevideo (1998) it was agreed to endorse
the development of guidelines for the implementation of the Interna-tional
Organization of Supreme Audit Institutions (INTOSAI) Audit-ing
Standards. The INTOSAI Auditing Standards Committee (ASC)
would carry out the development work in consultation with other
standing committees and working groups.
The Auditing Standards focus mainly on fi nancial auditing, but they
cover performance auditing as well. As many Supreme Audit Institu-tions
have pointed out, there is a need for special guidelines in per-formance
auditing, since it differs in character from fi nancial auditing.
It has therefore been thought wise to develop separate guidelines for
performance auditing.
A fi rst discussion was held at the Committee meeting in London
(2000), and after the meeting in Lisbon (2002) working-material was
sent to the Committee members for comments. The ongoing work
was supported by the Governing Board and the INCOSAI congress in
Seoul (2001).
At the Committee meeting in Stockholm (2002), it was decided to
send an exposure draft to all INTOSAI members for comments. A fi nal
draft was produced and thereafter approved by the Committee at its
meeting in Bratislava (2003). Throughout the process the Governing
Board has been informed of the progress of the work and approved the
presented working plans.
These Implementation Guidelines for Performance Auditing are the
result of the joint efforts of the members of the INTOSAI Auditing
Standards Committee, which has included the Supreme Audit Institu-tions
of:
Antigua and
Barbuda
Argentina
Austria
Australia
Azerbaijan
Brazil
Cameroon
Costa Rica
Denmark
Canada
Egypt
India
Lithuania
Japan
Namibia
Norway
Philippines
Portugal
Samoa
Saudi Arabia
Slovakia
Sweden;
Chairman
Tonga
Tunisia
Ukraine
United Kingdom
United States
Uruguay

I would especially like to recognize the SAIs of the United Kingdom
(English), Canada (French), Austria (German), Saudi-Arabia (Arabic)
and the United States (Spanish) for their support in proof-reading the
different language versions.
I am pleased to submit this document. It constitutes an important
step forward in the process of enhancing performance auditing among
government auditors. It should be considered a living document,
which has to be updated as practices progress. It is not a normative
or a technical document, or a handbook, but it contains a number of
guidelines and other information with practical implications that take
into consideration the special premises and features of performance
auditing. Even though these guidelines refl ect current best practices,
they will not fully be applicable to all INTOSAI members, due to dif-ferent
traditions and mandates. It is up to each member to determine
how to best apply and utilize these guidelines.
The purpose of this document is to:
• Describe the features and principles of performance auditing.
• Assist SAI performance auditors in managing and conducting per-formance
audits effi ciently and effectively.
• Provide a basis for good performance audit practices.
• Establish a framework for the further development of performance
audit methodology and professional development.
On behalf of the Auditing Standards Committee, I would like to
thank all INTOSAI Committees and members for their dedication and
cooperation in completing this project. I also thank my Committee
colleagues for their timely support and positive contributions to this
activity.
Stockholm, July 2004
Kjell Larsson
Auditor General, Swedish National Audit Offi ce
Chair of the INTOSAI Auditing Standard Committee

Table of contents
Introduction 7
Part 1: What is performance auditing? 11
1.1 What is performance auditing according to INTOSAI? 11
1.2 What is the special feature of performance auditing? 12
1.3 What ideas form the basis of performance auditing? 12
1.4 What are the basic questions in performance auditing? 13
1.5 What does auditing of economy, effi ciency and
effectiveness mean? 15
1.6 How does public management affect performance
auditing? 20
1.7 How does performance auditing relate to performance
measurement and programme evaluation? 21
1.8 Are there differences in analytical ambitions and
approaches? 25
1.9 Summary 29
Part 2: Government auditing principles applied to performance
auditing 31
2.1 How do the auditing principles apply to performance
auditing? 31
2.2 What are the general requirements for a performance
auditor? 36
2.3 Are there other important safeguards? 39
2.4 Summary 41
Part 3: Field standards and guidance: Initiating and planning
the performance audit 43
3.1 What are the overall steps in the performance auditing
cycle? 43
3.2 What does strategic planning involve? 44
3.3 What does planning of individual performance audits
involve? 47
3.4 Summary 56

Part 4: Field standards and guidance:
Conducting the performance audit 59
4.1 What characterises the main study process? 59
4.2 What has to be considered in the data collection
process? 60
4.3 What characterises the audit evidence and the audit
fi ndings? 62
4.4 How should a changeable and confl icting environment
be handled? 64
4.5 What is important when analysing data and drawing
conclusions? 66
4.6 Summary 68
Part 5: Reporting standards and guidance:
Presenting the performance audit result 69
5.1 What does the need to focus on the fi nal report involve? 69
5.2 What is required to make the report reliable? 70
5.3 What characterises a good and useful performance
audit report? 72
5.4 How should a performance audit report be distributed? 74
5.5 What purposes do follow-up processes serve? 75
5.6 Summary 76
References and bibliography 77
Appendices
1 Performance audit methodology 85
2 Performance audit criteria 103
3 Evidence and documentation 107
4 Communication and quality assurance 117
5 Performance auditing and information technology 121
6 Performance audits of activities with an environmental
perspective 127
7 Towards a system-oriented approach in performance
auditing: a theoretical framework 137

Introduction
Performance auditors can be faced with considerable variety and
ambiguity in their work. They require skills in analyzing activities and
management practices. They can be faced with the need to become
familiar with a wide range of organizational contexts and subject
matters. They need the ability to write logically and thoroughly on
complex issues. These guidelines can provide some assistance in these
areas, but much is incumbent on the performance auditors themselves
to develop their skills in these areas by other means.
The guidelines take into account relevant INTOSAI auditing stand-ards
and are based on generally accepted principles of performance
auditing, distilled from the experience of INTOSAI members.1 In order
to produce experience-based implementation guidelines, a study has
been made of standards and guidelines from a number of SAIs with
many years’ experience of performance auditing. Their experience
of conducting performance audits and implementing the Auditing
Standards has added valuable information, not least with respect to the
practical interpretation of the Auditing Standards.
It is not possible to produce guidelines applicable to all kinds of
performance auditing, since comparisons between the practices of per-formance
auditing in different countries show considerable variations
in mandate, organisation, and methods used. Guidelines in perform-ance
auditing cannot comprehensively embrace all possible approach-es,
methods and techniques, since in practice that would include every-thing
in the social sciences. Furthermore, performance audits deal with
a multitude of topics and perspectives covering the entire government
sector, and it would not be possible to develop detailed standards and
1Audit Director Tony Angleryd (Sweden) prepared the guidelines, but many INTOSAI members have
assisted in the processes. Performance Audit Standards from members and regional working groups
have been studied. The ‘Performance Auditing Guidelines’, approved at the 8th ASOSAI Assembly in
October 2000, should explicitly be mentioned.
IMPLEMENTATION GUIDELINES FOR PERFORMANCE AUDITING 7

8
procedures that work equally well in all these situations. In perform-ance
auditing it is not possible to produce a ‘cookbook’ type of manual
that can universally be followed for good results.
Consequently, some SAIs will fi nd guidelines of this type of lim-ited
value. For instance, they might be considered too ambitious for
auditors with little or no experience of dedicated performance audit
projects or program evaluations. As stated in the Auditing Stand-ards,
paragraph 1.0.6: ‘The SAI should apply its own judgments to
the diverse situations that arise in the course of government audit-ing.’
Moreover, paragraph 1.0.13 states: ‘Because of the approach and
structure of some SAIs, not all auditing standards apply to all aspects
of their work. For example, the collegial and juridical nature of the
reviews conducted by Courts of Account make aspects of their work
fundamentally different from the fi nancial and performance audits
conducted by SAIs, which are organized under a hierarchic system led
by an Auditor-General or a Comptroller General.’ This means that the
SAI itself should decide how and to what extent the guidelines are to
be used in its own audit practices and development work.2
What has been said above must not be taken as an argument against
any standardisation or guidelines, but when it comes to standardisa-tion
in performance auditing it is mostly a question of what to do,
rather than how to do it. For example, in designing a study one would
expect the auditors to make certain considerations and cover particular
aspects. How that is done must be decided on a case-by-case basis and
with consideration of the fact that methods and techniques have to be
applied with the necessary care that is commonly considered to be the
best practice in social sciences and auditing.
This document refl ects the experience of SAIs with a long tradition
and well-established standards of performance auditing. It deals with
performance auditing carried out as separate examinations or investi-gations;
i.e. performance auditing as a separate and professional activ-ity
that requires specialised skills, separate standards, special planning,
special reports, etc.3 Consequently, this document is aimed mainly at
2 In this paper, the various paragraphs of the Auditing Standards (2001) have been referred to as ‘AS’
followed by the respective paragraph number(s). The references are in italics. The term ‘regularity
(fi nancial) auditing’ has been abbreviated to ‘fi nancial auditing.’
3 This document provides general guidelines. Since performance auditing varies considerably between
different countries, it was considered sensible to make the guidelines less normative and detailed than is
traditionally the case.

those SAIs that are carrying out – or are planning to carry out – this
type of performance auditing.4
These guidelines consist of fi ve main parts.
Part 1 sets out the general framework for performance auditing,
PPPaaarrrttt 222 ddeefifi nneess aapppplliiccaattiioonn ooff aauuddiittiinngg pprriinncciipplleess ttoo ppeerrffoorrmmaannccee
auditing,
Part 3 provides standards and guidance for planning performance
audits,
Part 4 provides standards and guidance for conducting performance
audits,
PPPaaarrrttt 555 pprroovviiddeess ssttaannddaarrddss aanndd gguuiiddaannccee ffoorr pprreesseennttiinngg tthhee aauuddiitt
results.
TTThhheee AAAppppppeeennndddiiiccceeesss ccoonnttaaiinn ffuurrtthheerr iinnffoorrmmaattiioonn oonn hhooww ttoo ppllaann aanndd
conduct performance audits. The appendices also include informa-tion
on performance auditing in relation to information technology
(IT) and on conducting performance audits with an environmental
perspective. Further, a framework of system-oriented approaches in
performance auditing is presented.
4 This would, for instance, to some extent exclude the kind of continuous monitoring exercise that is
based on the concept of so-called performance indicators. However, non-regular (in-depth) studies on
topics such as whether performance measurement systems in government programs are effective and
valid or not are not excluded. (See section 1.7.)

Part 1: What is performance auditing?
1.1 What is performance auditing according to INTOSAI?
INTOSAI’s Auditing Standards (AS 1.0.38 and 1.0.40) state the
following:
‘The full scope of government auditing includes regularity and per-formance
audit’, and ‘Performance auditing is concerned with the audit
of economy, effi ciency and effectiveness and embraces:
(a) audit of the economy of administrative activities in accordance
with sound administrative principles and practices, and management
policies;
(b) audit of the effi ciency of utilisation of human, fi nancial and oth-er
resources, including examination of information systems, perform-ance
measures and monitoring arrangements, and procedures followed
by audited entities for remedying identifi ed defi ciencies; and
(c) audit of the effectiveness of performance in relation to achieve-ment
of the objectiveness of the audited entity, and audit of the actual
impact of activities compared with the intended impact’.
Performance auditing is based on decisions made or goals estab-lished
by the legislature, and it may be carried out throughout the
whole public sector.
Performance auditing is an independent examination of the effi ciency
and effectiveness of government undertakings, programs or organiza-tions,
with due regard to economy, and the aim of leading to improve-ments.

12
1.2 What is the special feature of performance auditing?
As stated in the Auditing Standards, performance auditing is not overly
subject to specifi c requirements and expectations. While fi nancial
auditing tends to apply relatively fi xed standards, performance audit-ing
is more fl exible in its choice of subjects, audit objects, methods,
and opinions. Performance auditing is not a regular audit with formal-ized
opinions, and it does not have its roots in private auditing. It is
an independent examination made on a non-recurring basis. It is by
nature wide-ranging and open to judgments and interpretations. It
must have at its disposal a wide selection of investigative and evalu-ative
methods and operate from a quite different knowledge base to
that of traditional auditing. It is not a checklist-based form of auditing.
The special feature of performance auditing is due to the variety and
complexity of questions relating to its work. Within its legal mandate,
performance auditing must be free to examine all government activi-ties
from different perspectives (AS 4.0.4, 4.0.21-23 and 2.2.16).
The character of performance auditing must not, of course, be taken
as an argument for undermining collaboration between the two types
of auditing.
Performance auditing does not have its roots in the form of auditing
common to the private sector. Its roots lie in the need for independent,
wide-ranging analyses of the economy, effi ciency, and effectiveness of
government programs and agencies made on non-recurring basis.
1.3 What ideas form the basis of performance auditing?
Public accountability means that those in charge of a government
program or ministry are held responsible for the effi cient and effective
running of such. Accountability presupposes public insight into the
activities of the program or ministry. Performance auditing is a way for
taxpayers, fi nanciers, legislatures, executives, ordinary citizens and the
media to ‘execute control’ and to obtain insight into the running and
outcome of different government activities. Performance auditing also
provides answers to questions such as: Do we get value for money or
is it possible to spend the money better or more wisely? A criterion of
good governance is that all public services (or all government pro-grams)
are subjected to auditing

Legitimacy and trust are essential values in all government under-takings,
and performance auditing may contribute to strengthening
these values by pro ducing public and reliable information on the
economy, effi ciency, and effectiveness of government programs. This
is facilitated by the fact that performance auditing is independent of
the government ministries whose activities are subject to the audit. In
this way, an independent and reliable view of the performance of the
audited program or objects is obtained. The performance audit does
not represent any vested interest and has no ties, fi nancial or other-wise,
to the audited objects. By producing independent assessments,
performance auditing may also serve as a basis for decisions on future
investments and activities. The basis for this instrument – providing
incentives for change by conducting independent analyses and assess-ments
of public sector performance – is the importance of learning
and reliable information. In a rapidly changing, complex world with
limited resources and many uncertainties, there is a need for perform-ance
auditing.
Certain ideas form the basis of performance auditing:
• One starting point is that it is important to assess the economy, ef-fi
ciency, and effectiveness in all government activities and, for that
purpose, an audit is needed, which examines and evaluate such
matters and which may contribute to better government spending,
better public services and better public accountability and manage-ment.
• Secondly, it is important to have reliable and independent informa-tion.
An examiner is needed who represents the public interest; who
can think and act independently in order to show and question the
current situation.
• Finally, an overview and insights into government activities and the
ability to infl uence and improve its performance are important. A
competent examiner is needed who can fulfi l this role, who will
promote incentives for learning and change and improved condi-tions
for decision-making.
1.4 What are the basic questions in performance auditing?
All government programs or undertakings (and most processes they
generate) can, at least in theory, be analysed with the use of a formula
that describes how to move from one position to another by certain

14
means in order to achieve specifi c objectives. In performance auditing,
this is often done by trying to answer two basic questions:
• Are things being done in the right way?
• Are the right things being done?
The fi rst question is primarily aimed at the ‘producer’ and is con-cerned
with whether policy decisions are being carried out properly.
This question is usually associated with a normative perspective, i.e.
the auditor wants to know whether the executive has observed the
rules or the requirements. In order to widen the analysis, the ques-tion
may be extended to whether the activities carried out are also
considered the most appropriate – provided that the right things are
being done. Until this stage in the process, performance auditing has
been mainly concerned with different aspects of the economy or the
effi ciency of operations.
The scope for analysis becomes considerably wider when the second
question – whether the right things are being done – is asked. In other
words, whether the adopted policies have been suitably implemented
or whether adequate means have been employed.
This kind of question refers to effectiveness or impact on society.
In fact, the question might even imply that a government undertaking
– or a chosen measure to achieve a certain objective – runs the risk of
being contested. A performance auditor might, for instance, fi nd a cho-sen
measure ineffective and inconsistent with objectives. However, the
moment auditors start asking whether the public commitment itself is
feasible at all they will also have to be cautious not to go beyond their
mandate by crossing the borderline into political territory.
The so-called input–output model is another means of illustrating
these interactions. The model assumes a fl ow as shown below.
Commitment Input Action/production Output Outcome
Purpose Resources Action Services Objectives
defined assigned done provided met
Outputs are the result of inputs and actions taken to achieve specifi c
goals. Theoretically, it should be possible for performance auditing to
scrutinize all components and relations in the input-output model, ex-cept
for the component on the far left. The two basic questions given
above are still relevant, as is the wide range of perspectives that can be
applied to answer them.

1.5 What does auditing of economy, efficiency and effectiveness
mean?
As stated above, performance auditing is mainly concerned with the
examination of economy, effi ciency, and effectiveness.5 According to the
Auditing Standards (AS 1.0.40), an individual performance audit may
have the objective of examining one or more of these three aspects.
Economy – keeping the costs low
According to the Auditing Standards, ‘economy’ means minimising
the cost of resources used for an activity, having regard to appropriate
quality.
Audits of economy may provide answers to questions such as:
• Do the means chosen or the equipment obtained – the inputs –
represent the most economical use of public funds?
• Have the human, fi nancial or material resources been used economi-cally?
• Are the management activities performed in accordance with sound
administrative principles and good management policies?
Even though the concept of economy is well defi ned, an audit of
economy is not that easy to conduct. It is often a challenging task for
an auditor to assess whether the inputs chosen represent the most eco-nomical
use of public funds, whether the resources available have been
used economically, and if the quality and the quantity of the ‘inputs’
are optimal and suitably co-ordinated. It may prove even more diffi cult
to be able to provide recommendations that will reduce the costs with-out
affecting the quality and the quantity of services.
Efficiency – making the most of available resources
Effi ciency is related to economy. Here, too, the central issue concerns
the resources deployed. The main question is whether these resources
have been put to optimal or satisfactory use or whether the same or
similar results in terms of quality and turn-around time could have
been achieved with fewer resources. Are we getting the most output – in
ttteeerrrmmmsss ooofff qqquuuaaannntttiiitttyyy aaannnddd qqquuuaaallliiitttyyy ––– fffrrrooommm ooouuurrr iiinnnpppuuutttsss aaannnddd aaaccctttiiiooonnnsss??? TThhee qquueess--
tion refers to the relationship between the quality and quantity of
services provided and the activities and cost of resources used to pro-duce
them, in order to achieve results.
5 Standards concerning ‘environmental considerations’ and ‘equity requirements’ are also taken into
account in performance auditing. (See Appendix 6.)

16
Clearly, any opinion or fi nding on effi ciency is usually only relative,
while occasionally ineffi ciency is immediately apparent. A fi nding on
effi ciency can be formulated by means of a comparison with similar
activities, with other periods, or with a standard that has explicitly
been adopted. Sometimes standards, such as best practices, are appli-cable.
Assessments of effi ciency might also be based on conditions that
are not related to specifi c standards – when matters are so complex
that there are no standards. In such cases, assessments must be based
on the best available information and arguments and in compliance
with the analysis carried out in the audit.
Auditing effi ciency embraces aspects such as whether:
• human, fi nancial, and other resources are effi ciently used;
• government programs, entities and activities effi ciently managed,
regulated, organised, executed, monitored and evaluated;
• activities in government entities are consistent with stipulated
objectives and requirements;
• public services are of good quality, client-oriented and delivered on
time; and
• the objectives of government programs are reached cost effectively.
The concept of cost-effectiveness concerns the ability or potential
of an audited entity, activity, program or operation to achieve certain
outcomes at a reasonable cost. Cost-effectiveness analyses are studies
of the relationship between project cost and outcomes, expressed as
cost per unit of outcome achieved. Cost effectiveness is just one ele-ment
in the overall examination of effi ciency, which might also include
analyses of, for example, the time in which outputs were delivered.
This, however, does not always coincide with the optimal timing with a
view to optimising impact.
In some cases it may prove diffi cult to totally separate the two
concepts – effi ciency and economy – from each other. They may both
directly or indirectly, concern whether, for instance, the audited entity:
• is following sound procurement practices;
• is acquiring the appropriate type, quality, and amount of resources
at an appropriate cost;
• is properly maintaining its resources;
• is using the optimum amount of resources (staff, equipment and
facilities) in producing or delivering the appropriate quantity and
quality of goods or services on time;
• is complying with requirements of regulations that govern/affect the
acquisition, maintenance and use of the entity’s resources; and

• has established a system of management controls.
In reality, audits of economy tend to focus on the fi rst three points.
The concept of effi ciency is mainly restricted to the question of wheth-er
the resources have been put to optimal or satisfactory use. Conse-quently,
effi ciency is mostly specifi ed in two possible ways: whether
the same output could have been achieved with fewer resources, or, in
other words, if the same resources could have been used to achieve bet-ter
results (in terms of quantity and quality of the output).
Financial auditing is also engaged in these issues, for instance when
auditing procurement practices. However, in fi nancial auditing the
scope is more limited. Unlike performance auditing, the objective
strictly relates to fi nancial accountability.
Effectiveness – achieving the stipulated aims or objectives
Effectiveness is essentially a goal-attainment concept.6 It is concerned
with the relationship between goals or objectives, outputs and impacts.
Are the stipulated aims being met by the means employed, the outputs
produced and the impacts observed? Are the impacts observed really the
result of the policy rather than other circumstances?
The question of effectiveness consists of two parts: fi rst, if the policy
objectives have been achieved, and second, if this can be attributed
to the policy pursued. In order to judge the extent to which the aims
have been achieved, they need to be formulated in a way that makes an
assessment of this type possible. This cannot easily be done with vague
or abstract goals. In order to judge the extent to which observed events
could be traced back to the policy, a comparison will be needed. Ide-ally,
this consists of a measurement before and after the introduction
of the policy and a measurement involving a control group, which has
not been subject to the policy.7
In practice, such comparisons are usually diffi cult to make, partly
because comparative material is often lacking. In such cases, one
alternative is to assess the plausibility of the assumptions on which the
policy is based. Often a less ambitious audit objective will have to be
chosen, such as assessing to what extent objectives have been achieved,
target groups have been reached, or the level of performance.
6 That is, the extent to which a program or entity is achieving its goals and objectives.
7 The term policy covers both government policy and agency policy (see section 2.1 and footnote 25).
There are always diffi culties involved in conducting performance audits when the policy objectives are
vague and abstract. For more information, see Appendix 2.

18
The auditor might seek to assess or measure effectiveness by com-paring
outcomes – or ‘impact’, or ‘state of things’ – with the goals set
down in the policy objectives. This approach is often described as
‘goal achievement’ analysis. However, when auditing effectiveness, one
should usually try also to determine to what extent the instruments
used have in fact contributed towards the achievement of the policy
objectives. This is effectiveness auditing in its ‘true’ application and
requires evidence that the outcomes, which have been observed, have
actually been caused by the action in question rather than by some
other factors. For example, if the policy objective is to reduce un-employment,
is an observed reduction in the numbers of unemployed
the result of the actions of the audited entity, or is it the result of a
general improvement in the economic climate over which the audited
entity had no infl uence? Here, the design of the audit must include
questions of attribution and be able to cope with the problem of ef-fectively
excluding external, intermediary variables.
Side effects SSiiddee eeffffeeccttss –– aa sseeppaarraattee aassppeecctt ooff ppeerrffoorrmmaannccee aauuddiittiinngg iiss tthhee uunniinn--
tended side effects of policy. The study of side effects is complicated by
the fact that they can be very diverse, since they are not limited by the
policy objectives. One possible way of limiting the scope of the inves-tigation
is to focus on those side effects, which, in other situations, one
tries to avoid (e.g. unfavourable environmental effects of economic
policy). This does not mean, however, that all side effects are undesir-able.
In auditing effectiveness, performance auditing may, for instance,
• assess whether government programs have been effectively prepared
and designed and whether they are clear and consistent;
• assess whether the objectives and the means provided (legal, fi nan-cial,
etc.) for a new or ongoing government program are proper,
consistent, suitable, or relevant;
• assess the effectiveness of the organizational structure, decision-making
process and management system for program implementa-tion;
• assess whether the program supplements, duplicates, overlaps, or
counteracts other related programs;
• assess whether the quality of the public services meets the people’s
expectations or the stipulated objectives;
• assess the adequacy of the system for measuring, monitoring and
reporting a program’s effectiveness;

• assess the effectiveness of government investments and programs
and/or their individual components, i.e. ascertain whether goals and
objectives are met;
• assess whether the observed direct or indirect social, economic and
environmental impacts of a policy are due to the policy or to other
causes;
• identify factors that inhibit satisfactory performance or goal fulfi ll-ment;
• analyse causes of fi ndings and observed problems in identifying
ways of making government activities and programs work more ef-fectively;
and
• identify the relative utility of alternative approaches to yield better
performance or eliminate factors that inhibit program effectiveness.
While a particular audit will not necessarily seek to reach conclu-sions
on all three aspects (i.e. economy, effi ciency, and effectiveness), it
may be of limited benefi t to examine aspects of economy or effi ciency
of activities in isolation without also considering, at least briefl y, their
effectiveness. Conversely, in an audit of effectiveness, the auditor may
also wish to consider aspects of economy and effi ciency: the outcomes
of an audited entity, activity, program or operation may have had the
desired impact, but were the resources employed to achieve this used
economically and effi ciently?
For the examination of effectiveness, it is generally necessary to
assess the outcome or impact of an activity. Thus, while a ‘system-based
approach’ may be useful (to assess, e.g. how the audited entity
measures and monitors its impact), the auditor will usually also need
to obtain suffi cient substantive evidence of the impact of the activity
or the program. Likewise, in order to assess the impact of an activity or
a government reform, it is in general always necessary to collect infor-mation
not only on the audited institutions and their activities and in-teractions,
but also on other stakeholders in the area. This is of course
of special interest when it is believed that actions of other stakeholders
may infl uence the impact.8
One specifi c aspect is the study of unintentional effects, especially if
these effects were negative. There is a problem of demarcation here, be-cause
these effects may spread into areas beyond the competence and
powers of the SAI. One way of limiting the scope might be to look at
8 The scope must be limited. The analyses, however, should not be too limited.

20
those unintentional effects that are being combated in other programs,
environmental side effects of an economic stimulation program, for
example.9
All other things being equal, economy is about keeping the cost low,
effi ciency is about getting the most or best output from available re-sources,
and effectiveness is about achieving the stipulated aims or
objectives.
1.6 How does public management affect performance auditing?
The form of public management employed will necessarily infl uence
priorities in performance auditing. In countries where public manage-ment
is mainly concerned with means and less involved with ends,
audits also tend to focus on whether rules have been observed and
enforced rather than whether the rules serve or are seen to serve their
intended purpose. In countries that have acknowledged management
by objectives and results, the audit focus is different. Public sector
management generally displays a combination of these philosophies.
As mentioned above, management by objectives and results tends
to promote interest in auditing effi ciency and effectiveness. As a result,
the auditor might not have to confront a traditional, rule-bound
government administration but an administration whose mandate
has been widened considerably in terms of how the intentions of the
legislature should be put into operation and which means should be
employed in order to achieve them.
Typically, the following questions would be of interest to a perform-ance
auditor:
• Is there a clear structure of performance goals and have the appro-priate
priorities and instruments been chosen for the use of public
funds?
• Is there a clear distribution of responsibility between the different
levels of authority, bearing in mind the principle of subsidiarity?
• Is there a general cost awareness and an orientation towards produc-tion
of services, putting citizens’ needs in focus?
9 For more information: Auditing Effi ciency, Offi ce of Auditor General (Canada), 1995 and Auditing
Policy Results Manual, Algemene Rekenkamer (Netherlands), 1997.

• Is there an adequate emphasis on management controls and report-ing
requirements?
The ministries and their subordinate bodies are responsible for
ensuring that good internal control routines are established. In this
context, it is the particular task of the performance auditor to keep an
eye on whether this responsibility has been properly taken care of. The
extent to which it has in fact also been observed by the auditee or the
auditees in their operations is for the fi nancial auditor to judge.10
In addition, a common objective of most governments today is to
improve the quality of public services, particularly as people’s expecta-tions
(often with reference to the service they receive from the private
sector) of what constitutes quality continue to increase. To promote
improvements of this type, many governments have embarked on
modernisation programs to deliver better services that are, for in-stance,
more easily accessible and convenient, provide citizens with
more choice, and are delivered more quickly. The quality of public
services is an increasingly important issue, which members of parlia-ments
and governments across the world expect the SAIs to address in
their performance audit reports.
Countries that have acknowledged management by objectives and re-sults
tend to focus more on performance than before. The form of
public management employed will infl uence the interest in perform-ance
auditing.
1.7 How does performance auditing relate to performance
measurement and program evaluation?
Both the executive branch and the legislature need evaluative informa-tion
to help them make decisions about the programs they are respon-sible
for-information that tells them whether, and in what important
ways, a government undertaking or program is working well or poorly,
and why. Many analytical approaches have been employed over the
years by agencies and others to assess the operations and results of
government programs, policies, activities, and organizations. Perform-ance
audit and evaluation studies are designed to judge how specifi c
10 In recent years, experience of auditing government administration policies and administrative
reforms has been frequently discussed by SAIs.

22
programs are working and thus may differ a great deal. One particular
aspect is the relationship between performance measurement, program
evaluation, and performance auditing.
Performance measurement
Performance measurement normally means the ongoing process of
monitoring and reporting on program accomplishments, particularly
progress towards pre-established goals. Performance measures may
address the type or level of program activities conducted (process), the
direct products and services delivered by a program (outputs), and/or
the results of those outputs (outcomes). Performance measurement
focuses on whether a program has achieved its objectives or require-ments,
expressed as measurable performance standards. Performance
measurement, because of its ongoing nature, can serve as an early
warning system to management and as a vehicle for improving ac-countability
to the public.
The ongoing process of ensuring that a government program or
body has met the targets set is a matter of internal management and
control, not a task for external auditors. It is the responsibility of the
fi nancial auditors – not the performance auditors – to confi rm that the
accounts are correct. However, in the area of performance measure-ment
– the check on the quality of performance-related information
produced by the executive branch for the legislature – both fi nancial
and performance auditors might be involved, either in separate activi-ties
or in joint audits.11 Performance indicators can sometimes also be
used as indicators or references in planning individual performance
audits. One topic for performance auditing is whether performance
measurement systems in government programs are effi cient and effec-tive.
For example, questions could be developed that address whether
the performance indicators measure the right things or whether the
performance measurement systems involved are capable of providing
credible measured results.12
11 Ad hoc and in-depth studies of performance measurement systems are typically a task for perform-ance
auditing in any SAI. The ongoing or regular performance reports of different government institu-tions,
however, may just as well be an audit task for the fi nancial auditors (sometimes in co-operation
with performance auditors, for instance if the fi nancial auditors have not been trained to conduct audits
of this type).
12 See, e.g. AAASSS 111...000...222777 aanndd 1.0.45.

Program evaluation and performance auditing
Program evaluations are individual systematic studies conducted to
assess how well a program is working. Program evaluations typically
examine a broader range of information on program performance and
context than is feasible to monitor on an ongoing basis. A program
evaluation may thus allow for an overall assessment of whether the
program works and what can be done to improve its results. Program
evaluations are one type of study that might be executed by a SAI un-der
the general heading of performance audits.
In recent years, the concept of program evaluation has been a
growing subject of discussion amongst SAIs. Whether or not program
evaluation is an important task for a SAI has been discussed. A special
group (INTOSAI Working Group on Program Evaluation) has been
set up to promote principles and guidance in this area. It is generally
accepted that program evaluation has objectives identical or similar to
those of performance auditing in that it seeks to analyse the relation-ship
between the objectives, resources, and results of a policy or pro-gram.
It has also been agreed that program evaluation is an important
task for a SAI that has the authority and competence to carry out such
studies.
Program evaluation has been described as an epitome of activi-ties
and methods that have aim to make exhaustive assessments of an
issue, using more or less sophisticated scientifi c approaches. Although
performance auditing may use the same approaches and methodolo-gies
as program evaluation, it does not, according to the INTOSAI
Working Group on Program Evaluation, necessarily engage in assess-ing
policy effectiveness or policy alternatives. In addition to examining
the impact of outputs, program evaluation may include issues such
as whether the stipulated aims are consistent with general policy. This
issue has been the subject of discussion among SAIs. Some SAIs have
the right to evaluate government and/or agency policy effectiveness
and include program evaluation in their performance audit mandate.
Others are not required to conduct such audits.
According to INTOSAI’s Working Group on Program Evaluation,
auditing and evaluation may be divided into the following seven cat-egories:
13
13 INTOSAI Working Group on Program Evaluation (1995), draft summary report.

24
• RRReeeggguuulllaaarrriiitttyyy aaauuudddiiittt::: aarree rreegguullaattiioonnss ccoommpplliieedd wwiitthh??
• EEEcccooonnnooommmyyy aaauuudddiiittt::: ddoo tthhee mmeeaannss cchhoosseenn rreepprreesseenntt tthhee mmoosstt eeccoonnoommiiccaall
use of public funds for the given performance?
• EEEffffififi ccciiieeennncccyyy aaauuudddiiittt::: aarree tthhee rreessuullttss oobbttaaiinneedd ccoommmmeennssuurraattee wwiitthh tthhee
resources employed?
• EEEffffffeeeccctttiiivvveeennneeessssss aaauuudddiiittt::: aarree tthhee rreessuullttss ccoonnssiisstteenntt wwiitthh tthhee ppoolliiccyy??
• EEEvvvaaallluuuaaatttiiiooonnn ooofff ttthhheee cccooonnnsssiiisssttteeennncccyyy ooofff ttthhheee pppooollliiicccyyy::: aarree tthhee mmeeaannss eemmppllooyyeedd bbyy
the policy consistent with the set objectives?
• EEEvvvaaallluuuaaatttiiiooonnn ooofff ttthhheee iiimmmpppaaacccttt ooofff ttthhheee pppooollliiicccyyy::: wwhhaatt iiss tthhee eeccoonnoommiicc aanndd
social impact of the policy?
• Evaluation of the effectiveness of the policy and analysis of causality:
are the observed results due to the policy, or are there other causes?
In practice classifi cations vary. One SAI with many years’ experience
of program evaluation is the General Accounting Offi ce of the US.
It defi nes four common types of program evaluations in performance
auditing:14
(1) Process evaluation
This assesses the extent to which a program is operating as intended.
Typically, it is concerned with the program activities’ conformity with
statutory and regulatory requirements, program design, and profes-sional
standards or customer expectations. It is increasingly important
to assess whether the quality of the operations – for instance applica-tion
forms, processing times, service deliveries and other client-ori-ented
activities – meets the people’s expectations.
(2) Outcome evaluation
This assesses the extent to which a program achieves its outcome-oriented
– and client-oriented – objectives. It focuses on outputs and
outcomes (including side effects and unintended effects) in order to
judge program effectiveness, but it may also put emphasis on quality
issues and client perspectives. An outcome evaluation may also assess
program processes in order to fully understand a program and how
outcomes are produced.
(3) Impact evaluation
This assesses the net effect of a program by comparing program out-comes
with an estimate of what would have happened in the absence
14 Performance Measurement and Evaluation: Defi nitions and Relationships (GAO/GGD-98-26),
General Accounting Offi ce (USA), 1998.

of the program. This form of evaluation is employed when external
factors are known to infl uence the program’s outcomes, in order to
isolate the program’s contribution to the achievement of its objectives.
(4) Cost-benefit and cost-effectiveness evaluations
These are analyses that compare a program’s outputs or outcomes
with the costs (resources expended) to produce them. When applied to
existing programs, they are also considered a form of program evalu-ation.
Cost-effectiveness analysis assesses the cost of meeting a single
goal or objective, and can be used to identify the least costly alternative
to meet that goal. Cost-benefi t analysis aims at identifying all relevant
costs and benefi ts.15
In the area of performance measurement both fi nancial and perform-ance
auditors might be involved. In some countries, an individual per-formance
audit may include many different kinds of studies and even
several program evaluations. In that sense program evaluation may be
considered one of many possible ‘tools’ that performance auditing uses.
Program evaluation is one type of study that might be executed by the
SAI under the general heading of performance auditing. It is an activity
of increasing interest and importance.16
1.8 Are there differences in analytical ambitions and approaches?
The mandate and orientation of performance auditing may, as stated
above, vary in different countries. A number of SAIs are not required
to execute performance audits or may consider themselves somewhat
limited in their capacity and experience in respect of these audits.
Other SAIs may have a long tradition of carrying out both advanced
performance audits and complex program evaluations. One of the
characteristics of auditing is the normative perspective where dis-crepancies
between ‘the norms and the reality’ – the actual fi ndings
– are expressed explicitly, and assessments and recommendations are
provided as ‘normative’. However, as well as being normative, perform-ance
auditing is usually descriptive, and may also include analytical
15 Some SAIs may as part of their mandate also include ‘Policy evaluation’ (the effectiveness of the
policies set), and a few SAIs conduct what they defi ne as ‘System evaluation’ (the appropriateness of the
systems adopted).
16 Other areas of increasing interest are performance audits of activities with an environmental perspec-tive
and performance auditing concerning information technology. (See Appendices 5 and 6.)

26
elements. (A performance audit may, for instance, ascertain the causes
of the difference between the conditions and the criteria.)
The results-oriented and the problem-oriented approaches
Performance auditing has various traditions and ambitions. Two ap-proaches
differ quite signifi cantly, although each is based on national
standards for performance auditing. They are called the results-ori-ented
and the problem-oriented approaches.
The results-oriented approach deals mainly with questions such as:
‘What is the performance or what results have been achieved, and have
the requirements or the objectives been met?’ In this approach, the au-ditor
studies performance (concerning economy, effi ciency, and effec-tiveness)
and relates observations to the given norms (goals, objectives,
regulations, etc.) or the audit criteria (more or less precisely defi ned
before the main study begins). If the criteria are diffi cult to determine,
the auditor may need to work with experts in the fi eld to develop cred-ible
criteria that, when applied, are objective, relevant, reasonable and
attainable. The audit criteria make it possible to provide assessments
on the fi ndings. In this approach, shortcomings are likely to be defi ned
as deviations from norms or criteria. Recommendations, if presented,
are often aimed at eliminating such deviations. The perspective is in
that sense basically normative.
The problem-oriented approach, on the other hand, deals primarily
with problem verifi cation and problem analysis, normally without
reference to predefi ned audit criteria.17 In this approach, shortcomings
and problems – or at least indications of problems – are the starting
point of an audit, not the conclusion.18 A major task in the audit is to
verify the existence of stated problems and to analyse their causes from
different perspectives (problems related to economy, effi ciency, and
effectiveness of government undertakings or programs). The problem-
17 In order to assess a problem one must fi rst understand it (and its possible intricate causes and impli-cations).
When it comes to analyses of complex problems of effi ciency and effectiveness, it is not always
possible to defi ne the audit criteria in the planning stage.
18 Indications of problems concerning effi ciency and effectiveness are often vague, subjective, and dif-fi
cult to defi ne and understand. Examples of possible indications of problems concerning the three Es
(economy, effi ciency, and effectiveness) are: rising costs resulting in demands for more resources; stated
imbalances between inputs and goals; lack of clarity in the allocation of responsibility between executive
bodies involved; ambiguities and contradictions in regulation; long waiting times or large backlogs;
stated lack of competence, criticism of management style, shortcomings in services and client-orienta-tion;
large numbers of complaints or appeals by the public; changes in external conditions; and indica-tions
of negative side effects of government programs.

oriented approach deals with questions such as: ‘Do the stated prob-lems
really exist and, if so, how can they be understood and what are
the causes?’ Hypotheses about possible causes and consequences are
formulated and tested.19 The perspective is analytical and instrumental;
the aim is to deliver updated information on the stated problems and
how to deal with them. The auditors are not restricted in their analy-ses.
20 All possible material causes are considered (only general goals
are taken for granted), so proposals to amend laws, regulations, and
structural design of government undertakings are not excluded, if is
shown that the existing structure give rise to severe and verifi ed prob-lems.
21
Thus, assessment in these two performance audit approaches might
be derived normatively (based on deviations from norms or criteria)
or analytically (based on analyses of the specifi c causes of problems).
In fact, it is the independent analysis that characterizes the problem-oriented
approach, while the results-oriented approach is mainly
characterized by its impartial assessment of whether given norms or
criteria have been met (even though it may involve analytical elements
as well). On the one hand, the results-oriented and the problem-ori-ented
approaches represent different audit traditions.22 On the other,
the approaches may also serve to illustrate the fact that performance
auditing includes various types of practical methods.23
Top-down and bottom-up perspectives
The perspectives of the two objectives may also vary. Performance
auditing is normally based on an overall owner perspective, that is, a
top-down perspective. It concentrates mainly on the requirements,
intentions, objectives and expectations of the legislature and central
government. In some countries, however, it is also possible – within
the framework of given objectives and premises – to add a ‘client-ori-ented
perspective’ (a focus on service-management, waiting-time, and
other issues relevant to the ultimate clients or consumers involved).
19 A hypothesis is a well-founded (testable) statement regarding causes and consequences of the prob-lem
to be audited (based on the assumption that the problem exists).
20 They are not limited to analyses of differences between conditions and audit criteria.
21 For more information on the problem-oriented approach, see for instance Handbook on Perform-ance
Auditing, RRV (Sweden), 1998.
22 They can be said to represent different levels of ambition.
23 The two methodological approaches can also been seen as linked to each other in terms of different steps
in an audit. Even if the problem-oriented approach by nature goes wider and deeper in its analytical ambi-tion,
the results-oriented approach may in its advanced form also allow for sophisticated analyses.

28
This might be viewed as an interpretation of the audit mission in order
to meet citizens’ interest in having SAIs focus on problems of real
signifi cance to the people and the community – a kind of bottom-up
perspective.
Focus on accountability or causes to the problems
Auditing is normally associated with accountability, but in perform-ance
auditing this is not always the case. Auditing accountability can
be described as judging how well those responsible at different levels
have reached relevant goals and met other requirements for which
they are fully accountable, (factors outside the control of the auditees
are not expected to infl uence the outcome). An alternative approach
is to focus on understanding and explaining the actual observations
that have been made during the audit. Instead of trying to fi nd out
who is at fault, it is possible to analyse the factors behind the problems
uncovered and to discuss what may be done about these problems.
This approach refl ects the idea that the overall aim of performance
auditing is to promote economy, effi ciency and effectiveness. The two
approaches represent different ideas to performance auditing; one in
which accountability (as in compliance and fi nancial auditing) is at the
centre of attention of the audit, while the other – which put emphasis
on economy, effi ciency and effectiveness – primarily concerns itself
with the subject matter of the audit causes of problems observed.
(Even if the main area of focus is not accountability, recommendations
arising from the audit as a rule encompass the question of division of
responsibility.)
Accountability auditing has the advantage that it is often easier
to carry out and that it corresponds to the conventional picture of
auditing. The problem, however, is that effi ciency and effectiveness
are complex issues which demand more comprehensive analyses (of
conditions and circumstances outside the control of the auditees).
Accountability auditing also involves risks – the perspective and scope
must be limited, which in turn unduly limits the possibility of mak-ing
an independent analysis. If, on the other hand, focus is placed on
problems observed and possible causes, this facilitates audits covering
the areas of responsibility of several different parties. Conditions are
thereby created for comprehensive analyses. It must be stated, however,
that this approach makes greater demands in terms of the skills of
auditors.

The message of this section is that there are also differences in meth-odological
approaches with respect to analytical ambitions. Generally
speaking, there are SAIs that have established high analytical ambitions
in their performance auditing, while others have settled for a lower
level.
Performance auditing should not be streamlined. Advanced perform-ance
auditing is complex investigatory work that requires fl exibility,
imagination and high levels of analytical skills. Streamlined procedures,
methods and standards may in fact hamper the functioning and the
progress of performance auditing. Consequently, standards – as well
as quality assurance systems – that are too detailed should be avoided.
Progress and practices must be built on learning from experience.
The orientation of performance auditing varies between SAIs. Two ap-proaches
differ more signifi cantly, namely the results-oriented and the
problem-oriented approaches. The results-oriented approach deals
with questions such as: ‘What is the performance, and have the objec-tives
been met?’ The problem-oriented approach deals primarily with
problem analysis. It deals with questions such as: ‘Do the stated prob-lems
really exist and, if so, what are the causes to the problems?’ Per-formance
auditing may apply both top-down and bottom-up perspec-tives.
Auditing is normally associated with accountability, but in
performance auditing this is not always the case. Performance auditing
should not be guided by standards that are too detailed and stream-lined.
This may hamper creativity and professionalism.
1.9 Summary
• Performance auditing examines the economy, the effi ciency and the
effectiveness of government programs and organizations and an-swers
questions such as: Do the inputs chosen represent the most
economical use of public funds? Are we getting the best services
from available resources? Are the aims of the policy being fully met,
and are the impacts the result of the policy? The perspectives and the
objects to be audited may vary, i.e. both individual agencies as well
as government-wide undertakings may be audited. Performance au-diting
is based on decisions made and goals set by the legislature,
and it may be carried out throughout the whole public sector.

30
• Performance auditing is not a regular audit with formalized opinions.
It is an examination made on an ad hoc basis. It is an audit that fo-cuses
on performance, rather than expenditure and accounting. It
has its roots in the requirements for independent analyses of the
economy, effi ciency, and effectiveness of government programs and
organizations. The special feature of performance auditing is partly
due to the variety and complexity of questions related to its work.
• All government activities can be analysed with the use of a formula
that describes how to move from one position to another by certain
means in order to achieve specifi c objectives. In performance audit-ing,
this is often done by trying to answer two basic questions: Are
things being done in the right way? Are the right things being
done?
• The ongoing process of ensuring that a government program or body
has met the targets set is a matter of internal management and con-trol.
However, in the area of performance measurement, both fi nan-cial
and performance auditors might be involved.
• Apart from examining the impact of outputs, program evaluation
may include issues such as whether the objectives are consistent
with the policy or with the options given for changing the policy in
order to achieve outcomes that are more effective. In some coun-tries,
performance audits may include many kinds of studies and
even several program evaluations. In that sense program evaluation
may be considered one of many possible ‘tools’ that performance
auditing uses.
• Performance auditing has various traditions. Two approaches differ
quite signifi cantly. The results-oriented approach deals mainly with
questions such as: ‘What is the performance or what results have
been achieved, and have the requirements or the objectives been
met?’ The problem-oriented approach deals primarily with questions
such as: ‘Do the stated problems really exist and what causes them?’
Auditing is usually associated with accountability, but in perform-ance
auditing this does not always have to be the case.
• Performance auditing should not be streamlined. It is investigatory
work that requires fl exibility, imagination and analytical skill. Stream-lined
and detailed procedures, methods and standards may in fact
hamper the functioning of performance auditing.

Part 2: Government auditing principles applied
to performance Auditing
In conducting a performance audit the auditors should follow the
INTOSAI Code of Ethics and Auditing Standards as well as relevant
SAI standards and guidelines applicable to performance auditing. The
INTOSAI general auditing standards states that the audit and the SAI
must be independent, possess required competence and exercise due
care (AS 1.0.6 and 2.2.1.). 24
2.1 How do the auditing principles apply to performance auditing?
The audit mandate and the general goals should be properly defined
Statutes generally lay down the audit mandate. Among other things it
regulates the extent to which a SAI can audit public sector programs
and organizations. Special regulations are often needed that specify the
conditions for performance auditing, for example, access to informa-tion
from sources other than the auditees, the ability to give recom-mendations,
the mandate to scrutinize government undertakings and
programs, and the effectiveness of legislation. The mandate ordinarily
specifi es the minimum audit and reporting requirements, specifi es
what is required of the auditor, and provides the auditor with author-ity
to carry out the work and report the results (AS 2.2.12, 2.2.19,
1.0.32-38, 1.0.42 and 1.0.47).
If possible, the mandate should cover the whole state budget, includ-ing
all relevant executive bodies and all corresponding government
programs or public services. Without suffi cient legal support, it might
even be considered illegal to publish justifi ed criticism of the effi ciency
and effectiveness of government programs, at least in respect to issues
that are politically sensitive. To avoid this situation – and suspicion of
24 For more information see these INTOSAI-documents: The Lima Declaration, The Code of Ethics and
Auditing Standards, and Independence of Supreme Audit Institutions.

32
self-censorship – the mandate needs to be both politically and publicly
supported (AS 2.2.18-20 and 2.2.23)
The audit objects (those that might be subjected to performance au-dits
by the SAI according to the mandate) can be described as ‘policy’,’
programs’, ‘organization’ pprrooggrraammss’’,, ‘‘oorrggaanniizzaattiioonn’’ aaannnddd ‘‘‘mmmaaannnaaagggeeemmmeeennnttt’’’... ‘‘‘PPPooollliiicccyyy’’’ iiss uussuuaallllyy ddeefifi nneedd
as an effort to achieve certain aims with certain resources and perhaps
within a certain time.25 A ‘ppprrrooogggrrraaammm’’’ ccaann bbee ddeessccrriibbeedd aass aa sseett ooff iinntteerr--
related means – legal, fi nancial, etc.– to implement a given govern-ment
or agency policy. ‘Organization’ can be defi ned in different ways,
but mostly it is taken to mean the aggregate of people, structures, and
processes that have the aim of achieving particular objectives. ‘Man-agement’
is generally taken to mean all the decisions, actions, and rules
for the steering, accounting and deployment of human, fi nancial, and
material resources. Management is often related to the internal opera-tions
of an organization. Policies and programs–decided by the leg-islature,
the executive or executive offi cials–may also have an internal
focus, relating to a specifi c organization (and its internal activities and
performance). But mostly their focus is wider and more external and
relates to activities of even non-governmental organizations (NGOs)
(and the impacts of the policies and programs in society, etc.).26
In many countries, the constitution or legislation gives the SAI the
explicit right to undertake some form of performance auditing. Some
SAIs may carry out examinations of the effi ciency and effectiveness of
complex government policies or undertakings, perhaps by in-depth
analyses of stated problems. Others are more limited in their approach.
As part of the explanation of standards, the Auditing Standards (AS
1.0.42) state: ‘In many countries, the mandate of performance auditing
will stop short of review of the policy bases of government programs.’
In these cases, performance auditing does not question the merits of
policy objectives but rather involves examinations of actions taken to
design, implement, or evaluate the results of these policies, and may
imply an examination of the adequacy of information leading to policy
decisions. Even in countries where the constitution or legislation does
not require the SAI to carry out audits of economy, effi ciency, and
25 The term policy covers in this guidelines – if nothing else is said – both government and agency
policy. The term policy can be used as equivalent to agency policy for SAIs who do not have the right
to review or evaluate government policy. (The term government undertaking covers both policy and
program.)
26 See for instance Manual of Performance Auditing, AG (The Netherlands), 1996.

effectiveness, current practice shows a tendency to include this sort of
work as part of fi nancial or regularity audits (AS 1.0.13, 1.0.42-43).
The general goals of performance auditing should also be defi ned in
the legislation or be decided on by the SAI. In general, SAIs may seek
to achieve one or more of the following general goals:
(1) To provide the legislature with independent examination of
the economical, effi cient, or effective implementation practices of
government policies or programs.
(2) To provide the legislature with independent, ad hoc analyses of
the validity and reliability of performance measurement systems, or
statements or self-evaluations about performance that are published
by executive entities.
(3) To provide the legislature with independent analyses of prob-lems
of economy, effi ciency, and effectiveness in government activi-ties
and thus contribute to improvements.
(4) To provide the legislature with independent assessments of the
intended and unintended direct or indirect impact of government
and agency programs and whether, and to what extent, stated aims
or objectives have been met or why they have not been met.
One common objective of performance auditing in many countries
– set by the legislator or the SAI itself – is to assess and improve the
functioning of government programs, central government itself and
any connected bodies (AS 1.0.20, 1.0.27, 1.0.40 and 4.0.25). Providing
recommendations is important in most countries. In others, recom-mendations
are not given at all, due to legal conditions and historical
traditions.27
Performance auditing must be free to select audit areas within
its mandate
According to the Auditing Standards (AS 2.2.10-19), a SAI must be
free to determine the areas covered by its performance audits. AS 2.2.8
states: ‘The SAI may give members of legislature factual briefi ngs on
27 Most SAIs provide recommendations in their performance audit reports. It has been claimed that
such a policy has inherent disadvantages. It could compromise the SAI’s independence and make
further examinations diffi cult. However, a SAI cannot be hold accountable for its recommendations,
and performance auditors can never claim to have found the only rational solutions (even if the recom-mendations
put forward are both logical and well founded, there are always other options). A SAI’s
recommendation can only be based on an assessment of what appeared at the time to be a rational, or
possibly the most rational, solution. Moreover, performance auditing is by nature a non-recurrent activ-ity.
It is therefore unlikely that a subject will be audited in the same way twice. For more information,
see Performance Auditing at the Swedish National Audit Bureau, 1993, pp 51.

34
audit reports, but it is important that the SAI maintain its independ-ence
from political infl uence, in order to preserve an impartial ap-proach
to its audit responsibilities. This implies that the SAI should
not be responsive, nor give the appearance of being responsive, to the
wishes of particular political interest.’ In paragraph 2.2.10 it is stated:
‘In some countries the audit of the executive’s fi nancial management
is the prerogative of the Parliament or elected Assembly; this may also
apply to the audit of expenditure and receipts at a regional level, where
external audit is the responsibility of a legislative assembly. In these
cases audits are conducted on behalf of that body and it is appropri-ate
for the SAI to take account of its requests for specifi c investigations
in programming audit tasks. It is nevertheless important that the SAI
remains free to determine the manner in which it conducts all its work,
including those tasks requested by the Parliament.’ It is also important
‘that there be no power of direction by the executive in relation to the
SAIs performance of its mandate’ (AS 2.2.14).
Performance audits should in general be ex post audits
The earliest point at which a SAI can examine effi ciency and effective-ness
is after the government has made a decision on the policy con-cerned
(this is more ore less implied in AS 4.0.22 and 4.0.25). An analy-sis
of objectives or an audit of policy preparation activities may be
carried out in some countries before the policy itself is implemented.
Even so, the problems that performance auditing focuses on – or aims
to eliminate – should be current problems in order to add value for the
user of the audit reports.
General aims of legislature should be taken for granted
Political decisions and goals established by the legislature are in general
the frame of reference, which form the basis of the audit criteria used
in performance auditing. It is not the role of a SAI to question these
decisions and goals. However, a SAI may, as a result of its fi ndings,
make critical comments on the goals, for example if they are incon-sistent
or if it proves impossible to follow up the extent to which they
have been achieved. Consequently, a performance audit report may in
fact question the merits of existing policies or decisions. The goals or
objectives may be too vague, in confl ict with other objectives, or based
on insuffi cient information. The policy may be ineffi cient and ineffec-tive,
and changes might be required if existing shortcomings are to be
overcome. On the other hand, it is defi nitely the role of performance

auditing to assess the economy, effi ciency and effectiveness of more
specifi c objectives and regulations established, for example, by govern-ment
agencies. (See AS 2.2.5 and 2.2.9.)
While performance auditing does not question political goals, it
can highlight the consequences of a given policy. It may also identify
and illustrate shortcomings resulting from confl icting goals. Thus,
performance auditing does not for example question the level of
compensation in social welfare systems. The auditors must have, as a
starting point, a set of problems that are related to economy, effi ciency,
and effectiveness in the welfare systems being audited. This might be
the case when, for example, a level of compensation in a given area has
unintended marginal effects in another area. The performance auditor
can then study the lack of coordination between different systems and
point out the resulting problems. If the actual level of compensation is
demonstrably different from the level that was originally set, the per-formance
audit can examine the reasons for this development.
High professional quality of work should be promoted and secured
INTOSAI’s Auditing Standards and its Code of Ethics states that all
government auditors should act with integrity, impartiality, objectiv-ity,
competence and professionalism. High ethical standards must be
exercised in order to serve the public interest best, and in AAASSS 222...222...333666 iitt
is stated, ‘Since the duties and responsibilities thus borne by the SAI
are crucial to the concept of public accountability, the SAI must apply
to its audits, methodologies and practices of the highest quality. It is
incumbent upon it to formulate procedures to secure effective exercise
of its responsibilities for audit reports, unimpaired by less than full
adherence by personnel or external experts to its standards, planning
procedures, methodologies and supervision.’
Performance audits are often complex undertakings, requiring a
wide range of skills, expertise, and experience. AAASSS 222...111...222666 ssttaatteess ‘‘BBeeccaauussee
of the importance of ensuring a high standard of work by the SAI, it
should pay particular attention to quality assurance programs in order
to improve audit performance and results.’ It is also stated that the SAI
should establish systems and procedures in order to ‘(a) confi rm [that]
the integral quality assurance processes have operated satisfactorily;
(b) ensure the quality of the audit report; and (c) secure improve-ments
and avoid repetition of weaknesses’ (AS 2.1.27). However, no
system for quality assurance can guarantee high quality performance
audit reports. It is, simply put, more important to have competent and

36
motivated staff than advanced systems for quality assurance. In other
words, systems for quality assurance should be relevant and easy to
manage, rather than complex and overly sophisticated.
According to INTOSAI, quality issues must be integrated in the
execution process. Even in the early planning stages, systems of quality
assurance might prove indispensable to ensure that the problems to
be addressed are material and well defi ned. The objectives, problems,
audit questions, and selected areas largely determine the quality of the
audit. The process of planning, and the various stages that make up the
decision-making process, ensure that quality is regularly assessed, since
certain conditions must be met before the audit can move forward.
Meticulous preparations are important to defi ne the audit questions,
the information needed, and the audit design (AS 2.1.27 and 3.1.1).
For more information, see Appendices 3 and 4.
The mandate of performance auditing should cover the state budget
and all corresponding government programs. The auditor must be free
to select audit areas within its mandate. Political decisions and goals
established by the legislature are the basic frame of reference. A per-formance
audit may, as a result of its fi ndings, question the merits of
existing policies. Performance audits are in general ex post audits that
deal with current issues. High levels of quality in the work must be
promoted and secured.
2.2 What are the general requirements for a performance auditor?
Performance auditors must posses specific professional skill
Performance auditing is an information-based activity, with profes-sional
values occupying a central position. These values include the
importance of auditors being given the opportunity to develop their
skills and attain good quality of results in their audits. This includes
creating an environment that is stimulating and that furthers quality
improvements (AS 1.0.45 and 2.1.9).
All auditors should possess adequate professional profi ciency to
perform their tasks (AS 2.2.1 and 2.2.33-38). The SAIs should recruit
personnel with suitable qualifi cations, adopt policies and procedures to
develop and train SAI employees to performance their tasks effectively,
prepare written guidance concerning the conduct of audits, support

the skills and experience available with the SAI and review the internal
procedures (AS 2.1.2).
The ability to recruit the right staff is a decisive factor in perform-ance
auditing. Each staff member is a unique investment. A perform-ance
auditor must be well educated, and in general it is required
that the auditor should have a university degree and experience in
investigative/evaluation work. Personal qualities are also of considera-ble
importance (analytical ability, creativity, receptiveness, social skills,
integrity, judgment, endurance, good oral and writing skills etc.). (AS
2.1.4 and 2.1.10.)
To become a performance auditor, a performance audit team-leader
or a performance audit manager, certain distinctive qualifi cations have
to be met. For instance, a performance auditor should be well edu-cated
in the social sciences and in scientifi c investigation/evaluation
methods. Special knowledge of the different functional areas to be
audited might also prove essential, but advanced skills in accounting
and fi nancial auditing are not always needed in performance auditing
or program evaluation. Where SAIs have organized their perform-ance
auditing separately from fi nancial auditing, it is quite accept-able
that personnel selected for performance auditing have different
backgrounds and skills than those selected for fi nancial auditing.28
To meet the quality requirements specifi ed in the Auditing Standards
(AS 2.2.36-39), the SAI should have a program to ensure that its staff
maintains professional profi ciency through continuous education and
training. A key factor in the development process is learning through
practical auditing work (AS 2.1.2, 2.2.37-38 and 2.1.16).
Continuous education and training may include such topics as cur-rent
developments in performance audit methodology, management
or supervision, qualitative investigation methods, case study analysis,
statistical sampling, quantitative data-gathering techniques, evaluation
design, data analysis, and reader-based writing. It may also include
subjects related to auditors’ fi eldwork, such as public administration,
public policy and structure, government administration policy, eco-nomics,
social sciences, or Information Technology science
(AS 2.1.6-10).
28 Unless being well experienced in performance auditing or similar work, a performance audit team-leader
or manager might run the risk of not being totally accepted (or respected) by the performance
auditors.

38
Qualifi cations for staff members who conduct performance audits
include
• knowledge of the methods applicable to performance auditing and
the education, skills, and experience needed to apply such knowl-edge;
• knowledge of government organizations, programs, and functions;
• skills to communicate clearly and effectively, orally and in writing;
and
• special skills depending on the nature of the specifi c audit (e.g.
statistics, information technology (IT), engineering etc, or expert
knowledge of the subject matter concerned (AS 2.2.33-38 and
2.1.11-12).
Performance auditing should be a team effort, since the issues
involved are complex. Consequently, not all members need to pos-sess
every skill mentioned above. Furthermore, it may not always be
possible for a SAI to recruit people who meet all the requirements. The
required skills may therefore be developed once a person is in service,
as long as candidates for appointment have clearly demonstrated the
potential and attitude for the kind of work that performance auditing
entails.
Effectiveness, professionalism and care must guide the audit work
The performance audit should be suffi ciently well defi ned and the
audit approach functional. The organization of the audit should satisfy
the general requirements of good project management (AS 3.0.2-3 and
3.1.1-3).
The performance audit must be carried out thoroughly, with the
aim of collecting relevant, reliable and suffi cient evidence in order to
enable anyone else to arrive at the same conclusions as the perform-ance
audit report. This calls for exercising sound judgment when
deciding the audit objective, what and when to audit, the approach and
methodology, the scope of the audit, the issues to be reported, and the
overall audit conclusion.
Good communication with the auditee(s) and experts from differ-ent
backgrounds is important during the entire audit process. Similar-ly,
performance audit managers must also be vigilant. It is important
that the factual basis of fi nal descriptions, analyses and recommenda-tions
is accurate. The report should be objective and balanced and have
a sober tone, with the purpose of adding value for the government. (AS
2.2.39, 3.5.1-2 and 4.0.22-25.)

The principles of proper administration should be observed. The
audit process should be well recorded. Important decisions made dur-ing
the course of the audit and the underlying considerations should
be recorded in writing. Accessible fi les and a logbook should be kept.
The main objectives of documentation – besides helping the auditing
team – are to record the audit evidence in support of conclusions and
decisions, to provide records to assist audit management and monitor-ing,
and to enable work to be reviewed by senior offi cers. Information
obtained during the audit should be treated as confi dential until the
report has been tabled (AS 2.2.39-40, 3.0.2-3, 3.1.1, 3.2.1, 3.5.2-7, and
4.0.24).
All government auditors should act with integrity, impartiality, ob-jectivity,
competence and professionalism. To meet these standards the
performance auditor must be adequately educated and have experience
of investigative/evaluation work. Personal qualities are also of consid-erable
importance. Effectiveness, professionalism and care must guide
the audit work.
2.3 Are there other important safeguards?
Although these guidelines set out a coherent basis for conducting a
performance audit, professional judgment (applied on the basis of
relevant rules and procedures) remains the most important ingredient
in performance audit work. The auditor should adopt an attitude of
professional scepticism throughout the audit, recognizing that circum-stances
may exist that could cause the information relating to perform-ance
to be materially misstated.
Various safeguards, both principal and practical, might have to be
applied in order to prevent material misstatements.
Reasonable assurance on the quality of information should be provided
A performance audit conducted in accordance with applicable audit-ing
standards must examine the quality of the information provided.
Performance auditing is increasingly dependent on the quality of
information produced by the auditees and others, often stored on elec-tronic
media. What is ‘reasonable’ depends on the situation, i.e. on the
kind of evidence at hand and the conclusions that can be drawn from
it. (AS 3.5.2)

40
The institutions concerned should be properly informed
The SAI should notify the government institutions responsible for or
involved in the audited program of the details of the audit, preferably
before it starts (AS 3.1.4). Due to the character of performance audit-ing,
it is important that senior offi cials are well acquainted with the
purpose of the audit.
The work performed should be properly supervised
The INTOSAI auditing standards state: ‘The work of the audit staff at
each level and audit phase should be properly supervised during the
audit, and documented work should be reviewed by a senior member
of the audit staff ’ (AS 3.2.1). When work is delegated to a member of
the audit team, the project manager should carefully direct, supervise,
and review the work delegated. All team members should understand
the objectives of the audit, the terms of reference of the work assigned
to them, and the nature of obligations imposed on them by applicable
auditing standards (AS 3.2.2).
Supervision of the performance audit team by senior members of
the audit staff involves directing, supporting and monitoring their
work to ensure that the audit objectives are met. (See Appendix 4.)
According to the Auditing Standards (AS 3.2.3), this involves ensuring
that
• all team members fully understand the audit objectives,
• audit procedures are adequate and properly carried out,
• international and national auditing standards are followed,
• audit evidence is relevant, reliable, suffi cient and documented, and
supports the audit fi ndings and conclusions; and
• audit budgets, timetables and schedules are met.
The use of experts requires special care
Experts are often used in performance auditing. Before using experts,
the auditor should ensure that the export has the necessary compe-tence
required for the purposes of the audit. An expert, if needed, is a
person or fi rm possessing special skills, knowledge, and experience in
a particular fi eld other than auditing. The auditor must ensure that the
expert is independent of the activity/program, and the experts should
be informed about the conditions and the ethics required. Although
the performance auditor may use the work of an expert as evidence,
the auditor retains full responsibility for the conclusions in the audit
report (AS 2.1.18, 2.2.43-45).

The auditors should notify the government institutions involved in the
audited program of the details of the engagement. They should provide
reasonable assurance that the information relating to performance is
reliable. Although the auditor may use the work of expert, the auditor
retains full responsibility for the conclusions.
2.4 Summary
• The mandate of performance auditing should be defi ned in the leg-islation,
and special regulations are often needed that specify the
conditions for performance auditing. The mandate should cover the
whole state budget, including all relevant government undertaking
and all relevant public services. The mandate should allow for audits
of both individual government institutions and large state-owned
enterprises of public interest as well as government-wide programs
from different perspectives. The performance auditor must be given
the freedom to select audit areas within the performance audit man-date.
• Political decisions and goals established by the legislature should be
the starting point for performance auditing. However, a SAI may, as
a result of its fi ndings, also make critical comments on them, for ex-ample
if goals are inconsistent or if it proves impossible to follow up
the extent to which they have been achieved.
• Performance audits should in general be ex post audits, but in some
countries an audit of policy preparation activities may be carried out
before the policy itself is implemented.
• All government auditors should act with integrity, impartiality, objec-tivity,
competence and professionalism but, due to its features, this
is of special importance in performance auditing. The performance
auditor must be well educated and have experience of investigative/
evaluation work. Personal qualities are also of considerable impor-tance
(analytical ability, creativity, receptiveness, social skills, integ-rity,
judgment and patience, as well as having good oral and written
skills).
• Performance auditing is a knowledge-based activity, and due to its
special features – and its close links to politics – high quality of work
is perhaps the most important single factor for recognition. To en-sure
high quality of work, the SAI should pay particular attention to
creating an environment for performance auditing that ensures in-centives
for good quality and quality improvements. A properly
functioning system for quality assurance is one of the important ele-ments
in this context.

42
• Effectiveness, professionalism and care must guide the audit work,
and the principles of proper administration should be observed. The
auditors should notify the government institutions involved in the
audited program of the details of the engagement.
• Before using experts, the auditor should ensure both that this is
necessary and that the experts are competent and independent. Al-though
the auditor may use the work of experts as evidence, the
auditor retains full responsibility for the conclusions.

Part 3: Field standards and guidance: Initiating and
planning the performance audit
3.1 What are the overall steps in the performance audit cycle?
As stated in AS 3.0.1:‘The purpose of fi eld standards is to establish the
criteria or overall framework for the purposeful, systematic and bal-anced
steps or actions that the auditor has to follow.’ The fi eld stand-ards
establish the framework for planning, conducting and managing
audit work (AS 3.0.2).
The performance audit cycle covers several steps. Broadly speak-ing,
it comprises the planning process, the execution process and the
follow-up process. The planning process is often divided into different
stages. The fi rst stage is strategic planning, where potential themes and
topics are analysed. Once a topic has been selected for performance
audit, a pre-study – resulting in a work plan for the main study – may
be undertaken to gather information in order to be able to design a
proposal for the main study.
Throughout the main study, the emphasis should be on the produc-tion
of a fi nal report to be considered by the government, the legisla-ture,
the executive bodies concerned, and the public. The report-writ-ing
process should, based on experience, be viewed as a continuous
one of formulating, testing and revising ideas about the topic. Issues,
such as the expected impact and value of the audit, should be consid-ered
throughout the audit. By setting deadlines for the writing process,
timely reporting may be enhanced.
Follow-up procedures identify and document audit impact and the
progress made in implementing audit recommendations. Such proc-esses
are vital to provide feedback to the SAI and the legislature. (AS
2.2.5-6)
The performance audit cycle involves several steps: strategic planning,
preparation work, the main study and follow-up activities.

44
3.2 What does strategic planning involve?
Performance auditing should be directed towards areas where an
external, independent audit may add value in promoting economy, ef-fi
ciency, and effectiveness. In fi nancial auditing, the audit objects (and
the perspectives to be applied) are often defi ned for the SAI by its own
basic legislation. As noted above, the SAI usually has greater freedom
in the choice of performance audit objects and audit approaches. The
SAI must carefully consider the strategy for selecting subjects for per-formance
audits that help to set priorities and make selections. Interest
in change shown by, for instance, the government may contribute to
this process (AS 2.1.21-22 and 3.1.1-2).
The choice of audit areas should take place without any outside
pressure (AS 2.2.14). The SAI must maintain its political neutrality, but
maintenance of the SAI’s independence does not preclude requests to
the SAI from the executive, proposing matters for audit. However, if it
is to enjoy adequate independence, the SAI must be able to decline any
such request (AS 2.2.16).
Strategic planning is the basis for the selection of audit topics and
possible pre-studies. The planning might be carried out with the fol-lowing
steps:
1. Determining the potential audit areas from which the strategic choices
are to be made. The selection of audit areas involves strategic choices
with consequences for the SAI. The number of potential areas is
considerable and the SAI’s capacity is limited. This means that
choices must be made with care (AS 2.1.21 and 2.2.38).
2. Establishing the selection criteria to be used for these choices. The main
selection criterion is probably the audit’s primary contribution to
the assessment and improvement of the functioning of central gov-ernment
and the bodies connected with it. (AS 3.0.1.)
As for step 2, the general selection criteria would be as follows:
• Added v AAddddeedd vvaaallluuueee::: TThhee bbeetttteerr tthhee pprroossppeeccttss ooff ccaarrrryyiinngg oouutt aa uusseeffuull aauuddiitt
of good quality, and the less the policy fi eld or subject has been
covered earlier by audits or other reviews, the greater the added
value might be. Adding value is about providing new knowledge and
perspectives.
• Important problems or problem areas: The greater the risk for conse-quences
in terms of economy, effi ciency, and effectiveness or public
trust, the more important the problems tend to be. A problem may
be judged important or material if knowledge about it would be

likely to infl uence the user of the performance audit report. Active
and problem-oriented monitoring makes it easier to identify areas
for audits.29
• RRRiiissskkksss ooorrr uuunnnccceeerrrtttaaaiiinnntttiiieeesss::: TThhee ssttrraatteeggiicc ppllaannnniinngg mmaayy bbee bbaasseedd oonn rriisskk
analysis, or – less theoretical – analysis of indications of existing or
potential problems. The stronger the public interest involved where
there is reason to suspect ineffi ciency, the greater the risks (the less
the knowledge), and the greater the uncertainty. The accumulation
of such indicators or factors linked to an entity or a government
program may represent an important signal to SAIs and should
induce them to plan audits whose range and scope will depend on
the indices detected. Factors that may indicate higher risk (or uncer-tainty)
could be the following:
– The fi nancial or budgetary amounts involved are substantial, or
there have been signifi cant changes in the amounts involved.
– Areas traditionally prone to risk (procurement, technology,
environment issues, health, etc, or areas of unacceptable risk) are
involved.
– New or urgent activities or changes in conditions (requirements,
demands) are involved.
– Management structures are complex, and there might be some
confusion about responsibilities.
– There is no reliable, independent, and updated information on
the effi ciency or the effectiveness of a government program.
Some SAIs may choose topics based on strategic choices rather than
selection criteria (for example, with regard to the type of perform-ance
audit, policy spheres, relationship with reforms within the public
sector etc). Sometimes these strategic choices may refl ect the consti-tutional
and legal conditions and the established traditions. The may
also refl ect ‘political realities’ (i.e. certain topics are not expected to be
subjected to auditing).
Consequently, strategic planning allows for different ambitions and
decisions. Linked to a SAI’s annual planning system, strategic plan-ning
may be a useful tool in setting priorities and selecting potential
audits to be executed. In another context, it may serve as a mechanism
to select future audit themes, as a basis for more detailed planning. It
may also serve as an instrument for strategic policy decisions on the
29 For further discussion on the selection of problems, see HHHaaannndddbbbooooookkk iiinnn PPPeeerrrfffooorrrmmmaaannnccceee AAAuuudddiiitttiiinnnggg RRRRVV
(Sweden), 1999, and Picking the winners, NAO (United Kingdom) 1997.

46
future direction of the audit. As an illustration, a SAI might outline the
following options for its future audit orientation.
Focus on compliance with laws and regulations to secure implementation
Focus on individual Focus on government
organizational units undertakings/programs
Focus on effectiveness to contribute to change and renewal
One possible strategic choice is to decide to contribute to the moderni-sation
of the government administration and focus on auditing gov-ernment
programs with material effectiveness problems. An alternative
choice might be to simply focus on auditing individual government
agencies and their compliance with administrative/economic regula-tions.
In a changing society, it is quite natural that public activities are
regularly reviewed to see whether they fulfi l the goals and solve the
problems for which they were created. As time proceeds, new de-mands
replace old. Since demands and conditions constantly change,
performance auditing will have to be prepared to monitor and follow
developments and trends, review priorities, and use new approaches
and methods. If a SAI is defi ned – or defi nes itself – as an instrument
for change, it is important that its priorities for performance auditing
refl ect the need for improvement in the public sector. For example, in
a situation with a large budget defi cit or old-fashioned management
style, performance auditing may provide contributions to savings,
better use of resources, or modernization of management. If, on the
other hand, problems concerning unemployment, environment, eq-uity,
transparency, services to the clients, etc. are in focus in the public
debate, performance auditing may prefer to give priority to such issues.
In other words, strategic planning may aim to do more than pro-duce
viable subjects for audits. Ideally it should integrate audit topics
– or audit themes – in an overall perspective.30 Some SAIs conduct
special studies to build up knowledge or skills – either within a single
area of government or in an area defi ned by an audit theme – to assist
the strategic planning process. 31
30 Auditing work on a particular theme has become more common in recent years. This is longer-term
work, which usually produces a number of interrelated audit reports.
31 For an example, see Strategy for Performance Audits in the Chemical Sector, Swedish SSttrraatteeggyy ffoorr PPeerrffoorrmmaannccee AAuuddiittss iinn tthhee CChheemmiiccaall SSeeccttoorr, NNaattiioonnaall OOfffifi ccee,,
2002.

The strategic planning exercise normally results in a coherent and
cogent audit program for the SAI. The program lists the audit areas
and provides a brief account of the provisional problems, questions,
and other arguments supporting each one of them. Ideally, the pro-gram
then serves as a basis for operational planning and resource
allocation.
Strategic planning is the basis for the selection of audit topics. Linked to
a SAI’s annual planning system, it may be a useful tool in setting priori-ties
and selecting audits. It may serve as a mechanism for selecting fu-ture
audit themes, and a basis for detailed planning. Finally, it may serve
as an instrument for strategic policy decisions on the future direction of
the audit. Planning might be carried out in the following steps: deter-mining
potential audit areas; establishing the selection criteria to be
used; and identifying the main sources of information for the potential
audits. The strategic planning exercise would normally result in a coher-ent
and cogent audit program for the SAI and serve as a basis for opera-tional
planning and resource allocation.
3.3 What does planning of individual performance audits involve?
According to the INTOSAI auditing standards (AS 3.1.1), the auditor
should plan the audit in a manner, which ensures that an audit of high
quality is carried out in an economic, effi cient and effective way and in
a timely manner. A well thought-out plan is in general indispensable
in performance auditing. Before starting the main study, it is conse-quently
important to defi ne the audit objectives, the scope, and the
methodology to achieve the objectives. This is often done in the form
of a pre-study.32
The purpose of a pre-study is to establish whether the conditions for
a main study exist and, if they do exist, to produce an audit proposal
with a work plan. Primarily, operational planning should be a tool for
directing the execution process. In addition, it provides background
knowledge and information needed to understand the entity, program,
or function. An appropriate audit work plan makes it easier, for in-stance,
to ensure that the performance audit coverage is comprehensive
and realistic. The pre-study should normally be carried out in a fairly
short period.
32 Some SAIs use the term ‘preliminary study’.

48
Three important steps
The most important steps in drawing up an audit proposal are the fol-lowing:
• Defi ning the specifi c issue to be studied and the audit objectives,
• Developing the scope and the design of the audit,
• Determining the quality assurance, the timetable, and the resources.
In practice, these steps cannot always be strictly separated and they
do not necessarily take place in the same order.
1. Defining the specific issue to be studied and the audit objectives
One initial step is the more precise defi nition of the topic or the prob-lems
to be audited. The motives and the objectives for the study must
be elaborated upon. This is a diffi cult and important step that involves
examining the subject matter in depth, by studying relevant literature,
documents and statistics, conducting interviews with major stakehold-ers
and experts etc, and analysing potential problem indications from
various viewpoints. It is important that the defi nitions are distinct.
Ambiguous or vague defi nitions must be avoided.33 Even minor
changes to the audit question or the problem to be studied may have a
major impact on the general scope of the audit.
In short, this step involves elaborating on the following two ques-tions:
What? What is the audit question or the problem to be studied?
Why? What are the audit objectives?
The wording of the basic audit question is an aspect in the examina-tion
process that is of great importance: it is a decisive factor for the
results of the audit. It can be thought of as the fundamental research
question into a government program that the auditors seek the answer
to. Consequently, it is important that it is based on rational and objec-tive
considerations. In general, a SAI must apply a holistic perspective
that best favours the public interest and the general mission for its
performance auditing.
Audit objectives relate to the reasons for conducting the audit and
should be established early in the execution process to assist in identi-fying
the matters to be audited and reported on. In determining objec-tives,
the audit team must take into account the roles and responsibili-ties
of the SAI and the expected net impact of the audit as defi ned in
the strategic audit plan. The audit objectives and scope are interrelated
33 For more information, see, e.g. Handbook on Performance Auditing, HHaannddbbooookk oonn PPeerrffoorrmmaannccee AAuuddiittiinngg, RRRRVV,, 11999999..

and should be considered together. It is good management practice to
discuss the scope of the audit scope with the audited entities at the ear-liest
opportunity.34 When defi ning the audit objectives, one criterion
might be to optimise the contribution made by the audit. A possible
tool in determining this contribution is to outline the expected conclu-sions.
If an audit takes place on request, the audit objectives might be
more or less determined or obvious (AS 3.1.3).
The detail in which the audit is to be planned is another decision to
be made. Careful advance planning will prevent problems in the way
the audit should be handled arising at a later stage. At the same time,
planning that is too detailed may sometimes inhibit innovative think-ing
and openness. Audits are carried out in a complex world, and it is
therefore rarely possible to devise a comprehensive audit design that
predicts the progress of a performance audit in every detail.
2. Defining the scope and the design of the audit
The next step in the design phase is the development of the scope and
design of the audit. As in fi nancial auditing, the audit approach for
performance auditing needs to be structured.35
Defining the scope and the specific questions or hypotheses to
examined
The scope defi nes the boundary of the audit. It addresses such things
as specifi c questions to be asked, the type of study to be conducted and
the character of the investigation.36 Further, it comprises the work of
collecting information and the analyses to be executed (AS 3.1.3-4). 37
34 In some cases it may also prove useful to explicitly clarify what is not going to be audited in the actual
study (what is not intended to be covered). This may contribute to reduce misconceptions or false
expectations among stakeholders.
35 In this step, it may sometimes be a good idea to study scientifi c work and theories concerning the
area to be audited or the questions to be answered.
36 See section 4.2 below. For more information, see Performance Audit Manual, CAG, Bangladesh, 2000
and Government Auditing Standards, GAO (US), 2002.
37 Sometimes a SAI may limit its performance audit to a ‘meta-evaluation’ (an evaluation of self-evalu-ations).
It must be understood, however, that such an approach is only feasible when the SAI auditor is
fully satisfi ed that the internal evaluation processes provide objective, timely, and comprehensive assess-ments
of the programs concerned.

50
The scope of an audit is determined by answering the following
questions:
What? What specifi c questions or hypotheses are to be examined?
What kind of study seems to be appropriate?38
Who? Who are the key players involved and the auditee(s)?
Where? Are there limitations in the number of locations to be
covered?
When? Are there limitations on the time frame to be covered?
Having defi ned the motives and the objectives for the study as well
as the general audit question or the problems to be considered, it is im-portant
to defi ne the specifi c questions to be answered or the hypoth-eses
to be examined (the plausible causes of the problem). In practice,
they will form the basis for the selection of data collection methods.
The auditor must also defi ne the character of the study: what kind of
study is needed to highlight the questions? (Some common methods in
performance auditing are presented in appendix 1.)
It must be stated, however, that conducting fi eldwork is a more of
a continuous learning process than a matter of simply collecting data.
And questions (or hypotheses) may have to be changed as the auditors
become more knowledgeable during the audit. However, during the
planning stage, the purpose is to systematically direct attention to what
the auditors need to know, and from where and how they can obtain
the information.
Auditability is an important requirement in the operational plan-ning
process. It defi nes whether a topic is suitable for a main study. As
objectives and scope vary from one audit to another, the audit team
needs to assess whether an audit can be carried out. An issue must be
both auditable and worth auditing in order to be included in the audit
scope. The auditor might, have to consider, for instance, whether there
are relevant approaches, methodologies, and criteria available and
whether the information or evidence required is likely to be available
and can be obtained effi ciently. Furthermore, reliable and objective
information should exist and there should be reasonable chances of
obtaining this information. Other aspects to be considered are compli-ance
with the audit mandate, resources, professional skills required,
and conditions in terms of timing. Personnel with relevant skills must
be available, and an audit should not be overruled by other studies al-ready
being made by other bodies (AS 2.1.23, 2.2.39, 3.0.3 and 3.1.1-4).
38 The study design (goal-attainment study, time-management study etc.). See Appendix 1.

Understanding the program
It is important to develop a sound understanding of the audited
program or the auditee’s business that is suffi cient to achieve the audit
objectives, facilitate the identifi cation of signifi cant audit issues and
fulfi l assigned audit responsibilities. This knowledge includes an un-derstanding
of:
• the character of the government program being audited (role and
function, activities and processes in general, development trends
etc);
• legislation and general programs and performance goals;
• organizational structure and accountability relationships,
• internal and external environment and the stakeholders;39
• external constraints affecting program delivery;
• earlier investigations in the fi eld; and
• management processes and resources.
The aim in the design phase is to develop a basic understanding of
the audited program. Obtaining the required knowledge is a continu-ous
and cumulative process of gathering and assessing information,
and relating the resultant knowledge to audit evidence at all stages of
the audit. It is important that auditors weigh the costs of obtaining
information and the additional value of the information to the audit.
Sources of information may include:
• enabling legislation and legislative speeches;
• ministerial statements, government submissions, and decisions;
• recent audit reports, reviews, evaluations, and inquiries;
• scientifi c studies and research (including that from other countries);
• strategic and corporate plans, mission statements, and annual re-ports;
• policy fi les and management committee and board minutes;
• organization charts, internal guidelines, and operating manuals;
• program evaluation and internal audit plans and reports;
• conference reports and minutes;
• viewpoints from experts in the fi eld;
• discussions with auditee management and key stakeholders;
• management information systems; and
• other relevant information systems as well as offi cial statistics.
39 Some SAIs may, if required, conduct stakeholder analyses at this stage in order to get a clear picture
of the actual situation from different perspectives.

52
Past reviews are often a useful source of information. They can
help avoid unnecessary work in examining areas that have been under
recent scrutiny and highlight defi ciencies that have not yet been rem-edied.
Discussions with senior program management to gain an overall
program perspective may also prove important. Other relevant sources
of information are:
• studies by industry and professional or special interest groups;
• inquiries or previous reviews by the legislature;
• information held by coordinating agencies or government commit-tees;
• work undertaken by other governments; and
• press coverage.
Defining the audit criteria
The audit criteria are intended to give direction to the assessment
(helping the auditor to answer questions such as ‘On what grounds is
it possible to assess actual behaviour?’ ‘What is required or expected?’
‘What results are to be achieved – and how?’ – by program?) Audit
criteria are standards used to determine whether a program meets or
exceeds expectations. 40
In fi nancial audits, transactions that are examined tend to be judged
by the auditor as being ‘correct’ or ‘incorrect,’ ‘legal’ or ‘illegal,’ etc.
Such criteria tend to be relatively closed and are usually prefi xed by,
for example, the legislation establishing the audited entity. (For more
information, see appendix 2.) For performance audits, however, the
choice of audit criteria is normally relatively open and formulated by
the auditor, and as mentioned earlier, criteria are often less important
in the problem-oriented approach. In the problem-oriented approach
it is more important to formulate testable (verifi able) hypotheses on
possible causes to the audit problem.41
Thus, in performance auditing, the general concepts of economy,
effi ciency, and effectiveness need to be interpreted in relation to the
subject matter, and the resulting criteria will vary from one audit to
another. In defi ning audit criteria, auditors must ensure that they are
relevant, reasonable, and attainable. Finally, every criterion is elaborated
in the form of questions. These questions are factual in character and
intended to describe or measure the practical situation to be audited.
40 Audit criteria should be reliable, objective, useful, complete and accepted.
41 For information on how to defi ne audit problems, and how to formulate and test hypotheses, see e.g.
Handbook in Performance Auditing: Theory and practice, The Swedish National HHaannddbbooookk iinn PPeerrffoorrmmaannccee AAuuddiittiinngg:: TThheeoorryy aanndd pprraaccttiicce, AAuuddiitt OOfffifi ccee,, 11999999..

Audit criteria can, for instance, be obtained from the following
sources:
• laws and regulations governing the operation of the audited entity;
• decisions made by the legislature or the executive;
• references to historical comparisons or comparisons with best prac-tice;
• professional standards, experiences, and values;
• key performance indicators set by the auditee or the government
• independent expert advice and know-how;
• new or established scientifi c knowledge and other reliable informa-tion,
• criteria used previously in similar audits or by other SAIs;
• organizations (inside or outside the country) carrying out similar
activities or having similar programs;
• performance standards or previous inquiries by the legislature; and
• general management and subject-matter literature.
The basis of the audit criteria may be considered from different
angles:
• depending on the case in point, the most authoritative sources will
either be offi cial standards (such as goals laid down in laws and
regulation, decisions and policies taken by the legislature or the
executive branch), or
• on the basis of the scientifi c grounds of the standards, greater
emphasis will be placed on specialist scientifi c literature and other
sources such as professional standards and best practices.
Sometimes audit criteria are easy to defi ne, for example when the
goals set by the legislature or the executive branches are clear, precise
and relevant. However, this is often not the case. The goals may be
vague, confl icting or non-existent. Under such conditions, the audi-tors
might have to reconstruct the criteria. One possibility is to apply a
‘theoretical’ approach, by allowing experts in the fi eld to answer ques-tions
such as: ‘what ought to be the ideal results under perfect con-ditions
according to rational thinking or best-known comparable prac-tice?’
Alternatively, to defi ne and obtain support for well-founded and
realistic criteria, it may prove helpful to apply an ‘empirical’ approach,
involving discussions with stakeholders and decision makers.42
42 It is sometimes advisable to avoid setting precise and detailed audit criteria in the design phase (since
the knowledge is limited). For more information, see appendix 2.

54
Methodological planning
Methodological planning involves many different activities. For in-stance,
it is important to distinguish between the audit program (the
type of investigation that is needed for the data collection) and the
data-collection techniques. Performance audits can draw upon a large
variety of data-gathering techniques that are commonly used in the
social sciences, such as surveys, interviews, observations, and studying
written documents.43 The aim is to adopt best practices, but practical
reasons such as availability of data may restrict the choice of methods,
i.e. the auditors may often have to settle for the second-best solution.
As a general rule, it is advisable to be fl exible and pragmatic in the
choice of methods.44 Practical considerations will also have to infl u-ence
the audit program. Sampling methods and surveys might allow
general conclusions to be drawn and case studies provide an opportu-nity
for in-depth studies.45
Even though the nature of performance audits requires careful
choices and combinations of methodologies for examining variables, it
is important to have an open mind during the execution process. The
selection of sources must not be rigid at this stage (AS 3.0.3, 3.1.1-4,
3.5.2-4, and 4.0.4). For more information, see Appendix 1.46
For performance audits in particular, the auditor will be concerned
about the validity and the reliability of methods to be used to collect
and analyse data:
• Validity: methods should measure what they are intended to measure.
• Reliability: fi ndings should remain consistent if studies are made
repeatedly in the same environment47
43 This implies that the performance auditors should take an active interest in, and be given time for, con-tinuous
improvement of their methodological skills, and follow methodological developments in various
research institutions, etc. This could, for instance, be done by arranging in-house seminars with external
professionals, and by reading articles and literature. In choosing methods for data collection, the auditors
should be guided by the purpose of the audit and the specifi c questions or hypotheses to be answered. For
an overview on using different techniques for collecting audit evidence and for analysis of information, see
Evidence-Gathering Techniques, OAG (Canada), 1994. For further discussion on methodology, techniques
and concepts in social research, see The Research Methods Knowledge Base, Trochim, 2002. See also
Evaluation, Rossi et al. 1999, and Case Study Research, Design and methods, Yin et al, 1994.
44 The reasons for being pragmatic in the choice of methods are elaborated in Designing Evaluations,
GAO (USA), 1991, chapter 2.
45 See Case Study Research, Design and Methods, Yin et al, 1994. Ways of selecting cases can be used in
Case Study Evaluation, GAO, (USA), 1990. For surveys, see Conducting Surveys, OAG (Canada), 1998.
Using statistical sampling, GAO UUssiinngg ssttaattiissttiiccaall ssaammpplliinngg, ((UUSSAA)),, 11999922 aanndd The Research Methods Knowledge Base, Trochim, 2002.
46 For designing performance audits, see HHHaaannndddbbbooooookkk iiinnn PPPeeerrrfffooorrrmmmaaannnccceee AAAuuudddiiitttiiinnnggg,, RRRRVV ((SSwweeddeenn)) aanndd
Designing vfm studies, a guide, NAO (United Kingdom), 1997.
47 For a further discussion, see e.g. The Research Method Knowledge Base, Trochim, 2002.

3. Determining quality assurance, timetable, and resources
The fi nal step is the determination of the quality assurance measures
to be taken in the audit, the timetable and the fi nancial budget.
Quality assurance
The INTOSAI Auditing Standards (((AAASSS 222...111...222777)) ssttaattee tthhaatt tthhee SSAAII sshhoouulldd
establish systems and procedures for quality assurance. Quality control
procedures should be designed to ensure that all audits are conducted
in accordance with relevant standards and policies.
A distinction might be made between ex ante (ongoing quality ar-rangements
while work is in progress for example through peer review,
‘co-readers’, use of expert panels, special committees and specialists or
experts in the fi eld concerned, and techniques such as issue analysis)
and ex post arrangements (such as independent reviews of published
reports to identify lessons learned and how reports are received by key
stakeholders and the benefi ts they drive from them). Some SAIs have
engaged individual scientists or scientifi c institutions to conduct ex
post assessments. Both ex ante, intermediate and ex post arrangements
have to be planned.48
The audit manager is responsible for the day-to-day management of
the audit, including detailed planning, execution of the audit, supervi-sion
of staff, reporting to SAI management and overseeing prepara-tion
of the audit report. Where more complex performance audits are
concerned, the SAI may consider appointing a steering committee to
guide the audit team and to monitor the progress of the audit. (See
appendix 4.)
Administrative planning
The audit team and a team leader have to be selected, and an activ-ity
plan is required. It is important to determine the timetable and
the resources needed. Relevant factors include the manner in which
the audit is organized, the expected costs (for both staff, on the basis
of predetermined rates, and equipment), and the expected comple-tion
time. The budget and timetable should be documented. Progress
against these targets should be monitored. The audit manager and SAI
management are responsible for ensuring that performance audit is
completed within budget and on time (AS 2.1.26-28, 2.2.36 and 3.1.4).
48 Building quality into the performance audit examinations requires many considerations. See for
instance Value for money handbook: a guide for building quality into VFM examinations, NAO, United
Kingdom, 2003.

56
Compliance with laws and regulations
When laws, regulations, and other compliance requirements pertaining
to the auditee are signifi cant to audit objectives, auditors should design
the audit to provide reasonable assurance of compliance with these
requirements. In performance audits, auditors should be alert to situ-ations
or transactions that could be indicative of illegal acts or abuse.
They may need to determine the extent to which such acts affect the
audit results. In complicated cases, specialists on these matters might
be required (AS 3.4.1-7).
The result of the pre-study – the main study proposal
The audit design phase results in a consistent audit proposal with a
work plan (AS 3.0.1-3). The quality of the work must be secured before
the decision to start a main study is taken.
Before starting the main study in performance auditing, it is important
to defi ne the audit objectives, the scope, and the methodology to
achieve the objectives. This is done in a form of a pre-study. The pur-pose
of a pre-study is to establish whether the conditions for a main
study exist and, if they exist, to produce an audit proposal with a work
plan. The most important steps in drawing up audit proposals are: de-fi
ning the specifi c issue to be studied and the audit objectives; develop-ing
the audit framework and the audit design; and determining quality
assurance, the audit team, the timetable, and the resources. The
auditee(s) should be informed of the objectives, scope, and time sched-ule
of the audit.
3.4 Summary
• The performance audit cycle involves strategic planning, preparation
work, main study and follow-up activities. Strategic planning is the
basis for the selection of audit topics or audits themes to be execut-ed.
It may also serve as an instrument for strategic decisions con-cerning
the direction of the audit. Planning might be carried out in
these steps: determining potential audit areas, establishing the se-lection
criteria, and identifying the main sources of information for
the potential audits.
• Before starting the main study in performance auditing, it is impor-tant
to defi ne the audit objectives, the scope, and the methodology
to achieve the objectives. This is done in form of a pre-study. The
purpose of a pre-study is to establish whether the conditions for a
main study exist and, if they exist, to produce an audit proposal with
a work plan.

• In planning performance audits, auditors should observe the follow-ing:
– Consider the signifi cance and the needs of potential users of the
audit report as well as other interested parties.
Obtain an understanding of the entity to be audited and of the prob-lems
to be scrutinized, including the context of the activities in ques-tion.
– Identify signifi cant fi ndings from previous audits and other investiga-tions
and reports that could affect the audit objectives (even ongo-ing
studies).
– Consider political objectives and the legal and regulatory environ-ments.
– Defi ne the topics or the problems to be studied, the entity to be
audited and the audit objectives, i.e. the expected effect of the
audit.
– Defi ne the basic audit questions – as well as the more specifi c ques-tions
– to be answered or the hypothesis to be tested.
– Establish the audit criteria. The audit criteria represent the normative
standards against which the audit evidence is judged. The criteria
will vary according to the specifi c audit subject and objectives, the
legislation governing the undertaking or the audited entity, the stat-ed
objectives, and the specifi c conditions that the SAI deems rele-vant
and important for the case.
– Determine the audit evidence that will answer the audit question:
the relevance, reliability and suffi ciency of any data available within
the audited entities should be evaluated. The possibility of collecting
the required evidence (data) should be tested.
– Identify potential sources of information that should be used in or-der
to verify hypotheses, gain better knowledge of the subject, or
obtain answers to audit questions, i.e., information that may be used
as evidence.
– Consider, if needed, help from experts (consultants, other auditors)
concerning how to secure quality in the audit. It is important to
evaluate the professional knowledge and skills required by the audit
team to carry out the audit.
– Provide suffi cient staff and other resources to perform the audit and
prepare a written plan. Select a suitable audit team. Decide upon a
budget for the resources needed to carry out the examination and
the timetable.
– Consider the possible conclusions and impacts of the examination.
The proposed outcome should be judged in terms of ‘usefulness’
and ‘feasibility’. The auditor should consider the views and interests
of the stakeholders.

Part 4: Field standards and guidance:
Conducting the performance audit
4.1 What characterizes the main study process?
The purpose of the main study is to implement the work plan, conduct
the audit and produce a high quality audit report.49A performance
audit does not consist of a series of clearly defi ned measures, opera-tions,
or sub-processes that are carried out separately and in sequence.
In practice, the processes evolve gradually through interaction with
one another, and are carried out simultaneously, allowing the methods
to develop in depth and become increasingly sophisticated (AS 2.2.39,
3.0.1 and 4.0.21).
Carrying out an audit may be seen as both an analytical and a
communicative process. In the analytical process, data are collected,
interpreted, and analysed. The communication process begins when
the audit is fi rst presented to the auditee, and continues as the audit
proceeds, as different fi ndings, arguments and perspectives are as-sessed,
and continues until the report has been fi nalized.50
An open and constructive dialogue is the ideal, but an audit may
provoke negative reactions. The auditor may face varying situations,
from openness and willingness to cooperate, to evasiveness and se-crecy.
It is therefore important that the auditors inform the auditee of
the audit’s objectives and methods. This does not mean that the audit
object should dictate conditions or in any other way control the execu-tion
process. Instead, it involves establishing a constructive process of
interaction. As a rule, the assistance of individuals from the auditee is
essential to an effective audit. An active dialogue during the audit with
49 The main study shall be carried out in compliance with the best practices with respect to techniques
and methodologies (AS 2.2.36-37, 1.0.14 and 1.0.46).
50 For more information, see for instance Performance Auditing at the Swedish National Audit Bureau,
RRV, (Sweden), 1994.

60
the auditee, experts and others makes it easier, for instance, to make
continuous checks of preliminary fi ndings (AS 2.2.25-26).
It is also important to conduct the audit with integrity. The work
plan must be followed (resources, time, and quality) and the audit
must be carried out in accordance with relevant decisions and stand-ards
(AS 2.2.39 and 3.0.2).
The purpose of the main study is to implement the work plan, conduct
the audit and produce an audit report. Performance audits must be car-ried
out in compliance with the best practices. Carrying out an audit
may be seen as both an analytical and a communicative process. An
open and constructive dialogue is the ideal, but integrity is also impor-tant.
The work plan must be followed and the relevant standards met.
4.2 What has to be considered in the data collection process?
Quality in data collection and documentation is vital, since perform-ance
audit is open to judgment. The Auditing Standards state that
‘Competent, relevant and reasonable evidence should be obtained to
support the auditor’s judgment and conclusion regarding the organi-zation,
program, activity or function under audit’ (AS 3.5.1). While
evidence in fi nancial audits tends towards being conclusive (yes/no
or right/wrong), this is seldom the case in performance audits. More
typically, performance audit evidence is persuasive (‘points towards the
conclusion that…’).51
The auditors must be creative, fl exible and careful in their search for
evidence. When working in areas where evidence is persuasive rather
than conclusive, it is sometimes useful to hold discussions in advance
with the experts in the fi eld the nature of the evidence to be obtained
and the way it will be analysed and interpreted by the auditor. This
approach reduces the risk of misunderstanding and may speed up the
process. It is also important that the auditors seek information from
different sources, since organizations, individuals in an organization,
experts, and interested parties have different perspectives and argu-ments
to put forward (AS 2.2.39, 3.4.5 and 4.0.24).
Data, information, and knowledge are, broadly speaking, similar,
linked concepts. Data is the primary tool. Data, which has been com-
51 For more information see Government Auditing Standards, GAO (USA), 2002.

piled, is transformed into information. Information, which is analysed
and understood, will become knowledge. Both qualitative and quan-titative
data may be collected for different purposes during an audit,
whether as part of the learning process, or in order to describe and
analyse an outcome or a problem.52
Based on general experience, it is important to distinguish between
the following components in the information gathering process:
• Questions formulated to be answered by the study.
• Study design, i.e. the type of study that is needed to answer the
questions set (time management, cost-benefi t, goal attainment
studies, etc.).
• Audit programs, i.e. the type of investigation that is needed for the
data-collection (such as sampling, case studies, secondary analysis,
inquiries, ‘before- and after analysis’, comparable evaluations, etc.).
• Data-collection techniques needed in order to answer the questions
set (study of documentation, meetings, questionnaires, interviews
etc.).
• Quantitative and qualitative analysis, applied to the data collected
(for deeper analysis of the information collected).
Data collection may be performed once or through ongoing meas-urements
(such as time series design, longitudinal analysis etc.). Infor-mation
may be gathered on the basis of physical evidence, documents
(including written statements), oral testimonies (interviews), or by
other means depending on the objectives of the audit. It will often be
necessary to collect both quantitative and qualitative data. The types
of data to be obtained should be explainable and justifi able in terms of
suffi ciency, validity, reliability, relevance, and reasonableness. Perform-ance
auditing may produce primary data (its own source material)
with the aid of questionnaires, surveys and direct observation. Howev-er,
a great deal of secondary data (material produced by others) is also
often used. The best available information should be gathered. The
auditors, however, must not be rigid in their requirements for exact-ness.
It might prove costly and unnecessary, i.e. a second-best solution
is often quite suffi cient and appropriate (AS 3.4.5 and 3.5.1-4).53
It is vital that the auditors adopt a critical approach and maintain an
objective distance to the information put forward. At the same time,
they must be receptive to views and arguments. The auditors must be
52 See HHHaaannndddbbbooooookkk ooonnn PPPeeerrrfffooorrrmmmaaannnccceee AAAuuudddiiitttiiinnnggg,, RRRRVV ((SSwweeddeenn)),, 11999988..
53 As the saying goes: ‘It is better to be vaguely right than exactly wrong’.

62
able to see things from different perspectives and maintain an open
and objective attitude to various views and arguments. If they are not
receptive, the auditors may miss the best arguments. This also under-scores
the importance of making rational assessments, in that the audi-tors
discount their own personal preferences and those of others. It is
therefore important for the auditor’s involvement to be expressed in a
process of refl ection and objective analysis rather than in a conviction
that certain standpoints are correct (AS 2.2.40, 3.5.1 and 4.0.24).
Where computer-processed data are signifi cant to the fi ndings of
the audit, it may be wise to take extra precautions in order to obtain
suffi cient, competent, and relevant evidence that the data are valid and
reliable. Additionally, when assessment of the reliability of an informa-tion
system is the primary objective of the audit, the auditors should
review the system’s general and application controls. Adding to this,
during a main study the auditors may obtain sensitive information
(such as opinions on management or politics). Consequently, they
should guarantee anonymity and not divulge people’s opinions (AS
2.2.37, 2.2.46 and 3.3.4).
The results of the fi eldwork and analysis, along with the audit plan-ning
paperwork, need to be documented, fi led, and cross-referenced to
permit audit mangers to review the work done and validate the conclu-sions
reached. A record of the work should be retained in the form of
working papers. Suffi cient, competent, and relevant evidence should be
obtained to afford a reasonable basis for the fi ndings and the conclu-sions
(AS 3.2.1 and 3.5.5-7).
Quality in data collection and documentation is vital. The auditors have
to be creative, fl exible and careful in their search for suffi cient evidence.
It is important to maintain an objective distance from information put
forward, but the auditors must also be receptive to views and argu-ments
and seek information from different sources and stakeholders.
4.3 What characterizes the audit evidence and the audit findings?
Evidence may be categorized as physical, documentary, testimonial,
or analytical. A direct inspection or observation of people, property,
or events obtains physical evidence. Documentary evidence consists
of information such as letters, contracts, accounting records, invoices,
and management information on performance. Testimonial evidence

is obtained through interviews or questionnaires. Analytical evidence
includes computation, comparisons, separation of information into
components, and rational arguments. Evidence should be suffi cient,
competent, and relevant. It is suffi cient if there is enough of it to sup-port
the audit fi nding. Evidence used to support a fi nding is relevant if
it has a logical, sensible relationship to that fi nding. It is competent if it
is consistent with facts (AS 3.4.5 and 3.5.1).
Audit fi ndings are the specifi c evidence gathered by the auditor
to satisfy the audit objectives, in order to be able to answer the au-dit
questions and verify the stated hypothesis, etc. Conclusions are
statements deduced by the auditor from those fi ndings, and recom-mendations
are courses of action suggested by the auditor relating to
the audit objectives. Audit fi ndings contain the following elements:
cccrrriiittteeerrriiiaaa ((‘‘wwhhaatt sshhoouulldd bbee’’)),, cccooonnndddiiitttiiiooonnn ((‘‘wwhhaatt iiss’’)),, aanndd eeeffffffeeecccttt ((‘‘wwhhaatt aarree
the consequences’ – observed as well as ‘reasonable and logical future
impact’), plus cccaaauuussseee ((‘‘wwhhyy iiss tthheerree aa ddeevviiaattiioonn ffrroomm nnoorrmmss oorr ccrriitteerriiaa’’)),,
when problems are found.54 However, all four elements are not always
required in an audit; the element ‘criteria’ is for instance not always
specifi cally addressed in the problem-oriented approach.55 The process
of analyzing evidence, developing fi ndings, and producing recommen-dations
to resolve identifi ed areas of poor practice is summarized in
the following diagram from ASOSAI guidelines.
Audit criteria (what should be)
Audit evidence (what is)
Audit finding (`what is´ compared with `what should be´)
Determine the causes and effects of the finding
Develop audit conclusions and recommendations
Estimate likely impacts of the recommendations wherever possible
54 For more information, see Government Auditing Standards, GAO (USA), 2002.
55 See section 1.8.

64
Once an audit fi nding has been identifi ed, two complementary
forms of assessment take place: the assessment of the signifi cance of
the fi ndings and the determination of the causes (of increased per-formance
or of the lack of performance where performance is below
that expected).
The auditors may also try to assess the consequences of the fi nding.
In many cases, the effect of a fi nding may be quantifi able. For example,
the cost of expensive processes, expensive inputs, or unproductive fa-cilities
can be estimated. Additionally, the effect of ineffi cient processes,
such as idle resources or poor management, may become apparent in
terms of time delays or wasted physical resources. Qualitative effects
as evidenced in a lack of control, poor decisions, or lack of concern for
service may also be signifi cant. The effect should demonstrate the need
for corrective action. The effect can also have occurred in the past,
be occurring now, or possibly occur in the future. To make a fi nding
stand, be certain that, if the effect occurred in the past, the situation
has not already been remedied to prevent it from recurring. If the ef-fects
are not readily identifi ed, the performance auditor may need to
assert potential effects.56
Evidence may be categorized as physical, documentary, testimonial, or
analytical. Evidence should be suffi cient, competent, and relevant. Audit
fi ndings provide answers to the audit questions. Conclusions are state-ments
deduced from the fi ndings. Comparing audit observations (con-ditions)
with audit criteria identifi es audit fi ndings. Once an audit fi nd-ing
has been identifi ed, two complementary forms of assessment take
place: the assessment of the signifi cance of the fi ndings and the deter-mination
of the causes.
4.4 How should a changeable and conflicting environment
be handled?
A performance audit may run for a long time, and there may be chang-es
in knowledge and reality from the point in time when it started. In
performance auditing it is often diffi cult to make a choice between the
directions set out in the work plan and the description of the audit’s
structure on the one hand, and the interest in studying questions that
56 For more information, see Performance Auditing Guidelines, ASOSAI, 2000.

arise at a later date on the other. To avoid getting caught up in details
and a fl ood of data, detailed assessments of the need for information
must be made both before and during the audit. Based on experi-ence,
this makes it easier to eliminate extraneous detail and irrelevant
approaches, and to sort or structure the information gathered. Again,
however, the auditors must not be rigid and avoid all unplanned data
gathering.
AAASSS 222...222...222666 ssttaatteess ‘‘IInn ccoonnttrraasstt ttoo pprriivvaattee sseeccttoorr aauuddiitt,, wwhheerree tthhee aauuddii--
tor’s agreed task is specifi ed in an engagement letter, the audited entity
is not in a client relationship with the SAI. The SAI has to discharge its
mandate freely and impartially, taking management views into consid-eration
in forming audit opinions, conclusions and recommendations,
but owing no responsibility to the management of the audited entity
for the scope or nature of the audits undertaken.’ However, auditors
should always seek to create good relationships with the auditees and
other interested parties.
To avoid unnecessary confl icts, auditors should listen and learn, and
try to understand the particular nature of the activity under audit. To
do so they must be able to look at an activity from different perspec-tives,
and maintain an open and objective attitude to views and objec-tions
put forward. Regular meetings and discussions with the auditee
are often a valuable part of the audit. If confl icts occur, efforts should
be made to air contradictory opinions with a view to making the fi nal
picture as true and fair as possible. The auditors should attempt to
establish open co-operation and interaction and an atmosphere of
confi dence with the auditee at the earliest opportunity. However, the
very nature of auditing and the importance of its independence means
that a clear limit must be set up to prevent the individual performance
auditor from becoming involved in the practical work of implement-ing
changes at the auditee (AS 2.2.25, 2.2.29, 4.0.24).
There are limits to how far a performance audit can go to corrobo-rate
statements and verify fi ndings. It is, generally speaking, impos-sible
to avoid criticism, even when the most sophisticated scientifi c
methods are used. The advantages of verifying an issue must, therefore,
be weighed against time constraints and resource consumption. At
the same time, the basic data must be of good quality (AS 3.1.1, 3.5.1,
4.0.23 and 2.1.26).

66
A well-designed work plan may help the auditors avoid getting caught
up in details and a fl ood of data. To avoid unnecessary confl icts, audi-tors
should try to understand the particular nature of the activity under
audit. If confl icts occur, efforts should be made to air contradictory opin-ions
with a view to making the fi nal picture as true and fair as possi-ble.
4.5 What is important when analysing data and drawing
conclusions?
Most audits involve some type of analysis in order to understand or
explain what has been observed. A wide range of models or methods
of analysis is used (AS 4.0.21). This could be done in the form of more
detailed statistical analysis, discussions on the fi ndings within the audit
team, studies of documentation and working papers etc. The analysis
might sometimes also require comparisons of fi ndings between for
instance subjects that work well and those that work less well; one or
more subjects and an overview; and the audited area and a similar
audit area in another country.
The fi nal stage in the analysis of data involves combining the results
from different types of sources. There is no general method for doing
this, but it is of central importance that the auditor works systemati-cally
and carefully in interpreting the data and arguments collected.
This phase involves weighing up arguments and assertions, consulting
experts, and making comparisons and analyses. As the work continues,
the draft report gradually takes shape.57 The notes and observations
are put into structured order, and as internal and external discussions
progress, text is drafted, assessed and rewritten; details are checked and
conclusions are discussed. There is a need for exchanges of informa-tion
based on discussion papers to discuss major issues that have
emerged during the course of the audit. These meetings may serve to
confi rm facts with the audited entities and to promote the develop-ment
of audit fi ndings and recommendations.
If possible, all the main arguments that can be envisioned should be
covered, so that no entirely new and possibly decisive arguments –
57 If the analyses are based on scientifi c knowledge and well-established theories, they will probably be
more solid and interesting. It is also often easier to interpret observations in the light of a well-known
theory. (‘There is nothing so practical as a good theory. ‘)

or facts – may be introduced at the fi nal stage of the audit. In a prop-erly
conducted performance audit, the arguments put forward are
balanced against the best possible counter arguments, and the various
contrasting views are weighed against each other. Experienced col-leagues
and contracted external advisors etc. can assist in this process.
The conclusions should be based on objectives, rationality and project-specifi
c criteria (AS 2.2.36, 2.2.39-40, 3.2.4, 3.5.2, 3.5.7 and 4.0.24).
Recommendations, if provided, should be argued in a logical,
knowledge-based and rational fashion. The cause of a fi nding forms
the basis for the recommendation. The cause may be outside the
control of the entity under audit, in which case the recommendation
should direct attention outside the auditee. It is important to ensure
that recommendations are practicable, add value and address the
objectives of the audit. In some cases it is also important to present the
arguments for and against various alternative proposals. By following
the underlying arguments, the reader will be better able to understand
the fi nal recommendations (AS 1.05 and 4.0.23-27).
Before publishing a performance audit report, the auditee(s) in-volved
should always be given the opportunity to examine its content.
The draft report provides the fi rst opportunity for the auditee to see
the full context of audit fi ndings, conclusions and recommendations in
written form. Where responses provide new information, the auditor
should assess this and be willing to modify the draft report, provided
the usual standards of evidence are met. Oral and written responses,
should, as far as possible, be documented. All disagreements must be
analyzed. The fi nal report must be balanced and fair (AS 3.2.4 and
4.0.23-24).
The fi nal stage in the analysis of data consists in combining results from
different types of sources. There is no general method for doing this. In
a properly conducted performance audit, the arguments put forward
are balanced against the best possible counter arguments, and the
various contrasting views are weighed against each other. The conclu-sions
should be based on objectives, rationality and project-specifi c
standards and criteria. Before publishing a performance audit report,
the auditee(s) should always be given the opportunity to examine its
factual content. The recommendations, if provided, have to be argued in
a logical, knowledge-based, and rational fashion. They should be di-rected
to remedy root causes of problems

68
4.6 Summary
In brief, when conducting an audit, the following should be consid-ered:
• Execute the work plan with integrity and care in a timely manner,
and in accordance with international and national standards for
performance auditing. Planning should continue throughout the
audit. Activities should be reviewed and modifi ed as the execution
process evolves.
• The project should be properly introduced to the auditee. An active,
open and constructive dialogue should be maintained with the au-ditee
and other interested parties during the audit. The auditee (or
the main executive entities involved in the undertaking to be scruti-nized)
should be involved in the process.
• Implement the audit scope with care. Quality in data collection,
analysis, and documentation is vital.
• Gather the best possible or most suitable information – facts as well
as opinions, arguments and refl ections – from different sources and
seek requisite knowledge and expertise. See that the work is charac-terized
by objectivity, impartiality, and sensitivity. Critically evaluate
information obtained and arguments put forward. All relevant facts
and arguments must be collected.
• Protect the integrity of persons providing information, ensure that
working papers are not disseminated incorrectly and, in all other
ways, observe high ethical standards.
• Recommendations, if provided, should be directed to remedy root
causes of problems. The results of the fi eldwork need to be docu-mented,
fi led, and cross-referenced. Evidence should be suffi cient,
competent, and relevant.
• Undertake adequate quantitative and qualitative analyses. Discuss
the analyses with senior auditors, stakeholders and experts in the
fi eld. Make analyses and assessments of observations on the basis
of political intentions, rational considerations, and criteria specifi c to
the audit. The fi ndings should form the basis for recommendations.
• Ensure that the factual basis of descriptions, analyses and recom-mendations
is accurate and that they are fair and well founded,
balanced and correctly communicated to the auditee. The auditor
should ensure that the recommendations, if provided, address the
objectives of the audit.

Part 5: Reporting standards and guidance:
Presenting the audit result
5.1 What does the need to focus on the final report involve?
According to the Auditing Standards, auditors should prepare writ-ten
audit reports (AS 4.0.7). Written reports should communicate
the results of audits to all levels of government, make the results less
susceptible to misunderstanding, make the results available for public
inspections, and facilitate follow-up to determine whether corrective
actions have been taken. The performance audit report is the product
on which the government, the legislature, and the public judge the SAI
performance audit function.
Most SAIs with long experience of performance auditing publish
individual reports, i.e. each performance audit is published separately.
Other SAIs which are not required to execute such audit projects or are
restricted from publishing all of their performance audit fi ndings, may
publish their observations and conclusions in summarized form in
their annual reports. The following guidelines are mainly applicable to
SAIs which are not restricted in their reporting.
Given the amount of reporting required during an audit, the
reporting process may be facilitated by the use of a continuous report-writing
process. This process may start at the beginning of the audit
with an outline that develops into discussion papers, which are then
brought together in the proposed report and further refi ned in the
fi nal audit report (AS 3.2.4).
The auditors should prepare written audit reports. Given the amount of
reporting required during an audit, the reporting process may be facili-tated
by the use of a continuous report-writing process.

70
5.2 What is required to make the reports reliable?
The audit report should be reliable and contain objectives, scope,
methodology and sources used, as well as audit fi ndings, conclusions
and recommendations.58 Any limitations on the scope of the work
and the reasons for this should be described. It should be easy for the
reader to understand the purposes of the audit and to properly inter-pret
the results. The report should be complete, accurate, objective,
convincing, and as clear and concise as the subject-matter permits (AS
4.0.8 and 4.0.24).
Being complete ccoommpplleettee rreeqquuiirreess,, ffoorr iinnssttaannccee,, tthhaatt tthhee rreeppoorrtt ccoonnttaaiinnss aallll
information and arguments needed to satisfy the audit objectives,
promote an adequate and correct understanding of matters and condi-tions
reported, and meet the report content requirements.59 It is vital
that the starting points of the audit and the methods used, as well
as important source material and conclusions, are described in the
fi nal audit report. The relationship between audit objectives, criteria,
fi ndings and conclusions needs to be verifi able, complete and clearly
expressed. If recommendations are to be provided, there needs to be a
clear link between the analysis or conclusions and the recommenda-tions.
Auditors should, within the audit objectives, report all signifi -
cant instances of non-compliance and signifi cant instances of abuse
that were found during or in connection with the audit (AS 4.0.87-8,
4.0.22).
AAAccccccuuurrraaacccyyy rreeqquuiirreess tthhaatt tthhee eevviiddeennccee pprreesseenntteedd sshhoouulldd bbee ttrruuee aanndd
comprehensive and that all fi ndings are correctly portrayed. The
need for accuracy is based on the need to assure readers that what is
reported is credible and reliable. One inaccuracy in a report can cast
doubt on the validity of an entire report and can divert attention from
the substance of the report. In addition, inaccurate reports can damage
the credibility of the SAI. In other words, a high standard of accuracy
requires an effective system of quality assurance. Reported evidence
should demonstrate the correctness and reasonableness of the mat-ters
reported. Correct portrayal means accurately describing the audit
58 The report could, for instance, include comparisons with audit criteria, and contain an analysis of
differences between what is observed and the audit criteria, including the causes and effects of the dif-ferences.
59 Certain information may be prohibited from general disclosure by law or regulation. Such informa-tion
may be provided on a ‘need-to-know basis’ only to persons authorized by law to receive it. How-ever,
it may be possible to include confi dential or sensitive material in a separate, unpublished report
(AS 4.0.8).

scope and methodology, and presenting fi ndings and conclusions in a
manner consistent with the scope of audit work (AS 4.0.23-24).
OOObbbjjjeeeccctttiiivvviiitttyyy rreeqquuiirreess,, aass mmeennttiioonneedd aabboovvee,, tthhaatt tthhee pprreesseennttaattiioonn ooff
the entire report be balanced in content and tone. A report’s credibil-ity
is signifi cantly enhanced when it presents evidence in an unbiased
manner. The report should be fair and not misleading, and should
place the audit results in perspective. This means presenting the audit
results impartially and guarding against the tendency to exaggerate or
overemphasize defi cient performance. Interpretations should be based
on insight and understanding of facts and conditions. One-sided pres-entations
should be avoided. Even though auditing by its very nature
has its focus on shortcomings, it is an advantage if the performance
audit reports can make room for both positive and negative fi ndings
and assessments (AS 4.0.7 and 4.0.23).
Being cccooonnnvvviiinnnccciiinnnggg rreeqquuiirreess tthhaatt tthhee aauuddiitt rreessuullttss sshhoouulldd bbee rreessppoonnssiivvee
to the audit objectives, the fi ndings presented persuasively, and the
conclusions and recommendations follow logically or analytically from
the facts and arguments presented. Facts should be presented sepa-rately
from opinions. The language used should not be tendentious or
suggestive, and the information presented suffi cient to convince the
readers to recognize the validity of the fi ndings, the reasonableness of
the conclusions, and the benefi t of implementing the recommenda-tions.
Different opinions and arguments should be represented (AS
4.0.7 and 4.0.24).
CCCllleeeaaarrr rreeqquuiirreess tthhaatt tthhee rreeppoorrtt bbee eeaassyy ttoo rreeaadd aanndd uunnddeerrssttaanndd ((aass
clear as the subject-matter permits). Technical terms and unfamiliar
abbreviations must be defi ned. Logical organization of material, and
accuracy in stating facts and in drawing conclusions, are essential to
clarity and understanding. Although fi ndings should be presented
clearly, the auditors must keep in mind that one of their objectives is
to be persuasive, and this can best be done by avoiding language that
generates defensiveness and opposition (AS 4.0.7-8).
Being cccooonnnccciiissseee rreeqquuiirreess tthhaatt tthhee rreeppoorrtt bbee nnoo lloonnggeerr tthhaann nneeeeddeedd ttoo
convey and support the message. Although scope may exist for consid-erable
judgment in determining the content of reports, those that are
complete, but still concise, are likely to achieve greater results. It must
be stated, however, that advanced studies often require longer reports.
One must also have in mind that the performance audit reports are not
only written for those who have special knowledge; they are also writ-ten
for those who need more information to understand the subjects.

72
A more comprehensive report might provide the reader with a better
understanding of the basis for the conclusions drawn, and thus add
value and creditability to the audit report. In a wider sense, compre-hensive
reports may strengthen a SAI’s capacity to serve the citizens’
interest in openness and transparency (AS 4.0.4).
The audit report should be reliable. The report should be informative
and, if provided, have logical and clear recommendations that are
linked to the audit objectives and the fi ndings. The auditors should re-port
the audit objectives, scope, methodology and sources used, as well
as audit fi ndings, conclusions, and recommendations. It should be easy
to understand the purposes of the audit and interpret the results. The
report should be complete, accurate, objective, convincing and as clear
and concise as possible.
5.3 What characterises a good and usable performance audit report?
Good performance audit reports should add value to the stakeholders
and meet the objectives set. They should provide accessible, concise,
and up-to-date information, which the government, parliament,
auditee, and other stakeholders can use to improve the economy, ef-fi
ciency, and effectiveness of the public sector: i.e. the report should
contribute to better knowledge and adequate improvements.60 Good
performance audit reports should be reader-based and well structured,
and the language should not be ambiguous. They should present fi nd-ings
objectively and fairly (AS 4.0.7). This requires that:
• there are separate presentations of fi ndings and conclusions;
• facts are presented and interpreted in neutral terms;
• different perspectives and viewpoints are represented;
• all relevant fi ndings, arguments, and evidence are included; and
• reports are constructive, and positive conclusions are presented.61
60 The form and content of all audit reports are founded on the following general principles: a suitable
title and a properly signed report, a clear presentation of objectives and scope, completeness, addressee
and identifi cation of subject matter (AS 4.0.7-8).
61 One possible structure of performance reports is the following: (1) Executive summary, (2) Introduc-tion
and audit design (including background, motives for the study, objectives, scope, methods), (3)
Description of the audit object, (4) Findings and analyses, (5) Conclusion and assessments, and (6)
Recommendation and Appendix.

In a wider sense – and to sum up – the quality of a performance
audit may be assessed by means of specifi c criteria, including those
considered below:
Materiality, relevance and objectivity
The topics dealt with should be material. The information given
should be relevant to the topic; the audit question or the problem
studied. Objectivity can be defi ned as ‘impartiality, balance and
neutrality’. When making decisions about scope, audit evidence,
signifi cance of observations, and conclusions, the auditor must have
an unbiased point of view and an objective state of mind. The audit
design should ensure that the selection of facts to be investigated and
presented in the report is balanced and unprejudiced. The fi ndings
should be infl uenced by evidence obtained and assembled in accord-ance
with relevant audit standards. Facts must not be suppressed, and
the auditor must not exaggerate minor shortcomings. Explanations
– especially from the auditee(s) – must always be sought and critically
evaluated (AS 1.0.9, 2.2.40, 3.5.4, 4.0.7, and 4.0.24-26).
Reliability, validity and consistency
Users should be able to trust the reliability and validity of reported
results. The data collection methods should be valid and reliable. The
audit design should be such that conclusions arise from the fi ndings
and the analysis, based on verifi ed facts and other information from
various sources. All the documents in the process must be well bal-anced
in their perspectives and judgments (AS 2.2.36, 2.2.39, 3.2.3,
3.4.5, 3.5.2, 4.0.8, and 4.0.22-25).
Transparency, usability and timeliness
A SAI must not be forced to withhold fi ndings and should, within its
legal mandate, be free to decide what to publish and how. The report
should provide accessible, concise and up-to-date information, which
the government, parliament, and government entities can use to im-prove
the way they function, i.e. the information provided should add
value. The audit questions should be answered. The points on which
the SAI expects action to be taken, and by whom, should be clearly
stated. Being timely requires that the report should be issued on time
in order to make the information available for timely use by manage-ment,
government, legislative offi cials and other interested parties
(AS 2.2.10-11, 3.1.1, 4.0.4-5, 4.0.7-8 and 4.0.21-22).

74
Good performance audit reports add value to the stakeholders and
meet the objectives set. They contribute to better knowledge and high-light
improvements needed. They are reader-based and well structured,
and the language is not ambiguous. Findings are presented objectively
and fairly. There are separate presentations of fi ndings and conclusions,
and facts are presented and interpreted in neutral terms. Different per-spectives
and viewpoints are represented, all relevant fi ndings, argu-ments
and evidences are included, and the reports are constructive; i.e.
positive conclusions are presented.
5.4 How should the performance audit reports be distributed?
Comprehensive reports and wide distribution of every report are keys
to the credibility of the audit function. In accordance with its funda-mental
role, each SAI must decide on how to best serve its own and the
public interest in distributing the audit reports, both in general and
for each report. If possible, all relevant audit fi ndings should be made
public (in individual performance reports or in the annual report
from the SAI). It is an advantage if the reports are available for public
discussion and criticism (AS 2.2.11).
The report should, if possible, be distributed to the auditee, the
government, legislative offi cials, the media and other interested parties.
Appropriate offi cials who may be included in the distribution include
those designated by law or regulation to receive such reports, those
responsible for acting on the fi ndings and recommendations, those of
other levels of government who have provided assistance to the auditee
and legislators (AS 4.0.8).
Publishing audit reports may cause misunderstandings. The me-dia
may misinterpret and exaggerate fi ndings, and as a consequence
frustrate the purpose of the audit. It is therefore – based on experience
– recommended that one provides the media with adequate and well-balanced
information backed by factual evidence, for instance in the
form of press releases.
Comprehensive reports and wide distribution of every report are keys
to the credibility of the audit function. If possible, each performance
audit should be published in a separate report.

5.5 What purposes do follow-up processes serve?
A follow-up process will facilitate the effective implementation of
report recommendations and provide feedback to the SAI, the
legislature and the government on performance audit effectiveness.
In following up the report, the auditor should maintain objectivity and
independence and thus focus on whether identifi ed weaknesses have
been corrected rather than on whether specifi c recommendations have
been implemented or not.62 The priority of follow-up tasks should be
considered in the context of the overall audit strategy as determined
by the strategic planning process (AS 4.0.26).
Following up on SAI recommendations may serve four main
purposes:
• increasing the effectiveness of audit reports–the prime reason for
following up audit reports is to increase the probability that recom-mendations
will be implemented;
• assisting the government and the legislature – following up may be
valuable in guiding the actions of the legislature;
• evaluation of SAI performance – following up activity provides a
basis for assessing and evaluating SAI performance; and
• creating incentives for learning and development – following up
activities may contribute to better knowledge and improved practice.
When a performance audit is completed, there are various oppor-tunities
for obtaining information on how it has been received, for
instance by observing reactions from audited bodies, parliament and
in the media. Internal and external conferences can be arranged to help
summarize experience and promote learning. Internal audit reviews
and evaluations may also be useful. In addition, external critics (sci-entists,
experts, and others) could be asked to scrutinize performance
audit reports or to give their opinions on the quality of the work
(AS 2.1.26-30).
Results from the follow-up of audit recommendations should be
recorded. Defi ciencies and improvements identifi ed in the follow-up
of audits should, if needed, be reported to the government or the
legislature.
62 It is recognized that isolating the impact of an audit report in the context of other signifi cant changes
is diffi cult. The key factor remains whether the audit recommendations have been carried out, and this
may often be the only measurable indicator of impact.

76
A follow-up process will facilitate the effective implementation of report
recommendations and provide feedback to the SAI. There different
ways of obtaining such information: internal reviews and evaluations,
conferences and seminars, special follow-up audits etc.
5.6 Summary
In brief, some reporting principles to be considered are the following.
• The results should be documented and the reporting should be
timely.
• The audit report should be well communicated to the auditee and
provide well-founded, objective and complete information, analyses
and assessments that add value for decision-makers and other
stakeholders.
• The report should be objective, well written, well structured, com-prehensive,
reliable, and contain relevant and usable conclusions.
• The report should be published and followed up in an objective
manner.

REFERENCES AND BIBLIOGRAPHY
International Organization of Supreme Audit Institutions (INTOSAI)
The Lima Declaration of Guidelines on Auditing Precepts, 1977.
Journal of Government Audit.
Auditing Standards Committee
Code of Ethics and Auditing Standards, 2001.
Sub-Committee on SAIs Independence
Independence of SAIs project, INTOSAI, 2001.
Asian organization of Supreme Audit Institutions, ASOSAI
Performance Auditing Guidelines (ASOSAI), 2000.
Working Group on Program Evaluation
Draft Summary Report, 1995.
Working Group on Environmental Auditing
Environmental Auditing in Europe 2000 – results of the EUROSAI and
INTOSAI questionnaires, 2000.
Guidance on conducting audits and activities with an environmental
perspective, 2001.
Environmental Audit and Regularity Auditing, 2002.
Working Group on IT Audit
Introducing Performance Audit of the use of EDP, 1997.
Auditing IT Service Management, Parts I-III, 2001.
Mandates of Supreme Audit Institutions, 2003.
IntoIT – The INTOSAI IT Journal (Vol. 1-17).

78
Working Group on Privatization
Guidelines in Best Practice for the Audit of Privatizations, 1998.
Guidelines in Best Practice for the Audit of Economic Regulation, 2001.
Guidelines in Best Practice for the Audit of Public/Private Finance and
Concessions, 2001.
Supreme Audit Institutions
National Audit Office of Australia
Performance Auditing With Vision And Purpose: The Way Things Ought
To Be, 1993.
Office of the Comptroller and Auditor General of Bangladesh
Performance Audit Manual, 2000.
Value-for-Money Audit in Bangladesh: Efforts and Challenges, 2002.
Office of the Auditor General of Canada
Auditing the Management of People: An approach to the Identifi cation
and selection of Audit Issues, 1991.
Evidence-Gathering Techniques, 1994.
Auditing Effi ciency, 1995.
Conducting Surveys, 1998.
VFM Audit Manual, 2000.
National Audit Office of Denmark
Planning and choice of methods for performance audit, 2001.
Measuring results of performance audit, 2002.
The State Audit Office of the Estonia
Measuring Performance Audit Effectiveness: The Case of Estonia, 2002.
The Ghana Audit Service
Performance Audit Guidelines, 2003.
Offi ce of Auditor General of Guyana
Developing Training Programs for the Introduction of VFM Audit in
OAG/Guyana, 2000.

State Comptroller’s Office of Israel
Studies in State Audit, State Comptroller’s Offi ce, 1995.
Netherlands Court of Auditors
Manual on Performance Audits, 1996.
Auditing of Policy Results Manual, 1997.
Office the Auditor General of Norway
Guidelines for Performance Auditing, Preface, 1998.
The Philippine Commission on Audit
Development of a Self-Study Guide for Philippine VFM Auditing, 2001.
National Audit Office of Sweden
System-Oriented Effectiveness Audit, 1985.
Performance Auditing at the Swedish National Audit Bureau, 1993.
Handbook in Performance Auditing. Theory and practice, 1999.
Quality Control Model for Performance Audit, 2003.
State Audit Office of Thailand
Strategy to Improve Planning for Performance Audits, 2000.
National Audit Office of the United Kingdom (NAO)
Collecting, Analysis and Presenting Data, 1996.
Picking the winners: A guide to vfm selection, 1997.
Designing vfm studies: A guide, 1997.
Benchmarking: How benchmarking can help in vfm examinations, 1997.
Focus groups: How to apply the techniques to vfm work, 1997.
Value for money: Handbook, 1997.
Performance measurement: What to look for in vfm studies, 1998.
Presenting data in reports – a guide. The Public Face of the NAO, 1998.
Getting beneath the surface: A Handbook in Using qualitative evidence in
value for money examinations, 1999.
Taking a Survey, 1999.
State Audit in the European Union, 2001.
National Audit Offi ce Sampling Guide, 2001.
A Code of Principles for the VFM Profession.

80
Writing smart recommendations.
Are we being served? A value for money guide.
Value for money handbook: a guide for building quality into VFM exami-nations,
2003.
United States General Accounting Office (GAO)
Causal Analysis: A Method to Identify and Test Cause Effect Relation-ships
in Program Evaluation, 1982.
Content Analysis: A Methodology for Structuring and Analysing Written
Material, 1982.
Case Study Evaluation, GAO/PEMD-91-10.1.9, 1990.
Prospective Evaluation Methods: The Prospective Evaluation Synthesis,
1990.
Designing Evaluations, GAO/PEMD-10.1.4.1991.
Using Structured Interviews Techniques, GAO/PEMD-10.1.5, 1991.
How to Get Action on Audit Recommendations, OP-9.2.1, 1991.
Quantitative Data Analysis, GAO/PEMD-10.1.11, 1992.
Using Statistical Sampling, GAO/PEMD-10.1.6, 1992
The Evaluation Synthesis, GAO/PEMD-10.1.2, 1992
Developing and Using Questionnaires.GAO/PEMD-10.1.7, 1993.
Best Practices Methodology, 1995.
Performance Measurement and Evaluation: Defi nitions and Relation-ships
(GAO/GGD-98-26), 1998.
Program Evaluation: Studies Helped Agencies Measure or Explain Pro-gram
Performance, 2000.
Total Quality Management for SAIs, 2002
Managing for results. Using Human Capital Management to Drive
Transformational Change, 2002
Government Auditing Standards, 2003.
Office of the Comptroller and Auditor General of Zimbabwe
Value for money audit manual and the quality assurance system for value
for money audits, 2001.
European Court of Auditors
Implementing guidelines for the INTOSAI Auditing Standards, 1997.

Literature
Richard Boyle and Donald Lemaire, eds, Building effective evaluation
capacity: Lessons from practice, New Brunswick, N.J, London,
Transaction, 1999.
Brown, Auditing Performance in Government: Concepts and Cases,
New Brunswick, New Transaction Books, 1982.
Campbell, D.T. and Stanley, J.C., Experimental and Quasi-Experimental
Designs for Research, Chicago, Rand McNally, 1966.
Coelli, Tom, Prasala Rao, D.S., and George Batteses: An introduction to
Effi ciency Analysis, Kluwer Academic Publishers, 1998.
Franklin J.E.and Trascher, J.H., An introduction to Program Evaluation,
New York, Wiley & Sons, Inc., 1976.
Greenbaum, T.L., The Handbook of Focus Group Research: Revised and
Expanded Edition, Lexinton Books, New York, NY, 1993.
May, Claire, Effective Writing: A Handbook for Accountants, 3rd edn,
Englewood Cliffs, New Jersey, Prentice Hall, 1992.
Mohr, L.B., Impact Analysis for Program Evaluation, Newbury Park,
Sage Publications Inc., 1992.
Naess, Arne, Communication and argument: Elements of Applied
Semantics, Universitetsförlaget, Tangen-Trykk, Drammen, 1981.
Pollit Christoffer and Hilkha Summa, Performance or Compliance?
Performance Audit and Public Management in Five Countries, OUP,
1999.
Rossi, Peter H. Freeman Howard E. and Lipsey, Mark W., Evaluation:
A Systematic Approach, 6th edn, Sage Publications Inc., Thousands
Oaks, California, 1999.
Trochim, William M., The Research Methods Knowledge Base, 2n edn,
Cornell University, Atomic Dog Publishing, 2002.
Vedung, Evert. Public policy and program evaluation, New Brunswick,
N.J., Transaction Publishers, 1997.
Weiss, Carol H., EEEvvvaaallluuuaaatttiiiooonnn::: mmmeeettthhhooodddsss fffooorrr ssstttuuudddyyyiiinnnggg ppprrrooogggrrraaammmsss aaannnddd pppooollliiiccciiieeess,,
Upper Saddled River, N.J., Prentice Hall, 1998.
Yin, Robert K., Case Study Research: Design and Methods, 2nd edn,
Applied Social Research Methods, Vol 5, Sage Publications Inc.,
Thousands Oaks, London, New Delhi, 1994.

Appendices

Appendix 1
Performance Audit Methodology
Introduction – performance auditing and the data collecting process
Performance auditors may deal with a multitude of topics and per-spectives
covering the entire government sector. Performance auditors
may also use and combine a large variety of methods for collecting
and processing information, and data may be collected for different
purposes during an audit.
Information and data are usually collected to make it possible to
understand and describe the audit object, assess and measure output,
identify shortcomings, describe and analyze cause and effect relation-ships,
test hypotheses and explain performance, and test arguments
and proposals. Data collection may also be an important part of the
learning process where the auditor tries to understand the studied area
and its problems.
It is not possible to describe all approaches, models and methods
used by performance auditors. The approaches vary and the restric-tions
are few. A short description is provided below of the most
commonly used approaches and methods. Some of them aim to build
knowledge and better understanding, while others are used to verify
and extract basic facts.63
Performance audits involve a number of different steps in the data
collection process, such as:
Planning
(the process of defining issues or problems to be studied)
AAuuddiitt Audit qquueessttiioonnss
questions (the questions to be answered)
Study SSttuuddyy ddeessiiggnn
design (the information needed and the study to be done)
Audit AAuuddiitt pprrooggrraamm
program (the type of investigations to be conducted)
Data collection
(the techniques for data collection to be used)
Analyses
(the explanations and the relationships to be explored)
63 The ambition here is to provide a general description of methods used in performance auditing. For
practical implementation there is a need for more detailed information that would include the presenta-tion
of audit practices. Whether such guidance should be provided by INTOSAI is a question for the
future. Much information is available on web-sides.

86
Even though these steps constitute the performance audit
methodology, it must be stated that a performance audit must also
always be based on such things as individual insight, experience,
imagination and creativity, in other word processes or events that
cannot be ‘mechanically followed or applied’.
1. Planning the audit
Data collection starts in the planning phase, when possible topics for
audits are monitored and selected. The search is more specifi c during
the feasibility study, when the main study is prepared. Studies of docu-ments
and interviews with different stakeholders are probably the most
common data-gathering techniques in this stage. There are several
methods used by SAIs to assist the planning process, such as:
• risk analysis;
• SWOT analysis; and
• problem analysis.64
2. Formulating the audit question or defining the audit problem
Having decided the subjects and the objectives of the audit, the ques-tions
to be answered by the study have to be defi ned. A few examples
of general audit questions common to performance auditing are the
following:
• Do the means chosen represent economical use of public funds?
• What causes the rapid increases in costs?
• Are the services provided of good quality and client-oriented?
• Why are the services not delivered on time?
• Are government programs implemented effi ciently?
• Are the goals and objectives of the government program met?
• What is the reason for the poor impact of the government program?
64 Risk Analysis is common in all forms of auditing, especially fi nancial auditing. It has always been an
important tool in internal auditing. SWOT Analysis (analysis of Strengths, Weaknesses, Opportunities
and Threats) is perhaps less common in government auditing, but some SAIs apply it in their planning
processes. Problem Analysis is mainly a tool for SAIs that have adopted a more problem-oriented ap-proach.
For more information, see for instance: Auditing: A Risk Analysis Approach. 5th Ed. Konrath,
Larry. F. Thomson Learning. 2002; Dynamic SWOT Analysis – the Developer’s guide. Richard Dealtry,
IPC, UK, 1994; and Handbook in Performance Auditing, HHaannddbbooookk iinn PPeerrffoorrmmaannccee AAuuddiittiinngg, RRRRVV,, 11999988..

In formulating the audit question, performance auditors must rely
on the information collected and their own skills and experience.
Techniques such as mind-mapping, brainstorming etc. are sometimes
used in this stage.65 The process requires both internal discussions
within the audit team (and with senior auditors and mangers) as well
as meetings with experts and stakeholders. After having formulated the
general audit question, the auditors have to break it down into specifi c
and testable sub-questions to be answered by the study, i.e. the more
specifi c questions concerning wwwhhhaaattt iiisss oorr wwwhhhyyy iiisss iiittt?? TThhiiss iiss aann iitteerraattiivvee
process.
In the problem-oriented approach, emphasis has to be put on how
to defi ne the problem properly. Usually the audit begins with problem
indicators of some kind (shortcomings in service, complaints, rising
costs etc.). At the next stage, the auditors tries to relate and link the
different problems to each other, and will then attempt to defi ne the
problem to be audited as precisely as possible. They will also formulate
testable hypotheses regarding possible causes of the problem.66
3. Defining the study design
The next step is to decide what kind of information is needed to
answer the questions. To be able to choose an approach that refl ects
reality and matches the questions set, a wide range of issues has to be
considered. Some common approaches in performance auditing are
the following:
1. Goal-attainment studies or out-come based studies67
Basic question: Are the programs achieving their overall goals?
These studies assess the extent to which a program achieves its out-come-
oriented – and client-oriented – goals or objectives. They focus
65 See for instance: Brainstorming: How To Create Successful Ideas. Charles Clarke. Paperback. 1990, and
Mapping the Mind. Rita Carter. Paperback. 2000.
66 For practical information on how to defi ne problems and formulate hypothesis etc., see Handbook in
Performance Auditing, Theory and practice. RRV. 1999.
67 For more information, see Weiss, C. H. Evaluation research: Methods for assessing program effective-ness.
Englewood Cliffs, N.J.: Prentice Hall. 1972; and Rossi, Peter H. Freeman Howard E. and Lipsey,
Mark W., Evaluation: A Systematic Approach, 6th edn, Sage Publications Inc., Thousands Oaks, Califor-nia,
1999.

88
on outputs and outcomes (including side effects and unintended
effects) in order to judge program effectiveness, but may also put
emphasis on quality issues and client perspectives. Common questions
include: How were the program goals established? What is the status
of the program’s progress? Will the goals be achieved within the times
specifi ed? The general steps in goal attainment or outcome-based stud-ies
include: defi ning the major outputs and outcomes to be studied,
specifying the measures to be observed that will suggest that the key
outcomes for the targeted clients are being achieved, and identifying
what information is needed to demonstrate this.
Studies of goal attainment are common in performance auditing.
They are mainly used to examine whether set goals are reached and to
establish, through a general assessment, whether there are any short-comings
in the work that has been done. Any deviations can be estab-lished
by relating outcome and effects to the stated goals and demands.
The performance auditor uses such studies primarily when the goals
are clearly defi ned and are used as a management instrument, when
there is only limited knowledge of the effects, and when more general
assessments may be needed.
Examination of outcomes against objectives formulated in general
terms may give an idea of the way these objectives have been met.
However, this kind of information does not usually produce satisfacto-ry
guidelines for a more detailed assessment of the entities in question,
or what is needed to improve the outcome. Under certain circumstanc-es,
studies of goal attainment may, however, provide a basis on which
to assess what changes are needed. This is the case, for example, where
the objectives are few, compatible, concrete, and directly related to a
specifi c means of control or single activity (a regulation, an informa-tion
campaign etc.).
Occasionally, the gggoooaaalll---mmmeeeaaannnsss aaannnaaalllyyysssiiisss iiss uusseedd,, ttoo iilllluummiinnaattee cceerrttaaiinn
shortcomings between resources committed and goals attained. A goal-means
analysis may also be used when there is reason to believe that an
audited entity has not acted effi ciently.
A ppprrrooogggrrraaammm lllooogggiiiccc mmmooodddeeelll ((oorr aa ppoolliiccyy iinntteerrvveennttiioonn tthheeoorryy)) mmaayy hheellpp tthhee
auditor to conduct goal-attainment studies (and other studies). The
model depicts the structure or logic of the program being audited. It
shows the program hierarchy in terms of objectives and responsibili-ties.
Starting with the highest-level program objectives and desired
effects, the program logic model moves down through subprograms,
subprogram components, and specifi c activities, with each lower level

element being logically related to one at a higher level. A program
logic model can help the auditor to obtain an understanding of the
performance audit issues as it focuses attention on the relationship
between the program’s objectives and sub-objectives and the outputs
and outcomes (impacts and effects) that result from the program. It
can help the auditor to identify and seek answers to questions such as:
• Do the objectives provide a clear understanding of the rationale
behind the program, of the products and services that are being
provided, and of the recipients of these goods and services?
• Do the objectives allow the identifi cation of measurable outcomes?
• Are the causal linkages between the hierarchical levels plausible?
In the planning phase, program logic models help the auditor to
understand the audited entity and to identify key program results and
the systems and operations that produce them. A program logic model
is mainly applicable when reality is simple and rational. The drawback
with the model is that reality seldom corresponds with these assump-tions.
2. Process-based studies68
Basic question: How does the program work?
Process-based studies are geared to permit full understanding of how
a government program works: how does it produce the results that is?
Typical questions to be addressed are: What are the steps and proce-dures
in the working process? Are the resources managed and utilized
economically and effi ciently? What is the general process that clients go
through with the program? What are the common complaints? What
do clients and staff considers being the strengths and the weakness
of the program? On what basis are the services needed? These kinds
of studies are common, and they are especially useful if programs are
long-standing and have changed over the years, and if signs of short-comings
are reported. The process-based approach involves many
kinds of investigations (such as time management-, resource-utiliza-tion-,
stakeholder-studies etc).
68 See for instance: Weiss, C. H. Evaluation research: Methods for assessing program effectiveness. Eng-lewood
Cliffs, N.J.: Prentice Hall. 1972, and Rossi, Peter H. Freeman Howard E. and Lipsey, Mark W.,
Evaluation: A Systematic Approach, 6th edn, Sage Publications Inc., Thousands Oaks, California, 1999.

90
3. Impact studies69
Basic question: What are the net effects of the government program?
Impact studies assess the net effect of a program by comparing
program outcomes with an estimate of what would have happened in
the absence of the program. This type of study is used when external
factors are known to infl uence the program’s outcomes, in order to
isolate the program’s contribution to the achievement of its objectives.
4. Cost-benefit studies and cost-effectiveness studies70
Basic questions: Do the program benefits exceed the costs, and are the
objectives met at the lowest possible costs?
Cost-benefi t studies are investigations of the relationship between the
costs and benefi ts of government projects or programs expressed in
monetary terms. For example, a cost-benefi t study might be used to
audit the effi ciency of investment projects (for instance road-build-ing
projects). The purpose of such a study is to determine whether the
benefi ts of an entity, program or project exceed its costs. Cost-benefi t
studies may be used:
• To obtain assurance that an analysis by the audit entity is reliable;
• To compare costs and benefi ts when both are known or can reason-ably
be estimated;
• To compare costs of alternatives when benefi ts can be assumed con-stant.
A cost-benefi t study should normally consider not only the tangible
(and relatively easily measurable) costs and benefi ts, but also the intan-gible
(and diffi cult to estimate) costs and benefi ts.
Cost-effectiveness CCoosstt--eeffffeeccttiivveenneessss ssttuuddiieess aarree ssttuuddiieess ooff tthhee rreellaattiioonnsshhiipp bbeettwweeeenn
project costs and outcomes expressed as costs per unit of outcome
achieved. While a cost-effectiveness study allows auditors to compare
the economic effi ciency of program alternatives, cost-effectiveness
studies are concerned with fi nding the cheapest means of accomplish-ing
a defi ned objective or the maximum value from a given expendi-ture.
In contrast to the economists’ version of cost-benefi t studies, in
cost-effectiveness studies the benefi ts may be expressed in physical
rather than monetary units: the effectiveness of a program in reaching
69 For more information, see for instance: Impact Analysis for Program Evaluation. Lawrence. B. Mohr.
Paperback. 1996, and Causal Analysis: A method to Identify and Test Cause and Effect Re lationships in
Program Evaluation. GAO. 1982.
70 See, Cost-benefi t and cost-effectiveness analysis. In D.S. Codray, H.S. Bloom and J.R. Light (Eds.),
Evaluation practice in review, San Francisco, CA: Jossey-Bass. 1987.

given substantive goals is related to the monetary value of the resourc-es
going into the program or activity.
5. Benchmarking studies 71
Basic question: Are things being done in accordance with best
practices?
Benchmarking is a process for comparing an organization’s (a pro-gram’s)
methods, processes, procedures, products, and services against
those of organizations (programs) that consistently distinguish them-selves
in the same categories. Benchmarking may be used to:
• stimulate an objective review of processes, practices, and systems;
• develop criteria and identify potentially better ways of operating;
and
• lend more credibility to audit recommendations.
6. Meta-evaluation studies 72
Basic question: Is the quality of the conducted evaluation acceptable?
The purposes of meta-evaluations are to judge the quality of evalua-tions,
to improve the quality of evaluations, and to promote the actual
use of evaluation research in the management process. The role of the
SAI is to examine the actual quality of evaluations undertaken and the
adequacy of, and procedural conditions for, evaluation. The criteria for
meta-evaluations will concern the quality of the evaluation research
undertaken and the way the evaluation function has been integrated
into the management process. Broadly speaking, there are two possible
criteria:
The scientifi c and epistemological quality of evaluation research:
theoretical, methodological, and technical criteria, which refl ect the
state-of-the-art. The theoretical requirements concern, among other
things, the formulation of the problem, the defi nition of the con-cepts,
the hypotheses, and the cohesion of the theory as a whole. The
methodological requirements imposed on evaluation research involve,
among others, the validity and reliability of the research results. The
requirements concern, among others, the operationalization of the
evaluation criteria, based on a determination of whether the situation
in the policy fi eld satisfi es the evaluation standards.
71 See, Benchmarking. How benchmarking can help in vfm examination. NAO. 1997.
72 See for instance: Cook, T. D. & Gruder, C. L. Meta-evaluation research. Evaluation Quarterly, 2 (1),
p. 5-51, 1978; and Scriven, M.: An introduction to meta-evaluation. Educational Product Report No 2.
1999.

92
Criteria of usefulness of the research for policy/management practice:
this means that an audit report should provide information that is im-portant
for an effective, effi cient, and legitimate approach to a certain
policy problem. In practical terms, the report should contain explicit
references to the need for information to be satisfi ed by the research, to
a problem in policy practice, to the research objectives associated with
policy practice, etc.
7. Other common types of studies
Both approaches and grounds for assessment must be adapted to the
multiple types of issues performance auditing deals with. One may
even say that the nature of performance auditing, to a large extent,
has helped to shape and defi ne audit topics and to select and design
models and methods to be applied in each audit. The performance
auditor must maintain an extensive network of contacts, which include
contacts in the fi eld of social science. The special nature of the work
requires performance auditors to monitor new research fi ndings and
acquire new knowledge, for example, keeping abreast of developments
and maintaining awareness of innovative approaches.
Some examples of such other common approaches are the following:
• Organizational studies: Studies of whether organizational structures,
processes and programs, etc., correspond to best practice.
• Specifi c service- and quality-management studies: Studies of serv-ice-
capacity and quality-assurance systems.
4. Defining the audit program
Investigations of large numbers of people are seldom carried out in
performance auditing. Sampling and case studies are, however, com-mon.
When case studies are included in an audit, the interest does not
focus on individual cases, but on the possible conclusions that may
be drawn from them. Some of the usual investigation designs are the
following:
1. Comparative investigation73
Comparative investigations are mainly used to examine development
trends and alternative conditions. Comparisons may be made over
73 For more information, see Ragin, C.C. The Comparative Method: Moving Beyond Qualitative and
Quantitative Strategies. Berkley: University of California Press. 1989.

time and between different outcomes or alternatives. Comparisons
can be made between subjects that work well and those that work less
well, between one or more subjects and a general picture, and between
similar areas in different countries.
2. Before and after investigation74
In a ‘before and after’ investigation, the situation before the program
was started is compared with that after program implementation. A
simple ‘before and after’ study is one in which one set of measurements
is taken before program participation and a second set is taken on the
same set of participants after suffi ciently long participation. Impact
is estimated by comparing the two sets of measurements. The main
drawback to this design is that the differences between before and after
measures cannot be confi dently ascribed to the program.
3. Sampling investigation 75
The audit fi ndings, conclusions, and recommendations must be based
on evidence. Since auditors seldom have the opportunity to consider
all information about the audited entity, it is crucial that the data col-lection
and sampling techniques are carefully chosen (AS 153).
The auditor must make a judgment as to whether sampling is an
appropriate way of obtaining some of the audit evidence required.
Among the factors that must be considered are:
• the number and relative sizes of the items in the population;
• the complexity of the questions to be answered in the sample, and
• the relevance and reliability of evidence produced with alternative
tests and procedures, and the relative cost and time involved in each.
As the auditor seeks to draw conclusions about a whole popula-tion
by testing a sample of items selected from it, it is essential that
the sample is representative of the population from which it is drawn.
A sample may be statistical or non-statistical; both require the use of
professional judgment.
The fi rst stage in planning the sample is to make an exact defi ni-tion
of the population. For statistical samples, it is important that it is
homogenous. It is also essential that the auditor clearly defi nes the spe-cifi
c audit objective that testing with the aid of the sample is designed
to achieve.
74 See for instance, Campbell, D.T. & Stanley, J.C. Experimental and Quasi-Experimental Designs for
Research. Chicago. Rand McNally. 1966.
75 See for instance: UUUsssiiinnnggg SSStttaaatttiiissstttiiicccaaalll SSSaaammmpppllliiinnnggg.. GGAAOO.. 11999922..

94
Moreover, the sample size must be determined. Throughout the
selection procedure, the auditor should regularly review whether the
sample selected is likely to adequately represent the population. As far
as possible, testing should follow a pre-determined questionnaire. As
errors or exceptions are found, it is necessary to consider their cause
and nature. The results must be evaluated and documented.
Sampling is often used to obtain evidence in performance audits.
While the objectives of the sampling exercise may be different, the
underlying principles are the same. In performance auditing, sampling
techniques are mostly used when essential facts cannot be obtained
in other ways and when there are demands for structured compari-son
and well-founded generalizations. Due to limited resources, the
samples are seldom very large, and it is usually necessary to limit the
number of questions to be answered. To obtain more in-depth knowl-edge,
the selection methods may be supplemented with other informa-tion-
gathering techniques, such as case studies.
4. Case study investigation 76
Case studies may be used to demonstrate that the assumed problems
exist and that they are not merely marginal problems. Case studies
can also be used to make in-depth analyses and comparisons. How-ever,
it is diffi cult to make statements based on case studies regarding
the frequency and extent of problems, even if the cases are selected to
illustrate general problems and are representative. However, in com-bination
with other methods, case studies may allow some general
conclusions to be drawn. Since such studies are preferable to large
investigations, they are often used in performance audits.
Among other things, case studies are used to:
• make a comprehensive in-depth analysis of complex problems;
• obtain illustrative examples to discuss and check against other infor-mation;
and
• In combination with general statistics, to illustrate and confi rm the
results of broader studies.
In choosing the number of cases to be studied, a balance must be
struck between the breadth and depth of the examination. The selec-tion
of case studies may include both diffi cult and successful cases, so
that comparisons may be made to identify underlying factors. Another
76 For more information, see: Case Study Evaluation. GAO. 1990. Yin, Robert K. Case Study Research.
Design and Methods. Second Edition. Applied Social Research Methods. VOL 5. Sage Publications,
Thousands Oaks, London, New Delhi, 1994.

type of case study is carried out to supplement more general informa-tion
obtained from the in-depth examination of an individual case.
Once a number of case studies have been compiled, the auditors
may circulate the fi ndings to the audited entities involved to obtain
their views on the extent to which the studies give a general picture
of the problems. The results of the case studies may also be discussed
with scientists, experts, and specialists in seminars and elsewhere. By
combining general statistics with the in-depth case studies–and verify-ing
them–the auditors may gain suffi cient acceptance of their fi ndings
to allow them to form the basis for some generalizations.
5. Quasi-experimental investigation 77
The essential feature of true experiments is the random assignment of
subjects to treated and untreated groups constituting the experimental
and control groups, respectively. A control group is a group of un-treated
subjects that is compared with experimental groups in terms of
outcomes. An experimental group is a group of subjects to whom an
intervention is delivered and whose outcome measures are compared
with those of control groups.
A quasi-experiment is a research design in which ‘experimental’ and
‘control’ groups are formed non-randomly. The use of the experimen-tal
method on practical and political problems has led to increases in
‘quasi-experimental’ methods, which attempt to eliminate as much as
possible the extraneous effects that make assessment of impact diffi -
cult, though without the full scientifi c rigor that a properly-conducted
experiment would.
The two common types of quasi-experimental designs involve
constructing control or comparison groups in an attempt to approxi-mate
random assignment. This is done by matching participating and
non-participating targets or by statistical adjustment of participants
and non-participants so they are as equivalent as possible in respect of
relevant characteristics.
77 See Campbell, D. T. & Stanley, J.C. Experimental and Quasi-Experimental Designs for Research.
Chicago. Rand McNally. 1966.

96
5. Selecting techniques for gathering information
Data collection may be performed once or through ongoing measure-ments
(time series design, longitudinal analysis). Information may
be gathered on the basis of physical evidence, documents (including
written statements), oral testimonies (interviews), or by other means
depending on the objectives of the audit. It will often be necessary to
collect both quantitative and qualitative data. The types of data to be
obtained should be explainable and justifi able in terms of suffi ciency,
validity, reliability, relevance, and reasonableness. Performance audit-ing
may produce primary data (its own source material) with the aid
of questionnaires, surveys, and direct observation. However, a great
deal of secondary data (material produced by others) is also often used.
This includes offi cial statistics, which are extracted and processed, and
the audited bodies’ own statistics are also often used, if considered
relevant and reliable.
Both quantitative and qualitative information and data are used
(and combined). Quantitative data are measurable or numerical and
may be used to illustrate or back up a statement. Qualitative informa-tion
and data may, for instance, refer to opinions and attitudes or gen-eral
observations, but it may refer to knowledge that is more profound.
In general, qualitative information of some kind is always needed in
order to give an analysis breadth and depth in performance auditing.
Some of the methods used for data gathering are the following.
1. File examination
Documents provide an effi cient way of collecting data, and fi le exami-nation
is likely to form the basis of many performance audits. Files
contain a wide range of types of evidence, such as the decisions of
offi cials, the ‘case records’ of program benefi ciaries, and the records of
government programs. It is important to establish the nature, location,
and availability of fi les at the outset of a performance audit so that they
can be examined cost effectively.
2. Secondary analysis and literature search
Secondary analysis may relate to the review of general research reports,
books and papers in the subject area of the program, or to studies that
are more specifi c and statistics in the area, including both historical
material and current publications. It is important to examine different

kinds of documents from the audited entity as well as past audits and
evaluations carried out by the SAI, or others as these may update and
enlarge the auditor’s working knowledge of a particular subject. Stud-ies
of documentation can give the performance auditor access to useful
material, but it is important to assess the reliability of the content of
the documents – whether the information they contain gives an objec-tive
or subjective picture, whether they give a multi-faceted picture, etc.
3. Surveys or questionnaires
A survey is a systematic collection of information from a defi ned
population, usually by means of interviews or questionnaires adminis-tered
to a sample of units in the population. Surveys are used to gather
detailed and specifi c information from a group of people or organiza-tions.
They are particularly useful when one needs to quantify infor-mation
from a large number of individuals on a specifi c issue or topic.
Questionnaires are mainly used to collect facts that are not available in
any other way and that are important as a reference to substantiate a
viewpoint. Questionnaires are thus used when comprehensive knowl-edge
is needed. Case studies and other in-depth methods are often
used to supplement questionnaires. A wide range of survey techniques
is available. The most commonly used are postal, Internet, telephone,
and in-person interviews. Questionnaires often require computer
processing and also assume a good knowledge of the area in question.
Properly used, the questionnaire technique is effective, though it can
be diffi cult and time-consuming to design questions and to process the
answers obtained. As a result, experts are often needed.
4. Interviews
An interview is basically a question and answer session to elicit specifi c
information. A great deal of performance audit work is based on
interviews, and different kinds of interviews are carried out at differ-ent
stages of the audit. The entire spectrum of interviews is used, from
fact-fi nding conversations and discussions, through unstructured in-terviews
(that is, with ‘open-ended’ questions), to structured interviews
that follow a list of closed questions:
• preparatory and inventory interviews;
• interviews to collect material and information;
• interviews to chart attitudes and arguments, and
• interviews to generate and assess ideas and suggestions

98
Interviews may be used both in the planning phase and in the
examination itself, to obtain documents, opinions and ideas that relate
to the audit’s objectives, to confi rm facts and corroborate data from
other sources, or to explore potential recommendations. Various kinds
of techniques are used, one or more people may be interviewed at the
same time, and the interviews may be over the telephone or in the
form of personal visits. They may be in-depth or more in the nature of
checking information.
To obtain the broadest possible view of reality, it is important to in-terview
people with different positions, perspectives and insights (staff
at both central and local levels, staff from different interested parties or
stakeholders, experts in the fi eld in question etc.).
Interviews are a way of gathering facts and information and gain-ing
support for a variety of arguments, but one cannot rely solely on
interviews. Although many of the details and statements obtained in
an interview will have to be checked before they can be used, in many
contexts interviews are still a useful way of collecting facts. The results
of the interviews must then be compiled and documented in a way
that facilitates analysis and quality assurance, for example by separate
grouping of the factual material; the problems, causes, consequences,
and proposals put forward (conditions, criteria, causes, effects, and
potential recommendations).
5. Seminars and hearings
Seminars are often used in the different phases of a performance audit,
not least when preliminary observations and conclusions are to be
discussed.
Seminars might be used for instance to:
• acquire knowledge of a specialist area;
• discuss problems, observations, and possible measures; and
• air arguments for and against different views and perspectives.
Seminars have the advantage of bringing together a large number of
people representing a wide range of knowledge and perspectives. This
gives better knowledge of the area in question. Arranging and running
seminars and hearings, which are similar to seminars, is demanding.
However, the purpose of hearings is mostly to invite or call for inter-ested
parties and experts to give their views on the area to be audited.

6. Focus groups, focal groups, reference groups and experts
Focus groups are a selection of individuals brought together to discuss
specifi c topics and issues. They are primarily used to collect qualitative
data – information that can provide insights into the values and opin-ions
of those individuals in the process or activity being audited. Focal
group techniques are used to obtain information on the implementa-tion
and impacts of government programs based on the perspectives
of the benefi ciaries and other stakeholders.
The pattern of the views could be expressed in a stakeholder model.
A stakeholder study could be used in the planning of the audit in order
to identify the main groups interested in a certain action developed by
government.
Reference groups may be composed of people drawn from within
or outside the SAI and are usually made up of experts and specialists.
They are sometimes used both in initiating an audit and during the
course of the audit as a source of viewpoints. Some experts are con-sultants
or short-term employees. The reference groups may include
experts employed directly by the SAI, consultants employed by the
audited entity or experts operating on an independent basis (e.g. aca-demic
researchers). The purpose of using experts is, to make technical
knowledge or skills that are essential for the achievement of the audit
objectives available to the audit team. This kind of expert assistance
has the advantage of allowing relevant information to be acquired
quickly. One drawback, however, is that it may be diffi cult to judge the
competence of experts, to obtain the appropriate expertise, to check
their work, or to evaluate the results produced by the experts.
7. Direct observations
Direct observations are not common in performance auditing. This
method is mainly used to gain insight into, and understanding of, the
way an operation is run; to obtain the views of staff in the fi eld, and
discuss and test ideas, and to add to, or make comparisons with, other
information.

100
The following table presents some of the methods mentioned for
collecting data during audits/evaluations (source: Carter McNamara,
1998).
Method Overall Purpose Advantages Challenges
Questionnaires,
surveys,
checklists
When there is a need to get,
quickly and/or easily, a great
deal of information from
people in a non-threatening
way
– Can be done anonymously
– inexpensive to administer
– easy to compare and
analyze
– administer to many people
– can obtain large amounts
of data
– many sample
questionnaires already
exist
– Might not get careful
feedback
– wording can bias client’s
responses
– are impersonal
– in surveys, may need
sampling expert
– do not give full story
Interviews When one wants to fully
understand someone’s
impressions or experience,
or learn more about their
answers to questionnaires
– Obtain full range and
depth of information
– develop relationship with
client
– can be fl exible with client
– Can take a long time
– can be hard to analyze and
compare
– can be costly
– interviewer can bias
client’s responses
Documentation
review
When one wants an
impression of how
the program operates
without interrupting the
program; from a review
of applications, fi nances,
memos, minutes, etc.
– Gives comprehensive and
historical information
– does not interrupt
program or client’s routine
in program
– information already exists
– few biases about
information
– Often takes much time
– info may be incomplete
– one needs to be quite
clear about what one is
looking for
– not fl exible means
to obtain data; data
restricted that which
already exists
Observation To gather accurate
information about how a
program actually operates,
particularly about processes
– View operations of a
program as they are
actually occurring
– can adapt to events as
they occur
– Can be diffi cult to interpret
behavior
– can be complex to
categorize observations
– can infl uence behavior of
program participants
– can be expensive
Focus groups Explore a topic in depth
through group discussion,
e.g. about reactions to an
experience or suggestion,
understanding common
complaints, etc.; useful in
evaluation and marketing
– Quickly and reliably obtain
common impressions
– can be effi cient way to
get range and depth of
information in short time
– can convey key
information about
programs
– Can be hard to analyze
responses
– need a good facilitator for
security and closure
– diffi cult to schedule 6-8
people together

6. Analyzing and interpreting information
Analyzing quantitative and qualitative data is an important step in all
performance audits. When analyzing data (whether from question-naires,
interviews, focus groups, etc.), the auditors should start by re-viewing
the audit objectives and the audit question. This will help the
auditors to organize their data and focus their analysis. In interpreting
the information, the auditors should attempt to put the information
in perspective, by comparing the results to audit criteria or to what is
generally expected.
It is important to study the information both in-depth and exten-sively.
A performance audit analysis must always be based on a good
understanding of the activity under audit and its working conditions.
However, no analysis can consider everything. Each analysis must be
built on common sense and a realistic view of what can, may, and must
be put forward.
When analyzing information or data from interviews, the main
problem is to allocate what has been said into different categories or
topics. In interviews the auditor is looking for common threads of in-formation,
things that fi t together, or examples of the same underlying
problem, issue, or concept. In this sense, qualitative (non-numerical)
analysis may be used to assess and explain auditee performance. By
reading documents, the auditors are able both to collect descriptive in-formation
and to systematically analyze the text and look for anything
that is relevant.
Information analysis is an intellectual, creative, and iterative process,
which includes both rational and irrational elements. It always involves
refl ections and discussions, brainstorming, and mostly non-quantita-tive
techniques such as content analysis, comparative analysis, analysis
with the aid of expert panels etc.78
Two examples of quantitative techniques are mentioned below.79
1. Descriptive statistics to understand data distribution
Data distribution may often be expressed in the form of a graph (bar
chart or curve) that shows all the values of a variable. The statistics
78 Most of these methods are described in books like: Weiss, Carol H. Evaluation: methods for studying
programs and policies. Upper Saddled River, N.J.: Prentice Hall. 1998; and Rossi, Peter H, and Freeman
Howard E, and Lipsey, Mark W. Evaluation A Systematic Approach. 6th edn, Sage Publications Inc,
Thousands Oaks, California. 1999.
79 See for instance, Quantitative Data Analysis. GAO 1992.

102
that describe data distribution can be powerful tools for audit analysis
and reporting.
There are three basic dimensions of data distribution that may be
important to an auditor:
• the ‘“central tendency’” of the distribution (mode, median, mean,
quartile level, etc.);
• the spread (variability or dispersion) of the data (minimum and
maximum values, tails, etc.); and
• the shape of the data (standard deviation, normal distribution, fl at
distribution, bi-modal distribution, etc.).
Data distributions may be used:
• to identify the level, spread, or shape of the data when this is more
important than a single ‘average’ number;
• to decide whether variable performance meets an audit criterion or
not;
• to interpret probability distribution to assess risk; and
• to assess whether sample data are representative of the population.
2. Regression analysis
Regression analysis is a technique for assessing the degree to which
variables are associated (correlated). Regression analysis may be used:
• to test a relationship that is supposed to hold true;
• to identify relationships among variables that may be causally re-lated
that might explain outcomes;
• to identify unusual cases that stand out among expected values; and
• to make predictions about values in the future.

Appendix 2
Performance Audit Criteria
Audit criteria are reasonable and attainable standards of performance
against which the economy, effi ciency, and effectiveness of activities
can be assessed. They refl ect a normative (i.e. ideal) model for the
subject matter under review. They represent best or good practice, a
reasonable and informed person’s expectation of “what should be.”
When criteria are compared with what actually exists, audit fi ndings
are generated. Meeting or exceeding the criteria might indicate “best
practice,” but failing to meet criteria would indicate that improvements
could be made.
Some characteristics of suitable criteria include the following.
RRReeellliiiaaabbbiiillliiitttyyy::: RReelliiaabbllee ccrriitteerriiaa rreessuulltt iinn ccoonnssiisstteenntt ccoonncclluussiioonnss wwhheenn
used by another auditor in the same circumstances.
Objectivity: Objective criteria are free from any bias of the auditor or
management.
UUUssseeefffuuulllnnneeessssss::: UUsseeffuull ccrriitteerriiaa rreessuulltt iinn fifi nnddiinnggss aanndd ccoonncclluussiioonnss tthhaatt
meet users’ information needs.
Understandability: Understandable criteria are clearly stated and are
not subject to signifi cantly different interpretations.
CCCooommmpppaaarrraaabbbiiillliiitttyyy::: CCoommppaarraabbllee ccrriitteerriiaa aarree ccoonnssiisstteenntt wwiitthh tthhoossee uusseedd
in performance audits of other similar agencies or activities and with
those used in previous performance audits of the entity being audited.
CCCooommmpppllleeettteeennneeessssss::: CCoommpplleetteenneessss rreeffeerrss ttoo tthhee ddeevveellooppmmeenntt ooff aallll ssiiggnniififi --
cant criteria appropriate to assessing performance.
AAAcccccceeeppptttaaabbbiiillliiitttyyy::: AAcccceeppttaabbllee ccrriitteerriiaa aarree tthhoossee tthhaatt iinnddeeppeennddeenntt eexxppeerrttss
in the fi eld, audited entities, legislature, media, and general public are
generally agreeable to.
Criteria can perform a series of important roles to assist the conduct
of a performance audit, including:
• forming a common basis for communication within the audit team
and with SAI management concerning the nature of the audit;

104
• forming a basis for communication with the auditee’s management;
• forming a basis for the data collection phase by providing a basis on
which to build procedures for the collection of audit evidence; and
• providing the basis for audit fi ndings and helping to add form and
structure to observations.
Their level of detail and the form they take often determines the
degree to which criteria are successful in serving these uses.
It is unrealistic to expect that those activities, systems, or levels of
performance in economy, effi ciency, and effectiveness areas will always
fully meet the criteria. It is important to appreciate that satisfactory
performance does not mean perfect performance, but is based on what
a reasonable person would expect, taking into account auditee circum-stances.
The audit criteria must be set objectively. The process requires
rational consideration and sound judgment. The auditors must for
instance:
• have a general understanding of the area to be audited, and be famil-iar
with relevant legal and other documents as well as recent studies
and audits in the area;
• have good knowledge of the motives and the legal basis of the
government program or activity to be audited and the goals and
objectives set by the legislature or the government;
• have a reasonable good understanding of the expectations of the
major stakeholders, and be aware of basic expert knowledge; and
• have a general knowledge of practices and experience in other
relevant or similar government programs or activities.
Moreover, it is often useful to obtain the input of auditee(s) man-agement
to the development of criteria. Disagreement about criteria
can then be identifi ed, discussed, and, perhaps, resolved at an early
stage. However, the facts and arguments presented by the auditee(s)
must be weighed against other relevant facts and arguments (from
other sources, experts etc.).
Goals set by the legislature or the executive branches are sometimes
vague or confl icting. Under such conditions, the auditors might have
to interpret the goals to make them more operational or measurable.
One possibility is to get experts and stakeholders in the fi eld to answer
questions such as: How should the goals and objectives best be inter-preted
and measured? What should be the expected results under the
given conditions? What is the best-known comparable practice? If the

goals are confl icting, one option – if other alternatives seem inappro-priate
– is to divide the audit project into several sequential studies,
covering one goal at the time. In cases of vague or long-term goals it
might sometimes be possible to narrow the scope somewhat and look
for short-term perspectives and direct criteria. (There are of course
also other options.)

Appendix 3
Evidence and Documentation
1. Audit evidence
Audit evidence is information collected and used to support audit
fi ndings. The conclusions and recommendations in the audit report
stand or fall on the basis of such evidence.80 Consequently, perform-ance
auditors must give careful thought to the nature and amount
of evidence they collect. All fi eldwork should be planned from the
perspective of acquiring evidence intended to support the fi ndings ap-pearing
in the fi nal report.
The INTOSAI auditing standards state: ‘Competent, relevant and
reasonable evidence should be obtained to support the auditor’s judgment
and conclusions regarding the organization, program, activity or function
under audit’ (AS 3.5.1).
1.1 Competence of evidence
Evidence is competent (valid and reliable) if it actually represents what
it purports to represent. Considering the following matters can assist
when assessments are made of the reliability of evidence:
• Corroboration of evidence is a powerful technique for increasing
reliability. This means that the auditor looks for different types of
evidence from different sources.
• Sources of evidence from outside the audited entity are – rightly or
wrongly – often viewed as more reliable than information generated
within the auditee.
• Documentary evidence is usually considered to be more reliable
than oral evidence.
• Evidence generated through direct auditor observation or analysis is
more reliable than evidence obtained indirectly.
80 The INTOSAI’s Code of Ethics and Auditing Standards, glossary.

108
• The reliability of auditee-generated information will partly be a
function of the reliability of the auditee’s management/internal con-trol
systems.
• Oral evidence that is corroborated in writing is more reliable than
oral evidence alone.
• Original documents are more reliable than photocopies.
1.2 Relevance of evidence
Relevance requires that the evidence bear a clear and logical rela-tionship
to the audit objectives and to the criteria. One approach to
planning for data collection is to list, for each issue and criterion, the
nature and location of evidence that is needed, as well as the audit
procedure that is to be implemented.
Sufficiency of evidence
The auditor should obtain suffi cient appropriate audit evidence to be
able to draw reasonable conclusions on which to base the audit report.
Suffi ciency is a measure of quantity of audit evidence. Appropriateness
is a measure of quality of audit evidence, its relevance to particular
criteria, and its reliability.
Evidence is suffi cient when there is enough relevant and reliable
evidence to persuade a reasonable person that the performance audit
fi ndings, conclusions and recommendations are warranted and sup-ported.
In determining whether documentary evidence is suffi cient,
the auditor must take account the status of the document.
The factors that dictate the strength of evidence required to support
an observation in performance auditing include:
• the level of materiality or signifi cance of the observation;
• the degree of risk associated with arriving at an incorrect conclu-sion;
• experience gained in previous audit examinations – or other inves-tigations
– on the degree of reliability of the auditee’s records and
representations:
• known auditee sensitivity to an issue; and
• the cost of obtaining the evidence relative to the benefi ts in terms of
supporting the observation.
Evidence gathered during a performance audit may be predomi-nantly
qualitative in nature and require extensive use of professional
judgment. Accordingly, the auditor would ordinarily seek corroborat-

ing evidence from different sources or of a different nature in making
assessments and forming conclusions.
When planning the audit, the auditor should identify the prob-able
nature, sources, and availability of audit evidence required. The
auditor should consider such factors as the availability of other audit
reports or studies and the cost of obtaining the audit evidence.
2. Characteristics of performance audit evidence
Auditors need to be aware of potential problems or weaknesses with
performance audit evidence. Potential problems include:
• evidence based on a single source (reliability, validity, suffi ciency);
• oral evidence not supported by documentation or observation (reli-ability);
• evidence that is not time-sensitive, i.e. too old and does not refl ect
changes (relevance);
• evidence that is too expensive to obtain relative to benefi ts (rel-evance
and suffi ciency);
• source of evidence has a vested interest in outcome (reliability);
• samples collected are not representative (relevance, validity, suffi -
ciency);
• evidence may be related to an isolated occurrence (validity, suffi -
ciency);
• evidence is incomplete, i.e. does not demonstrate a cause or effect
(reliability, suffi ciency); and
• evidence is confl icting (reliability).
Evidence can be categorized as to its type - physical, oral, documen-tary,
or analytical.
2.1 Physical evidence
Observing people and events or examining property are ways to obtain
physical evidence. The evidence can take the form of photographs,
charts, maps, graphs or other pictorial representations. A photograph
of an unsafe condition is far more compelling than a written descrip-tion.
When the observation of a physical condition is critical to achieving
the audit objectives, it should be corroborated. This may be achieved
by having two or more auditors make the observation, if possible
accompanied by representatives of the auditee.

110
2.2 Oral evidence
Oral evidence takes the form of statements that are usually made in
response to inquiries or interviews. These statements can provide
important leads not always obtainable through other forms of audit
work. They can be made by employees of the auditee, benefi ciaries
and clients of the program being audited, experts and consultants
contacted to provide corroborating evidence in relation to an audit,
and by members of the general public. Corroboration of oral evidence
is needed if it is to be used as evidence rather than mere background
information.
Corroboration of oral evidence could be obtained:
• by written confi rmation from the interviewee;
• by weight of multiple independent sources revealing the same facts; or
• by checking records later.
In assessing the reliability and relevance of oral evidence, the auditor
needs to assess the credibility of the interviewee; that is, the position,
knowledge, expertise and forthrightness of the person being inter-viewed.
2.3 Documentary evidence
Documentary evidence in physical or electronic form is the most
common form of audit evidence. It may be external or internal to the
auditee. External documentary evidence may include letters or memo-randa
received by the auditee, suppliers’ invoices, leases, contracts,
external and internal audits and other reports, and third-party confi r-mations.
Internal documentary evidence originates within the audited
entity. It includes items such as accounting records, copies of outgoing
correspondence, job descriptions, plans, budgets, internal reports and
memoranda, statistics summarizing performance, and internal policies
and procedures.
The reliability and relevance of documentary evidence needs to be
assessed in relation to the objectives of the audit. For example, the
existence of a procedures manual is not evidence that the manual is
put into practice. As with oral evidence, the position, knowledge and
expertise of the author or approver of the document may need to be
assessed.
Documents that are the output of management control systems (e.g.
the accounting system) will need to be assessed in light of the internal
controls that operate within that system. Auditors who intend to rely
on such evidence should assess the system’s internal controls.

2.4 Analytical evidence
Analytical evidence stems from analysis and verifi cation of data. The
analysis can involve computations, analysis of ratios, trends, and
patterns in data obtained from the auditee or other relevant sources.
Comparisons can also be drawn with prescribed standards or industry
benchmarks. Analysis is usually numerical, and considers, for example,
ratios of output to resources, or the proportion of the budget that is
spent. It can also be non-numerical in nature; for example, observing a
consistent trend in the nature of complaints made about an auditee.
3. The evidential process
Collecting evidence takes place during both the pre-study and ex-amination
phases of an audit. Work done in the pre-study phase also
constitutes part of the overall evidence.
Auditors should:
• examine the characteristics of data required;
• collect data relevant to achieve the audit objectives;
• collect data outlined in the audit work plan;
• collect data that is suffi cient and persuasive to logically support the
analysis, observations, conclusions and recommendations; and
• apply the standard of evidence to build a successful case ‘on the bal-ance
of probabilities’.
Sources of evidence are discussed below.
3.1 Policy statements and legislation
Auditors should gather policy documents, operating guidelines, manu-als,
ministerial directives, decisions on delegation, etc., and examine
the background leading to their promulgation. Auditors should also
consider changes to legislation and the document trail leading to the
need for change, such as, submissions, press clippings, complaints, case
histories and speeches.
3.2 Published program performance data
Published auditee budget statements provide evidence on the objec-tives
and performance of agencies. They include an auditee overview
and provide fi nancial and performance information.

112
3.3 Interviews
Interviews can be useful, but it is necessary to identify the right people
to provide information and to corroborate the oral information. Solid
preparation for the topic is essential and a list of questions prepared in
advance is useful; in some cases, it may be effective to supply this list to
the interviewee beforehand.
3.4 File examination
Information obtained from fi les provides strong evidence to support
recommendations. A list of fi les should be obtained from auditee reg-istry
systems. In addition, fi le information of relevance to a particular
work area may be found in that work area. Audit interviews may also
give hints on which fi les to seek and review.
Files, which may prove useful for review, include those on:
• strategic and operational planning;
• management control;
• Executive meeting minutes;
• complaints and disputes; and
• reviews and audits.
File evidence can provide strong support for fi ndings. File examina-tion
is time intensive and it is usually not possible to examine all fi les.
Judgment must be exercised concerning whether to examine a random
selection or a selection based on the purpose of the investigation.
Usually the latter approach would be adopted, but if time permits a
random sample of other fi les should be studied.
3.5 Management reports and reviews
Agencies usually generate a number of internal reports or reviews that
summarize for senior management the issues at the time, or propose
courses for action. Auditors should locate and analyze such reports.
Ways of identifying reports include interviews and examination of
minutes of management meetings.
3.6 Databases
Most agencies have one form or another of management information
system that collects relevant information for conducting operations.
These systems can be important sources of evidence, especially in
quantifying various matters.

3.7 External sources
Larger agencies may have sophisticated specialist libraries relevant to
their areas of responsibility. Literature searches on relevant topics and
key words can be particularly useful.
3.8 SAI sources
The auditor should not overlook evidence that has been collected in
previous audits or through information collected to support strategic
planning for the auditee.
3.9 Observation
The value of direct observation should not be overlooked. Observa-tion
of the general demeanor of staff can give information on pressure,
morale, or lack of work, which can then be followed up if appropriate.
However, careful consideration needs to be given to selecting activi-ties
or facilities to be physically inspected. These should be representa-tive
of the area under examination. Auditors should also be aware that
people perform differently when they are being observed.
This type of evidence can be regarded as ‘soft’ unless corroborated.
Photographs and video recordings increase the value of direct observa-tion.
Written detailed descriptions of the results of the observations are
recommended.
4. Documentation
INTOSAI auditing standards state that: ‘Auditors should adequately
document the audit evidence in working papers, including the basis and
extent of the planning, work performed and the fi ndings of the audit’ (AS
3.5.5). It is also stated that: ‘Adequate documentation is important for
several reasons. It will:
a) Confi rm and support the auditor’s opinions and reports;
b) Increase the effi ciency and effectiveness of the audit;
c) Serve as a source of information for preparing reports or answering
any enquiries from the audited auditee or from any other party;
d) Serve as evidence of the auditor’s compliance with auditing stand-ards;
e) Facilitate planning and supervision;
f) Help the auditor’s professional development;

114
g) Help to ensure that delegated work has been satisfactorily per-formed;
and
h) Provide evidence of work done for future reference’ (AS 3.5.6).
Working papers are all relevant documents collected and generated
during a performance audit. They should include: documents record-ing
the audit planning; the nature, timing, and extent of the audit pro-cedures
performed; and the results and the conclusions drawn from
the audit evidence obtained. Working papers should therefore contain
at least three sections: planning, execution and reporting.
Working papers serve as the connecting link between the fi eldwork
and the audit report and should be suffi ciently complete and detailed
to provide an understanding of the audit. Thus, they should contain
the evidence accumulated in support of the opinions, conclusions and
analysis supporting the recommendations in the report.
Working papers assist organization, facilitate access to the evidential
documentation and thus:
• assist in the planning and performance of the audit;
• facilitate effective management of individual audits and the audit
task;
• assist in the supervision and review of the audit work; and
• record evidence resulting from audit work performed to support the
audit opinion.
The auditor should adopt appropriate procedures to maintain the
confi dentiality and safe custody of the working papers and should
retain the working papers for a period suffi cient to meet the needs of
the legal and professional requirements of record retention.
5. It is better to be vaguely right than exactly wrong
It is seldom possible for performance auditors to get all information
they need. Available data might be inaccurate, incomplete or confl ict-ing.
Thus, auditors have to be creative in trying to fi nd data that at
least reasonably well describes what they are after. It may be possible
to obtain well-founded estimations, or there might be other ways to
fi nd data that ought to be useful for practical analysis and overall as-sessment.
Less-precise data – combined with logical deduction and
other information – might sometimes be suffi cient; for instance if
the purpose is to describe tendencies. Another possibility is to get the
auditees and other stakeholders to confi rm that the less-precise data

being collected provides a reasonable and fair picture of reality, i.e. the
information collected can be used as evidence of one type or another
and as a basis for further analysis and general conclusions.
Data collection is always a compromise between the ideal (the best
solution) and the reality (the second or the third best solution). Over-estimated
ambitions in trying to fi nd complete, accurate and exact
data may hamper the effectiveness of any performance audit. Efforts
to be exact may easily be in confl ict with ambitions to make an intel-ligent
analysis. The need to be exact must therefore always be weighed
against what is reasonable economical, and relevant for the purpose of
the data-collection.
Consequently, performance auditors should always try to be practi-cal
in their efforts to collect, interpret and analyze data. It is impor-tant,
however, that the reader of the audit report is informed about
the quality of all information that has been collected and how it was
gathered. This is even more essential when it comes to data that are less
precise. The auditors should not draw any other conclusion than that
permitted by the quality of the data.

Appendix 4
Communication and Quality Assurance
External relations and relations with auditees
The development of good and proper external relations is often a key
factor in achieving effective and effi cient audits of government pro-grams.
The progress and outcome of the audit will be enhanced if the
audit team can obtain good contact and foster confi dence by maintain-ing
a fully professional approach during the course of the audit.
SAI staff should seek to maintain good relationships with all stake-holders
involved, promote the free and frank fl ow of information, and
conduct discussions in an atmosphere of mutual respect and under-standing.
The SAI should use its powers of access to information tact-fully
and with due regard to the ongoing operational responsibilities.
The SAI should endeavor to give the audited entities reasonable notice
of its intention to commence an audit and should discuss the general
scope of the study with relevant offi cers.
Relations with the auditees
The audit manager in charge of the study – or the project leader –
normally makes the initial contact with the auditee(s) in order to ad-vise
them on matters such as:
• the objectives, timing, duration, and type of audit to be conducted;
• the intended offi ces or regions to be visited; and
• names, titles, phone numbers, etc. of the audit staff and an offi cial
contact person at the SAI for the audit.
The objectives of the meetings or conferences in the initial process
are to:
• establish suitable liaison arrangements at both the management and
working levels, including arrangements for progressive reporting of
tentative fi ndings;
• ensure that the auditee(s) clearly understands the audit objectives
and processes, including a description of access powers and safe-guards
on confi dentiality;

118
• outline the auditee’s responsibilities and clarify any queries or mis-understandings
the auditee may have; and
• make administrative arrangements for the audit team, such as access
to buildings, personnel, fi les, systems, and data.
At the conclusion of each audit, an exit meeting with the managers
at the auditee(s) ought to be arranged, for example in order to:
• discuss provisional audit fi ndings, conclusions and recommenda-tions
with auditee management and obtain management’s com-ments
on them;
• afford the auditee the opportunity to correct misunderstandings and
question the audit conclusions and fi ndings.
Quality assurance and quality control systems
INTOSAI auditing standards state that the SAI should establish sys-tems
to confi rm that integral quality assurance processes have operated
satisfactorily; ensure the quality of the audit report; and to secure qual-ity
improvements and avoid repetition of weaknesses. Quality assur-ance
refers to policies, systems, and procedures established by SAIs to
maintain a high standard of audit activity. It also refers to the require-ments
applicable to the day-to-day management of audit assignments.
Quality control, on the other hand, refers to reviews conducted to
assess the quality assurance system or the executed audit projects.
SAI quality assurance activities
As part of the SAI’s professional obligations it must establish and
support adequate systems of quality assurance activities. The systems
comprise structures, policies, and procedures designed to provide the
SAI with adequate assurance that the work undertaken by the SAI
meets professional requirements and standards. Quality assurance
activities include:
• Securing the quality of the planning; the planning of selected tasks
should be reviewed to ensure that adequate consideration has been
given to all matters considered essential.
• Securing the quality of the on-going work; the on-going work
should be subject to continual review. This review is essential to
maintain the quality of audit work and to promote learning and
feedback.

• Securing the quality of the fi nalized audit; all completed tasks
should be reviewed prior to signing any reports.
Quality assurance activities should be designed to ensure that all
audits are conducted in accordance with relevant auditing standards.
The objectives of quality assurance procedures should incorporate:
• professional competence and integrity;
• supervision and assignment of personnel to engagements;
• guidance and assistance;
• client evaluation; and
• allocation of administrative and technical responsibilities.
The SAI’s general quality assurance policies and procedures should
be communicated to its personnel in a manner that provides reasona-ble
assurance that the policies and procedures are understood, and im-plemented.
Events and elements in the audit work that require special
attention by the audit managers and the SAI managers are considered
below. Systems for quality assurance – especially systems which focus
on the fi nal stages of the audit process – are often time consuming and
costly. The ideal is a fully integrated system.
Planning and budgeting and the use of consultants
The audit manager needs to take into account factors such as quality,
resources, and timing in planning the audit. The budgets consist of
allocations for salaries, travel, consultants, and any other direct costs. If
the SAI seeks advice from external experts, the standards for exercise of
due care and confi dentiality of information will apply to such arrange-ments.
Monitoring and executing the audit
The managers should ensure that the audits are completed within
budget and on time, extending the budget if justifi ed. The audit
manager must be aware of risks to timely audit completion and ensure
audit work is relevant to the objectives and scope of the audit. The
development of the data-gathering process and the analytical work
has to be monitored. The audit manager should ensure that the audit
teams are able to maintain good and proper relations with auditee(s)
and other stakeholders.
Progress reporting and audit reporting
The audit manager should inform the SAI management on the
progress of the audit, with recommendations for corrective action if

120
needed. The manager must also ensure that the audit reports meet the
reporting standards.
Quality assurance review program – external and internal reviews
A quality assurance review program is a series of external and inter-nal
reviews of activities undertaken by the SAI – it assesses the overall
quality of the work performed and covers various issues and perspec-tives.
Consequently, the program has to be fl exible. The results of the
program should be reported to the SAI management at least annu-ally
(with a high volume of performance audits). A quality assurance
review may examine adherence to policy and procedures and identify
areas where there is any opportunity for improvements in these poli-cies
and procedures, or it may assess the quality of work performed to
meet specifi ed objectives or specifi c stakeholders’ perspectives. Quality
assurance reviews will generally address both adherence to specifi ed
processes and the quality of the work performed. The report on the
quality assurance review program should summarize the results of all
the reviews including the tasks selected (number and type), the fi nd-ings,
and any recommendations.
Quality assurance and quality development
It is important that all quality assurance/control activities have a high
degree of legitimacy among the auditors and that the procedures
and systems are not too sophisticated. Besides quality assurance and
control activities, perhaps the most effective way to promote quality in
audit work is to recruit competent staff and to create working condi-tions
that:
• stimulate quality development;
• promote openness, delegation and mutual trust within the organiza-tion;
and
• encourage the auditor’s own sense of responsibility.
In an information-based and professional activity such as perform-ance
auditing, it is generally more important to support the audit
teams in their efforts to achieve a good level of quality in their work,
than to supervise them in the traditional sense of the term.

Appendix 5
Performance Auditing and Information Technology
Introduction81
Information Technology (IT) is being increasingly used for public
sector program planning, execution, and monitoring. The sharing or
integration of information between agencies raises issues such as the
risk of security breaches and unauthorized manipulation of informa-tion.
Auditors should not only be aware of the uses of IT, they should
also develop strategies and techniques for providing assurance to
stakeholders about value for money from the use of IT, security of the
systems, existence of proper process controls, and the completeness
and accuracy of the outputs. Earlier, it was common in performance
auditing to focus on issues such as planning, development and main-tenance
of individual IT systems. Today, the perspective is broader. IT
systems are primarily seen as important components in all government
programs (e-government). The shift in perspective has consequences
for performance auditing in the area of information technology.
The value of good IT systems is that they can improve the economy,
effi ciency and effectiveness of existing programs and contribute to
better public services. IT systems can be an effi cient and effective pro-gram
delivery mechanism. They have the potential to deliver existing
services at reduced cost and to provide a range of additional services,
including program performance information, with greater effi ciency,
security, and control than is available in manual systems. However, IT
systems also have the potential to result in major systemic errors with
a resultant greater impact on agency performance than would be pos-sible
if manual systems are used.
This appendix highlights a range of important considerations for
performance auditing in an IT environment and is not intended to
81 This appendix is based on documents form INTOSAI’s IT-Committee and the guidelines for per-formance
auditing provided by ASOSAI. The experience of individual SAIs has also been considered.

122
replace detailed guidelines that SAIs may need to develop in order to
evaluate their auditees’ IT environment.
The approach to performance auditing in an IT environment should
involve the following interrelated processes:
• obtain an understanding of the auditees’ IT systems and determine
their signifi cance for the performance audit objective;
• identify the extent of IT systems auditing required to achieve the
performance audit objective (e.g. audit of IT-investment processes
and their links to business strategies, audit of systems development;
audit of environment and applications controls) and employ spe-cialist
information system/IT auditors to undertake the task; and
• develop and use, when appropriate, computer-assisted audit tech-niques
to facilitate the audit.
A performance audit in an IT environment should:
• assess whether the IT systems enhance the economy, effi ciency, and
effectiveness of the program’s objectives and its management, espe-cially
in relation to program planning, execution, monitoring, and
feedback;
• determine whether system outputs meet established quality, service
and cost delivery parameters;
• identify any defi ciencies in information systems and IT controls and
the resultant effect on the effi ciency, economy, and effectiveness of
performance;
• compare the IT system development and maintenance practices of
the auditee to leading practices and standards; and
• compare the IT strategic planning, risk management, and project
management practices of the auditee to leading practices and stand-ards
including corporate governance practices.
A high level model for auditing IT use
In auditing the use of IT in business programs, the auditor needs to
focus on certain aspects. Broadly speaking, it is possible to identify the
following components in a high-level model for auditing IT issues:
• The government’s requirements and other demands concerning the
use of IT for public administration services.
• The government’s IT-based business improvement strategies and
decision processes concerning IT-based business improvement
investment projects (implementation of IT).

• The government’s development projects, running and maintenance
of IT-based business/programs, IT systems and IT infrastructure
including aspects of security, integrity, control etc.
• The clients’ use of IT systems including aspects of usability and the
interaction between users and IT systems.
• The long-term effectiveness of IT systems and IT support in busi-ness
programs (value for money in using IT).
• The support from different actors in IT-related issues given to agen-cies
in the public administration including issues such as IT stand-ards,
IT technology etc.
• The main actors infl uencing government and agencies in IT-related
issues such as IT trends, IT knowledge etc.
Performance aspects of auditing in an IT environment
In many cases the most important issue of the audit is to establish
whether the IT system has enhanced the effi ciency with which the
auditee manages its programs and whether the IT system has benefi cial
results for the stakeholders.
The auditor may also be expected to assess if the IT systems have
facilitated improved program management. Some areas to be consid-ered
include:
• The IT investment process – especially the auditee’s innovation
system for creating, processing and deciding on IT investment
proposals – linkage to business strategy, management and planning
processes.
• IT should support the objectives and business strategy of the auditee
and, therefore, is an integral part of its operations.
• IT operations require highly qualifi ed staff.
• The contribution of IT to operations is measured in operational ef-fi
ciency terms.
• The benefi ts of IT may not be realized without appropriate changes.
• Normal value for money measures may be more diffi cult to apply.
In addition to assessing whether the auditee’s IT systems represent
value for money, the performance auditor may also be expected to
measure if the IT environment has contributed to transparency, ac-countability,
and good governance.
The audit may also contain more specialized IT issues, i.e. IT system
development and operational management.

124
Performance auditing involving IT system development
A performance audit involving IT systems development should deter-mine
if the audited entity:
• has the appropriate executive approval for the development of the IT
system, i.e., that IT management fi ts in the corporate governance of
the auditee;
• has appropriate project management processes in place to manage
the project;
• has met required targets of time, cost, system function, and value for
money;
• uses an appropriate system development methodology; and
• has processes in place, including the involvement of internal audi-tors,
to ensure that the new system includes all the necessary con-trols
and audit trails, and is likely to meet the requirements of the
auditee and its stakeholders.
Performance auditing involving operational IT systems
The following list contains some of the more important concerns that
the auditor would be expected to consider and should be modifi ed as
required for the specifi c entity being audited:
• the strategic and operational management of IT, including assur-ance
that IT is included in the overall corporate governance of the
auditee;
• IT project management, including the auditee’s record in meeting
legislative and other deadlines;
• risk management practices in relation to IT;
• IT system design, development, and maintenance controls;
• compliance with standards, including external standards;
• application controls;
• processing controls, including audit trails;
• business continuity arrangements;
• data integrity, including sampling of data (possibly using computer-assisted
audit techniques);
• access controls and the physical and logical security of networks and
computers, including Internet fi rewalls;
• controls as a safeguard against illegal software;
• performance management and measurement; and

• other issues that arise during the audit.
In making the assessment the auditor may:
• review fi les and other documents relevant to the development and
operation of the IT systems;
• use an appropriate software package to test the central and net-worked
computing system controls;
• test a sample of transactions (potential for using computer-assisted
audit techniques) to validate the systems and relevant controls; and
• interview the Auditor General and key staff members.
Planning
As with any audit, performance auditing in an IT environment needs
to be planned. The planning process should frame audit objectives
with reference to the objectives of the auditee in adopting/introduc-ing
IT systems and should include audit concerns relating to value for
money, controls and security. The planning phase should also identify
the IT systems and their roles in programs, computer systems and
software packages being used by the auditee. During this phase, audi-tors
also need to identify the major potential risks and exposures of IT
systems.
Performance auditing in an IT environment requires specialist skills,
and appropriately trained personnel with skills in IT, information sys-tems,
auditing, and accountancy should be dedicated to the task. The
services of consultants may be needed for the more specialized techni-cal
areas. The SAI may also need to consider acquisition of appropri-ate
hardware and software tools. The SAI staff will require extensive
training to remain abreast of technological developments and IT audit
techniques.
Computer-assisted audit techniques
Auditors are increasingly using computer-assisted audit techniques.
These techniques utilize custom developed software programs to assist
in the execution of the audit. They can be used for both sampling sys-tem
transaction data and for testing the system as a whole. Computer-assisted
audit technique tools can be developed to:

126
• access and extract information from auditee databases;
• total, summarize, sort, compare, and select from large volumes of
data in accordance with specifi ed criteria;
• tabulate, check and perform calculations on the data;
• perform sampling, statistical processing, and analysis;
• provide reports designed to meet particular audit needs;
• facilitate audit planning and control, such as electronic audit work-ing
papers that support effective indexing, review and reporting;
• conduct surveys through web questionnaires, and
• increase the analysis of audit evidence and fi ndings.
Computer-assisted audit techniques can be used to collect data,
validate the processes in the program or to analyze the data. The
auditors should develop these techniques and provide training for the
staff of the auditee. These automated audit tools should be developed/
modifi ed, bearing in mind the IT environment in the auditee and the
audit objectives. Computer-assisted audit techniques can also be uti-lized
in performance audits of both IT and non-IT environments.
Reporting
The performance audit report should be drafted to minimize the use
of technical terminology with a view to making it easily understand-able
to management, members of the legislature and the public. When
the use of technical terms is inescapable, they should be explained.
Audit report should be published on SAI’s websites or on CD in order
to give the report a wide distribution.

Appendix 6
Performance Audits of Activities with
an Environmental Perspective
1. Introduction
Over the past 20 years, global awareness of environmental issues has
grown rapidly, with particular emphasis on matters such as ozone de-pletion,
the destruction of rain forests and global warming. The greatly
increased knowledge and experience of environmental issues acquired
during this period has led to a rethinking of the role and responsibili-ties
of both governments (at national and local level and their associ-ated
agencies) and industries. Some of the crucial changes that have
taken place are:
• The expansion of environmental regulation by state and local au-thorities.
• The increasing cost of environmental protection for both the private
and public sectors. The resources spent by both sectors on pollution
control have increased, and both businesses and government bodies
are looking for more cost-effective ways of dealing with these issues.
• Environmental awareness among fi nancial institutions, both na-tional
and international. The pressure and scrutiny brought to bear
by these institutions provide governments and businesses with the
impetus to give environmental issues closer consideration.
• Following the United Nations Conference on the Environment in
Rio de Janeiro, governments and corporations around the world
have shown more concern about sustainable development.
The increasing concern that organizations which affect the environ-ment
in a negative way should be accountable for their actions has led
to requirements for the consequences of those actions to be reported.
In turn, expectations have grown that the representations made in
environmental reports should be subject to independent audit. As a
result of the implications of these expectations for SAIs, the subject
was taken up by INTOSAI.

128
At the XVth INCOSAI in Cairo 1995, it was decided that, using the
INTOSAI Auditing Standards as a basis, the INTOSAI Working Group
on Environmental Auditing should develop a guide containing guide-lines
and methodologies for the conduct of audits with an environ-mental
perspective. The result of the work, “Guidance on Conducting
Audits of Activities with an Environmental Perspective”, was approved at
the XVIIth INCOSAI in Seoul 2001.82 The purpose of the guide is to
provide SAIs with a basis for understanding the nature of such audit-ing
as it has so far developed in the governmental sphere. This basis is
intended to provide a starting point from which to create an approach
to the satisfactory discharge of environmental auditing responsibilities
within the context of each SAI’s jurisdiction.
The term ‘environmental auditing’ is a convenient label generally
used to describe one of a plethora of activities such as management
audits, product certifi cation, governmental control measures and many
other activities, which bear little or no relation to an external audit.
SAIs also often carry out activities that, by defi nition, do not qualify
as audits, but which contribute to better government. In this appendix
the term “environmental auditing” is used solely in the context of the
independent external audit.
At XV INCOSAI (Cairo), it was agreed that environmental audit-ing
is, in principle, not different from the audit approach as practiced
by SAIs and that it could encompass all types of audit. In this con-text,
audit attention may be devoted to, for example, the disclosure of
environmental assets and liabilities, compliance with legislation and
conventions – both national and international –as well as to measures
instituted by the audited entity to promote economy, effi ciency and ef-fectiveness.
Consequently, it is sometimes easier to talk about perform-ance
auditing with an environmental perspective.
2. Applying INTOSAI Code of Ethics and Auditing Standards
The INTOSAI Code of Ethics and Auditing Standards refl ect a con-sensus
of best practices among SAIs. Thus, it is clear that the standards
codify generally accepted professional practices, which are applied in
carrying out independent external audits and which may also
82 This short presentation is to a large extent based on that guide and other documents and reports
from INTOSAI’s Working Group on Environmental Auditing. The experience of individual SAIs has
also been considered.

encompass the audit of activities with an environmental perspective.
It follows that a SAI should – to the full extent appropriate – take the
INTOSAI Auditing Standards into account when planning, conduct-ing,
and reporting on an environmental audit.
2.1 Basic postulate of relevance and its consequences for performance
auditing with an environmental perspective
The SAI should consider compliance with the INTOSAI Code of Ethics
and auditing standards in all matters that are defi ned as material. Each
SAI should establish a policy to ensure that environmental audits are
of high quality. The SAI should recognize the global nature of environ-mental
matters and promote performance audits with an environmen-tal
perspective.
Government programs or entities that have an impact on the envi-ronment
may be categorized into the following three groups:
• Programs or entities whose operations directly or indirectly affect
the environment.
• Programs or entities whose operations directly or indirectly may
infl uence environmental policy formulation and regulation, whether
internationally, nationally or locally.
• Programs or entities whose tasks involve power to monitor and
control the environmental actions of others.
The government is responsible for determining what information it
needs to ascertain whether its environmental objectives are being real-ized.
In addition, the executive authorities (the government agencies)
and their managers are directly responsible for the correctness and suf-fi
ciency of information on the entity’s impact on the environment, be
it with regard to fi nancial performance, assets or liabilities, compliance
with legislation, or other prescriptions for its performance.
Situations are likely to arise in practice, however, where there is a
lack of legislation providing for the disclosure of relevant environmen-tal
information or where there is a lack of disclosure for some other
reason. In such situations, the SAI should report the shortcoming and
may also have to consider the possible effects on its audit report.
The full scope of government auditing, regularity (fi nancial and
compliance) and performance auditing, also applies to environmental
auditing. Performance auditing of environmental activities may for
instance assess:
• whether environmental programs are properly prepared;

130
• whether environmental policies and programs are conducted in an
economical, effi cient and effective manner;
• whether environmental programs are properly monitored and
evaluated; and
• whether environmental programs are effective.
Environmental auditing adds a special challenge to the expanding
role of auditors and their responsibility for improving and developing
new techniques and methodologies to assess whether the reasonably
and valid performance measures are used by the audited entity.
2.2 General auditing standards of relevance and their consequences
The auditor and the SAI must be, and must be seen to be, independ-ent
and objective in carrying out environmental audits. The auditor
and the SAI must also possess the required competence. The wider
the SAI’s mandate and the more discretionary it is in nature, the more
complex the task becomes of ensuring quality of performance across
the mandate. This applies directly to environmental auditing and may
often be addressed by making use of teams or by obtaining assistance
from experts in the fi eld. If the SAI employs external environmental
experts, it must exercise due care to assure itself of their competence.
SAIs and their auditors and others who carry out environmental
audits should demonstrate adequate knowledge of both environmental
matters and performance auditing/program evaluation.
2.3 Field and reporting standards
The auditor should plan the audit in a manner that ensures an audit
of high quality and collect relevant and reliable information on the
audited program or entity. This should, where applicable, include
relevant environmental information on:
• laws and regulations of relevance;
• goals and objectives set by the parliament or the government;
• the policies, objectives and practices established by the entities to be
audited; and
• the existence of environmental assets and liabilities and any changes
that may have occurred in them during the period under review.
The objective and scope of the environmental audit should be clearly
defi ned. The specifi c needs of environmental auditing may require
additional procedures to be carried out. It may be advisable to make
use of specialists in the SAI to carry out review of the planning and
fi eldwork from an environmental perspective.

3. Developing methods and practices
A SAI may undertake environmental audits under its mandate to carry
out performance audits. For performance audits, there are at least fi ve
different types of environmental focus such as:
• audits of government monitoring of compliance with environmen-tal
laws;
• audits of the performance of government environmental programs;
• audits of the environmental impact of other government programs;
• audits of environmental management systems; and
• evaluations of proposed environmental policies and programs.
Since the SAI may not be able to audit every program or entity
involved, it will need to carefully design a methodology that will allow
it to draw supportable conclusions on how a given function or activity
is implemented nationally. Some of the tools it may consider using are
fi eld visits, questionnaires, and statistical sampling.
3.1 Auditing government monitoring
In many countries, a lead environmental ministry and department (or
other agency of the executive government) is charged with ensuring
that environmental laws and objectives are properly implemented by
public and/or private entities. These laws may charge the environmen-tal
department with such activities as:
• issuing permits that limit the quantity or concentration of pollut-ants
discharged;
• monitoring compliance with such permits;
• monitoring environmental conditions to help identify other poten-tial
non-compliance with regulation;
• helping in the interpretation of regulations, and providing other as-sistance
to regulated entities to assist them in their compliance work;
and
• taking enforcement action when violations occur.
In some cases, the federal (national) government, state, or other
levels of government may delegate these environmental regularity
responsibilities. In addition, other types of executive government
departments, such as transport or agriculture, may also exercise certain
environmental regulatory responsibilities. The SAI is often charged
with examining how well these other departments exercise their envi-ronmental
responsibilities.

132
The data needed to support fi ndings and conclusions may be cen-trally
located and readily available. More usually, important informa-tion
may need to be collected from diverse locations and from numer-ous
governmental and non-governmental entities. A SAI should be
aware that environmental regulatory compliance data has proven to
be particularly susceptible to error in many countries. The absence of
reliable environmental data may itself become a central message of the
SAI’s report.
3.2 Auditing the performance
A government may be enabled by statute or other authority to carry
out (or fund other entities to carry out) a range of other programs or
activities to protect or improve the environment. A program or activity
of this type:
• may be the responsibility of a government ministry, a government
department or one or several government agencies with particular
interest in the environment; or
• may be the responsibility of, for example, a department or a govern-ment
agency for agriculture, through a program for assisting farmers
to adopt practices that minimize pollution.
A SAI may fi nd it useful to identify the international agreements on
environmental matters to which its government has agreed and then
identify which programs have been established to achieve them. Care
should be taken in selecting and scoping an audit of a governmental
environmental program. A SAI may also consider whether to focus its
attention on one main policy instrument or on many different policy
instruments. In selecting a performance audit, the SAI should pay par-ticular
attention to the availability of suffi cient, relevant and reliable
data. When planning its audit, the SAI should examine the risks and
materiality of government programs or activities, taking into account
the resources involved, the importance of the environmental problem
to be addressed, and the magnitude of the intended effect. The audi-tors
should also consider whether there are indications of problems of
effi ciency and effectiveness in the area to be studied.
Consideration of the scope and methodology of the audit should
address the availability of audit criteria, particularly when the program
is not subject to statutory requirements. The SAI may identify ways of
comparing the program’s arrangements to best management practices
or to practices used for similar environmental programs in the same
country or elsewhere. The SAI may also report the program’s achieve-

ments over time, against the program’s own targets, or targets or
benchmarks set by experts.
The auditor should bear in mind that environmental programs may
be aiming for impacts which:
• are individually small-scale but cumulatively large-scale;
• take a long time to have a noticeable effect; and
• are affected by signifi cant external factors (weather conditions, etc.).
3.3 Auditing the environmental impacts of other government programs
In addition to programs whose principal aim is to protect or improve
the environment, all activities affect the environment in one way or
another through their use of resources or consequences of their other
actions. Some government programs have signifi cant effects, which
may be positive or negative, intended or unintended.
The environmental effects of the activities can be highlighted as part
of a wide-ranging performance audit – of the economy effi ciency, and
effectiveness of a government activity, or as a narrowly defi ned study
focusing solely on the environmental impacts.
The SAI may review the adequacy of:
• the description of the program or activity, its environment, and the
baseline conditions;
• the completeness of the range of key impacts identifi ed;
• the data used to assess the likelihood of the impacts and their ex-pected
scale; and
• any proposals for measures to counter the impacts.
The SAI may wish to test for itself what impacts a government
activity may have on the environment, its likely scale, and any values
that can be placed on its costs and benefi ts. Discussions with experts
and searches of literature can identify commonly used evaluation
methodologies. It may also be helpful to identify and seek the views of
key stakeholders (e.g. residents in the area affected by the activity, key
environmental interest groups, and nongovernmental organizations
in the fi eld) as well as academics specializing in relevant evaluation
methodologies.
From the outset, the government may identify measures that coun-ter
or reduce environmental impacts. The SAI’s audit may address
whether these measures:

134
• have been put in place and are in accordance with leading practices
or best available information or technology and not entailing exces-sive
cost; and
• have had the preventive effect intended, and, if not, what actions the
government has taken instead.
In some cases, the counter measure may need to be suitable for
preventing or dealing with low-risk, major-impact occurrences, such
as unintended releases of radioactive substances. Accident and incident
procedures may be rarely used, but they need to be operable in case of
need. When such procedures are important, an audit may review:
• the procedures;
• the training of any staff involved;
• the frequency of testing the procedures; and
• Whether any arrangements required with third parties (suppliers,
emergency services, etc.) are up-to-date.
3.4 Auditing environmental management systems
Organizations are introducing environmental management systems
to ensure that they are systematically setting policies for continual im-provement
in environmental performance and are achieving the policy
objectives effectively. In considering whether to undertake an audit
of environmental management systems or not, a SAI should identify
existing government policy on establishing them. The SAI may decide
to audit complete environmental management systems for individual
government departments. Alternatively, the audit may focus on one
or more elements across a range of departments, agencies, or other
organizations within the SAI’s remit. The latter approach can be help-ful
for dealing with relatively small-scale matters and nonetheless offer
scope for signifi cant improvements across the government.
A SAI may consider whether it should audit and report on an actual
performance target set by the government. For an audit of this type,
the SAI could usefully investigate how the government compares with
practices elsewhere and with the government’s commitments and
practices to international agreements. A SAI may also consider whether
government monitoring of departments’ environmental management
systems and reporting of environmental performance makes them suf-fi
ciently accountable to the legislature and the public for meeting key
performance targets. The SAI could undertake an audit to identify the
level of performance and reasons for non-achievement of targets.

3.5 Evaluating proposed policies and programs
Some SAIs may be called upon to provide analyses or information on
proposed policies or programs to their legislatures. This may occur, for
example, when a national legislature refocuses its attention from the
question, ‘Is the program operating well in conformance with its statu-tory
requirements?’ to the more basic question, ‘Do the underlying
statutory requirements themselves need modifi cation to make them
more cost-effective or to improve them in other ways?’ Under these
circumstances, it may request the SAI to analyze alternative proposals
being considered.
Generally, such work poses both challenges and risks. In particular,
analyses of proposed policies or programs may sometimes require
special skills (specially-trained staff, consultants, expert panels, etc.).
Even with these added skills, the nature of such analyses involves risks
for the SAI, particularly if it is viewed as taking sides in debates over
matters of policy. The SAI may consider the following alternatives to
minimize such a risk:
• Provide factual information rather than judgments.
• Identify consensus among experts.
• Evaluate and comment on analysis of other organizations.
• Decline the request.
4. Establishing audit criteria
One concern for SAIs in carrying out environmental audits is deter-mining
the criteria against which the audited program or entities will
be assessed. It is important to ensure that the chosen criteria – if used
– will be generally accepted as relevant, complete and understandable.
A performance audit may need to be based on criteria from, in a
formal sense, both ‘authoritative’ sources (based on laws, documented
policies and goals, generally accepted standards, etc.) and ‘non-author-itative’
sources (academic literature, experts, indicators, or measures
used by similar entities or other entities engaged in similar activities,
etc.). See appendix 2.
The special risk that a SAI faces in conducting an environmental
audit is that the criteria it has used may be inapplicable, inappropriate
or biased. The greatest risk for the SAI will come from using non-authoritative
sources of criteria. On the other hand, such sources are
often both necessary and fruitful when deeper or broader analysis is

136
required. The ultimate test of the chosen criteria is that (like all audit
criteria), they are objective rather than subjective. The criteria should
also ensure the completeness of the performance indicators used (and
the performance indicators should be relevant, understandable and
reliable).
In the problem-oriented approach, however, the use of audit criteria
will play a less important role. Instead, the formulation of well-found-ed
hypothesis is of great importance.83
83 See for instance: Handbook in Performance Auditing. Theory and practice. 1999.

Appendix 7
Towards a system-oriented approach in performance
auditing: A theoretical framework
1. The system-oriented approach of effectiveness auditing84
Public services are complex, and the growing complexity of govern-ment
programs increases the incidence of confl icting goals and unin-tentional
side effects, caused by overlapping or coinciding functions.
Further, most central decisions on public programs must be made
without being completely sure that they will attain the stated objec-tives,
at least not at the fi rst attempt.
It is not an easy task to turn political goals into government pro-grams
and to design and implement measures that will obtain desired
results. It may even prove diffi cult to fi nd suitable methods to assess
the results and the effectiveness of government interventions. There
is a growing insight that there often is a large gap between what has
been decided politically and what will later be implemented and fi nally
achieved.
It is therefore essential to develop models to help performance audi-tors
in their efforts to analyze and evaluate the implementation and
effectiveness of government interventions. One common approach in
performance auditing is the so-called goal-means (aims and methods)
model. In this model implementation is seen as a process where gen-eral
goals are broken down into sub-goals or means by ministries and
executive authorities; authorities later being held accountable for how
they manage and produce the services.
The system-oriented approach to effectiveness auditing, presented
below, has it roots in the goal-means model. It is also based on ideas
and concepts from ‘system theory’, where government undertakings
or programs are seen as systems of interacting and functional inter-dependent
elements. Regulations, resources, government bodies etc.
84 For more information, see Towards A System-Oriented Approach In Performance Auditing: A Theoreti-cal
Framework. National Audit Offi ce of Sweden, RRV, 1985.

138
are all examples of elements that constitute a system of this type of a
government undertaking.
Defi ning a government undertaking as a system means that the
performance auditors have to apply a holistic perspective. In auditing
effectiveness of such systems, less emphasis is put on the accountability
of individual agencies. The focus is on the effectiveness of the systems
themselves.
2. The system-model and government undertakings
The system-model is presented below, step by step.
2.1 Production – the first component in the model
Production is the core of the system-model. In all production of
services, production and consumption occur simultaneously. Conse-quently,
clients, staff, working methods and resources are all part of the
production system. This is shown in the fi gure below.

The aim of production is to deliver services to society effi ciently.
This might, for instance: require motivated and competent staff;
rational use of resources; good practices; participating and well-in-formed
clients etc.

2.2 Administration – the second component in the model
The operation of the production system takes place within the frame-work
of an administration, involving perhaps several ministries and
many departments and agencies, as well as regional offi ces and local
authorities.

Implementation for performance auditing

More Related Content

What's hot (20)

Similar to Implementation for performance auditing (20)

More from Golden Saragih (9)

Recently uploaded (20)

Implementation for performance auditing