Day2 session3 multi-criteria-analysis

METHODS FOR APPRAISAL OF
ENVIRONMENTAL INVESTMENTS –
MULTI-CRITERIA ANALYSIS
(QUANTIFIABLE AND NON-
QUANTIFIABLE); SCORING SYSTEMS
David Toft
SST-CONSULT /Kommunalkredit Public Consulting
Promoting Clean Urban Public Transport in Kazakhstan:
Designing a Green Investment Programme
14 December 2016, Astana

• Technique to assist decision-making that takes into
account more than one decision criterion
– for example, financial analysis considers IRR and NPV, CBA
considers net benefits and ENPV
• Key elements:
– set of policy objectives
– set of policy options (programmes or projects)
– measureable criteria upon which policy options are assessed for
their compliance with policy objectives
Multi-Criteria Analysis (MCA)

• MCA establishes preferences between options by
reference to an explicit set of objectives for which
measurable criteria have been established to assess
the extent to which the objectives have been
achieved
• Enables aggregation of data on individual criteria
to provide indicators of the overall performance of
policy options

• MCA has advantages over informal judgment
unsupported by analysis in that:
– open and explicit
– choice of objectives and criteria are open to analysis and change
if deemed inappropriate
– scores and weights, when used, are explicit and developed
according to established techniques
– performance measurement can be sub-contracted to experts
– can provide an important means of communication, within the
decision making body and sometimes, later, between that body
and the wider community
– when scores and weights are used, it provides an audit trail
after MCA: A Manual, published by the Department for Communities and Local Government
at the City of London

• Criticisms of MCA include that it is not a tool to assist
decision-making, but to make decisions based on the
biases, preferences, and moral judgements of the
decision-makers themselves
after MCA: A Manual, published by the Department for Communities and Local Government at the City of London

• Establish the decision context. What are the aims of the
MCA, and who are the decision makers and other key
players?
• Identify the options
• Identify objectives and criteria that reflect the value
associated with the consequences of each option
• Describe the expected performance of each option
against the criteria
• Weighting. Assign weights for each of the criteria to reflect
their relative importance to the decision
• Combine the weights and scores for each of the options
to derive an overall value
• Examine the results
• Conduct a sensitivity analysis of the results to changes
in scores or weights
Steps in an MCA

Scoring, Weighting, Ranking
Procedure
Setting
criteria
Setting
scores
Weighting Ranking
- Identifying all
relevant criteria
- Group the criteria
- Check
completeness
- Redundancy
- Mutual
independence
- Excessive number
of criteria
- Setting numerical
scales for
evaluating criteria
- Assigning values
on interval scale
(e.g. 0 – 10 points)
- Common
direction: better
performance,
higher number of
scores
- Define relative
importance of the
criteria
- Swing-weighting
- Combining
weights and scores
- Project ranking

Scoring criteria
Environmental Project size Type
Unit efficiency:
(PM2.5 emissions
from old buses –
PM2.5 emissions
from new buses
(in kg)) / total
project costs
CNG buses
Project
Preparation
Business
plan
Between
100 and
200
replace
ments
> 200
replace
ments
Proposed system
Traffic lights
with priority
for public
transport
New bus
stops with
online
information
New bus
stops
Length
of new
bus
lanes
Other
< 100
replace
ments
LPG buses
Polluted city
Location
City centre
City centre and
outskirts
City and
connecting rural
areas
Diesel buses

0 = the lowest
preference
10 = the highest
preference
0 10
Scoring scale
Direction and
interdependence
Distance from city
centre (km)
Scores
Number of bus
replacements
Scores
Scores
0
10 10
0
10
0
Unit efficiency for
emission reduction –
preference for large
scale projects

Why weights matter
• Consider two cases regarding the criteria: “project size" and
“technical” and project “location”:
1. The selection of projects depends on the choice of technology
(1 = modern diesel; 10 = CNG)
2. Projects with other technologies are excluded in pre-appraisal
stage
• Assume that the weight of technical criteria in case 1 amounts to
0.7 (on the scale (0-1))
• In case 2, the difference between the lowest and the highest risk is
not so big as in case 1 (as the most risky projects are excluded
from evaluation)
• Thus, in case 2 the risk factor may be considered not as important
as in case 1. The relative importance of criterion B compared to
other criteria decreases

Scoring Table
C. Project
location
Sum of
scores
Number of
replacements
Unit efficiency
of
environmental
effect
Project
preparation
Project type
(technology
of new
buses)
Other
system
improvem
ents
City centre,
city, or rural
areas
Project 1 10 5 1 5 0 5 26
Project 2 10 5 1 10 2 5 33
Project 3 5 10 1 1 2 3 22
Project 4 1 0 1 10 2 0 14
A. Project size B. Technical

Assigning Weights – Case 1
C. Project
location
Sum of
weighted
scores
Weight 0.1
Number of
replacements
Unit efficiency
of
environmental
effect
Project
preparation
Project type
(technology
of new
buses)
Other
system
improvem
ents
City centre,
city, or rural
areas
Project 1 10 5 1 5 0 5 7.2
Project 2 10 5 1 10 2 5 12.1
Project 3 5 10 1 1 2 3 5.6
Project 4 1 0 1 10 2 0 8.8
0.2 0.7
Scale for weights: 0 – 1 (where 1 means
exclusive preference to the criterion)

Assigning Weights – Case 2
C. Project
location
Sum of
weighted
scores
Weight 0.1
Number of
replacements
Unit
efficiency
of
environme
ntal effect
Project
preparation
Project type
(technology
of new
buses)
Other
system
improvem
ents
City centre,
city, or rural
areas
Project 1 10 5 1 5 0 5 10.5
Project 2 10 5 1 10 2 5 13.3
Project 3 5 10 1 1 2 3 9.5
Project 4 1 0 1 10 2 0 5.8
0.5 0.4
Scale for weights: 0 – 1 (where 1 means
exclusive preference to the criterion)

Examples: Technical Appraisal Table
for Wastewater Management Project
Criteria Max. points Sum of
points
P1 Level of preparation; proposed solution Weight 0.15
P1.1 Project technological and management concepts agreed (see Feasibility Study) 1
P1.2 Project results in compliance with current regulations in force 1
P1.3 Receiver of waste and sludge has been identified and confirmed 1
P1.4 Environmental Impact Assessment complete 1
P1.5 Public tender for project implementation complete 1
P1.6 Decision on the conditions for construction and management of land plot 1
P1.7 Construction design complete 1.5
P1.8 Permits: water and construction permits obtained 1.5
Together P1 (sum of points multiplied by weight)

Criteria Max. points Sum of
points
P2 Significance of the effects, modernity of project Weight 0.35
P2.1 Protection of water intakes, located downstream from the wastewater treatment
plant (WWTP) outfall and further than 10 km from the WWTP
1
P2.2 Protection of water in recreational and water sport areas 1
P2.3 Protection of water in nature reserves 1
P2.4 Protection of water as the 3rd or 4th purity class 1
P2.5 Technology 3
Together P2 (sum of points multiplied by the weight)
P3 Significance of effects – protection of human health and environment Weight 0.5
P3.1 Protection of water intakes located downstream from the WWTP outfall and closer
than 10 km (1 point) or closer than 5 km (2 points) or closer than 3 km (3 points)
3
P3.2 Receiving water protection body, classified as 1st purity class (2 points) or 2nd
purity class (1 point)
2
P3.3 Sewage system development in built-up areas 1
P3.4 Final destination of sludge has been determined 1
Together P3 (sum of points multiplied by weight)
P Total sum of points  P

Scoring, Weighting and Ranking System
Used for Evaluation of ERDF Investments
in Poland
• European Regional Development Fund (ERDF) in financing local infrastructure
projects
• Importance of programming documents – Integrated Operational Programme for
Regional Development (IOPRD)
• Various schemes of intervention under the IOPRD:
– Transport systems (road modernisation, public transport)
– Environmental infrastructure
– Social infrastructure
– Tourism and culture, etc.
• Project selection process based on periodic calls for proposals for different
schemes
• Unified application forms
• Each project evaluated by three external experts
• Ranking lists created for each project scheme (e.g. environmental infrastructure)
• Some criticisms concerning the diminishing role of independent experts in project
evaluation

Criterion
Weight
Max.No
scores
Referenceto
the
application
form
Scores
awarded
Justification
I. Project impact on implementing
EU environmental directives
3 12 D6/SW
II. Project complementarity with
other projects in particular with
ERDF financed projects or
Cohesion Fund projects
1 4 D7/SW
III. Project sustainability and
institutional preparation (ensuring
financial and institutional
sustainability)
2 8 D10/SW
IV. Cost-effectiveness 4 16 SW
Examples of the Scoring, Weighing and
Ranking of ERDF Investments In Poland
[1] a) National Programme for Municipal Wastewater Treatment:
From 10 000 to 20 000 Population equivalent (P.E.) – 4 points
From 5 000 to 10 000 P.E. – 3 points
From 2 000 to 5 000 P.E. – 2 points
Under 2 000 P.E. – 1 point
b) Waste management plans:
From 10 000 to 20 000 inhabitants – 4 points
Under 2 000 inhabitants – 1 point
c) Air protection investments within the areas covered by Air Protection
Programmes – 4 points
d) Renewable Energy
Source capacity:
- Over 10 MW – 4 points
- From 5 to 10 MW – 3 points
- From 1 to 5 MW – 2 points
- Under 1 MW – 1 point

Criterion
Weight
Max.No
scores
Referenceto
the
application
form
Scores
awarded
Justification
V. Correctness of the indicators
presented
1 4 E2/SW
VI. Technical viability 1 4 SW
VII. Complex projects carried out by
more than one local government
units
2 8 D6/D9
Maximum number of scores 56
Name of the expert evaluating
Date
Signature

Day2 session3 multi-criteria-analysis

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Day2 session3 multi-criteria-analysis (20)

More from OECD Environment (20)

Recently uploaded (20)

Day2 session3 multi-criteria-analysis