Impact evaluation of Energy Efficiency and DSM programmes
1 like1,053 views
The document outlines an evaluation framework consisting of seven key analytic elements essential for conducting evaluations of public interventions and policy measures. It discusses the importance of defining policy measure theories, establishing relevant indicators, creating baselines, and assessing outputs, outcomes, and impacts related to energy efficiency programs. Furthermore, it emphasizes cost assessments and the levels of evaluation effort necessary for effective analysis and recommendations in energy savings calculations.
Impact evaluation of Energy Efficiency and DSM programmes
- 1. Evaluation Guidance& The IEA DSM Evaluation Guidebook Harry Vreuls Harry.vreuls@rvo.nl
- 2. Overview •Evaluation framework •7 key analytic elements –With experiences •Recent developments
- 3. Evaluation framework •Input TIME •Output •Outcome/results EXTERNALITIES •Impact
- 4. Evaluation framework for a normative evaluation Objectives Outputs Relevance Inputs Efficiency Utility & Sustainability Effectiveness Society Economy Environment Public Intervention Policy Measure / Programme / Project Evaluation Outcome/ Results Impacts Needs, Problems Issues Evaluation Questions Relevance: To what extent are the objectives justified in relation to needs? Effectiveness: To what extents have the expected objectives been achieved? Efficiency: Have the objectives been achieved at lowest cost? Utility & Sustainability: Do the expected or unexpected effects contribute to a net increase in social welfare and sustainability? Adapted from European Commission 1999 and Technolopolis France 2001
- 5. Key analytic elements •What should be addressed in an evaluation? •How to ensure that relevant topics are considered? •Seven key elements
- 6. Seven Key Elements 1.Policy measure theory 2.Indicatorsshowing the success 3.Baselinesfor selected indicators. 4.Assessment of outputs and outcomes 5.Assessment of energy savings, emissions reductions and other relevant impacts 6.Calculation of cost, cost-efficiency and cost- effectiveness 7.Choice of level for evaluation efforts
- 7. Provides the basic framework for the evaluation A. Specification of Policy Measure Domain –End-user market segment –Supply-side market segment –Participation status –Location B. Effects Hypotheses –Qualitative and quantitative effects –Timeframe over which effects are expected Key Element 1: Statement of Policy Measure Theory
- 8. Key Element 1: Statement of Policy Measure Theory
- 9. Type of Policy Measure (Example) Domain Specification (Target Group Examples) Effects Hypothesis (Examples) RegulationBuilding Code Enforcement Builders Builders increase frequency of using code- prescribed building methods. Building codes for new building also influence the existing buildings through retrofit Information ProgrammesLabelling Programme Purchasers of products subject to labelling Manufacturers and retailers of these products Awareness of energy-efficient equipment increases among targeted consumers. Manufacturers increase share of efficient models in catalogues. Residential Energy Audits Programme Homeowners in programme area Participants increase level of awareness and knowledge of efficiency opportunities in their homes. Participants implement targeted improvements more frequently than non-participants. Economic IncentivesEquipment Rebate Programme Purchasers of equipment covered by programme Vendors and installers Market share of efficient models increases. Proportion of vendors and installers promoting supported equipment increases. Voluntary AgreementsIndustrial Programme Owners of facilities in sectors covered by agreements Owners and managers increase awareness of efficiency opportunities. Owners increase adoption rate for efficient production technologies. Elements of Programme Theory by Type of Policy Measure
- 10. Key Element 2: Specification of Indicators •Inputs (resources) •Outputs: under the control of the management •Outcomes or results: initial up to long term elements •Impacts: tend to be longer-term elements •Cost indicatorslevel
- 11. Example from Case Red Hot (DK) Stated evaluation objectives What is the impact on energy saving behaviour of families of children involved? Produced output Assessed outcome indicators Energy impact Teaching material Prize competition No. of children/classes taught No. who browsed through material No. who read material No. who had discussions with family No. who remember seeing TV- campaign No. who changed energy saving awareness No. who turn off stand-by (Not assessed)
- 12. Key Element 3: Development of Baselines What would market actors who participated in (or who were exposed to) the programme have done in the absence of the programme? A range of baseline types: -Static -Dynamic
- 13. Key Element 3: Development of Baselines Type of Policy Measure/ Example Programme Baseline Development Strategies RegulationBuilding Code Enforcement Building code provisions covering the targeted building components and end uses. Energy AuditsResidential Programme Non-participants’ adoption of measures supported by the audits, properly adjusted for differences between the participant and non- participant groups. Labelling Programme •Market share of qualifying equipment in areas not covered by the labelling programme. •Historical trends in percentage of qualifying models sold by manufacturers and/or displayed by retailers. Economic IncentivesEquipment Rebate Programme •Market share of qualifying equipment in areas not exposed to rebate programmes, •Non-participants’ level of adoption of targeted technologies or end-use consumption, with appropriate statistical controls.
- 14. Key Element 4: Assessment of Output and Outcome TypeofPolicyMeasure /Example ExampleOutputIndicators RegulationBuilding Code Enforc. Number of residences inspected and certified. Information Energy Audits Resident. •Number of audits. •Number of courses for energy auditors. Labeling Program •% of equipment that contains a label. •% of qualifying models displayed with appropriate labels. Econ. IncentivesEquipment Rebate •% of eligible facilities that participate •Market share of qualifying products. Volunt. AgreementsIndustry •% facilities in the sector that sign the agreement. •%signatories that comply with the agreement.
- 15. Free-of-Charge Elec. Audits (DK) Stated evaluation objectives •What types of advice are implemented? •What is the lifetime of the implemented advice? Produced output Assessed outcomeindicators •Audits reports •Audit concepts •No. of audits offered •No. of audits •No. of implemented advice by type •Lifetime of implemented advice •Customer satisfaction with audits and auditors
- 16. Key Element 5: Assessment Impacts Benefits TypicalItemsRequired Energy •Net program energy savings •Gross program energy savings Environmental •Volume of emissions reductions •Unit value of emissions reductions Non-Energy •Volume of water and other non-fuel resource savings •Unit value of non-fuel resource savings •Non-energy benefits: increased productivity, increased safety, and accelerated collections
- 17. Key Element 5: Energy Savings Methods Method TypicalPolicyMeasures Engineering •Econ.incentives:tax-relatedmeasuresandrebates •Informationonprograms:labeling •EnergyAudits Engineering with building simulation modelling •Regulation:buildingcodes •Econ.incentives:tax-relatedmeasuresrebates •Informationprograms:labeling Engineering with monitoring •Economic Incentives: rebates •Energy Audits •Voluntary Agreements •Regulation: building codes and equipment standards BillAnalysis •EconomicIncentives:rebates •VoluntaryAgreements End-UseMetering •EconomicIncentives:rebates •VoluntaryAgreements
- 18. Key Element 5: Energy SavingsIndustrial Energy Audits (S-Korea) Classification Replace Transformer Efficient Motors Replacement Refrigerators Lighting Others Total Saving (MWh/Year) 549.5 2,077.6 1,416.8 509.7 6,183.9 10,737.5 Saving (million won) 52.7 215.8 121.5 42.5 1,006.2 1,438.7 Investment (million won) 353.9 768.0 690.0 207.6 2,061.0 4,080.5 Recovery Period (Year) 6.7 3.6 5.7 4.9 - 2.8
- 19. Seven Key Elements 1.Policy measure theory 2.Indicatorsshowing the success 3.Baselinesfor selected indicators. 4.Assessment of outputs and outcomes 5.Assessment of energy savings, emissions reductions and other relevant impacts 6.Calculation of cost, cost-efficiency and cost- effectiveness 7.Choice of level for evaluation efforts
- 20. Key Element 6: Assessment of Costs, Cost- Efficiency and Cost-effectiveness Costs TypicalItemsRequired Comments Admini- strative •Administrative personnel and overhead costs •Outsourced program administration •Marketing and promotional costs •Measurement and evaluation costs In most social cost- effectiveness schemes, incentives paid directly to customers or vendors are identified as transfer payments (not as costs) Incre- mental Incremental costs of measures implemented as a result of the programme. Cost estimates for both the energy-efficient measure and its baseline alternative. Others Measure-specific items, such as costs to properly dispose of used fluorescent ballasts and lamps or downtime for installation. Judgement whether the probable magnitude of these costs is sufficiently high to justify measurement expenses.
- 21. Key Element 7: Level of Evaluation Effort •Level A: Comprehensive evaluation: –outcome indicators including net behavioural change –impact indicators on energy savings –additional internal and external information sources are needed •Level B: Targeted evaluation: –including outcome indicators as gross behavioural change –some additional information sources •Level C: Programme review evaluation: –focus on input and output indicators, –only use existing (written) information sources.
- 22. Level of Evaluation Effort, Cases in Volume 2 Level Case Country B/C Building codes B B Energy Efficiency Regulations for Residential Equipment CA B Energy management scheme for large buildings DK C Minimum energy performance standards Korea A/B Energy Performance Standard (EPS) for houses NL C Local energy efficiency information centres B B Energuide for houses CA B Energy labelling of small buildings DK A Free-of-charge electricity audit DK A Project ‘Red-Hot’ (element of stand-by campaign) DK
- 23. Developments related to Key Element 5:Assessment Impacts; Harmonisation of energy savings calculations •More harmonisation •Starting of standardisation
- 24. Ongoing work •USA •Uniform Methods Project •regional initiatives e.g. NEEP and SEE Action •European Commission: Energy Efficiency Directive (EED) & National Energy Efficiency Action Plans (NEEAPs) •IEA DSM Task 21: harmonisation of energy savings calculations •International standardisation •European standardisation: CEN •International standardisation: ISO
- 25. Standarisation •CEN and ISO are in the process of creating generalstandards on energy savings and energy savings calculations and holding more general guidance •Publication of the CEN standard EN16212:2012; no foreseen work in the short time •ISO is working on different areas to provide very general standards including one on ESC (for actions/projects)
- 26. Energy savings calculation, key elements in practise •In almost all reports and studiesresearched in Task 21 several to all of the key elements in energy savings calculation are present •Baselines: the most critical common element •Unitary savings: still often a ‘new’ start point for savings •Saving life-time: not a major topic, but when used, often treated in different ways
- 27. IEA DSM Agreement: Evaluation Guidebook Volume 1: Guidebook •7 key analytic elements (chapter 1) •5 types of policy measures (chapter 2-6) •Conclusions and recommendations (chapter 7) Volume 2: Country reports with case examples: Sweden, Netherlands, Korea, Italy, France, Denmark, Canada and Belgium www.ieadsm.org
- 28. Thank you The 7 key analytic elements: 1.Statement of policy measure theory 2.Specification of indicators for evaluation 3.Development of baselines for indicators 4.Assessment of output and outcome 5.Assessment of energy savings and emissions reductions and other relevant impacts 6.Calculation of costs, cost-efficiency and cost-effectiveness. 7.Choice of level (evaluation efforts)