iFRAME

iFRAME
Neil Reeder, Clara Péron and Karl H Richter
- impact Framework for Real And Meaningful Evaluation -
Deriving a framework to assess the impacts generated by
ICT-enabled social innovation initiatives
promoting social investment for integrated approaches to the
provision of personal social services
Brussels - 25th February 2015

Structure of presentation
Key components to be included in framework
Recognized measurement ‘meta-methodologies’
Roles in impact measurement and implications
Choices of measurement methodologies
Existing good practice frameworks for key components
2
1
2
3
4
5
K

K
Scene setting
3
From interesting objects in the landscape … to an impactful image
5 August 1962 - 29°28'06.1"S 30°10'15.7"E
Time and place of Nelson Mandela’s capture by police
Be alert to different perspectives and levels

Components to elaborate a framework
ICT
(Enabler)
PSSGIs
(Personal Social
Services of General
Interest)
Social
innovation
(New approaches)
Social
investment
(Context &
drivers)
PSSGIs:
1. Education and training
2. Employability and employment
3. Social assistance
4. Social care and childcare
5. Social inclusion
4
1 Key components to be included in framework
Impact
assessment
(Non-financial
effects)
3 x levels of impact
4 x perspectives of impact
(Could also mean private capital via debt or
equity financial product; 3rd sector grant by
NGO, charity or foundation; social capital via
volunteering & the sharing economy)
Social investment:
• Public sector spend on
infrastructure or social programmes
• Context: Social Investment Package
K

There is a wide recognition of the key stages
involved in impact assessment
First three stages of impact assessment are:
5
2 Recognized measurement ‘meta-methodologies’
Set goals
Articulate the
desired impact
Develop
framework
Select metrics
Collect and
store data
Undertake processes
(EU GECES sub-group, G8 Social Impact Investment Taskforce, EVPA, SROI Network, others)
K
Analysis
Theory of Change augmented
by pattern recognition
No Rating systems
Is a statistical
approach used?
Yes SROINo
Stakeholder
views on
attribution
Cost Benefit Analysis
Multi-Criteria
Decision Analysis
Randomised
evaluations
Before and
after study
Propensity
scoring
Yes
Approaches Techniques/
Methodologies
Is an effort made
to assess change
post-intervention?

6
The logic map or theory of change guides most
impact measurement frameworks
• Emerging consensus advocates a logic map basis
(also known as theory of change/ theory of value creation)
Resources that are
deployed
Actions performed in
support of objectives
Tangible products and
services
Changes, or effects on
individuals
Changes, or effects on
society
• Past emphasis has been on inputs, activities and outputs
• Best practice focuses on outcomes and impact (or value added)
2 Recognized measurement ‘meta-methodologies’ N
• This approach is advocated numerous bodies
e.g. the GIIN, The SROI Network, G8 & GECES

Impact creator
• Sector
(e.g. Health, Retailing, Utility)
• Organizational type
(public sector, public company,
social enterprise, co-operative,
charity)
Funder
• Government commissioner
• Social investor
• Venture philanthropist
• Private investor
7
Funder
Impact
Creator
Impact
Beneficiary
Recipients
3 Roles in impact measurement and implications
There are key roles within the social investment
value chain (1/4)
Impact beneficiary & recipients
(of goods & services)
• Client (direct effects) /
People with priority needs
• Intermediary
• Staff
• Environment
• Local community
• Government agency (especially
welfare or criminal justice)
• Market rate charge
• Subsidised charge
• Free at point of use
Source: drawn from LSE working paper (2013) Reeder and Colantonio
N

value chain (2/4) – Micro level
Funder
(e.g. public sector commissioner,
foundation or investor)
provides resources to impact
creator & interprets the impact
for beneficiaries
Impact creator
(business, social enterprise,
public sector)
creates impact from
available resources
Impact beneficiaries
(client, workforce, community, etc.)
gain better outcomes
Impact
generation
2. Interpretation of impact
• apportioned pro-rata based
on proportion of funding
• implicit in logic map
• explicit in SROI
Recipients
(can be client of impact creators or other
relationship, can be beneficiary but not always)
receive goods or services
from impact creator
Revenue / resources
(goods & services)
8
Funding
- grant or subsidy
- repayable capital
- conditional payment
1. Interpretation of impact on impact creator
(e.g. improved organisational capacity, cashflow etc)
3. Financial return
(If applicable)
N

value chain (3/4) – Meso & Macro Levels
Wider group of
funders
provide resources to impact
creators & interpret the impact
for beneficiaries
Wider group of
impact creators
create impact from
available resources
Impact beneficiaries
gain better outcomes
Impact
generation2. Interpretation of impact
Wider group of recipients
receive goods or services
from impact creators
Revenue / resources
(goods & services)
9
Funding
1. Interpretation of impact
3. Financial return
N

10
value chain (4/4)
• Working data model - showing relationship between datasets
EngagedX Investment STandards (EXIST)
Source: from ‘Social Impact Investment: Building the Evidence Base’ published by the OECD Feb 2015 for the G8)
N

Choice of measurement technique/ methodology:
influences, considerations and perspectives
11
4 Perspectives on choices of measurement methodologies
Analysis
Theory of Change augmented
by pattern recognition
No Rating systems
Is a statistical
approach used?
Yes SROINo
Stakeholder
views on
attribution
Cost Benefit Analysis
Multi-Criteria
Decision Analysis
Randomised
evaluations
Before and
after study
Propensity
scoring
Yes
Approaches
Techniques/
Methodologies
• Techniques / methodologies vary
according to mind-sets of evaluators:
• Technocratic < > participative
• Integrative < > disparate
• No ‘one-size fits all’
Is an effort made
to assess change
post-intervention?
N

Key Principles of measurement put forth by GECES
• Avoid scope for “gaming” or perverse incentives:
with any measurement there are those that will try to play the system to enable
targets to be shown to be achieved
• Outputs should not be treated as outcomes or impacts:
outputs are not the objective in themselves, but a means to achieving that
• Inflexibility is problematic:
Any measurement system must embrace the need to change and improve
• Blend quantification with qualitative understanding:
The story of the changing of lives, attitudes and outlooks must always come first
• Important to maintain proportionality:
the effort and degree of accuracy achieved needs to be proportional to the usage
of that measurement
Source: GECES (2014)
4 Perspectives on choices of measurement methodologies
12
N

Good practice frameworks - Social impact generally (1/3)
• Key aspects:
• What – the activity being undertaken
• Whom – multiple stakeholders/ beneficiaries
• Why – the outcome area / social concern being targeted
• Which effect – contrasting with a counterfactual
• Key dimensions of value
• Segmentation by sector of value creation
• Impact varies over time and with location
• Across micro/ meso/ macro levels
• Incorporating direct and indirect effects
• Influenced by confidence in the evidence
13
5 Existing good practice frameworks for key components N

Good practice frameworks - ICT (2/3)
ICT – 7 principle roles
• Computation
• Communication
• Co-ordination
• Recollection
• Automation
• Evaluation
• Design
Source: Denning (2012)
14
Source: Carretero et al (2012)
Communication
- Online and telephone-based information and advice
- Link virtually with others for emotional or practical support
- Access to content and services in mother tongue
- Enable online communities and informal exchanges
Co-ordination
- Sustain coordination & cooperation with those involved in care
- Facilitate participation of multiple actors in caring such as volunteers
Recollection
- Self-training to acquire skills in managing long-term diseases
Automation
- Help carers to make care arrangement online
for example …
Metrics might include:
• Do you feel well informed?
• How often do you speak to family and friends?
• How easy is it to obtain support?
5 Existing good practice frameworks for key components N

Good practice frameworks - Social innovation (3/3)
Diffusion, transferability or scalability is a fundamental aim of
social innovation, but raises many issues:
• It’s effects can be direct or indirect
• Measurement needs to take a flexible and qualitative approach
• The measurement approach needs to reveal complex causal relationships
i.e it is a nonlinear process
• One note of caution – it is not easy to capture data on social innovation
because it cascades, it moves back, it cross-pollinates, it jumps stages …
15
5 Existing good practice frameworks for key components
Source: Kessler and Sen (2013); van Hemelrjick (2010); Lane et al (2009)
N

ICT
(Enabler)
PSSGIs
(Personal Social
Services of General
Interest)
Social
innovation
(New approaches)
Social
investment
(Context &
drivers)
Next steps: discussion points & working groups
• We have an emerging framework:
• Needs refinement
• Needs testing
• Need to resolve detail
• … it is not the final word on the subject
16
C
Impact
assessment
(Non-financial
effects)
• Key questions for this afternoon’s workshops:
• How to evaluate from different perspectives?
(ICT/ social innovation/ social investment/ PSSGIs)
• How to evaluate across different levels?
(micro/ meso/ macro)
• What is the best methodological approach?

iFRAME

More Related Content

What's hot (19)

Viewers also liked (12)

Similar to iFRAME (20)

Recently uploaded (20)

iFRAME

Editor's Notes