SlideShare a Scribd company logo
Methods for measuring
citizen-science impact
online
SwafS citizen-science cluster meeting on impact
10 November 2021
Uta Wehn, IHE Delft
Luigi Ceccaroni, Earthwatch
2
• MICS framework development for impact assessment
• Co-design of impact journeys in specific case-study sites
• General platform to measure impact
Content
Framework development
3
Literature
review
Initial
framework
Progress the state of the art
Increase accessibility
Set of qu
Targeted lit
review
Measuring Impact of Citizen Science
mics.tools
Wehn, U., Gharesifard, M., Ceccaroni, L. et al. (2021)
Impact assessment of citizen science: state of the art
and guiding principles for a consolidated
approach. Sustainability Science
https://doi.org/10.1007/s11625-021-00959-2
Framework development
4
Initial
framework
Progress the state of the art
Increase accessibility and
adoption
Set of questions to
capture the value of
impact indicators
Targeted literature
review and analysis
mics.tools
platform
Additional
guidance
Measuring Impact of Citizen Science
mics.tools
Framework development
5
Initial
framework
Progress the state of the art
Increase accessibility and
adoption
Set of questions to
capture the value of
impact indicators
Targeted literature
review and analysis
[mics.tools]
MICS platform
Additional
guidance
Measuring Impact of Citizen Science
mics.tools
Impact assessment in
the MICS
case studies
6
• Workshop with practitioners & researchers from NBSs held
at the 2019 River Restoration conference in Liverpool
• 11 Interviews with 10 CS project coordinators (WP2/WP3)
• Screened 503 projects, shortlisted 130 projects, in-depth
analysis of 24 reference projects for IA methodologies,
tools and indicators
• Review of 9 general IA frameworks, 77 CS-specific IA
frameworks & empirical insights -> CS literature review
collaboration with selected WeObserve Impact CoP
members, joint paper
Developing the MICS conceptual framework
Inductive
MICS
Conceptual
Framework
Deductive
Accommodate range of purposes of
citizen-science impact assessment
Non-linear impact journeys
rather than impact silos
IA data collection methods &
information sources
Relative vs absolute impact
2
1
3
4
Guiding principles for MICS conceptual framework
Comparison of IA results
across citizen-science projects
5 Cumulative enhancement of
the framework over time
6 Wehn, U., Gharesifard, M., Ceccaroni, L. et al. (2021) Impact assessment
of citizen science: state of the art and guiding principles for a consolidated
approach. Sustainability Science https://doi.org/10.1007/s11625-021-
00959-2
Implementing the MICS conceptual framework
8
Theory of change elements
9
MICS Indicator
Characteristics Description
Name of Indicator Informative short name for the indicator
Description Succinct description of the indicator (what is it an indicator of)
Domain
Main domain to which the indicator contributes insights [society,
science, economy, environment, governance]
Qualitative/Quantitative
Type of data (quantitative or qualitative).
In some cases, the basic data will be qualitative (interview transcripts,
national reports or similar) which require coding / categorisation in
order to be useful for monitoring purposes.
Primary/Secondary data
Indicator based on:
 Primary data (that needs to be collected) or
 Secondary data (already existing)
 estimates
Source of data
 Primary data: specification of data collection method (e.g.
survey, interview, focus group discussion, observation, etc.)
 Secondary data: specification of existing database, document
(specific page numbers, exact tables etc.), and direct links to
the data source in question (if available)
 Estimates (summaries): expert judgement, e.g. project
coordinator
Time-series
For primary data:
 Time-series required (yes/no; if yes, how frequent)
For secondary data:
 Time-series available (yes/no) and (if applicable) actual years for
which data are available
Unit of measurement (observation)
Unit of measurement (observation), e.g. nominal, ordinal, interval
Unit of analysis
Unit of analysis (e.g. country, organisation, individuals (e.g. citizens),
publications etc.)
Analytical level (logic model)
Analytical level in the intervention logic model to which the indicator is
oriented (i.e. context, input, process, output, outcome, impact).
Links with indicators in other
domains
Domain(s)/indicator(s) to which the indicator (also) relates
Data collection
Details on how data should be collected for this indicator
 Method (survey / questionnaire, data retrieved from
databases, structured/semi-structured/explorative interviews,
focus groups, observation, desk research, document analysis,
ethnographic field studies, etc.)
 Questions asked
 Type & # of respondents / informants, incl. size of this
population
Indicator building
Explanation of how results are calculated from collected data.
For quantitative sub-indicators, specify the unit (e.g. #, %, etc.)
Availability of data
Extent of data availability for the indicator and sub-indicators (e.g. low,
moderate, high)
Implementing the MICS conceptual framework
10
Economy - Indicators ID
Indicator Characteristics
Name of Indicator Company growth International trade & investment Innovation & research Competit
Primary/Secondary data Primary (surveys) and Secondary data Primary (surveys) and Secondary data Primary (surveys) and Secondary data Primary (s
Description <copy from D1.10>
Qual/Quantitative Quantitatve Quantitative & qualitative Quantitatve Quantitat
Source of data dedicated survey dedicated survey dedicated survey dedicated
Time-series Yes, survey should be repeated Yes, survey should be repeated Yes, survey should be repeated Yes, surve
Unit of measurement (observation) absolute values & nominal (see data collection) absolute value absolute value absolute v
Unit of analysis Organisation Organisation Organisation Organisat
Analytical level Outcome; Impact
Links with indicators in other domains n.a. n.a. n.a. n.a.
Data collection
Survey - questions:
How many jobs are currently directly related to [CO topic] and
enabling technologies?
What is the nature of these jobs? (junior, medior, senior
position(s)) [nominal]
How many of your products/services are relevant for the
provision of COs?
What was your organisation's annual turnover in [year]?
What is your organisation's market share in the business of
COs?
How many clients does your organsition have in the CO
business?
Survey - questions:
How many international clients does your organsition have in the
CO business?
What specific customer segments does your organisation serve
related to COs?
How much has your organisation invested in CO-related activities in
[year]?
Survey - questions:
How many IPRs related to COs and enabling technologies
(patents, trademarks, copyright, other know-how rights)
does your organisation hold?
How many CO-related research projects is your organisation
currently involved?
In total, what is your organisation's budget (income & own
investment) in these CO-related research projects?
Survey - q
What valu
does your
Which ma
How man
for CO-rel
How man
organisati
Indicator building
Items:
# subject-related jobs
Nature of jobs
# of CO related products/services
Turnover
Market share in the business of Cos
# of clients in CO business and enabling technologies
Indicators:
# of international clients CO business and enabling tech-nologies
Customer segments (sectors) related to CO
Amount of investment in CO-related activities
<explain relation of analysed results across items above>
Indicators:
IPR (patents, trademarks, copyright, other know-how rights)
# of CO-related research projects
Total budget of CO-related research projects
<explain relation of analysed results across items above>
Indicators
# of reven
# of mark
# of CO to
<explain r
Science &
Technology
Society Economy Environment Governance
MICS Indicator
Characteristics Description
Name of Indicator Informative short name for the indicator
Description Succinct description of the indicator (what is it an indicator of)
Domain
Main domain to which the indicator contributes insights [society,
science, economy, environment, governance]
Qualitative/Quantitative
Type of data (quantitative or qualitative).
In some cases, the basic data will be qualitative (interview transcripts,
national reports or similar) which require coding / categorisation in
order to be useful for monitoring purposes.
Primary/Secondary data
Indicator based on:
 Primary data (that needs to be collected) or
 Secondary data (already existing)
 estimates
Source of data
 Primary data: specification of data collection method (e.g.
survey, interview, focus group discussion, observation, etc.)
 Secondary data: specification of existing database, document
(specific page numbers, exact tables etc.), and direct links to
the data source in question (if available)
 Estimates (summaries): expert judgement, e.g. project
coordinator
Time-series
For primary data:
 Time-series required (yes/no; if yes, how frequent)
For secondary data:
 Time-series available (yes/no) and (if applicable) actual years for
which data are available
Unit of measurement (observation)
Unit of measurement (observation), e.g. nominal, ordinal, interval
Unit of analysis
Unit of analysis (e.g. country, organisation, individuals (e.g. citizens),
publications etc.)
Analytical level (logic model)
Analytical level in the intervention logic model to which the indicator is
oriented (i.e. context, input, process, output, outcome, impact).
Links with indicators in other
domains
Domain(s)/indicator(s) to which the indicator (also) relates
Data collection
Details on how data should be collected for this indicator
 Method (survey / questionnaire, data retrieved from
databases, structured/semi-structured/explorative interviews,
focus groups, observation, desk research, document analysis,
ethnographic field studies, etc.)
 Questions asked
 Type & # of respondents / informants, incl. size of this
population
Indicator building
Explanation of how results are calculated from collected data.
For quantitative sub-indicators, specify the unit (e.g. #, %, etc.)
Availability of data
Extent of data availability for the indicator and sub-indicators (e.g. low,
moderate, high)
Implementing the MICS conceptual framework
Guidance for participatory evaluation
in the case studies
11
Measuring Impact of Citizen Science
mics.tools
Co-creation of Impact Journeys
CS Impact Assessment Compendium per case study
‘One stop’ collection of Impact Assessment instructions & findings per
case study
Co-creating Impact Journeys
Case study – Outfall Safari, UK
12
… and Project managers
Citizen scientists…
Impact journey workshops (online)
13
Measuring Impact of Citizen Science
mics.tools
Refining & validating the impact journeys
Co-creating Impact Journeys
Case study – Outfall Safari, UK
14
Economy - IndicatorsID
IndicatorCharacteristics
Name of Indicator Company growth Internationaltrade & investment Innovation & research Competitiveness
Primary/Secondary data Primary(surveys)andSecondarydata Primary(surveys)andSecondarydata Primary(surveys)andSecondarydata Primary(surveys)andSecondarydata
Description <copyfromD1.10>
Qual/Quantitative Quantitatve Quantitative &qualitative Quantitatve Quantitative &qualitative
Source of data dedicatedsurvey dedicatedsurvey dedicatedsurvey dedicatedsurvey
Time-series Yes,surveyshouldbe repeated Yes,surveyshouldbe repeated Yes,surveyshouldbe repeated Yes,surveyshouldbe repeated
Unit of measurement (observation) absolute values &nominal(see data collection) absolute value absolute value absolute value
Unit of analysis Organisation Organisation Organisation Organisation
Analytical level Outcome; Impact
Links with indicators in other domains n.a. n.a. n.a. n.a.
Data collection
Survey- questions:
How manyjobs are currentlydirectlyrelatedto [CO topic] and
enablingtechnologies?
What is the nature of these jobs?(junior,medior,senior
position(s))[nominal]
How manyof yourproducts/services are relevant forthe
provisionof COs?
What was yourorganisation's annualturnoverin[year]?
What is yourorganisation's market share inthe business of
COs?
How manyclients does yourorgansitionhave inthe CO
business?
Survey- questions:
How manyinternationalclients does yourorgansitionhave inthe
CO business?
What specific customersegments does yourorganisationserve
relatedto COs?
How muchhas yourorganisationinvestedinCO-relatedactivities in
[year]?
Survey- questions:
How manyIPRs relatedto COs andenablingtechnologies
(patents,trademarks,copyright,otherknow-how rights)
does yourorganisationhold?
How manyCO-relatedresearchprojects is yourorganisation
currentlyinvolved?
Intotal,what is yourorganisation's budget (income &own
investment)inthese CO-relatedresearchprojects?
Survey- questions:
What value proposition(s)relatedto COs andenablingtechnologies
does yourorganisationhave?
Whichmarket segments does yourorganisationserve?
How manydifferent revenustreams does yourorgansiationhave
forCO-relatedvalue propositions?
How manypartners forCos andenablingtechnologies does your
organisationhave?
Indicator building
Items:
# subject-relatedjobs
Nature of jobs
# of CO relatedproducts/services
Turnover
Market share inthe business of Cos
# of clients inCO business andenablingtechnologies
<explainrelationof analysedresults across items above>
Indicators:
# of internationalclients CO business andenablingtech-nologies
Customersegments (sectors)relatedto CO
Amount of investment inCO-relatedactivities
<explainrelationof analysedresults across items above>
Indicators:
IPR (patents,trademarks,copyright,otherknow-how rights)
# of CO-relatedresearchprojects
Totalbudget of CO-relatedresearchprojects
<explainrelationof analysedresults across items above>
Indicators:
# of revenue streams
# of market segments served
# of CO topic-relatedpartners
<explainrelationof analysedresults across items above>
Availability of data low low low low
Statistical robustness No validationconducted
Feasibility / Replicability moderatelyresource demanding moderatelyresource demanding moderatelyresource demanding moderatelyresource demanding
Comments/caveats
Bottom up (online workshops) Science-based (MICS conceptual framework)
Co-creating Impact Journeys
Case study – Outfall Safari, UK
From impact journeys to monitoring schemes
15
Measuring of impacts
Indicators to measure
short- & long-term
impacts
Co-creation & validation
of impact journey
Co-creating Impact Journeys
Case study – Outfall Safari, UK
From impact journeys to monitoring schemes
mics.tools
16
Measuring Impact of Citizen Science
mics.tools
The MICS platform (still under development)
17
Measuring Impact of Citizen Science
mics.tools
The MICS platform (still under development)
18
Measuring Impact of Citizen Science
mics.tools
The MICS platform (still under development)
19
Measuring Impact of Citizen Science
mics.tools
The MICS platform (still under development)
20
Measuring Impact of Citizen Science
mics.tools
Citizen science
Agricultural and veterinary sciences
Art theory and criticism
Biological sciences
Chemical sciences
Earth sciences
Engineering
Environmental sciences
Information and computing sciences
Language, communication and culture
Law and legal studies
Mathematical sciences and statistics
Medical and health sciences
Philosophy and religious studies
Physical sciences
Psychology and cognitive sciences
Studies in human society
Technology
The MICS platform (still under development)
21
Measuring Impact of Citizen Science
mics.tools
The MICS platform (still under development)
22
Measuring Impact of Citizen Science
mics.tools
Environment
24
The MICS platform (still under development)
23
Measuring Impact of Citizen Science
mics.tools
What engagement approaches does the project use?
 I don’t know
Previous Next
 Gamification
 Real-time validation on
observation/data submission
 Notification of updates to contributed
observations
 Tips of the day
 Personalised recommendations
 Feedback on data quality
 Active and responsive social-media
presence ("responsive" in the sense
that questions from participants are
responded to)
 Storytelling or exchanging of
experiences
 Invitation to or suggestion of events
 Workshops
 Measurement campaigns
 Active and responsive pilot / case
study presence ("responsive" in the
sense that questions from participants
are responded to)
 A forum, or other platform on a
website
 Awareness raising (ad-hoc
demonstrations, meetings and
appearances from partners)
 Photo contests
 Social events not included in previous
options
 Creation of champions, ambassadors
or leaders
 Organic conversation
The MICS platform (still under development)
24
Measuring Impact of Citizen Science
mics.tools
Does the project have a formal engagement strategy?
 Yes
 No
 I don’t know
Want an example engagement strategy that you can use?
Previous Next
The MICS platform (still under development)
25
Measuring Impact of Citizen Science
mics.tools
Is the engagement strategy shared with participants?
 Yes
 No
 I don’t know
Previous Next
The MICS platform (still under development)
26
Measuring Impact of Citizen Science
mics.tools
Progress
Example 1 of additional guidance
27
Does the project have a code of ethics?
 Yes
 No
 I don’t know
Want a template code of ethics you can use?
Previous Next
Measuring Impact of Citizen Science
mics.tools
Example 2 of additional guidance
28
Do participants gain new knowledge from
taking part in the project?
 Yes, and it has been measured.
 Yes, but it hasn’t been measured.
 No
 I don’t know
Interested in learning how to measure this?
Previous Next
Measuring Impact of Citizen Science
mics.tools
The MICS platform (still under development)
29
Measuring Impact of Citizen Science
mics.tools
The MICS platform (still under development)
30
Measuring Impact of Citizen Science
mics.tools
The MICS platform (still under development)
31
Measuring Impact of Citizen Science
mics.tools
The benefits
32
Measuring Impact of Citizen Science
mics.tools
• A space to think about impact
• Resources and templates
• Guidance on impact-data collection
• An impact-oriented project description
• A quantified assessment
• Access to state-of-the-art AI algorithms
Website: mics.tools
Twitter: @MICSproject
Facebook: MICS Project
LinkedIn: MICS Project
Contacts
• Luigi Ceccaroni
Coordinator
lceccaroni@earthwatch.org.uk
• Uta Wehn
u.wehn@un-ihe.org
• Claire Williams
Project manager
cwilliams@earthwatch.org.uk
Measuring Impact of Citizen Science
mics.tools 33
This project has received funding from the European Union’s Horizon
2020 research and innovation programme under grant agreement
No 824711.

More Related Content

PDF
RDA Presentation to the International Federation of Library Associations
PPTX
ESIP Federation: Community-Driven, Collaborative Governance - Carol Beaton Me...
PDF
My FAIR share of the work - Diamond Light Source - Dec 2018
PDF
RDAP13 Mark Parsons: The Research Data Alliance: Making Data Work
PDF
African Open Science Platform
PDF
Data and communication of research: incentives and disincentives
PPTX
RDAP 16: Sustainability of data infrastructure: The history of science scienc...
PPTX
Rubrics for DMPs
RDA Presentation to the International Federation of Library Associations
ESIP Federation: Community-Driven, Collaborative Governance - Carol Beaton Me...
My FAIR share of the work - Diamond Light Source - Dec 2018
RDAP13 Mark Parsons: The Research Data Alliance: Making Data Work
African Open Science Platform
Data and communication of research: incentives and disincentives
RDAP 16: Sustainability of data infrastructure: The history of science scienc...
Rubrics for DMPs

What's hot (20)

PDF
Benefits of Open Data and Policy Developments, perspectives from research ins...
PDF
Mejias "Making it work globally"
PPT
Workshop intro090314
PPTX
NISO Virtual Conference Scientific Data Management: Caring for Your Instituti...
PPTX
FAIR data: what it means, how we achieve it, and the role of RDA
PPTX
The future of FAIR
PPTX
PSB2014 A Vision for Biomedical Research
PDF
Jonathan Breeze, Symplectic
PDF
Kane "The Past is Prologue: Managing Change to Support an Expanding Research ...
PDF
White paper for an Open Research Data Strategy in Botswana
PDF
Data coordination and the role of RDA
PPTX
From Data Sharing to Data Stewardship
PPTX
Meeting Federal Research Requirements for Data Management Plans, Public Acces...
PPTX
Certifying and Securing a Trusted Environment for Health Informatics Research...
PDF
Cousijns "Open Infrastructure: Foundations and Attributes"
PDF
Open Science Incentives/Veerle van den Eynden
PDF
Big Data for Library Services (2017)
PDF
Open science and data sharing: the DataFirst experience/Martin Wittenberg
PPTX
NISO Virtual Conference Scientific Data Management: Caring for Your Instituti...
Benefits of Open Data and Policy Developments, perspectives from research ins...
Mejias "Making it work globally"
Workshop intro090314
NISO Virtual Conference Scientific Data Management: Caring for Your Instituti...
FAIR data: what it means, how we achieve it, and the role of RDA
The future of FAIR
PSB2014 A Vision for Biomedical Research
Jonathan Breeze, Symplectic
Kane "The Past is Prologue: Managing Change to Support an Expanding Research ...
White paper for an Open Research Data Strategy in Botswana
Data coordination and the role of RDA
From Data Sharing to Data Stewardship
Meeting Federal Research Requirements for Data Management Plans, Public Acces...
Certifying and Securing a Trusted Environment for Health Informatics Research...
Cousijns "Open Infrastructure: Foundations and Attributes"
Open Science Incentives/Veerle van den Eynden
Big Data for Library Services (2017)
Open science and data sharing: the DataFirst experience/Martin Wittenberg
NISO Virtual Conference Scientific Data Management: Caring for Your Instituti...
Ad

Similar to Methods for measuring citizen-science impact (20)

PPT
Manila Workshop Strategies for web data dissemination
PDF
Measuring Progress: Indicators, Data Sources and Assessment | Laszlo Pinter, ...
PPTX
Measuring Progress: Indicators, data sources and assessment | Laszlo Pinter, ...
PDF
Measuring Progress | Charles Thrift
PPT
Introduction to Routine Health Information System Slides
PDF
Overview of methodologies
PDF
155-holguin Baseline for indicators for R&D and innovation in ICT a tool for ...
PPTX
CDI Seminar: 3d impact analysis
PDF
PPT
Information Technology and Data Management Systems: Choosing an Indicator F...
PPTX
TOPIC TWO Evaluation and monitoring in projects
PDF
Ensuring Sustainability of Clean Development Mechanism Projects for Global Su...
PPTX
Hsm 2 Health information systems 21 - Copy.pptx
PPTX
ENVS502_UnitIII.pptx environmental planning and management
PPTX
Science Indicators & Mapping of Science by Aman Kr Kushwaha
PPT
Chapter IV of project planning and (5).ppt
PPT
Official Statistics & Statistical Systems
DOC
Module 8 presenter notes
PPTX
Principles for monitoring and evaluation
PPT
Knowledge Engineering, Electronic Government and the applications to Scientom...
Manila Workshop Strategies for web data dissemination
Measuring Progress: Indicators, Data Sources and Assessment | Laszlo Pinter, ...
Measuring Progress: Indicators, data sources and assessment | Laszlo Pinter, ...
Measuring Progress | Charles Thrift
Introduction to Routine Health Information System Slides
Overview of methodologies
155-holguin Baseline for indicators for R&D and innovation in ICT a tool for ...
CDI Seminar: 3d impact analysis
Information Technology and Data Management Systems: Choosing an Indicator F...
TOPIC TWO Evaluation and monitoring in projects
Ensuring Sustainability of Clean Development Mechanism Projects for Global Su...
Hsm 2 Health information systems 21 - Copy.pptx
ENVS502_UnitIII.pptx environmental planning and management
Science Indicators & Mapping of Science by Aman Kr Kushwaha
Chapter IV of project planning and (5).ppt
Official Statistics & Statistical Systems
Module 8 presenter notes
Principles for monitoring and evaluation
Knowledge Engineering, Electronic Government and the applications to Scientom...
Ad

More from Luigi Ceccaroni (20)

PPTX
ProBleu: Promoting intergenerational change via citizen science and education...
PPTX
Digital twins of the environment: opportunities and barriers for citizen science
PDF
Harnessing the power of citizen science for environmental stewardship and wat...
PPTX
Citizen science, training, data quality and interoperability
PDF
Abrazo, integra tv 4all @ eweek2004 (final)
PPT
Abrazo @ congreso e learning e inclusión social 2004
PPT
Pizza and a movie 2002 aamas
PPT
Integra tv 4all 2005 - drt4all
PPT
In out pc media center 2003
PPT
Modeling utility ontologies in agentcities with a collaborative approach 2002...
PPT
Pizza and a movie 2002 aamas
PPT
The april agent platform 2002 agentcities, lausanne
PPTX
ILIAD and CoCoast @ Noordzeedagen 2021
PDF
MICS @ Geneva 2020
PDF
Metrics and instruments to evaluate the impacts of citizen science
PDF
COST Action 15212 WG5 - Standardisation and interoperability
PDF
The role of interoperability in encouraging participation in citizen science ...
PDF
Ontology of citizen science @ Siena 2016 11 24
PDF
Citclops/EyeOnWater @ Barcelona - Citizen science day 2016
PDF
Workshop - data collection and management
ProBleu: Promoting intergenerational change via citizen science and education...
Digital twins of the environment: opportunities and barriers for citizen science
Harnessing the power of citizen science for environmental stewardship and wat...
Citizen science, training, data quality and interoperability
Abrazo, integra tv 4all @ eweek2004 (final)
Abrazo @ congreso e learning e inclusión social 2004
Pizza and a movie 2002 aamas
Integra tv 4all 2005 - drt4all
In out pc media center 2003
Modeling utility ontologies in agentcities with a collaborative approach 2002...
Pizza and a movie 2002 aamas
The april agent platform 2002 agentcities, lausanne
ILIAD and CoCoast @ Noordzeedagen 2021
MICS @ Geneva 2020
Metrics and instruments to evaluate the impacts of citizen science
COST Action 15212 WG5 - Standardisation and interoperability
The role of interoperability in encouraging participation in citizen science ...
Ontology of citizen science @ Siena 2016 11 24
Citclops/EyeOnWater @ Barcelona - Citizen science day 2016
Workshop - data collection and management

Recently uploaded (20)

DOCX
Epoxy Coated Steel Bolted Tanks for Farm Digesters Supports On-Farm Organic W...
PDF
The Role of Non-Legal Advocates in Fighting Social Injustice.pdf
PPTX
Biodiversity.udfnfndrijfreniufrnsiufnriufrenfuiernfuire
PPTX
FIRE SAFETY SEMINAR SAMPLE FOR EVERYONE.pptx
PPTX
structure and components of Environment.pptx
DOCX
Epoxy Coated Steel Bolted Tanks for Leachate Storage Securely Contain Landfil...
DOCX
Epoxy Coated Steel Bolted Tanks for Beverage Wastewater Storage Manages Liqui...
PDF
Blue Economy Development Framework for Indonesias Economic Transformation.pdf
DOCX
Epoxy Coated Steel Bolted Tanks for Anaerobic Digestion (AD) Plants Core Comp...
PPTX
UN Environmental Inventory User Training 2021.pptx
DOCX
Epoxy Coated Steel Bolted Tanks for Dairy Farm Water Ensures Clean Water for ...
PPTX
9.-Sedatives-and-Hypnotics.pptxcccccccccccccccccccppt
PDF
Effect of salinity on biochimical and anatomical characteristics of sweet pep...
PDF
Insitu conservation seminar , national park ,enthobotanical significance
PPTX
Environmental Ethics: issues and possible solutions
PPTX
Conformity-and-Deviance module 7 ucsp grade 12
PPTX
Concept of Safe and Wholesome Water.pptx
DOCX
Epoxy Coated Steel Bolted Tanks for Crude Oil Large-Scale Raw Oil Containment...
PPTX
Session8a AR6 Findings Working Group I Vice-Chair Nana Ama Browne Klutse
PPTX
Making GREEN and Sustainable Urban Spaces
Epoxy Coated Steel Bolted Tanks for Farm Digesters Supports On-Farm Organic W...
The Role of Non-Legal Advocates in Fighting Social Injustice.pdf
Biodiversity.udfnfndrijfreniufrnsiufnriufrenfuiernfuire
FIRE SAFETY SEMINAR SAMPLE FOR EVERYONE.pptx
structure and components of Environment.pptx
Epoxy Coated Steel Bolted Tanks for Leachate Storage Securely Contain Landfil...
Epoxy Coated Steel Bolted Tanks for Beverage Wastewater Storage Manages Liqui...
Blue Economy Development Framework for Indonesias Economic Transformation.pdf
Epoxy Coated Steel Bolted Tanks for Anaerobic Digestion (AD) Plants Core Comp...
UN Environmental Inventory User Training 2021.pptx
Epoxy Coated Steel Bolted Tanks for Dairy Farm Water Ensures Clean Water for ...
9.-Sedatives-and-Hypnotics.pptxcccccccccccccccccccppt
Effect of salinity on biochimical and anatomical characteristics of sweet pep...
Insitu conservation seminar , national park ,enthobotanical significance
Environmental Ethics: issues and possible solutions
Conformity-and-Deviance module 7 ucsp grade 12
Concept of Safe and Wholesome Water.pptx
Epoxy Coated Steel Bolted Tanks for Crude Oil Large-Scale Raw Oil Containment...
Session8a AR6 Findings Working Group I Vice-Chair Nana Ama Browne Klutse
Making GREEN and Sustainable Urban Spaces

Methods for measuring citizen-science impact

  • 1. Methods for measuring citizen-science impact online SwafS citizen-science cluster meeting on impact 10 November 2021 Uta Wehn, IHE Delft Luigi Ceccaroni, Earthwatch
  • 2. 2 • MICS framework development for impact assessment • Co-design of impact journeys in specific case-study sites • General platform to measure impact Content
  • 3. Framework development 3 Literature review Initial framework Progress the state of the art Increase accessibility Set of qu Targeted lit review Measuring Impact of Citizen Science mics.tools Wehn, U., Gharesifard, M., Ceccaroni, L. et al. (2021) Impact assessment of citizen science: state of the art and guiding principles for a consolidated approach. Sustainability Science https://doi.org/10.1007/s11625-021-00959-2
  • 4. Framework development 4 Initial framework Progress the state of the art Increase accessibility and adoption Set of questions to capture the value of impact indicators Targeted literature review and analysis mics.tools platform Additional guidance Measuring Impact of Citizen Science mics.tools
  • 5. Framework development 5 Initial framework Progress the state of the art Increase accessibility and adoption Set of questions to capture the value of impact indicators Targeted literature review and analysis [mics.tools] MICS platform Additional guidance Measuring Impact of Citizen Science mics.tools Impact assessment in the MICS case studies
  • 6. 6 • Workshop with practitioners & researchers from NBSs held at the 2019 River Restoration conference in Liverpool • 11 Interviews with 10 CS project coordinators (WP2/WP3) • Screened 503 projects, shortlisted 130 projects, in-depth analysis of 24 reference projects for IA methodologies, tools and indicators • Review of 9 general IA frameworks, 77 CS-specific IA frameworks & empirical insights -> CS literature review collaboration with selected WeObserve Impact CoP members, joint paper Developing the MICS conceptual framework Inductive MICS Conceptual Framework Deductive
  • 7. Accommodate range of purposes of citizen-science impact assessment Non-linear impact journeys rather than impact silos IA data collection methods & information sources Relative vs absolute impact 2 1 3 4 Guiding principles for MICS conceptual framework Comparison of IA results across citizen-science projects 5 Cumulative enhancement of the framework over time 6 Wehn, U., Gharesifard, M., Ceccaroni, L. et al. (2021) Impact assessment of citizen science: state of the art and guiding principles for a consolidated approach. Sustainability Science https://doi.org/10.1007/s11625-021- 00959-2
  • 8. Implementing the MICS conceptual framework 8 Theory of change elements
  • 9. 9 MICS Indicator Characteristics Description Name of Indicator Informative short name for the indicator Description Succinct description of the indicator (what is it an indicator of) Domain Main domain to which the indicator contributes insights [society, science, economy, environment, governance] Qualitative/Quantitative Type of data (quantitative or qualitative). In some cases, the basic data will be qualitative (interview transcripts, national reports or similar) which require coding / categorisation in order to be useful for monitoring purposes. Primary/Secondary data Indicator based on:  Primary data (that needs to be collected) or  Secondary data (already existing)  estimates Source of data  Primary data: specification of data collection method (e.g. survey, interview, focus group discussion, observation, etc.)  Secondary data: specification of existing database, document (specific page numbers, exact tables etc.), and direct links to the data source in question (if available)  Estimates (summaries): expert judgement, e.g. project coordinator Time-series For primary data:  Time-series required (yes/no; if yes, how frequent) For secondary data:  Time-series available (yes/no) and (if applicable) actual years for which data are available Unit of measurement (observation) Unit of measurement (observation), e.g. nominal, ordinal, interval Unit of analysis Unit of analysis (e.g. country, organisation, individuals (e.g. citizens), publications etc.) Analytical level (logic model) Analytical level in the intervention logic model to which the indicator is oriented (i.e. context, input, process, output, outcome, impact). Links with indicators in other domains Domain(s)/indicator(s) to which the indicator (also) relates Data collection Details on how data should be collected for this indicator  Method (survey / questionnaire, data retrieved from databases, structured/semi-structured/explorative interviews, focus groups, observation, desk research, document analysis, ethnographic field studies, etc.)  Questions asked  Type & # of respondents / informants, incl. size of this population Indicator building Explanation of how results are calculated from collected data. For quantitative sub-indicators, specify the unit (e.g. #, %, etc.) Availability of data Extent of data availability for the indicator and sub-indicators (e.g. low, moderate, high) Implementing the MICS conceptual framework
  • 10. 10 Economy - Indicators ID Indicator Characteristics Name of Indicator Company growth International trade & investment Innovation & research Competit Primary/Secondary data Primary (surveys) and Secondary data Primary (surveys) and Secondary data Primary (surveys) and Secondary data Primary (s Description <copy from D1.10> Qual/Quantitative Quantitatve Quantitative & qualitative Quantitatve Quantitat Source of data dedicated survey dedicated survey dedicated survey dedicated Time-series Yes, survey should be repeated Yes, survey should be repeated Yes, survey should be repeated Yes, surve Unit of measurement (observation) absolute values & nominal (see data collection) absolute value absolute value absolute v Unit of analysis Organisation Organisation Organisation Organisat Analytical level Outcome; Impact Links with indicators in other domains n.a. n.a. n.a. n.a. Data collection Survey - questions: How many jobs are currently directly related to [CO topic] and enabling technologies? What is the nature of these jobs? (junior, medior, senior position(s)) [nominal] How many of your products/services are relevant for the provision of COs? What was your organisation's annual turnover in [year]? What is your organisation's market share in the business of COs? How many clients does your organsition have in the CO business? Survey - questions: How many international clients does your organsition have in the CO business? What specific customer segments does your organisation serve related to COs? How much has your organisation invested in CO-related activities in [year]? Survey - questions: How many IPRs related to COs and enabling technologies (patents, trademarks, copyright, other know-how rights) does your organisation hold? How many CO-related research projects is your organisation currently involved? In total, what is your organisation's budget (income & own investment) in these CO-related research projects? Survey - q What valu does your Which ma How man for CO-rel How man organisati Indicator building Items: # subject-related jobs Nature of jobs # of CO related products/services Turnover Market share in the business of Cos # of clients in CO business and enabling technologies Indicators: # of international clients CO business and enabling tech-nologies Customer segments (sectors) related to CO Amount of investment in CO-related activities <explain relation of analysed results across items above> Indicators: IPR (patents, trademarks, copyright, other know-how rights) # of CO-related research projects Total budget of CO-related research projects <explain relation of analysed results across items above> Indicators # of reven # of mark # of CO to <explain r Science & Technology Society Economy Environment Governance MICS Indicator Characteristics Description Name of Indicator Informative short name for the indicator Description Succinct description of the indicator (what is it an indicator of) Domain Main domain to which the indicator contributes insights [society, science, economy, environment, governance] Qualitative/Quantitative Type of data (quantitative or qualitative). In some cases, the basic data will be qualitative (interview transcripts, national reports or similar) which require coding / categorisation in order to be useful for monitoring purposes. Primary/Secondary data Indicator based on:  Primary data (that needs to be collected) or  Secondary data (already existing)  estimates Source of data  Primary data: specification of data collection method (e.g. survey, interview, focus group discussion, observation, etc.)  Secondary data: specification of existing database, document (specific page numbers, exact tables etc.), and direct links to the data source in question (if available)  Estimates (summaries): expert judgement, e.g. project coordinator Time-series For primary data:  Time-series required (yes/no; if yes, how frequent) For secondary data:  Time-series available (yes/no) and (if applicable) actual years for which data are available Unit of measurement (observation) Unit of measurement (observation), e.g. nominal, ordinal, interval Unit of analysis Unit of analysis (e.g. country, organisation, individuals (e.g. citizens), publications etc.) Analytical level (logic model) Analytical level in the intervention logic model to which the indicator is oriented (i.e. context, input, process, output, outcome, impact). Links with indicators in other domains Domain(s)/indicator(s) to which the indicator (also) relates Data collection Details on how data should be collected for this indicator  Method (survey / questionnaire, data retrieved from databases, structured/semi-structured/explorative interviews, focus groups, observation, desk research, document analysis, ethnographic field studies, etc.)  Questions asked  Type & # of respondents / informants, incl. size of this population Indicator building Explanation of how results are calculated from collected data. For quantitative sub-indicators, specify the unit (e.g. #, %, etc.) Availability of data Extent of data availability for the indicator and sub-indicators (e.g. low, moderate, high) Implementing the MICS conceptual framework
  • 11. Guidance for participatory evaluation in the case studies 11 Measuring Impact of Citizen Science mics.tools Co-creation of Impact Journeys CS Impact Assessment Compendium per case study ‘One stop’ collection of Impact Assessment instructions & findings per case study
  • 12. Co-creating Impact Journeys Case study – Outfall Safari, UK 12 … and Project managers Citizen scientists… Impact journey workshops (online)
  • 13. 13 Measuring Impact of Citizen Science mics.tools Refining & validating the impact journeys Co-creating Impact Journeys Case study – Outfall Safari, UK
  • 14. 14 Economy - IndicatorsID IndicatorCharacteristics Name of Indicator Company growth Internationaltrade & investment Innovation & research Competitiveness Primary/Secondary data Primary(surveys)andSecondarydata Primary(surveys)andSecondarydata Primary(surveys)andSecondarydata Primary(surveys)andSecondarydata Description <copyfromD1.10> Qual/Quantitative Quantitatve Quantitative &qualitative Quantitatve Quantitative &qualitative Source of data dedicatedsurvey dedicatedsurvey dedicatedsurvey dedicatedsurvey Time-series Yes,surveyshouldbe repeated Yes,surveyshouldbe repeated Yes,surveyshouldbe repeated Yes,surveyshouldbe repeated Unit of measurement (observation) absolute values &nominal(see data collection) absolute value absolute value absolute value Unit of analysis Organisation Organisation Organisation Organisation Analytical level Outcome; Impact Links with indicators in other domains n.a. n.a. n.a. n.a. Data collection Survey- questions: How manyjobs are currentlydirectlyrelatedto [CO topic] and enablingtechnologies? What is the nature of these jobs?(junior,medior,senior position(s))[nominal] How manyof yourproducts/services are relevant forthe provisionof COs? What was yourorganisation's annualturnoverin[year]? What is yourorganisation's market share inthe business of COs? How manyclients does yourorgansitionhave inthe CO business? Survey- questions: How manyinternationalclients does yourorgansitionhave inthe CO business? What specific customersegments does yourorganisationserve relatedto COs? How muchhas yourorganisationinvestedinCO-relatedactivities in [year]? Survey- questions: How manyIPRs relatedto COs andenablingtechnologies (patents,trademarks,copyright,otherknow-how rights) does yourorganisationhold? How manyCO-relatedresearchprojects is yourorganisation currentlyinvolved? Intotal,what is yourorganisation's budget (income &own investment)inthese CO-relatedresearchprojects? Survey- questions: What value proposition(s)relatedto COs andenablingtechnologies does yourorganisationhave? Whichmarket segments does yourorganisationserve? How manydifferent revenustreams does yourorgansiationhave forCO-relatedvalue propositions? How manypartners forCos andenablingtechnologies does your organisationhave? Indicator building Items: # subject-relatedjobs Nature of jobs # of CO relatedproducts/services Turnover Market share inthe business of Cos # of clients inCO business andenablingtechnologies <explainrelationof analysedresults across items above> Indicators: # of internationalclients CO business andenablingtech-nologies Customersegments (sectors)relatedto CO Amount of investment inCO-relatedactivities <explainrelationof analysedresults across items above> Indicators: IPR (patents,trademarks,copyright,otherknow-how rights) # of CO-relatedresearchprojects Totalbudget of CO-relatedresearchprojects <explainrelationof analysedresults across items above> Indicators: # of revenue streams # of market segments served # of CO topic-relatedpartners <explainrelationof analysedresults across items above> Availability of data low low low low Statistical robustness No validationconducted Feasibility / Replicability moderatelyresource demanding moderatelyresource demanding moderatelyresource demanding moderatelyresource demanding Comments/caveats Bottom up (online workshops) Science-based (MICS conceptual framework) Co-creating Impact Journeys Case study – Outfall Safari, UK From impact journeys to monitoring schemes
  • 15. 15 Measuring of impacts Indicators to measure short- & long-term impacts Co-creation & validation of impact journey Co-creating Impact Journeys Case study – Outfall Safari, UK From impact journeys to monitoring schemes
  • 16. mics.tools 16 Measuring Impact of Citizen Science mics.tools
  • 17. The MICS platform (still under development) 17 Measuring Impact of Citizen Science mics.tools
  • 18. The MICS platform (still under development) 18 Measuring Impact of Citizen Science mics.tools
  • 19. The MICS platform (still under development) 19 Measuring Impact of Citizen Science mics.tools
  • 20. The MICS platform (still under development) 20 Measuring Impact of Citizen Science mics.tools Citizen science Agricultural and veterinary sciences Art theory and criticism Biological sciences Chemical sciences Earth sciences Engineering Environmental sciences Information and computing sciences Language, communication and culture Law and legal studies Mathematical sciences and statistics Medical and health sciences Philosophy and religious studies Physical sciences Psychology and cognitive sciences Studies in human society Technology
  • 21. The MICS platform (still under development) 21 Measuring Impact of Citizen Science mics.tools
  • 22. The MICS platform (still under development) 22 Measuring Impact of Citizen Science mics.tools Environment 24
  • 23. The MICS platform (still under development) 23 Measuring Impact of Citizen Science mics.tools What engagement approaches does the project use?  I don’t know Previous Next  Gamification  Real-time validation on observation/data submission  Notification of updates to contributed observations  Tips of the day  Personalised recommendations  Feedback on data quality  Active and responsive social-media presence ("responsive" in the sense that questions from participants are responded to)  Storytelling or exchanging of experiences  Invitation to or suggestion of events  Workshops  Measurement campaigns  Active and responsive pilot / case study presence ("responsive" in the sense that questions from participants are responded to)  A forum, or other platform on a website  Awareness raising (ad-hoc demonstrations, meetings and appearances from partners)  Photo contests  Social events not included in previous options  Creation of champions, ambassadors or leaders  Organic conversation
  • 24. The MICS platform (still under development) 24 Measuring Impact of Citizen Science mics.tools Does the project have a formal engagement strategy?  Yes  No  I don’t know Want an example engagement strategy that you can use? Previous Next
  • 25. The MICS platform (still under development) 25 Measuring Impact of Citizen Science mics.tools Is the engagement strategy shared with participants?  Yes  No  I don’t know Previous Next
  • 26. The MICS platform (still under development) 26 Measuring Impact of Citizen Science mics.tools Progress
  • 27. Example 1 of additional guidance 27 Does the project have a code of ethics?  Yes  No  I don’t know Want a template code of ethics you can use? Previous Next Measuring Impact of Citizen Science mics.tools
  • 28. Example 2 of additional guidance 28 Do participants gain new knowledge from taking part in the project?  Yes, and it has been measured.  Yes, but it hasn’t been measured.  No  I don’t know Interested in learning how to measure this? Previous Next Measuring Impact of Citizen Science mics.tools
  • 29. The MICS platform (still under development) 29 Measuring Impact of Citizen Science mics.tools
  • 30. The MICS platform (still under development) 30 Measuring Impact of Citizen Science mics.tools
  • 31. The MICS platform (still under development) 31 Measuring Impact of Citizen Science mics.tools
  • 32. The benefits 32 Measuring Impact of Citizen Science mics.tools • A space to think about impact • Resources and templates • Guidance on impact-data collection • An impact-oriented project description • A quantified assessment • Access to state-of-the-art AI algorithms
  • 33. Website: mics.tools Twitter: @MICSproject Facebook: MICS Project LinkedIn: MICS Project Contacts • Luigi Ceccaroni Coordinator lceccaroni@earthwatch.org.uk • Uta Wehn u.wehn@un-ihe.org • Claire Williams Project manager cwilliams@earthwatch.org.uk Measuring Impact of Citizen Science mics.tools 33 This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 824711.