Measuring the Case for Creativity

Measuring Creativity:
Adobe Youth Voices (AYV) Case Study
February 2014

2
Abstract
This white paper explores the importance and challenges of measuring creativity in the social
sector. It situates the concept of “creative confidence” within the broader field of creativity
studies, and outlines promising approaches to benchmarking and/or developing indices for
creative confidence and the application of creative confidence among youth. This approach was
developed through an initial Mission Measurement evaluation of the Adobe Youth Voices
program between 2012 and 2013. Following a description of background, methodology, and
findings, the paper summarizes lessons learned and suggests best practices for evaluating
similar initiatives in education, technology, and media.
Photo Credits:
“Find-your-target-and-follow-it”: iEARN Class of 2012-2013
“Design Your World”: Intel Computer Clubhouse Class of 2012-2013
“Aboriginal Voice Still Strong”: Free the Children Class of 2012-2013

3
Contents
Why Creativity Matters............................................................................................................... 3
Measuring Creativity .................................................................................................................. 5
The Measurement Challenge..................................................................................................... 6
Articulating Outcomes and Measuring Creativity: Mission Measurement’s Approach to
Evaluating the Adobe Youth Voices (AYV) Program .................................................................. 7
About Adobe Youth Voices..................................................................................................... 8
Articulating Outcomes ............................................................................................................ 9
Measuring Program Outcomes: Mission Measurement’s Methodology..................................14
Key Evaluation Findings ........................................................................................................16
So What? Lessons Learned......................................................................................................18
Next Steps ................................................................................................................................19

4
Creative Class
Why Creativity Matters
In the last 15 years of rapid technological change and use of diverse media engagement
platforms, the study of creativity has gained popularity in relation to everything from pop culture
to global development. From Richard Florida’s inception of the Global Creativity Index to
Disney’s Creativity Studio phone application, the need to understand, measure, and even sell
creativity is more popular than ever.
Why focus on creativity? A growing
population of workers is now defined as
“the creative class.” This class has been
identified as contributing to national
economic growth. These individuals –
“More than 40 million Americans, roughly
one-third of all employed people…in
science and engineering, architecture and
design, education, arts, music, and
entertainment” – engage in problem
solving to create new, useful and
innovative products and solutions.1
In the
current global economic climate, their
creative skills have become even more
important. As Florida points out, (citing
TechCrunch), “In a time of high
unemployment, when traditional skills can
be outsourced or automated, creative
skills remain highly sought after and highly valuable.”2
The 2012 Adobe Creativity Gap study of 5,000 adults in the United States, United Kingdom,
Germany, France and Japan confirmed this sentiment. Eighty percent of surveyed adults
agreed that “unlocking creative potential is key to economic growth” and nearly two thirds felt
that “being creative is valuable to society”.3
Beyond general population consensus, employers directly report that they are looking for
employees capable of innovation and critical thinking skills. A 2013 Hart Research Associates
study of 318 U.S. employers found that 71% reported colleges should put more emphasis on
“the ability to innovate and be creative”.4
Nearly all those surveyed (93%) agreed that “a
1
Florida, Richard (2011). The Rise of the Creative Class. New York: Basic Books.
2
Ibid.
3
Adobe Systems Inc. (2012). The State of Create Study. Retrieved from:
http://www.adobe.com/aboutadobe/pressroom/pdfs/Adobe_State_of_Create_Global_Benchmark_Study.pdf
4
Hart Research Associates. (2013). It Takes More than a Major: Employer Priorities for College Learning and
Student Success. Retrieved from: http://www.aacu.org/leap/documents/2013_EmployerSurvey.pdf
Figure 1: Creative Class
Source: Martin Prosperity Index

5
candidate’s demonstrated capacity to think critically, communicate clearly, and solve complex
problems is more important than their undergraduate major”.5
As demand for creative job candidates builds, debates about creativity have increasingly
entered the education space. A majority of adults in various global economies feel their
education system does not support creativity. Some educationists have argued that from an
accountability standpoint, schools should be held responsible for cultivating creativity in
students just as equally if not more than math and reading skills.6
The aforementioned Adobe
study found that over half of respondents felt that “creativity is being stifled by their educational
system”.7
Others have focused on issues of access and participation, arguing that historically
disadvantaged groups are further left behind by inequitable access to technology and media
tools for learning.8
Measuring Creativity
In light of what Jenkins calls the “Participation Gap” and what the Adobe Systems benchmark
study dubbed as the “Creativity Gap”, schools, nonprofits, and corporations are developing new
initiatives and approaches to fostering creativity. As these programs and curricula are
introduced, leaders are concurrently challenged with measuring their impact.
Researchers have developed several approaches to measuring creativity - provisionally defined
here as “the type of creativity that makes people adapt to the constantly changing environment,
reformulate problems, and take risks to try new approaches to problems”.9
These approaches
include:
 Measuring how creative work is;
 Measuring how creative a person is; or
 Measuring the social or economic value of creative work.10
Measurement occurs at aggregate and individual levels, with researchers attempting to examine
creativity levels in societies, governments, educational systems, and among individuals.11
Measurement of creativity in societies and governments often focuses on understanding
environmental conditions believed to contribute to creativity, such as tolerance, heterogeneity,
and levels of educational attainment.12
At the global level, tools such as the Global Creativity
5
Ibid.
6
Jenkins, H. (2006). Convergence Culture: Where Old and New Media Collide. New York: New York University
Press.
7
Adobe Systems Inc. (2012). The State of Create Study. Retrieved from:
http://www.adobe.com/aboutadobe/pressroom/pdfs/Adobe_State_of_Create_Global_Benchmark_Study.pdf.
8
Jenkins, H. (2006). Convergence Culture: Where Old and New Media Collide. New York: New York University
Press.
9
Villalba, E. (2009). Is it really possible to measure creativity? A first proposal for debate. Retrieved from:
http://ec.europa.eu/education/lifelong-learning-policy/doc/creativity/report/intro.pdf.
10
Nilsson, P. (2012). Four Ways to Measure Creativity. Retrieved from:
http://www.senseandsensation.com/2012/03/assessing-creativity.html.
11
12
Ibid.

6
Index have additionally attempted to measure how creativity is linked to elements other than
economic competitiveness, including overall happiness and well-being.13
Measurement of creativity at the individual level often focuses on understanding the
characteristics of a creative person, including but not limited to:
 Autonomy
 Flexibility
 Preference for complexity
 Openness to experience
 Sensitivity
 Playfulness
 Tolerance of ambiguity
 Risk-taking and risk-tolerance
 Intrinsic motivation
 Self-efficacy
 Wide interest and curiosity.14
Researchers also measure creativity levels in individuals by assessing how creative their work
is. For example, assessments such as the Torrance Test utilize questions that require quick
creative work such as turning a black egg shape into an image that “no one else will think of”,
then score respondents’ creativity based on quantity and quality, with a focus on response
diversity and originality. See Figure 2 below for a sample test.15
13
Martin Prosperity Index (2011). Creativity and Prosperity: The Global Creativity Index. Retrieved from:
http://martinprosperity.org/media/GCI%20Report%20Sep%202011.pdf.
14
15
Blair, Elizabeth. (2013). “More Than 50 Years of Putting Kids’ Creativity to the Test”.
http://www.npr.org/2013/04/17/177040995/more-than-50-years-of-putting-kids-creativity-to-the-test. National Public
Radio.
Figure 2: Sample Torrance Test

7
Evaluators must begin with the
challenge of defining the desired
outcome of a given program or
initiative. Is the goal to increase
“creativity” or to produce a new
generation of students ready to
join the creative class?
The Measurement Challenge
Though frameworks exist, many academics and
professionals continue to face conceptual and practical
challenges when attempting to measure the impact of
programs designed to increase creativity.
Conceptually, as with most social impact programs and
initiatives, evaluators begin with the challenge of
defining the desired outcome of a given program or
initiative. Is the goal to increase “creativity” or to produce
a new generation of students ready to join the creative class? Are we looking for creativity in
individual work or a broader capacity to collaborate and problem solve with others?
Beyond identifying intended program or initiative outcomes, evaluators must also define their
measurement construct. As Shechtman et al., note in a discussion about measuring “grit” or
“perseverance”, “an important distinction for measurement purposes is whether perseverance is
conceptualized as a disposition or a set of processes.”16
Similarly, “being creative” can be
conceptualized as a mindset, a trait, an enduring tendency, or a set of processes that unfold
over time.17
Skills and attributes or evidence of character traits are often conflated but can be
measured separately.
Practically, as programs and initiatives scale their efforts to foster creativity, they face
challenges gathering data from a larger number of respondents. Assessments that require
review of student work can provide valuable information but can be costly and time consuming
to conduct regularly. Authentic assessment involving evaluation of creative student work or
problem solving abilities can also be difficult to standardize and scale.
16
Shechtman, N., DeBarger, A. H., Dornsife, C., Rosier, S., Yarnall, L. (2013). Promoting Grit, Tenacity, and
Perseverance: Critical Factors for Success in the 21
st
Century. Retrieved from:
http://www.ed.gov/edblogs/technology/files/2013/02/OET-Draft-Grit-Report-2-17-13.pdf.
17
Ibid.

8
Articulating Outcomes and Measuring Creativity: Mission
Measurement’s Approach to Evaluating the Adobe Youth Voices
(AYV) Program
In 2012, Mission Measurement was provided with the opportunity to tackle the conceptual and
practical challenges of measuring creativity by developing and managing ongoing evaluation of
the Adobe Youth Voices program - a large-scale, innovative effort developed by the Adobe
Foundation to address the challenge of cultivating creativity in youth.
About Adobe Youth Voices (AYV)
Adobe Youth Voices (AYV) is a global initiative of
the Adobe Foundation designed to increase youth
creativity through the use of advanced digital
media tools and tested storytelling techniques.
Launched in 2006 and currently in its 8th
year of
operation, the AYV program has reached more
than 153,000 youth in over 55 countries.18
The program was developed and
implemented in collaboration with Education
Development Centers (EDC) and 19
American and international nonprofit partners (grantees). Currently AYV works with 14 partners
that participated in a post-program evaluation between 2012 and 2013 (see Appendix A for a list
of current partners). 19
AYV provides each of its partners with funding, curriculum, software, training, and technical
assistance that enables design and implementation of AYV workshops, full-year, and/or half-
year programs20
held during or outside of regular school schedules.
Although each global partner maintains an individualized program structure, a shared student-
centered curriculum unites the program. All AYV educators participate in a series of professional
development activities via training on curriculum design and implementation catered to AYV.
This training, which takes place prior to implementation of each program, utilizes youth-
assessment tools and rubrics promoting various student-centered pedagogical approaches that
encourage youth to “Create with Purpose”. By teaching educators to facilitate the media making
process, AYV supports a content focus on various types of media making (e.g., video,
animation, photography, podcasting, graphic design, poetry, print, etc.).
18
Adobe Youth Voices. (2013). Retrieved from: http://youthvoices.adobe.com/about
19
Mission Measurement evaluation efforts between 2012 and 2013 involved 16 AYV partners only.
20
Programs lasting one to four months.
Figure 3: Adobe Youth Voices Sites
Source: http://youthvoices.adobe.com/about

9
While AYV offers training to educators,
it primary objective is to give youth a
voice. Youth are encouraged to select
media topics of personal significance
within their school or home
communities. Finished media projects
share stories and address issues youth
value, including but not limited to
political conflict, environmental
preservation, bullying, gender, and
drug abuse (see Figure 4 for more
detail).
Each year, Adobe also recognizes
youth potential and noteworthy projects
through complementary initiatives
including awards and scholarships.
Articulating Outcomes
At the onset of its work with AYV, Mission Measurement engaged in comprehensive stakeholder
discussions to identify and articulate the program’s high-value outcomes. The development or
identification of an Outcomes Inventory™ was a crucial first stage of this engagement, in which
the Team began by developing a robust understanding of the Adobe Foundation’s strategic
direction, goals, and expectations for its investments. Throughout the year, Mission
Measurement worked to develop this knowledge base by engaging program leaders and
strategic stakeholders in a series of interviews which enabled the Research Team to track the
evolving strategic direction of the program. Through these conversations, Mission Measurement
revealed that AYV not only seeks to develop individual youth creativity (potentially characterized
as the ability to generate many novel ideas), but rather a much broader, nuanced set of skills
that would enable and empower youth to succeed in their personal, educational, and
professional lives. As the team probed further, it found that the program was also designed to
ensure youth applied their gained skill sets to their education, community, and career pathways.
As a leading global creative company, Adobe has a unique perspective on the importance and
application of these skill sets and increasingly looked to the AYV program to support the
development of applied creativity. Given this focus, Mission Measurement honed in on an
emerging concept in academic and corporate social responsibility literature – Creative
Confidence.
After reviewing AYV curriculum and program documents, conducting a series of workshops with
Adobe staff, and drawing upon various articles and speeches (in particular, Tom and David
Kelley’s “Reclaim Your Creative Confidence”), Mission Measurement began to operationalize
creative confidence by outlining what this concept means for AYV youth. See Table 1 below.
Figure 4: Adobe Youth Voices Youth Project Topics
Source: Adobe Youth Voices “Youth Say What” Infographic
What sparks
youth passion,
globally?

10
YouthEducators
Application of creative
confidence
Development of creative
confidence
Development of technical
skills
Attitudinal orientation
toward creativity
Increase creative
confidence in
youth
Improve educator
skills in and
utilization of
technology and
media
Improve educator
ability to facilitate
the creative
process
Increase pool of
educators
committed to
facilitating the
development of
creative
confidence
Increase pool of
educators
facilitating the
development of
creative
confidence in
youth
Increase youth
understanding of
the importance of
creative
confidence
Increase youth
application of
creative
confidence in
education, career
or community
engagement
Increase youth
ability to use
media to express
themselves
Table 1: Defining Creative Confidence: An Initial Set of Key Skills and Characteristics
Youth exhibit skills in:
 Communication
 Collaboration
 Critical thinking
 Problem solving
 Ideation
Youth are able to:
 Innovate
 Multi-task
 Be flexible/adaptable
 Express their point of view
Youth have:
 Self-confidence
 Sense of youth voice
 Pride in their abilities
 Accountability
 Persistence and
motivation
As the team drew closer to implementing monitoring and evaluation tools, it more precisely
defined creative confidence as “the ability to harness creative skills to solve problems” and
committed to measuring five dimensions of creative confidence:
 Self-Expression – the ability to express a point of view;
 Ideation – the ability to ideate and innovate;
 Collaboration – the ability to engage others;
 Flexibility – the ability to adapt; and
 Persistence – the ability to stick with a challenge through completion.
This signature concept is featured among other high-value youth and educator outcomes that
Mission Measurement and AYV leaders identified as expected program outcomes during the
first stage of the engagement. See Figure 5 below.
Figure 5: Adobe Youth Voices’ Outcomes Inventory
1 2 3 4
5 6 7 8

11
Brainstorming Selecting
Audience
Reviewing
and Editing
Preparing for
Exhibition
The Outcomes Inventory reflects AYV’s intention not only to develop youth creative confidence
and improve educator ability to facilitate the creative process, but also captures the intended
development process, through which youth adopt a new attitudinal orientation, develop technical
skills, then cultivate and apply their creative confidence.
This paper focuses exclusively on the youth outcomes as a means to a) situate AYV’s work
within the broader field of creativity studies, and b) highlight suggested proxies for creative
confidence among youth.
In particular, the paper will focus on the following youth outcomes:
Outcome 2: Increase youth ability to use media for self-expression
Outcome 3: Increase creative confidence in youth
Outcome 4: Increase youth application of creative confidence in education, career, or
community engagement.
Outcome 2: Increase Youth Ability to Use Media for Self-Expression
Working closely with the AYV team, Mission Measurement initially identified several dimensions
relevant to the outcome of increasing youth ability to use media for self-expression. Ultimately,
the team focused on measuring:
 Proficiency with media technology;
 Proficiency with the media making process; and
 Media literacy.
Each metric was further defined by outlining key stages or indicators for proficiency. For
example, based on review of AYV curriculum and consultation with the AYV and EDC Team,
the Evaluation Team identified four key stages of the media making process: brainstorming,
selecting an audience, reviewing and editing, and preparing to exhibit media.
Figure 6: Media Making Process

12
The team additionally developed competency scales to measure proficiency. To measure media
making proficiency, the team utilized a scale that probed for degrees of independence in
performing certain tasks (e.g., did/did not complete, completed with or without the help of an
educator, or completed as the leader of a youth group). To measure media literacy, the team
probed for frequency with which youth reported engaging in key media literacy skills including
recognizing a media piece’s audience, message, technique, and quality.
Outcome 3: Increase Creative Confidence in Youth
A core intended outcome of the AYV program is the development of creative confidence in
youth. The program is premised on the belief that the guided use of advanced digital media
tools and development of storytelling techniques can ignite youth creative confidence and
provide the opportunity to use creative skills to solve problems.
AYV regularly collects youth and educator testimonials which confirm the validity of this
premise. As with many youth development programs, AYV educators can readily cite examples
of youth growth during and after the program – youth who began the program reluctantly and
blossomed into engaged, passionate, and effective creators by the end of the program year.
However, AYV leaders struggled to structure, aggregate, and report these individual stories of
success.
By defining key dimensions of creative confidence and using a blend of quantitative and
qualitative analysis, Mission Measurement created a framework to better capture program
impact and provided preliminary data suggesting a link between key program activities and
youth creative confidence.
As aforementioned, Mission Measurement and the AYV Team agreed on five creative
confidence dimensions:
1) Ideation: The ability to ideate and innovate
2) Collaboration: The ability to successfully engage others
3) Flexibility: The ability to adapt
4) Persistence: The ability to manage challenges through completion
5) Self-Expression: The ability to express a point of view.
The team operationalized each dimension to enable youth to self-report on their abilities and
has further refined its phrasing in the second year of evaluation:

13
Table 2: Creative Confidence Operationalization
Initial Concept
Articulation
Initial
Operationalization
Current
Operationalization
Ideation
The ability to ideate and
innovate
Generate original ideas Create new ideas
Collaboration
The ability to
successfully engage
others
Work well with other
people
Work with and be
inspired by others
Flexibility The ability to adapt
Adapt to changes and
new situations
Adapt to problems and
circumstances
Persistence
The ability to manage
challenges through
completion
Overcome challenges to
finish what I’ve started
Stick with a challenge
through completion
Self-
Expression
The ability to express a
point of view
Communicate my ideas
to others
Express my point of view
Youth reported their strength of confidence across all five creative confidence dimensions,
providing perspective on their self-perception. Though self-perception of confidence can be
viewed as an inherently subjective, problematic method of measuring skills and abilities, where
positive self-perception (a disposition or mindset) is itself an outcome, a confidence scale can
yield valuable data. As Tom and David Kelly write,
“At its core, creative confidence is about believing in your ability to create change in the
world around you. It is the conviction that you can achieve what you set out to do...this
belief in your creative capacity lies at the heart of innovation.”21
To capture program impact on youth behavior, youth were also asked to provide qualitative
feedback regarding whether the program affected their ability to express themselves (a
particular focus of the program as executed to date).
Outcome 4: Increase Youth Application of Creative Confidence in Education, Career, or
Community Engagement
The ultimate intended outcome of the AYV program is the application of creative confidence.
AYV leaders are not only interested in determining whether youth in the program gain specific
skills or a more confident outlook, but also whether the program can have a holistic and lasting
impact on the lives of youth.
21
Kelley, T and Kelly, D. (2013). Creative Confidence: Unleashing the Creative Potential Within us All. New York:
Crown Business.

14
Collaborating once again with AYV leaders, the Evaluation Team worked to further define
application of creative confidence by narrowing its focus to understanding if and how the
program increases youths’ educational, career, and community engagement.
Specifically, youth were asked if, because of the AYV program, they had witnessed change in
the following areas of interest:
Table 3: Survey Options for Application of Creative Confidence
Application Areas Indicators
Education
 Tried harder to succeed in school;
 Taken steps to apply to a college/university;
Career
 Learned career skills;
 Thought more about the type of career they want to have;
Community  Volunteered outside of AYV; or
 Learned more about an issue within their community.22
After defining program outcomes, Mission Measurement developed data collection tools to
measure AYV’s impact in its seventh program year. Data was primarily collected through two
online post-program surveys – one for educators and one for youth. Both surveys were
translated and distributed in seven languages to ensure optimal international response rates. 23
This measurement approach enabled the Team to collect data from program participants at
multiple geographically separated sites. Online distribution also ensured all partners could
participate in the evaluation, regardless of their physical access to AYV educators and youth.
Fundamental to the survey design was an effort to measure creative confidence conceptualized
as a set of processes and as a disposition. For example, youth were asked about their
participation in the media creation process and their level of engagement during media
consumption. Self-reported behaviors, such as leading one’s team through each stage of the
media making process, were used as skill development indicators. Additionally, the team
directly probed to understand youth self-perception on each of the creative confidence
dimensions. For example, youth were asked how confident they felt in their ability to “generate
new ideas”.24
These self-ratings provided insight into self-perceived character traits.
22
Survey question: “Because of my AYV experience, I have:”
23
The team opted for post-program measurement alone because many partners had already started programming,
therefore pre- and post-program measurement was not possible.
24
Survey question: “How confident are you in your ability to do the following things?”

15
Measuring Program Outcomes: Mission Measurement’s Methodology
Pragmatic and Timely Survey Sampling
Considering time, resources, and varied partner-level implementation schedules, Mission
Measurement engaged the AYV Team to devise a pragmatic survey sampling strategy to
measure AYV’s impact. The strategy aimed to ensure maximum participation of AYV educators
and youth globally and facilitate consistent, quick feedback from partners.
In order to evaluate a program of this geographic scope and implementation complexity, Mission
Measurement utilized AYV’s existing resources across partners and program sites. The
Evaluation Team worked with AYV partner coordinators to a) link survey responses to
appropriate partners and geographic locations, and b) match youth responses to surveys of
respective educators, when applicable.
Through continuous engagement, Mission Measurement was able to conduct the most
comprehensive surveying of AYV educators and youth to date. Program coordinators
successfully directed educators and youth participants from all 16 program year 7 partners, at
over 300 program sites in more than 50 countries to the surveys. Mission Measurement was
able to verify and analyze 3,603 youth responses (~10% response rate) and 490 educator
responses (~44% response rate).25
Though lower than desired, these survey response figures
were triple that of previous years.26
Additionally, Mission Measurement was able to match a subset of the youth survey sample with
educators who submitted a post-program survey. Of the 3,603 youth who completed the survey,
76% were matched to their corresponding educators. The ability to match these surveys offered
deeper insights into links between program implementation and resulting youth outcomes.
Despite success reaching an international audience (400 youth at international AYV sites
completed the post-program survey, with 300 of these youth matched to an educator who also
participated in the evaluation process), Mission Measurement notes that nearly 87% of all youth
responses were generated at domestic sites. Given the overrepresentation of U.S. sites in the
survey, trends in outcomes by site and/or partner should be interpreted bearing this limitation in
mind.
Data Cleaning, Coding, and Analysis
Mission Measurement utilized an easily accessible and cost effective online survey host and
cleaned survey data in collaboration with program coordinators, who helped confirm active
versus non-active educators. To clean the entire dataset, the Evaluation Team took steps such
as removing responses from unconfirmed educators and their youth and removing incomplete
or inconsistent responses. The Evaluation Team subsequently examined relationships between
different variables and assessed trends in implementation and outcomes by running a series of
frequencies, cross-tabulations, and Chi-Square tests.
25
Youth response rate calculated from expected program reach numbers, as estimated using educator reports of
youth taught in program year 7. Educator response rate calculated from coordinator-provided lists.
26
In the prior year, analysis was conducted on 143 educator responses and 1,087 youth responses.

16
Key Evaluation Findings
Data from the three highlighted youth outcomes, coupled with educator and program reports on
implementation, begin to tell the story of AYV’s impact in its seventh program year.27
The team found that AYV reached over 30,000 youth in its 7th
program year and that 90% of
AYV educators reported working with youth for more than three hours per week. These
educators worked with youth to foster media making skills in a variety of settings (in- and out-of-
school), sometimes integrating curriculum into their existing courses, and in other cases,
providing separate classes and workshops to engage youth.
Despite varied levels of technology and internet access at the site-level, the team found that a
high percentage of youth program participants around the world reported completing three key
stages of media making (i.e., brainstorming, selecting audience, and reviewing and editing
media). Notably, over 90% of youth practiced brainstorming and over two-thirds of youth
selected an audience or revised their work. Youth completed and submitted over 2,000 media
projects to Adobe’s online community. Additionally, nearly 60% of AYV youth reported practicing
four key media literacy skills (i.e., recognizing audience, message, technique, and quality).
Approximately 51% and 62% of AYV youth surveyed think about message and quality when
viewing media, respectively.
More importantly, 87% of surveyed AYV youth reported being confident or somewhat confident
across the five creative confidence dimensions (ability to ideate, collaborate, be flexible, persist
through challenges, and self-express). Nearly 80% of AYV youth reported that their participation
in the program changed their behavior in their school, career, and/or community life. Taking the
opportunity to share AYV’s impact, youth offered:
• “I am able to come up with ideas… [AYV] will help me to go beyond what I would
normally think of to help out with simple but complex problems.”
• “I learned how to work as a group, and learn from other people's ideas. I think that, I
had learned much more people skills.”
• “Using the storyboards, video camera and the supplies around me helped me to create a
video for a book…We learned how to do things with the video camera and learned to be
creative by using the items around me.”
• “I had to learn more about my topic, so I learned to do some research for any project in
my life and that it is important to believe in yourself.”
• “I now know that I can express myself through my creations, rather than just staying
bottled up about things…this helped me to learn that it's okay to open up to other people
about the problems that I think are important.”
27
AYV educators responded to a post-program survey, which included questions about program inputs, technology
and internet availability.

17
These findings suggest that educators are fostering more than just media making skills: they are
also supporting skill development that can be applied outside of the program. As quoted above,
a common theme for youth across many partners was the use of media as a call to action about
issues of personal and community significance. Globally, youth in AYV use media to share
perspectives on issues ranging from promoting human rights to preserving the environment,
among others.28
Encouraging Creativity among Youth in Diverse Learning Environments
Outcomes data also provided a platform to begin exploring the connection between AYV
program implementation and youths’ development of technical skills, creative confidence, and
application of creative confidence to their daily lives.
As aforementioned, while AYV has a common curriculum, partners are given the flexibility to
implement the program in and outside of school, with different frequencies, durations, and
dosages. These implementation models contribute to the variations observed between partner
grantees. Specifically, there are differences in the levels of access to resources and tools (i.e.,
cameras, computers, etc.), and in the degree to which educators maintain fidelity to the AYV
curriculum and recommended practices. Though findings were mixed given inconsistent
response rates across partners, the team identified several preliminary findings to be further
explored in subsequent evaluation cycles.
For example, the team found that youth served by programs outside of the regular school day
reported stronger outcomes on the post-program survey.29
These youth were more likely to
report a) completion of all stages of the media making process, b) consistent use of media
literacy skills, c) confidence across all creative confidence dimensions, and d) impact on their
educational, career, or community engagement.
The Evaluation Team suspects that this trend may follow from a combination of factors. Out-of-
school programs may offer more scheduling flexibility and/or additional time specifically devoted
to AYV curriculum. Another possibility may be that the students attending programs outside of
school may self-select into these programs because they offer an outlet for creativity, which
arguably remains uncommon in existing academic curricula. Although these findings did not
prove significant through the Evaluation Team’s initial Chi-Square analysis, Mission
Measurement hypothesizes that the larger, more geographically diverse sample for its next
round of evaluation will provide deeper insights.
Within a flexible implementation model, the linkages between program type and outcomes
suggest that understanding youth interest and motivation may be a key factor in driving primary
program outcomes.
28
For a global depiction of youths’ media themes:
http://youthvoices.adobe.com/summit2013/img/AYV_Infographic_v18alt.pdf
29
Based on program setting information documented by AYV. The analysis included a comparison of partners which
run programming exclusively outside school ours against those that run programming exclusively in-school. Note,
some partners use both in-school and out-of-school approaches.

18
Stronger youth outcomes were also observed in programs taught by “high fidelity” educators -
educators who reported that over 50% of the youth in their program selected the topic of their
media project, used media tools, and completed a media project. The team observed that youth
taught by these educators reported higher levels of a) confidence across the creative
confidence dimensions, b) program impact on their educational, career, or community
engagement, c) consistent use of media literacy skills, and d) completion of all stages of the
media making process. These findings were also not found to be statistically significant, but did
show some trends that helped AYV think more broadly about the role that fidelity might play in
the larger, more diversified sample in the eighth program year.
So What? Lessons Learned
Ultimately, the AYV engagement allowed Mission Measurement to:
1) Advise AYV on how to standardize desired youth and educator outcomes, as
measured in a post-program survey;
2) Engage the AYV Team in a series of discussions regarding the degree to which the
program was achieving intended outcomes (discussions included possible
explanations for achievements and constraints);
3) Offer data-driven recommendations on which partners and/or geographic sites could
benefit from various types of additional support;
4) Provide feedback on inputs, outputs, and outcomes that may benefit from the
addition of targeted routine monitoring alongside the existing summative evaluation;
and
5) Utilize survey data to generate other exploratory analysis, including estimates of
overall portfolio reach and cost per youth outcome at the partner level.30
The development and implementation of this evaluation process for AYV yielded a multitude of
lessons relevant to program leaders and evaluators.
In relation to program development, the team found that:
1) The concept of “creative confidence” –the ability to harness creative skills to solve
problems - has become increasingly relevant as nonprofits and businesses look to
foster applied creativity. As leaders develop their program strategy, they can benefit
from a focus on a wider range of skills and competencies related to the capacity for
ideation, including the capacity for collaboration, persistence, self-expression, and
flexibility.
2) Programs that provide opportunities for participatory media and digital production may
serve as a particularly promising mechanism for developing youth creative

19
confidence. 31
Mission Measurement’s evaluation of the AYV program found that it
successfully fostered youth self-expression and prompted positive behavioral changes
including learning about and trying to address social issues. In the upcoming year, the
team will have the opportunity to conduct pre- and post- program assessment to more
rigorously monitor and evaluate youth growth across creative confidence dimensions
globally. However, preliminary research suggests that AYV’s youth-centered model
and emphasis on media making as a means for creative self-expression supports
positive youth outcomes across diverse modes of partner-level implementation.
Methodologically, the team found that:
3) Evaluators tasked with measuring the impact of programs designed to foster creativity
must begin their work by understanding the full range of intended program or initiative
outcomes. Creativity can be strictly defined as a capacity to generate original ideas,
but more often programs are also designed to foster critical thinking skills, problem
solving skills, or a host of other skills and abilities. Understanding the purpose and
scope of a program is a crucial first step to effective monitoring and evaluation.
4) Beyond collaboration during the initial phase of evaluation design, evaluators and
program leaders can benefit from engagement throughout the monitoring and
evaluation process. The joint development of competency scales, indexes and/or
benchmarks can support articulation of program goals and ensure that evaluation is
informed by experienced practitioners.
5) An understanding of partner needs and continuous engagement throughout the
survey fielding process can yield improved response rates. Mission Measurement
more than tripled the number of youth and educator responses to AYV’s post-program
surveys by ensuring its surveys were translated and providing program coordinators
with regular updates on survey completion.
6) A clear understanding of intended outcomes and universal impact indicators serve as
a means to evaluate a diverse portfolio of grantees. By using standardized
competency scales, Mission Measurement was able to measure and compare youth
and educator outcomes achieved by a diverse set of partners. This work informed
both program design and management approaches for Adobe Foundation.
As shown through Mission Measurement’s work with AYV, foundations can, and should expect
rigorous evaluation methods to support anecdotal data collection. This evaluation approach can
be pursued regardless of the challenges that come with a diverse portfolio of grantees.
31
Kafai, Y. and Peppler, K (2011). Youth, Technology, and DIY: Developing Participatory Competencies in Creative
Media Production. Review of Research in Education: 35(89).

20
Next Steps
By establishing a collaborative relationship with the AYV Team and implementing partners,
Mission Measurement has developed a platform for ongoing evaluation and engagement with
participants who are committed to measuring and benchmarking their progress.
Mission Measurement is now in the process of evaluating AYV’s eighth program year and has
developed and fielded pre- and post-program surveys for educators and youth. This tracking of
growth over time will supplement the continued process of matching youth surveys to those
from respective educators. Mission Measurement will also continue to explore linkages between
program implementation and high-level outcomes to determine if particular program practices
are linked to the development of creative confidence characteristics.
Finally, in an effort to fully capture holistic and long-term impact on AYV youth, the Evaluation
Team is working with AYV to explore ways to reach and engage AYV alumni to determine if and
how the AYV program has impacted their long-term educational and career aspirations, as well
as their level of community engagement. These evaluation efforts will continue to add to the
growing pool of research promoting more robust approaches to monitoring and evaluation of
creative confidence.

21
Appendix A:
Table 3: Current Adobe Youth Voices Partners
Partner Domestic /
International
Description
1
st
ACT Silicon
Valley
Domestic 1stACT Silicon Valley is a network of business, civic and arts leaders working
to foster cultural engagement and help create an authentic sense of place and
cultural identity for Silicon Valley. As a network of leaders, 1stACT's focus is to
make catalytic change across three areas: 1) Silicon Valley Arts & Culture -
support and strengthen arts and culture industry by providing a pipeline of
leadership, strategic investment funds, marketing and branding, and capacity
building; 2) Cultural Engagement - broaden the range of opportunities for
Silicon Valley workers, residents, children, and families to become engaged in
participatory cultural experiences; and 3) Vibrant Urban Core - realize a vision
of Downtown San Jose as Silicon Valley's urban core, which is anchored in big
deals and small wonders across five planning domains. http://www.1stact.org/
American India
Foundation
(AIF)
International The American India Foundation (AIF) is the largest diaspora philanthropy
organization focused on India and based out of the United States. Since
inception, AIF has benefited more than 1 million people, by investing in over
115 Indian non-governmental organizations. AIF partners with high quality
Indian NGOs to implement scalable and sustainable programs that
demonstrate effective solutions. In the process it builds a new generation of
social entrepreneurs and civil society leaders. The foundation works in tandem
with government agencies to translate these solutions into policy change. AIF
works out of New York and California in the US, and New Delhi and Bangalore
in India. In addition to its offices in the United States, AIF has eight chapters in
major metropolitan areas in the US. http://aif.org/
Bay Area
Video Coalition
(BAVC)
Domestic The Bay Area Video Coalition (BAVC) inspires social change by enabling the
sharing of diverse stories through art, education and technology.
http://www.bavc.org/
Chicago Public
Schools (CPS)
Domestic Chicago Public Schools, the third largest school district in the United States,
offers high quality world class education for the 21st century. Our mission is to
ensure that every child is on track to graduate prepared for success in college,
work and life. http://www.cps.edu/
Eastside
Educational
Trust
International Eastside Educational Trust is a highly acclaimed, award-winning, world record-
breaking arts education charity and London’s leading youth arts and media
provider. Eastside aims to educate and engage young people through direct
participation in the arts and media. We want to inspire all young people to
develop their potential, share their talents and enrich their lives and we do this
by enabling young people to work with professionals in neighborhoods where
are help is needed the most. http://www.eastside.org.uk/
Free the
Children (FTC)
International Free The Children is the world’s largest network of children helping children
through education, with more than one million youth involved in our innovative
education and development programs in 45 countries. Founded in 1995 by
international child rights activist Craig Kielburger, Free The Children has a
proven track record of success. The organization has received the World’s
Children’s Prize for the Rights of the Child (also known as the Children’s Nobel
Prize), the Human Rights Award from the World Association of Non-
Governmental Organizations, and has formed successful partnerships with
leading school boards and Oprah's Angel Network.
http://www.freethechildren.com/
International
Education and
Resource
International EARN is a non-profit organization made up of over 30,000 schools and youth
organizations in more than 130 countries. iEARN empowers teachers and
young people to work together online using the Internet and other new

22
Network
(iEARN)
communications technologies. Over 2,000,000 students each day are engaged
in collaborative project work worldwide. http://www.iearn.org/
ReelGrrls Domestic Reel Grrls empowers young women from diverse communities to realize their
power, talent and influence through media production. Our mission is to
cultivate voice and leadership in girls at a vulnerable age in their development.
We offer a variety of hands-on workshops for teenage girls in specific skills
including animation, cinematography, script writing and more.
http://www.reelgrrls.org/
Spy Hop
Institute for
Teachers
(SHIFT)
Domestic Spy Hop, SHIFT formerly known as the Interactive Digital Education Academy
(iDEA), offers K-12 educators immersive, high-quality, innovative professional
development and support to integrate the filmmaking process into core content
instruction. Participants receive a comprehensive, standards-linked filmmaking
curriculum and learn how to produce Public Service Announcements (PSAs),
instructional videos, stop motion projects, and other short films.
http://www.shiftworkshops.org/
TakingITGlobal
(TIG)
International TIG's missions is for youth everywhere actively engaged and connected in
shaping a more inclusive, peaceful and sustainable world. The organization's
mission is to enable a collaborative learning community which provides youth
with access to global opportunities, cross-cultural connections and meaningful
participation in decision-making by harnessing internet technologies to
cultivate youth leadership and engagement in social issues; bridging the
continuity gap that causes fragmentation between and within generations of
youth movements; championing the role of young people as key stakeholders
in all aspects of society. TIG facilitates global understanding and grow
leadership among youth to enhance their participation in social movements for
a better world. http://www.tigweb.org
TechBoston
within Boston
Public Schools
(BPS)
Domestic TechBoston is a department within the Boston Public Schools that supports
advanced technology courses in the district's schools. TechBoston's mission is
to inspire Boston Public School students by providing access to cutting edge
technology resources and opportunities that positively advance their academic
and career aspirations. http://www.techboston.org/
TechSpace International Inspiring young people to become digital creators through free access to
hardware, software, trained educators and creative mentors.
http://www.techspace.ie/
The Intel
Computer
Clubhouse
Network
International The Intel Computer Clubhouse Network
The Intel Computer Clubhouse Network is an international community of 100
Computer Clubhouses located in 20 different countries around the world. The
Computer Clubhouse provides a creative and safe out-of-school learning
environment where young people from underserved communities work with
adult mentors to explore their own ideas, develop skills, and build confidence
in themselves through the use of technology.
http://www.computerclubhouse.org/
Urban Arts
Partnership
(UAP)
Domestic Urban Arts Partnership advances the intellectual, social and artistic
development of underserved public school students through arts-integrated
education programs to close the achievement gap. Urban Arts Partnership
accomplishes its mission through a variety of interdisciplinary arts programs
that include in-school classroom integration, after-school programs, master
classes, professional development, summer programs, arts festivals, and
special projects. http://www.urbanarts.org

Measuring the Case for Creativity

More Related Content

What's hot (12)

Viewers also liked (18)

Similar to Measuring the Case for Creativity (20)

Measuring the Case for Creativity