Tools for Measuring Place-Based Systems Change
Download as PPTX, PDF3 likes717 views
This document discusses tools for measuring place-based and systems change initiatives. It emphasizes that evaluating systems change requires understanding events, patterns, and systemic structures. Frameworks presented measure changes in behaviors, relationships, policies, communication and resource allocation across stakeholders. Effective evaluation orients towards learning, involves community perspectives, and monitors changes at multiple levels over time.
Tools for Measuring Place-Based Systems Change
- 1. Tools for Measuring Place-Based Systems Change Christina Garcia, The James Irvine Foundation Justin Piff, Equal Measure May 31, 2017 Grantmakers for Effective Organizations Learning Conference
- 2. Goals Participants will: 1. Learn about the “systems mindset” needed to evaluate (and learn from!) systems change 2. Understand frameworks for evaluating systems change 3. Develop a plan for applying learning at home
- 3. "As someone whose job it is to address the end result, how can you not care about the beginning?" -Dan Barber Chef and author of The Third Plate
- 4. Introductions • Have you invested in place-based initiatives? • Have you invested in systems change initiatives? • Are you currently evaluating (or have you evaluated) these efforts?
- 5. The Systems Change Umbrella Collective Impact Comprehensive Community Initiatives Cross-Sector Partnerships Community Coalitions Community Development Community Partnerships
- 6. A Systems Change Mindset I want change and I want it now! (But yesterday would be nice!) Tackling and evaluating systems change requires the right mindset!
- 7. A Systems Change Mindset What traits or characteristics do funders and evaluators of systems change need?
- 8. A Systems Change Mindset Conditions for successful systems change evaluation • Orientation towards learning • Comfort with “process” and iteration • Cultural competency and community responsiveness
- 9. A Systems Change Mindset: What’s the difference? Just prove our strategy works. How can what we’re seeing improve what we do? Don’t deviate from the workplan. That question’s not important anymore, let’s adjust our approach. Let’s ask the board of directors how things are going. Let’s ask the students if they’re seeing a difference. I want to see client outcomes at scale. This work is complex, what are some signs that systems are changing? The “norm” Systems thinker
- 10. Articulating Systems Change Events Patterns Systemic Structures What we see What our data reveal What we want to change
- 11. How do we know when a system has changed? Events Patterns Systemic Structures Better Best Good We seek changes at all three levels
- 12. Systems affecting postsecondary success Events Patterns Systemic Structures Low postsecondary attainment rates Low rates among: Students from select school districts Students of color Students who attend community college First-generation college students Discriminatory disciplinary practices Assumption of “college going knowledge” Outdated/misaligned curricula or course sequencing
- 13. Small group exercise Identify an “event” you’re trying to change: • Employment • Academic outcomes • Health outcomes • Civic engagement What patterns/trends do you see among these events? • Population • Location • Timing What “systemic structures” are contributing to these patterns? • Individual or organizational relationships • Policies or practices • Incentives • Power structures • Communication/Information flow
- 14. Who determines how the system is functioning?
- 15. How do we evaluate these systems changes?
- 16. Key Question: What does evidence of systemic change look like? Observable indicators of embedded, sustainable changes to systemic structures: • Behaviors • Relationships • Policies or Practices • Flow of information/communication • Resource allocation Capturing changes among systemic structures Or else they won’t “stick”
- 17. Guidelines for data collection • Qualitative: Use interviews, focus groups, visuals, and other qualitative methods when you aren’t sure what the changes look like, or when you want to understand diverse and nuanced experiences with the system. • Quantitative: Capture, count, and track evidence – or signs – of systems changes when you know what you’re looking for. Capturing changes among systemic structures
- 18. REGIONAL HUBS OF EXCELLENCE Area players are connected to do — and gain — more together. PARTNER COMMUNITY PARTNER PARTNER COMMUNITY
- 19. Understand the formation and development of Linked Learning regional hubs to support learning Identify best practices, strategies, and lessons that can inform program strategy and development of field standards Provide support and insights to the James Irvine Foundation, Jobs for the Future, and Linked Learning hubs to support the success of the initiative Multi-case study, mixed-methods approach DE Learning- oriented Adaptable Responsive Linked Leaning Regional Hubs: Developmental Evaluation
- 20. Leadership & Partnership Development • Shared vision • Partnership composition • Governance • Partner roles and activities • Relationship-building • Commitment-building • Ways of working together • Understanding of regional issues Systems Change • Impact of regional partnerships on 1) deepening, 2) scaling, 3) aligning, and 4) adopting Linked Learning across sectors • Strategies and policy levers advancing Linked Learning • Areas that hub- initiated practice and policy changes offer the greatest potential for large-scale changes in student experiences Interacting Factors • Context of each region in shaping design, implementation, and effectiveness of hubs • Landscape of education and workforce partners influencing the work of the hubs • Alignment with parallel initiatives and/or pre-existing partnerships • Role of investment and technical assistance Linked Leaning Regional Hubs: Developmental Evaluation
- 21. Aspen Opportunity Youth Incentive Fund 21 communities across the US focused on improving outcomes for “opportunity youth” Key Question: “How does implementing a collective impact approach contribute to systemic shifts in communities that improve educational, work, and life outcomes for opportunity youth?” Annual partnership-level survey • 9 “systemic shifts” that demonstrate the system is changing to better support opportunity youth • Evidence of “systemic shifts” based on partner behaviors A series of site interviews and visits
- 22. Quantifying Systems Change for the Aspen Opportunity Youth Incentive Fund More effective integration of programs and organizations into pathways serving opportunity youth What changes among the system do you expect to see? What would tell you this has happened? Organizations implement shared intake forms Organizations use the same data systems to track youth participation across programs Staff from different organizations train one another Staff from organizations of different sectors train one another Organizations use the same assessment system to assess youth service needs
- 23. Quantifying Systems Change 57% 65% 40% 65% 67% 48% 0% 10% 20% 30% 40% 50% 60% 70% 80% Community 1 Community 2 Community 3 Year 1 Year 2 Effective integration of programs and organizations into pathways serving opportunity youth
- 24. National Evaluation of StriveTogether About 20 cradle-to-career partnerships from across the US Key Question: “How, and to what extent, have communities across the StriveTogether Cradle to Career Network built their civic infrastructure?” 83 Questions capturing partner perceptions of civic infrastructure in 9 areas • 5-point scale • Multiple perspectives re: experience in the community • What we would expect to see based on StriveTogether’s Theory of Action More than 2,500 surveys completed over 2 years
- 25. StriveTogether: Measuring approaches to systems change 70% 65% 63% 64% 60% 59% 55% 53% 48% 2016 2015 Shared Vision Data Use Partnership Partnership Community Capacity Structure Engagement Partner Partnership Partnership Practice Commitment Action Communication Change
- 26. Common Pitfalls in Assessing Systems Change Thinking you found the silver bullet Naval gazing Celebrating event-level changes while not changing systems Forgetting to look at patterns – changing overall event numbers may not address disparate outcomes for key groups
- 27. Small group exercise Identify one or two “systemic structures” you want to see change What evidence would you see that these structures have changed? • Behaviors • Relationships • Polices or Practices • Communication/Flow of information • Allocation of resources • Changes in knowledge or awareness Whose perspective should inform whether the system is changing?
- 28. Investing in and evaluating systems change requires a different mindset Evaluations should focus on learning and understanding Monitor events, patterns, and systemic structures Stakeholders will help define and understand systems change In Summary
- 29. GEO’s Systems Grantmaking Resource Guide: http://systems.geofunders.org/ Introduction to Systems Thinking, Daniel H. Kim: https://thesystemsthinker.com/wp- content/uploads/2016/03/Introduction-to-Systems-Thinking-IMS013Epk.pdf Three Keys to Unlocking Systems-level Change: https://ssir.org/articles/entry/three_keys_to_unlocking_systems_level_change Resources and Tools
Editor's Notes
- #2: Justin
- #3: Justin (funder asked me to put this in)
- #4: Justin This quote is from an unlikely source and a great example of systems thinking and how “place matters” How and where we grow and cultivate our food, including the crops we surround it with, can literally make it taste better.
- #5: Justin
- #6: Christina
- #7: Christina Provide examples of what each looks like from a grantmaking and evaluation perspective.
- #8: Christina to lead, Justin to take notes on flip chart Use flip chart to have folks identify the traits and briefly discuss/explain. Participants can offer suggestions for evaluators, funders, or both.
- #9: Christina Briefly explain each of these as they would apply to both the funder and the evaluator.
- #10: Christina Walk through the different sets of statements, pointing out how the 2nd statement of each set exhibits the systems mindset necessary for systems change evaluation.
- #11: Justin Here’s a framework often used to understand systems at the root of the issues we aim to address. We borrowed this from Daniel Kim’s “Introduction to Systems Thinking” b/c it provides a useful framework for unpacking how to evaluate the kind of systems change many of us are interested in. Events, What we see – these are the kinds of problems you’re trying to address (employment, homelessness, kindergarden readiness, etc.). Patterns, What our data reveal – We engage in systems change because the system isn’t working for some subpopulation/subset of people. Here, we need to unpack the “events” more to understand what patterns we’re dealing with (e.g., people of a certain status, age, socie-economic standing, georgraphy/neighborhood). Systemic structures – these are hardest to see, but these are what we really want to change when we talk about “systems change” – these are the unseen dynamics creating patters and ultimately exhibiting as the “events” we see. Things like ………………………………………..
- #12: Justin Once we’ve identified the systemic structures at play, we need to determine how to assess and understand them. Changes at all three levels are needed. Most philanthropies focus on the “events,” but we need to make sure that systemic structures are changing to ensure we’re not just applying a “band-aid” solution. But, if we only focus on “systemic structures,” we miss out on understanding whether these changes are actually having an impact. We’re going to talk more about how to evaluate changes among systemic structures.
- #13: Justin Walk through example.
- #14: Justin 8 mins. Work on, on your own. (10 minutes) Walk participants through the exercise and pass out worksheets. Using Handout: Pick an “event” you’re addressing. Identify the patterns you’ve observed and systemic structures influencing these. Get input from all participants in the group – there are no wrong answers. Worksheet will provide additional probes/discussion questions for the group to consider about the nature of systemic structures. Have a couple folks report out depending on time.
- #15: Christina…..introduce concept of including diverse stakeholder perspectives, including program/initiative beneficiaries, then direct folks back to tables to share and get input. Following up on exercise, have folks get additional input from table mates re: possible systemic structures affecting the event. (10 minutes) Have a few folks from the larger group report out to the whole group (5 minutes) Large Group Discussion: Reflecting on the exercise, ask folks to share the value of having multiple participants in their discussions? Did each bring a unique perspective to understanding the “issue?” We often engage in systems change because we think discrete, clear/cut strategies won’t work. As such, we need to understand the “system” from a variety of perspectives, including those influencing and influenced by the system. Each brings a different understanding of the issues at play, and each perspective is needed to change “the system.” Ask participants for examples of the types of people/stakeholders they would include in understanding how the system is functioning?
- #16: Justin Tee up next section. Now that we can identify systemic structures, we need to determine how to best understand/assess changes. We will provide a few examples of how we’ve approached this.
- #17: Justin 5 minutes Need to operationalize “systems change” by articulating the evidence we would see of a changed system. And, we also need some confidence that these changes will “stick” – observing one new behavior one time won’t cut it. So, we look for changes that are embedded and sustainable by nature – these have the power to affect “events” at scale. Note that most “events” have multiple systemic structures driving them. Measuring systems change needs to take these multiple factors into account (which is also why it’s so important to engage diverse stakeholders in identifying systemic structures).
- #18: Justin
- #19: Christina
- #20: Developmental evaluation approach: Is effective for initiatives that are innovative, in the early stages of development, and occur in complex shifting environments Focuses on learning, rather than accountability Is flexible and responsive Adapts learning questions and methods as the initiative evolves Identifies and shares emerging insights in a timely way Multi-case study approach Focuses on understanding each hub in its own context Focuses on identifying cross-cutting themes, strategies, and lessons Methods Interviews and discussions with anchors, key regional partners, JFF, and Irvine Site visits Anchor and key partner survey Field observations Document review Other methods, as needed (e.g., interviews and discussions with field experts, social network analysis, and spatial analysis)
- #21: Christina Animate one column at a time left to right.
- #22: Georgia
- #23: Justin
- #24: Justin Here’s how we quantify based on indicators of systems change to understand initial states and progress.
- #25: Bridget
- #26: JP Describe what we did here. Two year’s of data – remarkably consistent trends. Will hone in more on four aspects of this process.
- #27: Justin (Christina can editorialize) Naval gazing – a constant focus on systems change can leave you spinning your wheels – focusing so much on systems that you forget to look at impact. Celebrating event-level changes while not changing systems -- Need to make sure that it’s changes among systemic structures leading to changes we want, not band-aid or “one offs” Forgetting to look at patterns – changing overall event numbers may not address disparate outcomes for key groups – Critical to make sure that we look at patterns – our overall numers on things like employment, high school graduation, etc. may be improving, but are they improving equitably. It’s often underrepresented groups most affected by dysfunctional systems, so need to maintain an eye on who’s benefiting (and who’s not) from our strategies/interventions. Thinking you found the silver bullet – systems are complex – there likely is no silver bullet.
- #28: Justin Small group exercise to have participants identify the “indicators” of systems change based on their work during the first exercise Using Handout -- Have folks consider how to create evaluation and measurement approaches capturing both events and systems changes and relationship between changes in events and changes in the systems or increased clarity about those systemic issues. Depending on time, have a couple folks share what they came up with. Maybe one example. May also ask: What approach would you use to capture these changes?
- #29: Wrap-up