Tony Bryk - Bristol - Joining Improvement Science to NICs
3 likes5,090 views
The document discusses the need for improved educational practices through the integration of improvement science and networked communities to tackle complex systems. It emphasizes the importance of problem-specific, user-centered approaches to enhance learning outcomes while addressing performance variability. The goal is to create a more effective, measurable, and scalable educational system through disciplined inquiry and collaborative networks.
Tony Bryk - Bristol - Joining Improvement Science to NICs
- 1. Anthony S. Bryk University of Bristol May 21, 2014
- 2. 2 Triple Aims of Educational Improvement EFFICIENCY EFFECTIVENESS ENGAGEMENT Be0er Use of Resources Deeper Learning For All Students More Relevant
- 3. Last Decade: Performance Management Using Accountability to Drive Change – Set targets – Create incentives – Collect data/dashboards – Hold individuals accountable “Go figure it out or else…”
- 4. Last Decade: Evidence-based Practice Movement An academic has an idea He/she design and fine tunes an intervention An RCT field trial (5 years later) Evidence it can work Reviewed by What Works Clearing House Goes on an “approved list” Districts required or “incented” to buy only from approved list Educators “Implement with Fidelity” Practice Improves! But there is a problem, actually many problems.
- 5. Last Decade: Communities of Practice 5
- 6. Great Dynamism, But PuLng All of This Together 6 Resembles a Tower of Babel We have to find a be0er way to accelerate learning to improve.
- 7. I. Analogical Scavenger in hot pursuit!
- 9. 9 We can accomplish more together, than even the best of us can do alone. Complex systems problems that we now seek to solve
- 10. A Better Way: Integrating Two Big Ideas • Improvement Science Disciplines Efforts joined to • Structured Networks that Accelerate the Learning A Shi& Toward Learning Fast to Implement Well.
- 11. II.
- 12. Two Networked Improvement Communities ! 1. Extraordinary high failure rates in developmental mathematics courses in community colleges—a gatekeeper to opportunity. – Can colleges dramatically increase the success of these students to and through college-level math in one year of instruction? ! 2.The weak and incoherent supports by which individuals learn to teach in public school settings. – Can districts learn how to support new teachers to learn faster, better and hold on to them?
- 13. 13 III. Forming As a Community: Building Education’s Capacity to Improve
- 14. The Six Core Principles 14 ! 1. Make the work problem-specific and user-centered. ! 2. Variation in performance is the core problem to address. ! 3. See the system that produces the current outcomes. ! 4. We cannot improve at scale what we cannot measure. ! 5. Anchor practice improvement in disciplined inquiry. ! 6. Accelerate improvements through networked communities.
- 15. Taken Together 15 • Disciplined Inquiry • Rudiments a scientific community • Aim: systematic practice improvement
- 16. The Six Core Principles 16 ! 1. Make the work problem-specific and user-centered. ! 2. Variation in performance is the core problem to address. ! 3. See the system that produces current outcomes. ! 4. We cannot improve at scale what we cannot measure. ! 5. Anchor practice improvement in disciplined inquiry. ! 6. Accelerate improvements through networked communities.
- 17. How Do We Heal Medicine? Atul Gawande April, 2012 See the System
- 18. 18
- 19. Gawande’s Closing Observation Making systems work is the great task of my generation of physicians and scientists. But I would go further and say that making systems work — whether in healthcare, education, climate change, making a pathway out of poverty — is the great task of our generation as a whole.
- 20. And at the heart of making systems work is the problem of complexity… When you are confronted by any complex social system with things about it that you’re dissatisfied with and anxious to fix, you cannot just step in and set about fixing with much hope of helping. This realization is one of the sore discouragements …You cannot meddle with one part of a complex system without the almost certain risk of setting off disastrous events that you hadn’t counted on. If you want to fix something you are first obliged to seek to understand it…the whole system. -‐ Lewis Thomas, 1974 Biologist and Essayist
- 21. What We Should Work On: Task complexity Organizaonal complexity Complexity of work in 21st century instuons Need to “see the system”
- 22. The Invisible Complexity Schooling 22 The Invisible Complexity of Schooling
- 23. What We Should Work On: Task complexity Organizaonal complexity Consequences: • Breakdowns • Wide variability in outcomes Complexity of work in 21st century instuons Need to “see the system”
- 24. What We Should Work On: Task complexity Organizaonal complexity Consequences: • Breakdowns • Wide variability in outcomes Need to idenfy high-‐ leverage problems Design/develop/ refine quality work processes Complexity of work in 21st century instuons Need to “see the system”
- 25. I. Problem-‐ & User-‐Centered • What we tend to do now: a general issue comes into view and we jump on soluons. ! What is the specific problem we’re trying to solve?
- 26. 60-‐70% Students assigned to developmental math course. 80% Percent of these students that never get past this gate. 500,000 students in every cohort will never complete a college math requirement. 26 A Specific High-Leverage Problem to Solve
- 27. What We Should Work On: How We Work On It: Task complexity Organizaonal complexity Consequences: • Breakdowns • Wide variability in outcomes Need to idenfy high-‐ leverage problems Design/develop/ refine quality work processes Inherent indeterminism Must learn our way into improvement; “Change it to understand it.” Complexity of work in 21st century instuons Need to “see the system”
- 28. V. Engage in Disciplined Inquiry • Design-‐development orientaon, iterave cycles – The Driving Principle: Quick, minimally intrusive, and an empirical warrant – Mantra: Learn Fast, Fail Fast, Improve Fast!
- 29. Framing the Learning to Improve Challenge Current Situation Resistant Indifferent Ready Low confidence: Good idea, But how to make it work ???? Limited Capacity Very Small Scale Test Very Small Scale Test Very Small Scale Test Good base Capacity Very Small Scale Test Very Small Scale Test Small Scale Test High confidence: Good idea èè Can execute Limited Capacity Very Small Scale Test Small Scale Test Large Scale Test Good base Capacity Small Scale Test Large Scale Test Implement
- 30. Framing the Learning to Improve Challenge Current Situation Resistant Indifferent Ready Low confidence: Good idea, But how to make it work ???? Limited Capacity Very Small Scale Test Very Small Scale Test Very Small Scale Test Good base Capacity Very Small Scale Test Very Small Scale Test Small Scale Test High confidence: Good idea èè Can execute Limited Capacity Very Small Scale Test Small Scale Test Large Scale Test Good base Capacity Small Scale Test Large Scale Test Implement
- 31. Framing the Learning to Improve Challenge Current Situation Resistant Indifferent Ready Low confidence: Good idea, But how to make it work ???? Limited Capacity Very Small Scale Test Very Small Scale Test Very Small Scale Test Good base Capacity Very Small Scale Test Very Small Scale Test Small Scale Test High confidence: Good idea èè Can execute Limited Capacity Very Small Scale Test Small Scale Test Large Scale Test Good base Capacity Small Scale Test Large Scale Test Implement
- 32. Framing the Learning to Improve Challenge Current Situation Resistant Indifferent Ready Low confidence: Good idea, But how to make it work ???? Limited Capacity Very Small Scale Test Very Small Scale Test Small Scale Test Good base Capacity Very Small Scale Test Small Scale Test Modest Scale Test High confidence: Good idea èè Can execute Limited Capacity Small Scale Test Modest Scale Test Large Scale Test Good base Capacity Modest Scale Test Large Scale Test Implement Wide-spread
- 33. 33 Of Educational Reform Going Fast, but Learning Slow.
- 34. 34 We Are in Good Company Shil to Learning Fast to Implement Well
- 35. What We Should Work On: How We Work On It: Task complexity Organizaonal complexity Consequences: • Breakdowns • Wide variability in outcomes Need to idenfy high-‐ leverage problems Design/develop/ refine quality work processes Inherent indeterminism Must learn our way into improvement; “Change it to understand it.” Complexity of work in 21st century instuons Need to “see the system”
- 36. Developing a Quality Process Reliably at Scale Develop A Change Test under mulple condions Test under increasingly varied condions Make the change permanent Ini@al Hunches System Changes 1 school 1 administrator 5 schools Many administrators Enre vercal team A more diverse group of administrators District Wide All administrators Seeing Task Complexity Seeing Organizaonal Complexity Learning to improve feedback conversaons between principals and new teachers PLAN DO ACT STUDY
- 37. What We Should Work On: How We Work On It: Task complexity Organizaonal complexity Consequences: • Breakdowns • Wide variability in outcomes Need to idenfy high-‐ leverage problems Design/develop/ refine quality work processes Inherent indeterminism Must learn our way into improvement; “Change it to understand it.” Can’t always see all the consequences of what we do. Centrality of measurement: How will you know Δ is an improvement? Complexity of work in 21st century instuons Need to “see the system”
- 38. You Cannot Improve at Scale What You Cannot Measure • Need measureable targets – But, you just can not stand at the end of the line. • We need process measures ed to intermediate targets and key process changes.
- 39. TraditionalSequenceStatway Effects: Time to Complete a College Level Math Course 1 Year 2 Years Triple the success rate in half the time. 6% 51% 15%
- 40. Producve Persistence Belonging Uncertainty: Supporve social es Target: How do we measure it? Mindsets about the value of math Mindsets about potenal to learn math Anxiety Regulaon Study Skills Conceptual Task: reduce 40+ concepts to 5 core ideas focus on underlying malleable causes + change evidence Prac@cal Measurement: reduce 900 items to 26 “you have 3 minutes” A Primary Driver: key intermediate outcome
- 41. 41 ! Testing a Change Idea: A Starting Strong Package to Enhance Productive Persistence
- 42. What We Should Work On: How We Work On It: Task complexity Organizaonal complexity Consequences: • Breakdowns • Wide variability in outcomes Need to idenfy high-‐ leverage problems Design/develop/ refine quality work processes Inherent indeterminism Must learn our way into improvement; “Change it to understand it.” Can’t always see all the consequences of what we do. The Goal: Quality reliably at scale • Can we get an idea to work? • What will it take to make it work in many other contexts? Centrality of measurement: How will you know Δ is an improvement? Complexity of work in 21st century instuons Need to “see the system”
- 43. II.Variation in Performance Is the Problem to Solve • Crical queson is not: “What Works?” But rather: “How to advance quality among diverse teachers engaging varied populaons of students and working in different organizaonal contexts?” Goal: Achieve quality reliably at scale.
- 44. Belonging Uncertainty and Stereotype 12% 13% 14% 28% 40% 7% 11% 14% 50% 71% 0% 10% 20% 30% 40% 50% 60% 70% 80% Never Hardly Ever Somemes Frequently Always Pathways Dropout All students Black students “How olen, if ever, do you wonder: ‘Maybe I don't belong here?’” N = 714 math students
- 45. Variation in Pathways Success Rates by College (n=19) 45 1 23 4 5 6 7 8 9 11 1213 14 15 17 18 19 0% 50% 100% 0% 50% 100% StatwayStudents Non-Statway Matched Comparisons No improvement line We also have a failure, why? What can we learn? Triple success rate line
- 46. VI. Accelera@ng Improvement: Tap the Power of Networks • A source of innovaon • Mulple fast tests/refinements • Improvement diffusion A Learning Educa/onal System
- 47. A A Improvement Networks: Accelerate Learning in Pracce for Improvement A B A A A B A A A B A A A B C (Englebart,1994)
- 48. A System of Social Learning to Improve: Embracing Disciplined Inquiry Transla@onal Research Interven@ons (Alpha Labs) Will it work in our context and with out students, and if so, how? Expert Prac@@oner Knowledge (Subnet) Building robust clinical knowledge about effecve materials and instruconal pracces (PDSA). Learning from Network Data (Hub Analy@cs) Learning from observed variability. Discerning the unseen.
- 49. The Network Improvement Paradigm 49 Researchers vs Users “Knowers” “Doers” All Improvers What Works! How to Make It Work! Replicability as the new Gold Standard. “Script it” vs. “Every situation is unique” Develop Quality Processes to Support Complex Work Individual Autonomy As Most Prized Norm Working Together We Can Accomplish More " Implement Fast and Scale Wide Learn Fast to Implement Well " Focus on Standard Effect Size Focus on Sources of Variability in Performance " " " " Developing Practice-Based Evidence
- 50. The Network Improvement Paradigm “The problem that is managing quality is not just an intellectual endeavor; it is a pragmatic one.The point is not just to know what makes things better or worse; it is to make things actually better.” –Dr. Don Berwick, Founder Institute for Healthcare Improvement Learning Fast to ImplementWell to Achieve Quality Reliably at Scale. 50
- 51. It is all about accelerating how we learn in and through practice to improve. Our thanks to our foundation partners in this work: Carnegie, Gates, Hewlett, Kresge and Lumina. Also the Institute of Education Sciences and the National Science Foundation