Continuous Improvement: How systems design can benefit the data-driven design community
1 like345 views
This document discusses how systems design can support data-driven continuous improvement in K-12 education. It is a complex technical system with many interrelated parts. Data-driven decision making has shown promise in addressing achievement gaps but requires defining goals, measures, and actions. Systems design principles like defining measurable goals, mapping the system, and incorporating feedback loops can help educational organizations implement continuous improvement. However, goals and metrics must be carefully aligned to avoid unintended consequences from data-driven processes. Overall systems design may help scale data use to support small, ongoing improvements across the education system.
Continuous Improvement: How systems design can benefit the data-driven design community
- 1. Designing for Continuous Improvement in K12 How systems design can help data-driven design in complex-technical systems
- 2. Data-Driven Continuous Improvement in K12 "Data-Driven Decision-Making” Slavin et al, 2013 1. Gather assessment data 2. Analyze data to identify problems and their causes 3. Select actions to address those problems Gather Assessment Data Analyze Data Select the Right action Slavin, R. E., Cheung, A., Holmes, G., Madden, N. A., & Chamberlain, A. (2013). Effects of a data-driven district reform model on state assessment outcomes. American Educational Research Journal, 50(2), 371-396.
- 3. Instruction & Assessment Product UX Students Paper MAGICAL IMPROVEMENTS OCCUR HERE Education Companies Data-Driven Design in K12 classrooms Assessment Data Product Data UX
- 4. What is Data-Driven Design? 1. Define “the right” goals (or needs). 2. Define “the right” measures of those goals. 3. Take actions to modify the current system when goals aren’t met.
- 5. Software Product System Product Team Reflection Product Users Product Data Reports Product Design Optimization Software Model of Data-Driven Continuous Improvement
- 6. How Might We Design Systems to Support Data-Driven Continuous Improvement? Measure Outcomes Modify Designs Continuous Improvement Loop Simple Heuristic Advice: just make it easier to measure and modify “What are the areas of need?” “What can we do about it?”
- 7. Online Product Experiments (A/B Tests): A Method for Continuous Improvement B A random assignment Which design has best effect on outcomes? Designs Users Metrics Over 10,000/day run by Google, Facebook, Amazon, etc
- 8. Continuous Improvement in K12 "Data-Driven Decision-Making” Slavin et al, 2013 1. Gather assessment data 2. Analyze data to identify problems and their causes 3. Select actions to address those problems Gather Assessment Data Analyze Data Select the Right action Slavin, R. E., Cheung, A., Holmes, G., Madden, N. A., & Chamberlain, A. (2013). Effects of a data-driven district reform model on state assessment outcomes. American Educational Research Journal, 50(2), 371-396.
- 9. Why is there desire for “Data-Driven Decision Making” (DDDM) in K12? Very few interventions have been found to be effective in addressing the poverty achievement gap. In a review of 196 randomised field experiments, Data-Driven Decision Making was found to be one of the most effective approaches for improving high-poverty schools (Fryer, 2017). Fryer Jr, R. G. (2017). The production of human capital in developed countries: Evidence from 196 randomized field experimentsa. In Handbook of Economic Field Experiments(Vol. 2, pp. 95-322). North-Holland. Van Geel, M., Keuning, T., Visscher, A. J., & Fox, J. P. (2016). Assessing the effects of a school-wide data-based decision-making intervention on student achievement growth in primary schools. American Educational Research Journal, 53(2), 360-394.
- 10. DDDM is not “disruptive” or radical change. DDDM is all about small incremental improvements… ideal for complex social- technical systems
- 11. K12 Education is a Complex-Technical System
- 12. K12 Education is a Complex-Technical System Small-Group Interactions Text Interactions Computer Interactions Teacher Interactions Administrative Interactions PLCs Family Interactions Out-of-School Programs Government Policy Education Companies
- 14. Small-Group Interactions Text Interactions Computer Interactions Teacher Interactions Administrative Interactions PLCs “Professional Learning Communities” • Curriculum Coordinators • Schools • Districts • State Admin +
- 15. Small-Group Interactions Text Interactions Computer Interactions Teacher Interactions Administrative Interactions PLCs Government Policy Education Companies • Learning Standards • Textbooks • Assessments • Paper & Digital Products
- 16. K12 Education is a Complex-Technical System Small-Group Interactions Text Interactions Computer Interactions Teacher Interactions Administrative Interactions PLCs Family Interactions Out-of-School Programs Government Policy Education Companies Opportunities from Systems Design?
- 17. Systems Design
- 18. Role of data in systems design • Systems use data as feedback to guide action. • Measurements provide data to inform systems whether goals have been achieved. • Cybernetic systems?
- 19. Teach Next Topic Fail Success Assess Mastery Mastery Learning as a cybernetic system
- 20. Operate Exit Fail Success Test for Goal Miller, G. A., Galanter, E., & Pribram, K. H. (1960). Plans and the structure of behavior. New York, NY, US: Henry Holt and Co. Closed System TEST, OPERATE, TEST, EXIT Miller et al, 1960
- 21. Operate Exit Fail Success Test for Goal Open System Miller, 1960 Goal Controller
- 22. p92 p94 Argyris C 1990 ‘Overcoming Organisational Defences: Facilitating Organisational Learning’ Prentice-Hall: New Jersey, US Single and Double Loop Learning Argyris 1990
- 23. Instruction & Assessment Product UX Students Paper MAGICAL IMPROVEMENTS OCCUR HERE Education Companies Data-Driven Design in K12 classrooms Assessment Data Product Data UX
- 24. Instruction & Assessment Product UX Students Assessment Data Paper Teacher Actions in Response to Data? Product Data UX Administrative Interactions PLCs Education Companies Admin Actions in Response to Data? Product Team Actions in Response to Data?
- 25. Case #1: Supporting Data-Driven Improvement in Educational Product Companies
- 26. Classroom System Gather Data Analyze Data Actions for improvement Data is Inaccessible No Data or Invalid Data Data is not Actionable No Motivation for Data-Driven Improvement • Product doesn’t measure • Users don’t implement • Poor Data Experience • ? • ? Common Barriers to DDDM And some root causes
- 27. What are the educational goals? How do we assess we’ve achieved our goals successfully? What instructional designs might improve the assessments? "Backwards Design” Methodology (cite Understanding by Design) Goals Assessments Instruction
- 28. Key Question #1: Is the system capable of measuring success? Can the system measure whether it is achieving its goals? “If we want to report on student mastery for topic A, we need to collect valid test data on topic A”
- 29. Case #2: Development plans for a teacher-facing recommendation system
- 30. Mapping the System to Reinforcement Learning • What is the space of possible observations? (what we measure?) • What is the space of possible actions? (what can we do?) • What is the reward metric? (how do we quantify our goals?)
- 31. Class Data Dashboard Teacher Decide Next Task Class Assessment Data Dashboard Recommendations Teacher Decide Next Task Data from 1000s of other classes Decisions from 1000s of other teachers Mapping Classrooms to Reinforcement Learning Action Space Observation Rewards 1. Assessment Data Dashboard (Observation) 2. Digital Assignment Possibilities (Action Space) 3. Recommendation Success (Rewards) ? How to capture this?
- 32. Key Question #2: Is the action space sufficiently defined? The right data might be gathered and analysed appropriately …but is there a defined action in response? NEED methods for scaling-up system learning from rich human decision-making
- 33. Case #3: Automated Machine-Learning Optimization of an Educational Game
- 34. An AI system for design optimization successfully increased game engagement… Thousands of Online Players Machine Learning Online Game design improvements …but then spun out of control! Lomas, D., Forlizzi, J., Poonwala, N., Patel, N., Shodhan, S., Patel, K., Koedinger, K., Brunskill, E. (2016) Interaction Design as a Multi-Armed Bandit Problem. ACM CHI
- 35. Multi-Armed Bandit Problem Goal is to maximize payout from row of slot machines with unknown rates of reward (some machines pay out more) Need to balance exploration with exploitation 35 Lomas, J. D., Forlizzi, J., Poonwala, N., Patel, N., Shodhan, S., Patel, K., Koedinger, Brunskill, E. (2016). Interface Design Optimization as a Multi-Armed Bandit Problem. In CHI’16. Interface design as a multi-armed bandit problem…
- 36. 36 Ship targets Type to name a fraction Submarine targets Click to locate a fraction Instructional Design of “Battleship Numberline”
- 37. Design Factors Target Type: Ship or Sub Target Size: Small to large targets Time Limits: Small to large time limits Item Sets: The fractions to be estimated Tickmarks: Scaffolds for estimating Sequencing: Algorithms for presentation 37 Target Size TargetType Experiment 1: Assigned 10,832 players to 3 simultaneous experiments, each testing 6 different design factors (2x3: target type and target size)
- 38. 38 Success! Automatic optimization algorithms produced more engagement Bandit experiments produce greater total engagement Sum of Items Played
- 39. 39 Meta Experiment 1 Worst Design Ship 97 Best Design Ship 90
- 40. 40 Meta Experiment 2 Introduced ridiculously large “Bad Designs"
- 41. Software Integrate data with qualitative insights to find problems and rapidly test solutions Users Software Dashboard Design Optimization The Data-Experience Dialectic Data and qualitative insights must be integrated
- 42. With 10,000 subjects a day, we could run thousands of experiments each year But setting up, analyzing and acting upon experiments is hard. How might AI get involved? Could we partially automate design optimization and scientific experimentation?
- 43. Key Question #3: Is there a human-in-the-loop to keep alignment? If continuous improvement metrics are not aligned to goals and values, data-driven improvement (teams or AI) will produce unintended consequences NEED feedback loops about suitability of success metrics
- 44. Conclusions
- 45. Takeaways • From Designing Artificial Intelligence to Designing Intelligent Systems • Collaboration between teams and algorithms (not just for AI but for products)
- 46. A Definition of Intelligence used in AI “Intelligence measures an agent’s ability to achieve goals in a wide range of environments.” —Legg & Hutter, 2007 “an individual’s intelligence is related to their ability to succeed…” - Robert Sternberg
- 47. What can go wrong? • Poor goals (misalignment with values) • Poor metrics (misalignment with goals) • Poor actions responding to data • Misleading data and misreading data • Conflicting interests in stakeholders Unintended consequences! Accountability can create perverse incentives e.g. schools encourage low-performing students to drop out or to cheat (Schildkamp et al, 2012).
- 48. How can system designers help build the human into the loop? • UI for User Goal Setting • System mapping & communicating future systems • Processes to Negotiate Metrics for Success (not just for AI but for products)
- 50. Goals Evaluation ActPerceive Plan Target What should I do? How should I do it? Did it work?
- 51. Reinforcement Learning Input Reward EnvironmentCognition Output AI Definition of Intelligence “Intelligence measures an agent’s ability to achieve goals in a wide range of environments.”
- 52. Perception Action Evaluate Goals vs Metrics Measure Modify Goal World 1st Order: Closed System Measure Evaluate Values vs Goals Measure Modify 2nd Order: Learning System
- 53. Perception Action Compare Goals vs Measure Measure Modify Goal World Closed System