Ready by 21: Using Data in Coalitions

Editor's Notes

• #9: Background in research, nonprofits/collaboration Legitimacy building - capacity project, progression to research in coalitions
• #14: What is your definition of what you’re measuring? - How do you work out discrepancies between partners? Data-sharing agreements Legally, an organization has what they need to share Likely means “scrubbed” data and use of identifiers rather than names Likely means drafting & signing MOU Plan for actual data-sharing Cleaning and formatting – what form does the data need to be in? Repository? Software? Add in a little bit about software – case management – Sales Force, Efforts to Outcomes Distributing – how often are you supposed to upload and review
• #17: Who has access? Who provides the data, who’s responsible for knowing it, who has access to any particular system Who analyzes? Address the need for someone who has some statistical knowledge Ability to look across organizations/systems to make comparisons Can develop more complex analytics Example of materials – StriveTogether uses Lean Six Sigma; what do others do? How are decisions made? How often? By whom? Who’s responsible for understanding the data from each organization? How to evaluate your efforts? collective impact initiatives should evaluate data and the backbone Assuming you have these things in place – a plan and process for sharing – can start working through it as described in this exercise
• #18: Information on why CI initiatives collect data Reminder of how this structure is different from others - to align and evaluate what is and isn’t working versus creation of new programs Data that suggests impact rather than outcomes Can bring in some of Benn’s 6 purposes of evaluation Our experience: talk about the distinction we see between what people say they’re doing and what they actually do Visual option: have some key phrases/quotes from EC2C and Lodestar interviewees to talk about data to represent on slides What we see in benchmark reports
• #21: In Exhibit 3, the first column represents the percentage of parents who achieved the goal of reading to their child for 20 minutes a day. Which agency had the greatest percentage? Christopher House was highest, reporting that 23% of its parents read to their child at least 20 minutes a day. In Exhibit 3, the second column represents the percentage of parents who have been increasing their reading but are not yet at the goal of 20 minutes daily. Which agency showed the greatest gains there? The Chinese American Service League showed the greatest gains, with 43.33% of parents increasing their reading time. Christopher House was close behind at 41%. One of the interesting things to note is the Chinese American Service League is heading toward the goal that it had very little success with in the past, as shown by the fact that only one of its parents had read to their child every day for 20 minutes. In Exhibit 3, the third column is the combination of those who reached the goal and those moving toward it. Which agency is struggling the most in both? The Children’s Home + Aid Society is struggling most, with only 30.48% of its 105 surveyed families either increasing their reading to their child or achieving the goal of 20 minutes of reading a day. The agency is in a good position to learn from other CBC agencies who are getting better results and to set a SMART goal for improvement in the next academic year.
• #22: Taking a look at Exhibit 4, what kind of metric is the literacy indicator for school readiness? Is it a leading or lagging indicator? School readiness is a lagging indicator. The key idea here is that you can directly affect parental reading to children through a program, but you can’t directly affect literacy. It’s the result of an increase in another activity. Exhibit 4 compares students’ performance on literacy assessments over the course of the school year, from fall 2014 to spring 2015. There are three measures of change. The first two columns display the average scores from fall 2014 and spring 2015. The third column shows the average change over that time period. The fourth column indicates the percentage of children that showed at least a 0.5 increase, and the fifth column indicates the percentage of children that showed any increase. Whereas small increases might be due to maturation, the working theory is that a 0.5 increase is significant. Which agency had the greatest percentage of students with at least a 0.5 increase for ages 0–3? How do you make sense of the 100% scores for the Chinese American Service League and Gads Hill Center? For ages 0–3, 100% of Gads Hill Center and Chinese American Service League students showed a 0.5 increase or more in their literacy levels. One of the things for students to ponder is how much weight to give the 100% scores for the Chinese American Service League and Gads Hill Center for ages 0–3. They should note the small number of students in those groups: only five in the Chinese American Service League and nine in the Gads Hill program. With these small numbers, these programs may not be emblematic of best practices, but just provide lots of individualized attention. It’s noteworthy that these same two programs have the lowest percentage of students making gains of at least 0.5 for ages 3–5, where larger numbers of students are involved
• #23: What about ages 3–5? For ages 3–5, Children’s Home + Aid Society had the largest percentage of students with at least a 0.5 gain (87.8%).
• #24: Christopher House Breakout Group(s) report What’s the right narrative to accompany these data? Christopher House directors may be tempted to focus on just the agency’s strengths rather than its weaknesses. Although there are bright spots, like the percentage of children ages 3–5 who are making significant gains in reading, there are also concerns. For example, only 23% of parents are reading to their child for 20 minutes every night. Further, only 58.8% of children are making significant gains in the years 0–3. This suggests that there are both encouraging signs and opportunities for growth. What might make Christopher House’s data more or less comparable to those of the other CBC partners? Students will typically report that these data might be measured in different ways and so they are not strictly comparable. However, the case explicitly mentions that the CBC partners used the exact same measures, so it is important to correct them on this point. Other factors might include the degree to which different agencies only serve low-income students or whether Christopher House serves a greater number of students who speak English as a second language. Are changes in the literacy indicator for school readiness linked to changes in parents reading to their children? The charts provided in the case do not prove such a link, but the data-sharing system that CBC uses allows the member agencies to statistically analyze whether children whose parents made gains in reading to them every night also increased their literacy the subsequent year. Having a data system that tracks both leading and lagging indicators at the client level makes that possible. Chinese American Service League Breakout Group(s) report How should you interpret the results? Can you rely on the data related to parent reading? Do the data accurately reflect parents’ reading engagement? One key consideration in determining the comparative accuracy of parental reading rates is response rate to the parent survey. In particular, Christopher House’s survey response rate is relatively lower than the others, at 40%. Students could determine this by comparing Christopher House’s total number of children in Exhibit 4 (294) with its number of survey respondents in Exhibit 3 (116). Lower rates of response lend themselves to questions of response bias: Were parents who made no gains less likely to complete the survey, leading Christopher House to inflate its success? A second concern is the accuracy of the parents’ self-reported behavior. There may be a tendency for some parents to report what they had hoped to do rather than what they actually did. How can the data be pulled together to develop SMART goals? SMART goals are only as good as the data you collect. With faulty data, both the evaluation of the baseline and realistic progress are undermined. Erie Neighborhood House Breakout Group(s) report How do you make sense of the discrepancy between the leading (parent engagement) and lagging (literacy) indicators? There are a couple of ways students might go with this answer. Some student groups will focus on the difficulty of ensuring the accuracy of the leading indicator data. Others will question the logic of the theory that parents reading to their children is a determining factor of school readiness. A number of other factors could affect a child’s school readiness, including instructional quality in the classroom and unique challenges their population of students faces. Parent engagement alone may not be enough to overcome these issues. How can a SMART goal and plan for improvement be developed, taking the data into account? Students should be encouraged to think about a couple of factors. The first is a time lag, or how immediate is the impact of the leading indicator change on the lagging indicator. If this is the issue, then the data should show the correlation with parent reading the following year. The second is whether parent engagement is the right lever for Erie Neighborhood House. Because its students start with relatively high literacy scores in comparison to other agencies, maybe another intervention is needed. In this case, the SMART goal wouldn’t focus on parent reading at all.