Informing Progress: Insights on Personalized Learning Implementation and Effects

JULY 2017 Informing Progress John F. Pane Elizabeth D. Steiner Matthew D. Baird Laura S. Hamilton Joseph D. Pane Insights on Personalized Learning Implementation and Effects Funded by C O R P O R AT I O N  C O R P O R AT I O N The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. Guided by the belief that every life has equal value, the Bill & Melinda Gates Foundation works to help all people lead healthy, productive lives. In developing countries, it focuses on improving people’s health and giving them the chance to lift themselves out of hunger and extreme poverty. In the United States, it seeks to ensure that all people—especially those with the fewest resources—have access to the opportunities they need to succeed in school and life. Based in Seattle, Washington, the foundation is led by CEO Susan Desmond-Hellmann and Co-chair William H. Gates Sr., under the direction of Bill and Melinda Gates and Warren Buffett. RR-2042-BMGF The trademark(s) contained herein is protected by law. This work is licensed under a Creative Commons Attribution 4.0 International License. All users of the publication are permitted to copy and redistribute the material in any medium or format and transform and build upon the material, including for any purpose (including commercial) without further permission or fees being required. For additional information, please visit http://creativecommons.org/licenses/by/4.0/. This report is based on research funded in part by the Bill & Melinda Gates Foundation. The findings and conclusions contained within are those of the authors and do not necessarily reflect positions or policies of the Bill & Melinda Gates Foundation. PHOTO CREDITS  | Cover: PeopleImages/DigitalVision/Getty Images; page 5: JBryson/iStock/Getty Images Plus; page 7: Ableimages/ DigitalVision/Getty Images; page 8: Tyler Olson/Fotolia; page 19: asiseeit/E+/Getty Images; page 31: kali9/E+/Getty Images; page 32: WavebreakMediaMicro/Fotolia; page 34: PeopleImages/DigitalVision/Getty Images; page 38: STEEX/E+/Getty Images; page 39: Rob/ Fotolia; page 42: FatCamera/E+/Getty Images Table of Contents 2 Introduction 6 What Is Personalized Learning? 8 What Does Personalized Learning Look Like, and How Does It Differ from Practices in Schools Nationally? 9 Learner Profiles 12 Personal Learning Paths 16 Competency-Based Progression 20 Flexible Learning Environments 25 General Challenges to Implementing Personalized Learning 26 How Did Charter and District NGLC Schools Compare in Their Implementation of Personalized Learning? 33 How Did NGLC Schools Affect Student Achievement? 40 Implications and Policy Recommendations 45 References 46 Appendix A. Implementation Analysis Methods and Limitations 49 Appendix B. Achievement Analysis Methods and Limitations Informing Progress Insights on Personalized Learning Implementation and Effects 1  Introduction The Bill & Melinda Gates Foundation engaged the RAND Corporation to carry out a study of foundation-funded schools that are implementing personalized learning (PL). This is the third in a series of reports focused on PL school design characteristics, teacher and student perceptions, and student achievement. The basic concept of PL—instruction that is focused affordable and available in schools and developers have on meeting students’ individual learning needs while created software products that can support individual incorporating their interests and preferences—has been student learning. Much of this work was inspired by a longstanding practice in U.S. K–12 education. Examples Bloom’s (1984) article showing that human tutors include individualized education plans for students with providing individualized instruction to students can special needs, the use of data to make instructional produce large achievement gains relative to whole-class decisions for individuals or small groups, the use of instruction. In the context of a review of hundreds of support teachers and tutors, individual or group projects, studies of human and computer-based tutoring, VanLehn and diverse elective course offerings. (2011) made the important observation that mastery learning principles used by the tutors in Bloom’s (1984) More recently, options for personalization have increased article may account for a large part of their positive as personal computing devices have become more effect. The studies in VanLehn’s meta-analysis were not all conducted in K–12 schools, and many did not produce statistically significant results. Nonetheless, he found that systems that emulate the interactions of a human tutor Personalized learning prioritizes tended to produce positive achievement results. Some of these systems have undergone rigorous evaluation a clear understanding of the needs in K–12 schools with positive results (Brodersen and Melluso, 2017). and goals of each individual student For the most part, all of these personalization efforts and the tailoring of instruction to have been implemented within schools and classrooms that otherwise retain a traditional model of large-group address those needs and goals. These instruction to groups of roughly 20–30 similar-age needs and goals, and progress toward students. However, in recent years, it has become more common for schools to embrace schoolwide models of PL meeting them, are highly visible and that depart more radically from typical practice. These schools seek to allow what and how a student learns on easily accessible to teachers as well a daily basis to be less constrained by the needs of other students or by external requirements for grade-level as students and their families, are content coverage; to be driven largely by the individual frequently discussed among these student’s needs, interests, and context; and to be informed by ongoing conversations with the student and parties, and are updated accordingly. the adults in his or her life. In service of these objectives, the staffs at these schools are implementing a range of interconnected strategies in novel ways. 2 Informing Progress Insights on Personalized Learning Implementation and Effects In this report, we: than how they are described in theory. Therefore, the implementation component of our study seeks to ■■ explore what PL looks like in a small sample of describe how these schools were implementing PL, schools that have been focused on implementing PL understand some of the challenges and facilitators, and approaches schoolwide consider these alongside achievement findings to discern ■■ consider how the approaches to personalization in patterns that may be informative. these schools compare to a national sample that represents more-typical practice in the United States To learn about implementation, we interviewed school ■■ briefly discuss obstacles to PL implementation administrators, surveyed teachers and students, and collected instructional logs (brief surveys administered ■■ discuss how PL implementation differs between daily to teachers for several weeks during the school charter schools and traditional district schools in our year, focusing on instruction in that day’s lesson). We sample, and what factors seem to support or hinder also visited some of the schools to interview teachers implementation and students and observe classrooms. The surveys were ■■ describe how achievement growth for students in also administered to a national sample for comparison. these schools differs from growth for similar students Additional information about all of these data collection in other schools methods can be found in Appendix A of this report. ■■ discuss implications for policymakers, implementers, Student outcome analyses in this study rely on data from and funders. the Northwest Evaluation Association (NWEA) Measures of Academic Progress (MAP) mathematics and reading We collected data from the schools that received funding assessments, administered in the fall and spring of each from the Next Generation Learning Challenges (NGLC) school year in a subset of the NGLC schools. initiative in the Wave IIIa and Wave IV launch grants.1 This initiative was intended to support the development This study has numerous limitations, and so the of schools that took a highly personalized approach to findings should be interpreted cautiously. For example, learning. Our study began in fall 2012 and concluded in spring 2015. The schools participating in the NGLC program were not expected to implement a specific PL intervention. Although there were some general requirements, such as Key Findings allowing students to learn at varying rates, technology- ■ Schools in the NGLC sample were enabled learning, and incorporating flexibility in the pursuing a wide variety of practices learning environment, each school had the flexibility to focus on the learning needs of each to implement a PL model that would work best with its context, students, and goals. Of course, educational individual student in a supportive and interventions are often enacted differently in practice flexible way. 1 The NGLC initiative is managed by EDUCAUSE, a nonprofit as- ■ Schools were implementing specific PL sociation dedicated to advancing the use of information technol- ogy in higher education, in association with other organizational approaches to varying degrees, with partners, including the League for Innovation in the Community none of the schools looking as radically College, the International Association for K–12 Online Learning, and the Council of Chief State School Officers. NGLC receives different from traditional schools as primary funding from the Bill & Melinda Gates Foundation, with theory might predict. additional support from the William and Flora Hewlett Founda- tion, the Eli and Edythe Broad Foundation, and the Michael and Susan Dell Foundation. The initiative supports school districts, ■ There is suggestive evidence that charter management organizations, and partner organizations greater implementation of PL practices that embrace PL as a means to dramatically increase college readiness rates, particularly among low-income students and may be related to more positive effects students of color. To be considered for funding, these schools on achievement; however, this finding applied for a competitive grant. In their applications, schools were required to describe with specificity how their models requires confirmation through further would support PL. While all of these schools have a high degree research. of integrated technology as part of their school designs, they vary considerably in the methods and degrees to which they use technology to support PL. Informing Progress Insights on Personalized Learning Implementation and Effects 3  implementation data are limited by their self-report among other risks, selection bias or the implementation nature and small sample sizes, which can make it difficult of PL in the comparison group. In addition, because it to detect differences between groups. Comparisons may take a few years for new PL schools to optimize to national surveys are also limited by an untestable implementation, the results here may not reflect how assumption that they represent more-traditional well the NGLC schools will perform in the future. Readers practices, where PL is not being implemented as are encouraged to review the more-detailed discussion of intensively as in the NGLC schools. The achievement limitations in Appendix B.2 analyses use a research design that does not enable strong causal conclusions. Results can be influenced by, 2 Portions of this report are adapted from Pane et al. (2015). The Sample for Implementation Analyses The 40 NGLC schools in the implementation sample were ■■ more than three-quarters of the sample were charter predominantly located in urban areas (two were rural) schools and served large proportions of minority students from ■■ elementary and K–8 schools averaged about low-income families. Many of the schools started out 230 students per school, and middle and high schools serving a limited range of grades, with plans to expand averaged about 270 students annually until they reach their full enrollment and grade ■■ the median schoolwide proportion of students range. Key sample characteristics include the following, eligible for free or reduced-price lunch (FRL) was based on 2014–15 school year data provided by school 80 percent administrators: ■■ the median schoolwide proportion of students of ■■ 43 percent of the schools had been implementing color was 96 percent. PL for one year, 38 percent for two years, and 20 percent for three years Composition of schools in the implementation analysis About School Type 23% 10,600 students 78% District (n = 9) Charter (n = 31) 40 schools participating in NGLC Grade Level Elementary school (n = 5) 13% 6,145 students surveyed 48% 10% K–8 school (n = 4) Middle school (n = 12) 241 30% teachers surveyed High school (n = 19) Note: Percentages may not add to 100 percent due to rounding. 4 Informing Progress Insights on Personalized Learning Implementation and Effects The Sample for Achievement Analyses Of the 40 NGLC schools in the implementation analysis, older PL schools that had been operating for at least two only the 32 that administered the MAP assessment are years. Only 16 NGLC schools were included in those prior included in the achievement analysis. All of them were analyses, along with 46 additional schools that were not relatively new at implementing PL, having started in part of the NGLC program (among the non-NGLC schools, the 2012–13 academic year or later. Moreover, most of 18 had launched as new schools implementing PL in the these schools were new schools at the time they began same time frame as the NGLC schools; most of the 28 implementing PL. As such, this report discusses schools others launched in the prior decade, though we do not relatively early in the implementation process. The know exactly when they started to focus on PL). Table 1 results reported here are not directly comparable to compares the samples used for achievement analyses in achievement analyses presented previously in Pane et al. the two reports. (2015), which focused on a larger sample of somewhat Table 1: Comparison of achievement analysis samples in Pane et al. (2015) and this report Pane et al. (2015) This report Participating initiatives NGLC and other programs NGLC only School experience implementing PL At least two years At least one year Number of PL schools in sample 62 32 Percentage of charter schools in PL school sample 92% 75% Main achievement analysis 2-year span: 2013–15 1-year span: 2014–15 Approximate number of PL students in main achievement analysis 11,000 5,500 Approximate percentage of PL student sample in … grades 9–12 8% 31% … grades 6–8 23% 48% … grades K–5 69% 21% Informing Progress Insights on Personalized Learning Implementation and Effects 5  Section What Is Title Personalized Learning? Although there is not yet a widely shared definition of PL, we distilled this working What is the role definition from discussions with leading of technology in PL? practitioners in the field: In a variety of ways, technology holds promise to Personalized learning prioritizes a clear enable personalization to an extent that was not understanding of the needs and goals of possible at large scale in an earlier era. Technology’s each individual student and the tailoring of greatest role may be to manage the complexity of instruction to address those needs and goals. the personalization process. By occasionally providing These needs and goals, and progress toward instruction or supporting independent learning, meeting them, are highly visible and easily technology can also enable educators to take a accessible to teachers as well as students and more personalized approach in their own teaching their families, are frequently discussed among efforts and other activities they undertake to support these parties, and are updated accordingly. student learning and development. This aspiration contrasts with more-traditional instructional approaches, where efforts to meet individual students’ needs may take less priority than PL, such a diversity of models can be useful to help us having students work toward grade-level standards, learn which strategies and approaches, or combinations progress on pace with their grade-level peers, or thereof, appear to be most important for PL‘s success. To prepare for grade-level tests at the end of the year. organize our discussion we group the approaches used by In its ideal form, PL allows for greater variety in what the NGLC schools into four interdependent strategies. students are working on at any moment, while still setting ambitious goals for each student’s progress. The hypothesis, consistent with the research cited above, is Learner Profiles that personalized instructional approaches and strategies A learner profile is a record of each student’s will improve student outcomes in the short term (e.g., individual strengths, needs, motivations, stronger rates of growth in achievement) and in the progress, and goals based on data from all available long term (e.g., successful completion of a postsecondary sources. Learner profiles are available not only to degree or successful transition into a career). teachers, but also to students and their families, and are frequently reviewed, discussed, and updated to inform The best strategies for creating an educational the student’s educational plan. environment that is highly personalized have yet to be identified through research. The NGLC schools in this Personal Learning Paths study were taking a variety of approaches, some of which Informed by the learner profile, personal were extensions of traditional practices, often enhanced learning paths allow for flexibility in the by strategic use of technology for instruction and other specific paths students take through content to enact purposes, and some of which were more-significant their educational plan, while still holding them to high departures from common approaches. Each school expectations. Within parameters set by teachers, students integrated a set of approaches to create their unique can make choices about the content or structure of school model. At this early stage in the development of 6 Informing Progress Insights on Personalized Learning Implementation and Effects learning, and the school offers a variety of instructional approaches and curriculum materials, including support for meaningful learning experiences outside of school. Time is available during the school day for one-on-one academic support tailored to students’ learning needs, whether for remediation, help with grade-level content, or enrichment. Competency-Based Progression Competency-based progression enables personalized paths to run their natural course by removing external constraints on what material each student works on, when, and for how long. Each student’s progress toward clearly defined goals is continually assessed, and assessment occurs “on demand” when a student is ready to demonstrate competency. Assessment may take a variety of forms, such as projects or presentations, as well as more-traditional tests or quizzes. A student advances at his or her own pace and earns course credit (if applicable) as soon as he or she demonstrates an adequate level of competency. and to adopt approaches compatible with local context Flexible Learning and the population of students they served. Environments Flexible learning environments imply that In the next section, we use these four strategies as an the school adapts the use of resources such as staff, organizing framework for describing PL implementation space, and time to best support personalization. For in the NGLC schools. For each strategy, we first present example, elements of the learning space—size, classroom a vignette drawn from a school we visited, as an organization, and furniture—are designed to support example of relatively strong implementation of the implementation of PL. The structure of learning time and strategy. We then compare NGLC schools to a sample student grouping strategies are flexible, responsive to representing schools across the United States with student needs, and driven by data where appropriate. respect to implementing that strategy, discuss some of Technology is a key aspect of the school model and is the challenges NGLC schools reported, and briefly discuss available to all students; often schools provide a device to obstacles to PL implementation that cut across the four each student. strategies. Subsequent sections examine whether PL implementation differed between district-operated and As we discuss above, the schools approached PL charter schools in the NGLC sample. We then turn to an in a variety of ways, and did not necessarily plan analysis of achievement effects, and discuss implications to implement every strategy. Rather, they were all for policymakers, implementers, and funders. working toward the general goal of improving student achievement through PL, and were free to be creative The Four PL Strategies Are Interrelated Learner profiles maintain a rich and up-to-date record of student strengths, needs, goals, and progress; that information is used to define personal learning paths, which are appropriate and meaningful choices of material for each student to work on, with the necessary adult supports; competency-based progression enables these personalized paths to run their natural course by removing external constraints on what material each student works on, when, and for how long; and flexible learning environments enable schools to allocate resources in new ways to best support these processes. Informing Progress Insights on Personalized Learning Implementation and Effects 7  What Does Personalized Learning Look Like, and How Does It Differ from Section Title Practices in Schools Nationally? Implementation Analysis Methods The findings we present in this section comprise a However, we also made use of interviews with principals synthesis of the implementation data. The methods we and teachers, and focus groups with students; although used are described in greater detail in Appendix A, along these sources are less representative than the surveys, with a discussion of limitations. To briefly summarize our they provide a greater depth of information on key methods, we used a holistic approach to decide what aspects of implementation that help to clarify or information to present, focusing on meaningful evidence illuminate patterns we found in the survey data. We of differences (or similarities) between practices in the triangulated these sources with teacher logs and NGLC schools and in other schools nationally. Where classroom observation data where applicable. we were able to perform tests of statistical significance, we used those results to guide our decisions about When we discuss the interview data, we use terms such what material to present. In some cases, we describe as “many” and “most” to refer to more than half of differences that were not statistically significant but that interview respondents in the applicable group (e.g., were large in magnitude and qualitatively meaningful school leaders, teachers, or students) across schools, and in that they shed some light on substantive questions we use “several” or “some” to refer to less than half of about implementation. We relied heavily on teacher and respondents. Percentages reported here are based on student survey data because those sources are available survey results. The vignettes are drawn from the site visit for more of the sample and are thus most representative data, and the discussion of implementation challenges is of teachers’ and students’ attitudes and perceptions. drawn from survey, interview, and focus group data. 8 Informing Progress Insights on Personalized Learning Implementation and Effects  Learner Profiles In this section, we discuss: A learner profile is a record of each student’s individual ■■ strategies used by students and teachers to track and strengths, needs, motivations, progress, and goals based discuss learning goals on data from all available sources. Learner profiles are ■■ teachers’ use of student achievement data to available not only to teachers, but also to students and personalize instruction their families, and are frequently reviewed, discussed, ■■ usefulness of school data systems and teachers’ access and updated to inform the student’s educational plan. to data ■■ challenges of using learner profiles. KEY TAKEAWAYS ON LEARNER PROFILES NGLC schools showed higher levels of implementation Summary than the national sample in some ways. These findings suggest some important ways in which ■■ More NGLC teachers reported frequent receipt of the NGLC schools exhibited greater access to and use high-quality student data and extensive use of the of student data to inform personalized instructional data to personalize instruction. approaches. Both national and NGLC teachers reported ■■ More NGLC students reported using technology to receiving and using student data frequently and we track their learning progress. did not find differences in the use and characteristics of formal learner profiles. However, NGLC teachers reported receiving many types of student data (e.g., Schools in the two samples showed similar levels of implementation in other ways. data on students who have achieved mastery or need extra assistance) more frequently, and using them to ■■ Students in the two samples reported similar levels adjust instruction in ways consistent with PL practices to of discussion with teachers regarding their learning a greater extent than teachers in the national sample. progress or learning goals. These differences in data access and use could be related ■■ Teachers in the two samples reported similar rates to differences in the schools’ data systems, which in NGLC of keeping up-to-date documentation of student schools seemed more likely to contain student data which strengths, weaknesses, and goals. facilitated personalized instructional practices. According to principals, barriers to more-extensive use of student There were several challenges to implementation. data included difficulties measuring nonachievement ■■ Many NGLC schools struggled to use nonachievement constructs (e.g., behavior or socio-emotional skills) and data (e.g., behavior, attendance, socio-emotional integrating such data, along with other data generated skills) to inform instructional decisions and goal by curriculum products, into the school’s data system. setting, in part due to challenges measuring This made it harder to combine these inputs with those skills and integrating the results with other achievement data for instructional decisions and goal student data. setting. ■■ Data from digital curriculum programs were not always well integrated with other school data systems. Learner profiles: How do NGLC and national practices compare? More NGLC teachers reported frequent receipt of of varying achievement levels, including students who high-quality student data and extensive use of are far above or below grade level). NGLC teachers were the data to personalize instruction. NGLC teachers also more likely to report that they had access to high- reported that their schools’ data systems provided high- quality assessment data that helped them adapt the quality data useful for informing instruction (e.g., real- pace or content of instruction to meet students’ needs. time, actionable data, and information about students NGLC teachers, on average, reported receiving student Informing Progress Insights on Personalized Learning Implementation and Effects 9  VIGNETTE: What do learner profiles look like? School A is an urban charter high school that had been implementing PL for two years at the time of our visit. Students are able to check their grades using the school’s learning management system, PowerSchool, as well as their nonachievement data (e.g., behavior, attendance, socio-emotional skills) via SchoolRunner. These programs are accessible to students at home as well as in school and are also accessible to parents. Students reported that these two sources of information about their performance were updated frequently, were useful, and were easily accessible, for example, from their smartphones. Teachers said that they updated PowerSchool frequently—at least daily or weekly. A commitment to making student progress visible and accessible was also evident in posters and charts on classroom walls, which were used to track students’ progress toward mastering the college-ready ACT standards. Students and teachers reported that they drew on these multiple sources of data to drive conversations about student progress and set goals. Students reported that they discussed their grades and behavior with teachers, tracked progress, and set goals during Advisory, a daily time when students met one-on-one with teachers or caught up on classwork. Teachers reported different methods for ensuring that they met with all students during Advisory to discuss their grades and progress. One met weekly with each of her advisees; another let the students initiate meetings, but checked in with each student at least every two weeks. Several students described close relationships with their advisers, as illustrated in the quote. “With my Rise [Advisory] teacher, I talk to her every single day [about the progress I’m making in school]. I even text her. That’s how my bond is with her. And with teachers, I just have two where I constantly check in for my grades and how I’m doing, and they give me feedback on how I’m supposed to do better and I start improving my things better now.” —Student comment about advisory Support achievement and nonachievement data (such as data on More NGLC students reported using technology to student behavior or socio-emotional outcomes) more track their learning progress. However, students than a few times a month, versus approximately monthly in the two samples reported similar levels of in the national sample. NGLC teachers also reported discussion with teachers regarding their learning using such data to inform and personalize instruction to progress or learning goals. Students in the NGLC a greater extent (see Figure 1). However, a majority of sample were somewhat more likely to agree that they teachers in both samples reported that they had plenty kept track of their progress using technology (e.g., by of data but needed help translating those data into using an online gradebook or portfolio) most of the instructional steps. There were no differences in teachers’ time or always. There were no differences in how often reports of how easy their school data systems were to use. students reported discussing their learning progress with their teachers or working with their teachers to set personal goals for their own learning, but such practices were not widespread. “I would say we’re definitely better off in the sense that we’re gathering data constantly: their Teachers in the two samples reported similar rates [students’] homework and the assessments in of keeping up-to-date documentation of student the lessons, the assessment during the projects. strengths, weaknesses, and goals. When comparing the NGLC survey results with those from the national It’s just ongoing. It’s pretty fluid.” sample, we found no differences in several key aspects of —NGLC teacher comment about using learner profiles. Similar proportions of teachers reported student achievement data using frequently updated, shared documents, either paper or electronic (such as learner profiles and learning plans), to document each student’s strengths, weaknesses, 10 Informing Progress Insights on Personalized Learning Implementation and Effects and goals. Among teachers who reported using such planned to undertake this integration, but had not yet documents, their characteristics (e.g., whether they exist been able to do so. for all students or were frequently updated), were similar in the two samples. Data from digital curriculum programs were not always well integrated with other school data systems. Another common challenge, reported by NGLC Challenges of Learner Profiles principals, was that data from the school’s various digital Many NGLC schools struggled to use curricula and online materials were not well integrated nonachievement data to inform instructional with other data systems (e.g., the learning management decisions and goal setting, in part due to system where teachers recorded grades). This increased challenges measuring those skills and integrating the burden on teachers who wanted to retrieve and the results with other student data. Using multiple analyze these data. types of student data to inform instruction is a key feature of learner profiles. All NGLC schools collected nonachievement student data, but much of this was “In terms of nonacademic goals, we teach done informally. Few schools had robust systems for [students] design thinking, so we do want collecting these data, particularly on socio-emotional skills such as collaboration, critical thinking, or resilience, them to demonstrate the designer skills. And or using them to inform instructional decisions and those are things like critical thinking and understand student progress. Many school administrators collaboration. We haven’t yet figured out the told us in interviews that their schools had not yet best ways to measure those . . . ” pulled achievement and nonachievement data together —NGLC principal, on measuring into one cohesive document or system—the data were nonachievement skills often tracked in multiple systems. Most of these schools FIGURE Extent to which teachers used student achievement data 1 100 ■ Large extent ■ Moderate extent 80 Percentage of teachers 60 30% 33% 27% 27% 40% 40 33% 31% 40% 34% 31% 20 39% 37% 39% 39% 31% 19% 18% 22% 17% 14% 0 National NGLC National NGLC National NGLC National NGLC National NGLC sample sample sample sample sample sample sample sample sample sample Identifying topics Developing Reflecting on and Tailoring the Tailoring the pace requiring more or recommendations discussing learning content of of instruction to less emphasis in for tutoring or with my students instruction to individual students’ instruction other education individual students’ needs support services for needs particular students Note: N = 212–214 NGLC teachers; N = 525 national teachers; survey question: “This year, to what extent have you used student achievement/mastery data for each of the following purposes?” Response choices were on a scale of 1 (“My school doesn’t do this”) to 5 (“Used to a large extent”). Informing Progress Insights on Personalized Learning Implementation and Effects 11  Personal Learning Paths In this section, we discuss: tional plan, while still holding them to high expectations. ■■ the use of a variety of instructional activities, Within parameters set by teachers, students can make including tailored support choices about the content or structure of learning, and ■■ students’ ability to choose topics and instructional the school offers a variety of instructional approaches materials and curriculum materials, including support for mean- ingful learning experiences outside of school. Time is ■■ challenges of implementing personal learning paths. available during the school day for one-on-one academic support tailored to students’ learning needs, whether Personal learning paths allow for flexibility in the specific for remediation, help with grade-level content, or path students take through content to enact their educa- enrichment. KEY TAKEAWAYS ON PERSONAL LEARNING PATHS NGLC schools implemented more individual support than supports tailored to each student’s learning needs. Highly the national sample. personalized approaches, such as flexible paths through ■■ NGLC schools appeared to dedicate more time to content and extensive student choice in the content one-on-one, tailored support of student learning. or structure of learning were not common in either group, most likely because they can be time-consuming Schools in the two groups were more similar on other for teachers to develop and manage. Teachers also aspects of implementation. reported that the need to meet standards constrained ■■ Teachers and students in both samples reported the amount of choice they could offer to students, which relatively low levels of student choice of topics and also likely limited implementation of highly personalized materials. approaches. ■■ Teachers in both samples reported similar levels of tailoring instruction to student needs, although To implement a variety of instructional approaches, NGLC students reported slightly higher rates than NGLC schools reported adjusting instructional time to students nationally. focus on coaching and individual supports for students to ■■ Teachers in both groups agreed that their curriculum a greater extent than teachers did in the national sample, a difference that was perhaps facilitated by the fact that materials were of high quality. the NGLC schools built one-on-one supports for students, There were several challenges to implementation. such as an advisory period, into the school schedule. In ■■ NGLC teachers perceived limited time to develop schools that offered choice in path and content, students personalized lessons to be the biggest obstacle to often worked on different topics and assignments than implementing personal learning paths. their peers. While many students enjoyed the flexibility ■■ Finding high-quality standalone technology-based such choices offered, others observed that it made materials was a challenge. seeking help from (and collaboration with) peers difficult, ■■ Teachers perceived tension between offering student because students were all working on different things. choice and the need to address standards. Teachers in both groups reported that their curriculum materials were of high quality. Interview, log, and survey ■■ Extensive choice can make student collaboration data suggest that NGLC schools used a combination of challenging. standalone tech-based programs and teacher-developed curriculum materials; we do not have comparable data Summary for the national sample. Most NGLC schools implemented a variety of instructional approaches and focused on one-on-one academic Personal learning paths: How do NGLC and national practices compare? NGLC schools appeared to dedicate more time to Surveyed NGLC teachers reported using individual one-on-one, tailored support of student learning. tutoring, coaching, and support for a greater proportion 12 Informing Progress Insights on Personalized Learning Implementation and Effects  VIGNETTE: What do personal learning paths look like? School B is an urban charter middle school that serves grades 6 through 8 and had been implementing its PL model for three years at the time of our visit. At school B, students could take a flexible, personalized path through content via a “playlist”—a list of a variety of activities (e.g., readings, videos, practice problems, assignments) identified by the instructor and designed to help students learn a particular standard or skill. In these playlists, students were placed at the appropriate level of content based on a standardized test. Students used the playlists to choose which activities to complete as part of their course work, as described in the quote. Often, students within a class were working on a variety of different standards, and those who were working on a common set of standards were often grouped together for projects or group work. Students were exposed to a variety of instructional approaches, a strategy consistent with personal learning paths. For example, teachers utilized independent work with and without technology, group and independent projects, the playlists, and one-on-one and small-group work with the teacher. Instructional materials included, for example, online curricula, online games, hands-on projects, and textbooks. Teachers had autonomy to vary instructional approaches and materials in their classrooms as needed. Most class periods included time for one-on-one academic support—teachers would confer with some students while others were working—and a 30-minute period at the end of the day was reserved for teachers to work with selected students for additional one-on-one support. “First, in order to see what standard we’re working on we go to Canvas. Canvas shows all of our classes and once you click on the class, it has a list of the standards and [learning levels] students are on. Every [learning level], you go to it and click on the standard you’re working on and work on the assignments. Some people are on different [learning levels] so it’s based on what they’re working on and there are links to activities.” —Student comment about playlists of the lesson, while teachers in the national sample of both NGLC schools and schools nationally, but we did reported spending a greater proportion of class time not find the degree of student choice to be extensive. on large-group instruction. These results are shown in A majority of teachers in both groups indicated that it Figure 2. The administrator interview and site visit data was rare for students to choose their own instructional suggest that one-on-one academic support was built materials or the topic of the class focus (responding that into the daily school schedule in all the NGLC schools. In it occurred “not at all” or “to a small extent”). Although some schools, this took the form of an advisory period NGLC students reported slightly more choice, less than where students could receive individualized support one-third of students in both groups reported that from teachers, from peers, or from independent practice. they frequently made their own choices (responding Other schools scheduled “intervention time” as a class “most of the time” or “always” on items shown in period, where students sought help in the subjects in Figure 3). The student focus groups suggested that it was which they were struggling. Still others built one-on- one supports, such as independent practice with teacher “ . . . blended learning is when you have half support, into each class period. Although we cannot be certain that schools in the national sample did not of the class on the laptop while you’re doing schedule time for teachers to provide individualized lessons and targeting personalized learning. support, the results presented in Figure 2 suggest that They [students] get personalized learning there are differences in the structure of learning time from the computer and from me. With reading between the groups of schools in the two samples. rotations while they are on the computer I get Teachers and students in both samples reported to have my guided reading done while some relatively low levels of student choice of topics and are in their stations. It helps a lot.” materials. Survey results suggest that student choice in —NGLC teacher comment about variety the content and structure of their learning was a feature in instructional approaches Informing Progress Insights on Personalized Learning Implementation and Effects 13  somewhat more common for teachers to offer a choice opportunities to learn in different ways within a single in how students could complete specific assignments, as lesson, such as listening to the teacher, working in small illustrated by the student comment. groups, or working by themselves. Teachers in both samples reported similar levels Teachers in both groups agreed that their of tailoring instruction to student needs, although curriculum materials were of high quality. Access NGLC students reported slightly higher rates than to high-quality curriculum materials is a key support for students nationally. About two-thirds of teachers in creating personal learning paths. NGLC teachers agreed both groups reported that they adapted content and with those in the national sample that their curriculum provided a variety of instructional materials to suit materials were of high quality and met the learning individual students’ needs to a small or moderate extent. needs of all of their students. These data were collected For example, teachers reported using instructional at a time when teachers nationally were relying heavily approaches such as teacher-led large- and small-group on materials that they developed or found on the web instruction, individual tutoring, small-group collaboration as they attempted to align their curricula with new state and projects, or independent practice with and without standards (Opfer, Kaufman, and Thompson, 2016). digital content. NGLC students were slightly more likely than the national sample to report that they had Challenges of Personal Learning Paths “ . . . it’s the same topic, but you can choose to NGLC teachers perceived limited time to develop complete the task any way you want to. We personalized lessons to be the biggest obstacle to just had this assignment called The Pit and the implementing personal learning paths. Providing Pendulum. And, basically, [the teacher] split it personalized pathways and activities for students is a key feature of personal learning paths, but one that can up into two things. After we finished reading it be time-consuming for teachers. A majority of NGLC you can compare the book to anything media teachers reported that “an inadequate amount of time that compares to the book. . . . And then, also, to prepare personalized lessons for all students” was there was the second part to it where you can a major or minor obstacle to PL implementation, and build off the story and you can write from the half reported that “excessive amounts of time I need to torturer’s perspective . . .” spend developing personalized content” was a major or minor obstacle. Similarly, in site visit and administrator —NGLC student comment about choice interviews, many NGLC teachers and administrators FIGURE Teacher reports of activities used for more than a quarter of class time 2 during a typical lesson 100 ■ NGLC sample 80 Percentage of teachers ■ National sample 60 40 67% 20 35% 30% 17% 9% 10% 0 Large group instruction Individual tutoring Coaching and support Notes: N = 209–214 NGLC teachers; N = 525 national teachers; survey question: “During a typical class, for what percentage of the time do you utilize the following activities with students?” Respondents wrote percentages for each activity in open-ended text boxes. 14 Informing Progress Insights on Personalized Learning Implementation and Effects  FIGURE Student reports of choice in various aspects of instruction 3 ■ Always ■ Most of the time ■ Sometimes ■ Never/rarely 100 17% 15% 28% 34% 80 Percentage of students 43% 44% 39% 60 44% 38% 30% 39% 40 30% 27% 21% 27% 20 19% 17% 20% 19% 9% 12% 9% 12% 0 7% National NGLC National NGLC National NGLC sample sample sample sample sample sample I have opportunities to choose I have opportunities to choose During a single lesson, I have what instructional materials what topics I focus on in class. opportunities to learn I use in class. in different ways. Notes: N = 4,785–4,835 NGLC students; N = 864 national students; survey question: “The following questions ask about your classroom experiences. When you answer them, please think about your experiences with all of your classes in math, English/reading, science, and social studies this year, and mark the response that indicates your typical experience.” Response choices were on a five-point scale from 1 (“never”) to 5 (“always”). mentioned time as an important obstacle to implementing eight of those were mentioned by more than one school, highly flexible personal learning paths with frequent and the two most popular products were mentioned by opportunities for student choice. Clearly, it is time- nine schools each. consuming for teachers to develop personalized lessons in NGLC schools. One theoretical hope is that technology can Teachers perceived tension between offering help make implementation of PL—and personal learning student choice and the need to address standards. paths in particular—more efficient. While technology has Although some NGLC schools offered students a high made many features of personal learning paths possible, degree of flexibility in the paths they could take through such as using a playlist, the work of finding (or creating) content, in most schools students did not seem to have and organizing high-quality content and assignments many opportunities to choose the content or structure often remains in the teachers’ hands. of their learning. For instance, several teachers we interviewed reported that offering extensive student Finding high-quality standalone technology-based choice conflicted with the need to address grade-level materials was a challenge. Survey, log, and interview standards, and reported that what students learned was data suggest that staff in the NGLC schools pieced dictated by the appropriate standards (i.e., subject matter together their tech- and nontech-based curriculum and grade level) with little variation, thus limiting the and instructional materials and used a combination of extent to which teachers could offer choices to students. standalone tech-based programs and teacher-developed material. In interviews, teachers and administrators said Extensive choice can make student collaboration they rarely relied on one or two all-encompassing tech- challenging. Students at some NGLC schools could choose based curriculum products because it was difficult to which topics to work on within a given content area and find ones that were of high quality and effective in their which activities to complete. In focus groups, several school context. About half of surveyed teachers reported students said they generally liked having the flexibility that they often pulled materials from multiple sources or to work on different topics at a different time from their developed them themselves. NGLC teachers reported that classmates. But they also said it posed challenges for they searched for or created about half of their curriculum collaboration. As one student said, “ . . . sometimes it’s materials to supplement the curriculum provided to them. really good to have everyone do the same topic because Few products were common across schools. Sixty-two then everyone can help anybody; and when you all have different online or digital sources of curriculum materials different topics, it’s like she’s doing that and he’s doing and assessments were reported across the 40 schools. Only that, so we can’t talk about it, so it depends.” Informing Progress Insights on Personalized Learning Implementation and Effects 15  Competency-Based Progression In this section, we discuss: when, and for how long. Each student’s progress toward ■■ teacher use of competency-based practices clearly defined goals is continually assessed, and assess- ■■ student experience with competency-based practices ment occurs “on demand” when a student is ready to demonstrate competency. Assessment may take a variety ■■ challenges of competency-based progression. of forms, such as projects or presentations, as well as more-traditional tests or quizzes. A student advances at Competency-based progression enables personalized his or her own pace and earns course credit (if applicable) paths to run their natural course by removing external as soon as he or she demonstrates an adequate level of constraints on what material each student works on, competency. KEY TAKEAWAYS ON COMPETENCY-BASED PROGRESSION NGLC schools appeared to differ from the national ■■ NGLC schools did not award credit for partial mastery sample in some respects. of a course in a way that could be transferred to ■■ Although most teachers in each group reported other schools. using competency-based practices, NGLC teachers and students reported higher levels of such practices. Summary ■■ NGLC teachers were more likely than those in the A majority of teachers in both groups reported using national sample to require students to get through a competency-based practices to a moderate or large certain amount of material. extent. Teachers and students in the NGLC schools reported that competency-based practices were common, There were several challenges related to competency- allowing students to work at different paces and on based progression. different topics or skills at the same time. While this ■■ Many NGLC teachers said that allowing students finding is encouraging, implementing competency-based to progress at their own pace through content was progression is not without challenges. Some teachers challenging when students did not complete work at reported that organizing students into groups for the an acceptable pace. larger performance tasks could be difficult, because ■■ Competency-based grading systems were difficult students were in different places in learning the material. to explain to stakeholders and did not fit with In addition, principals and teachers said that competency- traditional reporting practices. based grading systems were difficult to explain to stakeholders and did not fit with traditional reporting practices. Competency-Based Progression: How do NGLC and national practices compare? Although most teachers in each group reported large extent. However, the NGLC teachers were more using competency-based practices, NGLC teachers likely to report using these practices to a great extent and students reported higher levels of such (Figure 4). In addition, NGLC students were more likely practices. Competency-based practices include enabling than students nationally to report that they always students to work at various paces and on different experienced practices consistent with competency-based topics than their classmates, giving them opportunities progression (Figure 5). Overall, though, neither group of to review or practice new material until they really students perceived these practices to be very common. understand it, requiring them to demonstrate that they understand a topic before moving on to a new topic, and NGLC teachers were more likely than those in enabling them to track their own progress. A majority of the national sample to require students to get teachers in both the national and NGLC samples reported through a certain amount of material. This result is using competency-based practices to a moderate or shown in Figure 4. Since the amount of time students 16 Informing Progress Insights on Personalized Learning Implementation and Effects  VIGNETTE: What does competency-based progression look like? School C, a charter high school with 9th grade students, was broadly implementing competency-based progression during its first year of operation. Students were aware of the goals and standards they were supposed to learn at the start of the year, and they met with teachers during the year to plan how they were going to meet those goals. According to the principal, teachers shared the goals of each course with students at the outset and students “project managed” to get the work done and track their progress. According to school staff, the curriculum was anchored in the Common Core State Standards and the Next Generation Science Standards. As students worked through the standards, they were periodically tested through brief assessments they called “comprehension checks,” as well as longer, more-complex “performance task” assessments. When those were accomplished with a score of at least 75 percent, students moved on to the final assessment for that standard. Comprehension checks and performance tasks could take several forms, but most often were a quiz or a small project. Students tackled the comprehension checks when they were ready (“…at different times…it’s one of the beauties of the school,” according to one student) and moved through the content at their own pace. Students were grouped in the same classes “as other people who are on the same pace with us,” as one student put it. Students progressed to the next standard upon demonstrating mastery. FIGURE Teacher reports of competency-based learning practices 4 ■ To a great extent 100 ■ To a moderate extent 80 Percentage of teachers 60 32% 29% 32% 44% 33% 49% 48% 40 43% 43% 35% 41% 45% 42% 20 33% 35% 19% 16% 16% 21% 19% 0 National NGLC National NGLC National NGLC National NGLC National NGLC sample sample sample sample sample sample sample sample sample sample I require students Different students I give students Students have When students to show that they work on different the chance to opportunities to are working understand a topic topics or skills at work through review or practice independently, before they can move the same time. instructional new material I require them to on to a new topic. material at a faster until they fully get through a or slower pace than understand it. certain amount of other students in material even if this class. they are working at their own pace. Notes: N = 210–212 NGLC teachers; N = 525 national teachers; survey question: “Please indicate the extent to which you agree with each of the following statements about your curriculum and instruction.” Response choices were on a four-point scale from 1 (“not at all”) to 4 (“to a great extent”). Informing Progress Insights on Personalized Learning Implementation and Effects 17  FIGURE Student reports of competency-based learning experiences 5 ■ Always ■ Most of the time ■ Sometimes ■ Never/rarely 100 17% 13% 15% 12% 21% 24% 33% 39% 80 Percentage of students 37% 39% 40% 43% 60 40% 41% 36% 32% 40 29% 28% 24% 26% 30% 22% 20 21% 21% 19% 22% 21% 10% 14% 13% 12% 8% 0 National NGLC National NGLC National NGLC National NGLC sample sample sample sample sample sample sample sample I am given the chance I am required I work on different I have opportunities to work through to show that topics or skills than to review or practice instructional material I understand a new what my classmates new material until at a faster or slower topic before I move are working on at I really understand it. pace than other on to a new topic. the same time. students in this class. Notes: N = 4,784–4,794 NGLC students; N = 864 national students; survey question: “The following questions ask about your classroom experiences. When you answer them, please think about your experiences with all of your classes in math, English/reading, science, and social studies this year, and mark the response that indicates your typical experience.” Response choices were on a five-point scale from 1 (“never”) to 5 (“always”). worked at their own pace was not the same between help students fill in gaps in their learning and access the two groups, this finding is difficult to interpret. grade-level content in preparation for state tests, echoing One possibility is that NGLC teachers wanted to ensure similar comments about offering students choice in their that students maintained focus and continued to work personal learning paths. hard in a self-paced environment. According to most of the teachers we interviewed, part of the rationale Challenges of Competency-Based was a belief that students may need to develop the skills necessary to work at their own pace. Many NGLC Progression school leaders told us that they were taking a “gradual Many NGLC teachers said that allowing students release” approach, in which students initially received to progress at their own pace through content was lots of support and structure, which decreased over challenging when students did not complete work time to allow students to take greater responsibility at an acceptable pace. Teachers and administrators for their own learning. Since most of the NGLC schools at many NGLC schools reported that allowing students were in their first or second year of implementation, it to progress through content at their own pace was is possible that we observed a greater degree of support challenging for several reasons. According to teachers, and structure than will be present in future years. An many students did not know how to organize their alternative explanation for requiring students to cover time so they would complete their work at a sufficient certain material is that teachers wanted to ensure pace. For example, at the end of the second semester coverage of curriculum content that the students would some students still had not completed work they were not have covered at their own pace. Some school leaders expected to do during the first semester. Many schools and teachers said they felt it necessary to set a pace to addressed this challenge by using pacing guides, or by 18 Informing Progress Insights on Personalized Learning Implementation and Effects specifying a minimum amount of work that students awarded credit when they had demonstrated mastery must complete in a certain time frame. of the material. Ideally, credit would be awarded in increments smaller than a course, and the credit would Competency-based grading systems were difficult be transferrable, eliminating the need for students to to explain to stakeholders and did not fit with repeat lessons or courses if they transfer schools. Like traditional reporting practices. Teachers and principals many schools, NGLC schools awarded credit for mastery reported that competency-based grading systems were when students completed a course. Where NGLC schools often challenging to explain to parents and community awarded mastery of material at a finer granularity than members. Principals also reported that competency-based a full course (such as individual learning standards), grades had to be converted into traditional “grades” that students were not able to take these credits with them were acceptable for state-level reporting and college when they transferred to another school, according applications, a challenge encountered by all the NGLC to principals. As a result, students who transferred schools implementing this strategy. schools would likely have to cover the material again. This challenge may stem from a lack of widely accepted NGLC schools did not award credit for partial standards for how to track completion of material in mastery of a course in a way that could be increments of less than a full course. transferred to other schools. One of the goals of competency-based systems is that students would be Informing Progress Insights on Personalized Learning Implementation and Effects 19  Flexible Learning Environments In this section, we discuss: Flexible learning environments imply that the school ■■ flexible use of school resources such as staff, space, adapts the use of resources such as staff, space, and time and time to best support personalization. For example, elements ■■ use of technology in instruction of the learning space—size, classroom organization, and ■■ frequency of adjusting student groups based on data furniture—are designed to support implementation of PL. The structure of learning time and student grouping ■■ challenges of flexible learning environments. strategies are flexible, responsive to student needs, and driven by data where appropriate. Technology is a key as- pect of the school model and is available to all students; often schools provide a device to each student. KEY TAKEAWAYS ON FLEXIBLE LEARNING ENVIRONMENTS NGLC schools appeared to differ from the national Summary sample in some respects. NGLC schools used space, staff, and time in ways that ■■ NGLC teachers reported more-flexible use of were different from schools in the national sample. resources such as space, staff, and instructional time These practices included creating learning spaces that to support PL. were open and flexible, using a variety of activities that ■■ Although both groups reported that technology were based on the needs of the student or the demands played a primary role in instruction, NGLC teachers of the lesson, using student achievement data to assign reported greater reliance on technology-based students to groups, and, among teachers who reported instructional materials than teachers nationally. grouping students by ability level, changing those groups ■■ Obstacles to teaching with technology were reported more frequently. to be less prevalent in NGLC schools. The role of technology in instruction was similar in both ■■ Although both groups considered student grouping samples, as was the use of data to assign students to based on data to be a key strategy, NGLC teachers groups. While these findings are encouraging, some reported adjusting those groups more frequently. NGLC principals, teachers, and students reported that creating and using flexible spaces in traditional school NGLC schools were similar to the national sample in buildings was challenging: Such spaces were often noisy, emphasizing student grouping and use of technology making it difficult for students to concentrate. Some (noted above). aspects of flexible scheduling also proved challenging ■■ Teachers and students in both groups reported for NGLC schools: Schools experienced barriers to flexible positive opinions about the school environment. scheduling at the school level but used time flexibly at the classroom level, and student grouping was more There were several challenges related to flexible learning flexible within classes than schoolwide. Teachers and environments. students in both groups reported similarly positive ■■ Creating flexible learning spaces in traditional school perceptions of the school environment, which could buildings was challenging. enable flexible use of resources in ways that support PL. ■■ NGLC schools experienced barriers to implementing flexible learning environments at the school level, but practices were more flexible at the classroom level. 20 Informing Progress Insights on Personalized Learning Implementation and Effects  VIGNETTE: What do flexible learning environments look like? School D, a charter school serving grades 6–8, had been implementing its PL model for one year at the time of our visit. The school was in a converted office building, with classrooms made out of modular walls that did not reach the ceiling and could be rearranged, and several large open spaces where students could work independently or in groups, and where the school gathered for “design challenges” (complex, interdisciplinary, long-term projects), and other whole-school events. A 5-week “trimester” allowed students who were struggling to solidify their skills, and permitted students who were on track to take interdisciplinary classes that went beyond the regular curriculum (such as public speaking, coding, and “myth busters”). By design, many teachers were cross-certified, enabling administrators to be flexible in how they organized classes, and to play to teachers’ strengths. Many projects were team-taught. Classes were organized in a block schedule, although class length fluctuated as the school experimented with the schedule. The schedule could be rearranged easily, even on short notice, to accommodate projects and whole-school design challenges. In classrooms, the structure of learning time was flexible, teachers had discretion to use the time as they saw fit, and students experienced a variety of instructional approaches and activities depending on the lesson. Students were grouped by learning level schoolwide. Administrators considered standardized test data and consulted with parents and students to make student grouping decisions. In classrooms, grouping was more fluid and often dependent on the lesson requirements. The school had a one-to-one technology model. Students used their own laptops or were supplied with Chromebooks, which they could take home. Students reported using technology “constantly” to monitor their progress, take tests, work on projects, communicate with teachers, do research, and complete assignments. In addition, students could attend classes virtually, using Adobe Connect, during unplanned school closures, such as snow days. “Who teaches what depends on certification area or talents working with low-achieving or high-achieving kids . . .” —Principal comment about staffing Flexible learning environments: reported that their schools had flexible schedules that were intended to facilitate PL. Many NGLC administrators How do NGLC and national practices also mentioned that teachers generally had flexibility to compare? use their classroom time in the way that was best suited NGLC teachers reported more-flexible use of to the lesson and the needs of the students. Teacher resources such as space, staff, and instructional log data confirm that NGLC teachers adjusted their time to support PL. NGLC teachers were somewhat instructional activities to suit the needs of the lesson more likely to report that their schools had large open or the student, though comparative log data are not spaces and comfortable furniture that could easily be available for a national sample. rearranged to facilitate PL. In addition, NGLC teachers were more likely to report that “co-teaching or job- NGLC teachers reported greater reliance on share” best described their teaching arrangement, technology-based instructional materials than although such arrangements were not common overall.1 teachers nationally. Teachers in both groups were NGLC teachers were less likely to report that scheduling equally likely to report that technology played a constraints were an obstacle to implementing PL in their primary role in instruction, but there is some evidence to schools; and many NGLC administrators we interviewed suggest that NGLC teachers relied on technology-based instructional materials for some activities to a greater 1 Co-teaching or job-share was defined as, “I am one of two or extent than teachers in the national sample. For example, more teachers who are jointly responsible for teaching the same NGLC teachers reported that students were engaged subject(s) to a group of students (for example, in the same class- room), all or most of the day and/or in a majority of classes.” in independent practice with software for a larger Informing Progress Insights on Personalized Learning Implementation and Effects 21  proportion of the lesson. In addition, when students were reported adjusting those groups more frequently. using technology, NGLC teachers reported that students Similar proportions of teachers in the NGLC and national more often used structured online curriculum materials; samples reported using student achievement and watched videos, animations, or simulations; solved multi- nonachievement data to assign students to groups within step, open-ended problems; and received immediate their classes. Similar proportions also reported grouping feedback on problem solutions, as shown in Figure 6. students of similar ability levels together. However, there is some evidence that NGLC teachers adjusted Obstacles to teaching with technology were student groupings more frequently. Among teachers reported to be less prevalent in NGLC schools. who reported grouping students of similar ability NGLC teachers were less likely to perceive the following levels together, NGLC teachers changed groups more operational and logistical obstacles to promoting student frequently: 29 percent reported changing groups weekly, learning using technology: compared with 4 percent of teachers in the national sample. ■■ an inadequate number of devices (e.g., laptops) ■■ problems with hardware Teachers and students in both the national and ■■ inadequate bandwidth NGLC schools reported positive opinions about the ■■ lack of opportunities to participate in professional school environment. Large majorities of teachers in development both samples agreed that administrators and teachers ■■ lack of flexibility in deciding how to use technology were focused on improving student learning, were in instruction supportive, and that teachers collaborated well with one another. NGLC teachers were more likely to report ■■ lack of support from technology specialists high levels of administrator support and trust, and that ■■ inadequate opportunities for teachers to provide teachers were highly focused on improving student input on how technology is used. learning, as shown in Figure 7. Teachers’ perceptions of students were similar and largely positive in both Although both groups considered student grouping samples, with majorities agreeing that students based on data to be a key strategy, NGLC teachers were respectful of other students and staff and were FIGURE Teacher reports of the extent to which students engaged 6 in certain activities while using technology 100 ■ To a great extent ■ To a moderate extent 80 Percentage of teachers 60 24% 37% 29% 40 30% 24% 25% 30% 20 40% 23% 31% 35% 21% 18% 14% 12% 10% 0 National NGLC National NGLC National NGLC National NGLC sample sample sample sample sample sample sample sample Using structured Watching videos, Solving multi-step, Receiving immediate curriculum materials animations, or open-ended feedback on problem online simulations problems or solutions conducting investigations Notes: N = 213–216 NGLC teachers; N = 525 national teachers; survey question: “For this question, we are interested in the activities students are engaged in when they are using technology. Please indicate the extent to which students are engaged in the following types of activities.” Response choices were on a four-point scale from 1 (“not at all”) to 4 (“to a great extent”). 22 Informing Progress Insights on Personalized Learning Implementation and Effects  FIGURE Teacher agreement with items related to school environment, NGLC and nationally 7 ■ Strongly agree ■ Agree 100 80 Percentage of teachers 41% 43% 42% 40% 60 60% 59% 54% 40 40% 49% 55% 20 43% 43% 31% 22% 21% 23% 0 National NGLC National NGLC National NGLC National NGLC sample sample sample sample sample sample sample sample Administrators at Administrators at Administrators at Teachers at my my school are highly my school are highly my school trust school are highly focused on student supportive of teachers to make focused on the learning. teachers. decisions about their mission of improving own instruction. student learning. Notes: N = 228–230 NGLC teachers; N = 525 national teachers; survey question: “Rate your level of agreement with each of the following statements about your school.” Response choices were on a four-point scale from 1 (“disagree strongly”) to 4 (“agree strongly”). motivated to achieve. However, in both groups, about schools were located in traditional school buildings that half of teachers did report that certain factors were often could not be reconfigured to use space in flexible obstacles to implementing PL, such as too many students ways. Nonetheless, a majority of administrators reported in classes, too much diversity in achievement levels, high that their schools contained some flexible learning levels of absenteeism, disciplinary problems, motivation, spaces. Where nontraditional spaces existed, using them behavior, or attendance. was not without challenges. In particular, staff and students in such schools reported that open spaces were Students had similarly positive perceptions of their school noisy, making it difficult to focus on instruction. environment. Large majorities of students in the national and NGLC samples reported positive feelings about their NGLC schools experienced barriers to implementing schools and learning environments, and the two groups flexible learning environments at the school level, were equally likely to agree that they felt supported but practices were more flexible at the classroom by their teachers in their school work and in preparing level. Principals reported that flexible grouping was for the future, as shown in Figure 8. Although a large rarely used at the school level, and in most schools majority of NGLC students expressed positive opinions, students were grouped by traditional grade level. Teacher they were somewhat less likely to report that they felt survey and interview data indicate that student grouping safe, comfortable being themselves, and an important was more flexible within classes than schoolwide. part of the school community. Teachers reported that data-based student grouping strategies were used frequently at the classroom level, Challenges of Flexible Learning where students were sometimes grouped homogeneously and sometimes heterogeneously, according to the goals Environments in NGLC Schools of the lesson. Creating flexible learning spaces in traditional school buildings was challenging. Most of the NGLC Informing Progress Insights on Personalized Learning Implementation and Effects 23  FIGURE Students’ opinions about their school environment 8 ■ Strongly agree 100 ■ Agree 80 Percentage of students 54% 49% 60 54% 50% 61% 52% 50% 45% 54% 57% 52% 50% 40 20 34% 38% 37% 31% 30% 29% 32% 25% 23% 26% 25% 22% 0 National NGLC National NGLC National NGLC National NGLC National NGLC National NGLC sample sample sample sample sample sample sample sample sample sample sample sample Teachers work All students are Teachers pay I feel safe in There is at least I am an important hard to make sure encouraged to go attention to all this school. one adult in this part of my school that all students to college. students, not just school who knows community. are learning. the top students. me well. Notes: N = 4,629–4,665 NGLC students; N = 864 national students; survey question: “How much do you agree with the following statements about your school?” Response choices were on a four-point scale from 1 (“strongly disagree”) to 4 (“strongly agree”). However, implementing flexible learning time at reported that it was difficult for logistical reasons. Staff the school level could also take the form of flexible at these schools reported that they struggled to create scheduling, where each student would have a unique classes of a reasonable size, or ensure that there were schedule that changed as often as weekly based on data enough teachers available to supervise students working about their learning needs. Only a few NGLC schools independently. Some school leaders found that the took this approach, and in those that did, the principals work of creating a new schedule each week was too burdensome. Teachers, however, reported that they were empowered to use their classroom time flexibly, which in most schools meant using a variety of instructional “We started off with teachers creating a strategies in accord with the needs of the lesson or different schedule for the students every week. the student. It just took too much time because they were doing it all by hand . . .” —Principal comment about flexible schoolwide scheduling 24 Informing Progress Insights on Personalized Learning Implementation and Effects General Challenges to Implementing Personalized Learning The teacher survey included several questions that addressed challenges to implementing PL. These questions asked about implementing PL in general and were not specific to any one of the four strategies; we therefore discuss them briefly in this section. NGLC teachers were less likely than teachers operational factors, such as lack of administrator support, nationally to report operational obstacles to pressure to cover specific material, lack of data, lack of implementing PL, such as scheduling constraints. flexibility in curriculum, and scheduling constraints, were We examined teacher perceptions of obstacles to obstacles. Pressure to cover specific material and lack implementing PL in the NGLC and national samples. of flexibility in the curriculum seemed to be the largest Although some of these conditions were not perceived as obstacles for teachers in the national sample, as shown in obstacles by a majority of teachers, teachers in the NGLC Figure 9. sample were less likely to report that environmental and FIGURE Teacher reports of obstacles to implementing PL 9 100 ■ Major obstacle 80 Percentage of teachers ■ Minor obstacle 60 27% 40 37% 37% 21% 26% 25% 20 38% 28% 18% 22% 21% 18% 19% 17% 14% 13% 7% 5% 6% 3% 0 National NGLC National NGLC National NGLC National NGLC National NGLC sample sample sample sample sample sample sample sample sample sample Lack of support Lack of flexibility Pressure to cover Inadequate data Scheduling from school in the curriculum specific material to help me constraints administration I am required as a result of personalize to teach state or district students’ standards or testing instruction requirements Notes: N = 217–219 NGLC teachers; N = 525 national teachers; survey question: “Please indicate the extent to which each of the following conditions is an obstacle to your efforts to promote personalized learning for students. If the condition does not exist in your school, please mark ‘not applicable.’” Response choices were on a four-point scale from 1 (“not applicable; condition does not exist in my school”) to 4 (“condition exists and is a major obstacle”). Informing Progress Insights on Personalized Learning Implementation and Effects 25  How Did Charter and District NGLC Schools Compare in Their Implementation of Personalized Learning? This section compares charter and district KEY TAKEAWAYS FROM THE implementation of COMPARISON OF CHARTER AND ■■ learner profiles DISTRICT SCHOOLS ■■ personal learning paths ■■ competency-based progression Implementation of PL in Charter ■■ flexible learning environments. and District Schools Within the NGLC Sample With early signs that PL holds promise for positive effects Charters appeared to have higher levels of on student achievement, there has been considerable implementation than district schools in some ways. enthusiasm about scaling up its implementation. ■■ Teachers reported more-frequent receipt and more- Although many of the earliest adopters of PL have been extensive use of actionable student data. charter schools, successful scale-up of this approach will ■■ More teachers reported adapting course content to inevitably include district-operated public schools, which meet students’ needs to a great extent. serve the vast majority of K–12 students in the United ■■ Teachers reported using small-group instruction for States. A manifest question is whether the positive results larger portions of the lesson. seen thus far in samples that are dominated by charter ■■ Teachers were more likely to agree that their schools are likely to generalize broadly. For example, curriculum materials were of high quality. charter schools comprised 92 percent of the sample that produced favorable results in the Pane et al. (2015) ■■ Teachers and students reported more-extensive use study. Are there attributes of charter schools that are of competency-based practices. particularly conducive to implementing a somewhat ■■ Flexible use of space and staff was more prevalent. radical innovation like PL, or should we expect scale-up ■■ Teachers reported incorporating more technology in districts to proceed with similar results to those seen into instruction and fewer obstacles to doing so. in these charters? We use the limited data available in ■■ Teachers reported greater use of data to group the current study to conduct a preliminary exploration students. of this topic. Our small sample consists of one-fourth ■■ Teachers and students reported more-positive district-operated and three-fourths charter schools. Here, perceptions on some dimensions of school we examine implementation similarities and differences environment. we observed between district and charter schools in our ■■ Teachers were less likely to report that student sample, and we examine achievement outcomes along factors, such as discipline, were major obstacles to PL. the same dimension in the next chapter. Although these analyses do not enable strong conclusions due to small sample sizes, and thus should be interpreted with great District schools appeared to have higher levels of caution, they may offer some observations that warrant implementation than charters in some ways. consideration by stakeholders interested in the scale-up ■■ District schools appeared to have more- of PL. comprehensive learner profiles. ■■ Teachers reported changing student groupings more frequently. 26 Informing Progress Insights on Personalized Learning Implementation and Effects Charter and district schools appeared to be similar in groups also reported using student achievement data for several ways. activities related to personalization, but charter teachers ■■ Teachers reported similar levels of use of learner reported more-extensive use of such data for many profiles. instructional activities. Charter teachers were also slightly ■■ Opportunities for student choice were uncommon. more likely to agree that their school data systems ■■ Teachers were equally likely to assign students to provided them with actionable data. Majorities of classes and groups by age or achievement. teachers in both district and charter schools agreed that data systems were easy to use and provided them with ■■ Teachers reported using classroom time flexibly and real-time, actionable data, but charter teachers expressed incorporating a variety of activities. stronger agreement, as shown in Figure 10. ■■ Teachers and students reported positive perceptions of the school environment. Teachers in both groups reported similar levels of use of learner profiles, but district schools Summary appeared to have more-comprehensive learner profiles. About half of charter and district teachers In general, charter schools tended to display more- reported that their schools used learner profile extensive implementation of many aspects of PL. District documents. Among teachers who reported that their schools displayed less-extensive implementation and schools used learner profiles, district and charter teachers tended to look more similar to the national sample, were equally likely to report that the profiles were suggesting lower implementation of novel PL practices. frequently updated and set forth a plan for students to Charter teachers reported greater use of key aspects of accomplish their learning goals. Charter teachers were learner profiles, such as more-frequent receipt and use of less likely to report that the profiles were comprehensive student data, and greater adaptation of course content and available for every student. to meet students’ needs. Charter teachers and students reported a greater extent of using and experiencing competency-based practices, such as being able to work Personal Learning Paths on different topics than others and at their own pace. More charter teachers reported adapting Key components of flexible learning environments, such course content to meet students’ needs as flexible use of staff and space, and use of technology, to a great extent. Surveyed teachers in both groups were reportedly more common in charter schools. reported a limited amount of student choice in the instructional materials and topics students used and Reported use of learner profile documents was the focused on, with about one-third of teachers responding same in both groups, although charter teachers’ that they provided such choice. As described above, responses suggest that their learner profiles were less NGLC teachers tended to make efforts to adapt course comprehensive. As with the national and district samples, content to meet students’ needs by providing additional opportunities for student choice were uncommon, assignments, resources, and activities for remediation although charter teachers reported using small-group or enrichment. Charter teachers reported using these instruction more frequently. Flexible use of class time approaches to a greater extent than district teachers. was common in both groups, as was using a variety of instructional strategies. Teachers and students in both Opportunities for student choice were uncommon groups reported similarly positive perceptions of the in both groups, but charter teachers reported school environment. using small-group instruction for larger portions of the lesson. Students in both groups reported that Due to numerous limitations, these findings should be they were not given a great deal of choice in the topics interpreted with great caution. or materials they used in their classes. Less than half of students reported that it was very or mostly true Learner Profiles that their teachers took their experiences and interests Charter teachers reported more-frequent into account when deciding what they would work on. receipt and more-extensive use of Similarly, about one-third of teachers in both groups actionable student data. In both groups, majorities of reported that students had opportunities to choose teachers reported receiving a variety of achievement and the instructional materials they used in class. However, nonachievement data at least a few times per month. there is some evidence that charter teachers attempted However, charter teachers reported receiving such data to address individual students’ needs. Charter teachers more frequently: approximately weekly. Teachers in both were less likely to report using large-group instruction for Informing Progress Insights on Personalized Learning Implementation and Effects 27  FIGURE Charter and district teachers’ opinions of their schools’ data systems, 10 with national sample results included as a reference 100 ■ Agree ■ Strongly agree 80 Percentage of teachers 41% 39% 60 35% 39% 38% 49% 53% 46% 39% 40 43% 47% 43% 38% 47% 38% 43% 42% 45% 20 37% 38% 25% 30% 24% 24% 22% 17% 21% 16% 17% 15% 0 Charter District National Charter District National Charter District National Charter District National Charter District National Our school’s data I have access to The data system Our school’s data Our school’s data system is easy high-quality provides real-time system includes system provides to use. assessment data data that is achievement information at a that help me adapt actionable. measures that level of detail that the pace or content provide helps me inform of instruction to information about my instruction. meet students’ needs. students of varying achievement levels. Notes: N = 151–153 charter teachers; N = 61–63 district teachers; N = 525 national teachers; survey question: “Please indicate your level of agreement with each of the following statements.” Response choices were on a five-point scale of 1 (“not applicable”) to 5 (“strongly agree”). For each item and group, 3 to 8 percent of teachers indicated the item was not applicable. large portions (i.e., more than 40 percent) of the lesson, although this percentage was still higher than in the and more likely to report using small-group instruction. national sample (Figure 11). Daily instructional logs seem to confirm this pattern: teachers reported using Charter teachers were more likely to agree that competency-based learning practices for more than their curriculum materials were of high quality. a small portion of the lesson but less than half of the Access to high-quality curriculum materials is a key lesson. Charter teachers reported using competency- enabler of personal learning paths. Majorities of charter based practices for a larger portion of the lesson overall, and district teachers agreed that their materials were of and that more of the lesson involved content students high quality and met the needs of all of their students, could experience at different levels of depth. Charter though charter teachers expressed slightly stronger students also reported experiencing some competency- agreement. Charter teachers also reported that a greater based practices to a greater extent than district proportion of their curriculum materials were provided students and students nationally. Charter students were to them. slightly more likely to report that they were required to demonstrate understanding of a topic before they Competency-Based could move on to the next one, and that they had Progression opportunities to practice or review until they fully understood the material. Charter and district students Charter teachers and students reported were equally likely to report that they could work on greater use of competency-based practices. different topics or skills than their classmates at the same Although a majority of teachers in both groups reported time and work at a different pace than other students in using competency-based practices to some extent, larger the class. District students reported experiencing most of proportions of charter teachers reported using these these practices to a similar extent as students nationally, practices to a great extent, as shown in Figure 11. In as seen in Figure 12. most cases, fewer district teachers reported using these practices to a great extent than did charter teachers, 28 Informing Progress Insights on Personalized Learning Implementation and Effects Charter and district teachers were equally likely under the supervision of another teacher” best described to assign students to classes and groups by age or their teaching arrangement, although such teaching grade level. Teacher logs of daily practices show that arrangements were not common overall. Principals of students were assigned to their class by age or grade several NGLC charter schools reported that noncertified level in about two-thirds of lessons; assignment to classes staff acted as advisers and mentors to students; we did or groups by achievement level was relatively uncommon. not hear about similar roles in most district schools. When students were grouped by achievement level, Together, these findings suggest that charters were teachers reported using homogeneous groups in more employing staff in unconventional roles to a greater lessons than heterogeneous groups. This is consistent extent than district schools. Finally, charter teachers were with what many administrators told us in interviews. somewhat less likely to report that scheduling constraints were an obstacle to implementing PL practices. Flexible Learning Charter teachers reported greater use of data to Environments group students, but district teachers reported Flexible use of space and staff was more changing groupings more frequently. A majority prevalent in charter schools than in district schools. of teachers in both groups reported using achievement Charter teachers were somewhat more likely to report data to assign students to groups within their classes, but that their school had nontraditional instructional spaces charter teachers reported doing so to a greater extent. (e.g., space with comfortable furniture, large open Charter teachers also reported grouping students of instructional spaces, open common areas for student use, similar ability levels together somewhat more frequently, and breakout rooms) and that those spaces facilitated a finding that is consistent with charter teachers’ greater PL practices. Charter teachers were also more likely to reported use of competency-based instructional practices. report that “co-teaching or job-share” or “working However, among the teachers who reported grouping FIGURE Charter and district teachers’ implementation of competency-based learning 11 practices, with national sample results included as a reference 100 ■ Great extent ■ Moderate extent 80 Percentage of teachers 34% 29% 60 28% 41% 37% 26% 53% 42% 40 49% 31% 48% 43% 20% 43% 35% 46% 47% 48% 20 36% 38% 40% 27% 26% 28% 25% 19% 16% 16% 21% 19% 0 Charter District National Charter District National Charter District National Charter District National Charter District National I require students Different students I give students the Students have When students to show that they work on different chance to work opportunities to are working understand a topic topics or skills at through review or practice independently, before they can the same time. instructional new material I require them to move on to a material at a faster until they fully get through a new topic. or slower pace than understand it. certain amount of other students material even if in this class. they are working at their own pace. Notes: N = 149–150 charter teachers; N = 59–61 district teachers; N = 525 national sample; survey question: “Please indicate the extent to which you agree with each of the following statements about your curriculum and instruction.” Response choices were on a four-point scale from 1 (“not at all”) to 4 (“to a great extent”). Informing Progress Insights on Personalized Learning Implementation and Effects 29  FIGURE Charter and district students’ experiences of competency-based learning practices, 12 with national sample results included as a reference 100 80 ■ Always Percentage of students ■ Most of the time 60 40 31% 30% 25% 24% 26% 26% 26% 30% 20 21% 22% 20% 21% 20% 17% 24% 22% 19% 18% 14% 10% 12% 15% 8% 12% 0 Charter District National Charter District National Charter District National Charter District National I am given the chance I am required I work on different I have to work through to show that I topics or skills than opportunities to instructional understand a topic what my classmates review or practice material at a faster before I move on are working on at new material until or slower pace than to a new topic. the same time. I really understand it. other students in this class. Notes: N = 2,964–2,978 charter students; N = 1,727–1,732 district students; N = 864 national students; survey question: “The following questions ask about your classroom experiences. When you answer them, please think about your experiences with all of your classes in math, English/reading, science, and social studies this year, and mark the response that indicates your typical experience.” Response choices were on a five-point scale from 1 (“never”) to 5 (“always”). students by ability level, district teachers reported Charter teachers reported incorporating more changing groupings more frequently. technology into instruction and fewer obstacles to doing so. Charter teachers were more likely to report Teachers in both groups reported using classroom that technology played a primary role in instruction and time flexibly and incorporating a variety of that students used structured curriculum materials online activities. Charter and district teachers reported similar to a moderate or great extent. The proportion of time use of instructional time and instructional activities. teachers reported using technology in the classroom Teacher-led small-group instruction and small-group was similar in the two groups. Regarding technology collaboration were used for more of the lesson than obstacles, the two groups reported similar levels of large-group instruction. Teachers’ reports of their daily logistical issues (e.g., lack of opportunity to participate in practice confirm that a variety of activities were used professional development), but district teachers reported across lessons, and for a small portion of the lesson, more hardware and infrastructure issues, as shown in suggesting that teachers changed activities frequently Figure 13. based on the needs of the student or the requirements of the lesson. Teachers and students in both groups reported positive perceptions of the school environment, with charter teachers reporting more-positive Activities used by teachers in both groups perceptions on some dimensions. Large majorities ✔ students working independently with and of NGLC teachers in both groups expressed positive opinions about the school environment. However, charter without technology teachers were more likely to strongly agree that their ✔ students working with teacher support with school had high levels of administrator support and trust, and without technology and that teachers were highly focused on improving ✔ students working in small groups with and student learning. Charter teachers also reported more- positive perceptions of family involvement. Students in without technology. NGLC charter and district schools had similarly positive 30 Informing Progress Insights on Personalized Learning Implementation and Effects  perceptions of their school environment. Large majorities of students reported positive feelings about their schools and learning environments. The two groups were equally likely to agree that they felt supported by their teachers in their school work and in preparing for the future, and that they felt safe, supported by teachers, comfortable being themselves, and an important part of the school community. Charter teachers were less likely to report that student factors, such as discipline, were major obstacles to PL. Although majorities of charter and district teachers reported positive perceptions of students, charter teachers were more likely to agree that students in their school were motivated to achieve, and less likely to report that student characteristics and behavior were obstacles to implementing PL, as shown in Figure 14. In most cases, charter teachers perceived that these student-related factors were obstacles at similar rates as teachers nationally. Student absenteeism appears to be a notable exception; charter teachers were much less likely to report that this was a major obstacle than district teachers and teachers nationally. FIGURE Charter and district teacher reports of obstacles to using technology, 13 with national sample results included as a reference ■ Major obstacle 100 ■ Minor obstacle 80 Percentage of teachers 60 38% 40 29% 30% 36% 32% 35% 34% 24% 20 23% 22% 30% 17% 18% 19% 18% 11% 24% 25% 25% 22% 11% 22% 14% 14% 7% 11% 6% 11% 6% 11% 11% 5% 5% 5% 13% 7% 8% 13% 6% 9% 0 3% Charter District National Charter District National Charter District National Charter District National Charter District National Charter District National Charter District National Slow internet An inadequate Inadequate Problems with Lack of Lack of support Inadequate connection or number of opportunities hardware, such flexibility in from technology opportunities inadequate computers or for teachers to as insufficient deciding how specialists or to participate bandwidth devices to provide input computing I can use other staff who in professional accommodate on how power or lack technology in can provide development all students technology of compatibility my instruction technical related to is used with software support technology use Notes: N = 160–163 charter teachers; N = 62–63 district teachers; N = 525 national teachers; survey question: “Please indicate whether the following conditions exist in your school and the degree to which each is an obstacle to your efforts to promote student learning using technology such as computers, smartphones, or tablets. If the condition does not exist in your school, please mark ‘not applicable.’” Response choices were on a four-point scale from 1 (“not applicable; condition does not exist in my school”) to 4 (“condition exists and is a major obstacle”). Informing Progress Insights on Personalized Learning Implementation and Effects 31  FIGURE Charter and district teacher perceptions of student-related obstacles to PL, 14 with national sample results included as a reference 100 ■ Major obstacle 80 Percentage of teachers ■ Minor obstacle 60 27% 30% 40 33% 32% 32% 26% 28% 25% 34% 35% 30% 20 35% 35% 30% 25% 22% 20% 25% 23% 21% 15% 15% 17% 0 5% Charter District National Charter District National Charter District National Charter District National Too many students Too much diversity High levels High levels of for whom I am in achievement of student student disciplinary responsible levels among my absenteeism problems students Notes: N = 153–155 charter teachers; N = 62–63 district teachers; N = 525 national teachers; survey question: “Please indicate the extent to which each of the following conditions is an obstacle to your efforts to promote personalized learning for students. If the condition does not exist in your school, please mark ‘not applicable.’” Response choices were on a four-point scale, from 1 (“not applicable; condition does not exist in my school”) to 4 (“condition exists and is a major obstacle”). 32 Informing Progress Insights on Personalized Learning Implementation and Effects How Did NGLC Schools Affect Student Achievement? Now our attention shifts from implementation of PL to an examination of achievement effects. To measure achievement effects over one academic year, the study analyzed mathematics and reading scores for all students in the NGLC schools who took the NWEA MAP assessments in fall 2014 and spring 2015. Similarly, to measure achievement effects over two academic years, the study analyzed scores for students who took the assessments in fall 2013 and spring 2015. MAP is an online adaptive test in which the test software adjusts the consecutive difficulty of questions in response to an individual student’s answer. If a student responds incorrectly, the next question is easier; if a student responds correctly, the test software progresses to a more difficult question. The MAP assessment can provide accurate information on a broad range of student ability from kindergarten to grade 11, including how much progress a student makes over the course of a school year. Each NGLC student in the achievement analysis was matched to a set of similar students to form a “virtual comparison group” (VCG). More details are available in Appendix B. Of the 40 NGLC schools, 32 had MAP data available for the one-year span, representing approximately 5,500 students, and 16 had data available for the two- We report achievement effects of year span, representing about 1,800 students. While these schools and students were all included when PL using effect sizes, a standard estimating effects for the NGLC schools in the aggregate, school-level results are not reported where data were way researchers measure the available for fewer than 30 students. All of these schools impact of an educational strategy. implemented PL schoolwide during the years they are included in the analyses. As discussed above, these results This allows researchers to make are not directly comparable to achievement analyses presented previously in Pane et al. (2015) because comparisons across research studies. the current sample is composed of mostly secondary schools relatively new to implementing PL, whereas the To assist with interpretation, we sample for the prior report, on average, had greater PL also translate the effect sizes into experience and a majority of elementary schools. the percentile rank of a PL student Overall Results who would have performed at the In this report, we focused our analysis of treatment effects primarily on the NGLC schools for which we have median (50th percentile) if they had data for the most recent year (2014–15). This one-year span has the greatest number of schools and students, been in a non-PL school. Informing Progress Insights on Personalized Learning Implementation and Effects 33  FIGURE Analyses show positive effects for NGLC 15 schools for the 2014–15 academic year ■ Mathematics 0.16 ■ Reading 0.14 0.12 0.1 Effect size 0.08 0.06 0.04 0.02 0 −0.02 Subject Percentile gain 3 3 Number of 5,539 5,474 students Note: Solid bar indicates statistical significance (p < 0.05) after adjustment for multiple hypothesis tests. Percentile gains translate the treatment effect sizes into the amount of improvement experienced by the median student. but also includes some schools that were in their first year of implementing PL. We estimated positive treatment effects of approximately 0.09 in mathematics and 0.07 FIGURE Students started below national in reading, as shown in Figure 15. Only the mathematics 16 norms and approached them by estimate is statistically significant. These effect sizes end of the academic year translate to gains of about 3 percentile points; Fall 2014 to Spring 2015 specifically, a student who would have performed at the ■ Fall 2014 median in the comparison group is estimated to have 60 performed 3 percentile points above the median in an ■ Spring 2015 NGLC school in both subjects. National norm 50 Percentile rank equivalent The average fall and spring student national percentile ranks are shown in Figure 16. Here, instead of using 40 statistical models with control variables to compare NGLC students with a matched set of students (the VCGs), we 30 simply compare their average performance to national norms for their grade. The figure shows that students 20 started the year significantly below national norms in both mathematics and reading, and gained a modest amount during the school year. In mathematics, students 10 gained about two percentile points but remained significantly below national norms; in reading, students 0 also gained about two percentile points and were Mathematics Reading performing approximately at national norms by spring. Note: Solid bars indicate statistically significant differences from national norms (p < 0.05) after adjustment for multiple hypothesis tests. 34 Informing Progress Insights on Personalized Learning Implementation and Effects Results Over Time Although our primary analysis focuses on a single year FIGURE Percentile rank changes over of academic growth over the 2014–15 academic year, 17 two academic years there were 16 NGLC schools that had been in operation For the 16 schools in the study that started the prior year, 2013–14, and administered the MAP implementing PL in 2013 or earlier, restricted assessment in both academic years. To examine growth to the students present for all four tests trajectories in those schools, we restricted the sample to students with test scores in fall and spring of both ■ Fall 2013 ■ Fall 2014 academic years, and examined their scores relative to 60 ■ Spring 2014 ■ Spring 2015 national norms. The results are shown in Figure 17. In both mathematics and reading, cumulative growth over National norm 50 the two years is evident. Students started significantly Percentile rank equivalent below national norms, gained ground after one academic 40 year, and gained further ground the second academic year, placing them above national norms (though not statistically significantly above) at the end of two years. 30 The largest gains on average appeared to occur in the second year, suggesting that PL systems may require some 20 experience before operating at their fullest potential. 10 Results by School Figure 18 displays treatment effect estimates for each 0 NGLC school for the 2014–15 academic year. In each Mathematics Reading subject, we included only schools for which we had Note: Solid bars indicate statistically significant differences from data on at least 30 students (dropping one school in national norms (p < 0.05) after adjustment for multiple hypothesis tests. mathematics). Where the estimates were statistically significant, the bars are solid. K–8 schools are colored red, elementary schools purple, middle schools orange, and Effects for Groups at Different Starting high schools blue. Superscripts next to the school number Achievement Levels indicate a district-run school. Overall, a slight majority of First, we examine the distribution of students in the schools were estimated to have positive effects, though NGLC sample based on their scores at the beginning of they are not always significant. Middle schools have the 2014–15 school year. We defined five levels (quintiles) strong representation among the schools with significant based on national norms, such that an equal number of positive estimates, and many of the district schools have students nationally are in each group, with each higher negative estimates. Figure 19 displays treatment effect quintile containing students with higher fall scores. If estimates for each NGLC school in operation for the NGLC students started out similar to national norms for two-year span of 2013–15, and with data from at least their grade level, we would expect each group to hold 30 students (dropping three schools in both mathematics about 20 percent of the sample. However, as we saw and reading). In both mathematics and reading, about in Figure 16, the NGLC sample as a whole was below half of the schools have positive treatment estimates. national norms in fall 2014. Figure 20 shows that about one-quarter of the NGLC students were in the lowest- Effects for Subgroups achieving group in both subjects. PL could have greater or lesser effects for various As a way of examining achievement effects within each subgroups of students. In this section, we examine effects of these groups, we calculate the fraction of NGLC by starting achievement level, grade level, gender, and students who surpassed their VCGs in raw score growth. whether the school is a district-operated or a charter If the NGLC sample grew similarly to students nationally, school. We lacked data to examine race/ethnicity or high- we would expect about 50 percent of NGLC students to poverty subgroups. surpass their VCGs, and 50 percent not to, simply due to random fluctuations. Figure 21 shows that across the Informing Progress Insights on Personalized Learning Implementation and Effects 35  FIGURE Treatment effect estimates for the 2014–15 academic year, by school 18 2014–15 effect sizes by school Mathematics Reading 28 0.35 1 0.63 19 0.31 30 0.42 32a 0.29 5 0.39 16 0.24 2 0.36 1 0.23 32a 0.32 2 0.22 19 0.24 13 0.18 28 0.22 17 0.18 16 0.20 11 0.17 7 0.18 29 0.16 4 0.17 24a 0.14 29 0.15 18 0.13 18 0.12 7 0.13 6a 0.10 17 0.09 24a 0.09 4 0.07 8 0.09 School ID School ID 25 0.03 11 0.08 15 0.01 25 0.07 13 0 5 0.02 9 −0.01 3 0.01 12 −0.01 22 0.01 6a −0.05 9 −0.03 31 −0.07 10a −0.04 8 −0.08 15 −0.05 −0.08 3 31 −0.05 −0.09 26a 14a −0.06 −0.12 10a 23a −0.09 14a −0.12 20 −0.09 22 −0.13 12 −0.11 20a −0.20 ■ K–8 −0.17 ■ Elementary 21 23a −0.21 ■ Middle 26a −0.20 21 −0.24 ■ High 27 −0.44 27 −0.51 −0.60 −0.40 −0.20 0.00 0.20 0.40 0.60 0.80 −0.60 −0.40 −0.20 0.00 0.20 0.40 0.60 0.80 Effect size Effect size Note: Solid bars indicate significance at the p < 0.05 level after adjustment for multiple hypothesis tests. a Indicates district schools. 36 Informing Progress Insights on Personalized Learning Implementation and Effects  FIGURE Treatment effect estimates for the 2013–15 two-year time span, by school 19 2013–2015 effect sizes by school Mathematics Reading 19 0.54 19 0.36 32a 0.29 2 0.25 17 0.24 32a 0.25 24a 0.15 17 0.19 2 0.13 0 0.14 25 0.02 24a 0.13 School ID School ID 10a −0.02 25 0.12 9 −0.07 19 −0.04 15 −0.12 15 −0.18 29 −0.18 10a −0.19 ■ K–8 3 −0.32 20a −0.21 ■ Elementary ■ Middle 20a −0.49 3 −0.23 ■ High 26a −0.52 26a −0.25 −0.60 −0.40 −0.20 0.00 0.20 0.40 0.60 0.80 −0.60 −0.40 −0.20 0.00 0.20 0.40 0.60 0.80 Effect size Effect size Note: Solid bars indicate significance at the p < 0.05 level after adjustment for multiple hypothesis tests. a Indicates district schools. achievement spectrum, more than half of NGLC students surpassed their comparison students. This suggests that FIGURE Percentages of students in each quintile PL is benefiting students of all ability levels. For the 20 of starting achievement based on lowest four quintiles, approximately 60 percent of NGLC national norms students surpassed their VCGs in both mathematics and Fall 2014 reading; for the highest quintile, the percentages are in the mid-50s. 30 ■ Mathematics ■ Reading 25 25% 24% Effects by Grade Span Percentage of NGLC students Figure 22 displays estimated treatment effects for 20% 20% three grade spans, K–5 (elementary), 6–8 (middle), 20 19% 19% 19% 19% 18% and 9–12 (high). With the exception of elementary 17% reading, the estimates are positive for both subjects, 15 though only the middle school mathematics estimate is statistically significant. The largest estimates are for the 10 middle school grades. These results contrast with those presented in Pane et al. (2015), where elementary schools 5 performed the strongest, probably due to differences in the two samples. 0 Bottom 2nd 3rd 4th Top quintile quintile quintile quintile quintile Informing Progress Insights on Personalized Learning Implementation and Effects 37  FIGURE Percentages of students surpassing 21 their virtual comparison groups by Effects by Gender quintiles of starting achievement We did not find evidence of differing PL treatment 2014–15 effects by gender. As mentioned above, limitations of the ■ Mathematics data do not enable us to examine other demographic ■ Reading subgroups, such as race/ethnicity or socioeconomic status. 70 Percentage of NGLC students within quintile Effects in District-Operated 60 and Charter Schools Among the NGLC schools with achievement data, eight 50 are operated by school districts and the remaining 24 are charter schools. The district schools are all high schools 40 and middle schools, whereas the charter schools include schools of all levels. To review, Figure 18 showed that 30 many, though not all, of the district schools had negative estimates of treatment effects, and Figure 22 showed 20 that estimated treatment effects vary depending on the grade levels served by the schools. 10 The analysis here calculates the average treatment effects for district and charter schools separately. As shown in 0 Figure 23, the charter schools performed similarly in Bottom 2nd 3rd 4th Top quintile quintile quintile quintile quintile both mathematics and reading, with estimated effects near 0.10 (only the mathematics estimate is significant). Note: Solid bars indicate statistically significant differences from national norms (p < 0.05) after adjustment for multiple The district schools have smaller estimates—about half hypothesis tests. as large in mathematics and near zero in reading. Due 38 Informing Progress Insights on Personalized Learning Implementation and Effects  FIGURE Estimated effects for elementary, FIGURE Estimated effects for district-operated 22 middle, and high-school grade bands 23 and charter schools 2014–2015 2014–2015 ■ District schools ■ Grades K–5 ■ Charter schools 0.3 ■ Grades 6–8 0.25 ■ Grades 9–12 0.25 0.2 0.2 0.15 0.15 0.1 Effect size Effect size 0.1 0.05 0.05 0 Mathematics Reading 0 −0.05 −0.05 −0.1 −0.1 −0.15 Mathematics Reading −0.15 −0.2 Note: Solid bar indicates statistical significance (p < 0.05) Note: Solid bar indicates statistical significance (p < 0.05) after after adjustment for multiple hypothesis tests. adjustment for multiple hypothesis tests. to small samples, the district estimates are particularly imprecise. This imprecision, along with the differences in the grade levels served by the two groups, suggests that these trends in district versus charter schools should be treated as suggestive but not conclusive. These Findings Withstand a Series of Rigorous Sensitivity Analyses To help evaluate the robustness of the findings discussed in this chapter, we performed a variety of sensitivity tests. These included analyses based on national norms of growth, restricting the VCGs to come from the same school type (charter or district) as the corresponding NGLC school, and examining the effect of test duration on results. The rationale, methods, and results of these tests are discussed in Appendix B. After evaluating the results of these sensitivity tests, we concluded that they support the results presented here and the substantive conclusions we are able to draw. Informing Progress Insights on Personalized Learning Implementation and Effects 39  Implications and Policy Recommendations Review of Implications Key Findings Although advocates and reformers have developed PL models, many of the component practices are relatively common nationally, making it difficult to ■ NGLC teachers reported higher levels of clearly identify what makes a school a PL school. implementation than teachers nationally At a theoretical level, PL is very different from the on some aspects of PL. instructional approaches that have been typical in K–12 ■ NGLC schools looked more like the schooling in the United States. It puts a primary focus on identifying each individual student’s strengths, needs, national sample on some more-difficult- goals, and progress; uses those to provide appropriate to-implement aspects of PL. and meaningful individualized instructional experiences ■ Barriers to PL implementation included with the necessary adult supports; removes constraints on what students work on, when, and for how long; and poor integration of data systems, reallocates resources to best support these processes. tensions between competency-based practices and meeting grade-level In this theoretical conception, schools that are high standards, and the time needed to implementers of PL approaches would look very different develop personalized lessons. from more-traditional schools. In practice, although there were some differences between the NGLC schools and ■ Students in NGLC schools experienced the national sample, we found that schools in our study positive achievement effects in were implementing PL approaches to a varying degree, mathematics and reading, although the with none of the schools looking as radically different from traditional schools as theory might predict. This is effects were only statistically significant due in part to the schools trying various combinations in mathematics. of strategies and features, rather than all of them; to ■ On average, students overcame gaps the newness of the schools in our study (most of which relative to national norms after two had been in existence for less than three years); and to external constraints, such as state or district policies. years in NGLC schools. Despite the lack of clear differences in the practices that ■ Students at all levels of achievement teachers reported implementing in NGLC schools and relative to grade-level norms appeared the national sample, it is important to note that all of the NGLC schools had adopted structures and systems to to benefit. support PL within three years, and all of them reported ■ Results varied widely across schools striving to emphasize personalization in their school and appeared strongest in the middle designs and operations. grades. At the same time, many of the core practices of PL ■ Implementation and effects of PL were also implemented to some extent in the national sample, which consisted of schools that do not clearly seemed higher in charter schools than in identify themselves as PL schools. These factors make it district schools in the sample. difficult to draw a clear line separating PL from non-PL 40 Informing Progress Insights on Personalized Learning Implementation and Effects schools. And, although early evidence suggests that these responses difficult to interpret. Finally, we do not have PL approaches may be quite promising for improving any information about the extent to which schools in the achievement for a broad range of students, at this early achievement comparison group were implementing PL stage in the development of the innovation it is not clear strategies. what PL practices or combination of practices have the greatest impact on students. In light of these limitations, this apparent relationship between the extent of PL implementation and student There is suggestive evidence that greater outcomes should be interpreted cautiously. The question implementation of PL practices may be related to of why district schools in the NGLC sample did not seem more-positive effects on achievement; however, to be implementing PL approaches to the same extent as this finding requires confirmation through further charter schools, however, is an important one that merits research. First, the NGLC sample shows both positive further investigation. treatment effects and higher levels of implementation than schools in the national sample. Second, within The positive student outcome effects found in this the NGLC sample, there is a trend toward smaller study may not occur quickly or in all contexts. As estimated effects for district schools as compared with policymakers, practitioners, and funders think about charters, and this is accompanied by lower levels of how they could use the results of this research to implementation of PL practices among district schools. enhance, expand, and support implementation of PL, it Specifically, the reason students in the NGLC schools is important to keep in mind that the positive student are outperforming students in the national comparison outcome effects found in this study may not occur quickly group may be because NGLC schools are implementing or in all contexts. The earlier report from this study (Pane more PL strategies or implementing the strategies to a et al., 2015), which found statistically significant positive greater extent than schools nationally; and this may also effects for PL schools in mathematics and reading, explain why charter-school students tend to outperform focused on a sample of 62 schools, many of which were district students within the NGLC sample. Third, we see experienced implementers of PL and part of large, well increasing effects on student achievement with longer established charter networks. This report focuses on a exposure to PL, as shown in Figure 17. smaller sample of schools that were newer (most had been open for less than three years) and, although mostly Although consistent with the data, this hypothesis charters, were generally starting new networks or were is somewhat speculative and should be interpreted part of smaller networks. While the effects reported here cautiously for several reasons. First, the number of district are generally positive, they are smaller, overall, than schools in the implementation sample was small—nine those in the earlier report. Furthermore, the effects are out of 40—and thus the differences we observe may not smaller for district schools than for charter schools in be broadly generalizable to schools implementing PL. both reports. Taken together with the implementation Second, we have limited information about schools in data presented here, it is possible that implementing the national comparison sample, such that differences PL in new schools and in district schools may be more in composition (such as the proportion of charters) challenging than in other contexts. Therefore, as PL could influence the NGLC–national comparisons. We strategies become more widely used and studied, it is also cannot rule out the possibility that students in the possible that not all schools will see gains as large as NGLC sample—and in charter schools in particular—are those in the current sample or the sample examined in experiencing greater achievement gains for reasons that Pane et al. (2015). are not related to the implementation of PL strategies. It is also possible that PL may be more challenging Policy Recommendations to implement in district schools. We do not have any In this study, we found that schools were implementing evidence that could confirm or elucidate this hypothesis, PL approaches to varying degrees, although all were and it is possible that district schools could see the effects attempting to implement PL strategies to some degree. of PL if their practices were more consistent with those This could be because schools chose to implement some reported in charters. In addition, our observations about PL strategies but not others, or because the schools in this the extent of implementation of PL practices rely largely sample, which are predominantly new schools, planned on surveys of teachers and students, where responses are to implement more PL strategies as their schools grew to self-reported. Not only do we have no objective way to full capacity. It could also be because there are barriers at confirm their perceptions, but the problem of reference the local or state levels that cause variation in the ability bias (West et al., 2016) could make comparisons among to fully implement PL strategies. Informing Progress Insights on Personalized Learning Implementation and Effects 41  Yet the results of this study suggest that PL has positive be completed in their entirety, in a specific order, or at effects, and there is a lot of momentum in the field a certain grade level can inhibit full implementation to spread it. Given the results of this study, and those of such strategies. Taking a more flexible approach described in Pane et al. (2015), we offer the following could also include revising policies to allow inclusion recommendations for policymakers, implementers, of multidisciplinary courses or projects, or revising and funders of PL that could help to support broader or eliminating seat-time policies. Such policies, while implementation of PL and enhance our understanding of remaining flexible, should ensure that all students are how to implement it effectively. exposed to rigorous, high-level content and should be monitored to ensure equitable outcomes. For State and District Policymakers Allow school staff to have some autonomy to Incorporate flexibility into policies related to design school schedules that support PL. Flexibility course progressions. Some PL strategies involve to design a schedule that supports the school model and allowing students to complete specific standards or vision of PL, and modify it as needed over the course sections of a course as one way of helping them catch of the year, can be a key component of successful PL up to grade level, and others rely on allowing students implementation. While some safeguards should be to progress to new content as they demonstrate in place to ensure adequate coverage of content in competency. NGLC teachers and school leaders described compliance with state standards, this flexibility might using such practices but also requiring students to get entail allowing schools to implement a longer school through a certain amount of material. In addition, NGLC day or year, customize the length or number of class principals reported that they were not yet able to award periods, or develop multidisciplinary classes or project- credit for mastery of specific standards or sections of a based classes. District policies that require uniform school course in a way that would prevent students from having schedules could inhibit this flexibility. District leaders and to repeat that material if they transferred to other other stakeholders might seek ways to enable flexibility schools. State or district policies that mandate courses in schools that are willing to innovate. 42 Informing Progress Insights on Personalized Learning Implementation and Effects Enable schools to hire staffs that are the best fit a traditional letter grade for reporting purposes. At for the school. As we described in Pane et al. (2015), the same time, nontraditional grading systems can be NGLC administrators reported that a key challenge was challenging to understand, so policymakers could work hiring and retaining teaching staff with both the right to ensure that the school has the necessary resources to level of skills and experience and who were a good clearly communicate their grading system to internal and fit for the school model. Although it is not yet clear external audiences. which qualifications are most important to consider when staffing PL schools, having the flexibility to hire Look to early adopters of PL for examples of large- staff that support the school model and fostering scale policy change. Policymakers at the state level working conditions that support retention both seem interested in exploring some of these recommendations important. Policymakers could consider revising teacher could look to states such as Vermont, New Hampshire, placement or hiring provisions, revising policies to and Kentucky, all of which are working to revise state support hiring staff in nontraditional roles, and allowing policies to support PL at the local level and implement schools flexibility in some work rules, within legal and PL strategies statewide. District-level policymakers could contractual limits, to enhance teacher retention. look to districts such as Fulton County, GA, Piedmont, AL, and Horry County, SC, which are working to implement Ensure that accountability policies value growth PL strategies districtwide. Policymakers at all levels could and other metrics of student success. As we reported also look to charter management organizations such as in Pane et al. (2015), several of the NGLC schools reported Rocketship and Summit, which aim to incorporate PL policy challenges related to implementing competency- approaches in their schools. based approaches. One example was state tests focused on grade-level standards, which might not be sensitive to For Implementers at the District learning content above or below grade level, and which and School Levels could have accountability consequences for teachers Provide teachers with the resources and time to or schools. Policymakers should consider refining their pilot new instructional approaches and gather systems to use assessments that can accurately measure evidence of how well they work. As we argue growth in learning and to use growth-related metrics elsewhere in this report, it is not yet clear which PL for accountability. States could take advantage of the strategies and practices are most likely to positively affect flexibility offered by the accountability provisions of the student outcomes. Therefore, it is important to ensure Every Student Succeeds Act (ESSA), which allow for the that teachers and school leaders have the flexibility, inclusion of statewide academic indicators that measure time, and resources (e.g., funding, support staff, access growth rather than relying exclusively on proficiency. to experts) to experiment with new instructional ESSA also permits states to adopt an additional indicator approaches, develop a systematic process for collecting of “school quality or success,” which could include and analyzing evidence of their effectiveness, and make measures that might capture useful information about changes as needed. schools adopting PL, such as access to advanced course content or postsecondary readiness. These indicators Provide teachers with time and resources to could provide a broader and more-balanced set of collaborate on developing curriculum and on information with which to gauge the performance of reviewing and scoring student work. If the school schools implementing PL models. staff prefer to develop their own curriculum materials, it is important to ensure that teachers have the flexibility, Revise grading policies to incorporate competency- time, and resources to collaborate on curriculum based approaches, and clearly communicate these development and score student work in ways that approaches to students, families, employers, and are minimally intrusive on their teaching duties. Time postsecondary education institutions. Competency- to collaborate on scoring student work is particularly based learning strategies often require nontraditional important in schools that use mastery-based grading approaches to grading student work or judging student systems, where the system’s norms and parameters may readiness to progress through content and may need to still be in development. include ways to assess learning using multidisciplinary coursework or projects. Innovative schools should have Identify a school staff member (or two) who is the flexibility to develop nontraditional grading systems comfortable with technology and has curriculum that support the school model, and policymakers could expertise to serve as a just-in-time resource for consider limiting the need to convert grades back to teachers. Some technology resources have the potential Informing Progress Insights on Personalized Learning Implementation and Effects 43  to enable key PL strategies, but integrating technology resources or support could help ease the burden of data into instruction can often be challenging for teachers. It entry and integration for teachers, allowing them to is therefore important that schools identify one or two focus more time on instruction. staff members who have the ability to support teachers in troubleshooting technology issues as they arise, creating For Funders technology-integrated lessons and projects, accessing Direct funding to technology developers who will and interpreting data from technology-based curriculum work with teachers and curriculum experts to materials, and developing classroom management plans design technology-based curriculum materials and to include technology. data systems that will support PL practices. For Provide resources and support for school staff to example, such efforts could include curriculum programs help them choose the most-appropriate digital or that incorporate multiple paths through content and nondigital curriculum materials. Many NGLC schools include high-quality assessments of competency. Ideally, reported that finding standalone technology-based such materials and systems would be adaptable to curriculum programs of high quality that were well suited students at a variety of learning levels and integrated for the school context was challenging. As a result, many with student information systems to provide a complete schools tended to rely on multiple technology-based picture of each student’s goals and progress. programs and teacher-developed materials, a situation Allocate funding for research that includes stronger that can make developing lesson plans time-consuming experimental designs and that systematically tests for teachers. In addition, the lack of curricula designed to specific PL strategies. As funders continue to invest in meet the needs of students performing at different levels PL, and administrators continue to adopt the strategy in can hinder teachers’ efforts to personalize instruction. states, districts, or other groups of schools, intentional Ensuring that school staffs have the necessary resources program design can enable more-rigorous evaluation (e.g., time, funding, extra staff) and support (e.g., methods than were available for the current study. In access to curriculum experts or other means of vetting, particular, implementing a well-defined PL model in a adapting, or combining materials) could help ease the sample of schools, with half of the sample randomly burden of curriculum development for teachers, allowing assigned to begin immediately, and the other half serving them to focus more time on instruction. as a control group for a set period of time, can enable Provide resources and support for school staff rigorous causal estimates of PL effects. Such a design to integrate multiple data systems. Although can rule out concerns of selection bias—that factors technology is a key enabler of PL, another barrier to other than PL are responsible for the effects measured widespread, effective PL implementation is that some in PL schools. Moreover, a clearly defined model of technologies have not yet developed to the point where implementation for PL schools can help to clarify they support PL by making some aspects of teaching uncertainties about how, and to what extent, PL differs more efficient. For example, many school data systems in from more-traditional practice, and, if results are positive, use in PL schools do not yet integrate achievement and enable clear specification of a model for replication and nonachievement data, shifting the burden of integrating scale-up. and interpreting those data onto teachers. Providing Acknowledgments The authors are grateful to the PL students, teachers, and administrators who voluntarily participated in project data collection; NGLC staff who facilitated connections with the schools; and NWEA staff who responded to our multiple requests for assessment data and helped us interpret them. We are also grateful to the following RAND staff who contributed to the research: Mollie Rudnick, Courtney Kase, Amanda Edelman, Andrea Bingham, Katharina Best, Christopher Paul, Evan Peet, Kyle Siler-Evans, Gerald Hunter, Melanie Rote, Suzette Gambone, and Stephanie Lonsinger. This document is vastly improved as a result of feedback we received on earlier drafts from Chandra Garber, Paul Hill, Cathy Stasz, Julia Kaufman, Patrick McEwan, and Brad Bernatek and his colleagues at the Bill & Melinda Gates Foundation. Any flaws that remain are solely our own. 44 Informing Progress Insights on Personalized Learning Implementation and Effects References Bloom, Benjamin S., “The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring,” Educational Researcher, Vol. 13, No. 6, 1984, pp. 4–16. Brodersen, R. Marc, and Daniel Melluso, Summary of Research on Online and Blended Learning Programs that Offer Differentiated Learning Options, Washington, D.C.: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Central, 2017. As of June 1, 2017: https://ies.ed.gov/ncee/edlabs/projects/project.asp?projectID=4499 Iacus, Stefano M., Gary King, and Giuseppe Porro, “Causal Inference Without Balance Checking: Coarsened Exact Matching,” Political Analysis, Vol. 20, No. 1, 2012, pp. 1–24. Opfer, V. Darleen, Julia H. Kaufman, and Lindsey E. Thompson, Implementation of K–12 State Standards for Mathematics and English Language Arts and Literacy, Santa Monica, Calif.: RAND Corporation, RR-1529-1-HCT, 2016. As of June 1, 2017: http://www.rand.org/pubs/research_reports/RR1529-1.html Pane, John F., Elizabeth D. Steiner, Matthew D. Baird, and Laura S. Hamilton, Continued Progress: Promising Evidence on Personalized Learning, Santa Monica, Calif.: RAND Corporation, RR-1365-BMGF, 2015. As of June 1, 2017: http://www.rand.org/pubs/research_reports/RR1365.html Pustejovsky, James E., and Elizabeth Tipton, “Small Sample Methods for Cluster-Robust Variance Estimation and Hypothesis Testing in Fixed Effects Models,” Journal of Business & Economic Statistics, 2016. Thum, Yeow Meng, and Carl H. Hauser, NWEA 2015 MAP Norms for Student and School Achievement Status and Growth, Portland, Oreg.: Northwest Evaluation Association, August 16, 2015. As of June 1, 2017: http://www.sowashco.org/files/department/rea/2015NormsReport_Reading.pdf VanLehn, Kurt, “The Relative Effectiveness of Human Tutoring, Intelligent Tutoring Systems, and Other Tutoring Systems,” Educational Psychologist, Vol. 46, No. 4, 2011, pp. 197–221. West, Martin R., Matthew A. Kraft, Amy S. Finn, Rebecca E. Martin, Angela L. Duckworth, Christopher F. O. Gabrieli, and John D. E. Gabrieli, “Promise and Paradox: Measuring Students’ Non-Cognitive Skills and the Impact of Schooling,” Educational Evaluation and Policy Analysis, Vol. 38, No. 1, 2016, pp. 148–170. Informing Progress Insights on Personalized Learning Implementation and Effects 45  APPENDIX A. Implementation Analysis Methods and Limitations To explore what PL looks like, we drew on interviews with teachers and administrators, focus groups with students, and classroom observations conducted during in-person site visits to nine NGLC schools to present brief vignettes that highlight exemplar implementation of the four key PL strategies. To explore how PL practice in NGLC schools differs from In some cases, we describe differences that were not practice nationally, we compared teacher and student statistically significant but that were large in magnitude survey responses from the NGLC schools with those from and qualitatively meaningful in that they shed some light surveys with nearly identical questions administered to on substantive questions about implementation. Student national samples of teachers and students.1 To facilitate and teacher survey results can be found in the online this comparison, we first weighted the national survey addendum to this report. results to more closely reflect the NGLC sample in terms of geographic locale (e.g., urban), grade level, subject When interpreting the implementation data, it is taught (by teachers), and gender (of students). We important to keep in mind the limitations of the data lacked the necessary data to include family income in sources, which rely on the self-reports of stakeholders the student survey weighting process, and the national who voluntarily participated. We had no independent sample appears to be somewhat more affluent than means of verifying the accuracy of their responses. the NGLC sample. Moreover, the NGLC surveys were Where response rates were lower, particularly for the conducted in the spring, and the national surveys were teacher survey and logs in some schools, responses conducted in the summer; responses may have been may not accurately represent the perceptions of the affected by differences in how clearly respondents could whole stakeholder group, limiting generalizability. remember details about the practices and experiences we Survey responses likely vary across several factors, such inquired about. We also present additional evidence of PL as grade-level configuration (e.g., elementary versus implementation from the NGLC administrator interviews, secondary schools), but we avoided breaking down the site-visit interviews, and teacher logs. data by these features because of the small numbers of respondents in some categories. Additionally, the self- We observed a trend toward larger PL treatment effects reported nature of the surveys may limit their ability to in charter schools than in district schools. We therefore accurately measure differences across schools. As just one examined our implementation data separately for charter example, West and colleagues (2016) have documented and district schools, where feasible, to explore questions a phenomenon known as reference bias, where such as whether levels of implementation differed in responses can be influenced by the respondent’s frame of these two contexts or whether factors that hindered or reference or social context. To illustrate, a student might facilitated implementation differed. answer a question about being “given opportunities to demonstrate my strengths and weaknesses,” with Throughout this report, we used a holistic approach the response option “mostly true.” The actual amount when deciding what information to present. We focused of such opportunities necessary to meet the threshold on presenting meaningful evidence of differences (or of “mostly true” can vary from student to student, similarities) between implementation of PL and more- influenced by their own experiences as well as the norms traditional practice. Where we were able to perform of the school or the attitudes of their peers. Thus, two tests of statistical significance we used those results to respondents who responded “mostly true” might actually guide our decisions about what material to present. be experiencing different levels of these opportunities. This can reduce the validity of comparisons of responses 1 The national survey was administered by Grunwald Associates. The items were identical to those administered in the NGLC between groups (such as students in NGLC schools versus schools except for tense; the NGLC items were in present tense a national sample, or teacher versus student reports of a because the survey was administered in the spring, whereas the PL implementation feature). national survey items were in past tense, referring to the pre- vious school year, to reflect the fact that students and teachers completed it during the summer. 46 Informing Progress Insights on Personalized Learning Implementation and Effects Although we weighted the national student and Site visit schools were selected based on fall 2014 teacher surveys to make the respondent profiles more administrator interviews and documentation. We similar to the PL samples, data limitations prevented us purposefully selected schools that varied on several from doing so with respect to family income, limiting dimensions: the extent to which the school was the comparability of the student survey samples. We implementing competency-based progression, extent to opted not to include years of teaching experience when which the school was implementing technology-based weighting the teacher survey to allow for the possibility PL, grade configuration, and organizational structure that hiring less-experienced teachers was something some (e.g., a school that was part of a charter management NGLC schools did intentionally (in other words, reliance organization versus one administered by a traditional on newer teachers might be considered part of the NGLC district). Teachers were randomly selected for the approach to operating schools, rather than an extraneous interviews and focus groups so that there was some factor that we would want to control for). We observed variation across grade level taught, subject taught, and differences in the mean years of teaching experience years of teaching experience. Students were selected for in each sample (3.9 years of experience for teachers in the focus group by a school administrator so that the the national sample; 2.7 years for teachers in the NGLC group would include students with a mix of ages and sample) and this difference could affect responses in learning levels, as well as students from both genders. ways that are not related to the implementation of PL practices. We also have no information about the extent to which the schools in the national sample were Administrator Interviews implementing PL strategies, and therefore comparisons We interviewed an administrator by telephone at each between the national sample and the PL sample should school, district, or charter management organization be interpreted with these limitations in mind. in the fall of the 2014–15 school year. We conducted a second set of telephone interviews in the spring The small number of district schools in the with an administrator at the school level, usually the implementation sample (nine) limits our ability to make principal or assistant principal. At site-visit schools, reliable comparisons between the district and charter the spring administrator interviews were conducted in schools in the NGLC sample. Moreover, the teacher person. The interviews helped gather other information workforce appears to differ between district and charter about instructional practices, including what types of schools, which could affect responses through reasons technology the school was implementing, whether the other than the implementation of PL practices. Overall, school used standards-based grading, and whether there charter teachers were less experienced than district were opportunities for learning outside of school. The teachers: 22 percent of charter teachers reported having interviews lasted one hour. one year of experience or less, compared with 5 percent of district teachers. Charter teachers were also more likely to have received their certification through a Teacher Logs nontraditional program (27 percent, compared with Teachers of mathematics and ELA were asked to 14 percent of district teachers). For those reasons, complete logs, which were brief, online surveys that district–charter comparison results should be interpreted included questions about daily instructional practice with caution. and the factors that influenced their teaching on a particular day. We administered the logs over two 10- day periods in 2014–15, once in the fall and once in the Site Visits spring, for a total of 20 logs per teacher. In the fall, the We conducted one-day site visits at nine schools in spring logs were distributed to a sample of 331 teachers, and 2015. The visits included a one-hour interview with the 228 teachers completed at least one log in which they principal, 45-minute individual interviews with three indicated that they had provided instruction that day, for instructional staff, a one-hour focus group with six to a response rate of 69 percent. In the spring, the logs were eight instructional staff, a one-hour focus group with six distributed to a sample of 330 teachers, and 189 teachers to eight students, and 10- to 15-minute observations of at completed at least one log in which they indicated that least two classrooms, one mathematics and one English they provided instruction that day, for a response rate of language arts (ELA). The purpose of the site visits was to 57 percent. gather in-depth information about implementation of the school model and instructional practices and to solicit student perspectives. Informing Progress Insights on Personalized Learning Implementation and Effects 47  The number of logs completed varied by teacher; missing spring results that cover a broader range of topics. We logs were due either to a response of “I did not provide distributed the fall survey to 9,294 students and the instruction today” or to noncompletion. Each day, spring survey to 9,058 students. Response rates were teachers answered a series of questions while focusing 71 percent and 69 percent, respectively. on their interactions with one student during the first 45 minutes of mathematics or ELA instruction. Teachers As with the teacher surveys, we developed many of were asked to focus on a different student for each day the items specifically for this study, but the surveys also that they completed the log. The rationale for asking included original or modified versions of items from teachers to focus on a single student rather than the the CCSR surveys; the High School Survey of Student entire class is that the instruction offered, and the nature Engagement, developed by the Center for Evaluation of the student-teacher interactions, can vary across and Education Policy at Indiana University; and the students. This variability is particularly likely to occur in PL Tripod survey, developed by Harvard University’s Ronald environments. Ferguson, to measure student opinions of teacher quality. Teacher Surveys National Surveys Teachers of mathematics and ELA were also asked to To provide comparative data for our teacher and student provide their perceptions about various aspects of the surveys, the Bill & Melinda Gates Foundation engaged models, including professional training and support, Grunwald Associates to administer the surveys to a access to resources, the quality of instructional and national sample. Those surveys were administered during curricular materials, use of different models of classroom the summer after the 2014–15 school year. The questions instruction, use of technology in the classroom, use on the survey were nearly identical to those on our of data to assess student progress, and obstacles to surveys, although the language was adapted to refer in implementation. The survey was distributed to a sample the past tense to the 2014–15 school year. of 330 teachers and the response rate was 74 percent. The teacher surveys were administered online in spring Analysis of Interview and Focus-Group Data 2015. Although most of the survey items were developed specifically for this study, a few were adapted from The analysis of the interview and focus-group data other RAND surveys or from surveys developed by the proceeded in several steps. First, interview notes were University of Chicago Consortium on Chicago School compared to the audio recording and cleaned to serve Research (CCSR). as a near-transcript of the conversation. The cleaned interview notes were then loaded into the qualitative analysis software package NVivo 10 and auto-coded by Student Surveys interview question (so that responses to specific interview Students were asked to describe their study habits, questions were easily accessible) as well as coded using a attitudes toward learning, perceptions about their thematic codebook developed by the evaluation team. school, level of access to technology, and other topics. Once the thematic coding was complete, we conducted The student surveys were administered online in the fall a second round of coding, analyzing the data according and spring of the 2014–15 school year to students in 36 to questions of interest (e.g., to what extent are schools schools with enrolled students who met the age criteria: implementing competency-based progression?). In this grades 6 and above or age 11 and older if the school did stage, we used an inductive coding process (i.e., codes not use traditional grade levels. The fall survey focused were derived from the data rather than a structured on study habits and attitudes toward learning; the codebook) to develop responses to the questions of spring survey supplemented these with the remaining interest. topics. Student responses to items that appeared on both surveys were similar, so this report focuses on the 48 Informing Progress Insights on Personalized Learning Implementation and Effects APPENDIX B. Achievement Analysis Methods and Limitations This study is designed to use the most rigorous method that can be applied to the situation. In particular, given the portfolio of NGLC schools, it was not possible to create randomly assigned treatment and control groups; nor did we have access to data from neighboring schools that might have matched the NGLC schools. Moreover, as new schools, they lacked a history of data from before they began implementing PL, which would have enabled other analytic methods for determining achievement effects. With these limitations, we determined that a matched comparison group design is the best available quasi-experimental method for estimating the effect of NGLC schools on student outcomes. If the NGLC students can be matched to comparison students who are equivalent at baseline, this method can produce unbiased estimates of the NGLC effect. To create the matched comparison group, NWEA drew magnitude of the effect. Because of these limitations, on its large national database of testing data to identify achievement results should be interpreted with some VCGs—comparison groups of students who had starting caution. performance similar to the PL students and who were attending schools serving similar populations. Details While the basic empirical strategy remains the same about the matching method and the statistical models we between this report and Pane et al. (2015), there were used to estimate results are described below. This process some refinements that arose from obtaining an updated enables us to make “apples-to-apples” comparisons of and richer data set from NWEA. The first change related learning growth between the students in the PL schools to the matching algorithm performed by NWEA, and and a similar population of students attending other other changes were to the analytic methods described schools. below. Limitations of Achievement Analysis Matching Method for Virtual Comparison Group However, there are limitations to this method. Previously, when searching for matches to create VCGs Although, as detailed below, we find that the observable for students in one PL school, NWEA would require characteristics of the comparison students are well all matches to come from schools outside the same matched to those of NGLC students in the study, the governing organization (generally speaking, a school comparison students could possess other unidentified or district or charter management organization). This unobserved differences from the NGLC students. Those restriction prevented contamination of the control differences could confound efforts to measure the impact group with other PL students in the same governing of PL. For example, parents of NGLC students might have organization, but conceptually could enable PL greater interest in nontraditional schooling environments students from one governing organization in the study and this could be related to how well their children to be included in the VCG for a different governing do, independently of the NGLC schools’ PL treatment. organization. At our request, NWEA updated their Differences like this are a type of selection bias that could VCG-matching algorithms to exclude students from affect estimates of treatment effects, in either a positive any governing organization in the study. As such, the or a negative direction. The VCG approach also assumes updated matching algorithm was as follows: For each that the students in the comparison group are attending NGLC student, NWEA created a VCG of up to 51 students more-traditional schools that are not using PL practices, from its database. Separate comparison groups were but there is no way to verify this assumption. If this created for the mathematics and reading tests and for assumption is not true—if any of the comparison schools each time span examined. The analysis uses fall scores were indeed using PL practices—estimates comparing as pretests and spring scores as posttests (from the NGLC students to VCG students could underestimate the same academic year for one-year analyses, and from Informing Progress Insights on Personalized Learning Implementation and Effects 49  the following academic year for two-year analyses). The The basic intuition of the CEM approach is that treated following student and school-matching criteria were students are matched with control students based on applied to create the VCG.1 similarities in observables across several dimensions together, instead of collapsing the matching space into REQUIREMENTS FOR ALL VCG MATCHES a univariate distance metric, as is done with propensity ■■ Students have valid scores for the pretest and the score matching. This method is robust even if a control posttest. student is used as a match for multiple treatment students: only the closeness of the match is relevant. The ■■ Students are not in any of the governance process creates weights that reflect how often control organizations containing schools in the PL sample. students are repeated and the size of each treated ■■ Schools have the same locale classification (e.g., student’s comparison group. urban, suburban, rural, etc., according to the National Center for Educational Statistics Public Specifically, treated students all receive a weight of 1, School Universe Survey). while control students are given a weight equal to the ■■ Students are the same gender and in the same grade sum of the inverse of the size of their VCG group for as the treatment-group students to whom they are each time they are in a treated student’s VCG. Equation 1 matched. shows the definition of these weights. APPROXIMATE MATCHING CRITERIA ⎧ ⎪ ■■ Schools differ by no more than 15 percentage points ⎪⎪ 1 if Ti = 1 on the portion of students participating in the FRL wi = ⎨ 1 program. ∑ i ∈ VCG j } VCG j ⎪ j if Ti = 0 ■■ Students scored similarly on the pretest MAP ⎪ { assessment. Preference is given for students with ⎪⎩ the same pretest score, but this can be expanded to where i indexes students, j indexes each VCG group within five points on NWEA’s RIT scale if necessary student i appears in, and |VCGj| is the number of VCG to find matches.2 students in that group; Ti is a treatment indicator equal ■■ Number of days elapsed between the pretest and to one for PL students and zero for VCG students; and posttest testing differs by no more than 18 days. wi is the weight for student i. For example, consider a control student who appears in two treated students’ Refinements to the Statistical VCG groups. The first VCG group she appears in has 50 Estimation Strategy control students, and the second VCG group she is in has 48 control students. The weight for this control student would be 1 + 1 ≈ 0.0408 . NWEA also provided unique identifiers for each VCG student so that we could observe cases where the same 50 48 VCG student was selected to match more than one PL After calculating these weights, the data set is reduced student and we could account for this duplication in our to having one observation per student test score, instead analysis. To do so, we now use a type of Coarsened Exact of retaining multiple records, as was done with the Matching (CEM) estimator (Iacus et al., 2012). CEM allows within-estimator used in Pane et al. (2015). The weights us to analyze a data set with one record per student are then applied in a weighted linear regression, as test event, instead of multiple records for VCG students described below. The CEM estimator used here departs matched to more than one treated student. It also more slightly from that of Iacus et al. (2012), in that matching closely reflects and capitalizes on the matching algorithm cells are created around each treated student instead of enacted by NWEA. across all of the data points, and thus may overlap across treated students; however, the general intuition of the approaches is the same. 1 NWEA first identified all student records that met these crite- ria, and, if there were more than 51, took a random sample of The dependent variable in the weighted regression is 51 of those records. the gain from pretest to posttest in the MAP assessment- 2 NWEA’s RIT (Rasch Unit) scale is a stable equal-interval vertical scale score. We standardized test scores using mean scale designed to allow items of different difficulty levels to be placed on a common scale. A student’s RIT score indicates the and standard deviations of the pretest scores by grade, level of question difficulty a given student is capable of answer- so that the pretest scores have a mean of zero and a ing correctly about 50 percent of the time. 50 Informing Progress Insights on Personalized Learning Implementation and Effects standard deviation of one within each grade level, and district level and we use both the treatment and VCG posttest scores reflect the standardized growth. Because clusters instead of clustering on treated schools, as was of small samples and volatility of scores in the highest done previously. grades, we classified grades 11 and 12 into a single “late high school” group for the grade-level indicators. We Finally, we updated our analysis to make use of more up- then divided the standardized growth by the number to-date norms published in Thum and Hauser (2015). of days elapsed between pretest and posttest, to account for variation in the time elapsed, to obtain a Number of Schools and Students in standardized measure of growth in achievement per day. Achievement Analysis Samples We regressed the standardized growth in achievement per day on treatment status and the following covariates: Table B.1 displays the number of schools and students an indicator of whether the school is district-operated, entering into the overall analysis of mathematics and the school-level percentage of students eligible for FRL, reading for the 32 NGLC schools in the 2014–15 analysis, and student-level indicators of grade level and gender. and the 16 NGLC schools in the 2013–15 analysis. Students We then scaled the treatment effect back up to a year had to remain in one of the NGLC schools in our sample by multiplying the coefficient on treatment by the to be included in the analysis. The table indicates average number of elapsed days for the sample (across the students’ grade level at the start of the relevant both treatment and VCG). None of the exactly matched time span. covariates are included in the regression, but are Table B.2 displays the number of schools and students implicitly controlled for. entering into the comparison of charter-operated and In a second change to the analysis, we now use a district-operated NGLC schools for the 2014–15 academic clustering algorithm and degrees of freedom estimators year. In this analysis there were 24 charter schools that are more robust when there are small numbers of covering all grade spans, and eight district schools clusters (Pustejovsky and Tipton, 2016). We cluster at the covering the middle and high-school grades. Table B.1. Number of schools and students in aggregate analyses     Number Number of Students by Grade at Start of Time Span Group of   Schools K 1 2 3 4 5 6 7 8 9 10 11 12 NGLC 32 253 251 159 160 89 259 1,133 955 529 1,149 454 82 1 Reading VCG 5,040 5,807 6,724 5,341 5,797 3,062 4,918 32,082 22,634 15,415 23,402 12,503 1,643 51 2014–15 NGLC 32 250 255 153 158 84 258 1,235 928 528 1,159 410 77 44 Math VCG 4,837 5,122 6,995 5,264 5,682 2,876 4,467 31,187 23,324 15,058 24,515 11,754 1,893 1,496 NGLC 16 78 52 56 76 55 100 554 309 69 394 65 1 Reading VCG 2,723 1,443 1,919 1,986 2,913 2,219 3,488 11,982 7,004 1,055 9,753 1,040 51 2013–15 NGLC 16 91 43 52 70 52 102 555 304 111 395 66 36 Math VCG 2,745 2,304 1,843 1,634 2,680 1,802 3,217 11,268 6,859 1,376 9,822 1,593 1,172 Table B.2. Number of NGLC schools and students in the 2014–15 charter-district analysis Number Number of Students by Grade Group of Schools K 1 2 3 4 5 6 7 8 9 10 11 12 Charter 24 253 251 159 160 89 259 886 612 232 621 85 75 Reading District 8 247 343 297 528 369 7 1 Charter 24 250 255 153 158 84 258 873 603 235 565 86 68 38 Math District 8 362 325 293 594 324 9 6 Informing Progress Insights on Personalized Learning Implementation and Effects 51  Assessment of Balance Between the analyses. The extent to which these alternative analyses Treatment Group and the VCG produced similar or different estimates than our main analyses could help validate the treatment estimates or The VCG is intended to be very similar to the study group place likely bounds on true treatment effects. in terms of students’ observable characteristics prior to treatment. This is true by construction for the criteria that were matched exactly (namely, the grade level of Analyses Based on Conditional Growth Norms the student and the urbanicity of their school). For the First, we used an alternative method for estimating approximate matching criteria, we examined whether treatment effects using conditional expected growth the groups appear to be the same. Table B.3 shows estimates based on norms (CGN) calculated by NWEA. balance on variables that were approximately matched. CGN uses students’ starting scores and elapsed time We present both the unweighted VCG means (after to predict a typical posttest score based on normative restricting the sample to retain only one observation per data from a national sample (for more on the CGN VCG student, per subject, per year) and the weighted methodology, see Thum and Hauser, 2015, p. 38). The VCG means, wherein we weight the means using the CGN method does not consider other factors that are part CEM weights described above. We also present the of the VCG matching, such as student gender, schoolwide standardized difference, calculated by dividing the measures of poverty (e.g., FRL), and geographic locale. difference by the standard deviation of the variable For each relevant subgroup (school, grade span, or for the pooled sample (treatment and VCG). We find overall), we estimated the average difference between standardized differences that are always smaller than the treated students’ realized growth and their CGNs, 0.25 in the unweighted comparison, and particularly under the assumption that national norms generally close for baseline scores. Restricting the sample to one represent typical growth in schools that are not NGLC observation per VCG student caused some of this minor schools. imbalance relative to the data initially received from NWEA. The standardized differences are even smaller after weighting, reflecting the role that the weights play Restriction to the Same School Type in helping to restore balance. In a second sensitivity analysis, we set additional constraints for the VCG matching. Many of the NGLC Sensitivity Analyses schools are charter schools, which may tend to enroll a select group of students. As one example, families To help evaluate the robustness of the main findings make an affirmative decision to enroll their children. discussed above, we performed a variety of sensitivity Family involvement in education might influence student Table B.3. Balance between NGLC and VCG groups on variables not exactly matched NGLC Unweighted VCG Weighted VCG Subject Variable Mean Mean Difference Std. Diff. Mean Difference Std. Diff. Start RIT 204.77 202.93 1.84 0.07 204.71 0.07 0.00 Reading FRL 74.94 74.86 0.09 0.00 74.38 0.56 0.03 Elapsed days 243.56 238.03 5.54 0.20 241.40 2.17 0.08 2014–15 Start RIT 211.63 210.13 1.50 0.05 211.58 0.05 0.00 Math FRL 74.95 74.36 0.58 0.03 74.42 0.52 0.02 Elapsed days 242.40 237.63 4.77 0.17 240.11 2.30 0.08 Start RIT 203.81 201.38 2.43 0.09 203.77 0.04 0.00 Reading FRL 79.25 75.81 3.44 0.16 77.24 2.01 0.09 Elapsed days 591.56 593.02 −1.46 −0.04 589.31 2.25 0.07 2013–15 Start RIT 212.05 208.12 3.93 0.14 211.99 0.06 0.00 Math FRL 78.84 75.57 3.27 0.15 76.66 2.17 0.10 Elapsed days 592.09 594.48 −2.40 −0.07 589.89 2.20 0.06 Note: the unweighted VCG columns show sample characteristics after restricting to one observation per VCG student, per subject, per year. 52 Informing Progress Insights on Personalized Learning Implementation and Effects achievement in positive ways unrelated to the schools’ Figure B.1. Test durations for NGLC and VCG students influence on achievement. To the extent VCGs are drawn 70 from schools that are not charter schools, there is the time spent on test (minutes) potential that a difference in family involvement, or in 60 Average amount of other factors that might influence students to enroll in 50 schools of choice, could bias the results. We investigated this concern by attempting to make the treatment and 40 control groups more similar on such factors. We only 30 kept VCG students that had the same school type (district or school of choice) as their treated NGLC student. 20 We compared the treatment effect estimate using the 10 matched school-type VCG with that from the standard VCG matching criteria that ignore choice. The concern 0 Mathematics Reading about unmeasured differences between choice and nonchoice schools would gain credence if the schools- ■ Fall 2014 NGLC ■ Spring 2015 NGLC of-choice VCG produces meaningfully lower treatment ■ Fall 2014 VCG ■ Spring 2015 VCG effect estimates than the standard VCG analysis. assess the risk that anomalous test duration growth for Filtering and Alternative Time Span Analyses some students might influence the estimated treatment Finally, we discovered concerning patterns in test effects. First, we began by filtering out outlying test duration (the amount of time students spend taking the durations both among the treated students and the test) among students and schools in the study. Briefly, VCGs. We used the following filters: some student-test events had very long durations or large ■■ Filter 1: Drop if fall or spring test durations are changes in test duration between the fall pretests and below 5th percentile or above 95th percentile for spring posttests. This raised concerns that differences grade and subject (national duration, provided in in duration, or testing conditions that drive changes in personal communication by NWEA). duration, might influence estimates of the treatment effect of attending an NGLC school. ■■ Filter 2: Drop if the change in test duration from fall to spring exceeds the national 90th percentile of To that end, we performed a set of sensitivity analyses change in test duration for grade and subject. related to duration to gain a better understanding ■■ Filter 3: Drop if the durations meet the criteria of of how anomalies in test duration might be affecting both filters 1 and 2. treatment effect estimates. We applied filters to remove students with anomalous test durations or anomalous If an NGLC student met a filter’s criteria, all of the changes in test duration between pretest and posttest. VCG records for that student were also filtered out. We also applied filters at the school level based on However, if a VCG student was filtered we did not drop aggregate patterns of test durations of the participating the corresponding NGLC student, or other VCG records students. Finally, we examined the use of a different time that did not meet filter criteria. Table B.4 presents the span in spring-to-spring because this pretest–posttest percentages of NGLC and VCG student records that were pair tends to have less discrepancy in test duration. The filtered out. In every case, more VCG records are filtered difference in duration change is less dissimilar between out than NGLC records. Filter 3, by construction, filters NGLC and VCG students than was observed between out the smallest fraction of students. the PL and VCG students in Pane et al. (2015). Figure B.1 shows that NGLC students generally spent more time on Table B.4. Percentages of NGLC and VCG students the tests than their VCG counterparts. However, in both dropped by filters subjects, the fall-to-spring time increases were about the same for NGLC and VCG students, at 16 percent. Filter 1 Filter 2 Filter 3 Subject NGLC VCG NGLC VCG NGLC VCG Although the average durations presented in Figure B.1 Mathematics 30% 54% 21% 45% 14% 35% do not suggest concerns about duration, we applied the Reading 32% 56% 22% 47% 14% 35% same filtering methods as in Pane et al. (2015) to further Insights on Personalized Learning Implementation and Effects Informing Progress 53  Although some changes in test duration could reflect problematic. Moreover, if we believe that the fall or inappropriate test administration conditions, in some spring test durations are so short or so long as to result in cases these changes might be due to factors that could invalid scores, these alternative durations may also suffer legitimately be attributed to treatment effects, such as from the same problem. academic growth that results in more-difficult (and more time-consuming) items being administered in the spring, An additional problem is that, if most of the treatment or increases in students’ willingness to persist through effect happens in the first year of exposure to the school challenging test content. Where this is the case, it would or to NGLC, then this will be missed by not starting from be incorrect to filter such students out, and the treatment a baseline fall score.3 effect would be biased if part of the treatment were Although these alternate time spans use two-year data increasing student human capital in ways that would to create additional estimates of one-year effects, they appear to result in anomalous test duration or change differ in important ways from estimates made from in duration. To that end, we additionally evaluated one-year data. In addition to the differences already the overall treatment effect, where instead of filtering noted, the data have differences both in the treatment individual students out, we only filtered out anomalous students included (students need to have been present in schools. We used two methods to filter out schools: the NGLC schools for both years and tested at least three ■■ calculate average durations by subject and grade for times, as opposed to needing the students present just all students in the school and filter out the school if for the two tests in the same year for the one-year span) filter criteria are met as well as having a potentially different set of VCGs. For these reasons, we considered the comparison of the ■■ filter out a school if over 40 percent of students in different spans with each other, but did not directly that school meet filter criteria. compare them to the filtered treatment effect estimates. By using the 2013–15 span to get the needed data, we Using these filtered data sets, we applied the same are restricted to using the 16 schools available. statistical models used previously to estimate treatment effects overall and for each school, for each subject, and time span. Results of Sensitivity Analyses As an alternative to filtering, we can use multi-year CONDITIONAL GROWTH NORMS data to estimate treatment effects using time spans First, we estimated treatment effects using CGNs, shown other than fall to spring. An alternative to the fall-2014- in Figure B.2. To interpret these results, we focus on to-spring-2015 span is the spring-2014-to-spring-2015 the fact that the CGN analysis estimates similar positive span. The purpose for this is that the differences in test effects as the main analysis. We interpret this as helping duration are generally between fall and spring, with to validate the main results. Although the CGN analyses fall durations typically shorter than spring durations. vary slightly from the VCG method, we focus less on Therefore, using spring-to-spring time spans alleviates the magnitudes of the CGN estimates because the the issue. This also reflects a common span used in other VCG method is more rigorous in carefully developing a educational achievement analysis, where only spring matched comparison group, as opposed to benchmarking achievement tests are available. against national norms as is done in the CGN method. The VCG estimates in this chart differ from the main However, there are potential problems with this analysis because CGN estimates were not provided for alternative. First, the new span includes summer, and some students (e.g., 12th graders), and those students researchers have found evidence that students experience were dropped from both analyses for the sake of using a test-score declines over the summer. If summer declines consistent sample for this sensitivity test. are an outgrowth of differences in testing conditions and not related to actual learning, then including summer may result in a more accurate measure of learning during 3 Also, on a more technical note for our current data, when we the school year because the pretest and posttest are use spring pretests, the students are not matched to their VCGs administered under more-similar conditions. However, it on this pseudo-baseline. To account for this, we also evaluate a treatment effect where we drop all VCGs not within three points may be that some of this summer loss is true loss of the on the RIT scale (approximately 95 percent of VCGs are with- achievement that accrued the prior school year, which in plus-or-minus three points of the PL student’s score on the should be attributed to the schools and their practices, interim spring test, while an even higher proportion of VCGs are within plus-or-minus three for the true baselines on which they in which case, time spans that include summer are more were matched). 54 Informing Progress Insights on Personalized Learning Implementation and Effects Figure B.2. Comparison of VCG and CGN methods RESTRICTING THE COMPARISON TO SCHOOLS 2014–15 OF THE SAME GOVERNANCE ■ VCG ■ CGN Next, we examined treatment effects using a VCG 0.12 composed of students only from the same governance 0.1 structure (district or charter) as the corresponding NGLC school. Figure B.3 presents these results. The results 0.08 are virtually the same in both subjects. We conclude Effect size that these results help to affirm the treatment effects 0.06 estimated by the standard VCGs. 0.04 DURATION ANALYSIS 0.02 We applied a variety of student-level and school-level 0 filters to remove anomalous test durations from the Mathematics Reading analysis. Applying the filters at the student- and school levels yields a range of estimates. Figure B.4 focuses on Note: Statistical tests of significance were not performed for this sensitivity analysis. the main analytic sample and displays the unfiltered estimate and confidence interval in yellow, and the median filtered estimate and its confidence interval in red. The blue bars show the range of the filtered Figure B.3. Comparison of VCG and estimates (but not their confidence intervals). In both same-governance VCG analyses subjects, the median filtered estimate is smaller than the 2014–15 unfiltered estimate. For mathematics, the median filtered 0.16 estimate is positive and statistically significant, and none of the filtered estimates are negative. For reading, the 0.14 median filtered estimate is positive but significantly 0.12 indistinguishable from zero, with two of the nine filters 0.1 producing negative estimates. The decrease in the Effect size 0.08 treatment effect from unfiltered to the median filtered estimate is 23 percent for mathematics and 39 percent 0.06 for reading. 0.04 0.02 Figure B.4. Analyses with Test Duration Filters 0 −0.02 0.2 Mathematics Reading Note: Statistical tests of significance were not performed for 0.15 this sensitivity analysis. Effect size ■ Unrestricted VCG 0.1 ■ VCG restricted to same governance 0.05 0 −0.05 Mathematics Reading Note: Statistical tests of significance were not performed for this sensitivity analysis. ■ Range of filtered estimates ● Unfiltered estimate   ● Median filtered estimate Informing Progress Insights on Personalized Learning Implementation and Effects 55  Figure B.5. Alternative time span comparison Finally, we looked at alternative spans for the subset of NGLC schools that had been operating for at least two 0.3 years. Figure B.5 presents these results. Results from the 0.25 main fall-2014-to-spring-2015 analysis are shown in blue, with the alternative span of spring 2014 to spring 2015 0.2 shown in red. The spring-to-spring analysis produces 0.15 smaller treatment estimates, particularly in mathematics. 0.1 The fall-to-spring results differ from the estimates for the Effect size whole sample because only a subset of students had the 0.05 requisite set of scores to participate in this sensitivity test. 0 −0.05 −0.1 −0.15 −0.2 Mathematics Reading Note: Statistical tests of significance were not performed for this sensitivity analysis. ● Spring 2014–Spring 2015 ● Fall 2014–Spring 2015 56 Informing Progress Insights on Personalized Learning Implementation and Effects  CHILDREN AND FAMILIES The RAND Corporation is a nonprofit institution that helps improve policy and EDUCATION AND THE ARTS decisionmaking through research and analysis. ENERGY AND ENVIRONMENT HEALTH AND HEALTH CARE This electronic document was made available from www.rand.org as a public service INFRASTRUCTURE AND of the RAND Corporation. TRANSPORTATION INTERNATIONAL AFFAIRS LAW AND BUSINESS NATIONAL SECURITY POPULATION AND AGING PUBLIC SAFETY Support RAND SCIENCE AND TECHNOLOGY Browse Reports & Bookstore TERRORISM AND Make a charitable contribution HOMELAND SECURITY For More Information Visit RAND at www.rand.org Explore the RAND Corporation View document details Research Report This report is part of the RAND Corporation research report series. RAND reports present research find- ings and objective analysis that address the challenges facing the public and private sectors. All RAND reports undergo rigorous peer review to ensure high standards for research quality and objectivity. Limited Electronic Distribution Rights This document and trademark(s) contained herein are protected by law as indicated in a notice appearing later in this work. This electronic representation of RAND intellectual property is provided for non- commercial use only. Unauthorized posting of RAND electronic documents to a non-RAND website is prohibited. RAND electronic documents are protected under copyright law. Permission is required from RAND to reproduce, or reuse in another form, any of our research documents for commercial use. For information on reprint and linking permissions, please see RAND Permissions.