Data Driven

Editor's Notes

• #2: What is data-driven decision making? Data-driven decision making is the process of making choices based on appropriate analysis of relevant information. School district decision makers are using technology and professional expertise to improve instruction and operations. Why use data for decision making in K-12 education? Decisions in school districts have been made according to tradition, instinct, and regulations. More access to better information enables educational professionals to test their assumptions, identify needs, and measure outcomes. Schools are using data-driven decision making to provide more individualized instruction to students, track professional development resources, identify successful instructional strategies, better allocate scarce resources, and communicate better with parents and the community. How does a district decide what data to collect? Most districts are data rich. They have too much information in too many places to effectively use it. They have information about student records, student assessment, transportation services, food service, human resources, library automation, student health, special education, and curriculum management, to name a few. The challenge is to integrate these disparate systems and make the information available in timely, easy-to-understand reports so that decision makers can affect student performance. What is a data warehouse? A data warehouse is a storage facility integrating sources of vital information about every student and staff member in the school system. Providing easy access to this data is a crucial element of a data warehousing solution. At the same time, much of the information is highly confidential. Finding the right balance between access and security, flexibility and control, is an ongoing challenge for K-12 IT departments. What are some of the ways in which data reports can be structured? Reports need to be timely, tied to objectives, and available to people with the responsibility and ability to act on them. Data reports that show data in different ways such as tables, charts, graphs, and trends enable more people to access and understand the information. Most districts create a standard set of reports based on the key questions and indicators identified in the planning process. If possible reports should include longitudinal data so that teachers, principals, and administrators can compare results over time. What common data report formats are most useful to teachers? Web-based systems enable teachers to log-in and view a class or drill down to a student profile or flexible groupings of students. They can view assessment results tied to standards and assessment items. Teachers filter by period, course or NCLB filters such as ethnicity, gender, or second language learners. They have access to current and historical data as well as contact information for student, parents, and email links to other teachers. One district enables teachers to export contact information for mail merges. What common data report formats are most useful to principals? Principals use data on attendance, enrollment, student/teacher/parent satisfaction surveys, and test results to assess progress, allocate resources, and create school improvement plans. They look for information that is organized numerically rather than alphabetically, includes objective descriptions of data, visual displays of information, and query tools. What common data report formats are most useful to district personnel? District personnel use data to report results to federal and state agencies, most notably NCLB and state assessments. Data also helps district personnel determine the appropriate allocation of district resources, plan professional development, analyze district level interventions to achieve desired results, create school improvement plans, and assess the overall progress toward strategic goals. To use data more effectively in decision making, district personnel need access to data across information systems, for example: linking financial data with student assessments helps to refine resource allocations, connecting human resources with student assessment helps identify professional development needs, etc. The data must be available in both aggregate and disaggregate formats, allowing administrators to drill down by school, department, classroom, student demographics, etc. Why is data-driven decision making so important to No Child Left Behind? With the right data at the right time to inform decisions about resources, grouping, and instruction, schools are more likely to meet their Adequate Yearly Progress (AYP) requirements and comply with NCLB. The first years of No Child Left Behind (NCLB) required school districts to collect more data, in more detail and disaggregate it to show the progress toward achieving state standards. If teachers and administrators are going to be able to keep students from falling behind, they need to know what’s working and what students are learning during the year. What data is being collected by states and districts? The State Educational Technology Directors Association (SETDA) created a set of data elements to help state education departments meet the data reporting requirements of NCLB and to generate comparative national data. The data elements are divided into sections based on NCLB requirements. Each section contains key questions, indicators of the answer, and data elements that can be collected to measure the results. (See the Data Collection Project at www.setda.org .) What are the major barriers to effective use of data in decision making for school districts? Lack of training and interoperability are the main barriers to more effective data-driven decision making, according to a survey conducted by Grunwald & Associates on behalf of CoSN in 2004. Lack of training: 50% Interoperability—systems that are unable to share or exchange data: 42% Lack of understanding of what to do with the data: 39% Absence of clear prioriies on what data should be collected: 36% Failure to collect data in a uniform manner: 35% Outdated technology/legacy systems: 31% Low quality data – inaccurate or incomplete: 24% Timing of data collection: 24% User interface is too complicated to understand reports: 22% What are the major misconceptions about effective use of data in decision making in school districts? Build it and they will use it. It is not enough to make data available. The district has to have a process in place for analyzing the information and getting it to the right decision maker at the right time with the power and resources to act on it. Teachers need to know how to analyze data and query systems. Teachers want to teach not crunch numbers. Districts that have successfully implemented DDDM in the classroom provide teachers with on-site support, timely reports, analytic tools, and planning teams. Test scores determine the quality of a school and a child’s education. Many factors contribute to the success or failure of a student. Emphasis on test scores can give the community the wrong impression about a school. It is up to the superintendent and principals to frame the discussion so that parents and community members understand how well schools are doing and what they need to do to improve. What is necessary for the systematic use of data for decision making? The district strategic planning process provides the framework for data-driven decision making. The district data warehouse is designed to aggregate and disaggregate data needed for the planning process. An interface is developed to give different members of the educational organization access to the reports and data related to their work. Data-driven decision making can be divided into three functional areas: collection, integration and dissemination of data; analysis and reporting of data, and; process and procedures for acting on the data. Once the system is in place, a process is developed for review, analysis, and planning. School teams meet with a cross-functional district team to define requirements for disaggregation and determine interim measures. Experts from the district planning and evaluation division meet with the area superintendent and members of the school planning team to discuss the specific data and help school teams understand it within the context of their school. The district sets benchmarks to help area superintendents and principals set goals and meet expectations. How long does it take for a district to establish a process? Districts using data-driven decision making estimate that they spend at least one year planning the system and developing community support for it. Building the system and rolling it out will take at least two years. Veteran planners recommend aligning the process with a major initiative that the stakeholders have control over and responsibility for, such as recruitment, achievement, or enrollment. Some risk should be required in order to prove to skeptics that the process works. Data-driven decision making is an on-going process rather than a one-time project. District staff members need to be open and honest about results and have the freedom and responsibility to test and try new strategies for improvement. Legitimate concerns about assessment tools, data, and curriculum should be acknowledged and addressed as the district refines the process. The result is a common understanding of what goes into the aggregate data and a process for helping each student meet the same standard for success. What type of technology is needed to implement systemic data processes? The hardware may include secure servers for storage and computing devices for input and output, and a secure network to store and access data. As computing devices evolve and develop, more options with increased mobility, security, and lower cost will most likely be available. To manage information about every student over time requires sophisticated data warehouse systems with integrated student information and assessment systems. At their most comprehensive, these integrated systems combine standards-based instructional resources with multiple assessment tools, data management and analysis systems, and professional development. Districts must also adopt methods for authenticating and validating data, safeguards and security to comply with privacy legislation and protect data, and business continuity plans in case of loss or system failure. What types of skills are needed to implement systemic data processes? Perhaps the most important part of data-driven decision making is enabling decision makers to use it. Colorful reports and expensive assessment packages will have no effect unless they are combined with leadership and effective professional development. The district needs both organizational and individual capacity for improvement. Administrators need training in continuous improvement processes and the opportunity to share ideas with peers to learn how to ask the right questions. Faculty and staff members need training to learn how to read data and apply it to their goals and objectives. Instructors need training in different instructional strategies to apply when the data shows that traditional methods are not working. Many districts have created staff positions within the district or at the school site to provide analysis and training. The hands-on support helps decision makers become more sophisticated in their use of data, and as analytic and instructional tools come online, they are ready to use them. Who are the key decision makers at the district level who should be involved in the data-driven decision making process? The superintendent sets priorities and leads the effort by setting measurable, realistic goals for using data. The IT department is responsible for managing the technology infrastructure, coordinating system planning and development, and providing access tools. Many districts have established research and assessment divisions to oversee testing, reporting, and evaluation. This group provides analysis to help principals and teachers use the data and develop assessment tools that are aligned to standards. Curriculum and instruction administrators often provide the training necessary for teachers and principals to use data reports for intervention and planning. Involving classroom teachers during the design and testing of systems builds support from the people who will use it to reach students. How much analysis teachers do themselves depends on the availability of tools, support, and training. Who are the key decision makers at the school level who should be involved in the data-driven decision making process? Principals are the change agents at the school site. Without their commitment, it will be difficult for data to become an integral part of instruction. Principals model data use and encourage it by sharing the benefits and successes. They help teachers become data-driven decision makers by scheduling time for teams to meet, plan, train, and conduct evaluation. Site-based specialists or support teams assist principals and teachers with data mining and analysis. They may have special expertise or training to query the data systems and produce reports needed to inform decisions. Who are the key decision makers at the classroom level who should be involved in the data-driven decision making process? In addition to using data for determining instruction, teachers can engage students in the decision making process by helping them view appropriate reports, set learning goals, and make decisions about how to meet their goals. How are assessment data used for decision making? Assessments used during the year may be formative assessments used for instructional interventions or benchmark assessments to determine progress against an external measure. It is essential for teachers, principals and others to know what kind of assessment they are using and the proper method of analysis based on the reliability and validity of the measure. How can teachers use assessment data to inform instruction? Benchmark assessments tied to state and district standards provide quick snapshots of where students are with regard to the progress they are expected to make. Interventions used by educators include reemphasizing skills, utilizing additional diagnostics to get at the root cause, changing instructional materials, and creating cohort groups within schools and classrooms of students who have a similar achievement gap or pattern to apply instructional strategies. Although teachers have always used tests and quizzes to track student progress, these measures did not necessarily relate to standards or the assessment systems did not provide results in a timely manner. How does a district keep momentum to change going? Planning for DDDM can become an all-consuming process with perpetual refinement of processes and adjustment to the technology. At some point, district leaders need to choose an area for improvement and begin the process. Staff members will have different levels of experience and interest in changing their practice to incorporate data. Leaders need to choose specific areas and celebrate success to keep the momentum toward long-term change.