Analyzing data the data is coming
- 1. The data is coming, the data is coming! A Self Help Guide to planning an effective Assessment Meeting: Early Release Days January 31st, February 13th, and March 16th
- 2. Mapping Team Scenarios In most situations your mapping team will be in one of the following three modes: 1. Developing common assessment(s) from scratch. 2. Modifying an existing common assessment for alignment with Key Outcomes. 3. Analyzing data from previously administered common assessments.
- 3. Based upon your team’s current status, click on the most applicable scenario! Quickly ! 1. Developing common assessment(s) from scratch. 2. Modifying an existing common assessment for alignment with Key Outcomes. 3. Analyzing data from previously administered common assessments.
- 4. Developing a CA from Scratch When developing a common assessment here are some things to keep in mind. Before the first question is created, it is important to have established Key Outcomes/Local Standards. • To read an explanation of this process click here. • To view an example visit the Chemistry Master Map in the District Entity 2006-2007 of Curriculum Mapper and download the Local Standards located at the top of August. *Although you are being provided with the materials necessary to complete this activity with a mapping team, it has been our experience that it can be efficiently and accurately completed by the facilitator with Scott and Matt’s assistance. This can save the group time and allow the mapping team to focus on minor revisions and the development of test questions.
- 5. Developing a CA from Scratch – Con’t Now that your Key Outcomes/Local Standards are finished and attached to the Master Map, you are ready to begin developing test questions. A good resource to share with your mapping team would be Stiggins’ chart titled, “Links Among Achievement Targets and Assessment Methods.” Click here to view. Stiggins’ document will help the mapping team determine the appropriate type(s) of assessment based on what they are assessing.
- 6. Developing a CA from Scratch – Con’t The following clip discusses the process the mapping team will engage in to create their common assessment. Double-click the video to begin. *Clip courtesy of the Assessment Training Institute
- 7. Developing a CA from Scratch – Con’t • Request Members bring sample questions pertaining to the Key Outcomes to the next meeting. • As facilitator, we suggest you monitor how many questions address each of the standards. The idea isn’t to have the same number of questions for each of your standards, but rather an appropriate number of questions relative to how much time was devoted to each of the standards on the map and during instruction.
- 8. Modifying a Common Assessment This section is for groups that had common assessments in place before local standards (derived from Key Outcomes) were developed. Now that you have local standards, you need to see if your assessments are assessing what you are teaching!
- 9. Modifying a Common Assessment When modifying a common assessment, one of the most important steps is ensuring that the test questions are aligned to the course’s Key Outcomes/Local Standards. The following language was taken from your Boot Camp Binder: 1. Examine the Common Assessments for alignment with Standards Now that the Standards are complete, it is time to work through the common assessments to see which questions relate to the different standards. During this process, the mapping team will determine if all of the questions on the common assessments fit into one of the standards (i.e. are we testing what we said we would teach?). If we find a question that does not fit into one of the standards that we created, there are two options. The first is to add another standard covering that topic. The second option is to either reword the question to make it more appropriate, or to get rid of the question all together. 2. Check the Thoroughness of the Common Assessment It may be a good idea to create a tally sheet of how many questions address each of the standards. The idea isn’t to have the same number of questions for each of your standards, but rather an appropriate number of questions relative to how much time was devoted to each of the standards. The mapping team may decide that some revision to the common assessment is necessary at this time.
- 10. Modifying a Common Assessment Some other things to consider: • Teacher Feedback – specifically comments regarding the test format (style of questions, language, length, etc…) or content issues (either questions on irrelevant material OR too few or no questions measuring a specific standard). • Collected Data (possibly from Mastery Manager) – please help to ensure that mapping team decisions are based on data which might include trends or consensus from the team and are not necessarily reactionary in nature. • Please view the Analyzing Data pages that follow for more specific information.
- 11. Analyzing Data Based on careful analysis and inspection, I have concluded that gasoline is not necessary for my current mode of transportation! How can we use data to make meaningful and informed decisions in regards to our master maps and common assessments? *Audio clip courtesy of Bravenet.com
- 12. Analyzing Data • As the facilitator, please remember to take time to celebrate any accomplishments that may be encountered when analyzing the data! • Here are a few things to remember as you examine data with your mapping team: • NEVER show teacher or student names to the group. • Any changes based on data should be based on trends or consensus from the team. Avoid rushing to changes without sufficient evidence.
- 13. Analyzing Data There are really Teacher Reflections: two forms of data •Teachers should be given an that could be opportunity to share resources and examined: reflections that have been documented teacher reflections on their diary maps. and/or assessment data. •Teachers may have comments on how thoroughly the assessment measured what was taught as described on the master map.
- 14. Analyzing Data Assessment Data If your group gave a common assessment using Mastery Manager, here are some resources to help you successfully show them some of the results. • Please remember to avoid showing teacher or student names on any reports that you choose. Begin by logging onto the Mastery Manager website. • Remember that your login is the first portion of your e-mail address (Ex. jhawk) and your default password was/is: chsd155 We strongly urge you to change your password ASAP to help protect the confidentiality of the data!
- 15. Analyzing Data Generating Reports • Locate the name of the assessment that you plan on examining in the list of assessments. • Click on the Reports Button on the right side of the screen. • A new screen will appear allowing you to choose many different ways to display your data. In order to show the test results for ALL students taking the assessment lumped together, please follow the directions on the next page!
- 16. Analyzing Data #1 - Choose the #2 - Leave “All Teachers” this on choice at the top “Combined of the list. Selected Sections.” #4 - Click on #3 - Choose the Generate Reports “All Sections” to gain access to choice at the top the data reports. of the list.
- 17. Analyzing Data After choosing “All Teachers” and “All Sections” and clicking on the generate reports button, this screen appears allowing you access to a variety of reports. This reminds you of the assessment which you are examining. The Item Analysis report is a great report for any assessment that does not yet have local standards linked to test questions. The Item Analysis by Learning Objective report is a great report for assessments that have established local standards and have them linked to test questions. Please remember not to show any reports that display teacher or student names! Also, please only show reports that you are familiar with to avoid accidentally showing a report that may embarrass a colleague.
- 18. Analyzing Data If your common assessment does not currently have local standards assigned to test questions, then the most relevant report would most likely be the Item Analysis report. This report shows information on how students responded to all multiple choice questions. Please remember that by selecting all teachers and all sections this combines data for all teachers that gave this common assessment. It is possible to attach local standards to questions in Mastery Manager even after you have given the test. Once this is done, new reports with more detailed information will also become available without the need to re-scan any student answer sheets.
- 19. Analyzing Data Sample Size Remember, this shows This column shows combined data for all statistically how difficult teachers giving this the question was. The common assessment. number relates to the percentage of students This area shows the that got the question number of students correct. Anything that chose each highlighted in red response as well as means that less than the corresponding 50% of the students got percentage. Green that question correct. represents what the correct answer was while yellow Average Number of students leaving represents the most score on the the question blank. commonly chosen incorrect answer. assessment. Number & % of students that received a multimark answer (most commonly from poor erasing).
- 20. Analyzing Data If your common assessment does have local standards attached to its test questions, then another report will offer some very detailed information for your mapping team. Please check out the Item Analysis by Learning Objective report. This report breaks the test into pieces showing how students performed on questions related to each of the local standards. Hear ye, hear ye! The question, “How do I know if my students have learned it?” has been answered!
- 21. Analyzing Data Displays the standard being measured, the percentage of students mastering this The Mastery Cut Score standard, and which test refers to the percentage a questions were specifically student must earn to aligned with this standard. demonstrate mastery. Displays the Item Analysis for all of the questions measuring this standard. If 70% of students or more master the standard, this bar will be green. If 50-69% master the standard it will be yellow. It less than 50% master the standard, the bar will be red.
- 22. In Closing… Thanks for your hard work and leadership! Your efforts on the front lines have allowed us to move forward and improve instruction and assessment for all our students! Remember it’s about answering the question, “How will we know if our students have learned what our mapping teams have determined as important to the course?” Keep up the good work! If you have any questions or need any assistance, please don’t hesitate to contact: Scott Kubelka 455.8500 x31 Matt Timmerman 455.8500 x43