Confronting Data Manipulation
- 1. Confronting Data Manipulation on Your Campus Dr. Will Miller Director of Institutional Research and Effectiveness Flagler College
- 3. Context • We rely on others for data • IPEDS • Outcomes reports • Closing the Loop • Course evaluations • Technology has eliminated the paper trail
- 4. Data Manipulation • Comes in a variety of flavors • Different than data error! • Survey responses • Course evaluations • Course outcomes • Closing the Loop reports • Or more troubling forms
- 5. Flagler College • Founded in 1968 • Baccalaureate only • Approximately 2700 students • U.S. News Top Regional • The unicorn of admissions • Increase applicant pool • Increase cohort size • Decrease merit scholarships • Increase class profile
- 6. But… • Unicorns aren’t real
- 7. The Swift Fall of the Unicorn • Background of the offender • New Director of IRE and New Director of College Writing in Summer 2013 • During a random audit of student placements from fall semester, we notice a vast majority of students were under-placed in writing courses according to information in Student Management System • Initially blame advisors • But, test scores in the system do not match hardcopy test scores in student files • The two cases that sealed his fate
- 8. Internal Process • Notification of VPAA (February 7) • More analysis (Weekend of February 8-9) • Audit of data entry changes shows one user affiliated with all changes on weekends in September for numerous years (beginning in 2004) • Update of VPAA (February 9 around 2am) • Notification of President (February 10 at 8am) • More analysis • Final notification of President (February 12 at 8am) • Trip with VPEM (February 13) • Removal (February 14) • Measure Impact (Weekend of February 15-16) • Notify Cabinet on February 17 at 8am
- 11. Contributing Factors • Relationships • Whistle blower • Access to databases • Test score entry • Policies on paper v. policies in practice • Silos
- 12. The Public Process • External investigators • External data auditors • Crisis communication consultants • Openness and transparency • Stakeholders • Campus community
- 15. Reporting and Response • Open and transparent • Easier to tell all at the beginning • No sugar coating • Responses from community • Responses from stakeholders • Accrediting bodies • Government • Publications
- 16. Impact • Faculty • Students • Staff • Institution as a whole
- 17. And the Other Shoe Falls • Students aren’t impacted…if they take math and English first semester • Which some students don’t • 223 students end up misplaced in math, English, or in a few cases both due to test score fraud • A whole new process • Pull transcripts, courses taken, time of registration v. time of data manipulation, determine remediation while consulting with faculty, presenting remediation, and in some cases scrub transcripts • Interests DoE
- 18. Lessons Learned Any press does not necessarily make for good press Be open and transparent Be selective in choosing investigators Do not worry about cost…you’ll be remembered by how you respond
- 19. Lessons Learned Do not sugarcoat Policies exist to be followed Do not hold press conferences Some people don’t need access to some things
- 20. Lessons Learned Do not blindly trust
- 21. Lessons Learned It can be a positive impetus for change • We’re actually a unicorn • Decreased applications, increased yield, made the class (and then some), and increased profile • Re-established relationships between enrollment management and faculty • Personnel changes
- 22. Questions