Caveon webinar lessons learned at atp and nces
Download as PPS, PDF1 like1,822 views
Caveon is organizing several upcoming events focusing on test security, including conferences and webinars featuring industry experts discussing lessons learned and best practices to combat testing irregularities. Key topics include managing media during breaches, data forensics for test integrity, and recommendations for strengthening test security measures. The document highlights the importance of proactive measures and collaboration among stakeholders to prevent cheating and enhance testing practices.
![STEVE FERRARA – PEARSON
• [Speaking primarily as a former Maryland SAD]
• School Personnel Not Well Equipped to Handle
Investigations
• Don’t have Training
• Teaching Draws on Nurturing and Supporting
Skills](https://image.slidesharecdn.com/caveonwebinarlessonslearnedatatpandnces-120330151042-phpapp02/85/Caveon-webinar-lessons-learned-at-atp-and-nces-23-320.jpg)
Caveon webinar lessons learned at atp and nces
- 1. Upcoming Caveon Events • NCME in Vancouver, April 12-16 – NATD Breakfast Plenary, Dr. John Fremer, April 13 • CCSSO TILSA SCASS at National Harbor, MD • Conference for the Statistical Detection of Potential Test Fraud, University of Kansas, May 23-24 • National Conference on Student Assessment in Minneapolis, June 26-29 • International Testing Commission Conference in Amsterdam, July 3-5 • National Collegiate Testing Association in Minneapolis, August 1-4 www.caveon.com
- 2. Caveon Online • Caveon Security Insights Blog – http://www.caveon.com/blog/ • Twitter – @Caveon • LinkedIn – Caveon Company Page – Caveon Test Security Group • Please contribute! • Facebook – Will you be our “friend?” – “Like” us! www.caveon.com
- 3. Caveon Webinar Series “Lessons Learned at ATP and NCES” Steve Addicott Dr. John Fremer Vice President President Caveon Caveon Consulting Services March 28, 2012
- 4. Agenda • ATP 2012 Security Lessons Learned – Steve Addicott • NCES 2012 Security Lessons Learned – Dr. John Fremer • Q&A
- 5. About Caveon…. • Our mission: To help protect the tests, programs, and reputations of our clients, some of the most important test programs and test publishers in the world – Certification & Licensure – State DOEs & School Districts – DOD – International • Entering our 9th year of operation • Provide practical, measureable security enhancements www.caveon.com
- 6. Test Security Highlights from ATP Theme: • A dozen sessions focused on security – Too many at the same time! • Caveon sessions with our clients – Conducting investigations – Managing the media in cheating scandals – How to implement Data Forensics – Embedded verification tests • Security Committee breakfast – Standing room only, despite the 7am start time!
- 7. Key Industry Veterans Focusing on Security • Liberty Munson, • Ashok Sarathy, Microsoft GMAT • Chad Buckendahl, • Steven Barkley, Alpine Testing CFP • Mark Poole, Pearson • Daniel Eyob, VUE GMAT • Cathy Donath, • Nikki Eatchel, Donath Group Questar • Ray Nicosia, ETS
- 8. Conducting Investigations Ben Mannes, Director of Test Security, ABIM Lessons Learned • Was a security breach (due to a possible vulnerability in your policies or infrastructure) identified during the incident? • If so, what can be done to change policies, procedures, or technologies to fix it? • A cost/benefit analysis should be done to address the vulnerability identified by the incident. • Are there other ramifications to the organization from the incident?
- 9. Gimme Shelter: Managing the Media Storm of a Security Breach Jim Vasalek, Sr Director of Public Affairs, LSAC NON-Traditional Media Blogs, bulletin boards, chat rooms, file-sharing sites, and social media No professional code of ethics No editors No fact checkers No obligation to get your side of the story Can keep a story alive well after traditional media has lost interest, even well after it actually has died May be more influential with your key audiences than traditional media
- 10. Implementing Data Forensics Aimee Hobby Rhodes, Director of Exam Security, CFA Institute Collect All the Evidence You Can!! • Use multiple statistical analyses – Similarity Analysis, Erasure Analysis, Prior score analysis (gains/losses), identical response analysis, etc • Suspected Cheating/Incident Reports • Seating Chart/Floor plans • Candidate Connections • Anything else you can find!
- 11. Implementing Data Forensics Jennifer Semko, Partner, Baker & McKenzie Your Candidate Agreement is foundational! • Are candidates on notice that – sharing items is a breach? – studying from recalled items is improper? – Data Forensics may be used? • Did you reserve the right to: – Invalidate scores? Suspend access to examination? Take other action? • What are the grounds for action? Is there a “catch all”? • Do you: – regularly review your agreement language? – have uniform security procedures and policies in place?
- 12. Embedded Verification Test Liz Burns, Senior Manager of Technical Certifications Programs, Juniper Networks Key Benefits of New Security Tactic, EVT • Legally defensible “proof” for enforcement actions – Does not require “intent” • Enforces against unqualified candidate • Works on broad range of content theft • Fairly inexpensive to do • Provides intelligence on vulnerability of your exams • Can do “results hold” for proactive enforcement – Changes definition of “PASS” to reaching the cut score AND the fraud detection threshold
- 13. Social Media • Large emphasis placed on Social Media at the conference. – twitter hash tags, QR codes, or LinkedIn links • Many were tweeting and blogging about the conference during live sessions – Getting the word out instantaneously. • Get on the train….NOW! – be a part of the online conversation – how information is being shared.
- 14. Security Committee Breakfast • Standing room only…really – Early morning didn’t deter anyone – Despite myriad “celebrations” the evening before • Lots of important work – Security Survey – Enforcement – Lobbying – Standards • All volunteers – Pat yourself on the back!!
- 15. JOHN FREMER – CAVEON TEST SECURITY STATE ASSESSMENTS IN TRANSITION The Perfect Test Security Storm
- 16. SECRETARY DUNCAN • “States, districts, schools, and testing companies should have sensible procedures in place to ensure tests accurately reflect student learning.” • Quoted in recent Atlanta Journal Constitution article – “Cheating our children: Suspicious school test scores across the nation.”
- 17. ETS – CARSWELL WHITEHEAD • All High Stakes Tests are Subject to Attempts to Gain an Unfair Advantage • Must Build in Processes to Detect Irregularities – Before – During – After
- 18. GREG CIZEK – UNC, CHAPEL HILL PREVENTION OF IRREGULARITIES IN ACADEMIC TESTING: • Test Providers / Contractors * Clear definition of cheating * Clear, educator-referenced materials * Web-based qualification utility, database? * Less corruptible formats * CBT/CAT delivery
- 19. DAVID FOSTER – KRYTERION SEVEN DETECTION PRINCIPLES 1. Focus…Concentrate on high-risk threats first 2. Adapt…Match detection methods to the threat 3. Backup…Use layers of detection methods 4. Predict…Watch out for new threats 5. Filter…Rule out other explanations 6. Evaluate…Use detection to evaluate security 7. Plan…Set up to succeed
- 20. BRIAN JACOBS - UNIV OF MICHIGAN DETECTING IRREGULARITES • Benefits of statistical analysis – Relatively low cost – Covers the entire population – Measure of the “extent” of the irregularity – e.g., likelihood it would have occurred by chance – Systemic patterns as well as individual cases of concern • Limits of statistical analysis – Like preliminary screens for rare diseases, statistical analyses to detect cheating can have a high rate of false positives and false negatives • Critical to complement statistical analyses with other methods – Can never identify the individual responsible for the manipulation
- 21. JAMES LIEBMAN – COLUMBIA UNIVERSITY LAW SCHOOL • Keep an open line for reports of infractions • Teachers as allies • Duty to report immediately • Multiple locations (principal, monitors, local test office, state test office, “special investigations”) • Allow anonymity; offer confidentiality
- 22. SCOTT NORTON – LA DEPT OF ED Areas for Improvement at State Level • More state oversight is needed for district-led investigations. • Standardization across states may be needed for established procedures such as erasure analysis. • Better information is needed about other statistical analysis procedures for detecting suspect patterns of responses, unusual gains or losses, etc.
- 23. STEVE FERRARA – PEARSON • [Speaking primarily as a former Maryland SAD] • School Personnel Not Well Equipped to Handle Investigations • Don’t have Training • Teaching Draws on Nurturing and Supporting Skills
- 24. John Fremer – Caveon Test Security Ten Recommendations Moving Forward 1. Acknowledge the seriousness of security issues 2. Expect cheating and plan to be proactive 3. Use multiple detection methods and forensic statistics 4. Minimize testing windows 5. Strengthen the chain of custody
- 25. Ten Recommendations Moving Forward…Cont. 6. Increase the emphasis on security training 7. Allocate adequate resources for test security 8. Pilot techniques for detection of cheating 9. Continue to learn from others 10. Monitor new advances in anomaly detection and prevention (e.g. “Epidemiological Model”)
- 26. RESOURCES • CCSSO/ATP – Operational Best Practices • ATP Security Committee • Caveon Test Security – Blog – Webinars • (In development) NCME Guidelines • (In development) TILSA Guidebook for State Assessment Directors on Data Forensics
- 28. Continue The Conversation - Follow us on twitter @caveon for updates and events - Check out our blog…www.caveon.com/blog/ - LinkedIn Group – Join our Caveon Test Security group to ask questions and join discussions about test security topics - Slides of this and past Webinars are available at www.caveon.com/resources/webinars
- 29. Thank you! Steve Addicott Dr. John Fremer Vice President President Caveon, LLC Caveon Consulting Services steve.addicott@caveon.com john.fremer@caveon.com