Everything you need to implement a data forensics program
1 like522 views
This document provides an overview and agenda for a presentation on implementing a data forensics program to monitor test security. The presentation covers introducing data forensics and its uses in monitoring for security breaches and investigating potential breaches. It also discusses obtaining budget and stakeholder support, the legal foundation and issues around data forensics, and managing investigations. The agenda includes sections on introduction to data forensics, program implementation, obtaining budget and support, legal issues, and managing investigations.
![Admissibility of Expert Evidence in Court
• Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S.
579 (1993)
– ―[U]nder the Rules [of Evidence] the trial judge must
ensure that any and all scientific testimony or evidence
admitted is not only relevant, but reliable….‖
• Kumho Tire Co. v. Carmichael, 526 U.S. 137 (1999)
– ―The objective of [the trial court’s gatekeeper] requirement
is to ensure the reliability and relevancy of expert
testimony. It is to make certain that an expert … employs
in the courtroom the same level of intellectual rigor that
characterizes the practice of an expert in the relevant
field.‖](https://image.slidesharecdn.com/everythingyouneedtoimplementadataforensicsprogram-120927141827-phpapp02/85/Everything-you-need-to-implement-a-data-forensics-program-33-320.jpg)
Everything you need to implement a data forensics program
- 1. ―Everything You Need to Know to Implement a Data Forensics Program‖ Presenters: Dennis Maynes - Chief Scientist, Caveon Test Security Jennifer Ancona Semko - Partner, Baker & McKenzie Kerri Davis - Anti-Piracy Program Manager, Microsoft Learning Presented September 27th, 2012
- 2. Agenda Introduction to Data Forensics Program Implementation Obtaining Budget and Support Legal Foundation of Data Forensics and Navigating Legal Issues Managing Investigations 002
- 3. Presented by: Dennis Maynes INTRODUCTION TO DATA FORENSICS 003
- 4. Introduction • Purpose of a data forensics program – Measure and manage security risks – Ensure fair and valid testing – Use statistics to monitor and to investigate • Purpose of security initiatives – Mitigate losses and liability • Illustrations of loss – November 2007, Denver, de-icers – July 2011, Atlanta, 178 educators 004
- 5. Data Forensics • Science of examining data to find potential security risks • There are clues in the data relating to: – Collusion – Use of recalled questions – Rogue review courses – Testing sites with poor security – Exams and items that have become exposed • ―We balance probabilities and choose the most likely. It is the scientific use of the imagination.‖ – Sherlock Holmes, The Hound of the Baskervilles 005
- 6. Examples of Statistics • Aberrance or person-fit – (pre-knowledge) • Similarity – (collusion) • Erasures – (tampering) • Gains – (pre-knowledge) • Shared e-mails – (improper coordination) • Foreign tests – (extra ―help‖) • Response time – (braindumps) • Score differences – (pre-knowledge) 006
- 7. Test Security Threat Scale Statistical Anomalies Testing Irregularities Security Violations Security Breaches Test Fraud 008
- 8. Test Fraud Taxonomy • Content Theft and/or Disclosure • Collusion and/or Providing information during the exam • Violation of Proctoring and/or Administration Rules • Tampering and/or Manipulating the Score Distribution • Based on Amrein-Beardsley, A., Berliner, D. C. & Rideau, S. (2010). Cheating in the first, second, and third degree: Educators' responses to high- stakes testing. 009
- 9. Data Forensics Uses • Two modes – Monitor for security breaches – Investigate potential breaches • Inform investigations • Take corrective actions – Score invalidations – Test site closures – Replace test items • Manage security health • Monitor security risk levels 010
- 10. Data Forensics Monitoring • Examine ALL of the data • Must correct for multiple comparisons – Bonferroni Correction – Probability for threshold is /n – Example: if n = 10,000 and is .05 – use .0000005 • Probabilities allow – Objective measures – Ensemble statistics – Error rate control 011
- 11. Data Forensics Inference • To invalidate scores, most psychometricians require – An eye-witness account and – Probability less than one in ten thousand. • Hypothetical question: Suppose two individuals submitted identical 500 word essays – would you act? • Basis for action – Strength of the evidence – Is the score trustworthy? If you accept statistics to determine candidate competence, why would you reject statistics to determine score trustworthiness? 012
- 12. Circumstantial Evidence • Requires an inference or deduction • Seek ―disconfirming‖ or plausible explanations • Value of multiple pieces of evidence • Collect and document all the evidence • Apply policy consistently for each case ―Circumstantial evidence is a very tricky thing. It may seem to point very straight to one thing, but if you shift your own point of view a little, you may find it pointing in an equally uncompromising manner to something entirely different.‖ – Sherlock Holmes, The Boscombe Valley Mystery 013
- 15. Presented by: Kerri Davis PROGRAM IMPLEMENTATION 014
- 16. Test Security is a Process Data Forensics can be used to improve the process Assess overall test security risks Identify strengths and weaknesses in the process Document the test security process Take steps to improve the test security process The goal of a Data Forensics program should be the improvement of test security. The primary purpose of Data Forensics is NOT to apprehend and punish potential cheaters. 015
- 17. General implementation approach • Establish policy and precedent: scope of your program, budget, legal, planned outcomes • Obtain stake-holder support • Create infrastructure • Create agreements • Reports • Revise exam policies • Implementation – Conduct pilots and dry runs – Train staff – Perform data forensics analyses – Review and revise 016
- 18. Stakeholder support is critical • Explain how the statistical analysis works • Present results from data forensics analyses – How many test takers were cheating? – How many locations had weak security? • Outline overview of cheating evidence • Explain work flow • Outline costs associated with enforcement • Address individual questions/concerns 017
- 19. Summary • Think of a 3-tiered approach to your data forensics implementation: 1. Establish scope 2. Propose a budget 3. Create an action plan • Use this approach to formulate your program and to gain stakeholder support • Don’t be afraid to revise as you go; using data forensics is a process itself.
- 20. OBTAINING BUDGET AND SUPPORT 020
- 21. It’s not ―IF‖… it’s ―WHEN‖ • Don’t wait for a breach to occur before you seriously think about security. • Don’t be in a position of explaining (to your stakeholders, the public, or the press) why you are not protecting the integrity of your exams. Successful implementation of a Data Forensics program will anticipate inquiries by the media and the public in order to communicate that the program is pursuing a proper course for ensuring the tests are administered fairly and securely. 021
- 22. Assemble the Security Team • Identify key personnel from the affected departments: – Exam Development – Psychometrics – Exam Administration – Legal – Risk Management – Scoring/Grading – Professional Conduct 022
- 23. Make the Case for Security • Exam scores mean nothing if candidates can gain an unfair advantage by cheating. • Share highly-publicized examples of individuals gaining an unfair advantage – Prevention: Avoid being a news story • Present the Cost of Security vs. Insecurity – Cost of Development – Reputational Harm – Threat to the Public 023
- 24. Plan for Successful Implementation • Propose a security budget • Identify key individuals, their roles, and their time commitment • Develop and document process flows • Anticipate and overcome obstacles – Fear of statistics • Don’t understand them • Don’t understand how to use them – Fear of what people might think – Lack of familiarity with score review process 024
- 25. Questions at this point?
- 27. Presented by: Jennifer Ancona Semko LEGAL FOUNDATION OF DATA FORENSICS AND NAVIGATING LEGAL ISSUES 028
- 28. Why is this important? 029
- 29. The First Brick: the Candidate Agreement • Contract: An agreement between two or more persons which creates an obligation to do or not to do a particular thing. A legal relationship consisting of the rights and duties of the contracting parties. Black’s Law Dictionary, Sixth Edition • Your agreement with test takers defines the relationship • Memorializes your (and their) rights and obligations • If done properly, makes expectations (and remedies) clear 030
- 30. What does your agreement say? • Are candidates on notice that sharing items is a breach? • Are candidates on notice that studying from recalled items is improper? • Did you reserve the right to invalidate scores? Suspend or permanently ban access to the examination? To take other action? • Are candidates on notice of the possible use of data forensics? • What are the grounds for action? Is there a ―catch all‖? • Do you regularly review your agreement language? • Do you have uniform security procedures and policies in place? • Are candidates required to cooperate in investigations? 031
- 31. Successfully Using Data Forensics •Can you defend your actions? –Do you have to prove ―cheating‖? •Contract law – ―good faith‖ –Language of agreements –Documented policies & procedures –Taking all steps to show ―good faith‖ •Will you need expert testimony? •General deference to exam programs –State actors: due process 032
- 32. Admissibility of Expert Evidence in Court Federal Rule of Evidence 702, Testimony by Experts If scientific, technical, or other specialized knowledge will assist the trier of fact to understand the evidence or to determine a fact in issue, a witness qualified as an expert by knowledge, skill, experience, training, or education, may testify thereto in the form of an opinion or otherwise, if: (1) the testimony is based upon scientific facts or data, (2) the testimony is the product of reliable principles and methods, and (3) the witness has applied the principles and methods reliably to the facts of the case. 033
- 33. Admissibility of Expert Evidence in Court • Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 (1993) – ―[U]nder the Rules [of Evidence] the trial judge must ensure that any and all scientific testimony or evidence admitted is not only relevant, but reliable….‖ • Kumho Tire Co. v. Carmichael, 526 U.S. 137 (1999) – ―The objective of [the trial court’s gatekeeper] requirement is to ensure the reliability and relevancy of expert testimony. It is to make certain that an expert … employs in the courtroom the same level of intellectual rigor that characterizes the practice of an expert in the relevant field.‖
- 34. Deference . . . within limits
- 35. Deference to Exam Programs Murray v. ETS, 170 F.3d 514 (5th Cir. 1999) (SAT Exam) • Louisiana basketball player; needed 820 on SAT • Scored 700, then 1300 – Similarity to nearby student (3 in 100 million odds) – Scored 800 on retake • ―ETS’s contract with Murray clearly and explicitly reserved to ETS the right to withhold any scores ETS had reason to believe were not valid. The only contractual duty ETS owed to Murray was to investigate the validity of Murray’s scores in good faith.‖ 034
- 36. Deference to Exam Programs Langston v. ACT, 890 F.2d 380 (11th Cir. 1989) (ACT Exam) • Alabama football player; scored 10 on ACT; then 20 • Inconsistent with GPA; unusual similarity to nearby student • ―Under the governing law, the outcome of plaintiff’s case does not turn on whether or not plaintiff cheated on his exam, but only on whether or not ACT carried out its contractual obligations in good faith.‖ 035
- 37. MANAGING TEST SECURITY INVESTIGATIONS 025
- 39. Managing Investigations • Do your investigators have all they need to be effective? – Corporate support (budget, effective legal counsel, training) – Software/services – Support from other departments (Psychometrics, Test Development, Registration & Credentialing, etc.) • What procedure is in place to select cases for investigation? – Is it prudent/efficient to investigate all matters? • What metrics exist to determine the success/efficiency of an investigation? – Did the investigation glean the desired information? – When does an organization ―close‖ an investigation? 027
- 40. Managing Investigations • Who within (or outside) your organization conducts investigations? When? – May depend on investigation type: exam-day incidents, ongoing copyright infringement, collusion, proxy testing • At what point do you involve legal counsel, board members, or other departments? • Do your policies and procedures reflect what is needed to manage investigations? – Are candidates obligated to cooperate? • How are results reported? Who makes sanctions decisions?
- 41. Gather Evidence • Similarity analysis, gains analysis, other statistics • Reports of security incidents • Seating charts and chain of custody of materials • Review test taker associations and connections • Review access logs to secured exam content • Review score histories of test takers and locations • Review test booklets for signs of ―work‖ • Responses by test center staff & test takers • Adherence to security policies 018
- 42. Evaluate Evidence • Do alternative explanations exist? • Are candidate explanations/responses convincing? • Could test fraud have occurred? • Are the test results trustworthy? – Evaluation depends upon trustworthiness of the scores, NOT an inference of behavior. The proper use of Data Forensics is to certify the trustworthiness of the test results and the integrity of the test administration. It is NOT proper to use these results to place a label, such as ―cheater,‖ on an individual. 019
- 43. Wrap up & key takeaways • Data Forensics – Measure and manage risks – Ensure fair and valid testing – Mitigate losses and liability • Implementation – Security is a process, not a state – Policy—what will we do with the results? – Breaking down organizational ―fiefdoms‖ • Support – Not ―if‖, but ―when‖ – Nobody plans to fail, but…
- 44. A long and winding road…. Key Takeaways, cont. • Legal – Agreement is your foundation • Even in K-12! – Don’t be ―arbitrary and capricious‖ – Consistent and uniform • Investigations – ―Go/No Go‖ decision criteria • Constrained resources – Focus on the results, not the behaviors • Statistically ―Indeterminate results‖ vs ―You’re a cheater!‖
- 45. Got questions?
- 46. kerrid@microsoft.com jennifer.semko@bakermckenzie.com dennis.maynes@caveon.com Caveon Confidential. Do not share without permission.
- 47. Hope to see you soon… • CCSSO TILSA SCASS – Oct 1-5 – Indianapolis • Next month’s webinar – ―Do It Yourself Security Audits and Security Investigations‖ – Tuesday, Oct.16, noon EDT • ICE (Institute for Credentialing Excellence) – Nov 6-9 – Palm Springs
- 48. Want more? LinkedIn group ―Caveon Test Security‖ twitter @caveon for updates, news, connect Blogs! Caveon Security Insights – www.caveon.com/blog Cheating In The News – www.caveon.com/citn www.caveon.com/resources/webinars - to see past sessions Contact skyler.weisenburger@caveon.com for slides, comments, and questions about this session
Editor's Notes
- #3: Introduction to Data Forensics - DennisProgram Implementation - KObtaining Budget and Support - KManaging Investigations - JLegal Foundation of Data Forensics and Navigating Legal Issues - J
- #18: Test Security is a ProcessWhetheryou already have a program in place, or are just starting out, it’s important to realize that data forensics will be one factor in your security plans. Data Forensics can be used to improve the processYou can take your program to the next level; data forensics can help you understand your gaps and needs in order to enhance any efforts you’re already doing. Assess overall test security risksIdentify strengths and weaknesses in the processData forensics provides that holistic view of what is going on in your testing program—the behaviors and trends that indicate where gaps lay.Document the test security processOnce you have the forensics piece in place, it can help you document the process overallTake steps to improve the test security process
- #19: First you will want to look at your program overall, if you have one, and identify its strengths and weaknesses. Then you can lay a foundation to stakeholders on how data forensics will help your program—whether its for the sole purpose of identifying security risks, for enforcement, for metrics, for etc. You may want to have a conversation with your legal team on your goals with data forensics, and get parameters around what you can do with the data. You might also talk to your exam delivery provider about any forensics they do, and how you can work with them. This is the stage where you set the scope of data forensics within your overall security program.Next, suggest a budget—will you do forensics on all your exams, part of your exams, part of your program, etc. This is important so that stakeholders know exactly what they are paying for and what metrics you are hoping to provide.At this point you should prepare your stakeholders for their support by having a clear scope, a legally-defensible plan for what to do with the data, a budget, and what kind of reporting or metrics you hope to provide. Once you have stakeholder support, you can begin to create infrastructure. Create your legal foundation by getting the right agreements in place. For instance, this may mean revising a candidate policy to include the use of forensics. Learn how to read reports and train any staff. Study the data and revise what you are measuring, if it is not helpful. At this point you may be fine-tuning your overall plan and policy as you learn how forensics work.Last, you’ll want to implement any changes to your program, or enforcement actions, that you planned to do as a result of forensics. These changes may not be immediate; you may want to have a period of “data collecting” only. But be sure to deliver on what you promised to stakeholders.
- #20: Also important to do your homework regarding legal limitations. You don’t want to make promises you can’t keep.
- #23: So probably the biggest concern you might get from a stakeholder is a question as to why you need data forensics. Can’t we just catch the cheaters without it? This is where you stress that the data forensics is much more than just “going after cheaters”—it’s about a holistic, independent look at your whole security program. The strength in data forensics lies in finding those hidden vulnerabilities, so that you can head them off before there are breaches. Think of it as preventative security spend rather than reactive. The other concern you will probably get is around the legal defensibility of such a program, and this is where having a clear scope and action plan, that you have cleared with your legal team, can help.
- #25: Again, this is where you really stress the importance of proactive planning vs reactive—prevention over enforcement. The cost of breaches are both tangible: rescoring, redeveloping exams, re-administering; and intangible: reputation, integrity
- #26: To sum:
- #34: Jennifer to edit this with new links/sources
- #39: Jennifer
- #43: [not sure I understand how this slide relates to data forensics]?
- #44: This slide speaks to the heart of “what are you going to do with your information?” could there be any other explanation for why test anomalies occurred? Depending on the size of your program, your data forensics will most likely lead to more follow-ups with schools, teachers, proctors, or exam delivery providers, rather than giving you a clear case to do xyz.