BCC 2003 - Reintroducing Watchlist Statistics
0 likes567 views
This document discusses face recognition technology and biometrics. It summarizes the results of tests on face recognition systems, distinguishing between verification, watchlist, and identification tasks. Key metrics for each task are defined such as false accept rate, detection rate, and rank-n identification. The document urges using standardized tests and proper terminology to evaluate biometrics systems.
BCC 2003 - Reintroducing Watchlist Statistics
- 1. Reintroducing Watchlist Statistics Duane Blackburn Federal Bureau of Investigation dblackburn@fbiacademy.edu Biometrics Technology
- 2. Face Recognition Debate Fall 2001 Could the hijackers have been spotted? Biometrics Technology
- 3. Face Recognition Debate Fall 2001 Can it track me? Gregory Tara Kaye Nancy DOB: 4/21/74 Dorsey Leflore Dorothy Divorced Mother of 2 Shedd 2002 Income: $45,328 Matthew Millner Dennis Ricks Joe Grammer Gerald Elinor Galbraith Pardini Allan Olliff Wilfred Moesch Clayton Tayler Weddell Persky Biometrics Technology Nicolas Marocco
- 4. Face Recognition Debate Fall 2001 Or will this happen? Osama bin Laden Biometrics Technology
- 5. Why the confusion? • Confusing – Verification vs. Identification • incorrect usage of statistics – Identification (open set) vs. Identification (closed set) – False alarm rate, false accept rate, false match rate, etc. Biometrics Technology
- 6. Statistics Shortcomings • ROC (Verification) – Making a claim to your identity • CMC (Identification) – Each probe is in the database • Are measuring closed set identification • Neither relates to the problem! – Solution: develop statistics that measure this problem. Biometrics Technology
- 7. Chronologically • Face Recognition at a Chokepoint - Scenario Evaluation Results. http://www.dodcounterdrug.com/facialrecognition • FRVT 2002. http://www.frvt.org. • Both available in the Biometrics Catalog, http://www.biometricscatalog.org. Biometrics Technology
- 8. First Step • Forget what you know! Biometrics Technology
- 9. Baseline Performance Face Recognition Vendor Test 2002 (FRVT 2002) - http://www.frvt.org Biometrics Technology
- 10. Three Tasks • Verification – Are you who you say you are? • Watchlist – Are you in my database? If so, who are you? • Identification – You are in my database, can I find you? Biometrics Technology
- 11. Verification Threshold = 0.8 Claim Response = 0.75 Threshold = 0.7 Claim Response = 0.65 Threshold = 0.6 • Determines if the claimed identity of a face is correct. • Metrics: – Probability of Correct Verification • 1- (False Reject Rate) – False Accept Biometrics Technology
- 12. Verification Case #1 - Want a low false accept rate False Accept Rate = 0.1% Verification 1 0.9 Probability of Correct Verification 0.8 0.7 Cognitec 0.6 0.5 Eyematic 0.4 Identix 0.3 0.2 0.1 0 1 False Accept Rate = 0.1% Biometrics Technology
- 13. Verification Case #2 - Want a high verification rate Probability of Verification = 95% Verification Probability of Correct Verification = Cognitec 1 Eyematic Identix 95% 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 False Accept Rate Biometrics Technology
- 14. Verification ROC Verification Probability of Correct Verification = 95% Cognitec 1 Eyematic Identix 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 False Accept Rate Verification 1 Probability of Correct 0.8 Verification 0.6 Cognitec Eyematic 0.4 Identix 0.2 0 1 False Accept Rate = 0.1% Biometrics Technology
- 15. Watchlist Threshold = 0.8 0.75 Threshold = 0.7 0.65 Threshold = 0.62 Database 0.60 0.55 0.25 Biometrics Technology
- 16. Watchlist Threshold = 0.8 0.75 Threshold = 0.7 Database 0.65 0.70 0.25 0.55 Biometrics Technology
- 17. Watchlist Case #1 - Want a high detect and identification rate Detect and Identify = 75% Watchlist Detect and Identification Rate = Cognitec 1 Eyematic 75% Identix 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 False Alarm Rate Biometrics Technology
- 18. Watchlist Case #2 - Want a low false alarm rate False Alarm Rate = 0.1% Watchlist Detection and Identification Rate 1 0.9 0.8 0.7 Cognitec 0.6 0.5 Eyematic 0.4 Identix 0.3 0.2 0.1 0 1 False Alarm Rate = 0.1% Biometrics Technology
- 19. Watchlist Watchlist Detect and Identification Rate = 75% Cognitec 1 Eyematic Identix 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 False Alarm Rate Watchlist 1 Detection and Identification 0.8 0.6 Cognitec Rate Eyematic 0.4 Identix 0.2 0 1 False Alarm Rate = 0.1% Biometrics Technology
- 20. Identification 0.75 Database 0.65 0.80 0.70 0.25 0.55 Biometrics Technology
- 21. Identification Database That’s watchlist, not identification! Biometrics Technology
- 22. Identification • I know you are in the database, but who are you? – Closed universe test. • Compares your picture to all pictures in the database and lists them according to similarity. • Metrics: – How often is the top answer correct? – How often is the answer in the top 2? – How often is the answer in the top 3? – How often is the answer in the top n? Biometrics Technology
- 24. Three Tasks • Verification – Are you who you say you are? • Watchlist – Are you in my database? If so, who are you? • Identification – You are in my database, can I find you? Make sure you use the correct terms and the correct statistics! Biometrics Technology
- 25. Government Biometric References • Biometrics Catalog - http://www.biometricscatalog.org • FRVT - http://www.frvt.org • Biometric Consortium – http://www.biometrics.org Biometrics Technology