![Closed-Set Identification (1:N)
17
➢ Cumulative Match Characteristic (CMC) curve:
○ summarizes the overall performance by reporting recall over a range of cutoff
points
○ represents the expectation of finding the correct match in the top r matches,
where r is the rank considered in the final ranking result
○ Formally, the CMC can be defined as follows:
where
● i ∈ [1...N] are the probes to match
● Gi represents the position of the probe i in the gallery set sorted in descending
order with respect to the scores](https://image.slidesharecdn.com/01-introduction-241125143951-d542afef/85/Deep-Face-Recognition-Tutorial-at-SIBGRAPI-2018-17-320.jpg)
![Generic Pipeline
22
[from OpenBR tutorial]
ROC CMC IET
Evals](https://image.slidesharecdn.com/01-introduction-241125143951-d542afef/85/Deep-Face-Recognition-Tutorial-at-SIBGRAPI-2018-22-320.jpg)
![Convolutional Neural Networks
25
Keys: Huge training dataset & complex network structure
DeepFace
[Taihman et
al. 2014]
FaceNet
[Schroff
et al. 2015]
4.4M training
images
200M training
images](https://image.slidesharecdn.com/01-introduction-241125143951-d542afef/85/Deep-Face-Recognition-Tutorial-at-SIBGRAPI-2018-25-320.jpg)
![IARPA Janus Benchmarks A
30
[from Klare et al. CVPR’15]](https://image.slidesharecdn.com/01-introduction-241125143951-d542afef/85/Deep-Face-Recognition-Tutorial-at-SIBGRAPI-2018-30-320.jpg)
![IARPA Janus Benchmarks A
31
IJB-A brings up the question: “How do we build a model of a subject given multiple
heterogeneous media?”
[from Klare et al. CVPR’15]
For the first time a subject’s set contains:
➢ Still images
➢ Frames from multiple videos](https://image.slidesharecdn.com/01-introduction-241125143951-d542afef/85/Deep-Face-Recognition-Tutorial-at-SIBGRAPI-2018-31-320.jpg)
Deep Face Recognition Tutorial at SIBGRAPI 2018
- 1. Deep Face Recognition Iacopo Masi* USC ISI Tutorial at SIBGRAPI 2018 Yue Rex Wu USC ISI Tal Hassner Open University of Israel Prem Natarajan USC ISI
- 2. Introduction Deep Face Recognition: A tutorial SIBGRAPI 2018
- 3. Tutorial Program and Schedule 3 ➢ PART I: ○ Introduction (9:00 - 9:30) ○ Face Alignment, Preprocessing and Training Collections (9:30 - 10:30) ➢ Break (Relax, coffee or questions) (10:30 - 10:45) ➢ PART II: ○ Network Architecture and Loss Functions (10:45 - 11:40) ■ Small Break (5 min) ○ Face Matching (11:45 - 12:00)
- 4. Scope of the tutorial 4 ➢ This Talk: ○ What is Face Recognition? ○ Why Face Recognition? ○ When to use it? ○ Who is using it? (Applications and Impacts) ➢ Next Talks: ○ How to perform it effectively? ■ Training Collections ■ Face Preprocessing ○ Network Architecture and Loss Function ○ Face Matching
- 5. Machine Vision Face Recognition 5 ➢ Face Recognition as a Computer Vision Problem par excellence ○ “I have a (face) image, I would like to know who is in the image” ○ “Given two face images, can we tell if it is the same person?” ➢ Easy for humans… ○ ….but unfortunately, given a face image, machines see this:
- 6. 6 ➢ Face Recognition as a Computer Vision Problem par excellence ○ “I have a (face) image, I would like to know who is in the image” ○ “Given two face images, can we tell if it is the same person?” ➢ Easy for humans… ○ ….but unfortunately, given a face image, machines see this: Machine Vision Face Recognition
- 7. 7 Machine Vision Face Recognition ➢ Face Recognition as a Computer Vision Problem par excellence ○ “I have a (face) image, I would like to know who is in the image” ○ “Given two face images, can we tell if it is the same person?” ➢ Easy for humans… ○ ….but unfortunately, given a face image, machines see this:
- 8. When it all started... 8 Everything started with Bledsoe in 1966: “This recognition problem is made difficult by the great variability in head rotation and tilt, lighting intensity and angle, facial expression, aging, etc. Some other attempts at facial recognition by machine have allowed for little or no variability in these quantities. Yet the method of correlation (or pattern matching) of unprocessed optical data, which is often used by some researchers, is certain to fail in cases where the variability is great. In particular, the correlation is very low between two pictures of the same person with two different head rotations.”
- 9. The face image formation process is a complex, entangled interaction F of: ➢ Nuisances: ○ Pose changes ○ Illumination variations ○ Deformations caused by Expression ○ Texture and Shape changes caused by Aging ➢ Identity of the subject Face recognition needs to reverse-engineer the image formation process: given I, it needs to restore , without being “fooled” by other confounding factors present implicitly in the images (pose, illumination, expression, age). PIE-A:Pose, Illumination, Expression, Age 9 Pose Illumination Expression
- 10. Why it is so important 10 ➢ Security ➢ Video-surveillance Applications ➢ Entertainment ➢ Smart shopping
- 11. Importance of Biometrics 11 Iris - Constrained Fingerprint - Constrained Face - Partially Unconstrained Body - Totally Unconstrained Faces in the wild - Totally Unconstrained
- 12. Security / Video Surveillance - Facts: 12 ➢ London's subway on July 7, 2005: It took thousands of investigators weeks to parse the city's CCTV footage after the attacks. (CNN, April 27, 2013) ➢ Boston Marathon on April 15, 2013: Investigators sifted through hundreds of hours of video, looking for people “doing things that are different from what everybody else is doing”. The work was painstaking and mind-numbing: One agent watched the same segment of video 400 times. The technology came up empty even though both suspect’s images exist in official databases. (The Washington Post, April 20, 2013) ➢ How many CCTV Cameras are there globally? According to IHS, there were 245 million professionally installed video surveillance cameras active and operational globally in 2014. ➢ CCTV cameras on Britain's roads capture 26 million images every day. (The Guardian, Jan 23, 2014 ) FBI, photo illustration by Sean Gallagher drivers license photo student ID photo images published by the FBI of the bomber
- 15. Scenarios and Modalities: 15 ➢ 1:N Face Identification (Probe vs Gallery -- Probe vs Watchlist) ○ Closed-Set (all probes have a mate in the gallery/watchlist) ○ Open-Set (a probe may not have a mate in the gallery/watchlist) ➢ 1:1 Face Verification ➢ Video-based Face Recognition ➢ Face Clustering Face Verification Face Identification probe gallery
- 16. Evaluations and Metrics 16 ➢ 1:N Face Identification ○ Closed-Set: CMC (Cumulative Matching Characteristic) ○ Open-Set: IET (Identification Error Tradeoff Curve) ➢ 1:1 Face Verification: ROC (Receiver Operating Characteristic)
- 17. Closed-Set Identification (1:N) 17 ➢ Cumulative Match Characteristic (CMC) curve: ○ summarizes the overall performance by reporting recall over a range of cutoff points ○ represents the expectation of finding the correct match in the top r matches, where r is the rank considered in the final ranking result ○ Formally, the CMC can be defined as follows: where ● i ∈ [1...N] are the probes to match ● Gi represents the position of the probe i in the gallery set sorted in descending order with respect to the scores
- 18. Closed-Set Identification (1:N) 18 ➢ Normalized Area Under the Curve (nAUC): ○ It gives an overall score of how well a method performs over all ranks ○ calculated as the total area under a CMC divided by 100 × IDs, where IDs is the total number of gallery individuals. ○ Formally, the nAUC can be defined as follows: nAUC
- 19. Open-Set Identification (1:N) 19 IET curve: Identification Error Trade-Off plots the: ➢ DIR (Detection and Identification Rate) vs. ➢ FAR (False Alarm Rate) Total probes in Gallery Total probes NOT in Gallery IET is a function of two parameters (the score, and the rank). To plot a curve it is necessary to fix one of them. Usually IET is reported at rank=20.
- 20. Verification (1:1) 20 ➢ Receiver Operating Characteristic: result annotations 0 1 0 0 0 0.2 0.9 0.1 0.7 0.3 False Positive Rate True Positive Rate 0,0 1,1 Total Population = 5
- 21. Verification (1:1) 21 Total Population = 5 result annotations 0 1 0 0 0 0.2 0.9 0.1 0.7 0.3 False Positive Rate True Positive Rate 0,0 1,1 result annotations 0 1 0 0 0 1 1 1 1 1 result annotations 0 1 0 0 0 0 0 0 0 0 result annotations 0 1 0 0 0 0 1 0 1 0 score>0 then 1 else 0 A B C score>1 then 1 else 0 score>0.6 then 1 else 0 ● TP=1/1 (FN=0/1) ● FP=4/4 (TN=0/4) ● TP=0/1 (FN=1/1) ● FP=0/4 (TN=4/4) ● TP=1/1 (FN=0/1) ● FP=1/4 (TN=3/4) C B A 1,0.25 Positive Negative 1 4
- 22. Generic Pipeline 22 [from OpenBR tutorial] ROC CMC IET Evals
- 23. From constrained to unconstrained... 23 Faces in the lab Subjects comply with recognition system FR Vendor Test (FRVT) 2006 (Celebrity) Faces in the wild Subjects comply with the photographer, but not with recognition system Labeled Faces in the Wild (LFW) 2008 Faces in videos In any one frame, subjects comply with neither the photographer nor the recognition system YouTube Faces (YTF) 2011
- 24. From constrained to unconstrained... 24 Labeled Face in the Wild (LFW) L. Wolf, T. Hassner, and Y. Taigman. Descriptor based methods in the wild. ECCV, 2008
- 25. Convolutional Neural Networks 25 Keys: Huge training dataset & complex network structure DeepFace [Taihman et al. 2014] FaceNet [Schroff et al. 2015] 4.4M training images 200M training images
- 26. Convolutional Neural Networks 26 Verification results of CNNs-based systems on LFW. Performance started to approach saturation easily (99%).
- 27. Convolutional Neural Networks 27 In-the-wild data Face recognition system
- 28. Is Face Recognition solved? 28 Deep learning approaches
- 29. Is Face Recognition solved? 29 In-the-wild data Face recognition system
- 30. IARPA Janus Benchmarks A 30 [from Klare et al. CVPR’15]
- 31. IARPA Janus Benchmarks A 31 IJB-A brings up the question: “How do we build a model of a subject given multiple heterogeneous media?” [from Klare et al. CVPR’15] For the first time a subject’s set contains: ➢ Still images ➢ Frames from multiple videos
- 32. IARPA Janus Benchmarks A 32 Probe templates (query) Gallery templates (prior) Same person?
- 33. 33 IARPA Janus Benchmarks A Far wider pose distribution LFW Janus
- 34. 34 Up Next: How to build a face recognition pipeline?
- 35. 35 References 1. W. W. Bledsoe, “The model method in facial recognition,” Panoramic Research Inc., Palo Alto, CA, Rep. PR1, vol. 15, no. 47, p. 2, 1966. 2. W. Bledsoe, “Man-machine facial recognition: Report on a large-scale experiment, panoramic research,” Inc, Palo Alto, CA, 1966 3. L. Wolf, T. Hassner, and Y. Taigman. Descriptor based methods in the wild. ECCV, 2008 4. B. F. Klare, B. Klein, E. Taborsky, A. Blanton, J. Cheney, K. Allen, P. Grother, A. Mah, M. Burge, and A. K. Jain, “Pushing the frontiers of unconstrained face detection and recognition: IARPA Janus Benchmark-A,” in CVPR, 2015 5. P. J. Phillips, P. Grother, and R. Micheals, “Evaluation methods in face recognition,” in Handbook of Face Recognition. Springer, 2011, pp. 551–574 6. OpenBR project - http://openbiometrics.org/ 7. G. B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller, “Labeled faces in the wild: A database for studying face recognition in unconstrained environments,” UMass, Amherst, Tech. Rep. 07-49, October 2007 8. L. Wolf, T. Hassner, and I. Maoz, “Face recognition in unconstrained videos with matched background similarity,” in CVPR, 2011 9. F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in CVPR, 2015 10. Y. Taigman, M. Yang, M. Ranzato, and L. Wolf, “Deepface: Closing the gap to human-level performance in face verification,” in CVPR, 2014