Face Recognition: From Scratch To Hatch / Эдуард Тянтов (Mail.ru Group)
Download as PPTX, PDF1 like2,216 views
The document discusses advancements in face recognition technology, detailing methods such as the Viola-Jones algorithm and deep learning-based approaches like MTCNN and Faster R-CNN, highlighting their performance metrics. It emphasizes the importance of data quality and diversity for effective model training, particularly in overcoming biases in recognition accuracy. Furthermore, it introduces various techniques for optimizing inference speed and enhancing feature discriminability, such as TensorRT and metric learning with center loss.
Face Recognition: From Scratch To Hatch / Эдуард Тянтов (Mail.ru Group)
- 1. Face Recognition: From Scratch To Hatch Tyantov Eduard, Mail.ru Group
- 2. Face Recognition in Cloud@Mail.ru Users upload photos to Cloud Backend identifies persons on photos, tags and show clusters
- 5. edges object parts (combination of edges) object models
- 8. Auxiliary task: facial landmarks – Face alignment: rotation – Goal: make it easier for Face Recognition Rotate
- 9. Train Datasets Wider – 32k images – 494k faces Celeba – 200k images, 10k persons – Landmarks, 40 binary attributes
- 10. Test Dataset: FDDB Face Detection Data Set and Benchmark – 2845 images – 5171 faces
- 11. Old school: Viola-Jones Haar Feature-based Cascade Classifiers Haar-like features eyes darker nose lighter Examples
- 12. Viola-Jones algorithm: training for each patch AdaBoost ensemble features 160k valuable 6k weighted sum Face or NotDataset
- 13. Viola-Jones algorithm: inference for each patch Stages Face Yes Yes Stage 1 Stage 2 Stage N Optimization – Features are grouped into stages – If a patch fails any stage => discard
- 14. Viola-Jones results OpenCV implementation – Fast: ~100ms on CPU FDDB results 0.45 – Not accurate
- 15. 1. Pre-trained network: extracting features New school: Region-based Convolutional Networks Faster RCNN, algorithm 1 CNN 2 RPN 4 Classifier Face ? 3 Roi-pooling Feature Maps 3 2. Region proposal network3. RoI-pooling: extract corresponding tensor4. Classifier: classes and the bounding box
- 16. Comparison: Viola-Jones vs R-FCN Results – 92% accuracy (R-FCN) FDDB results Viola-Jones (opencv) 0.45 HOG (dlib) 0.7 R-FCN 0.92 – 40ms on GPU (slow)
- 17. Face detection: how fast We need faster solution at the same accuracy! Target: < 10ms
- 18. Alternative: MTCNN 1 Different scales 2 Proposal CNN 3 Refine CNN 4 Output CNN Cascade of 3 CNN 1. Resize to different scales2. Proposal -> candidates + b-boxes 3. Refine -> calibration4. Output -> b-boxes + landmarks
- 19. Comparison: MTCNN vs R-FCN MTCNN + Faster + Landmarks - Less accurate - No batch processing Model GPU Inference FDDB Precision (100 errors) R-FCN 40 ms 92% MTCNN 17 ms 90%
- 20. TensorRT
- 21. What is TensorRT NVIDIA TensorRT is a high-performance deep learning inference optimizer Features – Improves performance for complex networks – FP16 & INT8 support – Effective at small batch-sizes
- 22. TensorRT: layer optimizations 2. Horizontal fusion 3. Concat elision 1. Vertical layer fusion
- 23. TensorRT: downsides 1. Caffe + TensorFlow supported 2. Fixed input/batch size 3. Basic layers support
- 24. Batch processing Problem Image size is fixed, but MTCNN works at different scales Solution Pyramid on a single image
- 25. Batch processing Results – Single run – Enables batch processing Model Inference ms MTCNN (Caffe, python) 17 MTCNN (Caffe, C++) 12.7 + batch 10.7
- 26. TensorRT: layers Problem No PReLU layer => default pre-trained model can’t be used Retrained with ReLU from scratch Model GPU Inference ms FDDB Precision (100 errors) MTCNN, batch 10.7 90% +Tensor RT 8.8 91.2% -20%
- 27. Face detection: inference Target: < 10 ms Result: 8.8 ms Ingredients 1. MTCNN 2. Batch processing 3. TensorRT
- 28. Face Recognition
- 29. Face recognition task – Goal – to compare faces Latent SpaceCNN Embedding close distant Unseen – How? To learn metric – To enable Zero-shot learning
- 30. Training set: MSCeleb – Top 100k celebrities – 10 Million images, 100 per person – Noisy: constructed by leveraging public search engines
- 31. Small test dataset: LFW Labeled Faces in the Wild Home – 13k images from the web – 1680 persons have >= 2 photos
- 32. Large test dataset: Megaface – Identification under up to 1 million “distractors” – 530 people to find
- 34. Metric Learning
- 35. Classification CNN Embedding Classify – Train CNN to predict classes – Pray for a good latent space close distant
- 36. Softmax – Learned features only separable but not discriminative – The resulting features are not sufficiently effective close
- 37. We need metric learning – Tightness of the cluster – Discriminative features
- 38. Triplet loss Features – Identity -> single point – Enforces a margin between persons Anchor Positive Negativepositive + α < negative minimize maximize
- 39. Choosing triplets Crucial problem How to choose triplets ? Useful triplets = hardest errors Pick all positive Too easy Hard enough Solution Hard-mining within a large mini-batch (>1000)
- 41. Choosing triplets: trap Anchor Positive positive ~ negative minimize maximize Negative
- 43. Choosing triplets: trap Selecting hardest negative may lead to the collapse early in training
- 44. Choosing triplets: semi-hard Pick all positive Too easy Semi-hard Too hard positive < negative < positive + α
- 45. Triplet loss: summary Overview – Requires large batches, margin tuning – Slow convergence Opensource Code – Openface (Torch) • suboptimal implementation – Facenet, not original (TensorFlow) LFW, % Megaface Openface (Torch) 92 - Our (Torch) 99.35 65 Google’s Facenet 99.63 70.5
- 46. Center loss Idea: pull the points to class centroids
- 47. Center loss: structure – Without classification loss – collapses CNN Embedding Classify Softmax Loss λ Center Loss Pull – Final loss = Softmax loss + λ Center loss
- 48. Center Loss: different lambdas λ = 10-7
- 49. Center Loss: different lambdas λ = 10-6
- 50. Center Loss: different lambdas λ = 10-5
- 51. Center loss: summary Overview – Intra-class compactness and inter-class separability – Good performance at several other tasks Opensource Code – Caffe (original, Megaface - 65%) LFW, % Megaface Triplet Loss 99.35 65 Center Loss (Torch, ours) 99.60 71.7
- 52. Tricks: augmentation Test time augmentation – Flip image Embedding Flipped Embedding Final Embedding Average – Average embeddings – Compute 2 embeddings
- 53. Tricks: alignment Rotation Kabsch algorithm - the optimal rotation matrix that minimizes the RMSD LFW, % Megaface Center Loss 99.6 71.7 Center Loss + Tricks 99.68 73
- 54. Shades on At one point we used shades augmentation How to – Construct several sunglass textures – Place them using landmarks
- 55. Tricks: adding Eye loss CNN Embedding Person Softmax Loss Center Loss – We can force CNN to learn specific discriminative features – For celebrities eye colors are available in the Internet Eye Loss Eye Color
- 56. Eye loss: summary *Adding simple features doesn’t help, i.e gender LFW, % Megaface Center Loss + Tricks 99.68 73 Center Loss + Eye 99.68 73.5
- 57. Angular Softmax ||X||= 1 On sphere Angle discriminates ||W||= 1 b=0
- 59. Angular Softmax: different «m» m=1 m=3
- 60. Angular softmax: summary Overview – As describes in the paper: doesn’t work at all – Works using sum of losses (m=1,N) over training • only on small datasets! LFW, % Megaface Center Loss 99.6 73 Center Loss + Eye 99.68 73.5 A-Softmax (Torch) 99.68 74.2 Opensource Code – Caffe (original) – Slight modification of the loss yields 74.2%
- 61. Metric learning: summary Softmax < Triplet < Center < A-Softmax A-Softmax – With bells and whistles better than center loss Center loss Overall – Rule of thumb: use Center loss – Metric learning may improve classification performance
- 62. Fighting errors
- 63. Errors after MSCeleb: children Problem Children all look alike Result Embeddings are almost single point in the space
- 64. Errors after MSCeleb: asian Problem Face Recognition’s intolerant to Asians Reason Dataset doesn’t contain enough photos of these categories
- 65. How to fix these errors ? It’s all about data, we need diverse dataset! Natural choice – avatars of social networks
- 66. A way to construct dataset Face Detection Pick largest Face Recognition+ Clustering Cleaning algorithm 1. Face detection2. Face recognition -> embeddings3. Hierarchical clustering algorithm4. Pick the largest cluster as a personIterate after each model improvement
- 67. MSCeleb dataset’s errors MSCeleb is constructed by leveraging search engines Joe Eszterhas Joe Eszterhas and Mel Gibson public confrontation leads to the error Mel Gibson =
- 70. MSCeleb dataset’s errors Dataset has been shrinked from 100k to 46k celebrities Random search engine Corrected
- 71. Results on new datasets Datasets – Train: • MSCeleb (46k) • VK-train (150k) A-Softmax on dataset Megaface MSCeleb 74.2 MSCeleb cleaned 76.2 – Test • MegaVK • Sets for children and asians MegaVK 58.4 60 + VK 77.5 87.5
- 72. Ensemble Final Embedding Concat CNN-2 CNN-1 A-Softmax model Megaface MegaVK Best single 77.5 87.5 Ensemble of 2 79.6 88.6
- 73. Workaround still … Children are still challenge for the model
- 74. Workaround Algorithm 1. Construct dataset with children 2. Compute average embedding 3. Every point inside the sphere – a child 4. Tighten distance threshold there Results This allows softening the overall threshold
- 75. How to handle big dataset It seems we can add more data infinitely, but no. Problems – Memory consumption (Softmax) – Computational costs – A lot of noise in gradients
- 76. Softmax Approximation Algorithm 1. Perform K-Means clustering using current FR model Dataset K-Means Children Women Men Smaller sets
- 77. Softmax Approximation Algorithm 1. Perform K-Means clustering using current FR model CNN Embedding Predict cluster Predict person MenPerson Softmax 2. Two Softmax heads: 1. Predicts cluster label 2. Class within the true cluster Cluster Softmax Men
- 78. Softmax Approximation Pros Push Push Harder negative 1. Prevents fusing of the clusters 2. Does hard-negative mining 3. Clusters can be specified • Children • Asian
- 79. Fighting errors on production
- 80. Errors: blur Problem • Detector yields blurry photos • Recognition forms «blurry clusters» Solution Laplacian – 2nd order derivative of the image
- 82. Errors: body parts Detection mistakes form clusters
- 83. Errors: diagrams & mushrooms
- 84. Fixing trash clusters There is similarity between “no faces”! CNN Embedding Embedding No «firing» features
- 85. Fixing trash clusters «Trash» has small norm Faces Trash Softmax loss Motivation Softmax encourage big embedding’s norms Results – ROC AUC 97% – Better then Laplacian for blurry
- 87. Fun: new governors Recently appointed governors are almost twins, but FR distinguishes them Dmitriy Gleb
- 88. Over years Face recognition algorithm captures similarity across years Although we didn’t focus on the problem
- 89. Over years
- 90. Summary 1. Use TensorRT to speed up inference 2. Metric learning: use Center loss by default 3. Clean your data thoroughly 4. Understanding CNN helps to fight errors
- 92. Auxiliary
- 93. Best avatar Problem How to pick an avatar for a person ? Solution Train model to predict awesomeness of photo
- 94. Predicting awesomeness: how to approach Social networks – not only photos, but likes too
- 95. Predicting awesomeness: dataset Awesomeness (A) = likes/audience A=18% A=27% A=75%
- 96. Results – Mean Aveage Precision @5: 25% – Data and metric are noisy => human evaluation Predicting awesomeness: summary High score Low score
- 97. Predicting awesomeness: incorporating into FR One more branch in Face Recognition CNN Small overhead awesomeness embedding face
- 98. Appendix
- 99. Histogram loss Idea – Compute similarities positive and negative pairs – Maximize a probability that a randomly sampled positive pair has smaller similarity than negative one Loss = the integral of the product between the negative distribution and the cumulative density function for the positive distribution (shown with a dashed line)
- 100. Histogram loss Results – Got nice distributions – Doesn’t improve the triplet results – LFW on 97.7%
Editor's Notes
- #8: «Detect all faces on the image at all scales and rotations»
- #10: http://vis-www.cs.umass.edu/fddb/
- #11: http://vis-www.cs.umass.edu/fddb/
- #12: https://docs.opencv.org/master/d7/d8b/tutorial_py_face_detection.html , http://blog.dlib.net/2014/02/dlib-186-released-make-your-own-object.html
- #16: R-FCN: https://arxiv.org/pdf/1605.06409.pdf , Faster: https://arxiv.org/pdf/1506.01497v3.pdf
- #17: «Another approach – HOG features (dlib)» «not far from state-of-the-art (Google reports 94%)»
- #19: Offtop: обучается первая сеть на 12x12, вторая – 24x24, 3 – 36x36 – hard negative mining для 2 и 3 для исправления ошибок выдает первая 5x1x1, а на пирамиде тензор, дальше составляется батч для 2 и 3 (батч над батчами не делали) https://kpzhang93.github.io/MTCNN_face_detection_alignment/paper/spl.pdf
- #20: «Open source Pre-trained model (Caffe) No original training code «
- #24: «but not Torch/PyTorch We’ve retrained the CNN because of PreLu absense» basic- actively devolping
- #25: TODO: возможно показать что мы не подряд засовываем в сеть, а сразу в кучу, хз не перегруз ли
- #26: табличка в одном месте
- #31: https://www.microsoft.com/en-us/research/project/ms-celeb-1m-challenge-recognizing-one-million-celebrities-real-world/
- #33: мб добавить
- #34: Top results ~ 80% Best – 91.7%
- #38: «All faces of person are project to a single point»
- #39: Chad Smith
- #45: TODO: про альфа лишнее ?
- #48: https://ydwen.github.io/papers/WenECCV16.pdf, 65% из статьи
- #53: TODO: убрать текст ?
- #54: (Better than dlib)
- #55: TODO: не выпилить но доделать нормально фотка чувака точки + очки
- #57: «пол например не является discriminative – пробовали» 71.5% -> 75.5% или +2 71.5 vs 73, 75,5 из-за датасета
- #58: СЛОВА «сказать что норма – это деление на сумму»
- #59: To discriminate better -> tighten clusters
- #61: http://proceedings.mlr.press/v48/liud16.pdf
- #62: http://proceedings.mlr.press/v48/liud16.pdf
- #65: “Рассказать про идею созданую по мотивам корпоратиа”
- #66: VK – easy API OK – hard access to API, rate-limits Facebook – just closed to mining
- #67: Cleaning algorithm Face Detection Face Recognition to get embeddings Hierarchical clustering algorithm with Soften the distance threshold Pick the largest cluster as a person iterate after each model improvement
- #68: this leads to improved perf on megaface = 75%?
- #75: цифра лучше ? чуть меньше текста или на 2 слайда разбить TODO: мы обнаружили что дети компанкто расположены
- #77: https://arxiv.org/pdf/1609.04309.pdf - paper: “Efficient softmax approximation for GPUs”
- #79: «какой профит помимо решения проблемы»
- #81: https://docs.opencv.org/3.0-beta/doc/py_tutorials/py_imgproc/py_gradients/py_gradients.html It’s a face, but without enough facial information
- #83: “If the detector makes a mistake, recognizer gone crazy. Detection errors often form a cluster.”
- #85: «софтмакс стремится увеличить скалярное произведение»
- #86: maximizing scalar product <w_i, x> for i class We normalize embeddings only for distance
- #89: “Social networks contains avatars over users’ lives”
- #95: Data we know How many likes Who likes
- #97: http://people.cs.uchicago.edu/~larsson/fractalnet/overview.png
- #98: block 6 нарисовать
- #100: https://arxiv.org/pdf/1611.00822.pdf