Using nasal curves matching for expression robust 3D nose recognition
- 1. Using Nasal Curves Matching for Expression Robust 3D Nose Recognition Authors: Mehryar Emambakhsha , Adrian N Evansa and Melvyn Smithb a University of Bath, UK b University of the West of England, UK
- 2. Outline • Introduction and background • Nasal Curves Matching • Preprocessing • Landmarking and curves finding • Feature selection • Experimental results • Classification algorithms results • FRGC v2.0 Experiment III results • Conclusion • Summary • Future work
- 3. Outline • Introduction and background • Nasal Curves Matching • Preprocessing • Landmarking and curves finding • Feature selection • Experimental results • Classification algorithms results • FRGC v2.0 Experiment III results • Conclusion • Summary • Future work
- 4. Introduction and background • Why the nose region? • Easy to detect • Hair occlusion robustness • Difficult to conceal • Relatively expression invariant .... … but not all parts!
- 5. Introduction and background ICP one-to-one matching on multiple nasal regions K. Chang, W. Bowyer, and P. Flynn. Multiple nose region matching for 3D face recognition under varying facial expression. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(10):1695–1700, 2006.
- 6. ICP one-to-one matching on the cropped nasal region using curvature maps H. Dibeklio lu, B. Gökberk, and L. Akarun. Nasal region based 3D face recognition under pose and expressionǧ variations. In Proceedings of the Third International Conference on Advances in Biometrics, pages 309–318, 2009. Introduction and background
- 7. Distances between the geodesic contours H. Drira, , B. Amor, M. Daoudi, and A. Srivastava. Nasal region contribution in 3D face biometrics using shape analysis framework. In Proceedings of the Third International Conference on Advances in Biometrics, pages 357–366, 2009. Introduction and background
- 8. Outline • Introduction and background • Nasal Curves Matching • Preprocessing • Landmarking and curves finding • Feature selection • Experimental results • Classification algorithms results • FRGC v2.0 Experiment III results • Conclusion • Summary • Future work
- 9. Why curves are better than direct 3D surface? • Lower complexity than direct 3D • Potentially more robust against expression variation and spike noise • Can correspond to more invariant regions Preprocessing Feature selection Landmarking and curves finding Raw 3D data Expression robust curves Nasal curves matching
- 10. Outline • Introduction and background • Nasal Curves Matching • Preprocessing • Landmarking and curves finding • Feature selection • Experimental results • Classification algorithms results • FRGC v2.0 Experiment III results • Conclusion • Summary • Future work
- 11. Preprocessing Resampling and denoising • Resampling to remove noise from the X and Y maps • Denoising: • Holes conditional morphological filling • Missing data Cubic interpolation • High frequency and spike noise Median filtering with 2.31mm 2.31mm maskᵡ Nasal curves matching
- 12. Nasal curves matching Facial region cropping and alignment • Sphere (r=80mm) centred on the nose tip, intersected with the face surface • PCA iteratively applied to the cropped face • Depth map resampled in every iteration • The nose tip is iteratively re-detected using the shape index thresholding A. Mian, M. Bennamoun, and R. Owens. An efficient multimodal 2D-3D hybrid approach to automatic face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(11):1927–1943, 2007. Preprocessing
- 13. Nasal region cropping • Three cylinders are intersected with the face surface • Convex hull of AND found Preprocessing AND Convex hull Cropped nose Cylinders’ intersections with the face surface Nasal curves matching
- 14. Outline • Introduction and background • Nasal Curves Matching • Preprocessing • Landmarking and curves finding • Feature selection • Experimental results • Classification algorithms results • FRGC v2.0 Experiment III results • Conclusion • Summary • Future work
- 15. Landmarking and curves finding 16 landmarks are detected on the nasal region Nasal curves matching
- 16. • Detecting L9 (nose tip) • The filled nose region is multiplied by thresholded shape index • The result is dilated • Point with the highest depth value is detected as L9 M. Emambakhsh and A. Evans. Self-dependent 3D face rotational alignment using the nose region. In 4th IET International Conference on Imaging for Crime Detection and Prevention 2011 (ICDP 2011), pages 1–6, 2011. Landmarking and curves finding Filling Convex regions Multiplication Dilation Nasal curves matching
- 17. Landmarking and curves finding • Detecting L1 • Set of orthogonal planes, passing L9 are intersected with the nose surface • The intersections’ minima are detected (green curve) • The maximum of the minima is chosen as L9 Nasal curves matching
- 18. • Detecting L5 and L13 (alar) • Second set of orthogonal planes, passing L9 are intersected with nose surface • The intersections’ gradient is computed and its maxima are detected (green points) • Outliers are removed using K- means (K=2) clustering • Those whose x and y values are closest to L9’s are selected as L5 and L13 Landmarking and curves finding Nasal curves matching
- 19. • Other landmarks are selected by dividing the lines connecting L1 to L5, L5 to L9, L9 to L13 and L13 to L1 into 4 sections. Landmarking and curves finding Nasal curves matching
- 20. Nasal curves matching • Curves are found by intersecting planes with the nose surface, passing different pairs of the landmarks. • 75 curves are created. Landmarking and curves finding
- 21. Outline • Introduction and background • Nasal Curves Matching • Preprocessing • Landmarking and curves finding • Feature selection • Experimental results • Classification algorithms results • FRGC v2.0 Experiment III results • Conclusion • Summary • Future work
- 22. Nasal curves matching • Forward sequential feature selection (FSFS) is used to detect the most robust curves in different expressions. • Those curves which produce the highest rank-one recognition, based on the leave-one-out method are kept. • Genetic algorithm-based feature selection is also used. • The rank-one recognition rate is the cost function, while the binary weights are the parameters: Feature selection
- 23. Outline • Introduction and background • Nasal Curves Matching • Preprocessing • Landmarking and curves finding • Feature selection • Experimental results • Classification algorithms results • FRGC v2.0 Experiment III results • Conclusion • Summary • Future work
- 24. Experimental results • Datasets • Face Recognition Grand Challenge (FRGC) v2.0 • 557 unique subjects • Images obtained in three different time periods: Spring 2003, Fall 2003 and Spring 2004 • Bosphorus dataset • 105 unique subjects • Various expressions: neutral, happy, surprise, fear, sadness, anger and disgust • Bosphorus Feature selection • FRGC v2.0 Nose recognition evaluation
- 25. Experimental results • Landmarking accuracy and consistency • The nose tip is translated to the origin. • The Euclidean distance between different landmarks are computed per class. • The overall mean and standard deviation of the distances L1: 3.3620 ± 1.6798 mm L5: 2.3557 ± 1.1822 mm L13: 2.4259 ± 1.2691 mm C. Creusot, N. Pears, and J. Austin. A Machine-Learning Approach to Keypoint Detection and Landmarking on 3D Meshes. International Journal of Computer Vision, pages 1–34, 2013.
- 26. Experimental results • FSFS results • 28 curves produce the highest rank-one recognition rate. • These curves are the least sensitive ones to expression variation.
- 27. Experimental results • Comparison with GA’s result • FSFS outperforms GA in terms of convergence and computational cost. • FSFS selects fewer curves, with higher rank-one recognition rate.
- 28. Outline • Introduction and background • Nasal Curves Matching • Preprocessing • Landmarking and curves finding • Feature selection • Experimental results • Classification algorithms results • FRGC v2.0 Experiment III results • Conclusion • Summary • Future work
- 29. Experimental results • Subspace projection methods projection to a 277- dimensional space • Principal Component Analysis (PCA) • Linear Discriminant Analysis (LDA) • Kernel Fisher’s Analysis (KFA) polynomial (Poly) and fractional power polynomial (FPP) kernels are used • Multi-class Support Vector Machines (Multi-SVM) • One-vs.-all scenario for multi-classification • Linear kernels • Bootstrap aggregation decision trees (TreeBagger) • 119 trees are aggregated • Direct city-block (CB) distance 6 methods Classification algorithms
- 30. Experimental results • The number of training samples are varied per class. • KFA-Poly outperforms all the classification methods. Classification algorithms
- 31. Outline • Introduction and background • Nasal Curves Matching • Preprocessing • Landmarking and curves finding • Feature selection • Experimental results • Classification algorithms results • FRGC v2.0 Experiment III results • Conclusion • Summary • Future work
- 32. Experimental results • Spring 2003 Training • Fall 2003 + Spring 2004 Test FRGC v2.0 Experiment III results
- 33. Experimental results • Comparison of equal error rate (EER) for the best performing KFA-Poly curve K. Chang, W. Bowyer, and P. Flynn. Multiple nose region matching for 3D face recognition under varying facial expression. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(10):1695–1700, 2006. FRGC v2.0 Experiment III results
- 34. Experimental results • Rank-one recognition rate comparison FRGC v2.0 Experiment III results
- 35. Outline • Introduction and background • Nasal Curves Matching • Preprocessing • Landmarking and curves finding • Feature selection • Experimental results • Classification algorithms results • FRGC v2.0 Experiment III results • Conclusion • Summary • Future work
- 36. Conclusion: Summary • In this work the potential of the nasal curves for 3D nose recognition is evaluated. • A robust landmarking algorithm is proposed and curves are created by planes intersections with the nose surface. • The most invariant curves are selected by applying FSFS over the Bosphorus dataset. • Various classifiers are used on the feature space. • KFA-Poly generated the highest rank-one recognition rate and lowest EER.
- 37. Outline • Introduction and background • Nasal Curves Matching • Preprocessing • Landmarking and curves finding • Feature selection • Experimental results • Classification algorithms results • FRGC v2.0 Experiment III results • Conclusion • Summary • Future work
- 38. Conclusion: Future work • Curves extension to other facial parts • Application in pattern rejection • Robustness • Against partial/self- occlusion • Training-free matching • Other feature selection methods • Applying rank-level fusion techniques on the curves • Other feature spaces, not just depth! • Gabor-wavelets • Local binary patterns (LBP) • …
- 39. Thank you!
Editor's Notes
- #5: Some variation, so not all nose invariant …. Motivation for work, set of curves and select invariant ones?
- #7: Need this one..don’t think I’ve seen it. Used Bospherous
- #8: FRGC subset
- #10: Can use to select regions of nose more invarant
- #12: Resample eg rows, find cols with min std dev, used to calc gradient and then resample depth map. vv for cols Filling compared to orig and used if less then thresh. Preserves natural holes, eye corners, nostrils, mouth parts etc
- #13: Depth map resampled to 80x80?? Last 2 step improve on Mian et al.
- #14: And of the 3 intersetions?
- #19: 3D dist used in clustering, xy dist for selection. Red crosses are new ones? Could you replot with red dots? Ref to paper that this was improvement on.
- #26: Compares favourably with ref but no learning.
- #27: 50 pts per curve used for all curves
- #30: I’ll probably need a bit of background on these?
- #31: FRGC merged folders using differt traning samples. Subjects removed if <n. KFA poly best
- #33: Split into neutral and varying. KFA poly still best! Neural better than varying
- #34: Ref Chang’s paper. Mention difference between text and fig?
- #40: Could rework to focus on the things we’re working on – extending curves across face, better feature selection => higher recognition performance