![Our Facial Model
Nice to use certain
features, but how do
we get them?
• Face tracking, based
on a system
developed by Tao and
Huang [CVPR98],
subsequently used by
Cohen, Sebe et al
[ICPR02]
• First, landmark facial
features (e.g., eye
corners) are selected
interactively](https://image.slidesharecdn.com/automaticfacialemotionrecognition-241014171507-a692b9eb/85/Automatic-Facial-Emotion-Recognition-ppt-11-320.jpg)
![The Classifiers - TAN
• TAN models are such a subclass
• Advantage: There exist an efficient
algorithm [Chow-Liu] to compute the
optimal TAN model](https://image.slidesharecdn.com/automaticfacialemotionrecognition-241014171507-a692b9eb/85/Automatic-Facial-Emotion-Recognition-ppt-26-320.jpg)
Automatic Facial Emotion Recognition.ppt
- 1. Automatic Facial Emotion Recognition Aitor Azcarate Felix Hageloh Koen van de Sande Roberto Valenti Supervisor: Nicu Sebe
- 2. Overview INTRODUCTION RELATED WORK EMOTION RECOGNITION CLASSIFICATION VISUALIZATION FACE DETECTOR DEMO EVALUATION FUTURE WORKS CONCLUSION QUESTIONS
- 3. Emotions Emotions are reflected in voice, hand and body gestures, and mainly through facial expressions
- 4. Emotions (2) Why is it important to recognize emotions? • Human beings express emotions in day to day interactions • Understanding emotions and knowing how to react to people’s expressions greatly enriches the interaction
- 5. Human-Computer interaction • Knowing the user emotion, the system can adapt to the user • Sensing (and responding appropriately!) to the user’s emotional state will be perceived as more natural, persuasive, and trusting • We only focus on emotion recognition…
- 6. Related work Cross-cultural research by Ekman shows that some emotional expressions are universal: • Happiness • Sadness • Anger • Fear • Disgust (maybe) • Surprise (maybe) Other emotional expressions are culturally variable.
- 7. Related work (2) Ekman developed the Facial Action Coding System (FACS): Description of facial muscles and jaw/tongue derived from analysis of facial anatomy
- 8. Facial Expression Recognition • Pantic & Rothkrantz in PAMI 2000 performed a survey of the field • Recognize a generic procedure amongst all systems: • Extract features (provided by a tracking system, for example) • Feed the features into a classifier • Classify to one of the pre-selected emotion categories (6 universal emotions, or 6+neutral, or 4+neutral, etc)
- 9. Field overview: Extracting features Systems have a model of the face and update the model using video frames: • Wavelets • Dual-view point-based model • Optical flow • Surface patches in Bezier volumes • Many, many more From these models, features are extracted.
- 10. Facial features We use features similar to Ekmans: • Displacement vectors of facial features • Roughly corresponds to facial movement (more exact description soon)
- 11. Our Facial Model Nice to use certain features, but how do we get them? • Face tracking, based on a system developed by Tao and Huang [CVPR98], subsequently used by Cohen, Sebe et al [ICPR02] • First, landmark facial features (e.g., eye corners) are selected interactively
- 12. Our Facial Model (2) • A generic face model is then warped to fit the selected facial features • The face model consists of 16 surface patches embedded in Bezier volumes
- 13. Face tracking • 2D image motions are measured using template matching between frames at different resolutions • 3D motion can be estimated from the 2D motions of many points of the mesh • The recovered motions are represented in terms of magnitudes of facial features
- 14. Related work: Classifiers • People have used the whole range of classifiers available on their set of features (rule-based, Bayesian networks, Neural networks, HMM, NB, k-Nearest Neighbour, etc). • See Pantic & Rothkrantz for an overview of their performance. • Boils down to: there is little training data available, so if you need to estimate many parameters for your classifier, you can get in trouble.
- 15. Overview INTRODUCTION RELATED WORK EMOTION RECOGNITION CLASSIFICATION VISUALIZATION FACE DETECTOR DEMO EVALUATION FUTURE WORKS CONCLUSION QUESTIONS
- 16. Classification – General Structure Java Server Classifier Visualization Video Tracker (C++) x1 x2 . . xn Feature Vector
- 17. Classification - Basics • We would like to assign a class label c to an observed feature vector X with n dimensions (features). • The optimal classification rule under the maximum likelihood (ML) is given as:
- 18. Classification - Basics • Our feature vector has 12 features • Classifier identifies 7 basic emotions: • Happiness • Sadness • Anger • Fear • Disgust • Surprise • No emotion (neutral)
- 19. The Classifiers • Naïve Bayes • Implemented ourselves • TAN • Used existing code We compared two different classifiers for emotion detection
- 20. The Classifiers - Naïve Bayes • Well known classification method • Easy to implement • Known to give surprisingly good results • Simplicity stems from the independence assumption
- 21. The Classifiers - Naïve Bayes • In a naïve Bayes model we assume the features to be independent • Thus the conditional probability of X given a class label c is defined as
- 22. The Classifiers - Naïve Bayes • Conditional probabilities are modeled with a Gaussian distribution • For each feature we need to estimate: • Mean: • Variance: N i i N x 1 1 N i i N x 1 2 1 2 ) (
- 23. The Classifiers - Naïve Bayes • Problems with Naïve Bayes: • Independence assumption is weak • Intuitively we can expect that there are dependencies among features in facial expressions • We should try to model these dependencies
- 24. The Classifiers - TAN • Tree-Augmented-Naive Bayes • Subclass of Bayesian network classifiers • Bayesian networks are an easy and intuitive way to model joint distributions • (Naïve Bayes is actually a special case of Bayesian networks)
- 25. The Classifiers - TAN • The structure of the Baysian Network is crucial for classification • Ideally it should be learned from the data set using ML • But searching through all possible dependencies is NP-Complete • We should restrict ourselves to a subclass of possible structures
- 26. The Classifiers - TAN • TAN models are such a subclass • Advantage: There exist an efficient algorithm [Chow-Liu] to compute the optimal TAN model
- 27. The Classifiers - TAN • Structure: • The class node has no parents • Each feature has as parent the class node • Each feature has as parent at most one other feature
- 28. The Classifiers - TAN
- 29. Visualization • Classification results are visualized in two different ways • Bar Diagram • Circle Diagram • Both implemented in java
- 30. Visualization – Bar Diagram
- 31. Visualization – Circle Diagram
- 32. Overview INTRODUCTION RELATED WORK EMOTION RECOGNITION CLASSIFICATION VISUALIZATION FACE DETECTOR DEMO EVALUATION FUTURE WORKS CONCLUSION QUESTIONS
- 33. Landmarks and fitted model
- 34. Problems • Mask fitting • Scale independent • Initialization “in place” • Fitted Model • Reinitialize the mesh in the correct position when it gets lost Solution? FACE DETECTOR
- 35. New Implementation Movie DB OpenGL converter Capture Module Face Detector Face Fitting Send data to classifier Lost? Repositioning yes no Classify and visualize results Solid mask
- 36. Face Detector • Looking for a fast and reliable one • Using the one proposed by Viola and Jones • Three main contributions: • Integral Images • Adaboost • Classifiers in a cascade structure • Uses Haar-Like features to recognize objects
- 37. Face Detector – “Haar-Like” features
- 38. Face Detector – Integral Images • A = 1 • B = 2-1 • C = 3-1 • D = 4-A-B-C • D = 4+1-(2+3)
- 39. Face Detector - Adaboost Results of the first two Adaboost Iterations This means: • Those features appear in all the data • Most important feature: eyes
- 40. Face Detector - Cascade All Sub-windows T T T Reject Sub-window F F F F 1 2 3 4
- 41. Demo
- 42. Overview INTRODUCTION RELATED WORK EMOTION RECOGNITION CLASSIFICATION VISUALIZATION FACE DETECTOR DEMO EVALUATION FUTURE WORKS CONCLUSION QUESTIONS
- 43. Evaluation • Person independent • Used two classifiers: Naïve Bayes and TAN. • All data divided into three sets. Then two parts are used for training and the other part for testing. So you get 3 different test and training sets. • The training set for person independent tests contains samples from several people displaying all seven emotions. For testing a disjoint set with samples from other people is used.
- 46. Evaluation • Person dependent • Also used two classifiers: Naïve Bayes and TAN • All the data from one person is taken and divided into three parts. Again two parts are used for training and one for testing. • Training is done for 5 people and is then averaged.
- 47. Evaluation •Person dependent •Results Naïve Bayes:
- 49. Evaluation • Conclusions: • Naïve Bayes works better than TAN (indep: 64,3 – 53,8 and dep: 93,2 – 62,1). • Sebe et al had more horizontal dependencies while we got more vertical dependencies. • Implementation of TAN has probably a bug. • Results of Sebe et al were: TAN: dep 83,3 indep 65,1 NB is similar to ours.
- 50. Future Work • Handle partial occlusions better. • Make it more robust (lighting conditions etc.) • More person independent (fit mask automatically). • Use other classifiers (dynamics). • Apply emotion recognition in applications. For example games.
- 51. Conclusions • Our implementation is faster (due to server connection) • Can get input from different camera’s • Changed code to be more efficient • We have visualizations • Use face detection • Mask loading and recovery
- 52. Questions ?