J01116164
1 like368 views
This document provides a literature review of various techniques for automatic facial expression recognition. It discusses approaches such as principal component analysis (PCA), linear discriminant analysis (LDA), independent component analysis (ICA), 2D PCA, global eigen approaches using color images, subpattern extended 2D PCA, multilinear image analysis, color subspace LDA, 2D Gabor filter banks, and local Gabor binary patterns. It provides a table comparing the performance and disadvantages of these different methods. Recently, tensor perceptual color frameworks have been introduced that apply tensor concepts and perceptual color spaces to improve recognition performance under varying illumination conditions.
![IOSR Journal of Computer Engineering (IOSR-JCE)
e-ISSN: 2278-0661, p- ISSN: 2278-8727Volume 11, Issue 1 (May. - Jun. 2013), PP 61-64
www.iosrjournals.org
www.iosrjournals.org 61 | Page
A Literature review on Facial Expression Recognition Techniques
Ms.Aswathy.R
Post Graduate Scholar,
Computer Science Department, Nehru college of Engineering , India
Abstract: Automatic facial expression recognition has become a progressive research area since it plays a major
role in human-computer-interaction. The facial expression recognition finds major application in areas like
social interaction and social intelligence. A review of various techniques used in facial expression recognition
like principal component analysis (PCA), linear discriminant analysis (LDA) etc is done in this paper. The
survey is represented in tabular form for quick reference
Index Terms— Automatic facial expression recognition, Social Intelligence, Human-Computer interaction.
I. Introduction
In the recent years, many improvements have been accomplished in the areas like face recognition, face
tracking and it anticipated the interest in facial expression recognition field. Facial expressions are produced as a
result of distortions of facial features due to the constriction of facial muscles. The facial expressions are
examined for identifying the basic human emotions like anger, fear, happiness, sadness, disgust and surprise.
These expressions can vary between individuals. The facial expression recognition systems consists of four
major steps. Face detection and normalization phase detects the face and lighting effect is reduced. Next step is
feature extraction which extracts the features and irrelevant features are eliminated in feature selection process.
Final step is classification where the expressions are classified in to six basic emotions.
Approaches like Principal Component Analysis [1], Linear Discriminant Analysis [2], Independent
Component Analysis [3] constitute a major part in the facial expression recognition techniques. These methods
are 1-dimensional in nature. Therefore 2- dimensional Principal Component Analysis [4] is introduced. Since
these techniques are applicable only in gray scale images, Global Eigen Approach [5] and Sub pattern Extended
2-dimensional Principal Component Analysis [6](E2DPCA) can be extended by traditional approaches to color
space. Multilinear Image Analysis [7] introduced tensor concept which allows more than one factor variation in
contradiction to PCA. Color Subspace Linear Discriminant Analysis [8] also uses tensor concept but in color
space which improves the accuracy. For achieving greater performance, another technique called Gabor Filer
Bank [9] is used which outperforms all the other methods. Local Gabor Binary Pattern [10] has improved
recognition rate than the gabor filter bank technique.
Many studies have revealed that the overall performance is enhanced by color component embedding. But if
RGB color space is used, accuracy depend on the angle and light source which reduces the recognition
performance. Therefore RGB color space is not always suitable for color information processing. Perceptually
uniform color systems can address this problem. Therefore a novel tensor perceptual framework[11] for facial
expression recognition is introduced in this paper for better performance. This is done on perceptual color space
and can be examined under slight variations in illuminations.
II. Techniques Used For Facial Expression Recognition
A. Principal Component Analysis
Principal Component Analysis (PCA), also known as the eigen face approach is one of the popular method for
facial expression recognition[1]. The major goal of PCA is to reduce the dimensionality for effective face
indexing and retrieval. Also, PCA uses linear projection,which maximize the projected sample scattering [2]. In
this, the identity of the person is the only varying factor.PCA faces difficuilty if other factors like viewpoint,
lighting are varied.
B. Fisher’s Linear Discriminant
Under severe variation in facial expression and illumination Fisher‟s Linear Discriminant (FLD) is more
suitable. FLD reduces the scattering of projected sample since it is a class specific method[2]. Error rate is
reduced when compared to PCA.
C. Independent Component Analysis
Both PCA and LDA generate spatially global feature vectors. But for effective facial expression
recognition spatially localized feature vectors is needed. Therefore Independent Component Analysis (ICA)](https://image.slidesharecdn.com/j01116164-140428230800-phpapp01/85/J01116164-1-320.jpg)
![A Literature review on Facial Expression Recognition Techniques
www.iosrjournals.org 62 | Page
generates statistically independent basis vector[3].The average recognition rate is improved. But ICA is
computationally expensive than PCA.
D. 2Dimensional Principal Component Analysis
In PCA, feature extraction is done based on 1D vectors. Therefore the image matrix need to be
transformed into vector. 2dimensional Principal Component Analysis (2DPCA) uses 2D matrix instead of 1D
vector[4]. The recognition rate of 2DPCA is higher than PCA. But the storage requirement for 2DPCA is higher
than PCA since 2DPCA needs more coefficients for image representation.
E. Global Eigen Approach using Color Images
Conventional facial expression recognition techniques like PCA, LDA etc uses only the luminance
information in face images. Global Eigen Approach uses the color information in face images[5].RGB color
space does not provide any improvement in recogniton rate. In HSV color space, H component is removed since
it reduces recognition rate. YUV colorspace provides high recognition rate.
F. Sub pattern Extended 2-Dimensional PCA
The recognition rate of PCA is low and has small sample size problem. For gray facial expression
recognition, 2DPCA is extended to Extended 2DPCA. But E2DPCA is not applicable for color images.
Therefore Sub pattern Extended 2-Dimensional PCA (SpE2DPCA) is introduced for color face recognition[6].
The recognition rate is higher than PCA, 2DPCA, E2DPCA and problem of small sample size in PCA is also
eliminated.
G . Multilinear Image Analysis
Facial expression recognition needs different factors like pose, lighting, expressions to be considered.
But the conventional PCA addresses only variations in single factor. Multilinear image analysis use multilinear
algebra[7]. In this, the concept of „Tensor faces‟ is used, which separates different factors underlying the
formation of an image. Recognition rate
is greater when compared to PCA approach. Color information is not incorporated in multilinear image analysis.
H . Color Subspace Linear Discriminant Analysis
The 1DLDA AND 2DLDA are extended in color space to improve the face recognition accuracy. A 3-
D color tensor is used to generate color LDA subspace[8]. Horizontal unfolding increases the recognition rate
for 2DLDA while vertical unfolding improves recognition rate for 2DPCA.The performance evaluation of
various color spaces is not done.
I. 2D Gabor filter bank
The Gabor filtering is considered as one of the most important feature extraction technique in facial
expression recognition[9]. Gabor filter bank performs better in terms of recognition rate than the other methods
like PCA, LDA etc[11]. The major limitation of gabor filter is its bandwidth limitation ie. maximum bandwidth
is limited to one octave. Gabor filters loss high and low frequency information since it is band pass in nature
J. Local Gabor Binary Pattern
Appearence based features are being used for face recognition since it encodes specific details about
human faces. In this facial image is divided into sub blocks and similarities between sub blocks is obtained[10].
An important advantage of Local Binary Pattern (LBP) is its illumination tolerance. In Local Gabor Binary
Pattern(LGBP) method, LBP is extracted from gabor filters for feature vector generation. LGBP achieves better
performance than gabor filter method[11].
All these techniques have improved the performance of the facial expression recognition system
considerably by incorporating color component and tensor concepts. But none of the color spaces like RGB
provide provision for head pose and lighting variations.Therefore perceptual color space like CIELab and
CIELuv is used in this paper to improve the performance in varying conditions. The color images are
represented using 3D matrix. But 2D filtering operation is needed to be applied to a 3D array, which is complex
in nature. As a solution to this problem, 2D filters are to be applied three times over the three color components
in images. Instead of doing this, tensor concept is used. The filtering operation is directly applied to the tensor
i.e generated from the color image.
This paper introduces a Tensor Perceptual Color Framework (TPCF) [11] where color image
components are horizontally unfolded to 2D tensors using multilinear algebra and tensor concepts. Log-gabor
filters are used for feature extraction since it overcome the limitations of gabor filter based method. For feature
selection mutual information quotient method is utilized. Multiclass linear discriminant analysis classifier is](https://image.slidesharecdn.com/j01116164-140428230800-phpapp01/85/J01116164-2-320.jpg)
![A Literature review on Facial Expression Recognition Techniques
www.iosrjournals.org 64 | Page
References
[1] L. Sirovich and M. Kirby, “Low Dimensional Procedure for Characterization of Human Faces,” J. Optical Soc. Am., vol. 4, pp. 519-
524, 1987.
[2] P. N. Belhumeur, J. P. Hespanha, and D. J. Kriegman,“Eigenfaces vs. Fisherfaces: recognition using class specific linear projection,”
IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 711–720, Jul. 1997.
[3] B.A. Draper, K. Baek, M.S. Bartlett, J.R. Beveridge, “Recognizing Faces with PCA and ICA,” Computer Vision and Image
Understanding: special issue on face recognition, in press.
[4] J. Yang, D. Zhang, A. F. Frangi, and J. Y. Yang, “Two-dimensional PCA: A new approach to appearance-based face representation
and recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 26, no. 1, pp. 131–137, 2004.
[5] L.Torres, J. Reutter, and L. Lorente, “The importance of the color information in face recognition,” in Proceedings. 1999
International Conference on Image Processing, 1999. ICIP 99., vol. 3. IEEE, 1999, pp. 627–631.
[6] Chen S., Sun Y. and Yin B., “A Novel Hybrid Approach Based on Sub-pattern Technique and Extended 2DPCA for Color Face
Recognition,” 11th IEEE International Symposium on Multimedia, pp. 630-634, 2009.
[7] M. Thomas, C. Kambhamettu and S. Kumar, (2008), Face recognition using a color subspace LDA approach, Proceedings -
International Conference on Tools with Artificial Intelligence.
[8] M. A. O. Vasilescu and D. Terzopoulos, “Multilinear image analysis for facial recognition,” in Proc. Int. Conf. Pattern Recognit.,
Quebec City, QC, Canada, Aug. 2002, pp. 511–514
[9] Barbu, T, Gabor filter-based face recognition technique, Proceedings of the Romanian Academy, vol.11, no. 3, pp. 277 – 283, 2010.
[10] S. Moore and R. Bowden, “Local binary patterns for multi-view facial expression recognition,” Comput. Vis. Image Understand.,
vol. 115, no. 4, pp. 541–558, 2011.
[11] S. M. Lajevardi and H. R. Wu, “Facial Expression Recognition in Perceptual Color Space” IEEE Transactions on Image Processing,
vol. 21, no. 8, pp. 3721-3732, 2012.](https://image.slidesharecdn.com/j01116164-140428230800-phpapp01/85/J01116164-4-320.jpg)
J01116164
- 1. IOSR Journal of Computer Engineering (IOSR-JCE) e-ISSN: 2278-0661, p- ISSN: 2278-8727Volume 11, Issue 1 (May. - Jun. 2013), PP 61-64 www.iosrjournals.org www.iosrjournals.org 61 | Page A Literature review on Facial Expression Recognition Techniques Ms.Aswathy.R Post Graduate Scholar, Computer Science Department, Nehru college of Engineering , India Abstract: Automatic facial expression recognition has become a progressive research area since it plays a major role in human-computer-interaction. The facial expression recognition finds major application in areas like social interaction and social intelligence. A review of various techniques used in facial expression recognition like principal component analysis (PCA), linear discriminant analysis (LDA) etc is done in this paper. The survey is represented in tabular form for quick reference Index Terms— Automatic facial expression recognition, Social Intelligence, Human-Computer interaction. I. Introduction In the recent years, many improvements have been accomplished in the areas like face recognition, face tracking and it anticipated the interest in facial expression recognition field. Facial expressions are produced as a result of distortions of facial features due to the constriction of facial muscles. The facial expressions are examined for identifying the basic human emotions like anger, fear, happiness, sadness, disgust and surprise. These expressions can vary between individuals. The facial expression recognition systems consists of four major steps. Face detection and normalization phase detects the face and lighting effect is reduced. Next step is feature extraction which extracts the features and irrelevant features are eliminated in feature selection process. Final step is classification where the expressions are classified in to six basic emotions. Approaches like Principal Component Analysis [1], Linear Discriminant Analysis [2], Independent Component Analysis [3] constitute a major part in the facial expression recognition techniques. These methods are 1-dimensional in nature. Therefore 2- dimensional Principal Component Analysis [4] is introduced. Since these techniques are applicable only in gray scale images, Global Eigen Approach [5] and Sub pattern Extended 2-dimensional Principal Component Analysis [6](E2DPCA) can be extended by traditional approaches to color space. Multilinear Image Analysis [7] introduced tensor concept which allows more than one factor variation in contradiction to PCA. Color Subspace Linear Discriminant Analysis [8] also uses tensor concept but in color space which improves the accuracy. For achieving greater performance, another technique called Gabor Filer Bank [9] is used which outperforms all the other methods. Local Gabor Binary Pattern [10] has improved recognition rate than the gabor filter bank technique. Many studies have revealed that the overall performance is enhanced by color component embedding. But if RGB color space is used, accuracy depend on the angle and light source which reduces the recognition performance. Therefore RGB color space is not always suitable for color information processing. Perceptually uniform color systems can address this problem. Therefore a novel tensor perceptual framework[11] for facial expression recognition is introduced in this paper for better performance. This is done on perceptual color space and can be examined under slight variations in illuminations. II. Techniques Used For Facial Expression Recognition A. Principal Component Analysis Principal Component Analysis (PCA), also known as the eigen face approach is one of the popular method for facial expression recognition[1]. The major goal of PCA is to reduce the dimensionality for effective face indexing and retrieval. Also, PCA uses linear projection,which maximize the projected sample scattering [2]. In this, the identity of the person is the only varying factor.PCA faces difficuilty if other factors like viewpoint, lighting are varied. B. Fisher’s Linear Discriminant Under severe variation in facial expression and illumination Fisher‟s Linear Discriminant (FLD) is more suitable. FLD reduces the scattering of projected sample since it is a class specific method[2]. Error rate is reduced when compared to PCA. C. Independent Component Analysis Both PCA and LDA generate spatially global feature vectors. But for effective facial expression recognition spatially localized feature vectors is needed. Therefore Independent Component Analysis (ICA)
- 2. A Literature review on Facial Expression Recognition Techniques www.iosrjournals.org 62 | Page generates statistically independent basis vector[3].The average recognition rate is improved. But ICA is computationally expensive than PCA. D. 2Dimensional Principal Component Analysis In PCA, feature extraction is done based on 1D vectors. Therefore the image matrix need to be transformed into vector. 2dimensional Principal Component Analysis (2DPCA) uses 2D matrix instead of 1D vector[4]. The recognition rate of 2DPCA is higher than PCA. But the storage requirement for 2DPCA is higher than PCA since 2DPCA needs more coefficients for image representation. E. Global Eigen Approach using Color Images Conventional facial expression recognition techniques like PCA, LDA etc uses only the luminance information in face images. Global Eigen Approach uses the color information in face images[5].RGB color space does not provide any improvement in recogniton rate. In HSV color space, H component is removed since it reduces recognition rate. YUV colorspace provides high recognition rate. F. Sub pattern Extended 2-Dimensional PCA The recognition rate of PCA is low and has small sample size problem. For gray facial expression recognition, 2DPCA is extended to Extended 2DPCA. But E2DPCA is not applicable for color images. Therefore Sub pattern Extended 2-Dimensional PCA (SpE2DPCA) is introduced for color face recognition[6]. The recognition rate is higher than PCA, 2DPCA, E2DPCA and problem of small sample size in PCA is also eliminated. G . Multilinear Image Analysis Facial expression recognition needs different factors like pose, lighting, expressions to be considered. But the conventional PCA addresses only variations in single factor. Multilinear image analysis use multilinear algebra[7]. In this, the concept of „Tensor faces‟ is used, which separates different factors underlying the formation of an image. Recognition rate is greater when compared to PCA approach. Color information is not incorporated in multilinear image analysis. H . Color Subspace Linear Discriminant Analysis The 1DLDA AND 2DLDA are extended in color space to improve the face recognition accuracy. A 3- D color tensor is used to generate color LDA subspace[8]. Horizontal unfolding increases the recognition rate for 2DLDA while vertical unfolding improves recognition rate for 2DPCA.The performance evaluation of various color spaces is not done. I. 2D Gabor filter bank The Gabor filtering is considered as one of the most important feature extraction technique in facial expression recognition[9]. Gabor filter bank performs better in terms of recognition rate than the other methods like PCA, LDA etc[11]. The major limitation of gabor filter is its bandwidth limitation ie. maximum bandwidth is limited to one octave. Gabor filters loss high and low frequency information since it is band pass in nature J. Local Gabor Binary Pattern Appearence based features are being used for face recognition since it encodes specific details about human faces. In this facial image is divided into sub blocks and similarities between sub blocks is obtained[10]. An important advantage of Local Binary Pattern (LBP) is its illumination tolerance. In Local Gabor Binary Pattern(LGBP) method, LBP is extracted from gabor filters for feature vector generation. LGBP achieves better performance than gabor filter method[11]. All these techniques have improved the performance of the facial expression recognition system considerably by incorporating color component and tensor concepts. But none of the color spaces like RGB provide provision for head pose and lighting variations.Therefore perceptual color space like CIELab and CIELuv is used in this paper to improve the performance in varying conditions. The color images are represented using 3D matrix. But 2D filtering operation is needed to be applied to a 3D array, which is complex in nature. As a solution to this problem, 2D filters are to be applied three times over the three color components in images. Instead of doing this, tensor concept is used. The filtering operation is directly applied to the tensor i.e generated from the color image. This paper introduces a Tensor Perceptual Color Framework (TPCF) [11] where color image components are horizontally unfolded to 2D tensors using multilinear algebra and tensor concepts. Log-gabor filters are used for feature extraction since it overcome the limitations of gabor filter based method. For feature selection mutual information quotient method is utilized. Multiclass linear discriminant analysis classifier is
- 3. A Literature review on Facial Expression Recognition Techniques www.iosrjournals.org 63 | Page used for classifying the selected features. TPCF can effectively recognize the facial expressions under different illumination conditions and thus performance can be improved. Table1: Comparison table on literature survey III. Conclusion This paper presents a literature survey on the various techniques involved in facial expression recognition. There are various techniques that can be used for the purpose. These methods are measured on the basis of recognition rate. Higher the recognition rate, greater the performance. The tensor perceptual color framework has the highest recognition rate and has highest performance. IV. Acknowledgment This work is supported and guided by my research guide. I am very thankful to my research guide Ms. Shinu Acca Mani, Assistant Professor, Computer Science Department, Nehru college of Engineering and Research Centre, India for her guidance and support. NAME METHOD PERFORMANCE DISADVANTAGES Low-Dimensional Procedure for Characterization of Human Faces Principal Component Analysis Recognition rate is low Only single factor can be varied Eigenfaces vs. Fisherfaces : recognition using class specific linear Projection Fisher‟s Linear Discriminant Recognition rate higher than PCA Global feature vectors are generated Recognizing Faces with PCA and ICA Independent Component Analysis Recognition rate is improved compared to PCA and FLD Computationally expensive than PCA Two-dimensional PCA: A new approach to appearance- based face representation and recognition 2-Dimensional Principal Component Analysis Recognition rate is higher than PCA Storage requirement is higher than PCA The importance of the color information in face recognition Global Eigen Approach using Color Images YUV color space has highest recognition rate RGB color space does not provide any improvement in recognition rate A novel hybrid approach based on sub-pattern technique and E2DPCA for color face recognition Subpattern Extended 2- Dimensional Principal Component Analysis Recognition rate higher than PCA,2DPCA Variation in lighting, pose are not considered Face Recognition using a Color Subspace LDA approach Color Subspace Linear Discriminant Analysis Recognition rate is higher than 2DPCA and LDA Variation of performance in color spaces is not evaluated Multilinear Image Analysis for Facial Recognition Multilinear Image Analysis Recognition rate higher than PCA Less performance than Color Subspace LDA Gabor Filter Based Face Recognition Technique 2-Dimensional Gabor Filter Bank Higher recognition rate than PCA, LDA,2DPCA,Global Eigen Approach Low and high frequency component attenuation Local binary patterns for multi-view facial expression recognition Local Gabor Binary Pattern Better recognition rate than Gabor filter bank Color information is not included
- 4. A Literature review on Facial Expression Recognition Techniques www.iosrjournals.org 64 | Page References [1] L. Sirovich and M. Kirby, “Low Dimensional Procedure for Characterization of Human Faces,” J. Optical Soc. Am., vol. 4, pp. 519- 524, 1987. [2] P. N. Belhumeur, J. P. Hespanha, and D. J. Kriegman,“Eigenfaces vs. Fisherfaces: recognition using class specific linear projection,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 711–720, Jul. 1997. [3] B.A. Draper, K. Baek, M.S. Bartlett, J.R. Beveridge, “Recognizing Faces with PCA and ICA,” Computer Vision and Image Understanding: special issue on face recognition, in press. [4] J. Yang, D. Zhang, A. F. Frangi, and J. Y. Yang, “Two-dimensional PCA: A new approach to appearance-based face representation and recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 26, no. 1, pp. 131–137, 2004. [5] L.Torres, J. Reutter, and L. Lorente, “The importance of the color information in face recognition,” in Proceedings. 1999 International Conference on Image Processing, 1999. ICIP 99., vol. 3. IEEE, 1999, pp. 627–631. [6] Chen S., Sun Y. and Yin B., “A Novel Hybrid Approach Based on Sub-pattern Technique and Extended 2DPCA for Color Face Recognition,” 11th IEEE International Symposium on Multimedia, pp. 630-634, 2009. [7] M. Thomas, C. Kambhamettu and S. Kumar, (2008), Face recognition using a color subspace LDA approach, Proceedings - International Conference on Tools with Artificial Intelligence. [8] M. A. O. Vasilescu and D. Terzopoulos, “Multilinear image analysis for facial recognition,” in Proc. Int. Conf. Pattern Recognit., Quebec City, QC, Canada, Aug. 2002, pp. 511–514 [9] Barbu, T, Gabor filter-based face recognition technique, Proceedings of the Romanian Academy, vol.11, no. 3, pp. 277 – 283, 2010. [10] S. Moore and R. Bowden, “Local binary patterns for multi-view facial expression recognition,” Comput. Vis. Image Understand., vol. 115, no. 4, pp. 541–558, 2011. [11] S. M. Lajevardi and H. R. Wu, “Facial Expression Recognition in Perceptual Color Space” IEEE Transactions on Image Processing, vol. 21, no. 8, pp. 3721-3732, 2012.