Facial Emotion Detection Project
- 1. AI Facial Emotion Detection By: Ellie Pierson, Richard Thomas, Renold Thomas, Luca Netter, Bergen Cloninger, Emily Joseph Instructor: Laikh Tewari
- 2. Sections to include ● Intro + context - Richard ● Problem setup and approach - Emily ○ Raw pixels vs extracted features ● Models (epochs, other hyperparams): - Luca, Bergen, and Ellie ○ KNN ○ Log reg ○ Neural Net ○ CNN ○ Transfer Learning ○ Eye+eyebrow model ○ Comparison table ● Conclusion - Renold
- 3. What we are doing? ● Creating an AI to read emotions ○ Helps kids with autism ○ Driving assistance/safety ○ Help teachers in online teaching
- 4. Can you label the images below using these 5 emotions? ● Happy ● Sad ● Surprised ● Angry ● Neutral 1 2 3 4 5 6
- 5. Can you label the images below using these 5 emotions? ● Happy ● Sad ● Surprised ● Angry ● Neutral ✓ 1 2 3 4 5 6 Happy Angry Surprised Sad Angry Neutral
- 6. How can we use AI to identify these emotions?
- 7. Setup and Approach ● Use the best model that predicts emotions most accurately Landmarks and distances Models Evaluating Results! 1 2 3 4 ● Calculate distances between each facial landmark ● Find landmarks on different face images ● Training models using data consisting of faces and labels (knn, lr, dt, neural networks) ● Evaluate model performance using confusion matrix, epoch/accuracy graphs, and accuracy scores
- 8. Model Input Data EXTRACTED FEATURES Each image has 64 facial landmarks The # of distances between landmarks: 64+63+62+61… + 1 = 2278 Then we condense those 2278 distances into just 20 features! INPUT: 20 distance values for each OR RAW PIXELS Each image in the data set is 48*48 pixels (in grayscale) 48*48 = 2304 INPUT: 2,304 pixels for each image
- 9. KNN example ● We used a KNN model (K nearest neighbors) to establish a baseline accuracy ● A KNN model is a type of classification where an algorithm tries to predict what a new data point will look like based off of existing data ● KNN classification tends to be fast but inaccurate ● With this model, the accuracy comes out to be 43.55%
- 10. Log Reg Example ● Logistic regression is a model that classifies based off of a probability. ● Logistic regression creates a curve shaped like an S. ● An ‘S-Curve’ cannot extend the class number above or below 0-1. ● With this model, the accuracy comes out to be 36.8%
- 11. Neural Network ● A Neural Network is a model based off of how the human brain works, and uses input, hidden, and output layers. ● Each layer consists of some amount of neurons, which are used to test and train information by sending the information through every possible path between the layers. It filters the information which in this case is images, and attempts to correctly predict the emotion ● With this model, the accuracy comes out to be 51.7%, which is almost as good as a human, but just shy.
- 12. Neural Network Example ● Here is an example of a Neural Network: ● Ours looks similar to this, but with many more neurons in each layer.
- 13. CNN Example ● Takes an image ● Filters through a set of layers ● Predicts a label Through different variables: Our best model showed: Validation Accuracy = 68% , Loss = 0.85
- 14. Transfer Learning ● CNN Data Set →Model →Test Result ● Val Acc: 68.2% ● Better than most tested previous models
- 15. Eye + Eyebrow Model We wanted to see how our models would do if we were only given landmarks and distances above the nose as if we had a mask on. Our models will only receive inputs of eyes and eyebrows landmarks. Can you tell what emotion this is? Happy! K Nearest Neighbors: 38.35% Logistic Regression: 38.15% Decision Tree: 34.35% Neural Network: 42.65%
- 16. Comparison Table The most accurate model was a pre trained VGG model using Transfer Learning at 68.2% accuracy. Our trained CNN model was close behind at 68.0% accuracy.
- 17. Future Applications Emotion Detection could be used for: ● Help kids with autism ● In learning environments (i.e. Zoom, Google Meet)
- 18. If we had 1 more week ● Create a live camera feature
- 19. Thank You