Inspirit AI Facial Emotion Detection Project (Dec 2021)
- 1. AI Facial Emotion Detection By: Ellie Pierson, Richard Thomas, Emily Joseph, Renold Thomas, Luca Netter, Bergen Cloninger Instructor: Laikh Tewari
- 2. What we are doing? ● Creating an AI to read emotions ○ Helps kids with autism ○ Driving assistance/safety ○ Help teachers in online teaching
- 3. Can you label the images below using these 5 emotions? ● Happy ● Sad ● Surprised ● Angry ● Neutral 1 2 3 4 5 6
- 4. Can you label the images below using these 5 emotions? ● Happy ● Sad ● Surprised ● Angry ● Neutral ✓ 1 2 3 4 5 6 Happy Angry Surprised Sad Angry Neutral
- 5. How can we use AI to identify these emotions?
- 6. Setup and Approach ● Use the best model that predicts emotions most accurately Landmarks and distances Models Evaluating Results! 1 2 3 4 ● Calculate distances between each facial landmark ● Find landmarks on different face images ● Training models using data consisting of faces and labels (knn, lr, dt, neural networks) ● Evaluate model performance using confusion matrix, epoch/accuracy graphs, and accuracy scores
- 7. Model Input Data EXTRACTED FEATURES Each image has 64 facial landmarks The # of distances between landmarks: 64+63+62+61… + 1 = 2278 Then we condense those 2278 distances into just 20 features! INPUT: 20 distance values for each OR RAW PIXELS Each image in the data set is 48*48 pixels (in grayscale) 48*48 = 2304 INPUT: 2,304 pixels for each image
- 8. KNN example ● We used a KNN model (K nearest neighbors) to establish a baseline accuracy ● A KNN model is a type of classification where an algorithm tries to predict what a new data point will look like based off of existing data ● KNN classification tends to be fast but inaccurate ● With this model, the accuracy comes out to be 43.55%
- 9. Log Reg Example ● Logistic regression is a model that classifies based off of a probability. ● Logistic regression creates a curve shaped like an S. ● An ‘S-Curve’ cannot extend the class number above or below 0-1. ● With this model, the accuracy comes out to be 36.8%
- 10. Neural Network ● A Neural Network is a model based off of how the human brain works, and uses input, hidden, and output layers. ● Each layer consists of some amount of neurons, which are used to test and train information by sending the information through every possible path between the layers. It filters the information which in this case is images, and attempts to correctly predict the emotion ● With this model, the accuracy comes out to be 51.7%, which is almost as good as a human, but just shy.
- 11. Neural Network Example ● Here is an example of a Neural Network: ● Ours looks similar to this, but with many more neurons in each layer.
- 12. CNN Example ● Takes an image ● Filters through a set of layers ● Predicts a label Through different variables: Our best model showed: Validation Accuracy = 68% , Loss = 0.85
- 13. Transfer Learning ● CNN Data Set →Model →Test Result ● Val Acc: 68.2% ● Better than most tested previous models
- 14. Eye + Eyebrow Model We wanted to see how our models would do if we were only given landmarks and distances above the nose as if we had a mask on. Our models will only receive inputs of eyes and eyebrows landmarks. Can you tell what emotion this is? Happy! K Nearest Neighbors: 38.35% Logistic Regression: 38.15% Decision Tree: 34.35% Neural Network: 42.65%
- 15. Comparison Table The most accurate model was a pre trained VGG model using Transfer Learning at 68.2% accuracy. Our trained CNN model was close behind at 68.0% accuracy.
- 16. Future Applications Emotion Detection could be used for: ● Help kids with autism ● In learning environments (i.e. Zoom, Google Meet)
- 17. If we had 1 more week ● Create a live camera feature
- 18. Thank You
Editor's Notes
- #4: Kids with autism are able to make eye contact and read other people emotions through AI glasses. It improves their social skills. Cars can use this feature to read driver’s emotions and send personalized alerts to tell the driver if they are drowsy or if they are mad so they don’t drive recklessly. Teachers could use this to read students emotions and receive feedback to change the teacher’s approach to their teaching.
- #5: Mention how to answer: chat, annotate, think
- #9: Know how condensing features works: Standardization is the process of putting different variables on the same scale. It is a transformation that centers the data by removing the mean value of each feature and then scale it by dividing (non-constant) features by their standard deviation. After standardizing data the mean will be zero and the standard deviation one. We can use sklearn's inbuilt function which will help us to standardize our train data: Dimensionality reduction helps us find a low-dimensional representation of the data that retains as much information as possible. Principal Component Analysis (PCA) is one such technique.PCA is a technique used to emphasize variation and bring out strong patterns in a dataset.
- #10: bergen cloninger the baseline accuracy from this “quick and dirty” algorithm is fairly low, the best accuracy being 43.55% a confusion matrix is a summary of the results of a classification problem. Ideally, a confusion matrix would have no discrepancy along the diagonal, as the predicted labels would perfectly match the true labels. The confusion matrix for this example highlights the diagonals but still has some major flaws
- #11: bergen cloninger In a linear regression model, the trend line predicts that some points would have a probability of above 1 or below 0, which is impossible. A logistic regression curve solves this problem by “squishing” the line down in between the 0 and 1, allocating a probability for every point. In both the KNN and Logistic regression examples, the raw pixel data had a lower accuracy than the extracted features
- #12: luca-
- #15: We used a pre trained vgg model and using our cnn data set to fine tune the vgg model which gives us our test result
- #16: Ellie
- #17: Ellie
- #19: To recognize our emotions in real time using a camera