Toward a Dialogue System Using a Large Language Model to Recognize User Emotions with a Camera
0 likes43 views
The document discusses the development of a dialogue system using a large language model to recognize user emotions via facial expressions, aimed at improving visitor interactions during office relocations. It highlights the system's ability to respond based on emotional states such as happiness, anger, or sadness, and suggests potential applications in various fields like university operations and sports psychology. Challenges remain in enhancing emotion recognition accuracy and integrating speech recognition into the system.
Toward a Dialogue System Using a Large Language Model to Recognize User Emotions with a Camera
- 1. Toward a Dialogue System Using a Large Language Model to Recognize User Emotions with a Camera Tokushima University Hiroki Tanioka
- 2. 2 MOTIVATION Topic Our office moving Need To show visitors where to move Issue How to deal with visitors (offline users)
- 3. 3 Office Moving Information ★ Here!! HOW TO SOLVE? I have a question... ① ★ Here!! Map Not user friendly as she/he have to move around.
- 4. 4 HOW TO SOLVE? Office Moving Information ★ Here!! Online Support ② It would be better solution, but the fact that you actually visited the site means it may not be an online solution. I have a question...
- 5. 5 HOW TO SOLVE? Office Moving Information ★ Here! ! Can I help you? ③ The best solution would be to recognize the urgency and emotional state of the user, as online responses do not adequately convey this. I have a question... Can I help you? Using a camera like Duckietown
- 6. 6 FACIAL EXPRESSION RECOGNITION The same facial expression does not yield the same emotional state. Faces (upper left: normal, upper right: smiling, lower left: angry, and lower right: sad) of Japanese woman who visiting for consultation.
- 7. 7 HOW TO BUILD? • FER (Facial expression recognition in Python) • gpt-3.5-turbo (a model of ChatGPT) • Adding recognized emotion information in JSON format to the ChatGPT prompt FER
- 8. 8 HOW TO TEST? • Two types of dialogue scenario CASE A: ”Hello” + Emotion CASE B: Counseling for a friend + Emotion • FBot prototype system is developed in Python script on Google Colaboratory
- 9. 9 RESULT (CASE A): “Hello.” + Emotion Response (beginning of example) Under stand Worry -ing Encour aging Hello! It’s nice to hear from you. How are you feeling today? If there’s something on your mind or if you’d like to chat --- about anything, I’m here to listen and support you. Remember, your emotions are valid, and it’s okay to express how you feel. Take your time, and feel free to share whatever you’re comfortable with. Hello. It’s great to see you feeling happy! How can I assist you today? ✔︎ Hello! I’m here to listen and support you. If you’d like to talk about what’s on your mind or if there’s anything I can do to help, feel free to share. Remember, it’s okay to feel a range of emotions, and I’m here to offer assistance in any way I can. Take a deep breath, and know that you’re not alone. ✔︎ ✔︎ ✔︎ Hello! I’m here for you. If you’d like to talk about what’s been bothering you or if there’s anything specific you need help with, please feel free to share. Remember, it’s important to express your emotions and seek support when you’re feeling down. I’m here to listen and offer my assistance in any way I can. Take your time, and know that you are not alone. ✔︎ ✔︎ ✔︎
- 10. 10 RESULT (CASE B): “Thank you.”+ Emotion Response (beginning of example) Under stand Worry -ing Encour aging You’re welcome! I’m here to support you. If you ever need to talk more about comforting your friend or if there’s anything else on your mind, feel free to reach out. Remember, your compassion and care for your friend are truly valuable. Take care of yourself too during this time. Sending you positive thoughts and warm wishes. ✔︎ You’re welcome! I’m glad to see you feeling happy. If you have any more questions or need further assistance in the future, feel free to reach out. Remember, your kindness and support can truly brighten someone’s day. Keep spreading positivity and joy. Take care and stay happy! ✔︎ ✔︎ You’re welcome! It’s important to acknowledge your feelings and know that it’s okay to experience a range of emotions. If you ever need someone to talk to or seek comfort, don’t hesitate to reach out. Remember, you are not alone, and there are people who care about you and want to support you. Take care of yourself and remember to be kind to yourself during this time. Sending you positive thoughts and strength. ✔︎ ✔︎ ✔︎ You’re welcome! I’m here to support you and offer guidance whenever you need it. Remember that you are a caring friend, and your efforts to comfort and support others are truly admirable. If you ever want to talk more or share your feelings, feel free to reach out. Keep spreading kindness and positivity, even during challenging times. Take care, and stay strong. Sending you virtual hugs and support! ✔︎ ✔︎ ✔︎
- 11. 11 SUMMARY AND REMAINING ISSUES Developed a system for AI agents that sends emotional data from facial expressions with user inquiries to LLMs. Enabled the system of LLMs to respond differently based on emotions like happiness, anger, or sadness. Potential applications include receptionist tasks, university operations, sports psychology, and information security. Enhancing methods to summarize and interpret variable emotional data from user queries. Improving facial expression recognition accuracy, including for users wearing masks. Integrating speech recognition into the multimodal dialogue system.