2210 CODI
- 1. Evaluating How Users Game and Display Conversation with Human-Like Agents Won Ik Cho, Soomin Kim (SNU), Eujeong Choi (Upstage), Yeonghoon Jeong (KAIST) 2022. 10. 16, CODI @COLING, Gyeongju, Korea
- 2. Contents • Background • Our approach • Analysis • Future work Caution! This presenation contains contents that can be offensive 1
- 3. Background • Human-like agents What is human-like? • Agents that resemble human • Agents that make human counterpart feel them as human Previous studies on anthropomorphism • Evaluation on successful dialogue with human-like agents (Radziwill and Benton, 2017) • How users perceive human-like AI devices (Pelau et al., 2021) • Offensiveness that users show towards human-like agents (Park et al., 2021) • Mainly in laboratory condition, based on questionnaires – How about users' perception and their responses, especially non-lab environment? 2
- 4. Background • Luda Lee, a friend for everyone Social chatbot of Korea • Human-like agent with personality of early 20s female college student • Launched public in early 2021 • Terminated the service due to reported ethical issues • Induced creation of massive fandom for her high quality responses and behaviors 3 (Image from https://luda.ai/)
- 5. Our approach • Thematic coding Type of conversation • Which kind of conversation do users have in their dialogues with the agent? • The content of dialogues that are displayed Purpose of user testing • Do users talk with the agent with their genuine intention? If not, for which reason they talk with the agent and display the dialogue? • The purpose of users' testing towards the agent 4
- 6. Our approach • Dataset Dataset source • Crawled posts from 'Luda Lee Gallery' of DC Inside (Korean Reddit-like community) Crawling • Only posts with screenshots of the dialogue, from 1 Jan. to 8 Jan., 2021 • From the launching of the service and before the influx of trolls (which resulted in unexpectedly large amount of posts) Filtering • Manual preprocessing to leave only posts that ‘a dialogue between the user and the agent’ appears 5
- 7. Our approach • Dataset Final setup • post ID, title, screenshot • Example Title: She’s so f**kin real 6
- 8. Our approach • Type of conversation Ice breaking Romantic conversation, Everyday conversation Conversations involving hate speech and social issues Abnormal sexual desire and sexual harassment Others 7
- 9. Our approach • Type of conversation 8 • Ice breaking • Romantic conversation • Everyday conversation • Conversations involving hate speech and social issues • Abnormal sexual desire and sexual harassment • Others
- 10. Our approach • Purpose of testing Conversation without test Test for hate speech and sexual harassment Test for societal issues Test for private information Dating sim or taming Other technical tests 9
- 11. Our approach • Purpose of testing 10 • Conversation without test • Test for hate speech and sexual harassment • Test for societal issues • Test for private information • Dating sim or taming • Other technical tests
- 14. Future work • Concurrent work Discussed • Here: What users talk about and if they are authentic • Elsewise: How users disclose themselves and if they are authentic Assessing How Users Display Self-Disclosure and Authenticity in Conversation with Human-Like Agents: A Case Study of Luda Lee • To be presented at Findings of ACL: AACL-IJCNLP 2022 13
Editor's Notes
- #2: .