![Chatbot Components
• Why are users
interacting with your
chatbot? What tasks are
they trying to complete?
What goals are they
trying to achieve?
• Example:
• The user wants to
reset their email
password.
Intents Utterances
Entities Dialog
• What are the nouns the
user might want to take
action on? What are
synonyms for those
nouns?
• A minimum of five user
examples should be
created for each intent,
but 10-15 is better.
• Examples:
• #Entity : synonyms
• Email : mail, outlook,
notes, verse, e-mail,
apple mail
• For each intent, create a
list of user examples or
how your users might
state their intentions.
• Examples:
• I need to reset my
@email password.
• I want to change my
@email password.
• How do I change my
@email password?
• Error says to reset my
@email password.
• Matches the user’s
utterance (user’s stated
goal) with a dialog
response (chatbot’s
response).
• Example:
• [user] How do I reset
my email password?
• [chatbot] Outlook now
uses your w3id and
password to sign in.
To change your w3
password ….](https://image.slidesharecdn.com/uxpachatbotinaboxworkshop-220621221331-adbf27e6/85/Workshop-Chatbot-in-a-box-Introduction-to-conversation-design-and-conducting-research-on-chatbots-26-320.jpg)
![Response Types: Disambiguation intent
Confusion between 2 or more intents, with similar training phrases (utterances). Create
an intent that helps system find the correct intent.
When there is a vast list of supported options, conversational and open ended. But when
the supported list is small, just list the options in a menu – don’t play 20 questions.
“I want to buy something”
Huge list of things you can buy
Good open ended/conversational:
What is it you want to buy?
Bad menu:
[Food] [Water] [Cars] [Weapons] [Rocks]
[Plants] [Pets] [Livestock] [Real
estate] [Bitcoin] [Bridges] [Gardening]
[Books] [Movies] [Tickets] [Vacations]
[Storage] [Tech] [Photo] [Drug] […] […]
“I want to reset my password”
Lots of types, just 3 supported
Good menu:
Which password do you want to reset?
[Mac] [iPhone] [IBMid]
Bad open ended/conversational
What kind of password is it?](https://image.slidesharecdn.com/uxpachatbotinaboxworkshop-220621221331-adbf27e6/85/Workshop-Chatbot-in-a-box-Introduction-to-conversation-design-and-conducting-research-on-chatbots-84-320.jpg)
![Entity/slots
@toppings:
[value: synonyms]
meat: pepperoni, sausage, ham, anchovies, chicken
veggies: black olives, green olives, onions, bell peppers, jalapenos, artichoke
cheese: mozzarella, white cheddar, asiago, three cheese blend
values synonyms
Entity label (category)](https://image.slidesharecdn.com/uxpachatbotinaboxworkshop-220621221331-adbf27e6/85/Workshop-Chatbot-in-a-box-Introduction-to-conversation-design-and-conducting-research-on-chatbots-92-320.jpg)
![Responses within the dialog
- Excellent choice! Pizza is our specialty. Do you want toppings? Yes | No
Yes: Alright. Three toppings come for free. Each additional topping is $.50. List of
available toppings: [list]
No: jump to size
- Sounds like you want a [variable] pizza. Do you want any other toppings? Yes | No
Yes: Alright. Three toppings come for free. Each additional topping is $.50. List of
available toppings: [list]
No: jump to size
- Let's make sure I have this right. You want a pizza with [selected options]. Is that correct?
Yes: Great! What size do you want? [@size]
No: Oops! Let's go over the toppings again. What toppings do you want on your pizza?
- Ok, so you want a large. Is that correct? Yes | No
Yes: Would you like anything else? We also have appetizers, desserts, and drinks.
No: Sorry, I must have mozarella in my ears. What size pizza do you want?](https://image.slidesharecdn.com/uxpachatbotinaboxworkshop-220621221331-adbf27e6/85/Workshop-Chatbot-in-a-box-Introduction-to-conversation-design-and-conducting-research-on-chatbots-94-320.jpg)
![Entity
@uses:
[value: synonyms]
antiseptic: clean wounds, cleaning wounds, boo boo, ouchie
cooking: baking, salad dressing, marinade
drinks: tea, milk, coffee, kool-aid, lemonade
mead: alcohol, cocktail](https://image.slidesharecdn.com/uxpachatbotinaboxworkshop-220621221331-adbf27e6/85/Workshop-Chatbot-in-a-box-Introduction-to-conversation-design-and-conducting-research-on-chatbots-104-320.jpg)
![Bonus intents!
Disambiguation intent:
Choose a type of honey
Intent name:
#chooseType
Entity:
@honeyType:
[value: synonyms]
honeyType: acacia, buckwheat, clover, eucalyptus, orange blossom, tupelo, sage
Utterances:
- I want to try some @honeyType @honey
- My favorite @honey is @honeyType
Chit chat:
- On dear, hunny has so many uses, doesn’t it?
- So wonderful you enjoy hunny as much as I do](https://image.slidesharecdn.com/uxpachatbotinaboxworkshop-220621221331-adbf27e6/85/Workshop-Chatbot-in-a-box-Introduction-to-conversation-design-and-conducting-research-on-chatbots-106-320.jpg)
Workshop: Chatbot in a box - Introduction to conversation design and conducting research on chatbots
- 1. CIO Design, IBM Chatbot in a box: Introduction to conversation design and conducting research on chatbots Presenters: Jon G. Temple, Ph.D., Jason Telner, Ph.D., & Patricia Orsini Contributing material: Dabby Phipps Workshop@UXPA IBM, CIO Design
- 2. Who we are Dabby Phipps (co-author; PoohBot creator) Senior UX Designer/Researcher Patricia Orsini AI Experience Content Designer Jason Telner, PhD Senior UX Researcher Jon G. Temple, PhD IBM Design Principal
- 3. Welcome! • Introductions • Today’s agenda • Group activity: What do you want out of this workshop?
- 4. What is a chatbot / conversational agent / virtual agent / digital assistant? “I don’t know that. Let me search the web for you…”
- 5. The first chatbot was created in the mid 1960’s…
- 6. Meet ELIZA: The first chatterbot • ELIZA - natural language conversational program (1966) by Joseph Weizenbaum • Simulates Rogerian psychotherapist • ELIZA takes the user input, looks for keywords and synonyms, creates a response - often a transformation of the user’s input2 • ELIZA is not intelligent and can’t learn. • May not make sense or be grammatically correct • Comp Sci 101 project in Pascal 1https://www.masswerk.at/elizabot/
- 7. …as of 2018, there were over 3k chatbots …on Facebook alone
- 8. Chatbots Today • Software programs - leverage artificial narrow intelligence (ANI) • Answer user questions or perform certain tasks within a narrow scope • Natural language processing (NLP) - utterance • For chatbots, the key is natural language understanding (NLU) - to understand the user’s “intent” or what they’re trying to achieve3 https://www.bold360.com/features/chat-and-messaging
- 9. Why the Explosive Growth? • Primarily the emergence of two technologies: • Messenger applications • Artificial Intelligence / Machine Learning
- 10. Quick Facts1 • It's estimated that 1.4 billion people now use chatbots on a fairly regular basis. (Acquire) • In 2020 it was estimated that $5 billion would be invested in chatbots by 2021. (Chatbots Magazine) • Up to 64% of internet users surveyed said round-the-clock support is the biggest benefit of chatbots. (Drift) • If the chatbot can't solve a problem, 79% of people want access to a real person.
- 11. Chatbot Myth Watson on Jeopardy (2011)5
- 12. What Users Really Want • Businesses may want an AI with personality or conversational • Users aren’t looking for a virtual buddy because they are lonely • Users just want answers or assistance - efficiently as possible • For example, people using an IT help system rated the same system different based solely on outcome (NPS: 31 v. -32) • Best solution ever! – if it helps me; otherwise, it is the worst!! • If chatbots or a conversational UI can provide the appropriate answers faster, they will be preferred
- 13. The Pros and Cons of Chatbots: The Business Perspective Pros • Automate simple conversations • Available 24x7 • Multiple customers simultaneously • Great for small businesses • User in control • NLP conversational style of interaction Cons • Hard to support broad domains • Poor performance leads to decreased user confidence • Takes time and effort to build the knowledge corpus • May require natural language when not needed
- 14. The User Perspective: What Do They Like/Dislike? Likes • High availability • Not having to talk to a person • No wait time • Able to use conversational language • Still seems high tech and kind of cool Dislikes • Not able to talk to a person • Forced to use conversational language when other approaches may be more efficient • Being told it will search the web • Failure to use context • An indirect answer: “Watch this 10-min video!” ”Read this long document!” • Having to guess the chatbot’s domain • Failing to get an answer / not having expectations met (high risk of abandonment) • A complex and cluttered UI that belies the promised simplicity of a conversation
- 15. Relevant Design Roles UX researcher Visual design Interaction/UX design Conversation design
- 16. How these roles work together UX researcher The Business: “I want a chatbot!” Validate business assumptions with users Discovery phase – What do users need? Write new cognitive intents / entities/ dialogs Validate with users UX researcher UX researcher Conversational designer
- 17. Activity (2 mins) Think back to a recent experience with a chatbot • What did you find useful? • What needed improvement?
- 18. The knowledge of chatbots and the conversation designer
- 19. Where does the chatbot's knowledge come from? • Cognitive theories of mind - “homunculus problem" as a straw man • Cognitive processes - little people inside your head • Recursion - “what’s inside the head of the homunculus?” • Yet - this is true for chatbots: knowledge comes from conversation designers (aka people) • Chatbots aren’t reading textbooks, forums and writing own answers • Machine learning - (if used) to improve the query
- 20. Chatbot knowledge It comes from people!* *Soylent Green (1973)
- 21. Role of Conversation Designer • Conversation designers need to determine: • Chatbot’s purpose • Type of interaction (i.e., playful, neutral, professional) • The user intents within the domain • Craft concise responses - engage the user and address their need
- 22. Be engaged. Feel empowered. Get excited. Conversation Design IBM | CIO Design The job of the conversation designer Engage the user Make it a good experience Keep them coming back
- 23. Determine the Chatbot’s Purpose • End users • Solve user problems • Business goals • Orgs’s vision • Specific job or jobs for the chatbot
- 24. Tone of Voice • Dimension of formality: • Informal & friendly: "Hi there, how can I help you?” • More formal, still friendly: "Good morning. How may i help you today? Type your question in the box below to get started." • Formal, not very friendly: "This service is designed to answer your question in the box below.” • Consider: Your brand, target audience, tone of other content sources used by the solution, nature of the specific questions and responses
- 25. What Are Your Users’ Intentions? • Why are users interacting with your chatbot? What tasks are they trying to complete? What goals are they trying to achieve? • Once you’ve identified the intents, the next step is to create a list of user examples or how your users might state their intentions. • A minimum of five user examples should be created for each intent, but 10-15 is better.
- 26. Chatbot Components • Why are users interacting with your chatbot? What tasks are they trying to complete? What goals are they trying to achieve? • Example: • The user wants to reset their email password. Intents Utterances Entities Dialog • What are the nouns the user might want to take action on? What are synonyms for those nouns? • A minimum of five user examples should be created for each intent, but 10-15 is better. • Examples: • #Entity : synonyms • Email : mail, outlook, notes, verse, e-mail, apple mail • For each intent, create a list of user examples or how your users might state their intentions. • Examples: • I need to reset my @email password. • I want to change my @email password. • How do I change my @email password? • Error says to reset my @email password. • Matches the user’s utterance (user’s stated goal) with a dialog response (chatbot’s response). • Example: • [user] How do I reset my email password? • [chatbot] Outlook now uses your w3id and password to sign in. To change your w3 password ….
- 27. Common intents Insurance Real Estate Finance/Banking Retail Employee/HR Receive insurance quote Find information on available property Ask about a charge Learn about product or service details Learn about benefits Change insurance plan Find information on listing a property Pulling up information on accounts Check on order status and delivery times Find out information on internal ticketing processes Get help with insurance policy information Provide recommendations on real estate vendors Report fraud Find out how to return a product Find out subject matter experts within an organization Upgrade service or contract Find out return policy Review information on a ticket Learn about service details Report missing products Find information about customers and relationships Troubleshoot an issue Find information on organization’s services and products
- 28. Example Utterance: Show me yesterday’s financial news Intent: #showNews Entities/slots: @dateTime @newTypes Response: < Stocks are down again. Here are the financial news. >
- 29. Chatbot Flow Overview • Intents - user’s intention or what they want to do (verb) • Entities - keywords modify the intent (noun) - the what • Dialog - response generated by the chatbot based on its understanding of the user’s input using intents and entities User input (utterance) Bot assesses user intent Bot identifies best solution Bot provides response (dialog)
- 30. Role of Training and Machine Learning • Training does not create chatbot knowledge • Machine learning and training improves discoverability • If a user asks a question, machine learning improves the likelihood the correct answer will be found (if it exists) • The AI assigns a “confidence level” • Should the confidence level not meet the threshold (e.g. .60), the response isn’t displayed
- 31. How to identify user needs for cognitive intents: Research!
- 32. Intent Creation Process Prioritize backlog Identify end user phrases for the issue Determine entities Talk to SME’s & review support materials Generate utterances Create flow charts Write dialog Work with developers to map out API integrations Add intents, entities, and dialog to chatbot system Manual and automated testing
- 33. Example: Researching a new intent Identify top call drivers from support tickets “We get a lot of calls about resetting two types of passwords. I have a script that I read to callers”. Interview human support agents Input from SME’s Search data User feedback New content/products Data science analysis “We have an API that can be used to reset the intranet password easily; but to reset your email password, you need the vendor to provide us new services”. “I have too many different kinds of passwords to keep track of and I often forget to change them on time”. — Ima Uzer Top call drivers for August: 1. Email 2.Reset intranet password 3. VPN… Top search queries: 1. Outlook, HCL Notes, Apple mail 2. Checkpoint 3.Intranet password change, reset… Added 200K Macs to user base. New support content written for: Mac account access, Filevault, Screentime, Apple ID… Data science report: Patterns found in multiple sources - surveys, searches: password reset complexity in multiple contexts, including… Reset password
- 34. Overview of user research for chatbots –User needs and pain points “empathy” research to determine intents User interviews Contextual inquiry Journey Maps Personas Friction analysis Surveys Task completion /Help desk calls Comment analysis NPS scores
- 35. • Testing intents with users o Creating a user test plan o Best practices for testing intents o Using field metrics once your chatbot is deployed Chat bot in a box | UXPA 2022 Overview of user research for chatbots
- 36. Historical data –NPS scores, task completion/help desk calls and survey comment analysis 36
- 37. 0 1 2 3 4 5 6 7 8 9 10 0% 0% 0% 14% 14% 0% 0% 0.0% 0% 30% 43% Promoters – Detractors = NPS = 73-28 = 45 How likely are you to recommend application X to a friend or colleague? • Net Promoter Score (NPS) – A measure of how likely someone is to recommend a service or application • Typically asked in surveys • Should have a minim N of 30 respondents for acceptable Margin or Error • Comments most useful for recognizing reasons for low NPS i.e. detractor comments Net Promoter Score (NPS)
- 38. w3U – IT Support User Research IBM | w3 Unification 4Q20 4Q21
- 39. Top Task Failures Goal Completion Analysis IBM | CIO Design 0 50 100 150 200 250 300 IT Support: Ticket_open_update_close_status Search: Other Search: IT_support_help IT Support: Contact_support_call_chat IT Support: Other News: Other IT Support: Password_access_login Top Task Failures No Partially Yes
- 40. w3U-IT Support: Goal Completion . Other:
- 41. w3U-IT-Support: Survey Comments – Primary Category Reasons Goal Not Completed + Suggestions for Improvement • 26%: Ticket & Help Desk issues • 21%: Search + Specific Topic issues • 7%: Technical issues (Failed to load, Too Slow, Login Failed, etc) • 7%: Missing features • 4%: Difficult / Cumbersome / Complicated 127 responses were collected, 127 responses were provided to this question. A random sample of 127 responses were analyzed, which resulted in 132 comments.
- 42. w3U-IT-Support: Survey Comments – Secondary Category Reasons Goal Not Completed + Suggestions for Improvement
- 43. User interviews and contextual inquiry 43
- 44. User interviews • Better for discovery research • Asking users questions typically with a moderator 1:1 • Great for understanding your users, their tasks, sentiments, needs and pain points. • Great for understanding which intents might be helpful for their needs and pain points • Can be used for persona creation and journey mapping • Limited in how many can conduct but get rich feedback Chat bot in a box | UXPA 2022
- 45. Contextual inquiry • Better for discovery research • Observing users in the work environments and the tasks they do • Can observe their needs and pain points for their tasks • Can observe and ask questions • Often can be a segway to journey maps • Limited in how many can conduct but get rich feedback Chat bot in a box | UXPA 2022
- 46. User interview scenarios for discovery research 1) You find out that Slack is currently down, where do you go to learn about and resolve this problem. Probe on: • Where would you go first? Why there? • Where else would you go? • Any problems/difficulties? • How could it be improved? 2) You are experiencing a problem with JIRA and cannot update your work items. Where do you go to learn about and resolve the issue? Probe on: • Where would you go first? Why there? • Where else would you go? • Any problems/difficulties? • How could it be improved? 3) You cannot access your Outlook calendar and urgently need to find out your meetings for the day. Where do you go to learn about and resolve the issue? Probe on: • Where would you go first? Why there? • Where else would you go? • Any problems/difficulties? • How could it be improved? Chat bot in a box | UXPA 2022
- 47. User interview scenarios for discovery research Scenario Pain points for searching for solution You find out that Slack is currently down, where do you go to learn about and resolve this problem. • Not clear when search “Slack support” in w3 search where to find solution in search results must search for “support” in w3 search then get support options • No phone/chat listed directly on support page • Might visit Slack.com instead of using w3 • Nothing about outages on IT support page • Nothing to directly create ticket or chat/call support fom IT support page You are experiencing a problem with JIRA and cannot update your work items. Where do you go to learn about and resolve the issue? • Do not see correct w3 search results for “JIRA update work items” or JIRA support” –JIRA help desk query not accurate result • Would first go to JIRA website as external product • Need search “support” in w3 search, then get support options, nothing directly on page no chat or phone, must search JIRA, then • Prefer use Google search than w3 search is better • expect see outages/notifications on w3 page You cannot access your Outlook calendar and urgently need to find out your meetings for the day. Where do you go to learn about and resolve the issue?
- 48. Journey mapping for pain points 48
- 49. Journey mapping for pain points • Useful for visualizing a user's workflow in their tasks from beginning to end and plotting where their pain points and problem areas occur in the task flow • Data can be gathered from user interviews and contextual inquiries • Can be useful when performing friction analysis in terms of breaking the task down and which points to keep aware of for pain points and increased friction Chat bot in a box | UXPA 2022
- 50. Journey map for searching for IT information for application problem User navigates to w3 and look for notifications/outages User goes to w3 search for issue and look for relevant results User navigates to get support options and chat/phone/community/create ticket Phase 1 Phase 2 Phase 3 Phase 4 1 2 3 4 5 6 User navigates to w3 website User looks for notifications/outages User searches w3 for issue User tries to find relevant search result User navigates to IT support to get support options User finds appropriate application they need help with 7 User begins chat…not clear they will get a ticket Pain point Pain point Chat bot in a box | UXPA 2022
- 51. Search w3 for “Slack support” or “Slack is down” Search w3 for “support” Slack@IBM channel Slack.com Slack at IBM - Support Slack User Guide w3 Support Support Chat/call IBM Help IT Support (formerly Help@IBM) Get support options Slack app Chat/call IBM Help Search for outages./notifications Call co-worker Google search “Slack is down” Search for IT information for Slack is down Post question in channel Look for anything on outages or notifications Ask if their Slack is down Slack system status page Chat bot in a box | UXPA 2022
- 52. Search w3 for “JIRA cannot update work items” or “JIRA not updating” Search w3 for “support” JIRA.com No useful search results IT Support (formerly Help@IBM) Get support options Search for JIRA Chat/call IBM Help Search for outages notifications Google search JRA Search for IT information for JIRA cannot update items Create ticket only if cannot resolve Atlassian page Search issue get documentation Search w3 for “JIRA support” JIRA help desk query Nothing useful Atlassian page Expect FAQ common issues encountered Chat bot in a box | UXPA 2022
- 53. Search w3 for “cannot see Outlook calendar” IT support link Got o Slack Go to Outlook calendar first results Search for Outlook calendar Read relevant articles if nothing then chat/call support Chat/call support Create ticket last resort only if articles, chat/call cannot solve or do not give them ticket after chat/call Search for IT information for Outlook cannot view calendar Post question in channel –UXRA channel about Outlook or use IBM support Slack channels IT support tab if nothing from Slack channels Search w3 for “IT support” Get support options Outlook Go to Outlook calendar first results Support- Community link post question in there Chat/call support Chat/call support assumes create ticket at the end. Get support options Outlook Chat/call support assumes create ticket at the end. Chat bot in a box | UXPA 2022
- 54. Personas 54
- 55. Personas • Can be created from data collected from user interviews, contextual inquiry and surveys • User personas are realistic representations of your key user groups that you can use for reference when making design and feature decisions. • At their core, personas are about creating products with specific users in mind. • The personas should reflect the current state of a product and how users interact with it today; they should not be used to demonstrate how a user will interact with the system in the future. Chat bot in a box | UXPA 2022
- 57. Friction analysis and PURE method • Friction analysis is a quantitative method that breaks tasks down into individual steps and provides a quantitative rating of the complexity and difficulty in completing a task. Chat bot in a box | UXPA 2022
- 58. Friction analysis and PURE method Benefits Drawbacks • Working under time constraints • Large number of tasks that need to be tested and compared • Not feasible to test users due to time constraints or length of testing required • Quantitative measure of task complexity and pain points most important • Just doing simplified friction analysis lacks deeper qualitative feedback you get from user interviews • You need at least one more rater for interrater reliability, need to resolve disputes Chat bot in a box | UXPA 2022
- 59. Friction analysis and PURE method • PURE = Practical User Experience Rating by Experts • Measure of friction, 3-point score • Total of scores for each step in a task • We will modify it for CIO purposes Chat bot in a box | UXPA 2022
- 60. Friction analysis and PURE method w3DA December 2021 w3DA May 2021 Including the steps link in the w3DA results in more direct solution rather than going to devices first. “I want to badge in at an IBM printer” Chat bot in a box | UXPA 2022
- 61. Surveys 61
- 62. Surveys • Can be used to validate findings from discovery research and for personas. • Can be used to gather qualitative and quantitively data around user needs, pain points, sentiments and attitudes toward towards applications, user tasks, and chatbots. • Can collect data from hundreds of respondents • Can be useful for creating personas using cluster analysis. • Can perform sentiment analysis on the open-ended questions. Chat bot in a box | UXPA 2022
- 63. Method • Survey was fielded from March 8 –March 16, 2022. • Survey took 5-10 minutes to complete. • 600 IBMers were sent an invite to the survey recruited from the UXRA participant database (DIG). • Used the UXRA survey platform to send out the invites. • 265 respondents responded to the survey (got nearly the 30% completion rate as expected). Chat bot in a box | UXPA 2022
- 64. Rank ordering of importance of various features that would be most useful for an IBM conversational AI Top ranking of features for conversation al AIs at IBM were focused on IT support Some comments that had this question been worded using “digital assistants” the rankings would have been different and more focused on tasks rather than conversational features. Chat bot in a box | UXPA 2022
- 65. Explanation for ranking of features for an IBM conversational AI 18.9% 15.0% 14.2% 7.1% 5.5% 4.7% 4.7% 4.7% 3.9% 3.9% 3.9% 3.1% 3.1% 2.4% 0% 5% 10% 15% 20% 25% Support for problems Order of usefulness Day to day activities most important New hire tasks most important Other items are easy Schedule meetings most important No need for it answer questions need help fast Notifications most important Most complex outages Great for automating tasks Random % of Total Categorized Comments (N=127) with 90% Confidence Intervals 253 responses were collected, 119 responses were provided to this question. A random sample of 119 responses were analyzed, which resulted in 127 comments. Question: Explain the reason for your ranking. Top reasons for rankings of the items were support for problems most important, ranked for order of usefulness, and day to day activities most important. Chat bot in a box | UXPA 2022
- 66. Activity (15 mins) • Create a test plan with appropriate methods for identifying user needs
- 68. What Makes a Chatbot Good? • Objectives as a conversation designer: • Easy • Intuitive • Intentional language choices • Feel more like a conversation with another human • User walks away feeling satisfied
- 69. Writing Tips • Active voice • Use contractions • Keep sentences short • Use formatting to improve readability (bulleted lists, paragraphs, commas, dashes) • Check for misspelled words and other grammar issues • Avoid “Yes" and "No" answers • Incorporate part of the user's question in response • Be succinct and accurate answers (link out to website for longer responses)
- 70. Erika Hall's Eight Principles for Conversation(al) Design Cooperative Turn-based Goal-oriented Context-aware Quick & clear Truthful Polite Error-tolerant
- 71. Principle 1: Cooperative • Back and forth flow • Mimic how people have conversations - pattern understood by the user
- 72. Principle 2: Goal-oriented • Main purpose - help users reach their goals • Chatbot should reflect this the user’s intentions • Help your customers achieve their goals helps your business achieve its goals
- 73. Principle 3: Context-aware • Don’t rely on overly automated messages • Can frustrate users • Does not feel like a natural conversation • Builds confidence in the user User: What are your holiday hours? Chatbot: Our normal hours of operation are …..
- 74. Principle 4: Quick and clear • Conversational, but as efficient as possible • Use plain language and keep it simple • Offer logical next steps • Keep the conversation moving • Stay on point User: How many flavors of ice cream do you have? Chatbot: We have a huge selection of ice cream flavors so we’re sure we have something for everyone.
- 75. Principle 5: Turn-based • Replies - succinct and to the point. No monologues! • Do not send a string of responses • Best conversations are back and forth dialogs • Be clear if response from the user is needed • Validate user input to avoid errors
- 76. Principle 6: Truthful • Validate are accurate and truthful • Plan for dialog maintenance. Answers change over time - may need to be updated • Verify it is behaving as expected • Responses should align with user expectations
- 77. Principle 7: Polite • Don’t be rude • Be respectful of your user’s time • Users come to a chatbot with a goal. Stay on topic • Don’t try to push your own agenda if not aligned • Once the chatbot has solved the user’s issue, offer to help with other issues, next steps
- 78. Principle 8: Error-tolerant • To errr is human, to forgive divine. • Teach chatbot common spelling alternatives or errors • Disambiguate the request as needed User: How do I reset my pwd? Chatbot: To reset your password…
- 79. Example of a poorly designed chatbot experience Hi, how can I assist you today? I need help with my wifi connection. I’m not sure how to help you with that. Can I speak with an agent? We have a promotion happening for 20% your monthly bill… Here are a few links that may be helpful: www.IT.resourcesdoc.com www.helpIT.com Hi, I need help with my internet connection… Hi, I need help viewing my Internet bill Your bill can be found by going to your billing under your account from the main menu. The chatbot provides no introduction and does not instruct what it can and cannot help with The chatbot provides no alternatives and is not empathetic to its lack of knowledge The chatbot ignore the request for an agent and references an irrelevant promotion The chatbot pushes generic links, nothing specific, more searching required on part of user and no better than a search with results Answer is disorganized, lacks steps and sequencing.
- 80. Best practices designing chatbots • Make sure your users know upfront what the chatbot can do and set expectations. • Providing specific answers to specific questions, don't just ignore the user’s question or push generic links. The snippets in Google search -- those are quite specific answers, short concise, and well formatted. • Present solutions in their task sequence of operations and label the step number in the solution with a number. • Consider removing obvious solutions or solutions the user could easily find elsewhere. • Ensure the chat bot follows up quickly with users after it gives an instruction or action and does not leave them hanging. • Ensure your chat bot has an engaging personality that resonates with your users. • Use buttons to allow users to quickly choose options instead of typing them out.
- 81. Best practices designing chatbots • Chat bot replies should be written from first person perspective as a human would respond. • Ensure your chat bot breaks up information by mimicking the flow of regular human messaging conversations • Ensure buttons are functional at all points of the conversation • Have your chat bot account for multiple word choices, phrasings and start options to allow for flexibility of interaction with your users. • Design your chat bots for possible misunderstandings at every step • Have the chat bot clarify the user’s question or answer if it detects an ambiguity before offering an answer, through a related question to remedy misunderstandings. • Consider including a ”did you mean” or please try rephrasing your question if the chat bot does not completely understand the user’s question. • Do not lock your users into a conversation with your chatbot and make it easy for your user to get to a human agent or back a main menu.
- 83. Response Types • Substantive - answers the question (q: what's your favorite movie? a: My favorite movie is Bladerunner b/c …). Core topics. • Deflecting - responds to the question without actually answering (q: what's the weather like in london? a: humans must be fascinated by the weather b/c they're always asking me about it!) Non-core topics. • Redirecting - redirects the user back to the core areas (q: what do you think of religion? a: religion is not my area of expertise, but I do know about ABC banking cards!)
- 84. Response Types: Disambiguation intent Confusion between 2 or more intents, with similar training phrases (utterances). Create an intent that helps system find the correct intent. When there is a vast list of supported options, conversational and open ended. But when the supported list is small, just list the options in a menu – don’t play 20 questions. “I want to buy something” Huge list of things you can buy Good open ended/conversational: What is it you want to buy? Bad menu: [Food] [Water] [Cars] [Weapons] [Rocks] [Plants] [Pets] [Livestock] [Real estate] [Bitcoin] [Bridges] [Gardening] [Books] [Movies] [Tickets] [Vacations] [Storage] [Tech] [Photo] [Drug] […] […] “I want to reset my password” Lots of types, just 3 supported Good menu: Which password do you want to reset? [Mac] [iPhone] [IBMid] Bad open ended/conversational What kind of password is it?
- 85. System statements • Welcome statements – sets the tone; consider return visits and time of day; may also inform user about the chatbot domain. • Interleaving solutions – follows final answers and encourage more questions; consider variations and useful tips. • Topping and tailing statements – indicates trust level, or whether more info is needed. • Return statements – return the user back to a process flow. • Did not know statements (low confidence or no match) “Hi, I’m HAL 9000. Welcome to Discovery Two. I see this is your second trip to Jupiter. You may ask me anything about your stay on the ship.” “Unfortunately, it is not possible to open the pod bay doors at this time. Got another question, just ask me!” “Here’s some information I found on the internet about how much spare oxygen an escape pod can hold. Check it out!” “Going back to where we were before you were accidentally ejected into space, would you like to resume our chess game? You were winning.” “I’m sorry but I do not know how you will survive without oxygen.”
- 86. Chit Chat & Small Talk • Exclamations (yes, no, okay, sure, oh) • Personal • Topic-based (e.g. Weather, geography) • Threats, insults, and pleas for help • Explicit and offensive language • Compliments • Complaints • Fallback or Repair (I don't understand, what do you mean, why)
- 87. Planning Dialog Flow • Ok! Now we’re ready to start thinking about the chatbot’s dialog! • Before you start writing the exact words, think about how the conversation with your users will flow for each intent you’ve identified. • The mapping will help you understand if there are redundant dialog nodes across scenarios and how your scenarios might interact.
- 88. Best practices for dialog flow • Consider the most common situations – Conversational scripts need to account for the most common scenarios. User research can help identify these. • Prepare different versions of an answer – sounds less robotic and more like natural conversations, which are varied, diverse, novel, or dynamic. • Work on the sequence of dialogue – Once you know the scenarios, think through the sequence of the dialogue so appropriate answers will be provided for specific queries. Source: https://www.revechat.com/blog/chatbot-scripts/
- 89. Practice Activity (10 mins) Writing intents for ordering pizza
- 90. Instructions: Design an intent • 5-7 mins/3-5 mins discussion • Use the worksheet • User’s intent: Ordering a pizza with toppings (#orderPizza) • Generate up to 3 utterances for that intent • Define toppings as an entity (@toppings) • Create different values for that entity • Create some synonyms for each value
- 91. Utterances for #orderPizza • i want a pizza • i'd like a pizza • give me a pie • please give me a pizza • order a pizza pie
- 92. Entity/slots @toppings: [value: synonyms] meat: pepperoni, sausage, ham, anchovies, chicken veggies: black olives, green olives, onions, bell peppers, jalapenos, artichoke cheese: mozzarella, white cheddar, asiago, three cheese blend values synonyms Entity label (category)
- 93. Utterances for #orderPizza(updated) • i want a @toppings pizza • i'd like a pizza with @toppings • give me a @toppings pie • please give me a @toppings pizza • order a pizza pie and I want @toppings on it
- 94. Responses within the dialog - Excellent choice! Pizza is our specialty. Do you want toppings? Yes | No Yes: Alright. Three toppings come for free. Each additional topping is $.50. List of available toppings: [list] No: jump to size - Sounds like you want a [variable] pizza. Do you want any other toppings? Yes | No Yes: Alright. Three toppings come for free. Each additional topping is $.50. List of available toppings: [list] No: jump to size - Let's make sure I have this right. You want a pizza with [selected options]. Is that correct? Yes: Great! What size do you want? [@size] No: Oops! Let's go over the toppings again. What toppings do you want on your pizza? - Ok, so you want a large. Is that correct? Yes | No Yes: Would you like anything else? We also have appetizers, desserts, and drinks. No: Sorry, I must have mozarella in my ears. What size pizza do you want?
- 95. Dialog Flow
- 96. PoohBot demo
- 97. Intent
- 98. Entities
- 99. Synonyms
- 100. Dialog
- 101. Team Activity (20 mins) Writing intents for the uses of honey
- 102. Instructions: Design an intent • 15 mins/5 mins discussion • Divide into teams • Use the worksheets • User’s intent: Things to do with honey (#thingsToDo) 1. Generate up to 5 utterances for that intent 2. Define honey uses as an entity (@uses) 3. Create different values for that entity 4. Create some synonyms for each value 5. Craft some responses for the entity or entity:value (@uses or @uses:value) BONUS INTENTS 1. Write a disambiguation intent related to #thingsToDo 1.What is the new intent? 2.What is the entity? 3.Any synonyms? 4.What are the utterances? 2. Write an example of a chit chat response with PoohBot’s tone
- 103. Utterances for #thingsToDo • can I put @honey in my @uses? • can you @uses with @honey? • can you make @uses with @honey? • how is @honey @uses? • is @honey only good for @uses? • what @uses are good with @honey? • what @uses can you make with @honey?
- 104. Entity @uses: [value: synonyms] antiseptic: clean wounds, cleaning wounds, boo boo, ouchie cooking: baking, salad dressing, marinade drinks: tea, milk, coffee, kool-aid, lemonade mead: alcohol, cocktail
- 105. Responses in Dialog Entity level @uses Hunny has so many great uses I smear it on everything Entity:value level @uses:antiseptic Whenever I get a boo boo, I put hunny on it, but I always forget and lick it off @uses:drinks Tea just isn't the same without hunny @uses:mead Mr. Owl really likes mead but he always acts so silly afterwards @uses:desserts Hunny makes yummy desserts, especially baklava @uses:cooking Piglet likes to marinade himself in a sauce made with hunny
- 106. Bonus intents! Disambiguation intent: Choose a type of honey Intent name: #chooseType Entity: @honeyType: [value: synonyms] honeyType: acacia, buckwheat, clover, eucalyptus, orange blossom, tupelo, sage Utterances: - I want to try some @honeyType @honey - My favorite @honey is @honeyType Chit chat: - On dear, hunny has so many uses, doesn’t it? - So wonderful you enjoy hunny as much as I do
- 107. Validating cognitive intents with users
- 108. Creating a user test plan • What is the research problem? • What are goals of research session • What are you testing? • What are scenarios? • How will you test participants? • Number of participants (typically 6-10) • Participant requirements • Outline procedure • Introductory text to read to participants (length of session, expectations, privacy, tasks…) • Background questions • Exploratory tasks • Specific scenarios • Probe questions • Post test questions Chat bot in a box | UXPA 2022
- 109. Chat bot in a box | UXPA 2022
- 110. Chat bot in a box | UXPA 2022
- 111. Chat bot in a box | UXPA 2022
- 112. Best practices for testing intents • Test the Chatbot Onboarding Process • Test the Chatbot Using Generic Templates • Check Link Navigation • Test the Chatbot’s Conversational Flow o Does my chatbot understand the user’s questions? o Does it respond promptly to them? o Are its responses accurate enough and relevant? o Are there enough conversation steps or… too many? o Does it keep the user engaged? • Run a Chatbot-Error Handling Test • Test the Chatbot’s Context Awareness • Test Text with Alternative but Valid Responses
- 113. Field Metrics Months when users are most and least active with chatbot queries Months when new users come use the chatbot the most and least Months when users abandon the chatbot the most
- 114. Field Metrics Buttons were used more than typing in this chatbot. 25% of queries fell below the confidence threshold set an could not be answered.
- 115. Activity (15 mins) • Update your test plan with methods for validating intents
- 117. Activity (10 mins) Interactive demo: Your honey intents played back in PoohBot
- 118. Bonus content!!
- 119. How is writing intents for voice bots different than for chatbots? • As a general rule of thumb it is recommendation to design voice bots that minimizes demand on working memory and attention required because of their reliance on the auditory system. • Keep conversation short so users do not have to interrupt and less reliance on working memory • Ensure user responses are brief as well • Use brief pauses liberally so users can being speaking without interrupting • Ask only necessary questions • Obtain information one piece at a time in consumable pieces • Be sensitive to word synonyms • Be considerate to speech accents • Privacy and background noise might be issues
- 120. Failure to meet user expectations • New chatbots lack a full blown knowledge corpus on launch • Build out the knowledge corpus to answer direct questions (e.g., point to a long video for the answer) • In 2018, Gartner expected that 40% of first- generation chatbot/virtual assistant applications launched in 2018 would be abandoned by 20208 • A chatbot that fails to live up to user expectations will quickly lose adoption (abandonment)
- 121. Cognitive Intercept: Overview • With cognitive intercept9, embed the chatbot technology into an existing user experience (single point of entry) • Monitors in the background, assesses the user’s input • Intercepts if it has high confidence • Deploy a chatbot that covers a narrow scope of topics without frustrating the user • Continue to gather user input to increase its breadth and depth of coverage, as well as improving accuracy User initiates action High confidence match? Continue with standard experience Intercept Cognitive layer screens input in background Yes No
- 122. Cognitive Intercept: Real World Example - No Match • Where cognitive intercept has been used: • Web search (first integration point) • Voice Response Unit (VRU) • Online chat • Users enter their input in a standard search field • System evaluates the input as both traditional search terms and using NLP • If a high-confidence match is not found, the “chatbot” remains silent • Running in the background effectively manages expectations around what chatbots can or cannot deliver
- 123. Cognitive Intercept: Real World Example - Match • If the chatbot is trained on a particular user intent and finds a high-confidence match, then the chatbot “intercepts” and displays associated dialog • Full conversations using NLP are avoided by using close-ended prompts to represent user options Cognitive intercept benefits: • Manages user expectations • Provides a single point of entry • Reduces errors associated with NLP • Permits a graceful ramp up of the chatbot’s knowledge corpus
- 125. References https://www.masswerk.at/elizabot/ https://www.chatbotpack.com/how-eliza-chatbot-works/ https://www.bold360.com/learn/what-is-natural-language-processing https://www.smallbizgenius.net/by-the-numbers/chatbot-statistics/#gref https://www.csmonitor.com/Science/2011/0214/IBM-s-Watson-Can-a-computer-outsmart-a-Jeopardy!-braniac https://chatbotsmagazine.com/to-build-a-successful-chatbot-ask-these-5-questions-b7fe3776c74 https://marvelapp.com/blog/principles-of-conversational-design/ https://www.gartner.com/smarterwithgartner/2-megatrends-dominate-the-gartner-hype-cycle-for-artificial-intelligence- 2020/ Temple, J.G., Elie, C., Schenkewitz, L., Nezbedova, P., Phipps, D., Roberson, K., Scherpa, J., & Malatesta, M. (June, 2019. Not Your Average Chatbot: Using Cognitive Intercept to Improve Information Discovery. Paper presented at the UXPA International Conference, Scottsdale, AZ.