Chatbot testing
1 like714 views
This document discusses challenges in testing chatbots and proposes an approach. It describes how chatbots work using cognitive services and how they can be rule-based or AI-based. Key challenges in testing chatbots include dependency on large amounts of data, unpredictable user conversations, lack of testing strategy, and ensuring intelligence and security. The proposed approach includes testing chatbot APIs, regression, flows, domains, performance, security, and cognitive services. Advanced tools like an in-house chatbot named Ginger are demonstrated for testing. The importance of monitoring, automation, non-functional and domain-specific testing is emphasized.
Chatbot testing
- 1. © 2019 Nagarro – All rights reserved 1© 2019 Nagarro – All rights reserved Chatbot Testing: Challenges and approach for testing Chatbot applications Author: Rajni Singh & Neeraj Jain
- 2. © 2019 Nagarro – All rights reserved 2 Agenda Context Understanding Chatbots and real time examples of uncertainities Challenges Challenges in testing chatbot Testing Approach Scope, Approach & Tools and Use case
- 3. © 2019 Nagarro – All rights reserved 3© 2019 Nagarro – All rights reserved Understanding Chatbots, the evolution and its challenges Context 01
- 4. © 2019 Nagarro – All rights reserved 4 How it works? Chatbot application (Virtual assistant) Cognitive Services Business applications and Services http(s) Integration User input in plain language: • Text • Speech Channels allows user to connect to chatbot application Chatbot is a “computer program” which • Maintains a sensible and contextual conversation with a human • Operates on cognitive services such as NLP is used to make it more intelligent. • Processes business logic and prepare response to user input • Can take actions
- 5. © 2019 Nagarro – All rights reserved 5 Types of Chatbot Chabot can be broadly categorized in two categories. AI-BASED RULE BASED Built with artificial intelligence technique to understand the human language together with sentiments that does not rely on scripted conversation. It accepts free form of input and responds based on the existing domain knowledge with self-learning improvement with maturity. Chatbot behavior is based on a set of rules, flows, and triggers to respond to very specific commands being asked by the user. The conversation is usually scripted and chatbot responds to each question with a predefined rule and each step is picked with an explicit option. A simple example might be a chatbot that tells you the stock price on a given date.
- 6. © 2019 Nagarro – All rights reserved 6 Uncertainties: Why chatbot needs to be tested? Facebook Inc.’s chatbots hit a70% failure rate and zero emotional intelligence “Tay's responses have turned the bot into a joke, but they raise serious questions” In 2016, users were finding it impossible to unsubscribe WSJ Ticketmaster Breach Traces to Embedded Chatbot Software
- 7. © 2019 Nagarro – All rights reserved 7© 2019 Nagarro – All rights reserved Challenges 02 What could go wrong in Chatbot testing ?
- 8. © 2019 Nagarro – All rights reserved 8 03 0201 06 04 05 Dependency on humungous amount of data There is no defined input / output Continuous learning from past behavior Difficult to predict all scenarios AI applications are non- deterministic and probabilistic Continuous monitoring Shift from traditional testing to AI testing
- 9. © 2019 Nagarro – All rights reserved 9 Chatbot testing challenges Multichannel user experience Domain specific validations Uncertainty of user conversation Lack of robust testing strategy Chatbot intelligence not being questioned Chatbot security being compromised
- 10. © 2019 Nagarro – All rights reserved 10© 2019 Nagarro – All rights reserved Scope, parameters, and testing tools Approach 03
- 11. © 2019 Nagarro – All rights reserved 11 Chatbot testing scope API Regression Specialized testing Context remembrance and switching On Boarding & Personality Conversation flow Domain Specific Crowd Performance & Security NLP & Cognitive Services EnhancedUserExperience
- 12. © 2019 Nagarro – All rights reserved 12 Parameters for testing Navigation Error Management Speed Accuracy Onboarding Intelligence PersonalitySecurity
- 13. © 2019 Nagarro – All rights reserved 13© 2019 Nagarro – All rights reserved Meet Ginger Our in-house customized personal assistant
- 14. © 2019 Nagarro – All rights reserved 14 Any question about Nagarro Ask Ginger: Ginger Actionable insight that is customized to your role Ginger bytes: Notification about important event, conferences, announcement Ginger says:
- 15. © 2019 Nagarro – All rights reserved 15 Ask Ginger
- 16. © 2019 Nagarro – All rights reserved 16© 2019 Nagarro – All rights reserved Beyond traditional testing for AI-infused Chatbots
- 17. © 2019 Nagarro – All rights reserved 17 End-to-end automation testing framework Cloud/On-Premise Solution Conversational Management API Admin Linguistic Framework Intent Entity Synonyms Action Binding NLP Model Request Response Context Session User Configuration Security Training Integration LiveConnectors EnterpriseServices Training Set Data import-export
- 18. © 2019 Nagarro – All rights reserved 18 Goal completion rate Self-service rate AI and ML rates User retention rate Fallback rate Measure the performance of the bot
- 19. © 2019 Nagarro – All rights reserved 19 Data Model Manager RTMBAF Performance Monitoring Performance Tuning Performance MonitoringTraining Re-Train Measure Measure Enterprise Chatbot Model Read Client Interaction Watson LUIS Dialog flow RASA others Client Using Chatbot Client Questions NLP-based model life-cycle of the Chatbot
- 20. © 2019 Nagarro – All rights reserved 20 Advanced security testing 1 2 3 4 End-to-end encryption Two-factor authentication User authentication Intent authorization 5 Channel authentication 6 Compliance validation 7 Authentication timeout 8 Self-destructing messages
- 21. © 2019 Nagarro – All rights reserved 21 Domain-specific testing This testing should start with identifying the domain of the chatbot and then identify the keywords of that domain. A Chatbot built for IT should treat the word “Selenium” as an automation tool while a chatbot built for pharma industry should treat the “Selenium” as chemical.
- 22. © 2019 Nagarro – All rights reserved 22 Recommendations Monitor, measure and update (NLP, conversational etc.) Automate end to end (text, voice, etc.) Non-functional testing Domain-specific testing
- 23. © 2019 Nagarro – All rights reserved 23 HELPING YOU CREATE AN INTELLIGENT ENTERPRISE Thank You