How to Effectively Test Your Chatbot | Rasa Summit
0 likes165 views
The document discusses effective testing strategies for chatbots, highlighting issues such as manual testing, lack of centralized dashboards, and challenges with bot versioning and multilingual support. It emphasizes the need for automated testing, scenario creation, and proper entity mapping to improve chatbot resiliency and prevent failures. Additionally, it suggests tracking key performance indicators like retention rate and user feedback to enhance testing outcomes.
How to Effectively Test Your Chatbot | Rasa Summit
- 1. How to ”Effectively” ”Test” your Chatbot Soumya Mukherjee Director QA, DevOps & AIML Apty.IO
- 2. How are we doing our QA today • Testing is Blackbox for testers • Mostly manual testing done in organization • Conversational flow testing • Small Talk • Fallback checks • Integrations • Automation done on UI and API layer • Testing is mostly done on same training data • Models are trained by engineers and are not being monitored by QA • There are analytics tools available to monitor but it needs technical expertise for the QA • Result : More than 90% times bot breaks (no one understands when it will break), most of them fallback and get stuck - once bot is stuck it is stuck Q ? A
- 3. What are the issues in QA ? • Bots are evolving and continuous story creation is a problem • No tool manage story coverage • Your training data may not correspond to new stories or vice versa (it’s a mismatch) – most org keep training on the same data • Most automation tools offers record and playback (My stories are already written how to port is the question)
- 4. What are the issues in QA ? • No (unified) centralized dashboard present where QA can check (everything is quite scattered) • Intent Matching • Entity Testing – Slot identification • Entity Testing – Entity Validation • Confidence score • Confusion Matrix along with Precision/Recall/F1-Score • No easy way to reset the failed bot ! • Bot versioning is a mess and A/B testing becomes difficult • Multilingual bot QA is a challenge (have to make 2 separate bots) • High confidence score is also a problem as your bot will only predict same thing (if the data is same for multiple intents then it will predict the one with highest confidence score – may be incorrect) How to make sure your bot never breaks ?
- 5. How to make your test effective ? • Create scenarios for happy path, contextual questions, digressions, domain specific questions, stateless conversations • Map proper entities for common scenarios (example bus fee, tuition fee) – flow should change with entities in the stories • Automated tests should consume all stories and run them each time as part of regression testing • Story coverage visualization • For Manual Testing use Bot emulation product (like RasaX, Botfront) to test
- 6. How to make your test effective ? • Central dashboarding including : • Confusion matrix, Precision, Recall and F1-Score • Cumulative accuracy profile • Cross validation results • Perform Exhaustive testing (bot resiliency), Integration checks across platforms, Webhooks • Perform fault tolerance testing by performing performance testing (bot response, session management) & security testing (api interaction, typing speed check, punctuations, typo errors)
- 7. Other KPIs to track • Activity Volume • Bounce rate • Retention rate • Open sessions count • Session times (conversation length) • Goal completion rate • User feedback (sentiments) • Fallback rate (Confusion rate, reset rate & Human takeover rate)