A common question that is asked when deciding to test chatbots with Botium is, "What is the enhanced value of testing with Cyara/Botium verses testing with the vendor tools supplied?". Some of the key differentiators with Cyara/Botium are listed in this article.
Innovate Faster
- Runs at the API Level: Directly against NLP engine
- Regression testing done to assure you deliver accurate answers
- Continuous regression testing keeps up with that assurance
- NLP score testing - provides static and dynamic analytics to make sure the bot is understanding and working properly
- Testing at API level provides quick results for testing (100,000 test cases can run in minutes)
- Cyara/Botium provides testing on all channels, web, mobile, voice, apps for end 2 end testing
- Allows you to connect to device farms or providers
Mitigate Risk
- Security Testing
- With web or mobile interfaces you need to integrate security testing
- Easy to use in Cyara/Botium - turn it on and Cyara/Botium acts as a proxy between your tests and your bot (runs through an end to end test to your bot and the communication is running thru a security proxy and testing for around 300 vulnerabilities)
- The standards are provided and maintained by the web security community itself (OWASP)
- Reporting available within Cyara/Botium - easy to understand and evaluate the vulnerabilities
- Cyara/Botium provides passive and active scanning capabilities
- Active scanning should be done in your test environment because it tries to “break” the bot intentionally
- Performance Testing
- Run stress and load tests against bot
- Will generate a high load for a specific time (defined within Cyara/Botium)
- During a performance test, your Ops team should check logs and analytics on your system as well
- Goal is to discover the bots limits; start low with a number of parallel users and increase it over time to see where the breakpoints are
- Allows you to know when it’s in production, when action might need to be taken
Maximize Value
- Live Monitoring
- Test your main user cases in production
- Detect potential user errors with the bot before your users do
- Identify those bot use cases that are critical to the business and detect any issues early on
- Email notifications standard for alerting out of Cyara/Botium; optionally integrate from your CI-CD pipeline (Jenkins)
- Use Cyara/Botium to compare technologies to use for the Bot (DialogFlow vs IBM, etc)
- You pick a provider - then can compare another technology to see if there would be improvements
- Cyara/Botium can import conversation models from various design tools and could then compare auto-generated test cases for comparison
- During development, run crawler, users can create conversation flow to compare to unit testing
- During dev, Cyara/Botium has support for many speech processing systems to compare text to speech and speech to text iterations
- During training data creation, use Cyara/Botium to generate potential use cases based on what the bot is going to do - generate 1000’s of training data records in seconds; multiple languages, with humanification, and paraphrasing
Increase Efficiency
- Humanification - those real life scenarios that simulate what actual humans do (do this after happy paths are working)
- Intelligence/Context Testing - what does the bot remember and how long does it remember it for (during session, next hour, next day, next week, etc.)
- Profile testing -depends on the bot implementation, are all bots out there collecting and handling user info (who you are, your family if applicable - insurance example, data from crm) - may require using with other means of testing, check the database, call in, see if bot usage changed anything
- Navigation/Routing - sending user to the right place based on data pulled
- Escalation protocols - test human handovers