After a Bot is deployed, what is Cyara/Botium leveraged for?
Continuous development and improvements to the bot require continuous testing. So as a phase 1+ test, you will continue to test against your bot with both existing data and new utterance data.
I.E. If you track that your bot is not recognizing specific phrases asked by customers, you would continue to make changes to your bot and thus build up your testing data with new utterance examples and rerun the regression test with all of the collective data. Cyara/Botium sits within your devops lifecycle to allow for continuous development, testing and improvement. Cyara/Botium can automate your testing cycles as well providing the capability to run 1000’s of tests in minutes, as compared to manually testing your bot providing less coverage and is prone to errors.
Regression Test runs can be evaluated to compare a prior release to the new release. Additionally, Cyara/Botium can discover the bot once changes are built within the bot to create the new conversation flow design and all supporting test cases to provide regression testing.
What is the effort to set up and evaluate an existing bot?
Using your credentials into your existing bot, Cyara/Botium can download the conversational testing data and provide static analytics as compared to a standard language model.
Understanding Static/Dynamic Analytics
Static Analytics are driven from Botium by comparing an uploaded test set, existing test set, or manually created test set to a standard language model using proprietary algorithms written by Botium. These analytics have not been used in the running of a test against the client’s bot, yet. The results from static analytics are predicted based on what is likely to be interpreted correctly from a bot.
-
Overview Screen
-
Insights Screen
Notice the available analytics and tabs under the three main graphs
Note:Dynamic Analytics are available when you have used a Test Set to run against a specific Bot or NLU Engine and the data is different. It is “dynamic” because it is measured against an actual Bot.
These results are found under Test Results for the NLP test type. There you will notice the different tabs listed for the results. The dynamic results are the proof that what you have built works, and is proven with testing.
-
Confusion Matrix
