Welcome to the Botium Knowledge Center

Yes, related to test sets, test cases, test results. System Settings are configurable for each customer instance as a global setting: Clients will have the ability to configure the length of time to store results for. Configurable items include: (Cleanup Interval (in minutes) Keep Screenshots for x number of days for succ/fail test cases, Keep details of success/failed tests x number of days, keep NLP analytics in x number of days)

Cyara/Botium will verify context handling and the conversation flow of the bot. It can crawl the bot to discover the full conversational flow. Using Cyara/Botium you can identify flaws in the conversation flow before it goes into production. Run through thousands of test cases quickly and evaluate the results. Understand that when your bot is asked a question, does it reply the correct response, does it fall back to a default "I'm sorry" or misroute to the wrong intent.

Asserters can be used as additional validators for conversations. For example, if you want to check if the links sent by the bot are valid references, you can use the asserter called HyperLinkAsserter, which will try to reach the sent links. Every step in a multi-step conversation usually gets checked with an asserter: For more information on Asserters click here.

I want to book a flight, where from, where to?

The following asserters can be used to test flows with multi-turn logic:

• INTENT (arguments: intent name to look out for), used in #bot section, will assert that the bot answered with the specified intent.
• INTENT_CONFIDENCE (arguments: minimal accepted confidence, like “70” for 70%), used in #bot section, will assert that the bot answered with at least the specified minimal confidence.
• INTENT_UNIQUE (no arguments), used in #bot section, will assert that the recognized intent is unique (not alternate intent with same confidence identified).
• ENTITIES (arguments: expected entities like “from|to”, or minimal entities like “from|…” ), used in #bot section, will assert that bot answered with the specified entities.
• ENTITY_VALUES (arguments: expected entity values like “2018|2019”, or minimal entity values like “2018|…” ), used in #bot section, will assert that bot answered with the specified entity values.
• ENTITY_CONTENT (arguments: entity and expected values like location|Budapest|Vienna)
  • One ENTITY_CONTENT asserter checks only one entity. Use more asserters to check more.
  • Does not fail if the response has more values as specified in arguments.

Cyara/Botium supports cloud providers, but you can also connect to your self-hosted Selenium server. For connection to Saucelabs the following is supported: For Saucelabs, the Selenium Endpoint typically is https://ondemand.saucelabs.com/wd/hub, the Username and the Access Key you can find on the User Settings section in the Saucelabs dashboard. For Saucelabs Real Device Cloud (former TestObjects), you have to create a Test App first, switch to the Appium view and go to the Setup Instructions — you will find the Selenium Endpoint and the API Key there. For Saucelabs the device list is retrieved in real time via the Saucelabs API.

Depending on the support within the chatbot provider you can set up connectivity to multiple environments (development, UAT, production, etc.). Cyara/Botium supports testing at the API level for component testing and user Interface testing for system tests (website, social, etc.) & voice testing if desired.

You can reuse test sets across different environments by creating specific projects.

For NLP testing you can conduct static analytics against a standard language model against your existing training data by accessing the data via an API. Existing training data can be imported via file format as well (csv, josn, yaml, xlsm, zip). With Cyara/Botium, you can "improve" this data by using the AI Assisted Generator, Humanification Transformer, or Paraphraser, and then you may export it from Cyara/Botium and import it into your Chatbot as the "New and Improved" version of the training data.

By connecting to your bot, you can conduct NLP testing and review dynamic results based on how your bot responds to the NLU test data.

Sample Static Test Set Results: These analytic reports are provided comparing data in a test set to a standard language model:

We are making enhancements to the reporting and analytics portion details of the GDPR test results. For a better understanding of how Cyara/Botium tests against GDPR:

1. Automated Test: Ask your chatbot questions that it should be able to answer according to GDPR standards. For example: “What are you doing with my data?”
2. Next Step: Audit- Cyara/Botium enables you to conduct an audit, by answering a checklist of questions. For example: "Is there a data protection regulation for your chatbot?"
3. Based on these answers, Cyara/Botium generates GDPR test results and suggestions where correction or improvement is needed.
   1. Security: Assessing your chatbot’s ability to answer data security related questions. For example: “How do you protect my data?”
   2. Control: Determining how well customers can exercise their rights while interacting with your chatbot. For example: “Delete my personal data”
   3. Transparency: Ensuring your chatbot provides customers with full information regarding GDPR inquiries. For example: “What happens to my data?”
   4. Privacy: Checking that your chatbot can properly answer general questions about privacy. For example: “What is your privacy policy?”

Cyara/Botium crawler detects quick replies and buttons and simulates clicks on all of the options and follows the paths until it reaches the configured depth or found and open-ended question. You can then edit the flow to complete an answer to an open question and continue the crawl. A button appears in a conversation test case as such: