Table of Contents:
- Once connected to the client’s bot does data live on Cyara/Botium’s database?
- Does Cyara/Botium store transcripts on their cloud?
- What is Cyara/Botium testing for regression testing?
- How does Cyara/Botium test for a multiple step answer and entities?
- How do we test multi turn logic?
- Saucelabs being used for mobile and web testing. How is this connected?
- Where does testing occur?
- GDPR testing results
- If the conversation includes buttons, how would that appear?
Once connected to the client’s bot does data live on Cyara/Botium’s database?
Training data and test data as defined on Cyara/Botium resides in our platform. We don't provide any updates to the actual bot, but the means to update the bot by exporting data from Cyara/Botium.
Test Sets: Test Sets contain the data, scripts and other artifacts that represent your test cases. You can think of a test set as a folder where your test data lives.
Test Projects: You actually run the test cases by registering a Test Project and hitting the Start button. Within a Test Project you tell Cyara/Botium:
- which test cases you want to run
- on which chatbot channel
- when you want to run them
- and what happens with the test results
So a Test Project is composed from:
- a chatbot to define the communication channel
- one or more test sets, which are themselves containing the test cases
- configuration what aspects you want to test (test features)
- a preconfigured webhook to be integrated into your CI/CD pipeline
- and a historic view of the finished test results and test cases
Test ResultsA Test Result is the outcome of a single test execution, usually by starting test execution in a test project. You can see the total results as well as drill down to the single test cases and conversation steps causing trouble
Does Cyara/Botium store transcripts on their cloud?
Yes, related to test sets, test cases, test results. System Settings are configurable for each customer instance as a global setting: Clients will have the ability to configure the length of time to store results for. Configurable items include: (Cleanup Interval (in minutes) Keep Screenshots for x number of days for succ/fail test cases, Keep details of success/failed tests x number of days, keep NLP analytics in x number of days)
What is Cyara/Botium testing for regression testing?
Cyara/Botium will verify context handling and the conversation flow of the bot. It can crawl the bot to discover the full conversational flow. Using Cyara/Botium you can identify flaws in the conversation flow before it goes into production. Run through thousands of test cases quickly and evaluate the results. Understand that when your bot is asked a question, does it reply the correct response, does it fall back to a default "I'm sorry" or misroute to the wrong intent.
How does Cyara/Botium test for a multiple step answer and entities?
Asserters can be used as additional validators for conversations. For example, if you want to check if the links sent by the bot are valid references, you can use the asserter called HyperLinkAsserter, which will try to reach the sent links. Every step in a multi-step conversation usually gets checked with an asserter: For more information on Asserters click here.
How do we test multi turn logic?
I want to book a flight, where from, where to?
The following asserters can be used to test flows with multi-turn logic:
- INTENT (arguments: intent name to look out for), used in #bot section, will assert that the bot answered with the specified intent.
- INTENT_CONFIDENCE (arguments: minimal accepted confidence, like “70” for 70%), used in #bot section, will assert that the bot answered with at least the specified minimal confidence.
- INTENT_UNIQUE (no arguments), used in #bot section, will assert that the recognized intent is unique (not alternate intent with same confidence identified).
- ENTITIES (arguments: expected entities like “from|to”, or minimal entities like “from|…” ), used in #bot section, will assert that bot answered with the specified entities.
- ENTITY_VALUES (arguments: expected entity values like “2018|2019”, or minimal entity values like “2018|…” ), used in #bot section, will assert that bot answered with the specified entity values.
ENTITY_CONTENT (arguments: entity and expected values like
- One ENTITY_CONTENT asserter checks only one entity. Use more asserters to check more.
- Does not fail if the response has more values as specified in arguments.
Saucelabs being used for mobile and web testing. How is this connected?
Cyara/Botium supports cloud providers, but you can also connect to your self-hosted Selenium server. For connection to Saucelabs the following is supported: For Saucelabs, the Selenium Endpoint typically is https://ondemand.saucelabs.com/wd/hub, the Username and the Access Key you can find on the User Settings section in the Saucelabs dashboard. For Saucelabs Real Device Cloud (former TestObjects), you have to create a Test App first, switch to the Appium view and go to the Setup Instructions — you will find the Selenium Endpoint and the API Key there. For Saucelabs the device list is retrieved in real time via the Saucelabs API.
Where does testing occur?
Depending on the support within the chatbot provider you can set up connectivity to multiple environments (development, UAT, production, etc.). Cyara/Botium supports testing at the API level for component testing and user Interface testing for system tests (website, social, etc.) & voice testing if desired.
You can reuse test sets across different environments by creating specific projects.
For NLP testing you can conduct static analytics against a standard language model against your existing training data by accessing the data via an API. Existing training data can be imported via file format as well (csv, josn, yaml, xlsm, zip). With Cyara/Botium, you can "improve" this data by using the AI Assisted Generator, Humanification Transformer, or Paraphraser, and then you may export it from Cyara/Botium and import it into your Chatbot as the "New and Improved" version of the training data.
By connecting to your bot, you can conduct NLP testing and review dynamic results based on how your bot responds to the NLU test data.
Sample Static Test Set Results: These analytic reports are provided comparing data in a test set to a standard language model:
GDPR testing results
We are making enhancements to the reporting and analytics portion details of the GDPR test results. For a better understanding of how Cyara/Botium tests against GDPR:
- Automated Test: Ask your chatbot questions that it should be able to answer according to GDPR standards. For example: “What are you doing with my data?”
- Next Step: Audit- Cyara/Botium enables you to conduct an audit, by answering a checklist of questions. For example: "Is there a data protection regulation for your chatbot?"
- Based on these answers, Cyara/Botium generates GDPR test results and suggestions
where correction or improvement is needed.
- Security: Assessing your chatbot’s ability to answer data security related questions. For example: “How do you protect my data?”
- Control: Determining how well customers can exercise their rights while interacting with your chatbot. For example: “Delete my personal data”
- Transparency: Ensuring your chatbot provides customers with full information regarding GDPR inquiries. For example: “What happens to my data?”
GDPR is this running against an API?
GDPR testing can be done via API or via UX level.
If the conversation includes buttons, how would that appear?
Cyara/Botium crawler detects quick replies and buttons and simulates clicks on all of the options and follows the paths until it reaches the configured depth or found and open-ended question. You can then edit the flow to complete an answer to an open question and continue the crawl. A button appears in a conversation test case as such: