Welcome to the Botium Knowledge Center

Each Test Result provides

• Bot configuration information
• Test Cases Used
• Number Success
• Number Failed
• Duration of Test Run
• Ability to quickly rerun the test session
• Access to Test Cases to quickly view the associated test cases and their steps - viewing pass/fail results
• Ability to download the results

The Training Progress shows several KPIs of the test session with regards to NLP analytics

• Number of test cases, conversations steps, asserters
• Predicted intents and entities
• Precision/Recall/F1-Score
• Number of expected vs predicted intents and entities

If there is a second test session selected for comparison, the trends from one test session to the next are also shown.

For NLP/NLU Testing, Cyara/Botium provides several key insights:

• Confusion Matrix / Precision / Recall / F1-Score
• Entity Confidence Deviation
• Entity Confidence Risks
• Entity Utterance Distribution
• Intent Confidence Deviation
• Intent Confidence Distribution
• Intent Confidence Risks
• Intent Mismatch Probability
• Intent Utterance Distribution
• Training Progress
• Confidence Threshold

Sample Test Results:

Scenario: you are using Google Sheets for writing Botium test cases (same format as in Excel).

Solution: register your Google Sheet in Botium as “Download Link”.

Create Google Project and Enable the Sheets API

1. Go to the  Google Developers Console

2. Select your project or create a new one (and then select it)

3. Enable the Sheets API for your project

   • In the sidebar on the left, select APIs & Services > Library

   • Search for "sheets"

   • Click on "Google Sheets API"

   • Click the blue "Enable" button

Create Google Service Account

1. Open the Google Project you created above

2. In the sidebar on the left, select APIs & Services > Credentials

3. Click blue "+ CREATE CREDENITALS" and select "Service account" option

4. Enter name, description, click "CREATE"

5. You can skip permissions, click "CONTINUE"

6. Click "+ CREATE KEY" button

7. Select the "JSON" key type option

8. Click "Create" button

9. Your JSON key file is generated and downloaded to your machine (it is the only copy!)

10. Click "DONE"

11. Note your service account's email address (also available in the JSON key file)

Share your Google Sheet with the Google Service Account

1. Open your Google Sheet

2. Share the sheet with the service account’s email address from above

3. Note the sharing link

Register Download Link in Botium

• Use the Google Sheet sharing link as download link

• Post the content of the service account JSON file in the form

• Click on Save and check the configuration by clicking the Download button - it will download the Google Sheet as Excel file (without any formatting applied, as they don’t matter for Botium test cases anyway).

Attention: Google Sheets API Usage Limits

Botium is using the Google Sheets API for reading the Google Sheet. If there are usage limits (quotas) in place, you can check them as described here

Scenario: you are using Jira issues for test case management. The Botium test cases are attached to Jira issues (for example, as Excel files).

Solution: register the attachment download link together with Jira credentials in Botium as “Download Link”.

Create a Jira API Token

In Jira, create an API Token: https://id.atlassian.com/manage/api-tokens

Find Attachment Download Link

1. Use a link in this form to retrieve the issue details with the Jira Rest API: https://my-jira-site.atlassian.net/rest/api/latest/issue/issue-key

2. In the resulting JSON document, search for the attachment section and get the content attribute: https://my-jira-site.atlassian.net/secure/attachment/attachment-id/attachment-filename

Register Download Link in Botium

1. In Botium, navigate to the test set (or create a new one), enter the Remote Repositories section and click Register Download Link

2. Enter anything in the Download Link Name field

3. In the Download Link field, enter the URL from the attachment content attribute (step 2 from previous section)

4. Enter your Jira credentials in the Username / Password fields

   1. Username is your Jira login email address

   2. Password is the API Token created above

   3. Click Add Basic Auth Headers to generate the HTTP Basic credentials

5. Save and you are ready to go.

Botium will now download this Jira attachment everytime when running a test session.

Scenario: you are using Excel files hosted in Sharepoint for writing Botium test cases

Solution: register the Sharepoint link in Botium as Download Link

Create a Sharepoint App Registration (see here) and add JSON with clientId and clientSecret attributes here. Assign read permissions.

Create a Sharepoint App Registration

1. See here how to create a Sharepoint app registration

2. Take note of the client id and the client secret

Register Download Link in Botium

• In your Test Set, navigate to Linked Test Case Repositories and click on Register Download Link

• Copy&Paste the sharepoint link. It has the form:

  https://*mysharepoint*.sharepoint.com/sites/*mysite*/Shared Documents/*My Botium Excel.xlsx*

• Enter your client id and client secret and click on SET SHAREPOINT CLIENT HEADERS to generate the JSON headers

• Click on SAVE

• You can now test the credentials by clicking DOWNLOAD

• By using FETCH FILE CONTENT Botium will try to download the file and evaluate if it contains Botium content.

So, there are two very different things:

The first is, how does Botium identify that your Bot wants to show some buttons - this typically works out of the box, Botium recognizes the content types involved in delivering Microsoft Bot Framework content (including Adaptive Cards).

And the second part is, how does your bot backend logic expect that button clicks are happening ? There are several options when programming with the MS Bot Framework.

In most cases, the default Botium configuration works, as it is the way recommended by Microsoft:

In the "activity" delivered to your bot logic, the activity type is "event", and the activity attribute "name" holds the identifier for what button was actually clicked.

https://docs.microsoft.com/en-us/azure/bot-service/nodejs/bot-builder-nodejs-backchannel?view=azure-bot-service-3.0

So this is the default configuration in Botium.

We have seen other implementations, where button clicks are handled by the bot logic the same way as text input - for this, you have to configure Botium to use "message" as the activity type and "text" as the activity value field.

But it actually depends how you implemented it in your bot logic.

Find a description of the involved Botium capabilities in the Github repository for the Botium Directline3 connector.

The Botium Directline3 connector is configured as separate channel for an Azure Bot Service in the Azure Portal. Botium tests are always hitting the Azure servers - is it possible to run against a locally deployed Bot Framework app ? Yes.

Click here for more information on the Botium Directline3 connector

Most important capability settings are part of the Botium UI. For custom or special capabilities not used very often, you can add the capabilities to the capability list in Botium.