Welcome to the Botium Knowledge Center

Table of Contents:

• What do the Botium Test results provide?
• Can I use Google Sheets to manage my test cases?
• Can I use JIRA to manage my test cases?
• Can I use Sharepoint to manage my test cases?
• What CI/CD Pipelines does Botium Box support?
• How Can I Simulate Buttons with my Azure Bot?
• How can I test a Bot Framework Chatbot without going over Directline?
• I don't know how to set a capability in Botium Box.

Each Test Result provides

• Bot configuration information
• Test Cases Used
• Number Success
• Number Failed
• Duration of Test Run
• Ability to quickly rerun the test session
• Access to Test Cases to quickly view the associated test cases and their steps - viewing pass/fail results
• Ability to download the results

The Training Progress shows several KPIs of the test session with regards to NLP analytics

• Number of test cases, conversations steps, asserters
• Predicted intents and entities
• Precision/Recall/F1-Score
• Number of expected vs predicted intents and entities

If there is a second test session selected for comparison, the trends from one test session to the next are also shown.

For NLP/NLU Testing, Cyara/Botium provides several key insights:

• Confusion Matrix / Precision / Recall / F1-Score
• Entity Confidence Deviation
• Entity Confidence Risks
• Entity Utterance Distribution
• Intent Confidence Deviation
• Intent Confidence Distribution
• Intent Confidence Risks
• Intent Mismatch Probability
• Intent Utterance Distribution
• Training Progress
• Confidence Threshold

Sample Test Results:

Scenario: you are using Google Sheets for writing Botium test cases (same format as in Excel).

Solution: register your Google Sheet in Botium Box as “Download Link”.

Create Google Project and Enable the Sheets API

1. Go to the  Google Developers Console

2. Select your project or create a new one (and then select it)

3. Enable the Sheets API for your project

   • In the sidebar on the left, select APIs & Services > Library

   • Search for "sheets"

   • Click on "Google Sheets API"

   • Click the blue "Enable" button

Create Google Service Account

1. Open the Google Project you created above

2. In the sidebar on the left, select APIs & Services > Credentials

3. Click blue "+ CREATE CREDENITALS" and select "Service account" option

4. Enter name, description, click "CREATE"

5. You can skip permissions, click "CONTINUE"

6. Click "+ CREATE KEY" button

7. Select the "JSON" key type option

8. Click "Create" button

9. Your JSON key file is generated and downloaded to your machine (it is the only copy!)

10. Click "DONE"

11. Note your service account's email address (also available in the JSON key file)

Share your Google Sheet with the Google Service Account

1. Open your Google Sheet

2. Share the sheet with the service account’s email address from above

3. Note the sharing link

Register Download Link in Botium Box

• Use the Google Sheet sharing link as download link

• Post the content of the service account JSON file in the form

• Click on Save and check the configuration by clicking the Download button - it will download the Google Sheet as Excel file (without any formatting applied, as they don’t matter for Botium test cases anyway).

Attention: Google Sheets API Usage Limits

Botium is using the Google Sheets API for reading the Google Sheet. If there are usage limits (quotas) in place, you can check them as described here

Scenario: you are using Jira issues for test case management. The Botium test cases are attached to Jira issues (for example, as Excel files).

Solution: register the attachment download link together with Jira credentials in Botium Box as “Download Link”.

Create a Jira API Token

In Jira, create an API Token: https://id.atlassian.com/manage/api-tokens

Find Attachment Download Link

1. Use a link in this form to retrieve the issue details with the Jira Rest API: https://my-jira-site.atlassian.net/rest/api/latest/issue/issue-key

2. In the resulting JSON document, search for the attachment section and get the content attribute: https://my-jira-site.atlassian.net/secure/attachment/attachment-id/attachment-filename

Register Download Link in Botium Box

1. In Botium Box, navigate to the test set (or create a new one), enter the Remote Repositories section and click Register Download Link

2. Enter anything in the Download Link Name field

3. In the Download Link field, enter the URL from the attachment content attribute (step 2 from previous section)

4. Enter your Jira credentials in the Username / Password fields

   1. Username is your Jira login email address

   2. Password is the API Token created above

   3. Click Add Basic Auth Headers to generate the HTTP Basic credentials

5. Save and you are ready to go.

Botium Box will now download this Jira attachment everytime when running a test session.

Scenario: you are using Excel files hosted in Sharepoint for writing Botium test cases

Solution: register the Sharepoint link in Botium Box as Download Link

Create a Sharepoint App Registration (see here) and add JSON with clientId and clientSecret attributes here. Assign read permissions.

Create a Sharepoint App Registration

1. See here how to create a Sharepoint app registration

2. Take note of the client id and the client secret

Register Download Link in Botium Box

• In your Test Set, navigate to Linked Test Case Repositories and click on Register Download Link

• Copy&Paste the sharepoint link. It has the form:

  https://*mysharepoint*.sharepoint.com/sites/*mysite*/Shared Documents/*My Botium Excel.xlsx*

• Enter your client id and client secret and click on SET SHAREPOINT CLIENT HEADERS to generate the JSON headers

• Click on SAVE

• You can now test the credentials by clicking DOWNLOAD

• By using FETCH FILE CONTENT Botium Box will try to download the file and evaluate if it contains Botium content.

Yes. Bamboo has excellent support for scripting, so best option for running test cases on Botium Box is to use a script task with cURL or Wget.

And remember to map the build environment variables to the Botium Endpoint parameters — you definitely want to map the bamboo.buildKey variable to the Botium Box BUILDID parameter!

Yes. For running test case on Botium Box, either use a simple shell script with cURL or Wget, or use an HTTP Request Plugin. Botium output as JUnit XML can be visualized with the Test Results Analyzer Plugin. See here for instructions.

Yes. Just use a script task with cURL or Wget.

Yes. Just use a script task with cURL or Wget.

Most likely, yes.

So, there are two very different things:

The first is, how does Botium identify that your Bot wants to show some buttons - this typically works out of the box, Botium recognizes the content types involved in delivering Microsoft Bot Framework content (including Adaptive Cards).

And the second part is, how does your bot backend logic expect that button clicks are happening ? There are several options when programming with the MS Bot Framework.

In most cases, the default Botium configuration works, as it is the way recommended by Microsoft:

In the "activity" delivered to your bot logic, the activity type is "event", and the activity attribute "name" holds the identifier for what button was actually clicked.

https://docs.microsoft.com/en-us/azure/bot-service/nodejs/bot-builder-nodejs-backchannel?view=azure-bot-service-3.0

So this is the default configuration in Botium.

We have seen other implementations, where button clicks are handled by the bot logic the same way as text input - for this, you have to configure Botium to use "message" as the activity type and "text" as the activity value field.

But it actually depends how you implemented it in your bot logic.

Find a description of the involved Botium capabilities in the Github repository for the Botium Directline3 connector.

The Botium Directline3 connector is configured as separate channel for an Azure Bot Service in the Azure Portal. Botium tests are always hitting the Azure servers - is it possible to run against a locally deployed Bot Framework app ? Yes.

Click here for more information on the Botium Directline3 connector

The Bot Framework Emulator includes a Directline-compatible endpoint for providing the live chat.

1. Start the emulator and connect to your bot

2. It will tell you what port it is listing on for Directline connections

   
   
   

   Unfortunately it is not possible to configure this port, so the emulator will always run on another port.

3. In Botium Box, configure your Directline connector - use any value for the Directline token, as this won’t be validated.

4. In the Advanced Mode, add an additional capability

   1. Name DIRECTLINE3_DOMAIN

   2. Type Text

   3. Value http://127.0.0.1:39027/v3/directline

      1. Replace 127.0.0.1 with the hostname where the emulator is running, or with host.docker.internal if running it on the same workstation as your Botium Box

      2. Instead of 39027 use the port you are

That’s it.

This project is available on Github and also provides a Directline-compatible endpoint to connect to. Follow the installation instructions on Github and launch it. Similar to above, add a DIRECTLINE3_DOMAIN capability to tell Botium to use the local Directline endpoint instead of the one on the Microsoft servers.

The advantage over the Bot Framework Emulator is that it can run on a headless server as background process and the listen port can be configured. The disadvantage is that it is not officially supported by Microsoft.

The Botium Connector for Bot Framework Endpoint is available here: https://github.com/codeforequity-at/botium-connector-botframework

Most important capability settings are part of the Botium Box UI. For custom or special capabilities not used very often, you can add the capabilities to the capability list in Botium Box.

In the Connector Settings of the Chatbot, enable the Advanced Mode to bring up the raw Botium capability list. It is possible to add additional ones there.

It is possible set capabilities on Test Project level. Those capabilites are then added to execution only when a test session is started from this Test Project.

!!! info Setting capabilities on Test Project level is mostly used for enforcing environment-specific customizations (DEV / TEST / STAGING / PROD system)

In the Test Project settings there is the Environment Settings section.