Table of Contents:
- What do the Botium Test results provide?
- Can I use Google Sheets to manage my test cases?
- Can I use JIRA to manage my test cases?
- Can I use Sharepoint to manage my test cases?
- What CI/CD Pipelines does Botium support?
- How Can I Simulate Buttons with my Azure Bot?
- How can I test a Bot Framework Chatbot without going over Directline?
- I don't know how to set a capability in Botium.
What do the Botium Test results provide?
Each Test Result provides
- Bot configuration information
- Test Cases Used
- Number Success
- Number Failed
- Duration of Test Run
- Ability to quickly rerun the test session
- Access to Test Cases to quickly view the associated test cases and their steps - viewing pass/fail results
- Ability to download the results
The Training Progress shows several KPIs of the test session with regards to NLP analytics
- Number of test cases, conversations steps, asserters
- Predicted intents and entities
- Precision/Recall/F1-Score
- Number of expected vs predicted intents and entities
If there is a second test session selected for comparison, the trends from one test session to the next are also shown.
For NLP/NLU Testing, Cyara/Botium provides several key insights:
- Confusion Matrix / Precision / Recall / F1-Score
- Entity Confidence Deviation
- Entity Confidence Risks
- Entity Utterance Distribution
- Intent Confidence Deviation
- Intent Confidence Distribution
- Intent Confidence Risks
- Intent Mismatch Probability
- Intent Utterance Distribution
- Training Progress
- Confidence Threshold
Sample Test Results:
- Regression
- Performance
- NLP
and
- End2End Testing
- Monitoring
- Security
Can I use Google Sheets to manage my test cases?
Scenario: you are using Google Sheets for writing Botium test cases (same format as in Excel).
Solution: register your Google Sheet in Botium as “Download Link”.
Create Google Project and Enable the Sheets API
-
Go to the Google Developers Console
-
Select your project or create a new one (and then select it)
-
Enable the Sheets API for your project
-
In the sidebar on the left, select APIs & Services > Library
-
Search for "sheets"
-
Click on "Google Sheets API"
-
Click the blue "Enable" button
-
Create Google Service Account
-
Open the Google Project you created above
-
In the sidebar on the left, select APIs & Services > Credentials
-
Click blue "+ CREATE CREDENITALS" and select "Service account" option
-
Enter name, description, click "CREATE"
-
You can skip permissions, click "CONTINUE"
-
Click "+ CREATE KEY" button
-
Select the "JSON" key type option
-
Click "Create" button
-
Your JSON key file is generated and downloaded to your machine (it is the only copy!)
-
Click "DONE"
-
Note your service account's email address (also available in the JSON key file)
Share your Google Sheet with the Google Service Account
-
Open your Google Sheet
-
Share the sheet with the service account’s email address from above
-
Note the sharing link

Register Download Link in Botium
-
Use the Google Sheet sharing link as download link
-
Post the content of the service account JSON file in the form
-
Click on Save and check the configuration by clicking the Download button - it will download the Google Sheet as Excel file (without any formatting applied, as they don’t matter for Botium test cases anyway).

Attention: Google Sheets API Usage Limits
Botium is using the Google Sheets API for reading the Google Sheet. If there are usage limits (quotas) in place, you can check them as described here
Can I use JIRA to manage my test cases?
Scenario: you are using Jira issues for test case management. The Botium test cases are attached to Jira issues (for example, as Excel files).
Solution: register the attachment download link together with Jira credentials in Botium as “Download Link”.
Create a Jira API Token
In Jira, create an API Token: https://id.atlassian.com/manage/api-tokens
Find Attachment Download Link
-
Use a link in this form to retrieve the issue details with the Jira Rest API: https://my-jira-site.atlassian.net/rest/api/latest/issue/issue-key
-
In the resulting JSON document, search for the attachment section and get the content attribute: https://my-jira-site.atlassian.net/secure/attachment/attachment-id/attachment-filename
Register Download Link in Botium
-
In Botium, navigate to the test set (or create a new one), enter the Remote Repositories section and click Register Download Link
-
Enter anything in the Download Link Name field
-
In the Download Link field, enter the URL from the attachment content attribute (step 2 from previous section)
-
Enter your Jira credentials in the Username / Password fields
-
Username is your Jira login email address
-
Password is the API Token created above
-
Click Add Basic Auth Headers to generate the HTTP Basic credentials
-
-
Save and you are ready to go.

Botium will now download this Jira attachment everytime when running a test session.
Can I use Sharepoint to manage my test cases?
Scenario: you are using Excel files hosted in Sharepoint for writing Botium test cases
Solution: register the Sharepoint link in Botium as Download Link
Create a Sharepoint App Registration (see here) and add JSON with clientId and clientSecret attributes here. Assign read permissions.
Create a Sharepoint App Registration
Register Download Link in Botium
https://*mysharepoint*.sharepoint.com/sites/*mysite*/Shared Documents/*My Botium Excel.xlsx*

What CI/CD Pipelines does Botium support?
Does it support Bamboo ?
Yes. Bamboo has excellent support for scripting, so best option for running test cases on Botium is to use a script task with cURL or Wget.
And remember to map the build environment variables to the Botium Endpoint parameters — you definitely want to map the bamboo.buildKey variable to the Botium BUILDID parameter!
Does it support Jenkins ?
Yes. For running test case on Botium, either use a simple shell script with cURL or Wget, or use an HTTP Request Plugin. Botium output as JUnit XML can be visualized with the Test Results Analyzer Plugin. See here for instructions.
Does it support Codeship ?
Yes. Just use a script task with cURL or Wget.
Does it support Travis ?
Yes. Just use a script task with cURL or Wget.
Does it support build pipeline product X from vendor Y ?
Most likely, yes.
How Can I Simulate Buttons with my Azure Bot?
So, there are two very different things:
The first is, how does Botium identify that your Bot wants to show some buttons - this typically works out of the box, Botium recognizes the content types involved in delivering Microsoft Bot Framework content (including Adaptive Cards).
And the second part is, how does your bot backend logic expect that button clicks are happening ? There are several options when programming with the MS Bot Framework.
In most cases, the default Botium configuration works, as it is the way recommended by Microsoft:
In the "activity" delivered to your bot logic, the activity type is "event", and the activity attribute "name" holds the identifier for what button was actually clicked.
So this is the default configuration in Botium.
We have seen other implementations, where button clicks are handled by the bot logic the same way as text input - for this, you have to configure Botium to use "message" as the activity type and "text" as the activity value field.
But it actually depends how you implemented it in your bot logic.
Find a description of the involved Botium capabilities in the Github repository for the Botium Directline3 connector.
How can I test a Bot Framework Chatbot without going over Directline?
The Botium Directline3 connector is configured as separate channel for an Azure Bot Service in the Azure Portal. Botium tests are always hitting the Azure servers - is it possible to run against a locally deployed Bot Framework app ? Yes.
Click here for more information on the Botium Directline3 connector
Option 1: Use the Bot Framework Emulator
The Bot Framework Emulator includes a Directline-compatible endpoint for providing the live chat.
-
Start the emulator and connect to your bot
-
It will tell you what port it is listing on for Directline connections
Unfortunately it is not possible to configure this port, so the emulator will always run on another port.
-
In Botium, configure your Directline connector - use any value for the Directline token, as this won’t be validated.
-
In the Advanced Mode, add an additional capability
-
Name DIRECTLINE3_DOMAIN
-
Type Text
-
Value http://127.0.0.1:39027/v3/directline
-
Replace 127.0.0.1 with the hostname where the emulator is running, or with host.docker.internal if running it on the same workstation as your Botium
-
Instead of 39027 use the port you are
-
-
That’s it.
Option 2: Use offline-directline
This project is available on Github and also provides a Directline-compatible endpoint to connect to. Follow the installation instructions on Github and launch it. Similar to above, add a DIRECTLINE3_DOMAIN capability to tell Botium to use the local Directline endpoint instead of the one on the Microsoft servers.
The advantage over the Bot Framework Emulator is that it can run on a headless server as background process and the listen port can be configured. The disadvantage is that it is not officially supported by Microsoft.
Option 3: Connect to the Bot Framework Endpoint
The Botium Connector for Bot Framework Endpoint is available here: https://github.com/codeforequity-at/botium-connector-botframework
I don't know how to set a capability in Botium.
Most important capability settings are part of the Botium UI. For custom or special capabilities not used very often, you can add the capabilities to the capability list in Botium.
Chatbot Capabilities
In the Connector Settings of the Chatbot, enable the Advanced Mode to bring up the raw Botium capability list. It is possible to add additional ones there.
TestProject Capabilities
It is possible set capabilities on Test Project level. Those capabilites are then added to execution only when a test session is started from this Test Project.
!!! info Setting capabilities on Test Project level is mostly used for enforcing environment-specific customizations (DEV / TEST / STAGING / PROD system)
In the Test Project settings there is the Environment Settings section.
Comments
0 comments
Please sign in to leave a comment.