Welcome to the Botium Knowledge Center

Table of Contents:

• How can I inspect detailed test case logs in Botium Box?
• I want to run tests in parallel - how do I do this?
• What are the "Scripting Settings" in Botium Box?

There are situations where your test cases are failing, but you cannot see the reason why this is happening from the test results. It would be handy to get some more insights what’s happening there.

In the Downloads section of the test session, it is possible to get the full transcript of the test session as JSON or CSV file.

It contains the full detailed transcript from the test session (and it can get pretty large in file size).

• test result

• all test cases

  • … with result

  • … and with all conversation steps

    • … with user input

    • … or with expected response

    • … and actual response

If you still have troubles understanding what’s going on, it is possible to run a test session in Debug Mode to see the Botium Core output line by line. Use the Start Debug Test Session Now button.

In the Test Session Jobs section of the Test Session view, there is the progress of each background processing job shown (see I want to run tests in parallel - how do I do this ?).

Click on a job to open up the log output view.

By scrolling down and clicking on Read More you can open the full log output in a separate window.

The Botium Box Agents are the work-horses, which actually run the test cases

The number of agents depends on the Botium Box flavour.

• With the free Botium Box Mini plan, there is a pool of agents available, shared by all users

• With the paid plans, there are dedicated agents with allow parallelization

Every agent will run exactly one test case at a time.

What’s happening behind the scenes is:

• There is a queue with test cases to process

• Every agent processes the test cases it receives from the queue as quickly as possible

  • As soon as an agent is ready with a test case, it immediately pulls the next one from the queue

As an example, let’s consider a scenario where 5 Botium Box Agents are available, and you have two test sets, BIGTESTSET with 1000 test cases, and SMALLTESTSET with 100 test cases.

By default, without any further configuration, the behaviour in Botium Box is to equally distribute the load onto all available agents by creating batches of test cases. So let’s say you start to tests in BIGTESTSET, and right afterwards in SMALLTESTSET.

• The queue is filled with five batches of 200 test cases out of the 1000 ones from BIGTESTSET

• The workers begin to process the test cases - each worker takes one of the five batches

• In addition, the queue is filled with five batches of 20 test cases out of the 100 from SMALLTESTSET

The outcome in this scenario is:

• All agents are first working on test cases from BIGTESTSET in parallel

• Afterwards, the test cases from SMALLTESTSET are processed

So, although there is parallel execution, you have to wait for the BIGTESTSET to finish until the test cases from SMALLTESTSET are processed.

To prevent a scenario like above, it is possible to define the degree of parallelity. In the Test Project Settings section, under Test Execution, you can set the Parallel Jobs Count field.

This number gives the number of batches that Botium Box will push to the queue for the test set.

Let’s extend the scenario from above - for BIGTESTSET, the Parallel Jobs Count is set to 3, for the SMALLTESTSET, it is set to 1.

• When launching BIGTESTSET, there are three batches created (334/334/332)

• Three of the agents begin to process it, two agents are still waiting for something to do

• For SMALLTESTSET, there are two batches created (50/50)

• The two idle agents are processing them

So now the test cases from two different test sessions are processed in parallel.

If there is only one Botium Box Agent available, there is no parallel test execution available.

If there are more Botium Box Agents available, by default the test cases for a test session are equally distributed for parallel execution.

If you want to run test cases from multiple test sessions in parallel, you can specify the degree of parallelity with the Parallel Jobs Count field.

In Botium Box, there are some settings hidden behind the “Scripting Settings” - what are they and how can I use them ?

The Text Matching mode is the logic to use for comparing the bot response to the utterance (see capability).

Let’s say, your bot is answering a question for an order number with this text:

Your order number is PO-15342

When using text matching mode Substring matching, this BotiumScript will succeed:

#bot
Your order number is

And this one as well:

#bot
order number

As long as the bot response includes the snippet from BotiumScript, it is fine.

There is also the option to switch off the case sensitivity.

Using the text matching mode wildcard, you can introduce simple wildcard matching in your assertions by using the asterisk - so this one will succeed:

#bot
Your order number is *

And this one as well:

#bot
Your order number is PO-*

There is also the option to switch off the case sensitivity, which is the default if nothing else is selected.

Using the text matching mode regular expressions, assertion only succeeds if the BotiumScript snippet interpreted as regular expression is matching the bot response.

For example, you can assert that the order number follows your conventions:

#bot
Your order number is PO-[0-9]+

Recommending this tool to quick test regular expressions.

Using the Normalize Text before Matching option you can simplify the bot response before doing the assertions, to make it easier to compose regular expressions or substring patterns. HTML tags are removed, line breaks are removed, non printable characters are removed, … (for details, you can inspect the Convo.js#L456" Botium Core code).

Your bot is now returning a more sophisticated answer:

Your order number is <a href="my-site/show/15342">PO-15342</a>

&gt; <a href="my-site/show/15342">Show PO</a>
&gt; <a href="my-site/show/15342/items">Show Items</>

The Substring matching examples from above will continue to work, but the regular expression certainly won’t. When using the Normalize Text before Matching option, this bot response is converted to this text before the matching is done:

Your order number is PO-15342 > Show PO > Show Items

No line breaks, no HTML tags, no HTML entities. The BotiumScript snippets from above will continue to work.

If you are new to Botium and don’t know about convos, utterances and scripting memory, please read Botium in a nutshell first.

Enable the Expand Conversations option if your Test Set contains utterances files to be merged with the convo files. See this BotiumScript convo:

#me
HELLO_UTT

And this BotiumScript utterances:

HELLO_UTT
hello
hi
what's up

If Expand Conversations is disabled, the literal text HELLO_UTT is sent to the bot (which is most likely not what you want).

If it is enabled, the Convo Expansion Mode comes into play to select the number of utterances to actually send to the bot - all, the first one, or randomly select some of them (see capability).

When having a simple two-step conversation, it is not a big deal to send all the utterances. But when you have more complex convo, the number of possible combinations increments very quick when letting Botium use all of them.

You can find a sample test set for Botium Box showing these options in action by importing the test set from this URL into Botium Box.

There are cases where you are not really interested in asserting conversations with your bot. Maybe you are just interested in sending as much text as possible to the bot (for training purposes), or you just want to make sure the bot understands all of the given texts.

When you enable the Expand Utterances to Conversations option, all utterances in the test set are converted to simple 2-step conversations (text sent to bot, bot answer received). For example, you have these utterances file:

HELLO_UTT
hello
hi
what's up

Botium converts it into three convos:

#me
hello

#bot

#me
hi

#bot

#me
what's up

#bot

All three utterances are sent to the bot, and the answer is ignored.

If you want to make sure your bot actually understands what you sent, it is possible to specify the Incomprehension Utterances, the utterances your bot is returning in incomprehension cases.

For example, in addition to the HELLO_UTT above, you have an INCOMPREHENSION_UTT utterances list:

INCOMPREHENSION_UTT
Sorry, I don't understand
Can you please repeat ?
Please say it another way

Now, when you enter INCOMPREHENSION_UTT in the Incomprehension Utterances field, Botium converts the test set into these three convos:

#me
hello

#bot
!INCOMPREHENSION_UTT

#me
hi

#bot
!INCOMPREHENSION_UTT

#me
what's up

#bot
!INCOMPREHENSION_UTT

Now Botium makes sure that for every utterance from the HELLO_UTT list, the bot actually doesn’t return a “don’t understand”-like phrase.

The Scripting Memory in Botium has several use cases and can be enabled with this option:

• Extract and reuse text from a bot response

• Use dynamic content in BotiumScript

• Running a single convo multiple times with different parameters

The Scripting Memory Matching Mode option is for finetuning the text extraction logic - see capability.

Enable the Enable Test Parameter Store option if you have scripting memory files for parameterizing your convos in the test set.

You can find a sample test set for Botium Box showing these options in action by importing the test set from this URL into Botium Box.