Skip to main content
  1. Botium Support
  2. Getting started
  3. Botium in a Nutshell

Part 2: Writing Chatbot Tests

While the first part of this series gave a general overview of Botium, this part deals with the daily business in software testing and automation, writing test cases.

Hello, World!

The most basic test case in Botium consists of:

  1. Submitting a phrase possibly entered by a real user to the chatbot
  2. Checking the response of the chatbot with the expected outcome
In Botium, the test cases are described by conversation flows the chatbot is supposed to follow. For a sample “greeting” scenario, the Botium test case looks like this — also known as “BotiumScript”:
#me
hello bot!
#bot
Hello, humanoid! How can I help you?

BotiumScript is an easy-to-use scripting language, which is interpreted by Botium Core in a virtual machine. Don’t worry if you didn’t understand a word of that sentence, it doesn’t really matter from a user perspective.

You can write BotiumScript as:

  • plain text file with Notepad or any other text editor
  • Excel file
  • CSV file (comma separated values)
  • and more …

For details please consult the Botium Documentation.

Convos and Utterances

So, let’s elaborate the “Hello, World!” example from above. While some users will say “hello”, others maybe prefer “hi”:
#me
hi bot!
#bot
Hello, humanoid! How can I help you ?
Another user may enter the conversation with “hey dude!”
#me
hey dude
#bot
Hello, humanoid! How can I help you ?

And there are plenty of other phrases we can think of. For this most simple use case, there are now at least three or more BotiumScripts to write. So let’s rewrite it.

We name this file hello.convo.txt:
TC01 - Greeting
#me
HELLO_UTT
#bot
Hello, humanoid! How can I help you ?
Note: You may have noticed the additional lines at the beginning of the BotiumScript. The first line contains a reference name for the test case to make it easier for you to locate the failing conversation within your test case library.
And we add another file hello_utt.utterances.txt:
HELLO_UTT
hello bot!
hi bot!
hey dude
good evening
hey are you here
anyone at home ?
  • The first BotiumScript is a convo file — it holds the structure of the conversation you expect the chatbot to follow.
  • The second BotiumScript is an utterances file — it holds several phrases for greeting someone, and you expect your chatbot to be able to recognize every single one of them as a nice greeting from the user.
Note: Botium will take care that the convo, and utterances files are combined to verify every response of your chatbot to every greeting phrase.
So now let’s assume that your chatbot uses several phrases for greeting the user back. In the morning it is:
#me
HELLO_UTT
#bot
Good morning, humanoid! How can I help you this early ?
And in the evening it is:
#me
HELLO_UTT
#bot
Good evening, humanoid! How can I help you at this late hour ?
Let’s extract the bot responses to another utterances file:
BOT_GREETING_UTT
Good evening
Good morning
Hello
Hi
And now comes the magic, we change the convo file to:
#me
HELLO_UTT
#bot
BOT_GREETING_UTT

Utterances files can be used to verify chatbot responses as well. To summarize:

  • An utterance referenced in a #me-section means: Botium, send every single phrase to the chatbot and check the response
  • An utterance referenced in a #bot-section means: Botium, my chatbot may use any of these answers, all of them are fine

Where to place BotiumScript files ?

The BotiumScript files can be placed in any folder structure you consider to be meaningful for your project. Botium will just scan the base directory recursively and find all of the files holding convos and utterances.

Tip: You can even mix plain text files, CSV files and Excel files within a test suite.

If you've gotten this far and understood everything, you are pretty much ready for using Botium. The rest of this article points to some advanced features, but the usage of convos and utterances is the main concept behind BotiumScript.

Note: Even when using Botium Box, you have to understand BotiumScript — Botium Box provides assistance when writing your BotiumScript test cases, but for most efficient use you really have to understand it.

Its useful to understand Botium Script in order to get the most out of the Botium Platform. Botium provides assistance when writing your BotiumScript test cases, but for the most efficient use you really have to understand it.

Using Excel Instead of Text Files

You can use Excel files (*.xlsx) instead of plain text files for describing your convos and utterances. Take this conversation as example:



First Excel sample

You can download the Excel file to use as a template here.

The A-column corresponds to the #me tag, the B-column corresponds to the #bot tag. For adding more than one convo to a worksheet, just leave one row empty:



Second Excel sample

You can use utterances here as well (and everything else supported by text files):



Using utterances in Excel (Convo)

Using utterances in Excel (Utterances)

All you have to tell Botium is:

  • The sheet names where to look for convos
  • The sheet names where to look for utterances
  • The row and column index where to start looking
Note: Please note: the header rows (in the screenshots above) are not relevant to Botium, they are just for the sake of visual appearance. By default, Botium starts to look for convos and utterances at row number 2 and column 1.

You can spread your convos over as many sheets as you want (and we recommend to do so) — see Botium Documentation for details.

Note: On the plus side, using Excel helps in building your convo structure in a very intuitive, visual way. On the negative side, code repositories are way better in handling text files than in handling binary files (like Excel).

Scripting Memory

BotiumScript can remember parts of the chatbot response for later use. In this example, the order number is reused in a later conversation step:
#me
please send me two salami pizza
#bot
OK. Your order number is $orderNum
#me
What is the status order $orderNum
#bot
Your order will arrive soon
Note: Scripting Memory is disabled by default. See Botium Documentation for details how to activate.

User Interface Elements

Lots of chatbots out there are delivering not only text but also pictures, audio or video to the user. And lots of chatbots speed up the conversation by providing simple user interface elements such as buttons.
#me
please send me two salami pizza
#bot
Please select the size of the pizza
BUTTONS Kids|Normal|Family
MEDIA kids_pizza.png|normal_pizza.png|family_pizza.png
The special keywords BUTTONS and MEDIA tell Botium to look out for buttons and media files in the chatbot response.
Note: This concept in BotiumScript is called “asserter”, and you can inject your own custom asserters into BotiumScript (for example, to verify that the chatbot has actually added the order to the order management backend).
Similar behaviour is available on the other end of the conversation as well:
#bot
What Pizza do you want ? Please choose an option.
#me
BUTTON Salami
This will simulate a button click or option selection.
#bot
Did you like your Pizza ?
#me
No. It was ugly.
MEDIA pizza_ugly.png

And this will simulate a picture attachment.

Note: Button click simulation and media attachments are not supported by all Botium connectors. Please consult the documentation of the connector in question. The default behaviour is to use the button title as message text.

For technical details please consult the Botium Documentation.

Splitting Convos

To keep your convo files clean, convos can be split and included in other convos. This concept is called partial convo in BotiumScript.

Here is a file partial_login.pconvo.txt:
PARTIAL_LOGIN
#bot
Please tell me your customer access code!
#me
1234567
#bot
Thanks, you are logged in now.
And here another one show_info.convo.txt:
SHOW_INFO
#me
Show my registration info.
INCLUDE PARTIAL_LOGIN
#bot
OK, here is your registration info:

The conversation steps from the partial convo are inserted at the INCLUDE statement (the partial convo is selected by name).

Frequently Asked Questions

Doesn’t it take a long time to write test cases ?

Yes, absolutely, writing test cases takes a lot of time. Botium gives you some tools to speed up the process:

  • The Live Chat tool to capture a conversation with your chatbot and generate a convo file out of it
  • Several importers for generating convo files out of chatbot development tools (such as Dialogflow and IBM Watson Assistant)
  • If you have existing conversation logs, they usually can be converted to convo files with reasonable effort
  • The Botium Platform includes hundred thousand of prebuilt test cases for different domains

My chatbot responses include dynamic or alternating text — how to handle this with BotiumScript ?

Apart from separating your testcases in convo and utterances files, it is possible to fine tune how Botium actually verifies the chatbot responses:

  • fixed matching
  • lowercase matching
  • substring matching
  • regular expression matching
  • or you can even implement your own logic and hook it into Botium

Can I use BotiumScript for training my chatbot, and how ?

Absolutely. Just send all of your convos to the chatbot and the training begins. How this is actually done depends on the chatbot technology you have chosen. When doing it with IBM Watson Assistant for example, in the Watson Workspace you get a list of the utterances the chatbot did not understand and you are supposed to train them for intents or entities.

For some providers there are even converters available to convert BotiumScript utterances file to the native intent resolution format of the provider.

Can I add the BotiumScript files to a source code repository like git ?

Yes, and we encourage you to do so.

I wrote a lot of BotiumScript files, how can I finally send them to my chatbot ?

See part 3 of this series.

Was this article helpful?

0 out of 0 found this helpful