Skip to main content
  1. Botium Support
  2. General
  3. Best Practice

How to Prepare for Large Test Sets

So, you've invested quite some time into chatbot test coverage by writing thousands of Botium test cases. That's awesome! Here are some hints on how to structure your test sets to make it easier to manage a large number of test cases.

Selecting the Right File Format

The scripting language behind Botium test cases is called BotiumScript and supports several file formats:

Note: When using the visual Convo Editor in Botium, in the background the plain text format is used.
While it might seem tempting to use Excel for writing your test cases because everyone out there is familiar with using spreadsheets, this quickly becomes hard to oversee and maintain when there are hundreds of test cases spread over multiple worksheets.
Tip: We suggest using one of the text-based formats and setting up a Git integration with Botium.

Establish Naming Conventions

This is maybe a no-brainer. The suggested naming conventions are:

  • Utterance names should start with UTT_

  • Partial convo names should start with PCONVO_

  • Convo names should start with anything you choose for naming your test cases, for example TC04711_…

Convos and Utterances

Consider splitting up your test cases in convo files and utterance files, in case you already didn’t do so. There are several benefits:

  1. A clear separation of concerns between the conversational flow and the actual conversation content

  2. Extensive NLP testing with Botium can only be reached with massive amount of user examples, and this is only doable with separate utterance files

  3. Some of the awesome tools in Botium are working exclusively on utterance level:

It makes it easier to build up multi-lingual test scenarios (see below)

Partial Convos

With partial convos it is possible to reuse parts of a convo in multiple places. For software engineers, partial convos provide modularization and separation of concerns. Use them at least in any of these cases:

  • When having a common greeting / goodbye section in all of your test cases, you can have two partial convos to be included in the “real” test cases.

  • If you have to setup conversation context before starting some test cases, the setup process should be extracted in a partial convo. A typical example is a user login process.

The awesome thing is that in partial convos, you can do everything that you can do in full convos, including utterance expansion, using scripting memory, assertions, logic hooks and more. Read more about Partial Convos and How to Use Them.

Scripting Memory

There are three ways to use the Botium scripting memory.
  1. Extract and Reuse Dynamic Content in Test Cases: With scripting memory variables it is possible to extract dynamic content from chatbot responses in test cases and reuse this dynamic content.

    For more information, see Scripting Memory Variables.

  2. Inject Dynamic Content into Test Cases: With scripting memory functions it is possible to inject dynamic data into your test cases. Example usage:

    • Use current date and time in various formats

    • Generate random numbers in various formats

    • Inject system environment variables

    • Use test case metadata in test cases itself (test session name, test case name, …)

    • Extract custom payload

    • and more …

    See Scripting Memory Functions.

  3. Parametrize Test Cases: With scripting memory files it is possible to set parameters for test cases and let them run multiple times with different parameters.

    For more information, see Scripting Memory Files. This comes in handy when testing same conversation flow for different content, for example testing a shopping workflow for various products.

Test Set Dependencies

In Botium it is possible to define dependencies between test sets. When running test sessions, all content from dependent test sets is loaded as well. Possible use cases:

  • When working on multiple chatbot projects, you can have basic test cases for greeting, goodbye, smalltalk etc in a shared Smalltalk test set and set this as dependency for the specialized test sets

  • When working on multi-lingual chatbot projects you can have multiple test sets:

    • one shared test set holding the convo files with the conversational flow of the test cases - only utterance names are used here, no text content

    • for each supported language on test set holding the utterance files for this language used in the convo files, and maybe some additional language-specific test cases

    • the language test sets are dependent on the shared test set holding the conversational flow



Multi-Lingual Testing

Preparing Multi-Language and Multi-Environment Test Sets

Multi-Brand Testing

Preparing Multi-Brand and Multi-Environment Tests

Botium Crawler

Use the Botium Crawler for detecting uncovered branches and updating your test sets accordingly. A suggested setup is similar to the suggested development workflow:

  • Run Botium Crawler for the first time to detect all conversation flows

  • Copy the resulting test scripts into an empty test set and do a first-time export to the main branch of a linked Git repository

  • At time, re-run the Botium Crawler to detect new conversation flows

  • Copy the resulting test scripts into another empty test set and export to the same linked Git repository as before, but into a feature branch

  • Use Git pull requests to review the detected changes in the test set

Was this article helpful?

0 out of 0 found this helpful