Welcome to the Botium Knowledge Center

When designing a chatbot test strategy from scratch, there are often test case requirements like:

• We want to test the chatbot in multiple languages, but we don’t want to replicate the whole test set, as the conversations are basically the same in all our supported languages

• We want to run the tests in multiple enviroments - dev, test and production - and there are different user tokens to be used in each environment

If you can see yourself in one of the above points, then read on. This article will present a best practice technique to prepare Botium Box for those requirements.

These really are the basics of Botium - if you do not know what Convos and Utterances mean in Botium context, then please head over here. In short:

• With Convos you are describing the conversation structure of your tests, including user input and expected responses

• With Utterances you are describing lists of words or sentences

• Convos can reference Utterances to separate between the conversation structure and the actual phrasing

We will use this concept to handle multi-language requirements

With the Botium Scripting Memory it is possible to inject dynamically generated or static values into your test cases.

We will use this concept to set different conversation parameters for each environment your tests should run against.

In the end there will be a couple of test sets in Botium Box:

• There will be one test set holding the convo files valid for all conversations the should be available in all languages

• For each supported language, there will be a language-specific test set holding the utterances and any language-specific test cases

• For each environment, there will be a test set holding only the scripting memory

• For each combination of language + environment you have to run your tests, there will be one test set combining the partial test sets from above:

  • the convo files

  • the language-specific utterance files

  • the environment-specific scripting memory

Create a test set named Shared Convos in Botium Box. Add some first Convos in the Visual Convo Designer. The convos should map the conversation structure, but they should be free from any language-specific content. We have here a convo named TC_HELLO, which sends a default greeting to the chatbot, and expects a default greeting back - we will use utterance codes for it instead of literal phrases:

Here is the corresponding BotiumScript (for copy and paste):

TC_HELLO

#me
UTT_GREETING_DEFAULT

#bot
B_GREETING_DEFAULT

In another convo we send some kind of special greeting:

And again the BotiumScript:

TC_SPECIAL

#me
UTT_GREETING_SPECIAL

#bot
B_GREETING_DEFAULT

For each supported language, create a language-specific test set, name them for example Utterances EN and Utterances DE. To complete the shared convos from above, we need to define three utterance lists:

• UTT_GREETING_DEFAULT

• UTT_GREETING_SPECIAL

• B_GREETING_DEFAULT

In addition, these language-specific utterances should be free from environment-specific phrases, so we are already using the scripting memory in the UTT_GREETING_DEFAULT:

UTT_GREETING_DEFAULT
Chatid $chatid_default {"testId":"QA-$testcasename","testType":"regressionTest"}

Here we are using a scripting memory variable that will be different for each environment, the $chatid_default - and another one $testcasename to let Botium fill in the test case name on execution (see Botium Docs).

Similarily in the UTT_GREETING_SPECIAL, we will be using another variable $chatid_special

UTT_GREETING_SPECIAL
Chatid $chatid_special {"testId":"QA-$testcasename","testType":"regressionTest"}

In the utterance representing the chatbot response B_GREETING_DEFAULT we will use the variable $name_default, as the user names in the supported environments will be different - using a test user account in the test environment and a real user account (created specifically for testing) in the production environment is a common setting:

B_GREETING_DEFAULT
Hello $name_default, How can I help ?

Do the same for other supported languages. In german, this could look like this:

B_GREETING_DEFAULT
Hallo $name_default, wie kann ich helfen ?

The environment-specific parameters will be saved in Scripting Memory files. Create a test set named Params DEV and add a YAML-file named Scripting Memory:

In this file we define the variables that we used in the utterances above:

scriptingMemory:
  - header:
      name: dev
    values:
      $chatid_default: johnny@gmail.com
      $chatid_special: 4542343434343
      $name_default: Johnny

For the test set Params PROD this can look roughly the same, but with different variable values:

scriptingMemory:
  - header:
      name: prod
    values:
      $chatid_default: prodtester@gmail.com
      $chatid_special: 56565656565
      $name_default: ProdTest

Enable the Scripting Memory for these test sets in the Settings / Scripting Settings section

• Enable the switch Enable Scripting Memory

• Enable the switch Enable Test Parameter Store

Now create a test set EN DEV and set the Test Set Dependencies in Settings / Test Set Settings:

• Add the Shared Convos test set

• Add the Utterances EN test set

• Add the Params DEV test set

In a similar way you can now create test sets EN PROD, DE DEV and DE PROD by combinding the language- and environment-specific test sets.

You can see a nice graphical representation of the combined structure in the Flow section of the test set:

As a final step, use the Quickstart Wizard to create the test projects in Botium Box:

• Select the english language development version of the chatbot instance with the EN DEV test set

• Select eh german language production version of the chatbot instance with the DE PROD test set

• … and so on …

Everything is ready now for running your language- and environment-specific test sets.

The clear suggestion is to connect Botium Box to your own Git repository to setup a Continuous Testing pipeline. This principle can be applied to the structure established above as well.

In all of the Shared Convos, Utterances XX and Params XX test sets created above, set the Content Selection Type to Use Test Sets from Botium Box only , in order to not intervene with the Git repository content in the test sets you are developing within Botium Box.

For all of your Shared Convos, Utterances XX and Params XX test sets, now head over to the Linked Test Case Repositories section and register your Git Repository. You will need:

• Git Clone Url

• Username and password

• Git Branch (if not yet existing, don’t worry, you can create it later)

• And for each test set, use a separate folder in the Relative Path in Repository field

  • utterances_en for the Utterances EN test set

  • utterances_de for the Utterances DE test set

  • shared_convos for the Shared Convos test set

  • params_dev for the Params DEV test set

  • … and so on …

For all of the connected test sets you can now export the content to the Git repository by using the Export button in the Test Cases section.

Select the Git repository and the Git branch you want to export the content, enter a meaningful Git checkin comment and click the OK button. Botium Box will now export the repository content to the Git repository.

When doing this for the first few test sets, you will end up in some new directories in your Git repository.

You now have to repeat this export everytime you think it is required - once per day, once per sprint, on every change - that’s totally up to you. Remember that Git is there for tracking changes, so you should do this as often to get a meaningful tracking history in your Git repository.

Now we create a test set named EN DEV (Git) - the english-specific test set for the dev-environment loaded from the Git repository - head over to the Settings.

• In the Content Selection Type, choose Use Remote Repository only

• Enable the Scripting Memory and the Test Parameter Store as above for the Params XX test set

Afterwards, register the Git repository three times, each time pointing to one of three required folders:

• params_dev

• utterances_en

• shared_convos

To verify the content, now head over to the Test Set and click the Update Statistics & Insights button. This will make Botium Box download all of the content from the Git repository and build up some statistics and the flow chart for it.

In the flow chart you can now see the combined test set - the shared convos and the utterance examples.

As a final step, use the Quickstart Wizard to create the test projects in Botium Box.

• Select the english language development version of the chatbot instance with the EN DEV (Git) test set

• Select eh german language production version of the chatbot instance with the DE PROD (Git) test set

• … and so on …

Everything is ready now for running your language- and environment-specific test sets from a Git repository connected to Botium Box.