Welcome to the Botium Knowledge Center

When designing a chatbot test strategy from scratch, there are often requirements like:

• We are running a chatbot for all of our five brands. The conversations are basically the same for all of our brands, but there are slight differences.

• We have a development, a test and a production environment and we have to test on all of them

If you can see yourself in one of the above points, then read on. This article will present a best practice technique to prepare Botium Box for those requirements.

With the Botium Scripting Memory it is possible to inject dynamically generated or static values into your test cases. We will use this concept to set different conversation parameters for each environment your tests should run against.

For informatin about the scripting memory, see Botium Scripting Memory.

In the end there will be several new objects in Botium Box:

• There will be only one chatbot defined

• There will be one shared test set holding the test cases valid for all brands (with placeholders)

• For each brand, there will be a brand-specific test set with brand-specific test cases and brand-specific scripting memory

• For each combination of brand + environment you have to run your tests, there will be one test project combining:

  • the chatbot, enhanced with environment-specific capabilities

  • the brand-specific test set with the scripting memory files

  • the shared test set

In this example, we will use IBM Watson Assistant, but the same principle works for all supported technologies. We are connecting the chatbot to the Assistant's development workspace, so we can use it for developing the test cases. When running test cases later we will connect to the environment-specific Assistant workspaces by overwriting this from the Test Project.

Create a test set named Shared Convos in Botium Box. Add some first Convos in the Visual Convo Designer. The convos should map the conversation structure, and they should be free from any brand-specific content by using wildcards.

We have here a convo named TC_HELLO, which sends a default greeting to the chatbot, and expects a default greeting back:

Here is the corresponding BotiumScript (for Copy&Paste):

TC_HELLO

#me
Hello

#bot
Hello, this is Heinz, the chatbot of *. How can I help you ?

Note the use of the * as a wildcard - this is the spot where the brand name would be shown.

The above test case would assert that

• the chatbot introduces itself as Heinz

• and that there is any brand name included

But for your brands, you want to make sure that

• Each brand chooses a different name for the chatbot

• We want to additionally assert on the brand name (not accept just anything with the wildcard)

For each of the brands, create a new test set. The brand-specific parameters will be saved in Scripting Memory files. Create a test set named Params BRAND-1 and add a YAML-file named Scripting Memory:

In this file, we define the variables that we will use in our test cases, like chatbot name and brand name:

scriptingMemory:
  - header:
      name: heinz
    values:
      $chatbot_name: Heinz
      $brand_name: My first brand

For another brand, the test set Params BRAND-2 can look roughly the same, but with different variable values:

scriptingMemory:
  - header:
      name: anna
    values:
      $chatbot_name: Anna
      $brand_name: Another brand

Enable the Scripting Memory for these test sets in the Settings / Scripting Settings section

• Enable the switch Enable Scripting Memory

• Enable the switch Enable Test Parameter Store

The shared test cases from above now have to be changed to use the placeholders for the chatbot name and brand name, instead of just using a wildcard. Replace the corresponding spots in the test case with the variable name:

TC_HELLO

#me
Hello

#bot
Hello, this is $chatbot_name, the chatbot of $brand_name. How can I help you ?

This means that before running a test case, the variables are filled from the scripting memory files you defined upfront, therefore replacing those variables with concrete chatbot names and brand names for doing the assertions.

Now go the Test Projects / Register Test Project to combine everything from above and apply environment-specific settings.

• As Test Project Name choose something like BRAND-1 DEV

• Select the chatbot

• Select the Shared Convos Test Set and the brand-specific PARAMS BRAND-1 Test Set

• Save the Test Project and immediately head over to the Settings tab

In the Advanced section, it is now possible to overwrite the capabilities from the chatbot with the environment-specific settings. You can find the name of the capability (the basic configuration items for the Botium Connectors) either in the connector documentation, or in the Advanced Mode of the Chatbot connector settings.

In this case, we have to overwrite the IBM Watson Assistant workspace ID to connect it to a different workspace. Repeat the above steps for the other brands and environments, and name the Test Projects accordingly (BRAND-2 TEST, BRAND-1 PROD, ...).

Everything is ready now for running your brand- and environment-specific test cases.