Preparing Multi-Language and Multi-Environment Test Sets

Features and Techniques Overview

For setting up Botium we will use several techniques that are actually independent of each other, but in combination they are incredibly powerful.

• Convos and Utterances: These really are the basics of Botium - if you do not know what Convos and Utterances mean in Botium context, then please head over here. In short:

  • With Convos you are describing the conversation structure of your tests, including user input and expected responses

  • With Utterances you are describing lists of words or sentences

  • Convos can reference Utterances to separate between the conversation structure and the actual phrasing

  We will use this concept to handle multi-language requirements

• Scripting Memory / Test Parameter Store: With the Botium Scripting Memory it is possible to inject dynamically generated or static values into your test cases. We will use this concept to set different conversation parameters for each environment your tests should run against.

  For information about the scripting memory, see Using the Scripting Memory.

• Test Set Dependencies: In Botium it is possible to define dependencies between test sets and combine them into a single test set. We will use this technique to separate the different requirements into individual test sets and combine them as needed.

Resulting Test Sets

In the end there will be a couple of test sets in Botium:

• There will be one test set holding the convo files valid for all conversations the should be available in all languages

• For each supported language, there will be a language-specific test set holding the utterances and any language-specific test cases

• For each environment, there will be a test set holding only the scripting memory

• For each combination of language + environment you have to run your tests, there will be one test set combining the partial test sets from above:

  • the convo files

  • the language-specific utterance files

  • the environment-specific scripting memory

Step by Step

Now comes the interesting part - follow those steps to setup the basic structure in Botium:

1. Create a Shared Convos Test Set:

   Create a test set named Shared Convos in Botium. Add some Convos in the Visual Convo Designer. The convos should map the conversation structure, and they should be free from any language-specific content.

   1. We have here a convo named TC_HELLO, which sends a default greeting to the chatbot, and expects a default greeting back - we will use utterance codes for it instead of literal phrases:
      
      
      Note: Example BotiumScript (for Copy & Paste):

      TC_HELLO
      
      #me
      UTT_GREETING_DEFAULT
      
      #bot
      B_GREETING_DEFAULT

   2. In another convo we send some kind of special greeting:
      
      

      Note: Example BotiumScript:

      TC_SPECIAL
      
      #me
      UTT_GREETING_SPECIAL
      
      #bot
      B_GREETING_DEFAULT

2. Create Language-Specific Utterance Test Sets:

   For each supported language, create a language-specific test set, name them for example Utterances EN and Utterances DE.

   1. To complete the shared convos from above, we need to define three utterance lists:

      • UTT_GREETING_DEFAULT

      • UTT_GREETING_SPECIAL

      • B_GREETING_DEFAULT

      
      
      

   2. In addition, these language-specific utterances should be free from environment-specific phrases, so we are already using the scripting memory in the UTT_GREETING_DEFAULT:

      UTT_GREETING_DEFAULT
      Chatid $chatid_default {"testId":"QA-$testcasename","testType":"regressionTest"}

      Here we are using a scripting memory variable that will be different for each environment, the $chatid_default - and another one $testcasename to let Botium fill in the test case name on execution.

      Tip: See the list of Scripting Memory Functions in the article Using the Scripting Memory to learn more.

   3. Similarly, in the UTT_GREETING_SPECIAL, we will be using another variable $chatid_special:

      UTT_GREETING_SPECIAL
      Chatid $chatid_special {"testId":"QA-$testcasename","testType":"regressionTest"}

   4. In the utterance representing the chatbot response B_GREETING_DEFAULT we will use the variable $name_default, as the user names in the supported environments will be different - using a test user account in the test environment and a real user account (created specifically for testing) in the production environment is a common setting:

      B_GREETING_DEFAULT
      Hello $name_default, How can I help ?

      Note: Do the same for other supported languages. In German, this could look like this:

      B_GREETING_DEFAULT
      Hallo $name_default, wie kann ich helfen ?

   5. Enable the Scripting Memory: Enable the scripting memory for the test set in Botium Tools & Settings > Test Sets > Your Test Set > Configuration > Scripting

      • Enable the switch Enable Scripting Memory

      • Enable the switch Enable Test Parameter Store

      
      
      

3. Create Environment-Specific Scripting Memory Test Sets: The environment-specific parameters will be saved in Scripting Memory files:

   1. Create a Test Set named Params Dev, and add a YAML-file named Scripting Memory:
      Tip: Scripting Memory can be defined in other file formats as well - plain text, JSON, Excel - but we found that the YAML representation is the most concise one: easy to write and to understand.
   2. In this file we define the variables that we used in the utterances above:

      scriptingMemory:
        - header:
            name: dev
          values:
            $chatid_default: johnny@gmail.com
            $chatid_special: 4542343434343
            $name_default: Johnny

      
      
      
      Note: For the Test Set Params PROD this can look roughly the same, but with different variable values:

      scriptingMemory:
        - header:
            name: prod
          values:
            $chatid_default: prodtester@gmail.com
            $chatid_special: 56565656565
            $name_default: ProdTest

4. Enable the Scripting Memory: Enable the scripting memory for the Test Set in Botium Tools & Settings > Test Sets > Your Test Set > Configuration > Scripting

   • Enable the switch Enable Scripting Memory

   • Enable the switch Enable Test Parameter Store

   
   
   

5. Combine Test Sets into a Language-Environment-Specific Test Set: Create a new Test Set named EN DEV and set the following Test Set Dependencies

   • Add the Shared Convos test set

   • Add the Utterances EN test set

   • Add the Params DEV test set

   
   
   
   Note: In a similar way you can now create test sets EN PROD, DE DEV and DE PROD by combining the language and environment-specific test sets.

6. Create Test Projects: As a final step, use the Test Suite to create a test project in Botium:

   • Select the English language development version of the chatbot instance with the EN DEV Test Set

   • Select the German language production version of the chatbot instance with the DE PROD Test Set

   • and so on …

   Everything is now ready for running your language - and environment-specific test sets.