Skip to main content
  1. Botium Support
  2. Test Sets
  3. Testing Strategy and Test Case Authoring

Asserting on Custom Chatbot Content

When programming with chatbots or when testing chatbots you often notice that beside the plain text payload there is often additional payload delivered to and from the chatbot programming interface. Potential use cases are:

  • Navigation options (quick replies) presented to the user

  • Rich user interface elements to be shown on graphical output interfaces (websites, smartphone apps, Alexa Echo Show, ...)

  • Meta-Information about the conversation, such as context or session variables, user login information, ...

  • Internal processing information for programming and/or debugging purposes, such as NLU information

Standardized vs Custom Payload

For chatbot content that follows the standards of the chatbot engine, usually the Botium Connector is smart enough to use this additional payload - but there are also cases when this payload is fully customizable and cannot be supported out-of-the-box by Botium.

For example, Botium usually can handle:

  • Quick replies and other clickable elements

  • File and media attachments, such as pictures, videos, audio files

  • User interface elements such as carousels, adaptive cards and forms

  • NLU information such as intents, entities and confidence

Here is an example of chatbot content that is not supported out-of-the-box by Botium - to handle disambiguation by presenting other options to the user like or did you mean this (full content can be found at the end of this article):
{
  "sourceData": ['
    {
      "text": "hi",
      "intent": ...,
      "response_selector": {
        "all_retrieval_intents": [
          "faq",
          "chitchat",
          "out_of_scope"
        ],
        "out_of_scope": ...,
        "faq": {
          "response": {
            "id": -7277286380353916000,
            "response_templates": [
              {
                "text": "You can use Rasa to build assistants in any language you want!"
              }
            ],
            "confidence": 0.22302119433879852,
            "intent_response_key": "faq/languages",
            "template_name": "utter_faq/languages"
          },
          "ranking": ...,
        },
        "chitchat": ...,
      }
    },
    ...
  ],
  ...,
  "messageText": "Hey there, my name is Sara."
}
Botium includes the JSONPath Asserter for adding your own assertions for custom payload to your Botium test cases.

Showing Custom Payload

The easiest way to view a custom payload in Botium is to use the Live Chat feature.

When interacting with a live chat you can view the code for any step by clicking the code icon to the right </>.

When this is clicked, the actual payload is revealed. You can inspect it and use it for writing your own JSONPath Asserter statements.

Compose JSONPath Queries

Go the an online JSONPath evaluator and copy and paste the payload from Botium to the Inputs field. Strip it down to only include the sourceData attribute, as this is the element the Botium JSONPath Asserter operates on.
Note: It is important to only use the sourceData attribute from here on, otherwise your JSONPath queries will fail.
You can now use the JSONPath evaluator to write your JSONPath queries.
Example: In this example, I want to make sure that the Frequently Asked Questions are included in the alternative responses.
$[0].response_selector.faq.response.response_templates[0].text


Using the JSONPath Asserter

Using the Visual Test Case Editor in Botium, you can add a Generic JSONPath asserter to the convo step, using the JSONPath query from above, and the expected value of the retrieved payload element:

Example Botium Script: In BotiumScript, this looks like this:
My Convo

#me
hello

#bot
JSON_PATH $[0].response_selector.faq.response.response_templates[0].text|You can use Rasa to build assistants in any language you want!
Tip: If you do want to assert that the element exists, but not the actual content you can use the Generic JSONPath Count asserter instead:
My Convo

#me
hello

#bot
JSON_PATH_COUNT $[0].response_selector.faq.response.response_templates[0].text|=1
If you do want to assert that the element exists, but not the actual content you can use the asserter instead:

Appendix: Full JSON Content

{
  "sourceData": [
    {
      "text": "hi",
      "intent": {
        "id": 3679827850957151700,
        "name": "greet",
        "confidence": 0.9999986290931702
      },
      "entities": [],
      "intent_ranking": [
        {
          "id": 3679827850957151700,
          "name": "greet",
          "confidence": 0.9999986290931702
        },
        {
          "id": 6429607567961177000,
          "name": "ask_which_events",
          "confidence": 5.311148925102316e-7
        },
        {
          "id": 5176782347177008000,
          "name": "affirm",
          "confidence": 3.1072627848516277e-7
        },
        {
          "id": -9024360116534295000,
          "name": "ask_why_contribute",
          "confidence": 2.716884353048954e-7
        },
        {
          "id": 4192029954137472500,
          "name": "canthelp",
          "confidence": 2.655223738656787e-7
        }
      ],
      "response_selector": {
        "all_retrieval_intents": [
          "faq",
          "chitchat",
          "out_of_scope"
        ],
        "out_of_scope": {
          "response": {
            "id": 546640627353592800,
            "response_templates": [
              {
                "text": "I only understand English, I'm sorry."
              }
            ],
            "confidence": 0.5278691649436951,
            "intent_response_key": "out_of_scope/non_english",
            "template_name": "utter_out_of_scope/non_english"
          },
          "ranking": [
            {
              "id": 546640627353592800,
              "confidence": 0.5278691649436951,
              "intent_response_key": "out_of_scope/non_english"
            },
            {
              "id": -7551163730238903000,
              "confidence": 0.4721308648586273,
              "intent_response_key": "out_of_scope/other"
            }
          ]
        },
        "faq": {
          "response": {
            "id": -7277286380353916000,
            "response_templates": [
              {
                "text": "You can use Rasa to build assistants in any language you want!"
              }
            ],
            "confidence": 0.22302119433879852,
            "intent_response_key": "faq/languages",
            "template_name": "utter_faq/languages"
          },
          "ranking": [
            {
              "id": -7277286380353916000,
              "confidence": 0.22302119433879852,
              "intent_response_key": "faq/languages"
            },
            {
              "id": -5318412938595423000,
              "confidence": 0.1361294835805893,
              "intent_response_key": "faq/channels"
            },
            {
              "id": -7419326848540188000,
              "confidence": 0.10969941318035126,
              "intent_response_key": "faq/slots"
            },
            {
              "id": -2416206405690423300,
              "confidence": 0.10439935326576233,
              "intent_response_key": "faq/what_is_forum"
            },
            {
              "id": -302717316935957700,
              "confidence": 0.09726060926914215,
              "intent_response_key": "faq/ee"
            },
            {
              "id": -3056000738742988300,
              "confidence": 0.08571340143680573,
              "intent_response_key": "faq/opensource_cost"
            },
            {
              "id": -8330522239285509000,
              "confidence": 0.06913907080888748,
              "intent_response_key": "faq/rasax"
            },
            {
              "id": 17428305590793936,
              "confidence": 0.06566115468740463,
              "intent_response_key": "faq/tutorials"
            },
            {
              "id": -5573705506715309000,
              "confidence": 0.05603122338652611,
              "intent_response_key": "faq/python_version"
            },
            {
              "id": 4596325488943305000,
              "confidence": 0.052945058792829514,
              "intent_response_key": "faq/differencerasarasax"
            }
          ]
        },
        "chitchat": {
          "response": {
            "id": -6953636954150647000,
            "response_templates": [
              {
                "text": "I'm great! Thanks for asking."
              },
              {
                "text": "I'm good, thanks!"
              },
              {
                "text": "A little bit too warm, otherwise fine."
              },
              {
                "text": "A little bit cold, otherwise fine."
              }
            ],
            "confidence": 0.540128231048584,
            "intent_response_key": "chitchat/ask_howdoing",
            "template_name": "utter_chitchat/ask_howdoing"
          },
          "ranking": [
            {
              "id": -6953636954150647000,
              "confidence": 0.540128231048584,
              "intent_response_key": "chitchat/ask_howdoing"
            },
            {
              "id": 3264288962556806000,
              "confidence": 0.11934677511453629,
              "intent_response_key": "chitchat/ask_whatspossible"
            },
            {
              "id": 6374271166760188000,
              "confidence": 0.11392673850059509,
              "intent_response_key": "chitchat/ask_ishuman"
            },
            {
              "id": 1213346982764441600,
              "confidence": 0.06894567608833313,
              "intent_response_key": "chitchat/ask_whoami"
            },
            {
              "id": 6730836175972321000,
              "confidence": 0.03219909593462944,
              "intent_response_key": "chitchat/ask_weather"
            },
            {
              "id": -4468624430896687600,
              "confidence": 0.02988085150718689,
              "intent_response_key": "chitchat/ask_howbuilt"
            },
            {
              "id": 5888030632800154000,
              "confidence": 0.02959381602704525,
              "intent_response_key": "chitchat/ask_time"
            },
            {
              "id": 6485751845603756000,
              "confidence": 0.02904650755226612,
              "intent_response_key": "chitchat/ask_howold"
            },
            {
              "id": 1902869653804230100,
              "confidence": 0.024243080988526344,
              "intent_response_key": "chitchat/ask_languagesbot"
            },
            {
              "id": 992365625688249600,
              "confidence": 0.012689227238297462,
              "intent_response_key": "chitchat/ask_restaurant"
            }
          ]
        }
      }
    },
    {
      "recipient_id": "8efaea4c-7e24-4f38-ab65-821927337bf9",
      "text": "Hey there, my name is Sara."
    },
    {
      "recipient_id": "8efaea4c-7e24-4f38-ab65-821927337bf9",
      "text": "By chatting to me you agree to our [privacy policy](https://rasa.com/privacy-policy/)."
    },
    {
      "recipient_id": "8efaea4c-7e24-4f38-ab65-821927337bf9",
      "text": "If you're new to Rasa, I can help you get started! If you've already started building an assistant, you can also ask me about the different parts of Rasa 🐦"
    }
  ],
  "buttons": [],
  "nlp": {
    "intent": {
      "name": "greet",
      "confidence": 0.9999986290931702,
      "incomprehension": false,
      "intents": [
        {
          "name": "greet",
          "confidence": 0.9999986290931702
        },
        {
          "name": "ask_which_events",
          "confidence": 5.311148925102316e-7
        },
        {
          "name": "affirm",
          "confidence": 3.1072627848516277e-7
        },
        {
          "name": "ask_why_contribute",
          "confidence": 2.716884353048954e-7
        },
        {
          "name": "canthelp",
          "confidence": 2.655223738656787e-7
        }
      ]
    },
    "entities": []
  },
  "messageText": "Hey there, my name is Sara.",
  "channel": "default",
  "sender": "bot",
  "convoStepIndex": 1
}

Was this article helpful?

0 out of 0 found this helpful