Skip to main content
  1. Botium Support
  2. Test Suite
  3. NLP
  4. NLP Quality Metrics

Intent Mismatch Probability Risks

This section shows some charts visualizing the risk that some intents will be mismatched - meaning that the NLU engine predicts the correct intent, but with a confidence score very close to another one. Continue reading to learn more.

In real-life, a chatbot in this situation often responds with something like I am not sure what you mean - do you mean X or Y ? (In IBM Watson, this is called disambiguation).

Note: Issues with the CLARITY of the test results will be visualized here.


  • Utterances with low difference between primary and alternate predicted intent are classified as risky

  • Probability is not meant in the mathematical sense, but it is simply 1 minus the confidence score difference between primary and alternate predicted intent

  • The higher the value the higher the risk of mismatches

It basically is a measure of intent uniqueness: if mismatches happen, then either the training data is not tailored to the intents in question, or the intents are from a design view not really different. For example, when having “buy product A” and “buy product B” modeled as two different intents (“buy_A” and “buy_B”), the NLU engine may not be able to clearly decide between the two. In this case it would be a could decision to redesign the intent structure (having a “buy” intent and entities for “product A” and “product B”).

Interactive Radar Chart Insights

Clicking on the radar chart shows a list of intents with their confidence scores as predicted by the NLU engine.
Note: This only works if the NLU engine returns an alternate intents list.


Alerts

In general, there should be a significant difference between the confidence score of the primary and the alternate predicted intent. What is a significant difference depends on the application.

  • The more user examples for an intent, the more acceptable is a low difference

  • The more balanced the intents are expected to be, the less acceptable is a low difference

Actions

  • In general, the problematic utterances should be adapted or removed from one of the intents.

  • In cases where intents are overlapping, from a content point of view, redesigning the intent structure is usually the best option (see above).

Additional Tables

The Intent Mismatch Probability Risks, Mixed Intents show the intents where there are a notable number of mixings.

  • If Intent 1 is expected for an utterance and Intent 2 has been predicted, the pair Intent 1 / Intent 2 will be shown in this table



A special low-level analytics is possible with the Intent Mismatch Probability Risk, Alternative Intents table. This table lists:

  • Intent pairs which often appear in the alternate intents list of each other

  • If Intent 1 is very often part of the alternate intents of Intent 2 and ranked very high in the list, it will be shown here

  • In addition, the worst utterance for the intent pair is shown - the mismatched utterance with the highest confidence



Was this article helpful?

0 out of 0 found this helpful