This chart shows utterances with a high confusion probability based on confidence score. This is only eligible for NLU engines returning alternate intents with alternate confidence score.
Utterances with low difference between primary and alternate predicted intent are classified as risky
Probability is not meant in the mathematical sense, but it is simply 1 minus the confidence score difference between primary and alternate predicted intent
The higher the value the higher the risk of mismatches
It basically is a measure of intent uniqueness: if mismatches happen, then either the training data is not tailored to the intents in question, or the intents are from a design view not really different. For example, when having “buy product A” and “buy product B” modeled as two different intents (“buy_A” and “buy_B”), the NLU engine may not be able to clearly decide between the two. In this case it would be a could decision to redesign the intent structure (having a “buy” intent and entities for “product A” and “product B”).
In general, there should be a significant difference between the confidence score of the primary and the alternate predicted intent. What is a significant difference depends on the application.
The more user examples for an intent, the more acceptable is a low difference
The more balanced the intents are expected to be, the less acceptable is a low difference
In general, the problematic utterances should be adapted or removed from one of the intents.
In case intents are overlapping from a content point of view, redesigning the intent structure is usually the best option (see above).
The Intent Mismatch Probability Risks show the intents where there are a notable number of mixings
If Intent 1 is expected for an utterance and Intent 2 has been predicted, the pair Intent 1 / Intent 2 will be shown in this table
A special low-level analytics is possible with the Intent Mismatch Probability Risk, Alternative Intents table. This table lists:
Intent pairs which often appear in the alternate intents list of each other
If Intent 1 is very often part of the alternate intents of Intent 2 and ranked very high in the list, it will be shown here
In addition, the worst utterance for the intent pair is shown - the mismatched utterance with the highest confidence
Please sign in to leave a comment.