Skip to main content
  1. Botium Support
  2. Test Suite
  3. NLP
  4. NLP Quality Metrics

Intent Confidence Risks

This charts shows the intents of your test set with the weakest average confidence score.



A low average confidence score is an indicator that

  • either there are single utterances with very low confidence score having high impact on the average, due to the lack of training data for an intent

  • or for a large amount of utterances the NLU engine is not able to clearly decide for an intent

Note:

  • All test cases which resolved to an intent are collected, together with the confidence score

  • For all intents, the average confidence score is calculated

Alerts

If the average confidence score falls below a threshold (depending on the NLU engine typically between 0.4 and 0.6), this is a sign that the training data has to be balanced and/or sharpened for the intents under question.

Actions

  • In case there are single utterances having a high impact on the average, there is either additional training data needed for this intent, or the utterance is not specific enough and can be removed from the training data

  • Otherwise it is an indicator that the training data in general is not specific enough for the NLU engine, and it should be sharpened and enlarged by providing additional user examples for all weak intents.

Was this article helpful?

0 out of 0 found this helpful