Confidence Threshold is the lowest accepted confidence.
In Some Chatbot Engines you can set Confidence Threshold. If the Engine is not sure enough at classifying an intent (its confidence is below Confidence Threshold) then they will answer with incomprehension intent to show that they dont understand.
But you can deal with low confidence responses programmatically, dropping the indent, asking back, display alternatives, and so on.
This chart helps you to find the optimal Confidence Threshold:
Precision, Recall, and F1 Score without maximum means that the chatbot, or the test is unbalanced. It should be investigated further.
Confidence Threshold should be between the max of the Precision, and Recall
For general chatbots good Confidence Threshold is the max of the F1 Score
For a precise chatbot, like a Banking Bot we can find a Confidence Score between the max of the F1 Score, and the Precision using this Chart. (Win Precision but loose Recall.)
If a 'dont understand' answer is worse than an incorrect answer, like in a FAQ bot, then we can do the same between F1 Score, and Recall. (Win Recall but loose Precision.)
You can read about Precision, Recall and F1 score here: Quality Metrics for NLU/Chatbot Training Data.
Intent is recognized as correct if:
there is an Intent Asserter which does not fail.
There is no asserter and the response is not incomprehension
If the Chatbot Engine supports Confidence Threshold, then the best practice is to turn it off, or set it as low as possible BEFORE STARTING THE TEST SESSION to cover the largest spectrum of possible confidences!
You can examine the confidence with this chart. Increasing precision, and decreasing recall is expected. Otherwise deeper investigation is required.
On our screenshot is recall decreasing. Precision is increasing at the start, but after it it stops. It is because there where some false positive cases with confidence 1.
This Chart may be helpful if you want to compare more chatbot engines
Setting Confidence Threshold is fine tuning. Build chatbot, test it, set Confidence Threshold, and publish it.