Add warnings for wrong values of model_confidence parameter #8014
Conversation
dakshvar22
requested review from
a team and
joejuzl
and removed request for
a team
|
@akelad Could you review the documentation part? |
No, the normalization outside is only done for softmax model confidences to preserve the old behaviour. Primarily because the normalization for softmax is done slightly differently where it takes ranking length into account and then normalizes, whereas for inner we want to normalize first and then apply ranking length if needed. |
tczekajlo
added
status:model-regression-tests
and removed
status:model-regression-tests
labels
docs/docs/components.mdx
Outdated
| This parameter allows the user to configure how confidences are computed during inference. It can take two values: | ||
| 1. `softmax`: Confidences are in the range `[0, 1]` (old behaviour and current default). Computed similarities are normalized with the `softmax` activation function. | ||
| 2. `linear_norm`: Confidences are in the range `[0, 1]`. Computed dot product similarities are normalized with a linear function. | ||
| It is recommended to try using `linear_norm` as the value for `model_confidence` as it should be easier to tune thresholds for fallback then. |
There was a problem hiding this comment.
same comments here as above
docs/docs/migration-guide.mdx
Outdated
| - name: DIETClassifier | ||
| model_confidence: linear_norm | ||
| constrain_similarities: True | ||
| ... |
There was a problem hiding this comment.
same edits to this whole thing as the comments I made on the changelog
| ``` | ||
| - name: DIETClassifier | ||
| model_confidence: cosine | ||
| model_confidence: linear_norm |
There was a problem hiding this comment.
similar edits in this paragraph - but I don't think we should take out all reference to cosine here. We should just add a warning that the below is no longer true?
There was a problem hiding this comment.
Alright, I added back the possibility of adding cosine but changed the recommendation to linear_norm. Added an EDIT at the end about the deprecation.
docs/docs/policies.mdx
Outdated
| This parameter allows the user to configure how confidences are computed during inference. It can take two values: | ||
| 1. `softmax`: Confidences are in the range `[0, 1]` (old behaviour and current default). Computed similarities are normalized with the `softmax` activation function. | ||
| 2. `linear_norm`: Confidences are in the range `[0, 1]`. Computed dot product similarities are normalized with a linear function. | ||
| It is recommended to try using `linear_norm` as the value for `model_confidence` as it should be easier to tune thresholds for fallback then. |
Docs review Co-authored-by: Akela Drissner-Schmid <32450038+akelad@users.noreply.github.com>
| ## Rasa 2.3.3 to Rasa 2.3.4 | ||
|
|
||
| :::caution | ||
| This is a release **breaking backwards compatibility of machine learning models**. |
There was a problem hiding this comment.
@dakshvar22 Couldn't we have made that one non-breaking? Seem less than ideal to do a model breaking change in a micro release.
|
@dakshvar22 hey, i saw this warning and wanted to resolve it but it took some digging to understand that the change needed to be made in 3 places: TEDPolicy, DIETClassifier, and ResponseSelector. do you think the warning should clarify which pipeline/policy is triggering the warning? i wasn't able to figure it out until i came across this https://forum.rasa.com/t/model-confidence-linear-norm/42481/3 |
Proposed changes:
model_confidence=cosineandmodel_confidence=innerin all ML components.model_confidence=linear_normas a replacement which uses a linear normalization scheme on top of dot product similarities.Status (please check what you already did):
black(please check Readme for instructions)