Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

checkpoint_model ignores best model if just one metric stays the same #7286

Closed
Xwaler opened this issue Nov 16, 2020 · 2 comments
Closed

checkpoint_model ignores best model if just one metric stays the same #7286

Xwaler opened this issue Nov 16, 2020 · 2 comments
Assignees
Labels
area:rasa-oss/ml 👁 All issues related to machine learning type:bug 🐛 Inconsistencies or issues which will cause an issue or problem for users or implementors.

Comments

@Xwaler
Copy link

Xwaler commented Nov 16, 2020

This issue concerns the checkpoint_model parameter in every trainable component (DietClassifier, ResponseSelector, RulePolicy, etc...)
In order to save a model that has improved in every metric except one that has staled (can even stale at 1.00 accuracy !), the _does_model_improve function could compare with a superior or equal sign rather than superior strict.

float(current_results[key]) > self.best_metrics_so_far[key]

all_improved = all(
            [
                float(current_results[key]) >= self.best_metrics_so_far[key]
                for key in self.best_metrics_so_far.keys()
            ]
        )
@sara-tagger
Copy link
Collaborator

Thanks for the issue, @alopez will get back to you about it soon!

You may find help in the docs and the forum, too 🤗

@wochinge wochinge added area:rasa-oss/ml 👁 All issues related to machine learning type:bug 🐛 Inconsistencies or issues which will cause an issue or problem for users or implementors. labels Jan 29, 2021
@koernerfelicia koernerfelicia self-assigned this Apr 21, 2021
@tttthomasssss
Copy link
Contributor

tttthomasssss commented Jun 8, 2021

@dakshvar22 assigned as reviewer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:rasa-oss/ml 👁 All issues related to machine learning type:bug 🐛 Inconsistencies or issues which will cause an issue or problem for users or implementors.
Projects
None yet
6 participants