Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cross validation not working for RegexEntityExtractor with lookup tables #10448

Closed
4 tasks
mleimeister opened this issue Dec 2, 2021 · 2 comments
Closed
4 tasks

Cross validation not working for RegexEntityExtractor with lookup tables #10448

mleimeister opened this issue Dec 2, 2021 · 2 comments
Assignees
@kedz @mleimeister
Labels
area:rasa-oss 🎡 Anything related to the open source Rasa framework type:bug 🐛 Inconsistencies or issues which will cause an issue or problem for users or implementors.

Comments

@mleimeister
Copy link
Contributor

mleimeister commented Dec 2, 2021 •
edited by koernerfelicia
Loading

Rasa Open Source version

2.8.15

Python version

3.8

What operating system are you using?

OSX

What happened?

Based on this forum report, which I was able to reproduce, it looks like using rasa test nlu with cross validation does not properly work for RegexEntityExtractor with lookup tables. The lookup tables are not taken into account during the training and validation, leading to no entities being predicted.

The following is a minimal example config leading to the issue, since the training data of the user was fairly large:

config.yml

language: en

pipeline:
- name: WhitespaceTokenizer
- name: RegexFeaturizer
- name: LexicalSyntacticFeaturizer
- name: CountVectorsFeaturizer
- name: CountVectorsFeaturizer
  analyzer: char_wb
  min_ngram: 1
  max_ngram: 4
- name: DIETClassifier
  epochs: 100
  entity_recognition: False
- name: RegexEntityExtractor
  use_lookup_tables: True

policies:
- name: MemoizationPolicy
- name: RulePolicy

domain.yml

version: "2.0"

intents:
  - inform_name
  - greet
  - goodbye

entities:
  - name

slots:
  name:
    type: text
    influence_conversation: false
    auto_fill: true

responses:
  utter_ask_name:
  - text: "Hey! What's your name?"

  utter_greet_by_name:
  - text: "Nice to meet you {name}!"

  utter_goodbye:
  - text: "Ok, goodbye!"

session_config:
  session_expiration_time: 60
  carry_over_slots_to_new_session: true

nlu.yml

version: "2.0"

nlu:
- intent: inform_name
  examples: |
    - Hi I'm [Chris](name)
    - My name is [Laura](name)
    - I'm called [Sara](name)
    - You can call me [Peter](name)

- intent: greet
  examples: |
    - hey
    - hello
    - hi
    - hello there
    - good morning
    - good evening
    - moin
    - hey there
    - let's go
    - hey dude
    - goodmorning
    - goodevening
    - good afternoon

- intent: goodbye
  examples: |
    - cu
    - good by
    - cee you later
    - good night
    - bye
    - goodbye
    - have a nice day

- lookup: name
  examples: |
    - Chris
    - Sara
    - Peter
    - Jasmin

rules.yml

version: "2.0"

rules:

- rule: Ask name
  steps:
  - intent: greet
  - action: utter_ask_name

- rule: Greet by name
  steps:
  - intent: inform_name
  - action: utter_greet_by_name

- rule: Say goodbye
  steps:
  - intent: goodbye
  - action: utter_goodbye

Afterwards, the report shows no entities extracted by RegexEntityExtractor with confusion matrix

Source of the problem

Stepping through the code showed that when folds are generated from the training data in generate_folds, the TrainingData objects created here for don't have the lookup_tables parameter set, resulting in empty lookup tables for both train and test data.

Proposed solution

Add the lookup_tables parameter to take over the lookup tables from the original training data object.

Definition of done

  • Validate that the above change fixes the observed issue
  • Implement the fix
  • Add unit test to check correct behaviour
  • Add changelog

Command / Request

rasa test nlu --nlu data/nlu.yml --cross-validation --runs 1 --folds 2

Relevant log output

2021-12-03 13:21:14 INFO     rasa.cli.test  - Test model using cross validation.
/Users/matthias/Workspace/rasa/rasa/utils/train_utils.py:641: UserWarning: constrain_similarities is set to `False`. It is recommended to set it to `True` when using cross-entropy loss. It will be set to `True` by default, Rasa Open Source 3.0.0 onwards.
  rasa.shared.utils.io.raise_warning(
2021-12-03 13:21:16 INFO     rasa.nlu.model  - Starting to train component WhitespaceTokenizer
2021-12-03 13:21:16 INFO     rasa.nlu.model  - Finished training component.
2021-12-03 13:21:16 INFO     rasa.nlu.model  - Starting to train component RegexFeaturizer
2021-12-03 13:21:16 INFO     rasa.nlu.model  - Finished training component.
2021-12-03 13:21:16 INFO     rasa.nlu.model  - Starting to train component LexicalSyntacticFeaturizer
2021-12-03 13:21:16 INFO     rasa.nlu.model  - Finished training component.
2021-12-03 13:21:16 INFO     rasa.nlu.model  - Starting to train component CountVectorsFeaturizer
2021-12-03 13:21:16 INFO     rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer  - 26 vocabulary items were created for text attribute.
2021-12-03 13:21:16 INFO     rasa.nlu.model  - Finished training component.
2021-12-03 13:21:16 INFO     rasa.nlu.model  - Starting to train component CountVectorsFeaturizer
2021-12-03 13:21:16 INFO     rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer  - 267 vocabulary items were created for text attribute.
2021-12-03 13:21:16 INFO     rasa.nlu.model  - Finished training component.
2021-12-03 13:21:16 INFO     rasa.nlu.model  - Starting to train component DIETClassifier
/Users/matthias/Workspace/rasa/rasa/utils/tensorflow/model_data.py:750: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  np.concatenate(np.array(f)),
Epochs: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:09<00:00, 10.93it/s, t_loss=0.874, i_acc=1]
2021-12-03 13:21:26 INFO     rasa.nlu.model  - Finished training component.
2021-12-03 13:21:26 INFO     rasa.nlu.model  - Starting to train component RegexEntityExtractor
/Users/matthias/Workspace/rasa/rasa/shared/utils/io.py:97: UserWarning: No lookup tables or regexes defined in the training data that have a name equal to any entity in the training data. In order for this component to work you need to define valid lookup tables or regexes in the training data.
2021-12-03 13:21:26 INFO     rasa.nlu.model  - Finished training component.
2021-12-03 13:21:26 INFO     rasa.nlu.test  - Running model for predictions:
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 34.09it/s]
/Users/matthias/.pyenv/versions/3.8.6/envs/rasa/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1248: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/Users/matthias/.pyenv/versions/3.8.6/envs/rasa/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1248: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/Users/matthias/.pyenv/versions/3.8.6/envs/rasa/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1248: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/Users/matthias/.pyenv/versions/3.8.6/envs/rasa/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1248: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/Users/matthias/.pyenv/versions/3.8.6/envs/rasa/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1248: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 37.66it/s]
/Users/matthias/.pyenv/versions/3.8.6/envs/rasa/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1248: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/Users/matthias/.pyenv/versions/3.8.6/envs/rasa/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1248: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/Users/matthias/.pyenv/versions/3.8.6/envs/rasa/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1248: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/Users/matthias/.pyenv/versions/3.8.6/envs/rasa/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1248: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/Users/matthias/.pyenv/versions/3.8.6/envs/rasa/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1248: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
2021-12-03 13:21:26 INFO     rasa.nlu.model  - Starting to train component WhitespaceTokenizer
2021-12-03 13:21:26 INFO     rasa.nlu.model  - Finished training component.
2021-12-03 13:21:26 INFO     rasa.nlu.model  - Starting to train component RegexFeaturizer
2021-12-03 13:21:26 INFO     rasa.nlu.model  - Finished training component.
2021-12-03 13:21:26 INFO     rasa.nlu.model  - Starting to train component LexicalSyntacticFeaturizer
2021-12-03 13:21:26 INFO     rasa.nlu.model  - Finished training component.
2021-12-03 13:21:26 INFO     rasa.nlu.model  - Starting to train component CountVectorsFeaturizer
2021-12-03 13:21:26 INFO     rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer  - 17 vocabulary items were created for text attribute.
2021-12-03 13:21:26 INFO     rasa.nlu.model  - Finished training component.
2021-12-03 13:21:26 INFO     rasa.nlu.model  - Starting to train component CountVectorsFeaturizer
2021-12-03 13:21:26 INFO     rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer  - 184 vocabulary items were created for text attribute.
2021-12-03 13:21:26 INFO     rasa.nlu.model  - Finished training component.
2021-12-03 13:21:26 INFO     rasa.nlu.model  - Starting to train component DIETClassifier
/Users/matthias/Workspace/rasa/rasa/utils/tensorflow/model_data.py:750: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  np.concatenate(np.array(f)),
Epochs: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:06<00:00, 14.75it/s, t_loss=0.792, i_acc=1]
2021-12-03 13:21:33 INFO     rasa.nlu.model  - Finished training component.
2021-12-03 13:21:33 INFO     rasa.nlu.model  - Starting to train component RegexEntityExtractor
/Users/matthias/Workspace/rasa/rasa/shared/utils/io.py:97: UserWarning: No lookup tables or regexes defined in the training data that have a name equal to any entity in the training data. In order for this component to work you need to define valid lookup tables or regexes in the training data.
2021-12-03 13:21:33 INFO     rasa.nlu.model  - Finished training component.
2021-12-03 13:21:33 INFO     rasa.nlu.test  - Running model for predictions:
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 34.18it/s]
/Users/matthias/.pyenv/versions/3.8.6/envs/rasa/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1248: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/Users/matthias/.pyenv/versions/3.8.6/envs/rasa/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1248: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/Users/matthias/.pyenv/versions/3.8.6/envs/rasa/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1248: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/Users/matthias/.pyenv/versions/3.8.6/envs/rasa/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1248: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/Users/matthias/.pyenv/versions/3.8.6/envs/rasa/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1248: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 36.41it/s]
/Users/matthias/.pyenv/versions/3.8.6/envs/rasa/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1248: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/Users/matthias/.pyenv/versions/3.8.6/envs/rasa/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1248: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/Users/matthias/.pyenv/versions/3.8.6/envs/rasa/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1248: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/Users/matthias/.pyenv/versions/3.8.6/envs/rasa/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1248: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/Users/matthias/.pyenv/versions/3.8.6/envs/rasa/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1248: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
2021-12-03 13:21:34 INFO     rasa.nlu.test  - Accumulated test folds intent evaluation results:
2021-12-03 13:21:34 INFO     rasa.nlu.test  - Intent Evaluation: Only considering those 24 examples that have a defined intent out of 24 examples.
2021-12-03 13:21:34 INFO     rasa.nlu.test  - Classification report saved to results/intent_report.json.
2021-12-03 13:21:34 INFO     rasa.nlu.test  - Incorrect intent predictions saved to results/intent_errors.json.
2021-12-03 13:21:34 INFO     rasa.utils.plotting  - Confusion matrix, without normalization: 
[[ 5  2  0]
 [ 1 12  0]
 [ 0  0  4]]
2021-12-03 13:21:36 INFO     rasa.nlu.test  - Accumulated test folds entity evaluation results:
2021-12-03 13:21:36 INFO     rasa.nlu.test  - Evaluation for entity extractor: RegexEntityExtractor 
/Users/matthias/.pyenv/versions/3.8.6/envs/rasa/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1248: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/Users/matthias/.pyenv/versions/3.8.6/envs/rasa/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1248: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/Users/matthias/.pyenv/versions/3.8.6/envs/rasa/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1248: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/Users/matthias/.pyenv/versions/3.8.6/envs/rasa/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1248: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/Users/matthias/.pyenv/versions/3.8.6/envs/rasa/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1248: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
2021-12-03 13:21:36 INFO     rasa.nlu.test  - Classification report saved to results/RegexEntityExtractor_report.json.
2021-12-03 13:21:36 INFO     rasa.nlu.test  - Incorrect entity predictions saved to results/RegexEntityExtractor_errors.json.
2021-12-03 13:21:36 INFO     rasa.utils.plotting  - Confusion matrix, without normalization: 
[[ 0  4]
 [ 0 48]]
2021-12-03 13:21:37 INFO     rasa.model_testing  - CV evaluation (n=2)
2021-12-03 13:21:37 INFO     rasa.model_testing  - Intent evaluation results
2021-12-03 13:21:37 INFO     rasa.nlu.test  - train Accuracy: 1.000 (0.000)
2021-12-03 13:21:37 INFO     rasa.nlu.test  - train F1-score: 1.000 (0.000)
2021-12-03 13:21:37 INFO     rasa.nlu.test  - train Precision: 1.000 (0.000)
2021-12-03 13:21:37 INFO     rasa.nlu.test  - test Accuracy: 0.875 (0.125)
2021-12-03 13:21:37 INFO     rasa.nlu.test  - test F1-score: 0.871 (0.129)
2021-12-03 13:21:37 INFO     rasa.nlu.test  - test Precision: 0.873 (0.127)
2021-12-03 13:21:37 INFO     rasa.model_testing  - Entity evaluation results
2021-12-03 13:21:37 INFO     rasa.nlu.test  - Entity extractor: RegexEntityExtractor
2021-12-03 13:21:37 INFO     rasa.nlu.test  - train Accuracy: 0.923 (0.006)
2021-12-03 13:21:37 INFO     rasa.nlu.test  - train F1-score: 0.000 (0.000)
2021-12-03 13:21:37 INFO     rasa.nlu.test  - train Precision: 0.000 (0.000)
2021-12-03 13:21:37 INFO     rasa.nlu.test  - Entity extractor: RegexEntityExtractor
2021-12-03 13:21:37 INFO     rasa.nlu.test  - test Accuracy: 0.923 (0.006)
2021-12-03 13:21:37 INFO     rasa.nlu.test  - test F1-score: 0.000 (0.000)
2021-12-03 13:21:37 INFO     rasa.nlu.test  - test Precision: 0.000 (0.000)
@mleimeister mleimeister added type:bug 🐛 Inconsistencies or issues which will cause an issue or problem for users or implementors. area:rasa-oss 🎡 Anything related to the open source Rasa framework labels Dec 2, 2021
@kedz
Copy link
Contributor

kedz commented Dec 13, 2021

@kedz is reviewer

@mleimeister mleimeister mentioned this issue Dec 14, 2021
Merged
4 tasks
mleimeister added a commit that referenced this issue Dec 16, 2021
@mleimeister
Fix #10448 cross validation failing with lookup tables (#10524)
1c3f232 
* Unit test triggering the problem.
* Fix config file path
* Copy lookup tables to training folds in CV
* Fix linting
* Add changelog

Co-authored-by: Chris Kedzie <c.kedzie@rasa.com>
@mleimeister
Copy link
Contributor Author

Fixed in release 3.0.3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kedz kedz

@mleimeister mleimeister

Labels
area:rasa-oss 🎡 Anything related to the open source Rasa framework type:bug 🐛 Inconsistencies or issues which will cause an issue or problem for users or implementors.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@kedz @dakshvar22 @mleimeister