Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Converting Rasa-Demo to 2.0 Format #566

Merged
merged 58 commits into from
Oct 30, 2020
Merged

Converting Rasa-Demo to 2.0 Format #566

merged 58 commits into from
Oct 30, 2020

Conversation

b-quachtran
Copy link
Contributor

@b-quachtran b-quachtran commented Aug 24, 2020 •
edited
Loading

To do:

@b-quachtran b-quachtran linked an issue Aug 24, 2020 that may be closed by this pull request
Port Sara to 2.0 #565
Closed
@b-quachtran
Copy link
Contributor Author

@melindaloubser1 I've converted the NLU files to 2.0 (only thing missing is the lookup table file definition in the training data, since there doesn't seem to be support for that right now).

I've put down a few to-do tasks, lmk which ones you can take up and if there's anything missing from that list.

@indam23
Copy link
Contributor

indam23 commented Aug 24, 2020

Nice! The lookup tables are included inline in the yaml file now, no more txt files seperately. I can add those. I guess it makes sense to work in the order of the todos since they each depend on the previous to some degree. Would it be more helpful if I took an earlier one or a later one?

@b-quachtran
Copy link
Contributor Author

Yah, can you take on the story conversion and test out the Core converter tool? I can work on the Forms migration in parallel since those changes will mostly be in the domain file.

@indam23
Copy link
Contributor

indam23 commented Aug 25, 2020

sounds good!

@indam23 indam23 self-assigned this Aug 26, 2020
@b-quachtran b-quachtran marked this pull request as ready for review September 19, 2020 23:04
@b-quachtran b-quachtran requested a review from indam23 September 19, 2020 23:13
@b-quachtran
Copy link
Contributor Author

Data validation CI check is failing because of a bug with the new ResponseSelector format (RasaHQ/rasa#6728)

@indam23
Copy link
Contributor

indam23 commented Sep 28, 2020

@b-quachtran I'm seeing errors running rasa data validate - do you also?
A q about the test conversations: we really should have more test convos, but since we don't, how did you go about testing it all together?

@github-actions
Copy link
Contributor

Intent Cross-Validation Results (3 folds)

class support f1-score confused_with
micro avg 4951 0.8184 N/A
macro avg 4951 0.7871 N/A
weighted avg 4951 0.8114 N/A
chitchat 802 0.8582 nlu_fallback(31), out_of_scope(24)
faq 770 0.9057 nlu_fallback(19), enter_data(12)
enter_data 726 0.8535 nlu_fallback(39), out_of_scope(21)
out_of_scope 411 0.5022 nlu_fallback(79), chitchat(54)
how_to_get_started 335 0.8299 nlu_fallback(18), faq(10)
affirm 225 0.8092 nlu_fallback(19), enter_data(7)
technical_question 220 0.6010 nlu_fallback(26), faq(21)
contact_sales 159 0.8840 nlu_fallback(9), faq(5)
greet 147 0.8352 enter_data(9), chitchat(7)
signup_newsletter 141 0.9537 out_of_scope(4), nlu_fallback(3)
ask_which_events 106 0.9577 technical_question(1), faq(1)
deny 100 0.7940 nlu_fallback(5), enter_data(4)
install_rasa 97 0.8852 how_to_get_started(5), nlu_fallback(4)
human_handoff 69 0.8413 nlu_fallback(5), enter_data(4)
react_positive 62 0.4685 nlu_fallback(10), affirm(8)
nlu_info 62 0.9147 technical_question(1), faq(1)
switch 54 0.8952 nlu_fallback(3), affirm(1)
ask_how_contribute 53 0.8000 nlu_fallback(7), faq(2)
react_negative 47 0.6667 nlu_fallback(9), out_of_scope(3)
why_rasa 45 0.8706 faq(2), out_of_scope(2)
ask_question_in_forum 43 0.8642 nlu_fallback(4), faq(3)
bye 42 0.7671 nlu_fallback(4), affirm(3)
need_help_broad 41 0.7532 technical_question(5), nlu_fallback(3)
thank 39 0.8571 nlu_fallback(3), enter_data(2)
source_code 34 0.8308 nlu_fallback(3), chitchat(2)
canthelp 26 0.2927 chitchat(5), deny(5)
ask_why_contribute 21 0.7059 ask_how_contribute(4), why_rasa(2)
pipeline_recommendation 19 0.9189 nlu_fallback(2)
explain 16 0.4615 chitchat(3), nlu_fallback(2)
next_step 15 0.9286 chitchat(1), out_of_scope(1)
nlu_generation_tool_recommendation 14 0.8800 how_to_get_started(2), nlu_fallback(1)
restart 10 0.8000 technical_question(1), nlu_fallback(1)

Entity Cross-Validation Results (5 folds)

entity support f1-score precision recall
micro avg 1363 0.83093 0.85961 0.8041
macro avg 1363 0.70352 0.73095 0.6966
weighted avg 1363 0.81678 0.84093 0.8041
product 430 0.90825 0.90719 0.9093
language 297 0.96211 0.94194 0.9832
job_function 160 0.69360 0.75182 0.6437
name 155 0.71795 0.83051 0.6323
nlu_part 94 0.92462 0.87619 0.9787
company 89 0.28788 0.44186 0.2135
current_api 61 0.85950 0.86667 0.8525
location 42 0.78378 0.90625 0.6905
user_type 19 0.82609 0.70370 1.0000
entity 16 0.07143 0.08333 0.0625

@b-quachtran
Copy link
Contributor Author

@melindaloubser1 I'm seeing a bunch of story contradictions being thrown during training, working through them at the moment but probably won't be resolved until Monday.

@github-actions
Copy link
Contributor

Intent Cross-Validation Results (3 folds)

class support f1-score confused_with
micro avg 4951 0.8150 N/A
macro avg 4951 0.7981 N/A
weighted avg 4951 0.8088 N/A
chitchat 802 0.8453 nlu_fallback(39), out_of_scope(31)
faq 770 0.8885 nlu_fallback(23), enter_data(11)
enter_data 726 0.8505 nlu_fallback(34), out_of_scope(19)
out_of_scope 411 0.5205 chitchat(63), nlu_fallback(57)
how_to_get_started 335 0.8135 faq(18), nlu_fallback(16)
affirm 225 0.7982 nlu_fallback(17), enter_data(7)
technical_question 220 0.6085 nlu_fallback(28), faq(25)
contact_sales 159 0.8660 faq(4), out_of_scope(3)
greet 147 0.8432 nlu_fallback(7), enter_data(6)
signup_newsletter 141 0.9470 nlu_fallback(3), affirm(1)
ask_which_events 106 0.9671 faq(1), nlu_fallback(1)
deny 100 0.7579 enter_data(9), nlu_fallback(6)
install_rasa 97 0.8404 technical_question(6), nlu_fallback(6)
human_handoff 69 0.8682 nlu_fallback(4), contact_sales(3)
nlu_info 62 0.9365 technical_question(1), nlu_fallback(1)
react_positive 62 0.6903 nlu_fallback(6), affirm(5)
switch 54 0.8800 nlu_fallback(6), affirm(1)
ask_how_contribute 53 0.7835 nlu_fallback(7), chitchat(3)
react_negative 47 0.6914 out_of_scope(7), deny(4)
why_rasa 45 0.8049 nlu_fallback(5), faq(3)
ask_question_in_forum 43 0.9024 nlu_fallback(4), how_to_get_started(1)
bye 42 0.7606 affirm(6), nlu_fallback(6)
need_help_broad 41 0.7632 technical_question(6), nlu_fallback(2)
thank 39 0.9041 nlu_fallback(3), contact_sales(1)
source_code 34 0.8485 technical_question(2), nlu_fallback(2)
canthelp 26 0.5366 nlu_fallback(5), chitchat(3)
ask_why_contribute 21 0.6250 nlu_fallback(6), ask_how_contribute(3)
pipeline_recommendation 19 0.8421 nlu_fallback(3)
explain 16 0.4167 chitchat(4), nlu_fallback(4)
next_step 15 0.8667 affirm(1), nlu_fallback(1)
nlu_generation_tool_recommendation 14 0.9231 how_to_get_started(1), faq(1)
restart 10 0.9474 technical_question(1)

Entity Cross-Validation Results (5 folds)

entity support f1-score precision recall
micro avg 1363 0.8541 0.8832 0.8269
macro avg 1363 0.7392 0.7728 0.7202
weighted avg 1363 0.8407 0.8651 0.8269
product 430 0.9220 0.9095 0.9349
language 297 0.9718 0.9575 0.9865
job_function 160 0.7273 0.8254 0.6500
name 155 0.7527 0.8468 0.6774
nlu_part 94 0.9246 0.8762 0.9787
company 89 0.3529 0.5106 0.2697
current_api 61 0.9000 0.9153 0.8852
location 42 0.8608 0.9189 0.8095
user_type 19 0.9000 0.8571 0.9474
entity 16 0.0800 0.1111 0.0625

@github-actions
Copy link
Contributor

Intent Cross-Validation Results (3 folds)

class support f1-score confused_with
micro avg 4948 0.7990 N/A
macro avg 4948 0.7600 N/A
weighted avg 4948 0.7897 N/A
chitchat 802 0.8454 out_of_scope(32), nlu_fallback(32)
faq 767 0.9009 nlu_fallback(22), technical_question(8)
enter_data 726 0.8204 nlu_fallback(46), out_of_scope(30)
out_of_scope 411 0.4236 nlu_fallback(75), chitchat(62)
how_to_get_started 335 0.8186 nlu_fallback(19), faq(11)
affirm 225 0.7689 nlu_fallback(20), out_of_scope(9)
technical_question 220 0.6000 nlu_fallback(30), faq(27)
contact_sales 159 0.8797 nlu_fallback(12), faq(5)
greet 147 0.8339 enter_data(16), affirm(5)
signup_newsletter 141 0.9614 nlu_fallback(2), greet(1)
ask_which_events 106 0.9581 out_of_scope(2), nlu_fallback(1)
deny 100 0.6952 nlu_fallback(17), enter_data(8)
install_rasa 97 0.8462 how_to_get_started(6), nlu_fallback(5)
human_handoff 69 0.8455 nlu_fallback(6), deny(2)
nlu_info 62 0.8800 enter_data(2), how_to_get_started(2)
react_positive 62 0.6139 nlu_fallback(9), affirm(6)
switch 54 0.8824 nlu_fallback(4), enter_data(2)
ask_how_contribute 53 0.7579 nlu_fallback(12), chitchat(2)
react_negative 47 0.6667 out_of_scope(7), nlu_fallback(6)
why_rasa 45 0.7727 nlu_fallback(7), faq(1)
ask_question_in_forum 43 0.7949 faq(6), nlu_fallback(4)
bye 42 0.7246 nlu_fallback(8), out_of_scope(3)
need_help_broad 41 0.7654 nlu_fallback(3), technical_question(2)
thank 39 0.8493 nlu_fallback(4), enter_data(2)
source_code 34 0.8065 nlu_fallback(4), technical_question(2)
canthelp 26 0.4783 nlu_fallback(7), chitchat(3)
ask_why_contribute 21 0.6286 why_rasa(3), nlu_fallback(3)
pipeline_recommendation 19 0.8333 nlu_fallback(3), nlu_generation_tool_recommendation(1)
explain 16 0.2000 nlu_fallback(4), chitchat(4), why_rasa(3)
next_step 15 0.8571 nlu_fallback(2), affirm(1)
nlu_generation_tool_recommendation 14 0.7692 nlu_fallback(4)
restart 10 0.8421 why_rasa(1), nlu_fallback(1)

Entity Cross-Validation Results (5 folds)

entity support f1-score precision recall
micro avg 1451 0.8366 0.8537 0.8201
macro avg 1451 0.7090 0.7341 0.7014
weighted avg 1451 0.8225 0.8342 0.8201
product 518 0.8922 0.8670 0.9189
language 297 0.9605 0.9389 0.9832
job_function 160 0.6920 0.7752 0.6250
name 155 0.7645 0.8116 0.7226
nlu_part 94 0.9119 0.8889 0.9362
company 89 0.2857 0.4318 0.2135
current_api 61 0.8739 0.8966 0.8525
location 42 0.8649 1 0.7619
user_type 19 0.8444 0.7308 1
entity 16 N/A N/A N/A

@b-quachtran
Copy link
Contributor Author

@melindaloubser1 Context switching is working now and the rule conflicts are resolved. Can you review when you get a chance?

@indam23
Copy link
Contributor

indam23 commented Oct 20, 2020

Will do. Have we opened an issue for the nlu_fallback confusion during evaluation anywhere?

@github-actions
Copy link
Contributor

Intent Cross-Validation Results (3 folds)

class support f1-score confused_with
micro avg 4948 0.8095 N/A
macro avg 4948 0.7799 N/A
weighted avg 4948 0.8019 N/A
chitchat 802 0.8498 nlu_fallback(27), out_of_scope(26)
faq 767 0.9070 nlu_fallback(22), how_to_get_started(12)
enter_data 726 0.8235 nlu_fallback(53), out_of_scope(38)
out_of_scope 411 0.4520 chitchat(80), nlu_fallback(62)
how_to_get_started 335 0.8213 nlu_fallback(17), faq(9)
affirm 225 0.7891 nlu_fallback(20), out_of_scope(8)
technical_question 220 0.6110 nlu_fallback(25), faq(24)
contact_sales 159 0.8924 faq(7), nlu_fallback(3)
greet 147 0.8530 nlu_fallback(8), enter_data(7)
signup_newsletter 141 0.9541 nlu_fallback(3), out_of_scope(1)
ask_which_events 106 0.9623 chitchat(2), faq(1)
deny 100 0.8298 enter_data(8), nlu_fallback(5)
install_rasa 97 0.8432 nlu_fallback(9), how_to_get_started(7)
human_handoff 69 0.8682 nlu_fallback(7), enter_data(2)
nlu_info 62 0.9500 how_to_get_started(2), technical_question(1)
react_positive 62 0.5913 nlu_fallback(14), chitchat(8)
switch 54 0.8932 nlu_fallback(4), enter_data(3)
ask_how_contribute 53 0.7347 nlu_fallback(7), chitchat(5)
react_negative 47 0.6667 nlu_fallback(7), deny(3)
why_rasa 45 0.8049 nlu_fallback(6), faq(3)
ask_question_in_forum 43 0.8750 faq(3), nlu_fallback(3)
bye 42 0.7714 affirm(6), nlu_fallback(3)
need_help_broad 41 0.7500 technical_question(8), nlu_fallback(3)
thank 39 0.8333 nlu_fallback(4), affirm(1)
source_code 34 0.8060 technical_question(2), nlu_fallback(2)
canthelp 26 0.3810 chitchat(8), nlu_fallback(6)
ask_why_contribute 21 0.7273 ask_how_contribute(4), why_rasa(2)
pipeline_recommendation 19 0.8125 nlu_fallback(4), technical_question(2)
explain 16 0.4615 chitchat(4), nlu_fallback(4)
next_step 15 0.9286 affirm(1), nlu_fallback(1)
nlu_generation_tool_recommendation 14 0.8462 nlu_fallback(2), faq(1)
restart 10 0.6667 technical_question(3), canthelp(1)

Entity Cross-Validation Results (5 folds)

entity support f1-score precision recall
micro avg 1451 0.84396 0.8631 0.8256
macro avg 1451 0.70786 0.7300 0.7056
weighted avg 1451 0.82704 0.8381 0.8256
product 518 0.90669 0.8858 0.9286
language 297 0.96860 0.9513 0.9865
job_function 160 0.72109 0.7910 0.6625
name 155 0.76224 0.8321 0.7032
nlu_part 94 0.93069 0.8704 1.0000
company 89 0.20472 0.3421 0.1461
current_api 61 0.87395 0.8966 0.8525
location 42 0.77922 0.8571 0.7143
user_type 19 0.84444 0.7308 1.0000
entity 16 0.08696 0.1429 0.0625

@b-quachtran
Copy link
Contributor Author

Yeah it's in the discussion in the issue Akela made here

@koaning
Copy link

koaning commented Oct 22, 2020 •
edited
Loading

Just to make folks aware. I just tried running rasa nlu test locally and I'm seeing some strange bugs.

> rasa test nlu --config config-templates/just-sparse.yml --cross-validation --runs 1 --folds 2 --out gridresults/just-sparse
...
...
...
Traceback (most recent call last):
  File "/home/vincent/Development/rasa-demo/venv/bin/rasa", line 8, in <module>
    sys.exit(main())
  File "/home/vincent/Development/rasa-demo/venv/lib/python3.7/site-packages/rasa/__main__.py", line 116, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/home/vincent/Development/rasa-demo/venv/lib/python3.7/site-packages/rasa/cli/test.py", line 159, in run_nlu_test
    perform_nlu_cross_validation(config, nlu_data, output, vars(args))
  File "/home/vincent/Development/rasa-demo/venv/lib/python3.7/site-packages/rasa/test.py", line 276, in perform_nlu_cross_validation
    data, folds, nlu_config, output, **kwargs
  File "/home/vincent/Development/rasa-demo/venv/lib/python3.7/site-packages/rasa/nlu/test.py", line 1687, in cross_validate
    train,
  File "/home/vincent/Development/rasa-demo/venv/lib/python3.7/site-packages/rasa/nlu/test.py", line 1595, in combine_result
    ) = compute_metrics(interpreter, data)
  File "/home/vincent/Development/rasa-demo/venv/lib/python3.7/site-packages/rasa/nlu/test.py", line 1776, in compute_metrics
    interpreter, training_data
  File "/home/vincent/Development/rasa-demo/venv/lib/python3.7/site-packages/rasa/nlu/test.py", line 1310, in get_eval_data
    result = interpreter.parse(example.get(TEXT), only_output_properties=False)
  File "/home/vincent/Development/rasa-demo/venv/lib/python3.7/site-packages/rasa/nlu/model.py", line 398, in parse
    component.process(message, **self.context)
  File "/home/vincent/Development/rasa-demo/venv/lib/python3.7/site-packages/rasa/nlu/selectors/response_selector.py", line 382, in process
    self._resolve_intent_response_key(top_label) or top_label[INTENT_NAME_KEY]
  File "/home/vincent/Development/rasa-demo/venv/lib/python3.7/site-packages/rasa/nlu/selectors/response_selector.py", line 363, in _resolve_intent_response_key
    search_key = util.template_key_to_intent_response_key(key)
  File "/home/vincent/Development/rasa-demo/venv/lib/python3.7/site-packages/rasa/shared/nlu/training_data/util.py", line 145, in template_key_to_intent_response_key
    return template_key.split(UTTER_PREFIX)[1]

It's suggested here that it might be due to a data conversion error so I figured I'd check in to let folks know. This error occurs after the training phase is done, just before it starts with the validation bit.

EDIT

It seems to also fail by just calling rasa test nlu so it definitely seems to be going wrong there.

@indam23
Copy link
Contributor

indam23 commented Oct 26, 2020

@koaning I think this might be an environment thing, I'm not seeing that on 2.0.2

indam23
indam23 approved these changes Oct 26, 2020
View reviewed changes
Copy link
Contributor

@indam23 indam23 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

besides the false positive story conflicts (after ... and after action_listen ) this looks good!

@github-actions
Copy link
Contributor

Intent Cross-Validation Results (3 folds)

class support f1-score confused_with
micro avg 4950 0.8102 N/A
macro avg 4950 0.7797 N/A
weighted avg 4950 0.8013 N/A
chitchat 802 0.8506 nlu_fallback(31), out_of_scope(21)
faq 768 0.8986 nlu_fallback(17), how_to_get_started(7)
enter_data 726 0.8528 nlu_fallback(34), out_of_scope(27)
out_of_scope 411 0.4502 nlu_fallback(79), chitchat(73)
how_to_get_started 335 0.8338 chitchat(19), faq(14)
affirm 225 0.8065 nlu_fallback(26), out_of_scope(7)
technical_question 220 0.5397 faq(37), nlu_fallback(27)
contact_sales 159 0.8931 nlu_fallback(6), faq(5)
greet 147 0.8327 nlu_fallback(10), chitchat(8)
signup_newsletter 141 0.9455 nlu_fallback(4), enter_data(3)
ask_which_events 107 0.9714 chitchat(2), out_of_scope(1)
deny 100 0.7565 nlu_fallback(11), enter_data(6)
install_rasa 97 0.8482 how_to_get_started(8), technical_question(4)
human_handoff 69 0.8413 enter_data(4), nlu_fallback(4)
nlu_info 62 0.8833 technical_question(3), nlu_fallback(2)
react_positive 62 0.6000 nlu_fallback(9), affirm(5)
switch 54 0.8824 nlu_fallback(3), affirm(2)
ask_how_contribute 53 0.7789 chitchat(8), nlu_fallback(4)
react_negative 47 0.6744 deny(4), nlu_fallback(4)
why_rasa 45 0.8250 nlu_fallback(5), chitchat(2)
ask_question_in_forum 43 0.9024 faq(4), technical_question(1)
bye 42 0.7632 affirm(5), react_negative(3)
need_help_broad 41 0.7027 technical_question(6), nlu_fallback(6)
thank 39 0.8919 faq(2), nlu_fallback(2)
source_code 34 0.8254 nlu_fallback(4), chitchat(2)
canthelp 26 0.4681 nlu_fallback(5), chitchat(3)
ask_why_contribute 21 0.4667 nlu_fallback(8), out_of_scope(3)
pipeline_recommendation 19 0.9444 chitchat(1), nlu_fallback(1)
explain 16 0.4667 chitchat(3), nlu_fallback(2)
next_step 15 0.8966 nlu_fallback(2)
nlu_generation_tool_recommendation 14 0.8571 nlu_fallback(1), how_to_get_started(1)
restart 10 0.8000 N/A

Entity Cross-Validation Results (5 folds)

entity support f1-score precision recall
micro avg 1451 0.83716 0.85870 0.8167
macro avg 1451 0.71960 0.76439 0.6932
weighted avg 1451 0.82394 0.84339 0.8167
product 518 0.88785 0.86051 0.9170
language 297 0.96529 0.94805 0.9832
job_function 160 0.71383 0.73510 0.6937
name 155 0.75655 0.90179 0.6516
nlu_part 94 0.91667 0.89796 0.9362
company 89 0.29231 0.46341 0.2135
current_api 61 0.87179 0.91071 0.8361
location 42 0.79452 0.93548 0.6905
user_type 19 0.92308 0.90000 0.9474
entity 16 0.07407 0.09091 0.0625

@github-actions
Copy link
Contributor

Intent Cross-Validation Results (3 folds)

class support f1-score confused_with
micro avg 4950 0.8092 N/A
macro avg 4950 0.7855 N/A
weighted avg 4950 0.8026 N/A
chitchat 802 0.8537 nlu_fallback(37), out_of_scope(24)
faq 768 0.8982 nlu_fallback(21), technical_question(9)
enter_data 726 0.8262 nlu_fallback(40), out_of_scope(30)
out_of_scope 411 0.4670 chitchat(74), nlu_fallback(55)
how_to_get_started 335 0.8437 nlu_fallback(22), chitchat(9)
affirm 225 0.7864 nlu_fallback(15), enter_data(10)
technical_question 220 0.5949 faq(29), nlu_fallback(26)
contact_sales 159 0.8925 nlu_fallback(12), faq(5)
greet 147 0.8491 nlu_fallback(10), enter_data(7)
signup_newsletter 141 0.9493 nlu_fallback(7), out_of_scope(2)
ask_which_events 107 0.9763 chitchat(2), faq(1)
deny 100 0.7340 nlu_fallback(12), enter_data(6)
install_rasa 97 0.8254 technical_question(7), nlu_fallback(6)
human_handoff 69 0.8819 nlu_fallback(4), out_of_scope(3)
react_positive 62 0.5806 affirm(7), nlu_fallback(5)
nlu_info 62 0.8960 nlu_fallback(2), enter_data(2)
switch 54 0.8713 nlu_fallback(5), faq(2)
ask_how_contribute 53 0.8041 nlu_fallback(6), chitchat(4)
react_negative 47 0.6835 deny(5), nlu_fallback(4)
why_rasa 45 0.7955 nlu_fallback(5), chitchat(2)
ask_question_in_forum 43 0.8941 faq(3), need_help_broad(1)
bye 42 0.7887 nlu_fallback(7), affirm(3)
need_help_broad 41 0.7397 technical_question(7), nlu_fallback(3)
thank 39 0.8767 nlu_fallback(2), enter_data(2)
source_code 34 0.8387 nlu_fallback(3), chitchat(2)
canthelp 26 0.4348 deny(5), chitchat(3)
ask_why_contribute 21 0.6667 nlu_fallback(7), ask_how_contribute(2)
pipeline_recommendation 19 0.9189 nlu_fallback(2)
explain 16 0.4444 chitchat(4), nlu_fallback(2)
next_step 15 0.8571 affirm(1), nlu_fallback(1)
nlu_generation_tool_recommendation 14 0.8889 how_to_get_started(1), faq(1)
restart 10 0.7778 nlu_fallback(2), out_of_scope(1)

Entity Cross-Validation Results (5 folds)

entity support f1-score precision recall
micro avg 1451 0.8445 0.8688 0.8215
macro avg 1451 0.7238 0.7507 0.7114
weighted avg 1451 0.8323 0.8514 0.8215
product 518 0.9084 0.9075 0.9093
language 297 0.9636 0.9479 0.9798
job_function 160 0.7200 0.7714 0.6750
name 155 0.7491 0.8015 0.7032
nlu_part 94 0.9436 0.9109 0.9787
company 89 0.2656 0.4359 0.1910
current_api 61 0.8889 0.9286 0.8525
location 42 0.8312 0.9143 0.7619
user_type 19 0.9048 0.8261 1.0000
entity 16 0.0625 0.0625 0.0625

@github-actions
Copy link
Contributor

Intent Cross-Validation Results (3 folds)

class support f1-score confused_with
micro avg 4950 0.8087 N/A
macro avg 4950 0.7929 N/A
weighted avg 4950 0.8025 N/A
chitchat 802 0.8599 out_of_scope(31), nlu_fallback(23)
faq 768 0.9092 nlu_fallback(20), enter_data(14)
enter_data 726 0.8285 out_of_scope(37), nlu_fallback(34)
out_of_scope 411 0.4409 nlu_fallback(73), chitchat(64)
how_to_get_started 335 0.8256 nlu_fallback(17), chitchat(11)
affirm 225 0.7806 nlu_fallback(15), out_of_scope(10)
technical_question 220 0.5659 nlu_fallback(27), faq(22)
contact_sales 159 0.9073 nlu_fallback(7), faq(4)
greet 147 0.8357 out_of_scope(8), enter_data(7)
signup_newsletter 141 0.9433 nlu_fallback(4), out_of_scope(2)
ask_which_events 107 0.9665 nlu_fallback(2), chitchat(2)
deny 100 0.7451 nlu_fallback(6), enter_data(5)
install_rasa 97 0.8557 how_to_get_started(7), technical_question(5)
human_handoff 69 0.8906 nlu_fallback(4), out_of_scope(3)
react_positive 62 0.5085 affirm(9), nlu_fallback(8)
nlu_info 62 0.9333 faq(3), nlu_fallback(1)
switch 54 0.9143 nlu_fallback(2), enter_data(2)
ask_how_contribute 53 0.7835 nlu_fallback(6), chitchat(5)
react_negative 47 0.6835 deny(8), nlu_fallback(6)
why_rasa 45 0.8043 technical_question(3), nlu_fallback(2)
ask_question_in_forum 43 0.9048 faq(3), how_to_get_started(1)
bye 42 0.8333 affirm(3), out_of_scope(2)
need_help_broad 41 0.8158 technical_question(5), nlu_fallback(2)
thank 39 0.8571 nlu_fallback(3), out_of_scope(2)
source_code 34 0.8125 technical_question(4), chitchat(2)
canthelp 26 0.4783 deny(4), chitchat(3)
ask_why_contribute 21 0.6452 why_rasa(4), nlu_fallback(3)
pipeline_recommendation 19 0.8718 technical_question(1), nlu_fallback(1)
explain 16 0.4348 chitchat(4), nlu_fallback(3)
next_step 15 0.9333 nlu_fallback(1)
nlu_generation_tool_recommendation 14 0.9630 nlu_fallback(1)
restart 10 0.8421 technical_question(2)

Entity Cross-Validation Results (5 folds)

entity support f1-score precision recall
micro avg 1451 0.8490 0.8573 0.8408
macro avg 1451 0.7286 0.7521 0.7167
weighted avg 1451 0.8370 0.8401 0.8408
product 518 0.9071 0.8746 0.9421
language 297 0.9555 0.9355 0.9764
job_function 160 0.7029 0.7190 0.6875
name 155 0.7740 0.8248 0.7290
nlu_part 94 0.9479 0.9286 0.9681
company 89 0.3478 0.4898 0.2697
current_api 61 0.9076 0.9310 0.8852
location 42 0.8649 1 0.7619
user_type 19 0.8780 0.8182 0.9474
entity 16 N/A N/A N/A

@b-quachtran b-quachtran merged commit 338aafe into master Oct 30, 2020
@dakshvar22 dakshvar22 mentioned this pull request Dec 3, 2020
Closed
@indam23 indam23 deleted the rasa-2.0 branch May 17, 2021 09:12
hamzaziizzz pushed a commit to hamzaziizzz/rasa-demo that referenced this pull request Aug 10, 2024
@b-quachtran
Merge pull request RasaHQ#566 from RasaHQ/rasa-2.0
718550c 
Converting Rasa-Demo to 2.0 Format
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@indam23 indam23 indam23 approved these changes

Assignees

@indam23 indam23

Labels
nlu_testing_required
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Port Sara to 2.0
5 participants
@b-quachtran @indam23 @koaning @akelad @samsucik