Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ConvertTokenizer, CLS token & features as sequence #4996

Merged
merged 57 commits into from
Jan 13, 2020
Merged

ConvertTokenizer, CLS token & features as sequence #4996

merged 57 commits into from
Jan 13, 2020

Conversation

tabergma
Copy link
Contributor

@tabergma tabergma commented Dec 19, 2019 •
edited
Loading

Proposed changes:

  • Add ConveRT Tokenizer
  • Remove option use_cls_token from all tokenizers. Tokenizers add CLS token by default now.
  • Remove option return_sequence from featurizers. Featurizers return sequence by default. The feature vector of the CLS token contains the feature for the complete utterance.
  • Implement train and process of tokenizers in Tokenizer class. Subclasses just need to implement tokenize.

closes #4978

Status (please check what you already did):

  • added some tests for the functionality
  • updated the documentation
  • updated the changelog (please check changelog for instructions)
  • reformat files using black (please check Readme for instructions)

@tabergma
rename token offset to token start
02c61af
@tabergma
add convert tokenizer
595b7ef
@tabergma
add tests
fdeb9da
@tabergma
start updating the featurizer
48a84dc
@tabergma
Finish implementation of featurizer
9a1107c
@tabergma
update documentation
bf9950f
@tabergma
Merge branch 'master' into convert-tokenizer
cf04707
@tabergma
clean up
af3ce0f
@tabergma
add changelog entry
1a4eb39
@tabergma
Use convert tokenizer in convert pipeline.
3f76868
@tabergma
work on review comments
d3aed7c
@tabergma
Merge branch 'master' into convert-tokenizer
ce5ce96
@tabergma
Refactor convert tokenizer.
c9f525b
@tabergma
return cls_token vector in sequence_to_sentence_features if used
c491e48
@tabergma
use cls vector if present in embedding intent classifier
f8546e8
@tabergma
fix imports
73d6654
@tabergma
fix test
8fe5450
@tabergma
Merge branch 'master' into convert-tokenizer
48bb169
@tabergma
Merge branch 'convert-tokenizer' of github.com:RasaHQ/rasa into conve…
87092ae 
…rt-tokenizer
@tabergma
fix docs
8f3edd5
@tabergma
remove option use_cls_token from tokenizers
255b910
@tabergma
remove return sequence option from dense featurizers
dbb59e4
@tabergma
remove return sequence option from sparse featurizers
bfb036f
@tabergma
Merge branch 'master' into convert-tokenizer
76cbec3
@tabergma
cls token only present in text and response.
9a5d07b
@tabergma
use constants
6370c6c
@tabergma
update docs
4127485
@tabergma
extract cls token features in embedding intent classifier
17b9203
@tabergma
update changelog
eae8521
@tabergma
ignore cls token in entity extractors
bde46a5
@tabergma tabergma requested a review from Ghostvv January 2, 2020 15:18
@tabergma tabergma changed the title Convert tokenizer ConvertTokenizer, CLS token & features as sequence Jan 3, 2020
Ghostvv
Ghostvv approved these changes Jan 10, 2020
View reviewed changes
Copy link
Contributor

@Ghostvv Ghostvv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! add a couple of minor comments

dakshvar22
dakshvar22 approved these changes Jan 13, 2020
View reviewed changes
Copy link
Contributor

@dakshvar22 dakshvar22 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some minor suggestions. Great work! 🚀

tabergma and others added 6 commits January 13, 2020 10:53
@tabergma @dakshvar22
Update changelog/4978.removal.rst
80bef66 
Co-Authored-By: Daksh Varshneya <d.varshneya@rasa.com>
@tabergma @dakshvar22
Update docs/api/custom-nlu-components.rst
3244452 
Co-Authored-By: Daksh Varshneya <d.varshneya@rasa.com>
@tabergma @dakshvar22
Update docs/nlu/components.rst
021d367 
Co-Authored-By: Daksh Varshneya <d.varshneya@rasa.com>
@tabergma
review comments
22ffa50
@tabergma
Fixing requirements
fe87e38
@tabergma
fix test
189788b
@tabergma tabergma merged commit 71c1228 into master Jan 13, 2020
@tabergma tabergma deleted the convert-tokenizer branch January 13, 2020 19:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Ghostvv Ghostvv Ghostvv approved these changes

@dakshvar22 dakshvar22 dakshvar22 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Tokenizer for ConveRT
3 participants
@tabergma @Ghostvv @dakshvar22