Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix case of special tokens in encoder #19

Closed
benlipkin opened this issue Mar 2, 2022 · 2 comments
Closed

fix case of special tokens in encoder #19

benlipkin opened this issue Mar 2, 2022 · 2 comments
Assignees
Labels
bug Something isn't working lbs:encoders related to the encoder part of the library meta:pending-tests

Comments

@benlipkin
Copy link
Contributor

special tokens, e.g. , from tokenizer cause 1-off errors when using indices to extract sentence representations from context.

@aalok-sathe aalok-sathe self-assigned this Mar 2, 2022
aalok-sathe added a commit that referenced this issue Mar 2, 2022
…urrent stim representation. in partial fulfilment of #19 (excludes special tokens in individual stim length computation, but includes them in the overall context anyway, so extraction is still affected)
@aalok-sathe
Copy link
Contributor

Now what happens here is: the special tokens are chopped off from each stimulus when extracting stimulus-level representations evaluated within a context.
The remaining thing here is: being able to extract first-token/last-token/special-token representation for a single stimulus, because now special tokens are chopped off by default since in context they represent the whole context rather than any stimulus

@aalok-sathe aalok-sathe added lbs:encoders related to the encoder part of the library bug Something isn't working labels Mar 25, 2022
@aalok-sathe
Copy link
Contributor

whoops, that was an incorrect reference to this issue. it should have been #18 instead

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working lbs:encoders related to the encoder part of the library meta:pending-tests
Projects
None yet
Development

No branches or pull requests

2 participants