Add bytelevel normalizer to fix decode when adding tokens to BPE #2361
Annotations
3 errors and 6 warnings
|
tokenizers/src/tokenizer/mod.rs#L865
field `splits` of struct `tokenizer::pre_tokenizer::PreTokenizedString` is private
|
|
tokenizers/src/tokenizer/mod.rs#L867
field `splits` of struct `tokenizer::pre_tokenizer::PreTokenizedString` is private
|
|
|
|
Node.js 16 actions are deprecated. Please update the following actions to use Node.js 20: actions/checkout@v2, actions-rs/toolchain@v1, actions-rs/cargo@v1, actions/setup-python@v4, actions/cache@v1. For more information see: https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/.
|
|
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/checkout@v2, actions-rs/toolchain@v1, actions-rs/cargo@v1, actions/cache@v1. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
|
|
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
|
|
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
|
|
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
|
|
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
|
This job failed
Loading