Derive `Clone` on `Tokenizer`, add `Encoding.into_tokens()` method (#… · huggingface/tokenizers@daf3616 · GitHub

Commit

Derive Clone on Tokenizer, add Encoding.into_tokens() method (#…

Browse files

…1381)

* Add `into_tokens()` method

* derive clone

* Update tokenizers/src/tokenizer/encoding.rs

---------

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>

Loading branch information

epwalsh and Narsil authored Nov 20, 2023

1 parent e3bcef2 commit daf3616