From daf361676bdfd14088f7e0bc087effc6a9cfdf3e Mon Sep 17 00:00:00 2001
From: Pete <epwalsh10@gmail.com>
Date: Mon, 20 Nov 2023 00:56:29 -0800
Subject: [PATCH] Derive `Clone` on `Tokenizer`, add `Encoding.into_tokens()`
 method (#1381)

* Add `into_tokens()` method

* derive clone

* Update tokenizers/src/tokenizer/encoding.rs

---------

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>