-
Notifications
You must be signed in to change notification settings - Fork 53
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Document different package options #48
Comments
Hello! I got a little swamped with (school) work recently, so my apologies for the lack of documentation and clarity. I will update the README.md soon, but here are the gist of the changes and the rationale: This repository maintains two packages.
The reason to port the The plan going forward is to converge the APIs of both libraries to be interchangeable, allowing isomorphic behaviour (#43) and add appropriate documentation soon (with an additional PR for benchmarking both packages). Will close the issue after that is done :) Hope that clears up! |
First off, you rock @dqbd 🔥 This makes a ton of sense, and no worries about being swamped w/ school / work. Totally understand and it's all part of open source :) Thanks for the thorough explanation – will update https://github.com/transitive-bullshit/compare-tokenizers and my other projects accordingly 🙏 |
It seems there are multiple NPM packages associated with this
tiktoken
port, and I wasn't able to find the differences clearly documented anywhere. (@dqbd/tiktoken
,js-tiktoken
, andtiktoken
).Langchainjs seems to be using
js-tiktoken
(reference and associated commit langchain-ai/langchainjs@d60eae5), so I'm going with that for now, but the readme on this project usestiktoken
instead ofjs-tiktoken
, and@dqbd/tiktoken
looks like it's still around.@dqbd would love any clarity you can provide here, and thank you again for your amazing work on this project 🙏
Also, what does the
js-tiktoken/lite
version actually do differently than the other packages?The text was updated successfully, but these errors were encountered: