Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use a JS based tokenizer for token counting #1239

Merged
merged 12 commits into from
May 17, 2023
Merged

Use a JS based tokenizer for token counting #1239

merged 12 commits into from
May 17, 2023

Commits on May 17, 2023

  1. Use a rewritten tiktoken for token counting

    @dqbd
    dqbd committed May 17, 2023
    Configuration menu
    Copy the full SHA
    ddabd00 View commit details
    Browse the repository at this point in the history

  2. Fix lint, add test

    @dqbd
    dqbd committed May 17, 2023
    Configuration menu
    Copy the full SHA
    b604d79 View commit details
    Browse the repository at this point in the history

  3. Fix type import, typo

    @dqbd
    dqbd committed May 17, 2023
    Configuration menu
    Copy the full SHA
    7f4fccd View commit details
    Browse the repository at this point in the history

  4. Remove @dqbd/tiktoken as "weak" dependency

    @dqbd
    dqbd committed May 17, 2023
    Configuration menu
    Copy the full SHA
    c036232 View commit details
    Browse the repository at this point in the history

  5. Move JS implementation of tiktoken into a separate package

    @dqbd
    dqbd committed May 17, 2023
    Configuration menu
    Copy the full SHA
    f11d433 View commit details
    Browse the repository at this point in the history

  6. Upgrade to js-tiktoken@1.0.2

    @dqbd
    dqbd committed May 17, 2023
    Configuration menu
    Copy the full SHA
    f741923 View commit details
    Browse the repository at this point in the history

  7. Replace all occurrences of @dqbd/tiktoken

    @dqbd
    dqbd committed May 17, 2023
    Configuration menu
    Copy the full SHA
    7714b0d View commit details
    Browse the repository at this point in the history

  8. Avoid bundling ranks, fix test with CRA

    @dqbd
    dqbd committed May 17, 2023
    Configuration menu
    Copy the full SHA
    37b4133 View commit details
    Browse the repository at this point in the history

  9. Use AsyncCaller, cache promises

    @dqbd
    dqbd committed May 17, 2023
    Configuration menu
    Copy the full SHA
    c783e81 View commit details
    Browse the repository at this point in the history

  10. Remove WASM configuration changes

    @dqbd
    dqbd committed May 17, 2023
    Configuration menu
    Copy the full SHA
    a4f942a View commit details
    Browse the repository at this point in the history

  11. Upgrade to 1.0.6 for CRA fixes, mark js-tiktoken/lite as an external … 

    …dependency
    @dqbd
    dqbd committed May 17, 2023
    Configuration menu
    Copy the full SHA
    7fbb3b1 View commit details
    Browse the repository at this point in the history

  12. Remove next.config.js

    @dqbd
    dqbd committed May 17, 2023
    Configuration menu
    Copy the full SHA
    d953f94 View commit details
    Browse the repository at this point in the history