Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SigLIP] Add fast tokenizer #29969

Open
wants to merge 29 commits into
base: main
Choose a base branch
from
Open

[SigLIP] Add fast tokenizer #29969

wants to merge 29 commits into from
+139 −54

Commits on Mar 30, 2024

  1. First draft

    @NielsRogge
    NielsRogge committed Mar 30, 2024
    Configuration menu
    Copy the full SHA
    d5d67b7 View commit details
    Browse the repository at this point in the history

Commits on Mar 31, 2024

  1. Fix more tests

    @NielsRogge
    NielsRogge committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    cbde88a View commit details
    Browse the repository at this point in the history

  2. Add test

    @NielsRogge
    NielsRogge committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    de444e9 View commit details
    Browse the repository at this point in the history

Commits on Apr 1, 2024

  1. Remove print statements

    @NielsRogge
    NielsRogge committed Apr 1, 2024
    Configuration menu
    Copy the full SHA
    009fdc6 View commit details
    Browse the repository at this point in the history

Commits on Apr 22, 2024

  1. Merge remote-tracking branch 'upstream/main' into add_siglip_fast_tok… 

    …enizer_bis
    @NielsRogge
    NielsRogge committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    f714af0 View commit details
    Browse the repository at this point in the history

  2. Address comments

    @NielsRogge
    NielsRogge committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    6cd05c2 View commit details
    Browse the repository at this point in the history

  3. Use regex

    @NielsRogge
    NielsRogge committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    d67e40f View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2024

  1. Merge remote-tracking branch 'upstream/main' into add_siglip_fast_tok… 

    …enizer_bis
    @NielsRogge
    NielsRogge committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    f576078 View commit details
    Browse the repository at this point in the history

  2. Rebase

    @NielsRogge
    NielsRogge committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    de04050 View commit details
    Browse the repository at this point in the history

  3. Fix more tests

    @NielsRogge @itazap
    NielsRogge authored and itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    844c95c View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2024

  1. remove strip in tokenize, keep characters used in special tokens, fix… 

    … tests
    @itazap
    itazap committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    50500a5 View commit details
    Browse the repository at this point in the history

Commits on Aug 24, 2024

  1. ruff and FRAMEWORK error fix

    @itazap
    itazap committed Aug 24, 2024
    Configuration menu
    Copy the full SHA
    8ba6e0b View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2024

  1. remove unnecessary assertNotEqual from t5 (and siglip), add copied from)

    @itazap
    itazap committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    bf4f6db View commit details
    Browse the repository at this point in the history

  2. rm copied from

    @itazap
    itazap committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    d850451 View commit details
    Browse the repository at this point in the history

  3. typo

    @itazap
    itazap committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    e73fa01 View commit details
    Browse the repository at this point in the history

Commits on Sep 24, 2024

  1. removing fast class

    @itazap
    itazap committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    cbe0a31 View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2024

  1. updated tests for fast

    @itazap
    itazap committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    d2b2339 View commit details
    Browse the repository at this point in the history

  2. remove dev test file

    Ita Zaporozhets authored and Ita Zaporozhets committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    6379f9d View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into add_siglip_fast_tokenizer_bis

    @itazap
    itazap authored Sep 30, 2024
    Configuration menu
    Copy the full SHA
    6f55733 View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2024

  1. Update src/transformers/models/auto/tokenization_auto.py

    @ArthurZucker
    ArthurZucker authored Oct 1, 2024
    Configuration menu
    Copy the full SHA
    05f8b5c View commit details
    Browse the repository at this point in the history

  2. Update tests/models/llama/test_tokenization_llama.py 

    Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
    @itazap @ArthurZucker
    itazap and ArthurZucker authored Oct 1, 2024
    Configuration menu
    Copy the full SHA
    e296021 View commit details
    Browse the repository at this point in the history

  3. Update src/transformers/models/siglip/__init__.py 

    Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
    @itazap @ArthurZucker
    itazap and ArthurZucker authored Oct 1, 2024
    Configuration menu
    Copy the full SHA
    0f9669b View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2024

  1. Update src/transformers/models/siglip/__init__.py 

    Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
    @itazap @ArthurZucker
    itazap and ArthurZucker authored Oct 2, 2024
    Configuration menu
    Copy the full SHA
    0bca141 View commit details
    Browse the repository at this point in the history

  2. add auto test

    Ita Zaporozhets authored and Ita Zaporozhets committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    80d2f46 View commit details
    Browse the repository at this point in the history

  3. fix test not to try importing Sigliptokenizerfast

    Ita Zaporozhets authored and Ita Zaporozhets committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    2133809 View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2024

  1. import pretrained instead of siglip

    Ita Zaporozhets authored and Ita Zaporozhets committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    3a0e825 View commit details
    Browse the repository at this point in the history

Commits on Oct 18, 2024

  1. rm llama change

    Ita Zaporozhets authored and Ita Zaporozhets committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    e975d96 View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2024

  1. Merge remote-tracking branch 'upstream/main' into add_siglip_fast_tok… 

    …enizer_tres
    @NielsRogge
    NielsRogge committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    dc8ed15 View commit details
    Browse the repository at this point in the history

  2. Make fixup

    @NielsRogge
    NielsRogge committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    dfada5a View commit details
    Browse the repository at this point in the history