-
Notifications
You must be signed in to change notification settings - Fork 9.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
llama3 custom regex split #6965
Commits on Apr 26, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 6fbab2d - Browse repository at this point
Copy the full SHA 6fbab2dView commit details -
Configuration menu - View commit details
-
Copy full SHA for d2cfc22 - Browse repository at this point
Copy the full SHA d2cfc22View commit details -
Configuration menu - View commit details
-
Copy full SHA for 54f93eb - Browse repository at this point
Copy the full SHA 54f93ebView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1c924e4 - Browse repository at this point
Copy the full SHA 1c924e4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4056dc5 - Browse repository at this point
Copy the full SHA 4056dc5View commit details -
Configuration menu - View commit details
-
Copy full SHA for c8e7d95 - Browse repository at this point
Copy the full SHA c8e7d95View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4c3e882 - Browse repository at this point
Copy the full SHA 4c3e882View commit details -
Configuration menu - View commit details
-
Copy full SHA for a5710a4 - Browse repository at this point
Copy the full SHA a5710a4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7e308ed - Browse repository at this point
Copy the full SHA 7e308edView commit details -
Configuration menu - View commit details
-
Copy full SHA for feeaf4f - Browse repository at this point
Copy the full SHA feeaf4fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 7535803 - Browse repository at this point
Copy the full SHA 7535803View commit details -
Configuration menu - View commit details
-
Copy full SHA for 36d9832 - Browse repository at this point
Copy the full SHA 36d9832View commit details -
Configuration menu - View commit details
-
Copy full SHA for 06d3e69 - Browse repository at this point
Copy the full SHA 06d3e69View commit details -
Configuration menu - View commit details
-
Copy full SHA for c56e19d - Browse repository at this point
Copy the full SHA c56e19dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 7a44e44 - Browse repository at this point
Copy the full SHA 7a44e44View commit details -
Configuration menu - View commit details
-
Copy full SHA for d999cf6 - Browse repository at this point
Copy the full SHA d999cf6View commit details -
Configuration menu - View commit details
-
Copy full SHA for aeafb43 - Browse repository at this point
Copy the full SHA aeafb43View commit details -
Configuration menu - View commit details
-
Copy full SHA for e1b2bf7 - Browse repository at this point
Copy the full SHA e1b2bf7View commit details -
Configuration menu - View commit details
-
Copy full SHA for ed42711 - Browse repository at this point
Copy the full SHA ed42711View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4907e41 - Browse repository at this point
Copy the full SHA 4907e41View commit details -
Configuration menu - View commit details
-
Copy full SHA for e8c206b - Browse repository at this point
Copy the full SHA e8c206bView commit details -
Configuration menu - View commit details
-
Copy full SHA for e989176 - Browse repository at this point
Copy the full SHA e989176View commit details -
Configuration menu - View commit details
-
Copy full SHA for e3f6dc7 - Browse repository at this point
Copy the full SHA e3f6dc7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9b4d63a - Browse repository at this point
Copy the full SHA 9b4d63aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 43e12ce - Browse repository at this point
Copy the full SHA 43e12ceView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1b9b79d - Browse repository at this point
Copy the full SHA 1b9b79dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 8791e94 - Browse repository at this point
Copy the full SHA 8791e94View commit details -
Configuration menu - View commit details
-
Copy full SHA for a774d70 - Browse repository at this point
Copy the full SHA a774d70View commit details -
Configuration menu - View commit details
-
Copy full SHA for c160818 - Browse repository at this point
Copy the full SHA c160818View commit details
Commits on Apr 27, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 96965f6 - Browse repository at this point
Copy the full SHA 96965f6View commit details -
Configuration menu - View commit details
-
Copy full SHA for ad92983 - Browse repository at this point
Copy the full SHA ad92983View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4434c9d - Browse repository at this point
Copy the full SHA 4434c9dView commit details -
Configuration menu - View commit details
-
Copy full SHA for a22645c - Browse repository at this point
Copy the full SHA a22645cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 2affd0b - Browse repository at this point
Copy the full SHA 2affd0bView commit details -
Configuration menu - View commit details
-
Copy full SHA for ce5485a - Browse repository at this point
Copy the full SHA ce5485aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 91eaa41 - Browse repository at this point
Copy the full SHA 91eaa41View commit details -
Configuration menu - View commit details
-
Copy full SHA for 581c4a0 - Browse repository at this point
Copy the full SHA 581c4a0View commit details
Commits on Apr 28, 2024
-
Configuration menu - View commit details
-
Copy full SHA for b97add5 - Browse repository at this point
Copy the full SHA b97add5View commit details -
Configuration menu - View commit details
-
Copy full SHA for d63cc90 - Browse repository at this point
Copy the full SHA d63cc90View commit details -
Configuration menu - View commit details
-
Copy full SHA for e972e6c - Browse repository at this point
Copy the full SHA e972e6cView commit details -
Configuration menu - View commit details
-
Copy full SHA for ee6d1b3 - Browse repository at this point
Copy the full SHA ee6d1b3View commit details -
jaime-m-p committed
Apr 28, 2024 Configuration menu - View commit details
-
Copy full SHA for e11fe2f - Browse repository at this point
Copy the full SHA e11fe2fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 7642973 - Browse repository at this point
Copy the full SHA 7642973View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4e3e6d8 - Browse repository at this point
Copy the full SHA 4e3e6d8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1c888eb - Browse repository at this point
Copy the full SHA 1c888ebView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1545550 - Browse repository at this point
Copy the full SHA 1545550View commit details -
Configuration menu - View commit details
-
Copy full SHA for 491f233 - Browse repository at this point
Copy the full SHA 491f233View commit details -
Configuration menu - View commit details
-
Copy full SHA for e8dd4a1 - Browse repository at this point
Copy the full SHA e8dd4a1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 02fd977 - Browse repository at this point
Copy the full SHA 02fd977View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0f9058c - Browse repository at this point
Copy the full SHA 0f9058cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 7808150 - Browse repository at this point
Copy the full SHA 7808150View commit details -
jaime-m-p committed
Apr 28, 2024 Configuration menu - View commit details
-
Copy full SHA for 5cc4b2c - Browse repository at this point
Copy the full SHA 5cc4b2cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 7b1210f - Browse repository at this point
Copy the full SHA 7b1210fView commit details -
already exists unicode_tolower()
jaime-m-p committedApr 28, 2024 Configuration menu - View commit details
-
Copy full SHA for 6e4d2af - Browse repository at this point
Copy the full SHA 6e4d2afView commit details -
jaime-m-p committed
Apr 28, 2024 Configuration menu - View commit details
-
Copy full SHA for 2a48873 - Browse repository at this point
Copy the full SHA 2a48873View commit details -
jaime-m-p committed
Apr 28, 2024 Configuration menu - View commit details
-
Copy full SHA for 0cf9ed3 - Browse repository at this point
Copy the full SHA 0cf9ed3View commit details
Commits on Apr 29, 2024
-
Configuration menu - View commit details
-
Copy full SHA for ef4cca9 - Browse repository at this point
Copy the full SHA ef4cca9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 43708d2 - Browse repository at this point
Copy the full SHA 43708d2View commit details -
Configuration menu - View commit details
-
Copy full SHA for c68d259 - Browse repository at this point
Copy the full SHA c68d259View commit details -
Configuration menu - View commit details
-
Copy full SHA for af05268 - Browse repository at this point
Copy the full SHA af05268View commit details -
Configuration menu - View commit details
-
Copy full SHA for c21ab18 - Browse repository at this point
Copy the full SHA c21ab18View commit details -
Configuration menu - View commit details
-
Copy full SHA for 866e394 - Browse repository at this point
Copy the full SHA 866e394View commit details -
jaime-m-p committed
Apr 29, 2024 Configuration menu - View commit details
-
Copy full SHA for a0c870d - Browse repository at this point
Copy the full SHA a0c870dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 120cf37 - Browse repository at this point
Copy the full SHA 120cf37View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9a7d430 - Browse repository at this point
Copy the full SHA 9a7d430View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6d6ce93 - Browse repository at this point
Copy the full SHA 6d6ce93View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3202676 - Browse repository at this point
Copy the full SHA 3202676View commit details -
Configuration menu - View commit details
-
Copy full SHA for 80cb312 - Browse repository at this point
Copy the full SHA 80cb312View commit details -
Merge remote-tracking branch 'upstream/gg/bpe-preprocess' into gg/bpe…
…-preprocess
jaime-m-p committedApr 29, 2024 Configuration menu - View commit details
-
Copy full SHA for b66cdd1 - Browse repository at this point
Copy the full SHA b66cdd1View commit details -
jaime-m-p committed
Apr 29, 2024 Configuration menu - View commit details
-
Copy full SHA for 5c38f6e - Browse repository at this point
Copy the full SHA 5c38f6eView commit details -
jaime-m-p committed
Apr 29, 2024 Configuration menu - View commit details
-
Copy full SHA for 1d8fcc0 - Browse repository at this point
Copy the full SHA 1d8fcc0View commit details
Commits on Apr 30, 2024
-
Add alternative regex for custom aplit llama3
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 2cd1eb0 - Browse repository at this point
Copy the full SHA 2cd1eb0View commit details -
jaime-m-p committed
Apr 30, 2024 Configuration menu - View commit details
-
Copy full SHA for 0c6d820 - Browse repository at this point
Copy the full SHA 0c6d820View commit details
Commits on May 3, 2024
-
Add bruteforce random tests for token encoding
jaime-m-p committedMay 3, 2024 Configuration menu - View commit details
-
Copy full SHA for 3e3e283 - Browse repository at this point
Copy the full SHA 3e3e283View commit details -
wip: fixing unicode codepoint ranges
jaime-m-p committedMay 3, 2024 Configuration menu - View commit details
-
Copy full SHA for 4d441e4 - Browse repository at this point
Copy the full SHA 4d441e4View commit details
Commits on May 4, 2024
-
Merge remote-tracking branch 'upstream/master' into gg/bpe-preprocess
jaime-m-p committedMay 4, 2024 Configuration menu - View commit details
-
Copy full SHA for 798b576 - Browse repository at this point
Copy the full SHA 798b576View commit details -
jaime-m-p committed
May 4, 2024 Configuration menu - View commit details
-
Copy full SHA for 69a49ac - Browse repository at this point
Copy the full SHA 69a49acView commit details -
Unicode tables: separator, lowercase, uppercase and whitespace
jaime-m-p committedMay 4, 2024 Configuration menu - View commit details
-
Copy full SHA for 8fd849e - Browse repository at this point
Copy the full SHA 8fd849eView commit details -
llama3 custom regex split: fix \s
jaime-m-p committedMay 4, 2024 Configuration menu - View commit details
-
Copy full SHA for 67832e5 - Browse repository at this point
Copy the full SHA 67832e5View commit details -
jaime-m-p committed
May 4, 2024 Configuration menu - View commit details
-
Copy full SHA for edf375d - Browse repository at this point
Copy the full SHA edf375dView commit details
Commits on May 7, 2024
-
jaime-m-p committed
May 7, 2024 Configuration menu - View commit details
-
Copy full SHA for a5fa2fe - Browse repository at this point
Copy the full SHA a5fa2feView commit details -
jaime-m-p committed
May 7, 2024 Configuration menu - View commit details
-
Copy full SHA for def3d13 - Browse repository at this point
Copy the full SHA def3d13View commit details -
Ignore special tokens for testing
jaime-m-p committedMay 7, 2024 Configuration menu - View commit details
-
Copy full SHA for 7761f8e - Browse repository at this point
Copy the full SHA 7761f8eView commit details
Commits on May 8, 2024
-
jaime-m-p committed
May 8, 2024 Configuration menu - View commit details
-
Copy full SHA for 70ca1fe - Browse repository at this point
Copy the full SHA 70ca1feView commit details -
Refactor random tokenizer test
jaime-m-p committedMay 8, 2024 Configuration menu - View commit details
-
Copy full SHA for 77cbb79 - Browse repository at this point
Copy the full SHA 77cbb79View commit details -
Configuration menu - View commit details
-
Copy full SHA for ea47119 - Browse repository at this point
Copy the full SHA ea47119View commit details
Commits on May 9, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 8de8b6d - Browse repository at this point
Copy the full SHA 8de8b6dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 12a7b69 - Browse repository at this point
Copy the full SHA 12a7b69View commit details