Code

Add fast tokenizer for BARTpho#17254

Code

Closed

datquocnguyen wants to merge 70 commits intohuggingface:mainhuggingface/transformers:mainfrom datquocnguyen:maindatquocnguyen/transformers:mainCopy head branch name to clipboard

+1,716-44

Commits

Commits on May 14, 2022

• Add BartphoTokenizerFast

  datquocnguyen

  committed
  b889418

  Copy full SHA for b889418

• Add BartphoTokenizerFast

  datquocnguyen

  committed
  4c64432

  Copy full SHA for 4c64432

• Add test for BartphoTokenizerFast

  datquocnguyen

  committed
  3496219

  Copy full SHA for 3496219

Commits on May 17, 2022

• Revise BARTpho slow and fast tokenizers to be independent

  datquocnguyen

  committed
  aa77c99

  Copy full SHA for aa77c99

• Fix formatting

  datquocnguyen

  committed
  f5931ed

  Copy full SHA for f5931ed

• Fix formatting

  datquocnguyen

  committed
  4655d6d

  Copy full SHA for 4655d6d

• Fix formatting

  datquocnguyen

  committed
  3618b77

  Copy full SHA for 3618b77

• Merge branch 'main' into main

  datquocnguyen

  authored
  a81aa03

  Copy full SHA for a81aa03

• Fix formatting

  datquocnguyen

  authored
  8bfea5a

  Copy full SHA for 8bfea5a

• Update src/transformers/models/bartpho/tokenization_bartpho_fast.py

  Show description for 069612f
  

  datquocnguyen

  and

  sgugger

  authored
  069612f

  Copy full SHA for 069612f

• Fix formatting

  datquocnguyen

  committed
  7cb6707

  Copy full SHA for 7cb6707

• Remove hardcoded value

  datquocnguyen

  committed
  d76fffa

  Copy full SHA for d76fffa

Commits on May 19, 2022

• Revert the new slow tokenizer to the original slow one

  datquocnguyen

  committed
  6c1b82c

  Copy full SHA for 6c1b82c

• Fix formatting

  datquocnguyen

  committed
  af0aa0e

  Copy full SHA for af0aa0e

Commits on May 21, 2022

• The fast tokenizer with the same tokenization strategy as the slow one

  datquocnguyen

  committed
  1b22570

  Copy full SHA for 1b22570

• Fix formatting

  datquocnguyen

  committed
  a915922

  Copy full SHA for a915922

Commits on May 22, 2022

• Add fast tokenizers for PhoBERT and BERTweet

  datquocnguyen

  committed
  8835d18

  Copy full SHA for 8835d18

• Fix formatting

  datquocnguyen

  committed
  b3677f5

  Copy full SHA for b3677f5

• Add require_torch

  datquocnguyen

  committed
  7d9d477

  Copy full SHA for 7d9d477

Commits on May 27, 2022

• Improved tokenization strategy for BartphoTokenizerFast

  datquocnguyen

  committed
  542cfe2

  Copy full SHA for 542cfe2

• Original BERTweet and PhoBERT tokenizers

  datquocnguyen

  committed
  cd82d13

  Copy full SHA for cd82d13

• Fix format

  datquocnguyen

  committed
  f59b4af

  Copy full SHA for f59b4af

Commits on May 29, 2022

• Improve get_added_vocabulary_hacking

  datquocnguyen

  committed
  0b25f8c

  Copy full SHA for 0b25f8c

• Merge pull request #1 from datquocnguyen/main

  Show description for 176e323
  datquocnguyen

  authored
  176e323

  Copy full SHA for 176e323

• Fix formatting

  datquocnguyen

  committed
  a7aba09

  Copy full SHA for a7aba09

Commits on May 30, 2022

• Fast tokenizers for PhoBERT and BERTweet

  datquocnguyen

  committed
  0047c83

  Copy full SHA for 0047c83

Commits on May 31, 2022

• Fix formatting

  datquocnguyen

  authored
  18e7684

  Copy full SHA for 18e7684

• Fix formatting

  datquocnguyen

  committed
  a41f761

  Copy full SHA for a41f761

Commits on Jun 1, 2022

• Merge pull request #2 from huggingface/main

  Show description for a65cef8
  datquocnguyen

  authored
  a65cef8

  Copy full SHA for a65cef8

• Merge pull request #4 from huggingface/main

  Show description for 7692138
  datquocnguyen

  authored
  7692138

  Copy full SHA for 7692138

Commits on Jun 2, 2022

• Merge pull request #5 from huggingface/main

  Show description for 64a27eb
  datquocnguyen

  authored
  64a27eb

  Copy full SHA for 64a27eb

• Merge pull request #6 from huggingface/main

  Show description for d0ad0de
  datquocnguyen

  authored
  d0ad0de

  Copy full SHA for d0ad0de

Commits on Jun 3, 2022

• Merge pull request #7 from huggingface/main

  Show description for 72651a2
  datquocnguyen

  authored
  72651a2

  Copy full SHA for 72651a2

Commits on Jun 10, 2022

• Merge pull request #8 from huggingface/main

  Show description for a14b6a5
  datquocnguyen

  authored
  a14b6a5

  Copy full SHA for a14b6a5

Commits on Jun 12, 2022

• Merge pull request #9 from huggingface/main

  Show description for 68c3148
  datquocnguyen

  authored
  68c3148

  Copy full SHA for 68c3148

Commits on Jun 17, 2022

• Merge pull request #10 from huggingface/main

  Show description for cf9a23c
  datquocnguyen

  authored
  cf9a23c

  Copy full SHA for cf9a23c

Commits on Jun 23, 2022

• Merge pull request #11 from huggingface/main

  Show description for 321c148
  datquocnguyen

  authored
  321c148

  Copy full SHA for 321c148

Commits on Jul 13, 2022

• Merge pull request #12 from huggingface/main

  Show description for d592599
  datquocnguyen

  authored
  d592599

  Copy full SHA for d592599

Commits on Jul 19, 2022

• Merge pull request #13 from huggingface/main

  Show description for 9630bce
  datquocnguyen

  authored
  9630bce

  Copy full SHA for 9630bce

Commits on Jul 26, 2022

• Merge pull request #14 from huggingface/main

  Show description for 5c0fdac
  datquocnguyen

  authored
  5c0fdac

  Copy full SHA for 5c0fdac

Commits on Jul 28, 2022

• Merge pull request #15 from huggingface/main

  Show description for 3da785c
  datquocnguyen

  authored
  3da785c

  Copy full SHA for 3da785c

Commits on Aug 7, 2022

• Merge pull request #17 from huggingface/main

  Show description for 2f0940f
  datquocnguyen

  authored
  2f0940f

  Copy full SHA for 2f0940f

Commits on Aug 8, 2022

• Merge pull request #18 from huggingface/main

  Show description for 99b1c05
  datquocnguyen

  authored
  99b1c05

  Copy full SHA for 99b1c05

Commits on Aug 9, 2022

• Merge pull request #19 from huggingface/main

  Show description for f29a771
  datquocnguyen

  authored
  f29a771

  Copy full SHA for f29a771

• Merge pull request #20 from huggingface/main

  Show description for 3f0bdce
  datquocnguyen

  authored
  3f0bdce

  Copy full SHA for 3f0bdce

Commits on Aug 10, 2022

• Merge pull request #21 from huggingface/main

  Show description for 0e79af5
  datquocnguyen

  authored
  0e79af5

  Copy full SHA for 0e79af5

Commits on Aug 13, 2022

• Merge pull request #22 from huggingface/main

  Show description for 0db5b71
  datquocnguyen

  authored
  0db5b71

  Copy full SHA for 0db5b71

• Merge pull request #23 from datquocnguyen/fast_tokenizers_BARTpho_PhoBERT_BERTweet

  Show description for c21aadb
  datquocnguyen

  authored
  c21aadb

  Copy full SHA for c21aadb

• Update test_tokenization_bartpho.py

  datquocnguyen

  authored
  d4a6fbb

  Copy full SHA for d4a6fbb

• Merge pull request #24 from datquocnguyen/fast_tokenizers_BARTpho_PhoBERT_BERTweet

  Show description for af834cf
  datquocnguyen

  authored
  af834cf

  Copy full SHA for af834cf

• Update test_tokenization_bartpho.py

  datquocnguyen

  authored
  1bd229f

  Copy full SHA for 1bd229f

• Merge pull request #25 from datquocnguyen/fast_tokenizers_BARTpho_PhoBERT_BERTweet

  Show description for 57a7f67
  datquocnguyen

  authored
  57a7f67

  Copy full SHA for 57a7f67

Commits on Aug 18, 2022

• Merge pull request #27 from datquocnguyen/tmp_branch

  Show description for 2140b76
  datquocnguyen

  authored
  2140b76

  Copy full SHA for 2140b76

• Merge pull request #28 from datquocnguyen/fast_tokenizers_BARTpho_PhoBERT_BERTweet

  Show description for 30275f1
  datquocnguyen

  authored
  30275f1

  Copy full SHA for 30275f1

Commits on Aug 19, 2022

• Merge pull request #29 from huggingface/main

  Show description for e56cb63
  datquocnguyen

  authored
  e56cb63

  Copy full SHA for e56cb63

• Merge pull request #30 from huggingface/main

  Show description for 0b61789
  datquocnguyen

  authored
  0b61789

  Copy full SHA for 0b61789

• Merge pull request #31 from datquocnguyen/fast_tokenizers_BARTpho_PhoBERT_BERTweet

  Show description for 0a4b3c1
  datquocnguyen

  authored
  0a4b3c1

  Copy full SHA for 0a4b3c1

Commits on Aug 23, 2022

• Merge pull request #32 from huggingface/main

  Show description for f214fa0
  datquocnguyen

  authored
  f214fa0

  Copy full SHA for f214fa0

• Merge pull request #33 from datquocnguyen/fast_tokenizers_BARTpho_PhoBERT_BERTweet

  Show description for 8303666
  datquocnguyen

  authored
  8303666

  Copy full SHA for 8303666

Commits on Sep 6, 2022

• Merge pull request #34 from huggingface/main

  Show description for 5a7b682
  datquocnguyen

  authored
  5a7b682

  Copy full SHA for 5a7b682

• Merge pull request #35 from datquocnguyen/fast_tokenizers_BARTpho_PhoBERT_BERTweet

  Show description for 6833da7
  datquocnguyen

  authored
  6833da7

  Copy full SHA for 6833da7

Commits on Sep 18, 2022

• Merge pull request #36 from huggingface/main

  Show description for 85ecfbd
  datquocnguyen

  authored
  85ecfbd

  Copy full SHA for 85ecfbd

• Merge pull request #37 from datquocnguyen/fast_tokenizers_BARTpho_PhoBERT_BERTweet

  Show description for 53a577e
  datquocnguyen

  authored
  53a577e

  Copy full SHA for 53a577e

Commits on Sep 20, 2022

• Merge pull request #38 from huggingface/main

  Show description for 391a440
  datquocnguyen

  authored
  391a440

  Copy full SHA for 391a440

• Merge pull request #39 from datquocnguyen/fast_tokenizers_BARTpho_PhoBERT_BERTweet

  Show description for 2a06fa7
  datquocnguyen

  authored
  2a06fa7

  Copy full SHA for 2a06fa7

Commits on Oct 19, 2022

• Merge pull request #40 from huggingface/main

  Show description for 8048f3a
  datquocnguyen

  authored
  8048f3a

  Copy full SHA for 8048f3a

Commits on Nov 5, 2022

• Merge pull request #41 from huggingface/main

  Show description for 0787507
  datquocnguyen

  authored
  0787507

  Copy full SHA for 0787507

• Merge pull request #42 from datquocnguyen/fast_tokenizers_BARTpho_PhoBERT_BERTweet

  Show description for c0726f1
  datquocnguyen

  authored
  c0726f1

  Copy full SHA for c0726f1

Commits on Nov 23, 2022

• Merge pull request #43 from huggingface/main

  Show description for 0f90212
  datquocnguyen

  authored
  0f90212

  Copy full SHA for 0f90212

• Merge pull request #44 from datquocnguyen/fast_tokenizers_BARTpho_PhoBERT_BERTweet

  Show description for 809f738
  datquocnguyen

  authored
  809f738

  Copy full SHA for 809f738