[FEA] Create separate API for loading the vocabulary file for the subword-tokenizer

[davidwendt]

Is your feature request related to a problem? Please describe.
The current subword_tokenize API in cudf accepts the vocabulary hash-file as a parameter. Profiling shows processing this file into GPU memory is about 14% of the overall execution time for 1M strings 15 bytes each using the CLX BERT vocab hash file (755KB).

Time(%)	Avg (ns)	Range Name
78.3	20994753.8	libcudf:subword_tokenize
14.2	3811397.6	libcudf:load_vocabulary_file
7.3	318007.7	libcudf:make_numeric_column

From some previous discussions with @VibhuJawa @raykallen @BartleyR, multiple subword_tokenize calls over batches of data are likely using the same vocabulary so loading the hash-file for each tokenize call would be wasteful.

Note that the load time is fixed so for a smaller set of strings, the cost against the overall time is higher. Here is a run with only 100K strings with same vocab file:

Time(%)	Avg (ns)	Range Name
53.5	3806982.7	libcudf:load_vocabulary_file
42.2	3007159.7	libcudf:subword_tokenize
4.1	47691.5	libcudf:make_numeric_column

Here the vocab file load is more than half the overall time.

Describe the solution you'd like
Create a separate API to load the hash-table for the tokenize. The resulting object could then be passed on each tokenize call that used the same vocabulary thereby speeding up the processing of multiple batches of data.

This may also align better with the hugging face tokenizer which at least stores the vocabulary file names as an attribute if not the preprocessed hash data somehow internally.

The libcudf API layer already has APIs that separate the file load from the tokenize:

• nvtext::load_vocabulary_file
• nvtext::subword_tokenize accepts a nvtext::hashed_vocabulary struct

So this feature request is to design and implement an equivalent cudf Python API appropriately.
@randerzander

[VibhuJawa]

Create a separate API to load the hash-table for the tokenize. The resulting object could then be passed on each tokenize call that used the same vocabulary thereby speeding up the processing of multiple batches of data.

I have a slightly unrelated question, How hard will it be to support vocab.txt too (instead of just the hash-table file) in a similar API ?

The reasons for this ask are:

a. HuggingFace users often just read the vocab.txt for tokenizing if we can help users skip the process of creating the hash-table it might make porting stuff easier for users and with the proposed feature here this is gonna be anyways be a one time cost so it should be fine .

b. It paves the road for the decode feature (essentially a dictionary lookup) which we don't have in our tokenizer right now ( And we have to rely on cpus to do)

c. The current process of creating hash-table may have some bugs in it (see:#5760 and #5765 ). It is also not under any kind of tests or CI (AFAIK), (feel free to correct me @raykallen / @BartleyR ), so anyways it might be worthwhile adding it to add it to a location in cudf so that it's easier to keep it updated under a CI with tests.

Also, note that the suggestion here is to have both options (vocab.txt as well as vocab-hash-table) which the user can provide as he wishes.

Please feel to shoot this idea down if it's expensive/difficult to implement, This more of a nice to have ask :-)

edit: Added point b on decode .

[davidwendt]

I have a slightly unrelated question, How hard will it be to support vocab.txt too (instead of just the hash-table file) in a similar API ?

This would involve incorporating this python file inside the cudf Python package and having someone maintain it.
https://github.com/rapidsai/clx/blob/267c6d30805c9dcbf80840f222bf31c5c4b7068a/python/clx/analytics/perfect_hash.py

@kkraus14

[github-actions]

This issue has been marked rotten due to no recent activity in the past 90d. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.

[kkraus14]

We had a call offline to discuss a plan on this and this is what we came up with as an initial plan of action:

1. Change load_vocabulary_file function to return shared_ptr
2. Expose load_vocabulary_file function in Python / Cython
3. Create opaque Python wrapper around hashed_vocabulary shared_ptr
4. Expose subword_tokenize call into Python / Cython as appropriate
5. Rework the hash_vocab_utils.py into Python as appropriate in order to generate the files necessary for the load_vocabulary_file function.

Follow up work after the above is finished:

• Expose subword_tokenize call that takes in a raw vocab.txt file (pipelines through above)
• Optimize hash_vocab_utils.py?
• Move hash_vocab_utils functionality to libcudf
• Implement 'hash_vocab_utils' functionality to the GPU

[VibhuJawa]

On the python end-user side, the requested API can loosely follow BertTokenizerFast and we will provide a subset of the functionality as first pass.

class SubwordTokenizer(hash_file: str, max_length: int = 64, stride: int = 48, do_lower: bool = True, do_truncate: bool = False, max_rows_tensor: int = 500):

The function to include in this class is encode:

def encode(Convert a cudf string series into sequence of ids)

## cudf series -> Tuple[cupy.core.core.ndarray, cupy.core.core.ndarray, cupy.core.core.ndarray]

The end user will call it like below:

ser = cudf.Series(['this is the', 'best book'])

tokenizer = SubwordTokenizer('voc_hash.txt', max_length=max_length, stride=stride)

tokens, masks, metadata = tokenizer.encode(ser)

CC: @BartleyR / @raykallen for inputs.