Supporting subword-nmt BPE tokenization

[xwli-chelsea]

Hi,

The BPE tokenization from subword-nmt (https://github.com/rsennrich/subword-nmt) is also a widely used tokenization algorithm. Are there plans to also support it in tf.text?

Thanks!

[thuang513]

BPE is supported in SentencePiece[1], you can use SentencePiece models in tf.text's SentencePiece tokenizer op [2].

[1] https://github.com/google/sentencepiece
[2] https://github.com/tensorflow/text/blob/master/docs/api_docs/python/text/SentencepieceTokenizer.md