New Tokenizer API

[erogol]

• Tokenizer API

  Tokenizer API is defined by the TTSTokenizer class. It is intended to provide all the text processing functionalities to a tts model. New tokenizers can also be added by subclassing the TTSTokenizer class.

• Phonemizer API

  Phonemizer API is defined by the BasePhonemizer class and implemented by the ESpeak and Gruut wrappers, ZH_CH,
  JP_JA phonemizers. New phonemizers can be added by implementing the BasePhonemizer class.

• BaseCharacters

  BaseCharacters class provides an API to define the model vocabulary and provide the dictionary to map characters to
  token IDs and back. There are two pre-defined classes inheriting from BaseCharacters. IPAPhonemes and Graphemes that respectively define the IPA phoneme character set for models using phonemes and grapheme set for models using raw characters.

• Punctuations class

  Punctuations class to strip out punctuations and restore them when needed.

• Language specific text normalization routines under TTS.tts.utils.text

  Under TTS.tts.utils.text there are folders for each language to accommodate the text normalization routines that
  are designed for the language.

• GlowTTS recipe and model using the new API

  Other models are not compatible with the new API currently.

TODO:

• Add Tokenizer tests
• Add Punctuation tests
• Add Phonemizer tests
• Add Characters tests
• Update rest of 🐸TTS for the new APIs

[synesthesiam]

I'll take a look this weekend, @erogol 👍

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. You might also look our discussion channels.

[thorstenMueller]

Still relevant.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. You might also look our discussion channels.

[erogol]

This is already merged after rebasing