First multilingual-multispeaker YourTTS model speaking Luxembourgish

[mbarnig]

I am proud to present you the first multilingual-multispeaker YourTTS model speaking Luxembourgish. I trained the Coqui-VITS model from scratch with a customized balanced dataset, containing 12.800 samples from 18 speakers and from 5 languages : Lëtzebuergesch, Deutsch, Français, English and Português. The model was trained without phonemes with the following character-set:

characters="abcdefghijklmnopqrstuvwxyz ßàáâãäçèéêëíîïóôõöùúûü",
punctuations="!'(),-.:;? ",
phonemes=None,

The dataset mbarnig/lb-de-fr-en-pt-12800-TTS-CORPUS is available as a zipped archive on my HuggingFace account.

The best-model mbarnig/lb-de-fr-en-pt-coqui-vits-tts, trained with 115.861 steps, can be downloaded from the same account, including the related configuration and encoding files. The Tensorboard of the model-training can be viewed in realtime in this model repository.

Last but not least I created a HuggingFace space mbarnig/lb_de_fr_en_pt_COQUI_VITS_TTS with a live inference demo of the model.

I started my first speech synthesis project in December 2002 with FreeTTS (JAVA) and a trial version of Nuance 0.8. My dream was to create a luxembourgish multilingual speech-synthesizer. It took 20 years to realize this dream. Thanks to the authors of the YourTTS-project, to the founders of Coqui.ai and to the whole Coqui-community who made this possible.

[tieincred]

@mbarnig Hi,
Congratulations!
I am trying to train a very similar model on a different language can you help me with how data needs to be formatted?

[mbarnig]

My data structure is equivalent to the mailabs format. This way you can use the dataset configuration used in the Coqui-multilingual-vits-tts recipe. You will find more details (for example the folder nesting) in my Dataset repository. The audio files are located inside a folder named wavs/. The dataset format in the csv file takes two columns, separated by a pipe (|). The first column specifies the name of the audio-file, without the extension .wav. The second column specifies the transcription of the audio-file in UTF-8 text.

[tieincred]

@mbarnig Thanks a lot!
I am sure this will be really, helpful.

[tieincred]

@mbarnig Sir, I saw your work and checked your website, it is inspirational. Your work has helped a lot for my work progress, I was wondering if you have explored training with using_d_vectors?
I am trying to do that because my final aim is for voice conversion which seemingly requires the model to be trained with d_vectors.
Also, since while I trained the model there was no speaker mapping file being saved, I was wondering if speaker mapping was being saved in your training? something similar to language_ids.
Thanks in Advance.

[mbarnig]

Sorry. My training trials with d_vectors were not yet successful.

[tieincred]

Okay, no problem
If mine work, I'll update!
Thanks again

[wannaphong]

@mbarnig Can you share the speaker encoder source? (CV all lang)
I wants the source for the references.

[mbarnig]

Sorry, I don't understand your question. What is CV all lang ?

[razzium]

Hello @mbarnig,

So cool !

How could I add it to Coqui to test it ?

Thanks.

[jevylux]

@mbarnig That's a good question, I would like to do the same.
I am working on a rhasspy3 implementation and just completed the TTS installation.
when doing a tts --list_models io can see a multilingual/multi-dataset/yours_tts model but i am not sure it is the correct one.

[jevylux]

I tried the following :
Download the files
best_model.pth, config.json, config_se.json, language_ids.json, speakers.pth and model_se.pth
into local folder /home/marc/.local/share/tts/mbarnig/lb-de-fr-en-pt-coqui-vits-tts

the execute the following command
tts --text "Saut Marc, wei geet et dir haut ?" --out_path output/path/speech.wav --model_path "/home/marc/.local/share/tts/mbarnig/lb-de-fr-en-pt-coqui-vits-tts/best_model.pth" --config_path "/home/marc/.local/share/tts/mbarnig/lb-de-fr-en-pt-coqui-vits-tts/config.json" --speakers_file_path "/home/marc/.local/share/tts/mbarnig/lb-de-fr-en-pt-coqui-vits-tts/speakers.pth" --speaker_idx 4

but I keep getting the error No such file or directory: '/home/marc/rhasspy3/speakers.pth'

according to the doc the --speakers_file_path should conatin a json file, but there is no speaker.json
FileNotFoundError: [Errno 2] No such file or directory: '/home/marc/rhasspy3/speakers.pth'