Loss stuck while training a Tacotron2 model on custom dataset in Hindi

[omkarade]

I am training a Tacotron2 model with my custom dataset in Hindi.
Dataset Details: 14hours of data, 22 KHz, 16 bit, Single male Speaker[5sec -13sec]

Issue: the loss seems to be stuck. EvalAudios/audio also noise but word can understand , but audios generated from test sentences are very noise and not understandable.
Question: Should I train more epochs or is there something else or I change model
my data is noise free

[omkarade]

My configuration -----

config = Tacotron2Config(
output_path=output_path,
logger_uri=None,
run_name='run',
project_name='A',
run_description='a',
print_step=234,
plot_step=100,
audio=audio_config,
model_param_stats=False,
wandb_entity=None,
dashboard_logger='tensorboard',
log_model_step=None,
save_step=8000,
print_eval=False,
test_delay_epochs=-1,
run_eval=True,
distributed_backend='nccl',
distributed_url='tcp://localhost:54321',
mixed_precision=False,
epochs=1000,
batch_size=32,
eval_batch_size=8,
grad_clip=5.0,
scheduler_after_epoch=False,
lr=0.0001,
optimizer='RAdam',
optimizer_params= {"betas": [0.9,0.998],"weight_decay": 1e-06},
lr_scheduler='NoamLR',
lr_scheduler_params={"warmup_steps":4000},
num_loader_workers=4,
num_eval_loader_workers=4,
use_noise_augment=False,
use_phonemes=False,
#use_espeak_phonemes= True,
phonemizer=None,
#phoneme_language=hi,
compute_input_seq_cache=False,
text_cleaner='hindi_cleaners',
enable_eos_bos_chars=False,
test_sentences_file='',
characters=character_config,
add_blank=False,
batch_group_size=0,
loss_masking=True,
#sort_by_audio_len=False,
#min_audio_len=1,
#max_audio_len=1000000,
#min_text_len=1,
#max_text_len=inf,
compute_f0=False,
compute_linear_spec=False,
use_language_weighted_sampler=False,
use_gst=False,
gst=None,
gst_style_input=None,
num_speakers=1,
num_chars=127,
r=2,
gradual_training=None,
memory_size=-1,
prenet_type='original',
prenet_dropout=True,
prenet_dropout_at_inference=False,
stopnet=True,
separate_stopnet=True,
stopnet_pos_weight=10.0,
max_decoder_steps=500,
encoder_in_features=512,
decoder_in_features=512,
decoder_output_dim=80,
out_channels=80,
attention_type='dynamic_convolution',
attention_heads=4,
attention_norm='sigmoid',
attention_win=False,
windowing=False,
use_forward_attn=False,
forward_attn_mask=False,
transition_agent=False,
location_attn=True,
bidirectional_decoder=False,
ddc_r=6,
use_speaker_embedding=False,
speaker_embedding_dim=512,
use_d_vector_file=False,
d_vector_file=False,
d_vector_dim=None,
seq_len_norm=False,
decoder_loss_alpha=0.25,
postnet_loss_alpha=0.25,
postnet_diff_spec_alpha=0.25,
decoder_diff_spec_alpha=0.25,
decoder_ssim_alpha=0.25,
postnet_ssim_alpha=0.25,
ga_alpha=5.0,

[thorstenMueller]

Eval alignment is looking good, but alignment for test phrases not. Are test sentences compareable with test phrases in length and structure? 48k steps is not much training. Maybe more training helps.

[omkarade]

loss is still constant.
--> STEP: 27/234 -- GLOBAL_STEP: 104130

--> EVAL PERFORMANCE
| > avg_loader_time: 0.00259 (-0.00065)
| > avg_decoder_loss: 16.03601 (-0.10252)
| > avg_postnet_loss: 17.01884 (-0.09623)
| > avg_stopnet_loss: 0.29321 (-0.00784)
| > avg_ga_loss: 0.00054 (-0.00015)
| > avg_decoder_diff_spec_loss: 0.35906 (+0.00026)
| > avg_postnet_diff_spec_loss: 0.29674 (-0.00084)
| > avg_decoder_ssim_loss: 0.67794 (-0.00284)
| > avg_postnet_ssim_loss: 0.66857 (-0.00257)
| > avg_loss: 9.06018 (-0.05980)
| > avg_align_error: 0.50825 (-0.02271)

[WeberJulian]

It's not what he asked, he asked about test sentences. It looks like test_sentences or test_sentences_file were not specified so it will default to english test sentence which will break the inference.

[omkarade]

i given test sentence are present in train data.

[WeberJulian]

Can you post a few samples of your data ?

[omkarade]

How good are the transcripts?

I use import speech_recognition as sr

[omkarade]

120

is silence also effect on model??

[WeberJulian]

Oh you used speech recognition for the transcript ? It's pretty bad then. Check if the quality of the transcript is good.

[WeberJulian]

trim the trailing silences, it can be really bad for tacotron

[omkarade]

or should I train glowtts ??

[omkarade]

now GLOBAL_STEP are 126360 but still loss's is constant
or should I train Hifi-GAN using Tacotron2 model.is it improve my model??

[WeberJulian]

yeah

[omkarade]

it is not possible to add punctuation in my transcript. Punctuation really matter? or should I train model without Punctuation?

[WeberJulian]

If you want your TTS model to sound natural with punctuation, then you need them.
You can aslo find models that punctuate text for you on hugging face

[omkarade]

@WeberJulian @thorstenMueller my text data is clean it is ok to use this cleaner -->
def hindi_cleaners(text):
return text

[manmaynakhashi]

@omkarade you can use that , use espeak phoneme for better result

[manmaynakhashi]

    use_phonemes=True,
    phonemizer="espeak",
    phoneme_language="hi",
    phoneme_cache_path=os.path.join(output_path, "commentry_cache"),

add this to you config and
install espeak using sudo apt-get install espeak

[omkarade]

how install espeak on google colab notebook ??

[omkarade]

@manmaynakhashi hello sir !
you train any Hindi model???

[omkarade]

@WeberJulian @thorstenMueller I need increase max_decoder_steps ?it's on 500 ?

Decoder stopped with max_decoder_steps 500

[thorstenMueller]

Yes, increasing the value should help when output is cut off at the end.

[WeberJulian]

Except if you have very long test sentence it won't help you, it's probably your model alignment that is broken. Are test sentences sounding ok?

[thorstenMueller]

Indeed, @WeberJulian is right. How long are the phrases you try to synthesize?

[omkarade]

thanks for reply @thorstenMueller @WeberJulian

[lalit-mohanani]

Hii @omkarade I am also facing the same issue. can you help me with that I am trying to train glowtts and tacotron2 with hindi dataset with ljspeech formator how can I modify recipies to train it for hindi.

[tan1abhi]

Hi , how can I use your model as a pre-trained model.