Cannot converge after fine-tuning the ASR model.

[Minxiangliu]

I use this citrinet_512.yaml and prepare label like this, and then use stt_zh_citrinet_512 to fine-tuning the model.
The training data is about 50 hours with train-clean-100.tar.gz in LibriSpeech, and the Chinese data is about 50 hours.
But it did not converge normally during training. What could be the reason for this?

Thank you for your time!

[titu1994]

You should directly fine-tune the stt_zh_citrinet_512 model by following the steps in the CTC Finetuning tutorial. That checkpoint should use CER by default as well which might be why you "word error rate" is high - mandarin required CER not WER

[Minxiangliu]

@titu1994 Hi,
My training ended at 500 epoch, but I think the model can still converge.

The results:

The questions:

1. Can I continue to fine-tune my final result of model? If so, how much can I suggest for LR or other settings?

2. How many epochs do you recommend for fine-tuning training for this new language? From my example, my 500 epoch is still not enough.

Many thank!

[titu1994]

Depends on dataset size. We train LS for 100-1000 epochs and see near 1-2% WER improvement when switching from 100 to 1000 epochs.

Yes you can continue finetuning, had the max lr as slightly higher than the final LR and continue training.

[Minxiangliu]

@titu1994 Hi,
Do you mean that the final LR before the model finishes training is 1.692e-9, and the LR I fine-tuned must be slightly higher by 1.692e-9, such as 1.692e-8? Or you can suggest me if it is the current LR, how much I can adjust to allow me to continue to fine-tune the model.
The result:

Thank you for your help!

[titu1994]

You should set a min lr of 1e-5 or 1e-6. This is too small LR.

[Minxiangliu]

OK, My model training experience is not enough, thank you for your opinion, I will try.