Cache aware streaming ASR - discrepancies between training and inference

[tobygodwin]

Hello,

Firstly thanks for putting together a great tool.

I've trained a cache aware streaming fastconformer model using the standard config and the standard ctc asr training script .

The model trains well, but when I go to evaluate the streaming performance using the cache aware streaming inference script there is a drastic difference between offline and streaming performance (13% and 70% respectively), which surprised me because I thought that I had limited the attention context using chunked_limited during training, and that if this was matched during inference then equivalent performance would be achieved. When I increase the --chunk_size during streaming inference, performance improves significantly to 19% which implies to me that the original model was trained without any restrictions on attention context.

I've confirmed that the default streaming configs are the same between training and inference (the standard parameters returned by setup_streaming_params() for my config). One thing I did notice is that cache_last_channel and cache_last_time are not being passed during the training forward pass, but they are during the streaming inference forward pass. I guess this results in a mismatch between training and inference which would lead to differences in performance.

Am I missing something? I would have thought that cache_last_channel etc should be included in the training procedures.

Any guidance would be much appreciated.

Many thanks

[VahidooX]

Which branch or nemo version are you using?
Caches are not need during training. They are being used during inference as whole input is not given at once. You should get the exact same accuracy with the streaming script. We have made some changes recently. Let me check everything and get back to you.

[tobygodwin]

thanks for your response.

I'm using the master branch on this commit:

52d3af7

[VahidooX]

I found a bug in updating the caches and fixed it in this PR:
#7034

Would you please try it out?

[tobygodwin]

Sorry for the delay. That worked perfectly! Thanks very much

[sangeet2020]

Hello @titu1994 ,

I am in the process of training this model for German language, and I obtained these training stats after 100 epochs.

Looking at the training loss and train wer, I believe the losses have not yet converged. Do you think something is wrong with the parameters/hyper-params. or does the training needs to continue?

Thank You

[titu1994]

Please don't comment on old discussions, open a new issue, since your discussion didn't seem related to this one.

In any case, it looks like youre convergence is slow. Try using a pretrained model to speed it up. If it's chunk aware specifically, then training longer will help)

[sangeet2020]

Apologies. Will do so. thank you