Long training time for training a model on flu

[ynuozhang]

Hello Dr. Hie,

I encountered a similar problem regarding a super long training time while trying to retrain your model. I had 2 Nvidia Tesla A100 40GB memory graphic cards, AMD EPYC 7763 CPU, and I allocated 450GB RAM. I used one GPU for PBS job submitting and left the other for tracking the graphic card usage.

Initially, when I didn't change anything but just ran python bin/flu.py bilstm --train --test > flu_train.log 2>&1, I encountered out of memory error. Therefore, I had to lower the batch size down to 700 for the program to run. However, for each epoch, the estimated time of completion was ~30h. I'm not sure if there was anything wrong with the settings.

I used watch -d -n 0.5 nvidia-smi to check the GPU usage and got the following message (an image can be found here https://github.com/ynuozhang/runningBiLSTM/blob/main/nvidia-smi.png):
NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4
0 NVIDIA A100-PCI... Off | 00000000:21:00.0 Off | 0 |
| N/A 64C P0 245W / 250W | 38628MiB / 40536MiB | 100% Default | Disabled |

It seemed like the GPU memory was in full usage. I'm not sure where the problem was.
I upload the log here: https://github.com/ynuozhang/runningBiLSTM/blob/main/flu_train_March11-2.log

Any suggestions will be super helpful.
Thanks in advance!

[ynuozhang]

Dear Dr. Hie,

Turns out what I need is the most up-to-date version of tensorflow-gpu (2.8.0). After I upgraded necessary packages instead of using specific versions, the training time fell into an acceptable range. Hopefully this may help others with the same issue with GPU.

Best Regards

[patience111]

Thank you,
one epoch on my GPU is ~ 2.8 hrs with 80k flu HA seqs (> 50k provided by Brian)

[patience111]

Oh, just saw your training record,
BTW, I think there maybe something abnorm for the acc, I don't think biLSTM could have such low acc based on my previous RNN experience.
My first epoch reached 0.97 when training the first biLSTM model and even 0.99 (first epoch) when training the second biLSTM model(which I used checkpoint and it may train based on the first model)

[ynuozhang]

I'm not sure about this (new to this field). My first epoch ends with 0.91 acc, and my second epoch starts with 0.97. There's some overfitting during the training, and overall acc is around 0.98.

[patience111]

Oh, first one with 0.91 makes sense.
just based on your pic, I can only see first one is 'accuracy: 0.0386' and I kind of worried that there might be something abnorm.

[ynuozhang]

Haha thanks. I just can't upload the full training log due to the file limit. Glad that the number makes sense!