Training 14 of max 60 epochs without progress

[eberrigan]

Hello!

I have an 8 GB NVIDIA GeForce RTX 2070 GPU. It is being recognized by root painter and training is happening (I am at 14 epochs) however the client and terminal both say "without progress"--
"""
Training 14 of max 60 epochs without progress
epoch validation duration 0.4068887233734131
epoch train duration 60.9912
"""
The segmentation results still look random

the metrics plot shows no improvement

I have 7 image of roots annotated and followed the instructions for labeling described here: https://github.com/Abe404/root_painter/blob/master/docs/mini_guide.md

Are there any tips for improving the training results? Are there any parameters I should tweak for my system?

Thank you for your help!

Elizabeth

[Abe404]

Hi Elizabeth,

Thanks for reaching out. I'm not yet sure what the problem is but I'm sure we can figure it out.

I can see from the screenshot that your GPU is setup correctly and the network is making predictions, but as you say they are still random. This is normal at the start of training. Typically the first model is created from random weights and will result in predictions such as these.

Are there any tips for improving the training results?

I created a couple of videos demonstrating how to train a model: https://github.com/Abe404/root_painter#videos (the youtube video includes model training later on in the video).

Its not 100% essential but if you have the time, I of course suggest reading the paper, including the protocol and advice in the supplementary material and then watching the videos and following a variant of the colab tutorial yourself.

The colab tutorial is availalbe here: https://colab.research.google.com/drive/104narYAvTBt-X4QEDrBSOZm_DRaAKHtA?usp=sharing

As you already have things setup on your own computer then you could try following the colab tutorial (after the initial setup of the server) but using your own local setup and not via colab.

At this stage in learning RootPainter a common error seems to be with annotation. The initial annotation needs to be assigned in a specific way. The colab tutorial includes screenshots showing how to annotate biopores. You could try out the process with the biopores dataset first, annotating as shown and described in detail in the tutorial just to check the setup is working and to familiarize yourself with the annotation protocol before moving onto roots. While the model is making these random predictions you should focus on annotating both classes clearly and not correctively (described more in the colab tutorial and videos).

You could download the biopores dataset from here https://zenodo.org/record/3754046/files/biopores_750_training.zip (it's pretty quick to train a biopores model). The biopores dataset has already been prepared to be used in training and is a quick way to test everything is working OK.

For debugging your specific issues I would ask to see more detailed server output (more of what you can see in your terminal or what is in your server log file in your sync directory) and to see some of your annotations so far (screenshots is fine). If you like you can share these or we could have a zoom/teams meeting and inspect them together but my first suggestion is to first try and follow the colab tutorial using your local setup with the biopores dataset as the colab tutorial also gives other advice that will come in useful for annotating and training RootPainter models.

It can sometimes take some time for the first trained model to be saved so perhaps you just need to annotate a couple more images, but it could also be that there is a problem with your annotation and there's always a possibility of something unexpected wrong with the software itself. It's hard to know from the information provided so far.

I hope that helps! Please let me know if you figure out the problem or require more guidance from me. I'd like to understand what the problem is as there may also be something that can be improved in the software or documentation to avoid this pitfall.

Kind regards,
Abraham

[eberrigan]

Hi Abraham,

Thanks for your kind reply.

I annotated two more images and let the model train for around 30 minutes after annotation and now my results are much better!

!

Now I will do the corrective annotations.

Thanks for the advice. Your video was helpful.

Best,

Elizabeth