Auto3DSeg fails with 5 fold but not 2 on multi-GPU setup

[ashleymicuda]

I've been trying to run the https://github.com/Project-MONAI/tutorials/blob/main/auto3dseg/notebooks/auto_runner.ipynb tutorial on a dual GPU setup with my own data (~400 GB). The autorunner works and trains when I have my dataset split into 2 fold, however when I increase to 5 fold I get the following error:

Notably, the code runs for a few hours then crashes. It seems that Auto3DSeg tries to load in all the data before training starts, however I can't figure out why it would run with 2 fold and not 5. I can run 5 fold on one GPU which suggests it's a multiprocessing error (as the screenshot also suggests). Any ideas of how to run the 5-fold on multi-GPUs or if there's something in the default settings I need to change?

[diazandr3s]

Hi @ashleymicuda,

Thanks for sharing this.
Could you please confirm the number of images/labels per fold when running the 2-fold and 5fold set up?

[ashleymicuda]

Thanks for the quick reply. It's the same dataset used for both. For 5 fold I have 400 in each and for 2 fold I had 1600 in one and 400 in the other.