Timeout while waiting for prepare_data to finish

[kfoynt]

📚 Documentation

I initialize two processes. The first process is the one that creates the data inside prepare_data.
The second process waits for the first process.

The problem is that when I generate a large dataset it takes more than 1800 seconds, which is the default DDP timeout.
So my script kept freezing.

I think that the timeout should be emphasized in the documentation when one uses prepare_data in the lightning data module.

Currently, it is mentioned that prepare_data is called only by one process, but it is not mentioned that the other processes are waiting for the first process to finish and that there is a timeout for this. Also, it is not mentioned when the other processes start counting the time. Is it at the beginning of prepare_data or at the end? It seems to be at the beginning, which is what caused my script to timeout.

The timeout error is also not very informative. It just says that it timed out. Given that it is not described how exactly prepare_data works it is difficult to interpret the error.

I guess that the developer assumed that prepare_data won't take that long, but when one deals with very large data, almost certainly it will need more than 1800 seconds.

cc @Borda @justusschock @awaelchli

[awaelchli]

@kfoynt Yes we are aware of that. Unfortunately, PyTorch doesn't allow one to configure/disable the timeout for a single collective (e.g. in this case the barrier), and so this is a pretty strict limitation. Maybe we can get away with making a new process sub group just for these barriers here and configure the timeout there to a very high number.

[kfoynt]

I think that even a warning in the documentation will do. It would have saved me a day of debugging for sure.