RandCropByPosNegLabeld triggers CPU memory overflow when num_sample is very large.

[Jingnan-Jia]

In the function of RandCropByPosNegLabeld, I found that when I set num_samples = 100 (randomly crop 100 3D patches from a high-resolution CT scan), the CPU memory would overflow. This is because all the samples would be generated before they are passed to the next transform which occupy a great number of CPU memory.

I think you can/should use generator to generate samples one by one or batch by batch with a lazy manner so that we can set num_samples any great number.

For example, do not return 100 patches in one time. just yield one patch in one time.

I think you can also learn the manner in sliding_window_inference where the patches are predicted patch by patch. In this manner, we can divide the whole image to as many as patches we need.

[Nic-Ma]

@ericspod @rijobro could you please help share some comments here?
Thanks in advance.

[rijobro]

Hi, I can see why this might be problematic for you, but unless I've misunderstood something I can't fully see why using a generator would solve the problem.

If generating 100 images surpasses your CPU memory limit, then generating them lazily won't solve that, because at some point or another you'll still have all 100 images in memory. Unless you plan on doing something like yielding and then deleting them one-by-one.

I'd be interested in seeing your code to see if we can figure out a use case.

[juampatronics]

I believe gradient accumulation would be a good solution. See for example here. Also Fastai and Pytorch Lightning provide implementations.