[Bug] tensorrt may randomly release imported resources before deallocate

[AllentDan]

Description

After inheriting tensort.IGpuAllocator and override deallocate function in a script. The resouces may get released randomly before deallocate execution.

Environment

TensorRT Version: 8.4.1.5
NVIDIA GPU: A100 or Geforce 1660
NVIDIA Driver Version: 470.103.01
CUDA Version: 11.3
CUDNN Version: corresponding to cuda 11.3
Operating System: ubuntu 18.04

Steps To Reproduce

here is the code implementation. And this is our work-around to fix it.

[zerollzeng]

@pranavm-nvidia Not sure whether this is an issue in our python api. Can you take a look first?

[pranavm-nvidia]

Because of how our bindings are implemented, you have to explicitly instantiate the base class in your __init__ function (i.e. you cannot use super()):

trt.IGpuAllocator.__init__(self)

I'll update the API docs to clarify this.

[AllentDan]

Because of how our bindings are implemented, you have to explicitly instantiate the base class in your __init__ function (i.e. you cannot use super()):

trt.IGpuAllocator.__init__(self)

I'll update the API docs to clarify this.

I tried the method, the error still exists in my local machine.

[AllentDan]

Since the bug sometimes is not easy to be triggered, you may try print(torch) inside the deallocate function. There will be some error logs as expected.

[pranavm-nvidia]

From my reading of the PR you linked, it sounds like torch or torch.cuda is somehow being unloaded. I don't think that's something TensorRT would be able to do.
Do you know when torch is being released?

[AllentDan]

At the end of the program, before TorchAllocator got released. Not only torch or torch.cuda, logging was also released before deallocate been called.

[pranavm-nvidia]

Can you try making sure that any TRT objects you create (engine, context, etc.) are scoped? If they're in the global scope, then they might be freed after the torch module has been unloaded.

[ttyio]

closing since no activitity for more than 3 weeks, thanks!