Tf32 warnings

[qingpeng9802]

about #6754 .

Description

show a warning if any thing may enable tf32 is detected

Types of changes

• Non-breaking change (fix or new feature that would not break existing functionality).
• Breaking change (fix or new feature that would cause existing functionality to change).
• New tests added to cover the changes.
• Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
• Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
• In-line docstrings updated.
• Documentation updated, tested make html command in the docs/ folder.

[qingpeng9802]

/black

[wyli]

thanks! please see some minor inline comments

[wyli]

/black

[wyli]

Thanks, it looks good to me, testing with more environments

[qingpeng9802]

The failed test is caused by #2161 (also see kornia/kornia#1951). Pytorch only initializes torch.cuda once if torch.cuda is called, and there is no way to revert the initialized state (pytorch/pytorch#28829). The only solution is letting users to adjust the position of os.environ["CUDA_VISIBLE_DEVICES"], which is recommend by a pytorch maintainer.

[qingpeng9802]

/black

[wyli]

The failed test is caused by #2161 (also see kornia/kornia#1951). Pytorch only initializes torch.cuda once if torch.cuda is called, and there is no way to revert the initialized state (pytorch/pytorch#28829). The only solution is letting users to adjust the position of os.environ["CUDA_VISIBLE_DEVICES"], which is recommend by a pytorch maintainer.

ok... in this case how about only calling detect_default_tf32 in the config printing ?

MONAI/monai/config/deviceconfig.py

Line 195 in 65cf5fe

def get_gpu_info() -> OrderedDict:

currently the import monai is a bit slow already.

[qingpeng9802]

Per my test, the function reset_torch_cuda_after_run in the commit 348f089 can resolve the torch.cuda issue. @wyli Could you trigger the GPU CIs to confirm?
Also, this new added warning increases time <0.005% so it should be fine.

[wyli]

/build

[qingpeng9802]

This failed case is kind of interesting🤔
Since this issue pytorch/pytorch#80876 is resolved in PyTorch 1.12.1, @wyli Could you trigger the GPU CI PT113 and PT210?

My test env is Pytorch version: 2.0.0+cu117 NVIDIA-SMI 515.65.01 Driver Version: 516.94 CUDA Version: 11.7, and this env can pass tests.test_set_visible_devices.

[wyli]

Sure, I’ll try to rerun them. I think the utility function in the PR looks good, but adding it and the workaround to monai/__init__.py is a bit risky. (I read the relevant GitHub discussions, most of them are trying to avoid unnecessary early calls to torch.cuda) @ericspod @Nic-Ma

[qingpeng9802]

The key of torch.cuda issue is calling to https://github.com/pytorch/pytorch/blob/v2.0.1/torch/cuda/__init__.py#L219. IMO, importlib.reload should be relatively safe on PyTorch(Python) side. The main risk is that it is unclear whether PyTorch's initialization operation on CUDA is idempotent. That is, we would expect get the same result on the CUDA state after twice initialization.

Actually, I have an ugly but safe solution: subprocess.check_output("nvidia-smi") as alternative 😅.

[qingpeng9802]

As the disscussion above, we prefer to use alt 2. Feel free to add any comment on alt 2. If alt 2. is okay, I can push an alt 2. commit.

Looks like PyNVML is a safe choice. There are some similar usages in PyTorch, such as https://github.com/pytorch/pytorch/blob/v2.0.1/torch/cuda/__init__.py#L771-L794

[wyli]

sure, I think we can go for option 2 if it works fine across the test platforms and the import timing doesn't increase too much python -X importtime -c "import monai"

[qingpeng9802]

/black

[wyli]

/build

[wyli]

/build