torch libtorch_cuda_linalg.so

[jamesdbrock]

Describe the bug

Missing libtorch_cuda_linalg.so at runtime.

To Reproduce

I don't have a minimal reproduction, sorry.

Error:

An error occurred during training.
Traceback (most recent call last):
  File "manufacia/training/pipeline.py", line 135, in run
  File "manufacia/training/patch_distribution_pipeline.py", line 354, in training_process_pipeline
  File "manufacia/training/patch_distribution_pipeline.py", line 167, in _run_training
  File "image_anomalies/patch_distribution.py", line 218, in train
  File "image_anomalies/patch_distribution.py", line 137, in patch_distribution_stats_online
RuntimeError: Error in dlopen: libtorch_cuda_linalg.so: cannot open shared object file: No such file or directory

Expected behavior

pyinstaller does correctly collect the object file _internal/torch/lib/libtorch_cuda_linalg.so.

I think that pyinstaller should symlink this shared object file in the _internal directory like this, but it doesn't:

libtorch_cuda_linalg.so -> torch/lib/libtorch_cuda_linalg.so*

Other libtorch object files are symlinked in the _internal directory:

libtorch_cpu.so -> torch/lib/libtorch_cpu.so*
libtorch_cuda_cpp.so -> torch/lib/libtorch_cuda_cpp.so*
libtorch_cuda_cu.so -> torch/lib/libtorch_cuda_cu.so*
libtorch_cuda.so -> torch/lib/libtorch_cuda.so*
libtorch_python.so -> torch/lib/libtorch_python.so*
libtorch.so -> torch/lib/libtorch.so*

Desktop (please complete the following information):

• OS: Ubuntu
• Python Version: 3.10
• Version of pyinstaller-hooks-contrib: 2024.09
• Version of PyInstaller 6.11.0

[jamesdbrock]

Possibly similar to #591

[jamesdbrock]

I can fix the symlink by adding this line at the end of my .spec file. Is there a better way to do this?

if is_linux:
    os.symlink('torch/lib/libtorch_cuda_linalg.so', 'dist/Manufacia/_internal/libtorch_cuda_linalg.so')

[rokm]

Other libtorch object files are symlinked to the top-level directory because they are linked by some other shared library. The torch/lib/libtorch_cuda_linalg.so is not (it is dynamically loaded at run-time via dlopen).

If adding the symlink to top-level directory solves the issue for you, then it is likely that the search location is incorrectly inferred from one of other symlinked shared libraries (instead of their "true" location). Instead of adding the symlink, does removing all libtorch* symlinks from top-level directory also solve the problem?

---

I think the minimal reproducer would be something that uses linalg on CUDA, e.g.

import torch

A = torch.randn(3, 3, device='cuda:0')
print(f"A={A}")
b = torch.randn(3, device='cuda:0')
print(f"b={b}")
x = torch.linalg.solve(A, b)
print(f"x={x}")

Can you confirm that this reproduces the problem on your system?

It does not seem to do so on mine, unless I deliberately remove _internal/torch/lib/libtorch_cuda_linalg.so (which means that in my case, the symlinks are not an issue, but the codepath does use libtorch_cuda_linalg).

If you can reproduce the issue on your system with this sample, what version of torch are you using, and how did you install it (pip, conda, etc.)?

[jamesdbrock]

Thanks for your attention, @rokm .

Instead of adding the symlink, does removing all libtorch* symlinks from top-level directory also solve the problem?

Yes that does solve the problem.

torch 1.13.1+cu117 installed by Poetry.

Reproduction

I used your script to make a reproduction.

Files

pyproject.toml

[tool.poetry]
name = "testmodule"
version = "1.0.1.0"
description = ""
authors = []

[tool.poetry.dependencies]
python = ">=3.10,<3.11"
torch = [
  { version = "1.13.1+cu117", platform = "linux", source = "torchcu117" },
]
numpy = "1.24.2"
pyinstaller = "^6.11.0"
pyinstaller-hooks-contrib = "^2024.9"


[tool.poetry.scripts]
testmodule = "testmodule.main:main"

[[tool.poetry.source]]
name = "torchcu117"
url = "https://download.pytorch.org/whl/cu117"
priority = "explicit"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

src/testmodule/main.py

import torch

def main():

    A = torch.randn(3, 3, device='cuda:0')
    print(f"A={A}")
    b = torch.randn(3, device='cuda:0')
    print(f"b={b}")
    x = torch.linalg.solve(A, b)
    print(f"x={x}")

if __name__ == "__main__":
    main()

src/testmodule/__init__.py

Commands

poetry install

poetry run testmodule

That should run the test program successfully.

Now we continue and package the test program with pyinstaller.

poetry shell

pyi-makespec src/testmodule/main.py --onedir

pyinstaller -y --clean main.spec

exit

dist/main/main

Running the pyinstaller-packaged program fails with

RuntimeError: Error in dlopen: libtorch_cuda_linalg.so: cannot open shared object file: No such file or directory

[rokm]

Thanks for additional details. I can reproduce the problem with older torch builds from https://download.pytorch.org/whl (e.g., 1.13.1+cu117, 2.0.1+cu117, and 2.1.2+cu118). Later versions seem to have fixed this, and wheels provided on PyPI do not seem to be affected, regardless of the version.

I guess the proper solution here is to suppress the symlink generation for libraries found in torch/lib. We already do the same for tensorflow and for nvidia packages (which are used by tensorflow and PyPI torch builds).

[jamesdbrock]

Thanks @rokm , I believe your PR #836 will solve this issue.

Another object file which suffers from this same issue, which I will write here so that searches will find this issue: torch/lib/libcudnn_cnn_train.so.8.