Can't find GPU for model training on Paperspace P4000

[afriedman412]

Hi --
I'm trying to train a basic NER model on a Paperspace P4000 server, Spacy 3.4.1 and python 3.9.16. I used the Quickstart template to created a base_config.cfg etc. Everything seems to be formatted correctly because it works fine when I set the config to use the CPU.

When I set it to use the GPU, it uses the CPU regardless.

python -m spacy init fill-config ./spacy_model_gpu/base_config_gpu.cfg ./spacy_model_gpu/config.cfg

python -m spacy train ./spacy_model_gpu/config.cfg --output ./output --paths.train ./spacy_model_gpu/train.spacy --paths.dev ./spacy_model_cpu/train.spacy

If I add --gpu-id 0 to the spacy train command as suggested here, it throws this error:

  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.9/dist-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/usr/local/lib/python3.9/dist-packages/spacy/cli/_util.py", line 74, in setup_cli
    command(prog_name=COMMAND)
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/typer/main.py", line 532, in wrapper
    return callback(**use_params)  # type: ignore
  File "/usr/local/lib/python3.9/dist-packages/spacy/cli/train.py", line 45, in train_cli
    train(config_path, output_path, use_gpu=use_gpu, overrides=overrides)
  File "/usr/local/lib/python3.9/dist-packages/spacy/cli/train.py", line 67, in train
    setup_gpu(use_gpu)
  File "/usr/local/lib/python3.9/dist-packages/spacy/cli/_util.py", line 581, in setup_gpu
    require_gpu(use_gpu)
  File "/usr/local/lib/python3.9/dist-packages/thinc/util.py", line 195, in require_gpu
    raise ValueError("No GPU devices detected")
ValueError: No GPU devices detected

nvidia-smi gives me this:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.73.05    Driver Version: 510.73.05    CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro P4000        Off  | 00000000:00:05.0 Off |                  N/A |
| 46%   37C    P8     6W / 105W |      2MiB /  8192MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

and here's some further due diligence for GPU existence:

>>> torch.cuda.is_available()
True
>>> torch.cuda.device(0)
<torch.cuda.device object at 0x7f4a4d97a880>
>>> torch.cuda.current_device()
0
>>> torch.cuda.device_count()
1
>>> torch.cuda.get_device_name(0)
'Quadro P4000'

Thanks!!

[danieldk]

Could you post the output of pip list? This error commonly occurs when CuPy isn't installed or when the installed CuPy version does not correspond to the installed CUDA version. To verify this, you could import cupy in a Python REPL and ask it to report the number of available CUDA devices:

>>> import cupy
>>> import cupyx
>>> cupy.cuda.runtime.getDeviceCount()
1

[afriedman412]

cupy was not installed

so I ended up installing cuda toolkit from these instructions:
https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=20.04&target_type=deb_network

and I got this error:

Errors were encountered while processing:
 /tmp/apt-dpkg-install-WIrrAn/116-nvidia-compute-utils-530_530.30.02-0ubuntu1_amd64.deb
 /tmp/apt-dpkg-install-WIrrAn/121-nvidia-utils-530_530.30.02-0ubuntu1_amd64.deb
W: Target Packages (Packages) is configured multiple times in /etc/apt/sources.list:50 and /etc/apt/sources.list.d/cuda-ubuntu2004-x86_64.list:1
E: Sub-process /usr/bin/dpkg returned an error code (1)

[danieldk]

Are you sure that CUDA isn't already installed? From the nvidia-smi output, it seems that the machine has CUDA 11.6, so you can install CuPy with:

pip install cupy-cuda11x

Also see https://docs.cupy.dev/en/stable/install.html#installing-cupy .

[afriedman412]

yep switched it to cupy-cuda11x and now it works!

(to anyone looking on, I had to upgrade attrs above 18.x.x to get around a TypeError)