-
Notifications
You must be signed in to change notification settings - Fork 88
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Dependency issue between nvidia-container-runtime and nvidia-docker2 #1708
Comments
If you're using a NVIDIA PPA, you have to set it to a higher Pin-Priority. Can't mix and match these dependencies. |
I added to the file Package: *
Pin: origin nvidia.github.io
Pin-Priority: 1002 That gave NVIDIA PPA higher pin priority |
Any reason you can't use the packages from our PPA? |
|
If you need me to update the repository then I will. Better than trying to work around it. |
But we include the docker runtime files in our PPA too, so there can't be any conflicts. |
I don't think you have nvidia-docker2:
Installed: 2.6.0-1
Candidate: 2.6.0-1
Version table:
*** 2.6.0-1 1002
1002 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 Packages
100 /var/lib/dpkg/status
2.5.0-1 1002
1002 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 Packages
2.4.0-1 1002
1002 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 Packages
2.3.0-1 1002
1002 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 Packages
2.2.2-1 1002
1002 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 Packages
2.2.1-1 1002
1002 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 Packages
2.2.0-1 1002
1002 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 Packages
2.1.1-1 1002
1002 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 Packages
2.1.0-1 1002
1002 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 Packages
2.0.3+docker18.09.7-3 1002
1002 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 Packages
2.0.3+docker18.09.6-3 1002
1002 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 Packages
2.0.3+docker18.09.5-3 1002
1002 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 Packages
2.0.3+docker18.09.5-2 1002
1002 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 Packages
2.0.3+docker18.09.4-1 1002
1002 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 Packages
2.0.3+docker18.09.3-1 1002
1002 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 Packages
2.0.3+docker18.09.2-1 1002
1002 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 Packages
2.0.3+docker18.09.1-1 1002
1002 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 Packages
2.0.3+docker18.09.0-1 1002
1002 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 Packages
2.0.3+docker18.06.3-3 1002
1002 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 Packages
2.0.3+docker18.06.2-2 1002
1002 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 Packages
2.0.3+docker18.06.2-1 1002
1002 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 Packages
2.0.3+docker18.06.1-1 1002
1002 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 Packages
2.0.3+docker18.06.0-1 1002
1002 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 Packages
2.0.3+docker18.03.1-1 1002
1002 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 Packages
2.0.3+docker17.12.1-1 1002
1002 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 Packages |
We ship |
I used to use To run it I had to pull the image ( docker run --gpus all --rm -it -p 8888:8888 -p 8787:8787 -p 8786:8786 \
rapidsai/rapidsai:0.19-cuda11.2-runtime-ubuntu20.04-py3.8 It would be interesting to know if Thanks for your insights :) |
That command will work with our |
Indeed it works.
It would be nice to have the Thanks :) |
But some images need |
If found out that if you don't want to mess with pop-os libraries you can try following this git: |
Are we going to be able to use nvidia-container-runtime in PopOS 21.04 with docker? I tried every possible option available but I always get broken packages. |
What do you mean ? For me it works fine (I have Pop OS 21.04 too). Here's nvidia-container-runtime:
Installed: 3.4.0-1pop1~1601325114~20.10~2880fc6
Candidate: 3.4.0-1pop1~1601325114~20.10~2880fc6
Version table:
*** 3.4.0-1pop1~1601325114~20.10~2880fc6 100
100 /var/lib/dpkg/status |
For e.g I have this issue:
I tried installing nvidia-container-runtime or install nvidia-docker2 following the NVIDIA guide and nothing works. What guide did you use to configure nvidia-container-runtime? |
@groenator That's strange Pop OS PPAs have |
Thanks, here is the output:
|
What's the output of |
Hi There, This is the output:
|
I removed the packages from my computer and tried to reinstall them but I couldn't I had the same issue as you. I'm trying to figure out the pb. I'll come back to you just give me a minute |
Sure! Thank you for your help. |
It seems a problem was introduced with Pop OS 21.04 but to resolve this problem you need to follow these steps :
Now |
Hi, Your solution did help, before I tried using the ubuntu20.04 distribution but I never was able to install it because I didn't pin the nvidia.github.io repo. What does it mean in terms of drivers now? Would I get the Nvidia drivers from the PopOS repo?
Thank you for your help. |
In github repo for pop-os/nvidia-container-runtime we can see a problem with the last commit, when I inspected it, it was with pop-os/staging/hirsute/binary-amd64 so this was introduced in Pop OS 21.04 (Hirsute). @mmstick Can you check this problem please ? |
I just wanted to chime in to say that on Pop!_OS 21.04 it seems that this issue may be resolved. Just now I had no problems installing Aside from having to hardcode distribution=ubuntu20.04 \
&& curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - \
&& curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list sudo apt update sudo apt install nvidia-docker2 sudo systemctl restart docker sudo docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi |
hmm, this no longer works for me on popOS! 21.10 / impish 😢 |
On 21.10 this now results in:
|
@afiaka87 thanks for confirming -- yes, that's the same error I am seeing. I also tried apt-pinning as in pop-os/nvidia-container-toolkit#1 with no luck, and with nvidia installation from the default ubuntu repos, always with the same error though. Anyone know if this impacts vanilla ubuntu-21.10 as well as popOS! 21.10? (I know officially NVIDIA is just supporting ubuntu LTS releases, but using those directions has worked well until now). |
guys, just follow official guide https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html it will install following repos : ➜ ~ cat /etc/apt/sources.list.d/nvidia-docker.list and just uncomment those experimental ones , 1.8.0~rc.1-1 will be installed which works great on PopOS 21.10 |
Thanks @hlacik , that worked like a charm! Much appreciated! (I did have to set my apt-pin preferences to prioritize nvidia over popOS as in pop-os/nvidia-container-toolkit#1, then it was happy to upgrade to 1.8.0~rc1-1 and everything was working again perfectly. |
@hlacik Thanks will give this a shot when I have time. Has been making it challenging to work on ML projects. |
Forgot to close this, since this has been resolved 2 weeks ago. |
I personally have had no luck with this approach, so for anyone else experiencing this, I am using the following workaround. I have modified /etc/nvidia-container-runtime/config.toml with "no-cgroups = true" as per NVIDIA/nvidia-docker#1447 which results in
By following NVIDIA/nvidia-docker#1447 (comment) and also adding in the devices I have managed to get it working for now.
|
@mmstick more details: |
@csimpi did you try this solution #1708 (comment) ? |
In popOS 20.04, I manage to solve it by editing apt preferences. Just edit the file like below, update repo and install nvidia-docker2. sudo nano /etc/apt/preferences.d/pop-default-settings
sudo apt update |
@bassemkaroui This is not a solution, this is an ugly workaround that nobody should use. Overriding API priorities without knowing that PopOS devs want to do in the future is the worst idea ever |
Just to be clear, apt priorities (not "API priorities") are a part of apt-based distributions, and a Pop!_OS developer (@mmstick) specifically suggested it higher up in the thread. If you need to use the NVIDIA PPA instead of packages provided by Pop!_OS, then doing this is the correct solution. Of course, it's recommended to just use the packages provided by Pop!_OS instead of the NVIDIA PPA. For Pop!_OS 21.10 and above, only the |
Distribution (run
cat /etc/os-release
):Related Application and/or Package Version (run
apt policy $PACKAGE NAME
):There is a dependency issue between
nvidia-container-runtime
in the PopOS PPA andnvidia-docker2
so I provide the policy of both below:Issue/Bug Description:
I think the PopOS PPA is using an outdated version of
nvidia-container-runtime
which should be 3.5.0 not 3.4.0. Sincenvidia-docker2
requires >=3.5.0 I think the PopOS PPA should be updated?Steps to reproduce (if you know):
Follow install instructions as per the NVIDIA container toolkit guide. Since the distribution
popos20.04
does not exist I substitute$ID$VERSION_ID
withubuntu20.04
so the GPG step is:Once I run
sudo apt-get install -y nvidia-docker2
I get:Expected behavior:
My coworkers followed these steps some months ago without issue and have the inference server with Triton running. So,
sudo apt-get install -y nvidia-docker2
should work without any workaround like the one described below.Other Notes:
I can force my way past this using
I say no to the first option and then yes to the second:
This fixes my issue and then
nvidia-docker2
is installed and works. But then I get this horror:And now PopOS will no longer update.
Ok, so let's try again but say no to the first 2 options and then yes to the third:
This not only installs nvidia-docker2 but I can go to the Pop shop and install OS updates without issue. But I think this workaround shouldn't be needed and a smooth customer experience where
sudo aptitude install nvidia-docker2
just works is preferred. It used to work after all. And I'd like to see PopOS take over the world, hence the bug report.The text was updated successfully, but these errors were encountered: