Fix nvidia-gpu with kvm-driver

[betaboon]

hello,

took me a while to figure this out, but this fixes kvm-nvidia-gpu support for me.

this might also resolve #6912

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: betaboon (e9e3f81)

[k8s-ci-robot]

Welcome @betaboon!

It looks like this is your first PR to kubernetes/minikube 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/minikube has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @betaboon. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[minikube-bot]

Can one of the admins verify this patch?

[medyagh]

Thank you ver much for fixing this !

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: betaboon, medyagh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [medyagh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[medyagh]

betaboon thank you for fixing this, do u mind signing the CLA

[betaboon]

just did.

FYI:
this only seems to work for Latest Legacy GPU version (390.xx series): [390.147]

it doesnt work for either of:

Latest Production Branch Version: 510.60.02
Latest New Feature Branch Version: [495.46]

this it seems like i can only get cuda9 here.

I'm tinkering around with getting 510 working.

if i figure that out, I'll send it in another PR tho ;)

[medyagh]

just did.

FYI: this only seems to work for Latest Legacy GPU version (390.xx series): [390.147]

it doesnt work for either of:

Latest Production Branch Version: 510.60.02
Latest New Feature Branch Version: [495.46]

this it seems like i can only get cuda9 here.

I'm tinkering around with getting 510 working.

if i figure that out, I'll send it in another PR tho ;)

Thank you for figuring it out, do u mind adding it our docs ?
https://minikube.sigs.k8s.io/docs/tutorials/nvidia_gpu/

and mention it currently been known to work with that specific series

I would happily review your PR if you decide to make it work with newer series,

btw I am curious what is the use case of using GPU for you if you like to share

[betaboon]

i figured out what is preventing newer drivers to work.
but I'm not yet sure how to fix this, as i am lacking the knowledge about buildroot.

here's the gist:

• current nvidia-drivers require Module.symvers to be present in the kernel-source directory.
• for Module.symvers to be build CONFIG_MODVERSIONS=y is required

here's what i tried (which lead to partial success):

• enable CONFIG_MODVERSIONS in linux_defconfig
• build and run iso (Module.symvers is not available in buildroot/output/build/linux-4.19.202, but NOT in the resulting iso)
• copy Module.symvers into vm using minikube cp
• change nvidia-driver-installer-daemonset to make Module.symver available in /usr/src/linux-4.19.202
• apply the nvidia-driver-installer-daemonset
• profit (aka driver install works with 510.60.02)

two possible solutions:

• figure out how to include Module.symvers in the iso
• change the entrypoint-script of the nvidia-driver-installer to run make modules which should recreate it

[spowelljr]

Not sure if this is the route you want to go, but if you want an example of how to copy a file into the ISO you can look at #12081.

If you look at buildkit-bin.mk you can see how files from the repo are copied into the ISO.

[betaboon]

ok i converted this PR to a draft for now.

@spowelljr thanks for the pointers.

i just pushed a commit that fixes it for the latest nvidia-drivers.
there is a hack in there tho.

this is what it does:

• change the kernel-config to enable CONFIG_MODVERSIONS so that Module.symversgets generated
• extend the buildroot linux-target to include Module.symvers in /lib/modules/...
• change the command of the nvidia-driver-installer-daemonset to monkeypatch /entrypoint.sh (this is the hack)

the hack could and should be removed, but for that the entrypoint.sh in the upstream-package should has to be adapted.

[betaboon]

@medyagh just fyi there is something to review here now :) (dont want to hurry you tho )

[medyagh]

thank you betaboon I look forward to see more contributions from you !