more vm-driver=none documentation on resolve.conf

[marcosdiez]

Very interesting issue...

On a new box in AWS, minikube was not working unless I used /run/systemd/resolve/resolv.conf as the resolv.conf (which was by the way solved in kubernetes itself here: https://github.com/kubernetes/kubernetes/pull/63691/files )

But VirtualBox, the same scripts that were provisioning the box in aws were failing if I used k8s v0.11.x, v0.12.x or v0.13.x. (Things did work on k8s v0.14.0)

grep -E "^nameserver" /run/systemd/resolve/resolv.conf |head -n 3 > /tmp/resolv.conf && sudo -E minikube --vm-driver=none start --extra-config=kubelet.resolv-conf=/tmp/resolv.conf

Did solve the problem, but of course it took me 3 days to find out. Therefore I want to share the knowledge and save other people's time.

At first I though this would be to specific to my setup, but my test was done with a new VirtualBox machine provisioning from the official ubuntu 18.04.02 iso, all that done with scripts, so I can replicate the problem and the solution and my local environment should have nothing to do with it. If more users could test this before it get's merged, maybe that's good idea.

[k8s-ci-robot]

Hi @marcosdiez. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[minikube-bot]

Can one of the admins verify this patch?

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: marcosdiez
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: priyawadhwa

If they are not already assigned, you can assign the PR to them by writing /assign @priyawadhwa in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[marcosdiez]

We have a chicken and egg problem here....
The command above works.
Then one restarts the machine.
Then /tmp is wiped
Then minikube does not load.

Saving the file in a place that lasts longer is better but still problematic because if the DNS address changes (and it should because minikube is tends to be used on laptops).

I am thinking about a noble, non over engineered solution here. Maybe echo nameserver 8.8.8.8 > resolv.conf and call it a day....

[marcosdiez]

I am still unsure what to do.

Here I solved the problem in the following by creating a file called minikube.sh with the following relevant lines:

• grep -E "^nameserver" /run/systemd/resolve/resolv.conf |head -n 3 > /tmp/resolv.conf
• sudo -E minikube start --vm-driver=none --extra-config=kubelet.resolv-conf=/tmp/resolv.conf
• sudo rm -rf /etc/systemd/./system/kubelet.service.d /etc/systemd/./system/multi-user.target.wants/kubelet.service

The first line creates a "proper working" resolv.conf
The second runs minikube using the aforementioned file
The the makes sure minikube does not restart.

That's not necessary on vmware player (although it causes no issues), but it is mandatory on virtualbox

Then again this is not a minikube but, but a kubernetes < 1.14 bug.

@tstromberg @afbjorklund what do you suggest ? Just document that ? Write golang code on minkube to "correct" resolve.conf ?

[marcosdiez]

Thinking a little bit more about this, I noticed that /etc/systemd/system/kubelet.service.d/10-kubeadm.conf and /lib/systemd/system/kubelet.service are created by minikube itself.
What are their purpose with the none driver ?

Is it a given that when one runs minikube, one wants it to load automatically upon reboot ?
Shall I write a PR to make that optional or disable it for good on with the none driver (kind of like my disable cache patch ?)

[marcosdiez]

I saw that we use systemd to launch kubelet and restart it in case it fails. That's fine. But do we really want it to relaunch on a machine restart ?

[tstromberg]

Related: #3511