Minikube with Podman driver fails to start on NixOS systems

[alias-dev]

Steps to reproduce the issue:

1. Install minikube and podman on a NixOS system.
2. Run minikube start --driver=podman --container-runtime=cri-o.

Full output of failed command if not minikube start:

• minikube v1.23.2 on Nixos 21.11 (Porcupine)
  • MINIKUBE_IN_STYLE=0
• Using the podman driver based on user configuration
• Starting control plane node minikube in cluster minikube
• Pulling base image ...
  E1019 20:20:59.191986 581071 cache.go:201] Error downloading kic artifacts: not yet implemented, see issue podman: load kic base image from cache if available for offline mode #8426
• Creating podman container (CPUs=2, Memory=2200MB) ...
  ! StartHost failed, but will try again: creating host: create: creating: create kic node: create container: sudo -n podman run --cgroup-manager cgroupfs -d -t --privileged --device /dev/fuse --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var:exec -e container=podman --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase:v0.0.27: exit status 125
  stdout:

stderr:
Error: statfs /lib/modules: no such file or directory

• Restarting existing podman container for "minikube" ...
• Failed to start podman container. Running "minikube delete" may fix it: podman inspect ip minikube: sudo -n podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
  stdout:

stderr:
Error: error inspecting object: no such container minikube

X Exiting due to GUEST_PROVISION: Failed to start host: podman inspect ip minikube: sudo -n podman container inspect -f minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container minikube

On NixOS /lib/modules exists at /run/current-system/kernel-modules/lib/modules, so the volume mount fails.

[azahi]

Run to this same issue today.

🤦  StartHost failed, but will try again: creating host: create: creating: create kic node: create container: sudo -n podman run --cgroup-manager cgroupfs -d -t --privileged --device /dev/fuse --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var:exec -e container=podman --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase:v0.0.28: exit status 125
stdout:

stderr:
Error: statfs /lib/modules: no such file or directory

The fix was to just create the /lib/modules directory manually because NixOS doesn't follow FHS. Although, it would be really nice if minikube didn't assume that there must be this directory on the host system. Is there any reason on why this volume is being mounted?

[afbjorklund]

Is there any reason on why this volume is being mounted?

I think it is mounted for modprobe purposes (looking for kernel modules).

modprobe br-netfilter

Others usually fail because of a lack of "config", but that's another story...

We added /proc/config.gz (IKCONFIG) as a workaround.

[afbjorklund]

Couldn't NixOS add a symlink, to make it self compatible ? Wonder how /bin/sh works.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten