Multus: error in invoke Delegate add - "flannel": open /run/flannel/subnet.env

[psaini79]

I am trying to configure 2 network card in a pod for testing purpose. I am using multus plugin and executed following steps:

1. Downloaded the Multus plugin and build it. Copied the file under /opt/cni/bin. I executed this step on both master as well as on worker node.
2. Following is the kubeadm version:
   kubeadm version: &version.Info Major:"1", Minor:"9", GitVersion:"v1.9.1+2.1.5.el7"
3. I followed Install Doc. I used multus.yaml.
4. When I create ngnix container, I get following error on worker node:

Apr  8 07:20:41 k8dind2 kubelet: E0408 07:20:41.043438   31024 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "nginx-whjjm_default" network: Multus: error in invoke Delegate add - "flannel": open /run/flannel/subnet.env: no such file or directory

Apr  8 07:20:41 k8dind2 kubelet: E0408 07:20:41.043487   31024 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "nginx-whjjm_default(28594443-3afd-11e8-824c-0000170195b8)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "nginx-whjjm_default" network: Multus: error in invoke Delegate add - "flannel": open /run/flannel/subnet.env: no such file or directory
Apr  8 07:20:41 k8dind2 kubelet: E0408 07:20:41.043506   31024 kuberuntime_manager.go:647] createPodSandbox for pod "nginx-whjjm_default(28594443-3afd-11e8-824c-0000170195b8)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "nginx-whjjm_default" network: Multus: error in invoke Delegate add - "flannel": open /run/flannel/subnet.env: no such file or directory

Apr  8 07:20:41 k8dind2 kubelet: E0408 07:20:41.043546   31024 pod_workers.go:186] Error syncing pod 28594443-3afd-11e8-824c-0000170195b8 ("nginx-whjjm_default(28594443-3afd-11e8-824c-0000170195b8)"), skipping: failed to "CreatePodSandbox" for "nginx-whjjm_default(28594443-3afd-11e8-824c-0000170195b8)" with CreatePodSandboxError: "CreatePodSandbox for pod \"nginx-whjjm_default(28594443-3afd-11e8-824c-0000170195b8)\" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod \"nginx-whjjm_default\" network: Multus: error in invoke Delegate add - \"flannel\": open /run/flannel/subnet.env: no such file or directory"

I have no idea how to fix it. Also, I have following physical network cards on my master and worker node:

ens3 : 10.0.20.xx
ens4 : 192.168.16.xx

Do I need to change IP series in multus.yaml?

[psaini79]

I also observed following things:

1. I can see only 1 multus container on master:
   bash-4.2# kubectl get pods --all-namespaces | grep multus
   kube-system kube-multus-ds-9jzrp 1/2 CrashLoopBackOff 6 10m

2. If I describe pod, I see following error:

Events:
  Type     Reason                 Age               From              Message
  ----     ------                 ----              ----              -------
  Normal   SuccessfulMountVolume  8m                kubelet, k8dind2  MountVolume.SetUp succeeded for volume "cni"
  Normal   SuccessfulMountVolume  8m                kubelet, k8dind2  MountVolume.SetUp succeeded for volume "run"
  Normal   SuccessfulMountVolume  8m                kubelet, k8dind2  MountVolume.SetUp succeeded for volume "flannel-cfg"
  Normal   SuccessfulMountVolume  8m                kubelet, k8dind2  MountVolume.SetUp succeeded for volume "multus-token-ldk62"
  Normal   Pulled                 8m                kubelet, k8dind2  Container image "quay.io/coreos/flannel:v0.7.0-amd64" already present on machine
  Normal   Created                8m                kubelet, k8dind2  Created container
  Normal   Started                8m                kubelet, k8dind2  Started container
  Normal   Pulled                 6m (x4 over 8m)   kubelet, k8dind2  Container image "quay.io/coreos/flannel:v0.7.0-amd64" already present on machine
  Normal   Created                6m (x4 over 8m)   kubelet, k8dind2  Created container
  Normal   Started                6m (x4 over 8m)   kubelet, k8dind2  Started container
  Warning  BackOff                3m (x12 over 7m)  kubelet, k8dind2  Back-off restarting failed container

[rkamudhan]

@psaini79 Flannel daemon is not running in your node.. Please check the flannel daemonset deployment in @dougbtv multus.yaml file

[rkamudhan]

@psaini79 @dougbtv I think somewhere command: [ "/bin/sh", "-c", "set -e -x; cp -f /etc/kube-flannel/cni-conf.json /etc/cni/net.d/10-multus.conf; while true; do sleep 3600; done" ] is failing in the flannel container.

[dougbtv]

Thanks for highlighting me -- Kural's right on the money, it sure looks like you don't have a Flannel daemonset running there.

I have a reference Flannel daemonset that uses Multus+Flannel that I use often, it's a jinja2 template that I use as a set of ansible playbooks -- so you'll need to kind of manually parse it. Notably, anything that's {{ in_double_curly_braces }} is a variable and you'll need to fill it out by hand if you're not processing the template.

However, this is the one: https://github.com/redhat-nfvpe/kube-ansible/blob/master/roles/kube-template-cni/templates/multus.yaml.j2

(edit: I realize you're using my gist from last year, but, you might want to try an updated one using this linked sample as it does get some regular use)

[psaini79]

Hello Everyone,

Thanks for providing input. I will check my env and will get back to you.

However, I would like to let you know that I am able to configure 2 network cards using following config file. I took this file from YYGCui user.:

{
"name": "multus-demo-network",
"type": "multus",
"delegates": [
{
"type": "flannel",
"name": "flannel.2",
"subnetFile": "/run/flannel/networks/2.env",
"dataDir": "/var/lib/cni/flannel/2",
"delegate": {
"bridge": "kbr1"
}
},
{
"type": "flannel",
"name": "flannel.1",
"subnetFile": "/run/flannel/subnet.env",
"dataDir": "/var/lib/cni/flannel",
"masterplugin": true,
"delegate": {
"bridge": "kbr0",
"isDefaultGateway": true
}
}
]
}

[dougbtv]

I believe that looks like it's going to use flannel, which will require a daemonset being up, or, at least I assume, when you get back in touch can you post the process you used in whole, e.g. did you require both spinning up the yaml from my own reference with flannel+multus, and then configuring that? Or is that packed inside the yaml? Thanks

[psaini79]

Hi,

I appreciate your help and using your new conf file, I am able to configure 2 network cards in the container using multus plug in.

Though I have few question and I am going test the same:

1. What will happen to my pod if there is any error in macvlan eth card? i.e. eth0 or flannel based IP? or both of them have issues?
2. Can I have 2 flannel based card running on 2 different physical network card? I am able to configure 2 network card inside the container using flannel driver but I don't know how to configure flannel driver on 2 different physical network card?
3. I observed that I can reach i.e. 'ping' Flannel based ip from one pod to another pod but I am unable to reach IPs allocated to macvlan based card. However, I am checking routing table and will debug it further.

[psaini79]

Hi,

Though I am able to see 2 network cards in container, I am unable to ping MACVLAN based Ip from other PODs or hosts.

If I check following on the worker node, I do not see any bridge allocated to MACVLAN:

bash-4.2# brctl show
bridge name     bridge id               STP enabled     interfaces
cni0            8000.0a580af40101       no              vethfc275506
docker0         8000.0242e2df3b04       no

I gave reference of ens4 but it seems that flannel and Macvlan are on same bridge i.e. cni. I used following file:

{
  "name": "multus-demo",
  "type": "multus",
  "delegates": [
    {
      "type": "macvlan",
      "master": "ens4",
      "mode": "bridge",
      "ipam": {
        "type": "host-local",
        "subnet": "192.168.1.0/24",
        "rangeStart": "192.168.1.10",
        "rangeEnd": "192.168.1.20",
        "routes": [
          { "dst": "0.0.0.0/0" }
        ],
        "gateway": "192.168.1.1"
     }
    },
    {
      "type": "flannel",
      "masterplugin": true,
      "delegate": {
        "isDefaultGateway": true
      }
    }
  ]
}

[dougbtv]

@psaini79 -- do you have any further information on this one? If not, could you please close that out?

[psaini79]

I looked at it sometime back and found the issue was in my env. I found that macvlan bridge did not work on few VMs as some policies was enforced at ARP level which didn't allow me to pass packets outside the VM.

Since the issue in my env , I am closing it and will reopen if I test the some new machine. I will add my feedback.
Thanks @dougbtv for your help.