multus cannot find delegate cni network in pod/deployment without annotations

[paolovisintin]

I'm configuring multus-cni in a this scenario type:
hosts: ubuntu 16.04 LTS
kubernetes version: 1.7
network: macvlan, weave
When i start a pod with annotations, everything is working fine
example :

apiVersion: v1
kind: Pod
metadata:
  name: multus-multi-net-poc
  annotations:
    networks: '[  
        { "name": "weave-conf" },
        { "name": "macvlan" }
    ]'
spec:  # specification of the pod's contents
  containers:
  - name: multus-multi-net-poc
    image: "busybox"
    command: ["top"]
    stdin: true
    tty: true

but when i start a standard pod, this is not able to reach the "running" state and in logs i can see:

Error deleting network: Multus: Err in  reading the delegates: failed to read container data in the path
ailed to read container data in the path(\"/var/lib/cni/multus/c5c70dfd300607437cebb12b87f21651040f27bac143a91f2197fc2b8ff21b7e\"): open /var/lib/cni/multus/c5c70dfd300607437cebb12b87f21651040f27bac143a91f2197fc2b8ff21b7e: no such file or directory"

my cni conf is:

{
    "name": "minion-cni-network",
    "type": "multus",
    "kubeconfig": "/etc/kubernetes/admin.conf",
    "delegates": [
      {
        "type": "weave-net",
        "hairpinMode": true
      }
    ]
}

is there any incompatibility with kubernetes 1.7 or any misconfiguration on my side ?
Thanks

[rkamudhan]

@paolovisintin So you could create network interface using the network object with pod annotation (Using usage-with-kubernetes-tpr-based-network-objects ), and want to use weave-net as the default network, if the pod is not having annotation networks right ? We fixed this issue in the bug fix #14. When you create pod without pod annotation networks, can you provide kubectl describe pod <podname> ??

[paolovisintin]

@rkamudhan exactly this is what I want.
this is the new pod created:

apiVersion: v1
kind: Pod
metadata:
  name: busybox
spec:  # specification of the pod's contents
  containers:
  - name: multus-multi-net-poc
    image: "busybox"
    command: ["top"]
    stdin: true
    tty: true

this is the error in the node:

root@kube02:~# tail -f /var/log/syslog
. . . 
kuberuntime_manager.go:457] Container {Name:multus-multi-net-poc Image:busybox Command:[top] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:default-token-gd02k ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath:}] LivenessProbe:nil ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:true StdinOnce:false TTY:true} is dead, but RestartPolicy says that we should restart it.
Sep  5 00:03:50 kube02 kubelet[2532]: W0905 00:03:50.938166    2532 cni.go:258] CNI failed to retrieve network namespace path: Cannot find network namespace for the terminated container "a5ba2464225bd1065327bf626f93b80d59bbd19d0b77b7039db2ba5526817e0c"
Sep  5 00:03:51 kube02 kubelet[2532]: E0905 00:03:51.091776    2532 cni.go:312] Error deleting network: Multus: Err in  reading the delegates: failed to read container data in the path("/var/lib/cni/multus/a5ba2464225bd1065327bf626f93b80d59bbd19d0b77b7039db2ba5526817e0c"): open /var/lib/cni/multus/a5ba2464225bd1065327bf626f93b80d59bbd19d0b77b7039db2ba5526817e0c: no such file or directory

and this is the describe pod

root@kube-master ~/g/e/k/cni# kubectl describe pod busybox
Name:		busybox
Namespace:	default
Node:		kube02.cloud.evox.it/172.16.201.22
Start Time:	Tue, 05 Sep 2017 00:03:33 +0200
Labels:		<none>
Annotations:	<none>
Status:		Pending
IP:
Containers:
  multus-multi-net-poc:
    Container ID:
    Image:		busybox
    Image ID:
    Port:		<none>
    Command:
      top
    State:		Waiting
      Reason:		ContainerCreating
    Ready:		False
    Restart Count:	0
    Environment:	<none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-gd02k (ro)
Conditions:
  Type		Status
  Initialized 	True
  Ready 	False
  PodScheduled 	True
Volumes:
  default-token-gd02k:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	default-token-gd02k
    Optional:	false
QoS Class:	BestEffort
Node-Selectors:	<none>
Tolerations:	node.alpha.kubernetes.io/notReady:NoExecute for 300s
		node.alpha.kubernetes.io/unreachable:NoExecute for 300s
Events:
  FirstSeen	LastSeen	Count	From				SubObjectPath	Type		Reason			Message
  ---------	--------	-----	----				-------------	--------	------			-------
  3m		3m		1	default-scheduler				Normal		Scheduled		Successfully assigned busybox to kube02.cloud.evox.it
  3m		3m		1	kubelet, kube02.cloud.evox.it			Normal		SuccessfulMountVolume	MountVolume.SetUp succeeded for volume "default-token-gd02k"
  3m		14s		16	kubelet, kube02.cloud.evox.it			Normal		SandboxChanged		Pod sandbox changed, it will be killed and re-created.
  3m		13s		17	kubelet, kube02.cloud.evox.it			Warning		FailedSync		Error syncing pod

Many thanks!

[paolovisintin]

Hello,

I think something went wrong on my previous build

this is what I've just done, for every node:

cd /root/
wget wget https://storage.googleapis.com/golang/go1.9.linux-amd64.tar.gz
tar zxvf go1.9.linux-amd64.tar.gz
export GOPATH=/root/go
export GOROOT=$GOPATH
export GOARCH=amd64
export GOOS=linux
export GOBIN=$GOPATH/bin
export PATH=$PATH:$GOBIN
mkdir git
cd git
git clone https://github.com/Intel-Corp/multus-cni.git
cd multus-cni/
./build
cp bin/multus /opt/cni/bin/

Now everything works perfectly!

Many thanks!

[rkamudhan]

@paolovisintin No problem :) @lmdaly also faced the same bug but we fixed it in the latest commit.

[paolovisintin]

I'm sorry, my previous close was wrong, I have been confused by a pod that created a the file /etc/cni/10-weave.conf and this became the default cni configfile so every new pod was using weave standard conf.

Deleting, in every node, /etc/cni/10-weave.conf the problem appears:

Sep  8 00:56:41 kube02 kubelet[1067]: I0908 00:56:41.549266    1067 kuberuntime_manager.go:457] Container {Name:multus-multi-net-poc Image:busybox Command:[top] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:default-token-gd02k ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath:}] LivenessProbe:nil ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:true StdinOnce:false TTY:true} is dead, but RestartPolicy says that we should restart it.
Sep  8 00:56:41 kube02 kubelet[1067]: W0908 00:56:41.553329    1067 cni.go:258] CNI failed to retrieve network namespace path: Cannot find network namespace for the terminated container "ac4e015715f04d7a71f3b40f29293b0be2bc370ef97b22497afb56619b01a63c"
Sep  8 00:56:41 kube02 kubelet[1067]: E0908 00:56:41.736521    1067 cni.go:312] Error deleting network: Multus: Err in  reading the delegates: failed to read container data in the path("/var/lib/cni/multus/ac4e015715f04d7a71f3b40f29293b0be2bc370ef97b22497afb56619b01a63c"): open /var/lib/cni/multus/ac4e015715f04d7a71f3b40f29293b0be2bc370ef97b22497afb56619b01a63c: no such file or directory
Sep  8 00:56:41 kube02 kubelet[1067]: E0908 00:56:41.738482    1067 remote_runtime.go:114] StopPodSandbox "ac4e015715f04d7a71f3b40f29293b0be2bc370ef97b22497afb56619b01a63c" from runtime service failed: rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod "busybox_default" network: Multus: Err in  reading the delegates: failed to read container data in the path("/var/lib/cni/multus/ac4e015715f04d7a71f3b40f29293b0be2bc370ef97b22497afb56619b01a63c"): open /var/lib/cni/multus/ac4e015715f04d7a71f3b40f29293b0be2bc370ef97b22497afb56619b01a63c: no such file or directory

Pod config:

apiVersion: v1
kind: Pod
metadata:
  name: busybox
spec:  # specification of the pod's contents
  containers:
  - name: multus-multi-net-poc
    image: "busybox"
    command: ["top"]
    stdin: true
    tty: true

Any help is appreciated!

[rkamudhan]

Please try the following conf file

{
    "name": "minion-cni-network",
    "type": "multus",
    "kubeconfig": "/etc/kubernetes/admin.conf",
    "delegates": [{
        "type": "weave-net",
        "hairpinMode": true,
        "masterplugin": "true"
    }]
}

[paolovisintin]

Working, this is the right conf:

{
    "name": "minion-cni-network",
    "type": "multus",
    "kubeconfig": "/etc/kubernetes/admin.conf",
    "delegates": [{
        "type": "weave-net",
        "hairpinMode": true,
        "masterplugin": true <--- with no quote
    }]
}

[uabfra]

Hi, when I use the multus config file to configure a default network cni, I also need to add a "name": "default netname" to the delegate to get it accepted. Maybe this is the version of the CNI I use? or Calico which is my default net.

[paolovisintin]

Hi @nders20000 , I'm correcly using default delegate weave with no name, have never used Calico, but I'll plan to test

[rkamudhan]

@paolovisintin I will work on this issue to skip the master plugin requirement for only one delegate option

[paolovisintin]

@rkamudhan awesome!
At the same time we're working to define a particular namespace (defined in the node) to pick network profiles!
This is useful if you don't want to use DHCP in MacVlan (as seems it's not working) and you have distributed kubernetes nodes in different ip-range areas!

[dougbtv]

I believe that this issue is deprecated by most recent changes in 3.0, if you have any concerns, feel free to open.