Docker 1.7 cannot mount secrets #3072

liggitt · 2015-06-10T21:05:46Z

When we started using secrets for deployments, we noticed that containers are not able to read mounted secrets.

The pod definitions contain Volume and VolumeMount definintions, and docker inspect shows the volumes as expected, but the container cannot read files from the mount point.

This surfaces (in the case of the deployer pod) as this error:

F0610 18:32:48.935073       1 deployer.go:65] User "system:anonymous" cannot get replicationcontrollers in project "myproject"

docker inspect <container> shows the volume mount:

...
        "Env": [
...
            "BEARER_TOKEN_FILE=/var/run/secrets/kubernetes.io/serviceaccount/token",
...
    "HostConfig": {
        "Binds": [
            "/openshift.local.volumes/pods/12f168c2-0fad-11e5-a1f9-525400553cbb/volumes/kubernetes.io~secret/deployer-token-2jxjw:/var/run/secrets/kubernetes.io/serviceaccount:ro",
...
        ],
...
    "Volumes": {
...
        "/var/run/secrets/kubernetes.io/serviceaccount": "/openshift.local.volumes/pods/12f168c2-0fad-11e5-a1f9-525400553cbb/volumes/kubernetes.io~secret/deployer-token-2jxjw"
    },
    "VolumesRW": {
...
        "/var/run/secrets/kubernetes.io/serviceaccount": false
    },
    "VolumesRelabel": {
...
        "/var/run/secrets/kubernetes.io/serviceaccount": "ro"
    }
...

The text was updated successfully, but these errors were encountered:

liggitt · 2015-06-10T21:08:00Z

@csrwng @smarterclayton was there a fix for the boot2docker tmpfs issue?

smarterclayton · 2015-06-10T21:09:01Z

The containerized flag in the kubelet should allow you to mount.

On Jun 10, 2015, at 5:08 PM, Jordan Liggitt notifications@github.com wrote:

@csrwng @smarterclayton was there a fix for the boot2docker tmpfs issue?

—
Reply to this email directly or view it on GitHub.

liggitt · 2015-06-10T21:18:17Z

curious that the kubelet doesn't complain about creating the mount

gravis · 2015-06-10T21:46:13Z

What is that ~ in kubernetes.io~secret?

"/var/run/secrets/kubernetes.io/serviceaccount": "/openshift.local.volumes/pods/12f168c2-0fad-11e5-a1f9-525400553cbb/volumes/kubernetes.io~secret/deployer-token-2jxjw"

sspeiche · 2015-06-10T21:48:59Z

Looks like this is what is keeping me from docker pulling latest and having the build successfully publish to the registry. My happy path dev exp based on docker launched origin isn't happy :(

smarterclayton · 2015-06-10T21:50:32Z

I have a todo to fix this - basically we need to set the containerized flag and then add it to the e2e tests so it doesn't break.

----- Original Message -----

Looks like this is what is keeping me from docker pulling latest and having
the build successfully publish to the registry. My happy path dev exp based
on docker launched origin isn't happy :(

Reply to this email directly or view it on GitHub:
#3072 (comment)

gravis · 2015-06-11T17:35:00Z

Any workaround available for this, until it's fixed for good?

smarterclayton · 2015-06-11T17:38:24Z

You have to write out a node config file and then set a kubeletArguments of "containerized" with "true" as the argument (you need to specify it as a nested string array in the yaml - kubeletArgument is map [string] -> []string)

kubeletArguments:
containerized:

true

----- Original Message -----

Any workaround available for this, until it's fixed for good?

Reply to this email directly or view it on GitHub:
#3072 (comment)

gravis · 2015-06-11T19:42:36Z

I can't write a difference node config, as it is created when the container starts :)
Any plan to update the v0.6 docker image with this?
Thanks

smarterclayton · 2015-06-11T19:44:19Z

It'll probably be in 0.6.1

----- Original Message -----

I can't write a difference node config, as it is created when the container
starts :)
Any plan to update the v0.6 docker image with this?
Thanks

Reply to this email directly or view it on GitHub:
#3072 (comment)

gravis · 2015-06-11T19:45:26Z

Ok thanks. The sooner the better, we're stuck with this :)

smarterclayton · 2015-06-11T20:01:42Z

Try #3112 - you'll need to build your own openshift/origin image from the branch with hack/build-release.sh and then hack/build-images.sh. Still testing myself.

----- Original Message -----

Ok thanks. The sooner the better, we're stuck with this :)

Reply to this email directly or view it on GitHub:
#3072 (comment)

gravis · 2015-06-11T20:11:48Z

👍 will test that. Thanks!

gravis · 2015-06-11T21:10:36Z

Rah, I can't compile a new version using boot2docker:

++ Building go targets for linux/amd64: cmd/openshift
        # github.com/openshift/origin/cmd/openshift
/usr/lib/golang/pkg/tool/linux_amd64/6l: running gcc failed: Cannot allocate memory

I could raise the memory on vbox, but this implies destroying the current vm, and I can't remove everything... I will wait for your tests then. Let me know if the new image can be pulled from somewhere.
Thanks

gravis · 2015-06-12T18:58:37Z

I have just rebuilt the image from master, and the registry won't start either:

W0612 18:56:12.839622       1 container_manager_linux.go:68] [ContainerManager] Failed to ensure Docker is in a container: failed to find pid of Docker container: exit status 1
E0612 18:56:17.800696       1 kubelet.go:1111] Unable to mount volumes for pod "docker-registry-1-deploy_default": exit status 1; skipping pod
E0612 18:56:17.817346       1 pod_workers.go:108] Error syncing pod 79031390-1134-11e5-874a-8277bc1719bf, skipping: exit status 1

I'm running openshift with:

$ docker run -d -name "origin" \
 --privileged --net=host \
 -v /:/rootfs:ro -v /var/run:/var/run:rw -v /sys:/sys:ro -v /var/lib/docker:/var/lib/docker:rw \
 openshift/origin start --public-master=$(boot2docker ip)

$ docker run -it --rm openshift/origin:latest version
openshift v0.6-179-gcc71b54
kubernetes v0.17.1-804-g496be63

Should I open another issue?

smarterclayton · 2015-06-12T19:13:49Z

Can you repro with --loglevel=5 and look for the same log line? It should print the mount output.

Did you rebuild the base images as well?

On Jun 12, 2015, at 2:58 PM, Philippe Lafoucrière notifications@github.com wrote:

I have just rebuilt the image from master, and the registry won't start either:

W0612 18:56:12.839622 1 container_manager_linux.go:68] [ContainerManager] Failed to ensure Docker is in a container: failed to find pid of Docker container: exit status 1
E0612 18:56:17.800696 1 kubelet.go:1111] Unable to mount volumes for pod "docker-registry-1-deploy_default": exit status 1; skipping pod
E0612 18:56:17.817346 1 pod_workers.go:108] Error syncing pod 79031390-1134-11e5-874a-8277bc1719bf, skipping: exit status 1
I'm running openshift with:

$ docker run -d -name "origin"
--privileged --net=host
-v /:/rootfs:ro -v /var/run:/var/run:rw -v /sys:/sys:ro -v /var/lib/docker:/var/lib/docker:rw
openshift/origin start --public-master=$(boot2docker ip)
$ docker run -it --rm openshift/origin:latest version
openshift v0.6-179-gcc71b54
kubernetes v0.17.1-804-g496be63
Should I open another issue?

—
Reply to this email directly or view it on GitHub.

gravis · 2015-06-12T20:07:46Z

I0612 20:06:36.080128       1 empty_dir_linux.go:38] Determining mount medium of /var/lib/openshift/openshift.local.volumes/pods/873a34bc-113e-11e5-bf0e-22549f88d0e6/volumes/kubernetes.io~secret/deployer-token-wjx82
I0612 20:06:36.092700       1 nsenter_mount.go:139] findmnt command: nsenter [--mount=/rootfs/proc/1/ns/mnt -- /usr/bin/findmnt -o target --noheadings --target /var/lib/openshift/openshift.local.volumes/pods/873a34bc-113e-11e5-bf0e-22549f88d0e6/volumes/kubernetes.io~secret/deployer-token-wjx82]
I0612 20:06:36.133421       1 empty_dir_linux.go:48] Statfs_t of %v: %+v/var/lib/openshift/openshift.local.volumes/pods/873a34bc-113e-11e5-bf0e-22549f88d0e6/volumes/kubernetes.io~secret/deployer-token-wjx82{1635083891 4096 4762473 3929050 3681369 1218224 1171153 {[0 0]} 242 4096 4128 [0 0 0 0]}
I0612 20:06:36.133538       1 docker.go:321] Docker Container: /origin is not managed by kubelet.
I0612 20:06:36.133534       1 empty_dir.go:202] pod 873a34bc-113e-11e5-bf0e-22549f88d0e6: mounting tmpfs for volume not-used with opts []
I0612 20:06:36.133580       1 nsenter_mount.go:79] nsenter Mounting tmpfs /var/lib/openshift/openshift.local.volumes/pods/873a34bc-113e-11e5-bf0e-22549f88d0e6/volumes/kubernetes.io~secret/deployer-token-wjx82 tmpfs []
I0612 20:06:36.133612       1 nsenter_mount.go:82] Mount command: nsenter [--mount=/rootfs/proc/1/ns/mnt -- /usr/bin/mount -t tmpfs -o  tmpfs /var/lib/openshift/openshift.local.volumes/pods/873a34bc-113e-11e5-bf0e-22549f88d0e6/volumes/kubernetes.io~secret/deployer-token-wjx82]
I0612 20:06:36.176765       1 nsenter_mount.go:86] Output from mount command: nsenter: failed to execute /usr/bin/mount: No such file or directory
E0612 20:06:36.176972       1 kubelet.go:1111] Unable to mount volumes for pod "docker-registry-1-deploy_default": exit status 1; skipping pod
I0612 20:06:36.177006       1 kubelet.go:2051] Generating status for "docker-registry-1-deploy_default"
I0612 20:06:36.177512       1 server.go:569] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"docker-registry-1-deploy", UID:"873a34bc-113e-11e5-bf0e-22549f88d0e6", APIVersion:"v1", ResourceVersion:"182", FieldPath:""}): reason: 'failedMount' Unable to mount volumes for pod "docker-registry-1-deploy_default": exit status 1
I0612 20:06:36.196478       1 kubelet.go:1990] pod waiting > 0, pending
E0612 20:06:36.196881       1 pod_workers.go:108] Error syncing pod 873a34bc-113e-11e5-bf0e-22549f88d0e6, skipping: exit status 1
I0612 20:06:36.197236       1 server.go:569] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"docker-registry-1-deploy", UID:"873a34bc-113e-11e5-bf0e-22549f88d0e6", APIVersion:"v1", ResourceVersion:"182", FieldPath:""}): reason: 'failedSync' Error syncing pod, skipping: exit status 1

Relevent part I think: Output from mount command: nsenter: failed to execute /usr/bin/mount: No such file or directory

gravis · 2015-06-12T20:09:48Z

mount is available in the container:

[root@boot2docker openshift]# ls /usr/bin/mount
/usr/bin/mount
[root@boot2docker openshift]# which mount
/usr/bin/mount

gravis · 2015-06-12T20:14:25Z

If it's the host mount command, it's /bin/mount, not /usr/bin/mount. You should of course use which mount instead of absolute paths.

gravis · 2015-06-12T20:16:53Z

rho
https://github.com/GoogleCloudPlatform/kubernetes/blob/5520386b180d3ddc4fa7b7dfe6f52642cc0c25f3/pkg/util/mount/nsenter_mount.go#L97

gravis · 2015-06-12T20:18:41Z

after aliasing mount on the host, I have this:

I0612 20:17:48.342643       1 nsenter_mount.go:82] Mount command: nsenter [--mount=/rootfs/proc/1/ns/mnt -- /usr/bin/mount -t tmpfs -o  tmpfs /var/lib/openshift/openshift.local.volumes/pods/873a34bc-113e-11e5-bf0e-22549f88d0e6/volumes/kubernetes.io~secret/deployer-token-wjx82]
I0612 20:17:48.353253       1 nsenter_mount.go:86] Output from mount command: mount: mounting tmpfs on /var/lib/openshift/openshift.local.volumes/pods/873a34bc-113e-11e5-bf0e-22549f88d0e6/volumes/kubernetes.io~secret/deployer-token-wjx82 failed: No such file or directory
E0612 20:17:48.353584       1 kubelet.go:1111] Unable to mount volumes for pod "docker-registry-1-deploy_default": exit status 255; skipping pod

luxas · 2016-03-04T19:46:32Z

It says it's shared:

83 15 179:2 /var/lib/kubelet /var/lib/kubelet rw,noatime shared:1 - ext4 /dev/root rw,data=ordered

Did you get different output?

csrwng · 2016-03-04T19:48:08Z

no, that's what it looks like for me as well.

luxas · 2016-03-04T20:57:45Z

Any ideas someone why docker doesn't validate it?

gravis · 2016-03-04T21:19:09Z

Do you use systemd to run docker?

luxas · 2016-03-05T05:42:32Z

@gravis Yeah

/usr/bin/docker daemon -H unix:///var/run/docker.sock -s overlay --exec-opt native.cgroupdriver=cgroupfs

gravis · 2016-03-05T13:31:21Z

Make sure your service file contains the line MountFlags=slave:

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket
Requires=docker.socket

[Service]
Type=notify
ExecStart=/usr/bin/docker daemon -H fd://
MountFlags=slave
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
TimeoutStartSec=0

[Install]
WantedBy=multi-user.target

marun · 2016-03-05T17:06:37Z

systemd defaults MountFlags to 'shared', so mount propagation (and secrets) would also work by removing MountFlags or setting to empty string. A nice way to set this is via a drop in unit file (e.g. /etc/systemd/system/docker.service.d/clear_mount_propagation.conf) to override the setting in the unit file without modifying it directly:

[Service]
MountFlags=

luxas · 2016-03-05T20:15:25Z

@marun @gravis @csrwng Thanks for your help! Managed now to run it... on a Raspberry Pi! I didn't tell you that, but all these commands ran on my Pi. Resetting MountFlags= was the solution to my problem. Now we've got rid of --containerized in kubernetes-on-arm 👍

gravis · 2016-03-05T21:48:22Z

Wow, nice!

liggitt · 2016-05-10T15:14:59Z

@pmorie is this still valid?

klaus · 2016-07-04T13:10:33Z

as of today, on stretch/sid debian it is still valid. Symptoms are a working cluster with secrets not being deployed to the pods. It can be seen as kube-dns won't fully start.

liggitt · 2017-04-28T03:19:58Z

closing due to age and lack of activity

liggitt added kind/bug Categorizes issue or PR as related to a bug. priority/P1 component/auth labels Jun 10, 2015

liggitt self-assigned this Jun 10, 2015

liggitt mentioned this issue Jun 10, 2015

Deployments fail with User "system:anonymous" cannot get replicationcontrollers error #2918

Closed

liggitt assigned smarterclayton and unassigned liggitt Jun 10, 2015

smarterclayton mentioned this issue Jun 11, 2015

Containerization support for the kubelet #3112

Merged

openshift-bot closed this as completed in #3112 Jun 12, 2015

luxas mentioned this issue Mar 8, 2016

Experiment run kubelet as docker container kubernetes/kubernetes#4869

Closed

smarterclayton modified the milestones: 1.2.x, 1.3.0 Jul 12, 2016

liggitt modified the milestones: 1.3.0, 1.4.0 Sep 1, 2016

smarterclayton modified the milestones: 1.4.0, 1.5.0 Jan 31, 2017

smarterclayton modified the milestones: 1.5.0, 1.6.0 Mar 12, 2017

liggitt closed this as completed Apr 28, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker 1.7 cannot mount secrets #3072

Docker 1.7 cannot mount secrets #3072

liggitt commented Jun 10, 2015

liggitt commented Jun 10, 2015

smarterclayton commented Jun 10, 2015

liggitt commented Jun 10, 2015

gravis commented Jun 10, 2015 •

edited by stevekuznetsov

Loading

sspeiche commented Jun 10, 2015

smarterclayton commented Jun 10, 2015

gravis commented Jun 11, 2015

smarterclayton commented Jun 11, 2015

gravis commented Jun 11, 2015

smarterclayton commented Jun 11, 2015

gravis commented Jun 11, 2015

smarterclayton commented Jun 11, 2015

gravis commented Jun 11, 2015

gravis commented Jun 11, 2015

gravis commented Jun 12, 2015

smarterclayton commented Jun 12, 2015

gravis commented Jun 12, 2015

gravis commented Jun 12, 2015

gravis commented Jun 12, 2015

gravis commented Jun 12, 2015

gravis commented Jun 12, 2015

luxas commented Mar 4, 2016

csrwng commented Mar 4, 2016

luxas commented Mar 4, 2016

gravis commented Mar 4, 2016

luxas commented Mar 5, 2016

gravis commented Mar 5, 2016

marun commented Mar 5, 2016

luxas commented Mar 5, 2016

gravis commented Mar 5, 2016 via email

liggitt commented May 10, 2016

klaus commented Jul 4, 2016

liggitt commented Apr 28, 2017

Docker 1.7 cannot mount secrets #3072

Docker 1.7 cannot mount secrets #3072

Comments

liggitt commented Jun 10, 2015

liggitt commented Jun 10, 2015

smarterclayton commented Jun 10, 2015

liggitt commented Jun 10, 2015

gravis commented Jun 10, 2015 • edited by stevekuznetsov Loading

sspeiche commented Jun 10, 2015

smarterclayton commented Jun 10, 2015

gravis commented Jun 11, 2015

smarterclayton commented Jun 11, 2015

gravis commented Jun 11, 2015

smarterclayton commented Jun 11, 2015

gravis commented Jun 11, 2015

smarterclayton commented Jun 11, 2015

gravis commented Jun 11, 2015

gravis commented Jun 11, 2015

gravis commented Jun 12, 2015

smarterclayton commented Jun 12, 2015

gravis commented Jun 12, 2015

gravis commented Jun 12, 2015

gravis commented Jun 12, 2015

gravis commented Jun 12, 2015

gravis commented Jun 12, 2015

luxas commented Mar 4, 2016

csrwng commented Mar 4, 2016

luxas commented Mar 4, 2016

gravis commented Mar 4, 2016

luxas commented Mar 5, 2016

gravis commented Mar 5, 2016

marun commented Mar 5, 2016

luxas commented Mar 5, 2016

gravis commented Mar 5, 2016 via email

liggitt commented May 10, 2016

klaus commented Jul 4, 2016

liggitt commented Apr 28, 2017

gravis commented Jun 10, 2015 •

edited by stevekuznetsov

Loading