404 error on creating workspace container.

[monaka]

[Short description of problem here]

Reproduction Steps:

1. Bootup che-host.
2. Open browser
3. Create workspace.
4. Dialog pop ups "404 error".

Expected behavior:

Creats workspace with no error.

Observed behavior:

[Retrieving the stack's image and launching it] frame show logs.

[DOCKER] Status: Image is up to date for codenvy/ubuntu_jdk8:latest 
[DOCKER] Step 1 : FROM codenvy/ubuntu_jdk8
[DOCKER] ---> ace032f7e9e3
[DOCKER] Successfully built ace032f7e9e3
[ERROR] Error response from docker API, status: 404, message: no such file or directory

The result of journalctl -fu docker is like this.

Jul 03 02:41:47 docker-node1 dockerd[5350]: time="2016-07-03T02:41:47.717006029Z" level=debug msg="Calling POST /containers/create"
Jul 03 02:41:47 docker-node1 dockerd[5350]: time="2016-07-03T02:41:47.717036629Z" level=debug msg="POST /containers/create?name=workspacefcp9i0yvotzomvyq_machinedu438ln9r7qr8jqy_che_default"
Jul 03 02:41:47 docker-node1 dockerd[5350]: time="2016-07-03T02:41:47.717696936Z" level=debug msg="form data: {\"AttachStderr\":false,\"AttachStdin\":false,\"AttachStdout\":false,\"CpuShares\":0,\"Env\":[\"CHE_API_ENDPOINT=http://che-host:8080/wsmaster/api\",\"CHE_PROJECTS_ROOT=/projects\",\"CHE_LOCAL_CONF_DIR=/mnt/che/conf\",\"CHE_WORKSPACE_ID=workspacefcp9i0yvotzomvyq\",\"USER_TOKEN=dummy_token\"],\"ExposedPorts\":{\"4401/tcp\":{},\"4403/tcp\":{},\"4411/tcp\":{}},\"HostConfig\":{\"Binds\":[\"/che/lib/linux_amd64/terminal:/mnt/che/terminal:ro,Z\",\"/che/lib/ws-agent.zip:/mnt/che/ws-agent.zip:ro,Z\",\"/che/workspaces/wksp-rjf0:/projects:Z\"],\"CpuShares\":0,\"ExtraHosts\":[\"che-host:172.17.42.1\"],\"Memory\":1.048576e+09,\"MemorySwap\":-1,\"MemorySwappiness\":-1,\"PortBindings\":{},\"Privileged\":false,\"PublishAllPorts\":true,\"ReadonlyRootfs\":false},\"Hostname\":\"\",\"Image\":\"eclipse-che/workspacefcp9i0yvotzomvyq_machinedu438ln9r7qr8jqy_che_default\",\"Labels\":{},\"NetworkDisabled\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Tty\":false,\"User\":\"\",\"Volumes\":{},\"WorkingDir\":\"\"}"
Jul 03 02:41:47 docker-node1 dockerd[5350]: time="2016-07-03T02:41:47.726759830Z" level=error msg="Handler for POST /containers/create returned error: no such file or directory"

Che version: 4.4.0
OS and version: CoreOS 1010.6.0 (MoreOS)
Docker version: 1.10.3 (client and server have some version.)
Che install: server (zip) (based Docker image)

Additional information:

I tried to regenerate same situation by hand.

core@docker-node1 ~/eclipse-che-test $ docker exec -it che-host /bin/bash
user@82ca2b95dcfd:/$ ls /che/lib/linux_amd64/terminal/                                                
che-websocket-terminal  index.html  term.js
</lib/linux_amd64/terminal:/mnt/che/terminal:ro,Z eclipse-che/workspacefcp9i0yvotzomvyq_machinedu438ln9r7qr8jqy_che_default                                  
Error response from daemon: no such file or directory
user@82ca2b95dcfd:/$ docker create --name tmp -v /che/lib/linux_amd64/terminal:/mnt/che/terminal:ro eclipse-che/workspacefcp9i0yvotzomvyq_machinedu438ln9r7qr8>
2bd6db0f3360d5483f8b47bed711c036673ab60ec98bf21b799f960a5a691b72

So It seems be caused Z option. Possible be caused by Docker daemon?

• Problem started happening recently, didn't happen in an older version of Che: Unknown
• Problem can be reliably reproduced, doesn't happen randomly: Yes, 100% reproduction to me.

[monaka]

SELinux is off on my CoreOS instance (default setting).

So the result of ls on my host is with ? like this.

CoreOS $ ls -dZ /home/core/che/workspace
? /home/core/che/workspace

Labels are enabled in the container mounted -v /home/core/che/workspace:/projects

che-host $ ls -dZ /projects
system_u:object_r:user_home_dir_t:s0 /projects

But it can't remount /projects on the workspace container

$ docker run -v /che/workspaces:/projects:Z eclipse-che/workspacefcp9i0yvotzomvyq_machinedu438ln9r7qr8jqy_che_default
docker: Error response from daemon: no such file or directory.

I got same error when I started the ws-agent container without mounting -v /home/core/che/workspace:/projects

[ghost]

@monaka it does look like a Docker related issue to me. Let's take out Che out of equation for a while. Can you run the following?

docker run -ti -v $(pwd):/projects:Z codenvy/ubuntu_jdk8 bash

FYI, I have started a fresh CoreOS node on Digital Ocean, with Docker version 1.10.3, build 8acee1b and my workspace has successfully started.

core@coreos-2gb-nyc3-01 ~ $ docker -v
Docker version 1.10.3, build 8acee1b
core@coreos-2gb-nyc3-01 ~ $ uname -a
Linux coreos-2gb-nyc3-01 4.5.7-coreos #2 SMP Tue Jun 28 09:06:34 UTC 2016 x86_64 Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz GenuineIntel GNU/Linux

[monaka]

it does look like a Docker related issue to me.

And/or possibly related on the host environment. e.g. SELinux is on or off.
(I'm not sure now. So I posted this issue.)

docker run -ti -v $(pwd):/projects:Z codenvy/ubuntu_jdk8 bash

It works on the CoreOS host.
I'll test it in containers later as I can't connect my Che environment now.

FYI, I have started a fresh CoreOS node on Digital Ocean,

Thank you for fine information.
I'm running my node on Azure. There can be had some differences in cloud-init.yml by providers.
But I see at least one CoreOS node runs Che. I'll keep inspecting.

[TylerJewell]

@monaka - any update?

[monaka]

I came back. I found a curious message when run the container without ':Z'. This issue may be caused my mis-configurations.

Jul 12 05:05:04 docker-node1 dockerd[901]: time="2016-07-12T05:05:04.330468856Z" level=warning msg="Auto-creating non-existent volume host path /che/workspaces/wksp-69bo, this is deprecated and will be removed soon"

[monaka]

I got it.
I wrongly assumed about paths for volumes. Each paths should be host's. Not container's.

I run the che-host container with '-v /home/core/che/workspaces:/che/workspaces' badly.
It should be '-v /home/core/che/workspaces:/home/core/che/workspaces'.

Now my Che instance is booting developer machine. So I close this issue. Thanks for your support.