Docker version check (['/usr/local/bin/docker', 'version', '--format', \"'{{.Server.Version}}'\"]) failed: Error response from daemon: client is newer than server (client API version: 1.24, server API version: 1.23)

[belonesox]

• Bug Report

OS / ENVIRONMENT

Ansible Container, version 0.9.1
Linux, stas-custis-desktop, 4.9.20-nrj-desktop-1rosa-x86_64, #1 SMP PREEMPT Sat Apr 1 22:43:39 UTC 2017, x86_64
2.7.12 (default, Feb  5 2017, 20:03:30) 
[GCC 5.3.1 20160412 (ROSA)] /usr/bin/python
{
  "ContainersPaused": 0, 
  "Labels": null, 
  "CgroupDriver": "cgroupfs", 
  "ContainersRunning": 3, 
  "NGoroutines": 46, 
  "LoggingDriver": "json-file", 
  "OSType": "linux", 
  "HttpProxy": "", 
  "DriverStatus": [
    [
      "Root Dir", 
      "/var/cache/docker/aufs"
    ], 
    [
      "Backing Filesystem", 
      "extfs"
    ], 
    [
      "Dirs", 
      "44"
    ], 
    [
      "Dirperm1 Supported", 
      "true"
    ]
  ], 
  "OperatingSystem": "ROSA Desktop Fresh R9 EE 2016.1 Desktop", 
  "Containers": 8, 
  "HttpsProxy": "", 
  "BridgeNfIp6tables": true, 
  "MemTotal": 33143623680, 
  "Driver": "aufs", 
  "IndexServerAddress": "https://index.docker.io/v1/", 
  "ClusterStore": "", 
  "ExecutionDriver": "", 
  "SystemStatus": null, 
  "OomKillDisable": true, 
  "ClusterAdvertise": "", 
  "SystemTime": "2017-06-26T14:45:13.65604626+03:00", 
  "Name": "stas-custis-desktop", 
  "CPUSet": true, 
  "RegistryConfig": {
    "InsecureRegistryCIDRs": [
      "127.0.0.0/8"
    ], 
    "IndexConfigs": {
      "docker.io": {
        "Official": true, 
        "Name": "docker.io", 
        "Secure": true, 
        "Mirrors": null
      }
    }, 
    "Mirrors": null
  }, 
  "ContainersStopped": 5, 
  "NCPU": 4, 
  "NFd": 25, 
  "Architecture": "x86_64", 
  "KernelMemory": true, 
  "CpuCfsQuota": true, 
  "Debug": false, 
  "ID": "I7TU:45R4:ZBEH:A6NU:BQ6G:3GLI:JCZW:JOMS:YHOM:YZAS:HAYR:G73D", 
  "IPv4Forwarding": true, 
  "KernelVersion": "4.9.20-nrj-desktop-1rosa-x86_64", 
  "BridgeNfIptables": true, 
  "NoProxy": "", 
  "ServerVersion": "1.11.2", 
  "CpuCfsPeriod": true, 
  "ExperimentalBuild": false, 
  "MemoryLimit": true, 
  "SwapLimit": false, 
  "Plugins": {
    "Volume": [
      "local"
    ], 
    "Network": [
      "bridge", 
      "null", 
      "host"
    ], 
    "Authorization": null
  }, 
  "Images": 33, 
  "DockerRootDir": "/var/cache/docker", 
  "NEventsListener": 0, 
  "CPUShares": true
}
{
  "KernelVersion": "4.9.20-nrj-desktop-1rosa-x86_64", 
  "Os": "linux", 
  "ApiVersion": "1.23", 
  "Version": "1.11.2", 
  "GitCommit": "a34a1d5", 
  "Arch": "amd64", 
  "GoVersion": "go1.7.4"
}

SUMMARY

"ansible-container build" failed on any configuration (for example, reproducable on «ansible-container init ansible.django-template»)

STEPS TO REPRODUCE

mkdir ansible-container-django
cd ansible-container-django
ansible-container init ansible.django-template
ansible-container build

ansible-container-bug.txt

EXPECTED RESULTS

Working example from introduction tutorial.

ACTUAL RESULTS

Failed with "Docker version check (['/usr/local/bin/docker', 'version', '--format', "'{{.Server.Version}}'"]) failed: Error response from daemon: client is newer than server (client API version: 1.24, server API version: 1.23)"

Yes, see attached ansible-container-bug.txt

[chouseknecht]

@belonesox

Thanks for opening this issue, and sharing your feedback.

I'm curious, if you define DOCKER_API_VERSION=1.23 in your local environment, if that would fix it.
We init the Docker client with version='auto', so I'm surprised you're seeing this error.

[j00bar]

This error is coming out of Ansible's exec of docker, not from Ansible Container's use of docker-py with the auto-version client.

Starting with Docker version 1.13 (about 6 months ago), newer clients could talk to older servers without issue (see: moby/moby#27745). Your Docker daemon is older than that (version 1.11.2). Best to upgrade your Docker daemon.

@chouseknecht - Probably worth it to simply say we require Docker daemon 1.13 or later in the docs than to try to detect/set a version.

[belonesox]

if you define DOCKER_API_VERSION=1.23 in your local environment

Yes, I have tried this, no effect.

This error is coming out of Ansible's exec of docker, not from Ansible Container's use of docker-py with the auto-version client.

I looks to me, that it is problem of new docker client inside Conductor container
…/container/docker/engine.py with hardcoded version
DOCKER_VERSION = '17.04.0-ce'

And this new client may be could talk to old servers but I need to specify somehow DOCKER_API_VERSION=1.23 inside container, before first connection from ansible.

simply say we require Docker daemon 1.13 or later in the docs

Hmm. Even Fedora 25 still on Docker 1.12.

May be better to specify somehow version of docker inside container, or DOCKER_API_VERSION inside container? (same problems will repeat on every docker API change).

[j00bar]

May be better to specify somehow version of docker inside container, or DOCKER_API_VERSION inside container? (same problems will repeat on every docker API change).

The problems won't repeat on every Docker API change. As you can see from the Moby issue I linked to above, they've eliminated this problem going forward from Docker engine versions 1.13 or later.

Our three options here are:

1. ... to write-in autodetection of the server API version and set an environment variable in the Conductor to match.
2. ... to write-in the ability to specify additional environment variables in the Conductor execution, document that this variable needs to be specified if you're using an older engine, and hope that people read that versus filing bugs.
3. ... to not support engine versions before 1.13, document that, and hope that people read that versus filing bugs.

@chouseknecht I leave it up to you which you prefer.

[chouseknecht]

The conclusion to the Moby issues is the following:

"auto negotiation" works with a header that's sent by the daemon; since that header is added in docker 1.13, docker will only automatically fall back to version 1.12, which is the last version that did not set that header (see cli/command/cli.go#L169-L172)

If you want the client to talk to an even older version of the daemon, you can manually force it to use this API version through the DOCKER_API_VERSION environment variable, e.g.;

DOCKER_API_VERSION=1.21

So my thought is to pursue option 1, where we auto-detect the server API version, and set DOCKER_API_VERSION in the conductor to match.