allow client to talk to an older server #27745

vieux · 2016-10-25T16:44:43Z

removed user-agent middleware (to prevent Client and server don't have the same version error)
removed check for client is newer than server error
added API-Version header on all API calls
Ping() always calls /_ping, not /v1.xx/_ping
started added a few if where flags/features are only supported in 1.25

vieux · 2016-10-26T00:57:02Z

$ docker version
Client:
 Version:      1.13.0-dev
 API version:  1.24 (downgraded from 1.25)
 Go version:   go1.7.3
 Git commit:   ec3a34b
 Built:        Wed Oct 26 00:54:51 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.2
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   bb80604
 Built:        Tue Oct 11 17:00:50 2016
 OS/Arch:      linux/amd64
 Experimental: false


$ docker create --stop-timeout 5 test
Server only supports API version stop timeout, "1.24" support requires at least 1.25

duglin · 2016-10-26T01:35:08Z

@vieux I don't understand that error message - perhaps a rewording would help?
Also, it seems a bit odd that while I see the Client and Server servers are different (1.13 vs 1.12), the API versions are the same - you'd think that if those matched then no error would be generated. No?

vieux · 2016-10-26T18:08:21Z

@duglin thanks, I'll reword the error message. and thanks good catch, the client server should either say 1.25 or 1.24 (downgraded from 1.25)

duglin · 2016-10-27T14:08:01Z

@vieux in this mode of operation am I correct that this is using the data itself (ie. incoming fields vs what the server knows about) to determine if it can support the incoming request rather than the version string itself?

bfirsh · 2016-10-27T20:38:46Z

@vieux As an extended response to your comment on #27830, instead of using a header, perhaps an additional API endpoint could be used that reports the latest API version.

Something like:

GET / HTTP/1.1

{
  "LatestVersion": "1.25",
  "SupportedVersions": [
    "1.25",
    "1.24",
    "1.23",
    ...
  ]
}

(could also be something like /_api-versions if we didn't want to block out the root for this purpose)

In a potential future where a version is required, this could be used as the entrypoint for clients to figure out what versions can be used. Seems more solid and explicit than using a header on another API endpoint.

vdemeester · 2016-10-27T23:58:01Z

cli/command/cli.go

+	return nil
+}
+
+// OriginalClientVersion returns the origin client version.


returns the original client version ? 😝

vdemeester · 2016-10-28T00:02:09Z

client/client.go

@@ -183,9 +183,9 @@ func (cli *Client) Close() error {

 // getAPIPath returns the versioned request path to call the api.
 // It appends the query parameters to the path if they are not empty.
-func (cli *Client) getAPIPath(p string, query url.Values) string {
+func (cli *Client) getAPIPath(p string, query url.Values, ignoreVersion bool) string {


ignoreVersion is only for the _ping endpoint, right ?

I wonder if it better to have a ignoreVersion bool or handle the _ping endpoint by itself in the method 😛

I think that's a good idea (have _ping not call this)

I think we still need to go through the rest of the function:

u := &url.URL{ Path: apiPath, } if len(query) > 0 { u.RawQuery = query.Encode() } return u.String()

I don't like this solution either, but from client.Ping to getAPIPath it goes through rougly 4 functions. not sure how to handle that better. I could replace the Context part by a bool but not sure it would be enough ?

and by that I mean:

Client.Ping -> Client.get
Client.get -> Client.sendRequest
Client.sendRequest -> Client.sendClientRequest
Client.sendClientRequest -> Client.newRequest
Client.newRequest -> Client.getAPIPath

The best I could see is to add a bool to newRequest and getAPIPath and do:

Client.Ping -> Client.sendClientRequestIgnoreVersion
Client.sendClientRequestIgnoreVersion -> Client.newRequest(ignoreVersion=true)
Client.newRequest -> Client.getAPIPath(ignoreVersion=true)

wdyt @dnephin @vdemeester

@vieux this client/request.go has a few issues I think. I've been meaning to refactor these methods for a while now. I think the problem we're seeing here is a symptom of those issues.

I refactored the send methods here: master...dnephin:refactor-client

That branch includes a commit which uses the refactored functions to call Ping() without the api version in the path. It removes the need to have the ignoreVersion parameter entirely.

If this works for you, would you like to cherry pick the first commit into this branch? Otherwise I can drop the change to ping.go and open it as a PR. Let me know which you prefer.

Opened #27916

vieux · 2016-10-28T00:42:19Z

@bfirsh the current idea here is to piggyback on the work on regarding experimental. that's how it was designed: call /_ping and get the experimental status from a header.

I'm not against adding this new endpoint ,but it seems a bit overkill to me.

WDYT @dnephin ?

dnephin · 2016-10-28T16:54:20Z

I think adding it to /_ping makes sense. We already have an api endpoint for returning version information at /version. We could use /version instead of /_ping (and have it respond without a version in the path), but I don't think we need a new one.

dnephin · 2016-10-28T16:58:50Z

cli/command/cli.go

+	keyFile               string
+	client                client.APIClient
+	hasExperimental       bool
+	originalClientVersion string


What do you think about another word for original ?

Maybe defaultVersion, maxSupportedVersion, or preferredVersion ?

dnephin · 2016-10-28T17:03:34Z

cli/command/commands/commands.go

@@ -72,13 +72,19 @@ func AddCommands(cmd *cobra.Command, dockerCli *command.DockerCli) {
 		hide(system.NewInspectCommand(dockerCli)),
 	)

+	if versions.GreaterThanOrEqualTo(dockerCli.Client().ClientVersion(), "1.25") {
+		cmd.AddCommand(system.NewVersionCommand(dockerCli))
+	}


hmm, why is the version command gated by server version?

Sorry, I tought "system." was for the new data management commands, I'll remove that.

dnephin · 2016-10-28T17:06:03Z

client/client.go

@@ -183,9 +183,9 @@ func (cli *Client) Close() error {

 // getAPIPath returns the versioned request path to call the api.
 // It appends the query parameters to the path if they are not empty.
-func (cli *Client) getAPIPath(p string, query url.Values) string {
+func (cli *Client) getAPIPath(p string, query url.Values, ignoreVersion bool) string {


I think that's a good idea (have _ping not call this)

dnephin · 2016-10-28T17:10:37Z

client/container_create.go

+
+	if config != nil && config.StopTimeout != nil && versions.LessThan(cli.version, "1.25") {
+		return response, fmt.Errorf("Server only supports API version %s, %q support requires at least %s", cli.version, "stop timeout", "1.25")
+	}


Could you make this a function, so that all the error messages have a consistent format and wording.

func NewVersionError(version, required, feature string) error { ... }

If you make it a method on Client you could even drop the version argument.

For the wording how about:

"stop timeout" requires API version 1.25, but the Docker server is version 1.24.

dnephin · 2016-10-28T17:13:11Z

client/ping.go

 	}

-	return true, nil
+	return true, apiVersion, nil


Instead of three return values, maybe a struct like we use for other responses?

dnephin · 2016-10-28T17:14:38Z

cmd/docker/docker.go

@@ -50,6 +48,11 @@ func newDockerCommand(dockerCli *command.DockerCli) *cobra.Command {
 	flags.StringVar(&opts.ConfigDir, "config", cliconfig.ConfigDir(), "Location of client config files")
 	opts.Common.InstallFlags(flags)

+	// flags must be the top-level command flags, not cmd.Flags()
+	opts.Common.SetDefaultOptions(flags)


I think this is wrong. The flags have not been parsed yet so the values that it sets are going to be wrong I believe.

dnephin · 2016-10-28T17:15:40Z

cli/command/cli.go

+}
+
+// Ping pings the server and retrieve the version to use and if experimental is enabled.
+func (cli *DockerCli) Ping() (err error) {


Ping() is maybe not the best name for this, how about making this an unexpected func that is called from Initialize() ? initializeVersion() ?

dnephin · 2016-10-28T17:16:42Z

cmd/docker/docker.go

@@ -50,6 +48,11 @@ func newDockerCommand(dockerCli *command.DockerCli) *cobra.Command {
 	flags.StringVar(&opts.ConfigDir, "config", cliconfig.ConfigDir(), "Location of client config files")
 	opts.Common.InstallFlags(flags)

+	// flags must be the top-level command flags, not cmd.Flags()
+	opts.Common.SetDefaultOptions(flags)
+	dockerCli.Initialize(opts)


same here, this is too early to call Initialize() because the flags haven't been parsed yet.

dnephin · 2016-10-28T17:22:14Z

cmd/docker/docker.go

+	// flags must be the top-level command flags, not cmd.Flags()
+	opts.Common.SetDefaultOptions(flags)
+	dockerCli.Initialize(opts)
+	dockerCli.Ping()


I think this presents a serious UX problem.

If I run docker or docker --help to find out how to connect to an alternative host I'm going to get a connection error instead of the help text.

We can't make any API calls before flag parsing is complete, because we won't know which host we're connecting to until after flag parsing. But flag parsing has to happen after we setup the child commands. Which means the experimental support also has this issue. I don't think we can ship 1.13 this way.

The order of events needs to be this:

setup child commands and flags

parse flags

initialize client

execute command

That means that flags and child commands can not depend on the server version.

I think we should do this:

do not try and ping the server

go back to the original plan with the version header, where we make a request first then fall back on error.

for experimental use Hidden: isExperimental on the experimental commands. Add an environment variable DOCKER_EXPERIMENTAL that enables showing the commands by setting isExperimental=true`.

If I run docker or docker --help to find out how to connect to an alternative host I'm going to get a connection error instead of the help text.

This is not correct, the error is not checked at this point. (At least that's how I planned it, maybe I did a mistake in the code)

In my option it works ok:

docker --help that is able to talk to an experimental server will display experimental commands and commands supported by the server

docker --help that is able to talk to a stable server will not display experimental commands and commands supported by the server

docker --help that is not able to connect to anything will not display experimental commands and all the commands known to the cli

ping @mlaventure since it's not related specifically to the API versioning, but also experimental.

Ok, I guess my example was wrong, but I think the rest is still correct.

The server that it checks will be either the default (or the one set by DOCKER_HOST maybe, I'm not sure if that's read at this time). The --host or -H flag isn't parsed yet, so the behaviour is wrong if you have to use --host.

This could lead to some really strange behaviour where there is both a local daemon and some cloud edition. The command line is based on the local daemon, but then after flag parsing it points at the remote engine, which results in the incorrect behaviour when using any unsupported flags, instead of the helpful error message from the client.

Hum, yes correct. It does not work with --host (it works with DOCKER_HOST)

oh ok I get what you mean @dnephin .

So how about we remove all the checks from the cli pkg, so we always display commands/flags but we error out in the client package only instead ?

That might fix the issue. I think we could still hide the commands with this strategy: #27745 (comment)

@dnephin WDYT about #28008

Pull request moby#27745 added support for the client to talk to older versions of the daemon. Various flags were added to docker 1.13 that are not compatible with older daemons. This PR adds annotations to those flags, so that they are automatically hidden if the daemon is older than docker 1.13 (API 1.25). Not all new flags affect the API (some are client-side only). The following PR's added new flags to docker 1.13 that affect the API; - moby#23430 added `--cpu-rt-period`and `--cpu-rt-runtime` - moby#27800 / moby#25317 added `--group` / `--group-add` / `--group-rm` - moby#27702 added `--network` to `docker build` - moby#25962 added `--attachable` to `docker network create` - moby#27998 added `--compose-file` to `docker stack deploy` - moby#22566 added `--stop-timeout` to `docker run` and `docker create` - moby#26061 added `--init` to `docker run` and `docker create` - moby#26941 added `--init-path` to `docker run` and `docker create` - moby#27958 added `--cpus` on `docker run` / `docker create` - moby#27567 added `--dns`, `--dns-opt`, and `--dns-search` to `docker service create` - moby#27596 added `--force` to `docker service update` - moby#27857 added `--hostname` to `docker service create` - moby#28031 added `--hosts`, `--host-add` / `--host-rm` to `docker service create` and `docker service update` - moby#28076 added `--tty` on `docker service create` / `docker service update` - moby#26421 added `--update-max-failure-ratio`, `--update-monitor` and `--rollback` on `docker service update` - moby#27369 added `--health-cmd`, `--health-interval`, `--health-retries`, `--health-timeout` and `--no-healthcheck` options to `docker service create` and `docker service update` Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

deviantony · 2017-01-23T01:16:17Z

Maybe I'm missing something but it looks like I cannot use a Docker 1.13 client to send an info request to a Docker 1.10 server.

Reproduction steps:

# Start a Docker 1.10 in Docker container and expose the Docker API port locally
$ docker run --privileged --name docker110 -d -p 7110:2375 docker:1.10-dind
e937dbff2126d02d0b33c337b50bf3069261a131c04eaf77dc219964a0efdf0d

# Query the Docker 1.10 engine
$ docker -H localhost:7110 info
Error response from daemon: client is newer than server (client API version: 1.24, server API version: 1.22)

# Docker version output
$ docker version
Client:
 Version:      1.13.0
 API version:  1.25
 Go version:   go1.7.3
 Git commit:   49bf474
 Built:        Tue Jan 17 09:50:17 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.13.0
 API version:  1.25 (minimum version 1.12)
 Go version:   go1.7.3
 Git commit:   49bf474
 Built:        Tue Jan 17 09:50:17 2017
 OS/Arch:      linux/amd64
 Experimental: false

thaJeztah · 2017-01-23T12:06:40Z

@deviantony correct, "auto negotiation" works with a header that's sent by the daemon; since that header is added in docker 1.13, docker will only automatically fall back to version 1.12, which is the last version that did not set that header (see cli/command/cli.go#L169-L172)

If you want the client to talk to an even older version of the daemon, you can manually force it to use this API version through the DOCKER_API_VERSION environment variable, e.g.;

DOCKER_API_VERSION=1.21 ./docker version

gcbartlett · 2017-01-23T17:20:54Z

@thaJeztah, regarding interoperability with servers older than 1.12 (the blog does not note that restriction by the way), even without the header, it appears that the client does know the server's API version (e.g. the error message displays it...server API version: 1.21), so couldn't it automatically perform the equivalent of setting DOCKER_API_VERSION to automatically support earlier server versions?

Or could docker-machine env be updated to also return the appropriate DOCKER_API_VERSION version?

thaJeztah · 2017-01-23T17:46:24Z

the blog does not note that restriction by the way

Thanks for noticing that; I'll see if I can find the right person to update the blog and clarify that

it appears that the client does know the server's API version

The client does not know the daemon version, but only prints the error; if you look at the error;

Error response from daemon: ......

So the client tried to connect to the daemon through API 1.24, but the daemon did not support anything above 1.22. While this information could be used for negotiation, the current design is to use a dedicated API endpoint for that.

Also note that DOCKER_API_VERSION was added for "debugging" purposes. Most things should work, but there may be corner-cases where the client does not take differences in the old API into account. Versions that are automatically negotiated with the 1.13 client do take differences into account (e.g. in some cases values need to be specified slightly different, or options are not supported by the older API version, and therefore silently ignored).

Or could docker-machine env be updated to also return the appropriate DOCKER_API_VERSION version?

This was implemented for a short while, but led to incorrect issues where the original machine was a 1.12 daemon, but upgraded to 1.13, but the environment variable still forced the client to use the old API.

gcbartlett · 2017-01-23T17:57:17Z

Thanks @thaJeztah. So is the recommendation for using older than 1.12 servers to continue to use dvm?

thaJeztah · 2017-01-23T18:00:47Z

I think the recommendation would be to update those servers 😇 , but yes, using dvm (i.e, the correct client version to match the server) is for now the best solution.

allow client to talk to an older server

Pull request moby/moby#27745 added support for the client to talk to older versions of the daemon. Various flags were added to docker 1.13 that are not compatible with older daemons. This PR adds annotations to those flags, so that they are automatically hidden if the daemon is older than docker 1.13 (API 1.25). Not all new flags affect the API (some are client-side only). The following PR's added new flags to docker 1.13 that affect the API; - moby/moby#23430 added `--cpu-rt-period`and `--cpu-rt-runtime` - moby/moby#27800 / moby/moby#25317 added `--group` / `--group-add` / `--group-rm` - moby/moby#27702 added `--network` to `docker build` - moby/moby#25962 added `--attachable` to `docker network create` - moby/moby#27998 added `--compose-file` to `docker stack deploy` - moby/moby#22566 added `--stop-timeout` to `docker run` and `docker create` - moby/moby#26061 added `--init` to `docker run` and `docker create` - moby/moby#26941 added `--init-path` to `docker run` and `docker create` - moby/moby#27958 added `--cpus` on `docker run` / `docker create` - moby/moby#27567 added `--dns`, `--dns-opt`, and `--dns-search` to `docker service create` - moby/moby#27596 added `--force` to `docker service update` - moby/moby#27857 added `--hostname` to `docker service create` - moby/moby#28031 added `--hosts`, `--host-add` / `--host-rm` to `docker service create` and `docker service update` - moby/moby#28076 added `--tty` on `docker service create` / `docker service update` - moby/moby#26421 added `--update-max-failure-ratio`, `--update-monitor` and `--rollback` on `docker service update` - moby/moby#27369 added `--health-cmd`, `--health-interval`, `--health-retries`, `--health-timeout` and `--no-healthcheck` options to `docker service create` and `docker service update` Signed-off-by: Sebastiaan van Stijn <github@gone.nl> Upstream-commit: 5d2722f83db9e301c6dcbe1c562c2051a52905db Component: cli

GordonTheTurtle added the status/0-triage label Oct 25, 2016

vieux force-pushed the cli_backward_compose_api branch from 242b025 to 13f5cf3 Compare October 25, 2016 17:18

vieux added area/api status/1-design-review impact/api and removed status/0-triage labels Oct 25, 2016

vieux force-pushed the cli_backward_compose_api branch 2 times, most recently from a86f4b7 to ec3a34b Compare October 26, 2016 00:54

vieux force-pushed the cli_backward_compose_api branch 5 times, most recently from 6ac2bb8 to 3395d09 Compare October 27, 2016 00:43

vdemeester mentioned this pull request Oct 27, 2016

Proposal: Docker Client Version Negotiation #21930

Closed

icecrime added this to the 1.13.0 milestone Oct 27, 2016

vieux mentioned this pull request Oct 27, 2016

Deprecate API calls without a version #27830

Closed

vieux force-pushed the cli_backward_compose_api branch from 3395d09 to bb78253 Compare October 27, 2016 19:06

vieux force-pushed the cli_backward_compose_api branch from bb78253 to 24f8419 Compare October 27, 2016 22:31

vdemeester reviewed Oct 28, 2016

View reviewed changes

vieux force-pushed the cli_backward_compose_api branch from 24f8419 to 56d08df Compare October 28, 2016 01:04

dnephin suggested changes Oct 28, 2016

View reviewed changes

vieux force-pushed the cli_backward_compose_api branch 2 times, most recently from c698021 to d65d16e Compare October 28, 2016 21:10

thaJeztah added the impact/changelog label Nov 9, 2016

vieux force-pushed the cli_backward_compose_api branch 3 times, most recently from 591eabe to e98e4a7 Compare November 9, 2016 00:59

thaJeztah added status/4-merge and removed status/3-docs-review labels Nov 9, 2016

vieux merged commit d3c780b into moby:master Nov 9, 2016

vieux deleted the cli_backward_compose_api branch November 9, 2016 02:27

dnephin mentioned this pull request Nov 12, 2016

Non-existent subcommand exit code behavior changed in 1.13+ (1.13.0-rc1) #28329

Closed

This was referenced Nov 16, 2016

Handling of --rm broken when talking to older daemons #28439

Closed

Bugfix: set cli.manualOverride when env var not empty #28647

Merged

thaJeztah mentioned this pull request Jan 16, 2017

Add version annotation to various flags added in 1.13 #30186

Merged

sflxn mentioned this pull request Jan 24, 2017

Update docker personality to accept API versions > 1.23 vmware/vic#3415

Closed

lox mentioned this pull request Jan 28, 2017

[WIP] Clean up images buildkite/docker-buildkite-agent#38

Closed

2 tasks

dnephin pushed a commit to dnephin/docker that referenced this pull request Apr 17, 2017

Merge pull request moby#27745 from vieux/cli_backward_compose_api

7fe223b

allow client to talk to an older server

j00bar mentioned this pull request Jun 26, 2017

Docker version check (['/usr/local/bin/docker', 'version', '--format', \"'{{.Server.Version}}'\"]) failed: Error response from daemon: client is newer than server (client API version: 1.24, server API version: 1.23) ansible/ansible-container#622

Open

This was referenced Oct 9, 2017

Docker no longer rejects API requests with version greater than daemon maximum #35145

Closed

Docker capability testing is broken with Docker >= 1.13.0 aws/amazon-ecs-agent#1010

Closed

thaJeztah mentioned this pull request Oct 10, 2017

Return 400 error if API client is too new #35150

Merged

olivierdalang mentioned this pull request Aug 27, 2019

Update olivierdalang/docker-certbot#6

Merged

thaJeztah mentioned this pull request Sep 30, 2020

client: use canonical names for HTTP Headers #41508

Merged

allow client to talk to an older server #27745

allow client to talk to an older server #27745

Conversation

vieux commented Oct 25, 2016

vieux commented Oct 26, 2016 • edited Loading

duglin commented Oct 26, 2016

vieux commented Oct 26, 2016

duglin commented Oct 27, 2016

bfirsh commented Oct 27, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vieux commented Oct 28, 2016

dnephin commented Oct 28, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dnephin Oct 28, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

deviantony commented Jan 23, 2017

thaJeztah commented Jan 23, 2017

gcbartlett commented Jan 23, 2017 • edited Loading

thaJeztah commented Jan 23, 2017

gcbartlett commented Jan 23, 2017

thaJeztah commented Jan 23, 2017

vieux commented Oct 26, 2016 •

edited

Loading

dnephin Oct 28, 2016 •

edited

Loading

gcbartlett commented Jan 23, 2017 •

edited

Loading