Clear Architecture field in platform constraint for arm architectures #34021

nishanttotla · 2017-07-07T22:15:31Z

This is a potential fix for moby/swarmkit#2294

This is a temporary requirement.

Also related: opencontainers/image-spec#661

aaronlehmann · 2017-07-07T22:22:14Z

I wonder if we could append all known ARM variants to platforms in this case.

nishanttotla · 2017-07-07T22:34:35Z

Ping @stevvooe

nishanttotla · 2017-07-07T22:35:21Z

@aaronlehmann based on the discussion in the SwarmKit issue, I don't think we should be doing that. This will eventually be fixed for images.

aaronlehmann · 2017-07-07T22:38:34Z

Wouldn't this PR cause ARM images to be scheduled on x86 and other platforms? I suppose it's better than not working at all, but I'm wondering if we can do better.

nishanttotla · 2017-07-07T22:46:34Z

@aaronlehmann yes, that will happen. I think until the manifest is fixed, it is better to err on the side of putting lesser constraints.

stevvooe · 2017-07-07T23:02:06Z

This fix needs to happen server-side:

If the platform comes from a manifest list or oci index, it is fine.
If it has to be resolved from the image config, it should undergo similar logic.

Wouldn't this PR cause ARM images to be scheduled on x86 and other platforms? I suppose it's better than not working at all, but I'm wondering if we can do better.

Yes, but right now, we have no way of resolving which version of arm an image belongs to.

estesp · 2017-07-11T21:11:58Z

It is strange that I'm just learning this, but a weird mismatch that we actually query the Linux UTS subsystem for the daemon's architecture setting, but all image handling is done with GOOS and GOARCH. It was confusing to me that an image was trying to match on something that Go doesn't even acknowledge (armv7l) which to me is more the root of the problem than the well understood issue with ARM variants.

Seems we need to at some point resolve matching to all use GOOS/GOARCH (with the added variant processing that is already available in OCI and also available when building a manifest list) so that we don't have some resolver of a bunch of random distro uname choices back to something "understood".

thaJeztah · 2017-07-12T06:50:02Z

I agree with all of the above; we should fix the current situation, because it's a regression from 17.03;

use "OS", if it's a known value
use architecture, if it's a known value
if not, ignore, and use the same behaviour as docker 17.03

Wouldn't this PR cause ARM images to be scheduled on x86 and other platforms? I suppose it's better than not working at all, but I'm wondering if we can do better.

This is what 17.03 did, so not a regression. Mixed architecture Swarm clusters are probably rare, and if someone currently uses a mixed architecture (or operating system) cluster, they probably use constraints already to influence scheduling.

In the reporter's case moby/swarmkit#2294, things worked fine in the old situation; the service used an image specific to the architecture, and could just be deployed.

justincormack · 2017-07-12T07:01:25Z

@estesp what Go chooses to do does not correspond to the real world requirements of users necessarily either. This will likely diverge more in future too, so I don't think it is a great idea just to base things on Go.

…

On 11 Jul 2017 22:12, "Phil Estes" ***@***.***> wrote: It is strange that I'm just learning this, but a weird mismatch that we actually query the Linux UTS subsystem for the daemon's architecture setting, but all image handling is done with GOOS and GOARCH. It was confusing to me that an image was trying to match on something that Go doesn't even acknowledge (armv7l) which to me is more the root of the problem than the well understood issue with ARM variants. Seems we need to at some point resolve matching to all use GOOS/GOARCH (with the added variant processing that is already available in OCI and also available when building a manifest list) so that we don't have some resolver of a bunch of random distro uname choices back to something "understood". — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#34021 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAdcPLn03_0IL1Rmxl3l_yUhwJ8XVDL6ks5sM-UngaJpZM4ORfra> .

estesp · 2017-07-12T14:57:01Z

@justincormack true, the connection to Go is not to take their view of OS/architecture as gospel, but realize that as Go programs, we have a few options for determining our current runtime architecture and OS; either ask the Go runtime, or try to divine it from Linux syscalls/environment data.

FWIW, IMHO, YMMV :) .. Go is a much more stable source of this information than Linux as distros have shown to be quite open to variance and finding their own way when it comes to embedded architectures and uname -a responses :)

To make this more concrete, there is an issue (I think it was over in the OCI spec repo) that lists about 60 different variants (strings) of arch-os tuples on ARM platforms running Linux. Go provides us 2: "arm" and "arm64". That leaves us having to find another method (using the proposed variant field in manifest list/OCI indexes) when building images for v6, v7, v8 specific ARM families, but at least it means we aren't trying to harmonize 60 different strings down to "arm" or "arm64" + a variant list of 3 items. Of course we will have to find a way to query the runtime "variant" when arch = {arm,arm64}, which IMO is the only missing piece, but that only applies to manifest list/index-based image resolving, not single manifest images which only provide us the two pieces of info: OS and arch.

If I haven't rambled on too long (don't answer that), another way to look at this bug is that you will never have a stable comparison between runtime.GOARCH and uname -a; and images are created using runtime.GOARCH but they are being compared to output from uname -a. It was already shown to be a problem in the old days because x86_64 (a Linux-ism) was being hardcoded in images, but runtime.GOARCH in Go is called "amd64" for the same platform.

stevvooe · 2017-07-13T22:08:45Z

@justincormack UTS doesn't exactly help here. Right now, it returns armv7l in certain cases, which I cannot find any information about, anywhere. UTS (or gnu triples) seem identify a toolchain, which can be very different for identical architectures.

Go's approach is at least pragmatic: they try to define a real machine target.

If you can identify a standard approach that actually works, then, by all means, suggest one. Laying this on the footsteps of Go's toolchain is fairly dismissive.

alexellis · 2017-07-27T07:39:48Z

I also tested 17.07.0-ce-rc1 this morning on armv7l and after upgrading from 17.05 (fully working with stacks and services) - I get 0/1 replicas after deploying a stack.

In addition there are no error messages on service inspect/ls/ps.

andrewhsu · 2017-07-27T21:19:12Z

@alexellis fyi the code changes in this PR was not included in the 17.07.0-ce-rc1 release because it has not been merged yet. Curious to know if you ran your test with a custom compile of the 17.07.0-ce-rc1 codebase with this PR patch applied?

alexellis · 2017-07-27T21:25:47Z

I think Stefan and myself tried the same install script from Eli @ Docker Inc via the #arm Slack channel-

curl -fsSL
https://raw.githubusercontent.com/seemethere/docker-install/update_for_170701rc1/install.sh
| sh

 Version:      17.07.0-ce-rc1
 API version:  1.31
 Go version:   go1.8.3
 Git commit:   8c4be39
 Built:        Wed Jul 26 21:00:23 2017
 OS/Arch:      linux/arm

All services from docker-compose.armhf.yml are pinned at 0/1 replicas.
https://github.com/alexellis/faas

DEBU[0116] no suitable node available for task module=node node.id=a82troq1z5gmjwnpdlw76k5v0 task.id=dcxqoaohw872432weilc9voc8

However with 17.05 all come out at 1/1 replicas.

 Version:      17.05.0-ce
 API version:  1.29
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:30:54 2017
 OS/Arch:      linux/arm

thaJeztah · 2017-07-27T22:47:13Z

@alexellis what does

docker service inspect -f '{{json .Spec.TaskTemplate.Placement}}' <service-id>

show for those services?

alexellis · 2017-08-11T18:19:57Z

{"Platforms":[{"Architecture":"arm","OS":"linux"}]}

alexellis · 2017-08-11T18:21:16Z

@thaJeztah ... it's still broken in the following version
@andrewhsu

Client:
 Version:      17.07.0-ce-rc2
 API version:  1.31
 Go version:   go1.8.3
 Git commit:   36ce605
 Built:        Mon Aug  7 23:53:53 2017
 OS/Arch:      linux/arm

Server:
 Version:      17.07.0-ce-rc2
 API version:  1.31 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   36ce605
 Built:        Mon Aug  7 23:47:08 2017
 OS/Arch:      linux/arm
 Experimental: false

justincormack · 2017-08-11T18:23:32Z

@alexellis nothing has changed since #34021 (comment) "fyi the code changes in this PR was not included in the 17.07.0-ce-rc1 release because it has not been merged yet. Curious to know if you ran your test with a custom compile of the 17.07.0-ce-rc1 codebase with this PR patch applied?"

Have you tested it? Does it fix the issue?

alexellis · 2017-08-11T18:24:23Z

I interpreted the comment to mean "tell me what build you're using?" and I replied.

Do you have a "custom compile" of Docker CE available for testing?

alexellis · 2017-08-20T18:09:05Z

I can confirm the fix works.. I deployed a full application (FaaS).

I tested it along with rallying some of the community to help with the ARM testing and opened additional issues in the docker-arm repo with the tag community-supports-docker.

alexellis/docker-arm#14 (comment)

@nishanttotla @estesp @StefanScherer

vattybear · 2017-08-20T19:59:26Z

Concur with @alexellis above - have similarly tested the fix today - commented with details on alexellis/docker-arm#17

andrewhsu · 2017-08-31T23:14:24Z

Is this PR a suitable stepping-stone step in the interim?

stevvooe · 2017-09-01T01:13:18Z

api/server/router/distribution/distribution_routes.go

@@ -123,12 +123,19 @@ func (s *distributionRouter) getDistributionInfo(ctx context.Context, w http.Res
 		if err == nil {
 			err := json.Unmarshal(configJSON, &platform)
 			if err == nil && (platform.OS != "" || platform.Architecture != "") {
+				if platform.Architecture == "arm" || platform.Architecture == "Arm" {


I was looking at this approach and doing it on the server side will create future problems for this endpoint. Clearing the architecture should be done on the client side, when setting up the constraint. That way, we can fix this in the future and still be able to support this version of docker.

stevvooe · 2017-09-01T18:25:05Z

LGTM

@nishanttotla Thanks!

cpuguy83 · 2017-09-01T19:55:16Z

client/service_create.go

@@ -98,6 +98,11 @@ func imageDigestAndPlatforms(ctx context.Context, cli DistributionAPIClient, ima
 	if len(distributionInspect.Platforms) > 0 {
 		platforms = make([]swarm.Platform, 0, len(distributionInspect.Platforms))
 		for _, p := range distributionInspect.Platforms {
+			// clear architecture field for arm. This is a temporary fix.
+			arch := p.Architecture
+			if arch == "arm" || arch == "Arm" {


if stringsToLower(arch) == "arm"

Also should this be HasPrefix?

In the images I've seen, it's only been "arm". We could use HasPrefix to be extra careful but I don't think it's necessary.

cpuguy83 · 2017-09-01T19:55:27Z

client/service_create.go

@@ -98,6 +98,11 @@ func imageDigestAndPlatforms(ctx context.Context, cli DistributionAPIClient, ima
 	if len(distributionInspect.Platforms) > 0 {
 		platforms = make([]swarm.Platform, 0, len(distributionInspect.Platforms))
 		for _, p := range distributionInspect.Platforms {
+			// clear architecture field for arm. This is a temporary fix.


Can you add more detail here as to why it's being cleared?

Signed-off-by: Nishant Totla <nishanttotla@gmail.com>

nishanttotla · 2017-09-01T20:09:30Z

@cpuguy83 addressed your comments.

cpuguy83 · 2017-09-01T20:23:50Z

Lint errors are being dealt with here: #34706

nishanttotla · 2017-09-01T23:14:18Z

This commit will also need vendoring into the docker/cli.

cpuguy83

LGTM

andrewhsu · 2017-09-05T22:50:06Z

Is this one ready to go or are there any other types of verification that is needed before merging?

vdemeester

LGTM 🐮

GordonTheTurtle added the status/0-triage label Jul 7, 2017

nishanttotla force-pushed the dont-set-architecture-constraint branch from 3cbd838 to 444fdca Compare July 7, 2017 22:16

nishanttotla force-pushed the dont-set-architecture-constraint branch from 444fdca to 5fa6df3 Compare July 10, 2017 16:55

GordonTheTurtle assigned LK4D4 Jul 22, 2017

nishanttotla mentioned this pull request Jul 26, 2017

17.06 won't deploy stack anymore. no suitable node (unsupported platform on 3 nodes) on armhf docker cluster moby/swarmkit#2294

Closed

vielmetti mentioned this pull request Jul 28, 2017

WOA W30: Docker, Go, Guix, naming issues WorksOnArm/worksonarm-news#42

Closed

alexellis mentioned this pull request Aug 20, 2017

Swarm is Broken on Docker CE after 17.05 - no services are scheduled. alexellis/docker-arm#14

Open

stevvooe reviewed Sep 1, 2017

View reviewed changes

cpuguy83 requested changes Sep 1, 2017

View reviewed changes

Clear Architecture field in platform constraint for arm architectures

772af60

Signed-off-by: Nishant Totla <nishanttotla@gmail.com>

nishanttotla force-pushed the dont-set-architecture-constraint branch from f172404 to 772af60 Compare September 1, 2017 20:08

cpuguy83 added status/2-code-review and removed status/0-triage labels Sep 1, 2017

vieux added rebuild/janky labels Sep 1, 2017

GordonTheTurtle removed rebuild/janky labels Sep 1, 2017

cpuguy83 approved these changes Sep 1, 2017

View reviewed changes

cpuguy83 added status/4-merge and removed status/2-code-review labels Sep 1, 2017

yongtang added rebuild/janky labels Sep 2, 2017

GordonTheTurtle removed rebuild/janky labels Sep 2, 2017

yongtang added the rebuild/windowsRS1 label Sep 2, 2017

GordonTheTurtle removed the rebuild/windowsRS1 label Sep 2, 2017

stevvooe approved these changes Sep 5, 2017

View reviewed changes

vdemeester approved these changes Sep 7, 2017

View reviewed changes

vdemeester merged commit ea220e7 into moby:master Sep 7, 2017

andrewhsu mentioned this pull request Sep 8, 2017

vndr docker/docker to ea220e7 to bring in fix for arm docker/cli#516

Merged

nishanttotla mentioned this pull request Sep 20, 2017

manifest list (multiarch) picks wrong arch on ARMv7 #34875

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clear Architecture field in platform constraint for arm architectures #34021

Clear Architecture field in platform constraint for arm architectures #34021

nishanttotla commented Jul 7, 2017

aaronlehmann commented Jul 7, 2017

nishanttotla commented Jul 7, 2017

nishanttotla commented Jul 7, 2017

aaronlehmann commented Jul 7, 2017

nishanttotla commented Jul 7, 2017

stevvooe commented Jul 7, 2017

estesp commented Jul 11, 2017

thaJeztah commented Jul 12, 2017

justincormack commented Jul 12, 2017 via email

estesp commented Jul 12, 2017

stevvooe commented Jul 13, 2017

alexellis commented Jul 27, 2017

andrewhsu commented Jul 27, 2017

alexellis commented Jul 27, 2017 •

edited

Loading

thaJeztah commented Jul 27, 2017

alexellis commented Aug 11, 2017

alexellis commented Aug 11, 2017

justincormack commented Aug 11, 2017

alexellis commented Aug 11, 2017 •

edited

Loading

alexellis commented Aug 20, 2017 •

edited

Loading

vattybear commented Aug 20, 2017

andrewhsu commented Aug 31, 2017

stevvooe Sep 1, 2017

stevvooe commented Sep 1, 2017

cpuguy83 Sep 1, 2017

cpuguy83 Sep 1, 2017

nishanttotla Sep 1, 2017

cpuguy83 Sep 1, 2017

nishanttotla commented Sep 1, 2017

cpuguy83 commented Sep 1, 2017

nishanttotla commented Sep 1, 2017

cpuguy83 left a comment

andrewhsu commented Sep 5, 2017

vdemeester left a comment

Clear Architecture field in platform constraint for arm architectures #34021

Clear Architecture field in platform constraint for arm architectures #34021

Conversation

nishanttotla commented Jul 7, 2017

aaronlehmann commented Jul 7, 2017

nishanttotla commented Jul 7, 2017

nishanttotla commented Jul 7, 2017

aaronlehmann commented Jul 7, 2017

nishanttotla commented Jul 7, 2017

stevvooe commented Jul 7, 2017

estesp commented Jul 11, 2017

thaJeztah commented Jul 12, 2017

justincormack commented Jul 12, 2017 via email

estesp commented Jul 12, 2017

stevvooe commented Jul 13, 2017

alexellis commented Jul 27, 2017

andrewhsu commented Jul 27, 2017

alexellis commented Jul 27, 2017 • edited Loading

thaJeztah commented Jul 27, 2017

alexellis commented Aug 11, 2017

alexellis commented Aug 11, 2017

justincormack commented Aug 11, 2017

alexellis commented Aug 11, 2017 • edited Loading

alexellis commented Aug 20, 2017 • edited Loading

vattybear commented Aug 20, 2017

andrewhsu commented Aug 31, 2017

stevvooe Sep 1, 2017

Choose a reason for hiding this comment

stevvooe commented Sep 1, 2017

cpuguy83 Sep 1, 2017

Choose a reason for hiding this comment

cpuguy83 Sep 1, 2017

Choose a reason for hiding this comment

nishanttotla Sep 1, 2017

Choose a reason for hiding this comment

cpuguy83 Sep 1, 2017

Choose a reason for hiding this comment

nishanttotla commented Sep 1, 2017

cpuguy83 commented Sep 1, 2017

nishanttotla commented Sep 1, 2017

cpuguy83 left a comment

Choose a reason for hiding this comment

andrewhsu commented Sep 5, 2017

vdemeester left a comment

Choose a reason for hiding this comment

alexellis commented Jul 27, 2017 •

edited

Loading

alexellis commented Aug 11, 2017 •

edited

Loading

alexellis commented Aug 20, 2017 •

edited

Loading