ko should produce multi-platform images

[jonjohnsonjr]

Right now we assume amd64/linux and use distroless.

This is blocked on full manifest list support in the library, but ideally supplying a multi-platform base image would produce a multiplatform ko image.

(this is mostly a placeholder for reference until I flesh this out more)

[jonjohnsonjr]

Blocked by: google/go-containerregistry#119

Some thoughts also here under "Fat ko": google/go-containerregistry#254

[jonjohnsonjr]

The general idea for this was to drive the platform selection via the base image. Specifying a manifest list as the base image should produce a manifest list. If you want to build for a single platform, you could specify that platform's image.

I like this approach because we don't have to add any additional configuration or flags or anything, it's totally data-driven. One drawback is that it might be surprising that if you specify e.g. busybox, you'd end up building a binary for every single platform, but we could print a warning or something to make that clear.

We currently just pass through environment variables to go build for GOOS and GOARCH to set the platform/os, but we could inspect the base image's configuration instead for architecture and os.

This will be a bit of a pain to rework some of ko's build/publish interfaces, but it shouldn't be too difficult.

We'll need to do something similar to crane copy where we dispatch different logic based on what we get back from remote.Get, since we want to handle manifest lists slightly differently from manifests.

[zxDiscovery]

May I ask I question? Am I must export the env GOARCH=ppc64le if I want to build the ppc64le image on power when I use the current ko after the PR #77 merge? Thanks.
@jonjohnsonjr

[jonjohnsonjr]

@zxDiscovery correct, if you use a ppc64le specific image, you don''t need to export GOARCH anymore. Let me know if that doesn't work for you.

[zxDiscovery]

I use ko to build the ppc64le image on ppc64le platform, but the image can't work normally when I didn't export GOARCH=ppc64le.

root@dusk1:~# docker run -it 666ffcf0b33c
standard_init_linux.go:190: exec user process caused "exec format error"

After I export GOARCH=ppc64le, the images can work normally.
My environment information as follows:

root@dusk1:~# uname -o
GNU/Linux
root@dusk1:~# uname -a
Linux dusk1.fyre.ibm.com 4.15.0-45-generic #48-Ubuntu SMP Tue Jan 29 16:27:02 UTC 2019 ppc64le ppc64le ppc64le GNU/Linux

@jonjohnsonjr

[jonjohnsonjr]

@zxDiscovery If you're not overriding the base image, it will use gcr.io/distroless/base by default, which is an amd64/linux image, which would cause that not to work. You'd have to specify a ppc based image in .ko.yaml to take advantage of that change, e.g. ubuntu@sha256:0dcd8df85597396276e73ec96deb5c67c03c09b69297f7f3eb1a278649ec8e9b

To demonstrate, building an image containing ko -- note the architecture in the resulting config.

$ cat .ko.yaml 
defaultBaseImage: ubuntu@sha256:0dcd8df85597396276e73ec96deb5c67c03c09b69297f7f3eb1a278649ec8e9b

$ crane config $(ko publish -B .) | jq  .
2019/09/05 10:02:15 Using base index.docker.io/library/ubuntu@sha256:0dcd8df85597396276e73ec96deb5c67c03c09b69297f7f3eb1a278649ec8e9b for github.com/google/ko/cmd/ko
2019/09/05 10:02:17 Building github.com/google/ko/cmd/ko
2019/09/05 10:03:09 Publishing gcr.io/jonjohnson-test/ko/ko:latest
2019/09/05 10:03:10 existing blob: sha256:5059b42e78fada41824113ed43dc1dc528fe012caf84ec97fcd07be81bfe7e79
2019/09/05 10:03:10 existing blob: sha256:82bd3abde49712c4b07e1806574ae883d0b3e4fe3ad89bbc1c55b9edb8b001b5
2019/09/05 10:03:10 existing blob: sha256:2ba46c90b764a32fffe0fab2a144f38747f96ac61145e599da966777c7003e29
2019/09/05 10:03:10 existing blob: sha256:72164b581b02b1eb297b403bcc8fc1bfa245cb52e103a3a525a0835a58ff58e2
2019/09/05 10:03:10 existing blob: sha256:e9c62a75817ffb7dc6e831c48ed1799686dd3e3bd1a24f8db40adc2003642be1
2019/09/05 10:03:12 pushed blob: sha256:e353234264ad9f10ec480f9711c0010d6601b0ff8eaa1ed3c6f0f9a45d5825b1
2019/09/05 10:03:14 pushed blob: sha256:0c6a7186f4030d8a0e31a799d41eea3df2622120aa551ffa53b296620bc51ac0
2019/09/05 10:03:14 gcr.io/jonjohnson-test/ko/ko:latest: digest: sha256:cfc98498038ebcd5a5aefd6fc4de54361338c735c2e46f3caf1edeb9b8081942 size: 1235
2019/09/05 10:03:14 Published gcr.io/jonjohnson-test/ko/ko@sha256:cfc98498038ebcd5a5aefd6fc4de54361338c735c2e46f3caf1edeb9b8081942
{
  "architecture": "ppc64le",
  "author": "github.com/google/ko",
  "container": "bbf212bf9f0adaf7b4f32b1ce38526ae00197a38d82a8ea1ee9ac30420fc650f",
  "created": "2019-08-15T02:48:38.575406165Z",
  "docker_version": "18.06.1-ce",
  "history": [
    {
      "created": "2019-08-15T02:48:22.821369106Z",
      "created_by": "/bin/sh -c #(nop) ADD file:b0543ee4842b7be023d75e419a665cb6a64e3d23676ef87846964a8583761c51 in / "
    },
    {
      "created": "2019-08-15T02:48:29.552132934Z",
      "created_by": "/bin/sh -c [ -z \"$(apt-get indextargets)\" ]"
    },
    {
      "created": "2019-08-15T02:48:33.2412921Z",
      "created_by": "/bin/sh -c set -xe \t\t&& echo '#!/bin/sh' > /usr/sbin/policy-rc.d \t&& echo 'exit 101' >> /usr/sbin/policy-rc.d \t&& chmod +x /usr/sbin/policy-rc.d \t\t&& dpkg-divert --local --rename --add /sbin/initctl \t&& cp -a /usr/sbin/policy-rc.d /sbin/initctl \t&& sed -i 's/^exit.*/exit 0/' /sbin/initctl \t\t&& echo 'force-unsafe-io' > /etc/dpkg/dpkg.cfg.d/docker-apt-speedup \t\t&& echo 'DPkg::Post-Invoke { \"rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true\"; };' > /etc/apt/apt.conf.d/docker-clean \t&& echo 'APT::Update::Post-Invoke { \"rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true\"; };' >> /etc/apt/apt.conf.d/docker-clean \t&& echo 'Dir::Cache::pkgcache \"\"; Dir::Cache::srcpkgcache \"\";' >> /etc/apt/apt.conf.d/docker-clean \t\t&& echo 'Acquire::Languages \"none\";' > /etc/apt/apt.conf.d/docker-no-languages \t\t&& echo 'Acquire::GzipIndexes \"true\"; Acquire::CompressionTypes::Order:: \"gz\";' > /etc/apt/apt.conf.d/docker-gzip-indexes \t\t&& echo 'Apt::AutoRemove::SuggestsImportant \"false\";' > /etc/apt/apt.conf.d/docker-autoremove-suggests"
    },
    {
      "created": "2019-08-15T02:48:36.972700111Z",
      "created_by": "/bin/sh -c mkdir -p /run/systemd && echo 'docker' > /run/systemd/container"
    },
    {
      "created": "2019-08-15T02:48:38.575406165Z",
      "created_by": "/bin/sh -c #(nop)  CMD [\"/bin/bash\"]",
      "empty_layer": true
    },
    {
      "author": "ko",
      "created": "0001-01-01T00:00:00Z",
      "created_by": "ko publish github.com/google/ko/cmd/ko",
      "comment": "kodata contents, at $KO_DATA_PATH"
    },
    {
      "author": "ko",
      "created": "0001-01-01T00:00:00Z",
      "created_by": "ko publish github.com/google/ko/cmd/ko",
      "comment": "go build output, at /ko-app/ko"
    }
  ],
  "os": "linux",
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:3bd6a3fabaf390af12bc5d3cd3391e4d5f3106a3059cb3edaf0bec7745abea5a",
      "sha256:ff27e67a5be08af21d77d57b007248cd460bd76da4dd8e6e0f6d618e07c73ad1",
      "sha256:657b0bf9ab813720ed1aefd911e396834a63fe81418177896271f8c7b0de9344",
      "sha256:5495fb6e2095216885c46ab60b657b7f3a88550f98999972deb0a7199d757ea2",
      "sha256:4f76c7cc15474e09ddbebb4b39759df491cde9af6c85700612e7ec756492d571",
      "sha256:197515112b55d2ad87e2972ba1a74eb4d572be875c4de05ed718344e309a3c59"
    ]
  },
  "config": {
    "AttachStderr": false,
    "AttachStdin": false,
    "AttachStdout": false,
    "Cmd": [
      "/bin/bash"
    ],
    "Healthcheck": null,
    "Domainname": "",
    "Entrypoint": [
      "/ko-app/ko"
    ],
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "KO_DATA_PATH=/var/run/ko"
    ],
    "Hostname": "",
    "Image": "sha256:577428f5b7e262703426d2407e60dc2f74b33adbfcc5602fb7e43a474cad5d13",
    "Labels": null,
    "OnBuild": null,
    "OpenStdin": false,
    "StdinOnce": false,
    "Tty": false,
    "User": "",
    "Volumes": null,
    "WorkingDir": "",
    "ExposedPorts": null,
    "ArgsEscaped": true,
    "NetworkDisabled": false,
    "MacAddress": "",
    "StopSignal": "",
    "Shell": null
  },
  "container_config": {
    "AttachStderr": false,
    "AttachStdin": false,
    "AttachStdout": false,
    "Cmd": [
      "/bin/bash"
    ],
    "Healthcheck": null,
    "Domainname": "",
    "Entrypoint": [
      "/ko-app/ko"
    ],
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "KO_DATA_PATH=/var/run/ko"
    ],
    "Hostname": "",
    "Image": "sha256:577428f5b7e262703426d2407e60dc2f74b33adbfcc5602fb7e43a474cad5d13",
    "Labels": null,
    "OnBuild": null,
    "OpenStdin": false,
    "StdinOnce": false,
    "Tty": false,
    "User": "",
    "Volumes": null,
    "WorkingDir": "",
    "ExposedPorts": null,
    "ArgsEscaped": true,
    "NetworkDisabled": false,
    "MacAddress": "",
    "StopSignal": "",
    "Shell": null
  }
}

[jonjohnsonjr]

I wonder if we'll need something like https://github.com/karalabe/xgo to support CGO for this 🤔

[jonjohnsonjr]

#38 has landed but we should document this.

[jonjohnsonjr]

#211