Unable to query registry for image status deploying using Microk8s in Fedora.

[sriveros95]

Bug

Current Behavior

Followed instructions for installing on Linux (Fedora 30) using MicroK8s.
Following instructions for Demo Project running garden deploy on the project root I get:

Deploy 

✔ providers                 → Getting status... → Done
✔ backend                   → Syncing module sources (2 files)... → Done (took 0.3 sec)
✔ frontend                  → Syncing module sources (6 files)... → Done (took 0 sec)
 frontend                  → Getting build status for v-abd44a6dc6...
 backend                   → Getting build status for v-cef11cdcd0...

Failed building frontend. Here is the output:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Unable to query registry for image status:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


Failed building backend. Here is the output:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Unable to query registry for image status:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

2 deploy task(s) failed!

See error.log for detailed error message

And error.log:

[2020-03-22T22:05:24.176Z] Error: Unable to query registry for image status: 
    at /snapshot/project/garden-service/tmp/dist/build/src/plugins/kubernetes/container/build.js:0
    at Generator.next (<anonymous>)
    at fulfilled (/snapshot/project/garden-service/tmp/dist/build/src/plugins/kubernetes/container/build.js:0)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
Error Details:
command: 'docker manifest inspect localhost:32000/demo-project/frontend:v-abd44a6dc6'
output: ''


[2020-03-22T22:05:24.191Z] Error: Unable to query registry for image status: 
    at /snapshot/project/garden-service/tmp/dist/build/src/plugins/kubernetes/container/build.js:0
    at Generator.next (<anonymous>)
    at fulfilled (/snapshot/project/garden-service/tmp/dist/build/src/plugins/kubernetes/container/build.js:0)
Error Details:
command: 'docker manifest inspect localhost:32000/demo-project/backend:v-cef11cdcd0'
output: ''


[2020-03-22T22:05:24.813Z] Error: 2 deploy task(s) failed!
    at Object.<anonymous> (/snapshot/project/garden-service/tmp/dist/build/src/commands/base.js:0)
    at Generator.next (<anonymous>)
    at /snapshot/project/garden-service/tmp/dist/build/src/commands/base.js:0
    at new Promise (<anonymous>)
    at /snapshot/project/garden-service/tmp/dist/build/src/commands/base.js:0
    at Object.handleTaskResults (/snapshot/project/garden-service/tmp/dist/build/src/commands/base.js:0)
    at DeployCommand.<anonymous> (/snapshot/project/garden-service/tmp/dist/build/src/commands/deploy.js:0)
    at Generator.next (<anonymous>)
    at fulfilled (/snapshot/project/garden-service/tmp/dist/build/src/commands/deploy.js:0)

Expected behavior

Deploy

Reproducible example

https://docs.garden.io/example-projects/demo-project

Additional context

I use sudo for runing docker commands

Your environment

garden version

0.11.8

kubectl version

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.5", GitCommit:"20c265fef0741dd71a66480e35bd69f18351daea", GitTreeState:"clean", BuildDate:"2019-10-15T19:07:57Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}

microk8s.kubectl version

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:14:22Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:07:13Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}

sudo docker version

Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b7f0
 Built:             Wed Mar 11 01:26:25 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b7f0
  Built:            Wed Mar 11 01:25:01 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

[edvald]

Thanks for the report @sriveros95! We will investigate. We should also get those automated tests for microk8s up and running...

[edvald]

Can you elaborate on "I use sudo for runing docker commands"? I don't know for sure that it is, but I'm wondering if that's somehow related.

[sriveros95]

Thank you @edvald !

Well I've seen mostly people allow docker to run without sudoing, but Fedora and others dont quite recommend that .

Why we don't let non-root users run Docker in CentOS, Fedora, or RHEL

So everytime I run something with docker y do sudo docker

[edvald]

I see. That does make sense I suppose, but it's not something we've accounted for in our code. I'm not quite sure how we might accommodate this actually, since we don't really want to advise running Garden as root, and I'm not sure if (or even how) it would be advisable for Garden to run docker commands with sudo.

So for now we basically require the ability to run docker commands without sudo when using a local docker instance. Of course you could use in-cluster building (https://docs.garden.io/guides/in-cluster-building), which does away with the local docker requirement entirely. It's not something we've explicitly tested on microk8s but I know that does work on minikube and Docker Desktop so it's worth a shot, and we're happy to help work that out if you find any issues. And of course, if you have the option to run a remote cluster for your development that may be good, just depends on your setup and requirements.

In any case, interesting, and maybe this requirement is something we should explicitly document if this is a common restriction.

[sriveros95]

I will take a look at in-cluster building.

Maybe people that sudo docker should sudo garden, unless well its unfeasible as you say you "don't really want to advise running Garden as root," I tried but I havent looked at microk8s with sudo so when I sudo garden It doesnt try microk8s.kubectl but normal kubectl.

Will keep playing as it seems an amazing tool
Thanks again @edvald

[milanpro]

This seems to be a problem with the use of the experimental docker command docker manifest inspect.

While debugging garden i found that the following code in src/plugins/kubernetes/container/build.ts:

      // Non-zero exit code can both mean the manifest is not found, and any other unexpected error
      if (res.code !== 0 && !res.output.includes("no such manifest")) {
        throw new RuntimeError(`Unable to query registry for image status: ${res.output}`, {
          command: "docker " + args.join(" "),
          output: res.output,
        })
      }

did not work as expected, since res.output is an empty string.
The actual error could be found in res.stderr, which then led to me being able to deploy again with the following guard:
res.code !== 0 && (res.stderr === undefined || !res.stderr.includes("no such manifest"))

The docker manifest inspect might have changed in the latest docker release, since it is experimental.

[milanpro]

@edvald This is a major blocker in our project, since we are only able to deploy using our version. Were you able to look into this issue?

[edvald]

Yes! #1729, hoping to release this tomorrow

[edvald]

It turned out that docker manifest inspect doesn't work at all for the microk8s registry, for reasons as yet unknown.

[milanpro]

Okay wierd, since for us the same error occurs with our production cluster (so not microk8s). Should i open another issue then with the above mentioned solution?

[edvald]

Oh I see. Yeah, that looks to be a separate issue then. Could you file another issue with some info about your setup? I'll try and get that sorted asap.

[stale]

This issue has been automatically marked as stale because it hasn't had any activity in 60 days. It will be closed in 14 days if no further activity occurs (e.g. changing labels, comments, commits, etc.). Please feel free to tag a maintainer and ask them to remove the label if you think it doesn't apply. Thank you for submitting this issue and helping make Garden a better product!