Cannot build from local image with buildctl (OCI Worker)

[virgile-hogman]

Description

Cannot build from a local image with buildctl which always tries to resolve to docker.io. This applies to a multi-stage setup or simply from a local image in a CI setup. The base image should not be expected to be pushed to any repository (either docker.io or even private). It is only available locally (docker images).

This works with docker build and even DOCKER_BUILDKIT=1 using image names (but not SHA ID in the last case). See also moby/moby#39769.

This looks similar to the problems with local cache described here: moby/moby#39003.

Steps to reproduce the issue:

1. Demo project available here: https://github.com/virgile-hogman/docker-buildkit
2. Run ./buildctl-build.sh from bash
3. See logs, fails to resolve FROM docker.io/library/bk1:latest

Describe the results you received:

+ buildctl build --frontend dockerfile.v0 --progress=plain --local context=src2 --local dockerfile=src2 --output type=docker,name=bk2
+ docker load
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 66B done
#2 DONE 0.1s

#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.1s

#3 [internal] load metadata for docker.io/library/bk1:latest
#3 ERROR: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

#4 [1/2] FROM docker.io/library/bk1:latest
#4 resolve docker.io/library/bk1:latest
#4 resolve docker.io/library/bk1:latest 0.4s done
#4 ERROR: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
------
 > [internal] load metadata for docker.io/library/bk1:latest:
------
------
 > [1/2] FROM docker.io/library/bk1:latest:
------
error: failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to build LLB: failed to load cache key: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
open /var/lib/docker/tmp/docker-import-259291410/repositories: no such file or directory

Describe the results you expected:

This should work as for docker build when using image names (with or without DOCKER_BUILDKIT=1. Note with SHA-ID it does not work either in the last case as described in moby/moby#39769).

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client:
 Version:           18.09.6
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        481bc77
 Built:             Sat May  4 02:35:57 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Output of docker info:

Native install with Ubuntu 18.04.
More info does not seem relevant for now, can provide if really needed.

Additional environment details (AWS, VirtualBox, physical, etc.):

Connected to a private repository on a different host but this should not have effect here since it works with standard docker build.

[tonistiigi]

Buildkit with oci worker does not provide an image store. This has been discussed in the past but would mean we would need to add image management commands in the API. You can use containerd worker with the containerd image store(managed with ctr) or docker/buildx with docker image store.

[virgile-hogman]

@tonistiigi thanks for the answer, but coming just from the "docker world" i don't know exactly how to use containerd from buildkit in practice, especially the fetching part.

In this example i have a Dockerfile depending on a first one built only locally. Let's assume it is stored with containerd. But then it's still buildkit that needs to load it, so how would it find it in that local store? Do you need to name the local image in a particular way?

[tonistiigi]

But then it's still buildkit that needs to load it, so how would it find it in that local store? Do you need to name the local image in a particular way?

The first build would export it with a name --output type=image,name=myname and then another build can refer to it by that name.

[virgile-hogman]

I tried this but the second build still fails for the same reason. I don't still understand how the loaders work and how buildkit is going to find such image. I'm using the dockerized version of buildkit:

docker run -d --privileged -p 1234:1234 moby/buildkit:latest --addr tcp://0.0.0.0:1234
export BUILDKIT_HOST=tcp://0.0.0.0:1234

The first build produces an image that i can "somehow" see with buildctl du -v though i don't see the image name. There are some IDs which are not prefixed by sha apart from the base image corresponding to the alpine image. What am i missing? I don't know how to use ctr with this. The first image seems to be stored somewhere but i can't tell where.

[tonistiigi]

moby/buildkit:latest image is only configured for the oci worker. To run the containerd worker you need to have containerd running and start buildkitd with --oci-worker=false --containerd-worker=true

[virgile-hogman]

Is there any simple tutorial or documentation for setting up such a configuration? I have never used containerd and it sounds quite obscure to an end-user not familiar with terms like rund, containerd, workers or OCI, this all sounds like internal stuff for container developers which i'm not. The nice thing with Docker is its convenience and accessibility, it can be used out of the box from a simple apt installation. Here i don't even know where to look.

[rickmak]

I face the same problem, and tried to switch from oci to containerd worker but fails to boot the buildkit deamon.

$ docker run --name buildkit --privileged -p 1234:1234 moby/buildkit --addr tcp://0.0.0.0:1234 --oci-worker=false --containerd-worker=true
buildkitd: failed to connect client to "/run/containerd/containerd.sock" . make sure containerd is running: failed to dial "/run/containerd/containerd.sock": context deadline exceeded

The image tag for moby/buildkit is 8ce8d3df040f. I am running on Mac Docker desktop (2.1.03) with Engine version 19.03.2

Is that I must build the buildkitd myself if I want to test with the containerd worker? Thanks for your help.

[gfrancesco]

I'm experiencing the same issue. Looking into the moby/buildkit image (8ce8d3df040f) looks to me that there is no trace of containerd executables.

@tonistiigi since you suggested to start containerd before buildkitd --oci-worker=false --containerd-worker=true, can you please explain the necessary steps?
I went through the docs of docker/buildx, moby/buildkit and containerd/containerd with no luck.

I'm also freshly started with docker, but from the official documentation buildx is described as an improved version of build, but I'm a bit puzzled on why it can fail the very simple (and common I think) use case of creating an image starting from a base image already available locally under docker images.

[tonistiigi]

As I wrote in #1142 (comment) moby/buildkit is made for oci worker. It does not come with containerd binaries. If you want to run containerd you can refer to their getting started guide https://github.com/containerd/containerd/blob/master/docs/getting-started.md#getting-started-with-containerd . If you have containerd running you can start buildkitd binary with configuration pointing to your containerd daemon. As containerd runs in host and requires clients to share the filesystem, normally you would run buildkitd in host as well in that case. There's an old example of how to run them both in containers in #192 (comment) but that's probably not what you want.

I'm also freshly started with docker, but from the official documentation buildx is described as an improved version of build, but I'm a bit puzzled on why it can fail the very simple (and common I think) use case of creating an image starting from a base image already available locally under docker images.

buildx doesn't have anything to do with the question in here that is about buildctl. in buildx results are automatically get/put to docker images with docker driver, and with container driver --load loads the build results into docker. buildx never interacts with containerd daemon, nor does running a containerd daemon have any influence on the output of docker images.

[gfrancesco]

Ok, I think I'm getting even more confused, maybe I'll describe my use case more in detail to explain how I ended up here.
I'm running docker desktop on Mac OS.

• I want to build a base image of a linux OS, for which I need to use the new buildkit experimental RUN --security=insecure in my Dockerfile
• From the official docs, docker buildx build "extends the docker command with the full support of the features provided by Moby BuildKit builder" that looked to me like an easy entry point to use BuildKit
• To be able to use RUN --security=insecure, I need to create a builder instance with the insecure entitlement enabled for the buildkitd daemon by:
  docker buildx create --name insecure-builder --buildkitd-flags '--allow-insecure-entitlement security.insecure'
  which creates and enables a builder using the docker-container driver
• Now I can build my first image with: docker buildx build -t fvg/test:latest --load --allow security.insecure -f test.Dockerfile .
  here's the sample test.Dockerfile:

# syntax = docker/dockerfile:experimental

FROM alpine:latest
WORKDIR /root
RUN --security=insecure touch test1.file
CMD ["/bin/sh"]

• The image is correctly visible (and usable) under docker image ls. Now I want to build a second image that is based on the first: docker buildx build -t fvg/test2:latest --load --allow security.insecure -f test2.Dockerfile .
  here's the test2.Dockerfile:

# syntax = docker/dockerfile:experimental

FROM fvg/test:latest
WORKDIR /root
RUN --security=insecure touch test2.file
CMD ["/bin/sh"]

• The build fails looking to resolve docker.io/fvg/test:latest, failing to load the local image.

[+] Building 2.6s (6/8)                                                                                                                                             
 => [internal] load .dockerignore                                                                                                                              0.0s
 => => transferring context: 2B                                                                                                                                0.0s
 => [internal] load build definition from test2.Dockerfile                                                                                                     0.0s
 => => transferring dockerfile: 265B                                                                                                                           0.0s
 => resolve image config for docker.io/docker/dockerfile:experimental                                                                                          0.7s
 => CACHED docker-image://docker.io/docker/dockerfile:experimental@sha256:888f21826273409b5ef5ff9ceb90c64a8f8ec7760da30d1ffbe6c3e2d323a7bd                     0.0s
 => => resolve docker.io/docker/dockerfile:experimental@sha256:888f21826273409b5ef5ff9ceb90c64a8f8ec7760da30d1ffbe6c3e2d323a7bd                                0.0s
 => ERROR [internal] load metadata for docker.io/fvg/test:latest                                                                                               0.6s
 => ERROR [1/3] FROM docker.io/fvg/test:latest                                                                                                                 0.6s
 => => resolve docker.io/fvg/test:latest                                                                                                                       0.6s
------
 > [internal] load metadata for docker.io/fvg/test:latest:
------
------
 > [1/3] FROM docker.io/fvg/test:latest:
------
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to solve with frontend gateway.v0: rpc error: code = Unknown desc = failed to build LLB: failed to load cache key: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

If I try the same using the default builder under docker buildx ls with driver docker it works without error, but then I cannot build using --allow security.insecure, which is not allowed apparently.

[tonistiigi]

@gfrancesco If you want buildx to access docker images it needs to use docker driver. It is true that for the --security it was just added lately and can't be enabled atm. in docker. The PR for that is in moby/moby#39144 and you need to wait for that fix in order to use these 2 features together. If you want to use --security today with the container driver you can use a registry or combine your requests into a multi-stage Dockerfile for the same behavior.

[gfrancesco]

Thanks for helping, I'll keep an eye on the PR and look into use a registry in the meantime 🙂

[caleb15]

moby/moby#39144 has been merged in yet it looks like I got the same problem on circleci:

The dockerfile is a multi-stage dockerfile.

partial docker build script:

export DOCKER_BUILDKIT=1

###############################################

BACKEND_TAG=${CIRCLE_BRANCH}-backend-v2

echo "Pull old image for layer caching"
# Return true to ensure branches without images can still build
docker pull ${AWS_ACCOUNT_ID}.dkr.ecr.us-east-1.amazonaws.com/15five/fifteen5:${BACKEND_TAG} || true

echo "Build new python backend docker image"
docker build -f Dockerfile.production --target python_backend \
    --tag 15five/fifteen5:${BACKEND_TAG} \
    --build-arg dynatrace_subdomain=${DT_ENDPOINT} \
    --build-arg use_dynatrace=1 \
    .

      - setup_remote_docker:
          version: 19.03.12

Output of the above is:

Allocating a remote Docker Engine
Requesting version: 19.03.12

Assigned Docker Engine request id: 216311031
  provisioning: .
Remote Docker engine created. Using VM 'default-61b1abb0-61a6-4349-9ccb-5ba12832ffab'
Created container accessible with:
  DOCKER_CERT_PATH=/tmp/docker-certs181859510
  DOCKER_HOST=tcp://35.227.29.140:2376
  DOCKER_MACHINE_NAME=216311031
  DOCKER_TLS_VERIFY=1
  NO_PROXY=127.0.0.1,localhost,circleci-internal-outer-build-agent,35.227.29.140:2376

Server Engine Details:
  Version:          19.03.12
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       48a66213fe
  Built:            2020-06-22T15:44:20.000000000+00:00
  OS/Arch:          linux/amd64
  Experimental:     false

docker build script log:

Pull old image for layer caching
Error response from daemon: manifest for 482205614324.dkr.ecr.us-east-1.amazonaws.com/15five/fifteen5:add_node_image-backend-v2 not found: manifest unknown: Requested image not found
Build new python backend docker image
[+] Building 0.0s (0/2)                                                         
[+] Building 0.1s (1/2)                                                         
 => [internal] load build definition from Dockerfile.production            0.1s
 => => transferring dockerfile: 1.09kB                                     0.0s
 => [internal] load .dockerignore                                          0.1s
 => => transferring context: 1.61kB                                        0.0s
[+] Building 0.3s (2/3)                                                         
 => [internal] load build definition from Dockerfile.production            0.1s
 => => transferring dockerfile: 1.09kB                                     0.0s
 => [internal] load .dockerignore                                          0.1s
 => => transferring context: 1.61kB                                        0.0s
 => resolve image config for docker.io/docker/dockerfile:experimental      0.1s
[+] Building 0.4s (2/3)                                                         
 => [internal] load build definition from Dockerfile.production            0.1s
 => => transferring dockerfile: 1.09kB                                     0.0s
 => [internal] load .dockerignore                                          0.1s
 => => transferring context: 1.61kB                                        0.0s
 => resolve image config for docker.io/docker/dockerfile:experimental      0.3s
[+] Building 0.6s (2/3)                                                         
 => [internal] load build definition from Dockerfile.production            0.1s
 => => transferring dockerfile: 1.09kB                                     0.0s
 => [internal] load .dockerignore                                          0.1s
 => => transferring context: 1.61kB                                        0.0s
 => resolve image config for docker.io/docker/dockerfile:experimental      0.4s
[+] Building 0.7s (3/4)                                                         
 => => transferring context: 1.61kB                                        0.0s
 => resolve image config for docker.io/docker/dockerfile:experimental      0.4s
 => docker-image://docker.io/docker/dockerfile:experimental@sha256:de85b2  0.1s
 => => resolve docker.io/docker/dockerfile:experimental@sha256:de85b2f3a3  0.0s
 => => sha256:61261561661960014533790f5d6c42f5b88362db7e005f3 0B / 8.88MB  0.1s
 => => sha256:de85b2f3a3e8a2f7fe48e8e84a65f6fdd5cd5183afa 1.69kB / 1.69kB  0.0s
 => => sha256:8c69d118cfcd040a222bea7f7d57c6156faa938cb61b476 521B / 521B  0.0s
 => => sha256:08fb2b2ca3d58e19d791e73dea16126df37608115532e74 897B / 897B  0.0s
[+] Building 0.8s (3/4)                                                         
 => => transferring context: 1.61kB                                        0.0s
 => resolve image config for docker.io/docker/dockerfile:experimental      0.4s
 => docker-image://docker.io/docker/dockerfile:experimental@sha256:de85b2  0.3s
 => => resolve docker.io/docker/dockerfile:experimental@sha256:de85b2f3a3  0.0s
 => => sha256:61261561661960014533790f5d6c42f5b88362db7e0 5.01MB / 8.88MB  0.2s
 => => sha256:de85b2f3a3e8a2f7fe48e8e84a65f6fdd5cd5183afa 1.69kB / 1.69kB  0.0s
 => => sha256:8c69d118cfcd040a222bea7f7d57c6156faa938cb61b476 521B / 521B  0.0s
 => => sha256:08fb2b2ca3d58e19d791e73dea16126df37608115532e74 897B / 897B  0.0s
[+] Building 0.9s (3/4)                                                         
 => resolve image config for docker.io/docker/dockerfile:experimental      0.4s
 => docker-image://docker.io/docker/dockerfile:experimental@sha256:de85b2  0.4s
 => => resolve docker.io/docker/dockerfile:experimental@sha256:de85b2f3a3  0.0s
 => => sha256:61261561661960014533790f5d6c42f5b88362db7e0 8.88MB / 8.88MB  0.2s
 => => sha256:de85b2f3a3e8a2f7fe48e8e84a65f6fdd5cd5183afa 1.69kB / 1.69kB  0.0s
 => => sha256:8c69d118cfcd040a222bea7f7d57c6156faa938cb61b476 521B / 521B  0.0s
 => => sha256:08fb2b2ca3d58e19d791e73dea16126df37608115532e74 897B / 897B  0.0s
 => => extracting sha256:61261561661960014533790f5d6c42f5b88362db7e005f35  0.1s
[+] Building 1.0s (3/4)                                                         
 => resolve image config for docker.io/docker/dockerfile:experimental      0.4s
 => docker-image://docker.io/docker/dockerfile:experimental@sha256:de85b2  0.5s
 => => resolve docker.io/docker/dockerfile:experimental@sha256:de85b2f3a3  0.0s
 => => sha256:61261561661960014533790f5d6c42f5b88362db7e0 8.88MB / 8.88MB  0.2s
 => => sha256:de85b2f3a3e8a2f7fe48e8e84a65f6fdd5cd5183afa 1.69kB / 1.69kB  0.0s
 => => sha256:8c69d118cfcd040a222bea7f7d57c6156faa938cb61b476 521B / 521B  0.0s
 => => sha256:08fb2b2ca3d58e19d791e73dea16126df37608115532e74 897B / 897B  0.0s
 => => extracting sha256:61261561661960014533790f5d6c42f5b88362db7e005f35  0.2s
[+] Building 1.2s (4/4)                                                         
 => resolve image config for docker.io/docker/dockerfile:experimental      0.4s
 => docker-image://docker.io/docker/dockerfile:experimental@sha256:de85b2  0.6s
 => => resolve docker.io/docker/dockerfile:experimental@sha256:de85b2f3a3  0.0s
 => => sha256:61261561661960014533790f5d6c42f5b88362db7e0 8.88MB / 8.88MB  0.2s
 => => sha256:de85b2f3a3e8a2f7fe48e8e84a65f6fdd5cd5183afa 1.69kB / 1.69kB  0.0s
 => => sha256:8c69d118cfcd040a222bea7f7d57c6156faa938cb61b476 521B / 521B  0.0s
 => => sha256:08fb2b2ca3d58e19d791e73dea16126df37608115532e74 897B / 897B  0.0s
 => => extracting sha256:61261561661960014533790f5d6c42f5b88362db7e005f35  0.3s
[+] Building 1.3s (4/4)                                                         
 => resolve image config for docker.io/docker/dockerfile:experimental      0.4s
 => docker-image://docker.io/docker/dockerfile:experimental@sha256:de85b2  0.6s
 => => resolve docker.io/docker/dockerfile:experimental@sha256:de85b2f3a3  0.0s
 => => sha256:61261561661960014533790f5d6c42f5b88362db7e0 8.88MB / 8.88MB  0.2s
 => => sha256:de85b2f3a3e8a2f7fe48e8e84a65f6fdd5cd5183afa 1.69kB / 1.69kB  0.0s
 => => sha256:8c69d118cfcd040a222bea7f7d57c6156faa938cb61b476 521B / 521B  0.0s
 => => sha256:08fb2b2ca3d58e19d791e73dea16126df37608115532e74 897B / 897B  0.0s
 => => extracting sha256:61261561661960014533790f5d6c42f5b88362db7e005f35  0.3s
[+] Building 1.5s (4/4)                                                         
 => resolve image config for docker.io/docker/dockerfile:experimental      0.4s
 => docker-image://docker.io/docker/dockerfile:experimental@sha256:de85b2  0.6s
 => => resolve docker.io/docker/dockerfile:experimental@sha256:de85b2f3a3  0.0s
 => => sha256:61261561661960014533790f5d6c42f5b88362db7e0 8.88MB / 8.88MB  0.2s
 => => sha256:de85b2f3a3e8a2f7fe48e8e84a65f6fdd5cd5183afa 1.69kB / 1.69kB  0.0s
 => => sha256:8c69d118cfcd040a222bea7f7d57c6156faa938cb61b476 521B / 521B  0.0s
 => => sha256:08fb2b2ca3d58e19d791e73dea16126df37608115532e74 897B / 897B  0.0s
 => => extracting sha256:61261561661960014533790f5d6c42f5b88362db7e005f35  0.3s
[+] Building 1.6s (4/5)                                                         
 => docker-image://docker.io/docker/dockerfile:experimental@sha256:de85b2  0.6s
 => => resolve docker.io/docker/dockerfile:experimental@sha256:de85b2f3a3  0.0s
 => => sha256:61261561661960014533790f5d6c42f5b88362db7e0 8.88MB / 8.88MB  0.2s
 => => sha256:de85b2f3a3e8a2f7fe48e8e84a65f6fdd5cd5183afa 1.69kB / 1.69kB  0.0s
 => => sha256:8c69d118cfcd040a222bea7f7d57c6156faa938cb61b476 521B / 521B  0.0s
 => => sha256:08fb2b2ca3d58e19d791e73dea16126df37608115532e74 897B / 897B  0.0s
 => => extracting sha256:61261561661960014533790f5d6c42f5b88362db7e005f35  0.3s
 => [internal] load metadata for docker.io/15five/fifteen5:base            0.1s
[+] Building 1.8s (7/10)                                                        
 => => sha256:8c69d118cfcd040a222bea7f7d57c6156faa938cb61b476 521B / 521B  0.0s
 => => sha256:08fb2b2ca3d58e19d791e73dea16126df37608115532e74 897B / 897B  0.0s
 => => extracting sha256:61261561661960014533790f5d6c42f5b88362db7e005f35  0.3s
 => ERROR [internal] load metadata for docker.io/15five/fifteen5:base      0.1s
 => ERROR [python_backend 1/4] FROM docker.io/15five/fifteen5:base         0.1s
 => => resolve docker.io/15five/fifteen5:base                              0.1s
 => CANCELED [internal] load build context                                 0.1s
 => => transferring context: 144.35kB                                      0.1s
[+] Building 1.9s (7/10)                                                        
 => [internal] load build definition from Dockerfile.production            0.1s
 => => transferring dockerfile: 1.09kB                                     0.0s
 => [internal] load .dockerignore                                          0.1s
 => => transferring context: 1.61kB                                        0.0s
 => resolve image config for docker.io/docker/dockerfile:experimental      0.4s
 => docker-image://docker.io/docker/dockerfile:experimental@sha256:de85b2  0.6s
 => => resolve docker.io/docker/dockerfile:experimental@sha256:de85b2f3a3  0.0s
 => => sha256:61261561661960014533790f5d6c42f5b88362db7e0 8.88MB / 8.88MB  0.2s
 => => sha256:de85b2f3a3e8a2f7fe48e8e84a65f6fdd5cd5183afa 1.69kB / 1.69kB  0.0s
 => => sha256:8c69d118cfcd040a222bea7f7d57c6156faa938cb61b476 521B / 521B  0.0s
 => => sha256:08fb2b2ca3d58e19d791e73dea16126df37608115532e74 897B / 897B  0.0s
 => => extracting sha256:61261561661960014533790f5d6c42f5b88362db7e005f35  0.3s
 => ERROR [internal] load metadata for docker.io/15five/fifteen5:base      0.1s
 => ERROR [python_backend 1/4] FROM docker.io/15five/fifteen5:base         0.1s
 => => resolve docker.io/15five/fifteen5:base                              0.1s
 => CANCELED [internal] load build context                                 0.1s
 => => transferring context: 144.35kB                                      0.1s
------
 > [internal] load metadata for docker.io/15five/fifteen5:base:
------
------
 > [python_backend 1/4] FROM docker.io/15five/fifteen5:base:
------
failed to solve with frontend dockerfile.v0: failed to solve with frontend gateway.v0: rpc error: code = Unknown desc = failed to build LLB: failed to load cache key: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Return code from docker build 1

Note that base image on circleci worked fine with buildkit enabled, so buildkit can work with circleci - it's just fails in the situation above for some reason 🤔

[vnedvyga]

But then it's still buildkit that needs to load it, so how would it find it in that local store? Do you need to name the local image in a particular way?

The first build would export it with a name --output type=image,name=myname and then another build can refer to it by that name.

@tonistiigi I'm trying to build image from local image with containerd image store and it works fine if the local image was builded by buildkit. But it fails if the local image was not builded by builtkit (for example: pulled and retagged or imported). It fails with:
error: failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
or with:
error: failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to read dockerfile: failed to mount XXX: snapshot XXX does not exist: not found
if I specify correct containerd namespace.

I suppose this happens because those images are not in local build cache. Is there a way to make it work somehow?
Thank you!

[jamshid]

I'm sorry I don't understand all this but just curious if this is the same root issue as I'm seeing in:
docker/for-mac#5873
where building FROM a local image, which itself was built FROM a debian image, fails with "pull access denied" only when using buildkit on an M1 Mac.