COPY rpc error: code = Unknown desc = not found: not found

[pwaller]

I have a very simple Dockerfile. The dir/subdir path contains lots of binaries, about 500MiB over 7,000 files.

FROM ubuntu:18.04

COPY dir/subdir /opt/subdir

It fails with the error message:

 => ERROR [2/2] COPY dir/subdir /opt/subdir
------
 > [2/2] COPY dir/subdir /opt/subdir:
------

rpc error: code = Unknown desc =  not found: not found

The failure was repeatable.

I have copied the directory in order to reproduce it from another location, and it does not reproduce there.

Subsequent to doing this, the docker build in the original location now works. I observed it re-upload the build context. So I suspect that something went wrong with the build context cache? It's possible that I interrupted the build context cache upload at some point, so I suspect there could be an issue with the build context cache being uploaded.

Sorry for the nebulous issue without a clear reproducer but I wanted to document that I had hit this issue. Please feel free to close if there are no usable leads here.

I suspect that clearing out the build cache would have helped.

I'm running docker for Mac.

Expand docker version and docker info

Client: Docker Engine - Community
 Version:           18.09.1
 API version:       1.39
 Go version:        go1.10.6
 Git commit:        4c52b90
 Built:             Wed Jan  9 19:33:12 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.1
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       4c52b90
  Built:            Wed Jan  9 19:41:49 2019
  OS/Arch:          linux/amd64
  Experimental:     true

Containers: 76
 Running: 0
 Paused: 0
 Stopped: 76
Images: 430
Server Version: 18.09.1
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
runc version: 96ec2177ae841256168fcf76954f7177af9446eb
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.125-linuxkit
Operating System: Docker for Mac
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 5.817GiB
Name: linuxkit-025000000001
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 40
 Goroutines: 112
 System Time: 2019-01-16T14:23:06.9713013Z
 EventsListeners: 2
HTTP Proxy: gateway.docker.internal:3128
HTTPS Proxy: gateway.docker.internal:3129
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

[pwaller]

Hi @tonistiigi - I saw the status/needs-more-info label (fair enough!), but no comment. Is there something in particular I should collect if this happens again?

[greghall76]

I'm seeing this as well. Building on Ubuntu with Docker version 18.09.1, build 4c52b90 and buildkit.

# syntax=docker/dockerfile-upstream:experimental
FROM        maven:3.5-jdk-8-alpine as build
 ... 
COPY settings.xml /root/settings.xml

[tonistiigi]

Is there something in particular I should collect if this happens again?

We need some way to reproduce it. Eg. maybe on load if this is flaky. Or if this requires specific state maybe the state directory can be sent. If this proves impossible we could probably prepare a debug build that would gather more information when this happens.

[mcesar]

I suspect the cause might be low disk space.

[cstrahan]

This might be related to my issue: #974

[cstrahan]

Definitely not low on space:

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p7  445G  194G  250G  44% /
/dev/nvme0n1p7  445G  194G  250G  44% /home
/dev/nvme0n1p6  969M  114M  789M  13% /boot
/dev/nvme0n1p1  197M  122M   76M  62% /boot/efi

[cstrahan]

@tonistiigi What can I do to help provide a repro? My team is blocked on this (we have to use BuildKit, or resort to hacks, to get SSH agent forwarding), and I'm willing to do anything that'll help resolve this.

[cstrahan]

I have an explanation of how exactly this bug is tickled: #974 (comment)

[tonistiigi]

This is likely fixed by #1023 .

#974 should be different and should already be fixed in latest versions.

[pwaller]

@tonistiigi #1023 sounds very much to me like the something that could have happened in my case. I'm happy to close the issue on that basis, since it was difficult to tickle in the first place. Thanks for looking at this, I'm glad to see plausible resolution :)

[quazgar]

Any workaround for this, apart from upgrading?

[tonistiigi]

@quazgar docker builder prune --filter "type=source.local" should probably do it but upgrading is the recommended path