Update buildkit #48

I noticed this when building a Dockerfile that failed because a file didn't exist, so went through error messages that looked like they had a duplicate "not found" in the output; [+] Building 0.9s (6/9) => [internal] load build definition from Dockerfile 0.2s => => transferring dockerfile: 306B 0.0s => [internal] load .dockerignore 0.1s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/library/alpine:latest 0.0s => CACHED [1/5] FROM docker.io/library/alpine 0.0s => [internal] load build context 0.6s => => transferring context: 701B 0.5s => ERROR [2/5] ADD no-such-file.txt / 0.0s ------ > [2/5] ADD no-such-file.txt /: ------ failed to compute cache key: "/no-such-file.txt" not found: not found Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

…ed pipes. Signed-off-by: Siebe Schaap <siebe@digibites.nl>

…ndows rather than syscall. Signed-off-by: Siebe Schaap <siebe@digibites.nl>

…ndowsPipeDialer(). Signed-off-by: Siebe Schaap <siebe@digibites.nl>

…th; allowed backslashes in host. Signed-off-by: Siebe Schaap <siebe@digibites.nl>

This updates all occurrences of Go 1.13 to Go 1.16; also updated the code that's used to redact credentials in URLs to use the Go implementation. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Support for Windows OpenSSH agent forwarding

- pkg/signal: ignore SIGURG on all platforms - pkg/archive: use v2 capabilities in layer archives - update ishidawataru/sctp to fix possible socket leak when bind fails docker/cli: - config: print deprecation warning when falling back to ~/.dockercfg Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

go.mod: update docker and docker cli to v20.10.7

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

cache: fix possible nil dereferences

git: support subdir component

hack: update proto generators for arm64

test: fix mirror cache handling

This allows using the resolver package without having to import the buildkit daemon configuration. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

move RegistryConfig to resolver package

update to go 1.16

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

vendor: fix broken gosum

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

add hack/shell helper for dev shell environment

Signed-off-by: ktock <ktokunaga.mail@gmail.com>

This provides the basic functionality for the parser to recognize and parse provided heredocs in supported commands. Signed-off-by: Justin Chadwell <me@jedevc.com>

Signed-off-by: Justin Chadwell <me@jedevc.com>

This modifies the command structures to support inline files, as well as provides the logic to compile them down into appropriate LLB definitions. Signed-off-by: Justin Chadwell <me@jedevc.com>

Signed-off-by: Justin Chadwell <me@jedevc.com>

Enable to apply registry configuration to stargz snapshotter

…ion) full diff: https://github.com/docker/docker/compare/v20.10.7..0ad2293d0e5b This applies the same / similar local changes to buildkit as were made in the branch that moby is currently vendoring: moby/buildkit@244e8cd...cpuguy83:update_libnetwork_import Unfortunately, this (again) requires a "replace" rule (probably until we tagged a release with this change). Go mod refuses to pick a version from master, and (without the replace rule), does weird, reverting docker to a very old version, predating the sirupsen rename: > [vendored 4/4] RUN --mount=target=/src,rw --mount=target=/go/pkg/mod,type=cache go mod tidy && go mod vendor && mkdir /out && cp -r go.mod go.sum vendor /out: #10 0.500 go: finding github.com/docker/docker 8dbd90ec00daa26dc45d7da2431c965dec99e8b4 #10 0.599 warning: ignoring symlink /src/examples/kube-consistent-hash #10 1.027 go: github.com/moby/buildkit/util/testutil/integration imports #10 1.027 github.com/docker/docker/testutil/daemon imports #10 1.027 github.com/docker/docker/opts imports #10 1.027 github.com/docker/libnetwork/ipamutils imports #10 1.027 github.com/docker/libnetwork/osl imports #10 1.027 github.com/Sirupsen/logrus: github.com/Sirupsen/logrus@v1.8.1: parsing go.mod: #10 1.027 module declares its path as: github.com/sirupsen/logrus #10 1.027 but was required as: github.com/Sirupsen/logrus ------ error: failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c go mod tidy && go mod vendor && mkdir /out && cp -r go.mod go.sum vendor /out]: exit code: 1 make: *** [vendor] Error 1 Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Signed-off-by: ktock <ktokunaga.mail@gmail.com>

Fix doc about Stargz Snapshotter and registry authentication

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

Dockerfile heredocs

Update docker/docker to master / v21.xx-dev (post libnetwork integration)

filesync: ensure sendclose is always called

session: make sure all token request keep correct context

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

Signed-off-by: Justin Chadwell <me@jedevc.com>

Fix file modes with remote ADD commands

remove() needs to be called while holding the manager lock Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

The locks usage is mixed up because two locks separate locks are actually needed. With a specific lock, calls to SetValue can be protected. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

cache: avoid concurrent maps write on prune

metadata: hold lock on storageitem update

fuse-overlayfs is available as an apk since Alpine 3.14. Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

vendor: update vt100 to fork and remove replace rule

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

This can be done as a separate change when needed. Also should analyze if this would affect the gogo incompatibility issues with newer proto. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

Upstream fixes needed for cleaner solution Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

Add opentelemetry support

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

dockerfile: ensure config resolve errors keep source location

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

remove go fork for building windows/arm64

Dockerfile: remove nsswitch as default in go1.16

This utility was moved to a separate package, which has no dependencies. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

replace uses of deprecated containerd/sys.RunningInUserNS()

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

integration: add common context base to all integration tests

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> (cherry picked from commit bc9a83144c83e9fd78007b7bfe92e8082c59d40e)

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

Hopefully this can be removed with a future upstream change that could make this configurable. The package also needs internal dependency that is copied in. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

Avoid client package having dependency on global detect package. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

update runc binary to v1.0.0 GA

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

Dockerfile: install fuse-overlayfs from apk

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

Fix this panic when copying from an image with no layers: ``` panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x50 pc=0xdd8c17] goroutine 326 [running]: github.com/moby/buildkit/cache/contenthash.(*cacheManager).Checksum(0xc0005ec030, 0x1682c00, 0xc000842140, 0x0, 0x0, 0xc0005d4023, 0x1, 0x0, 0x0, 0x0, ...) /src/cache/contenthash/checksum.go:95 +0x37 github.com/moby/buildkit/cache/contenthash.Checksum(0x1682c00, 0xc000842140, 0x0, 0x0, 0xc0005d4023, 0x1, 0x0, 0x0, 0x0, 0x0, ...) /src/cache/contenthash/checksum.go:59 +0xd5 github.com/moby/buildkit/solver/llbsolver.NewContentHashFunc.func1.1(0x0, 0x4425d6) /src/solver/llbsolver/result.go:59 +0x20a golang.org/x/sync/errgroup.(*Group).Go.func1(0xc00056a360, 0xc000594510) /src/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x59 created by golang.org/x/sync/errgroup.(*Group).Go /src/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66 ``` When the path is "/", we allow it because it's a noop. Based on moby#2185 Signed-off-by: Aaron Lehmann <alehmann@netflix.com>

Signed-off-by: Aaron Lehmann <alehmann@netflix.com>

llb: ensure meta resolver uses platform form constraints

Signed-off-by: Aaron Lehmann <alehmann@netflix.com>

Avoid nil pointer dereference when copying from image with no layers

Signed-off-by: Cory Bennett <cbennett@netflix.com>

ensure containerd io is complete and closed before returning

flightcontrol: reduce contention between goroutines

…vent leaks Signed-off-by: Cory Bennett <cbennett@netflix.com>

this allows progress.Controller to manage the writer lifecycle Signed-off-by: Cory Bennett <cbennett@netflix.com>

[moby#2112] progress.Controller should own the progress.Writer to prevent leaks

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

dockerfile: fix git version detection

Signed-off-by: Justin Chadwell <me@jedevc.com>

This avoids wrong metadata matches on small files Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

Add support for heredocs in the ONBUILD command

Allow none differ on local source to avoid false Dockerfile matches

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

Improve build errors and remove useless grpc prefix

progressui: print logs for failed step as summary in plain mode

Previously, heredoc names were restricted to simple alphanumeric strings. However, heredocs should support much more complex use-cases, including quoting anywhere, as well as allowing special symbols like `.` for easily expressing file extensions. This patch adds support for these more complex cases, by using the shell lexer to parse each heredoc name. Additionally, we include improvements to the lexer to optionally preserve escape tokens to avoid problems when lexing words that have already been lexed before. Signed-off-by: Justin Chadwell <me@jedevc.com>

Signed-off-by: Justin Chadwell <me@jedevc.com>

Finalize was only used outside the cache package in one place, which called it with the commit arg set to false. The code path followed when commit==false turned out to essentially be a no-op because it set "retain cache" to true if it was already set to true. It was thus safe to remove the only external call to it and remove it from the interface. This should be helpful for future efforts to simplify the equal{Mutable,Immutable} fields in cacheRecord, which exist due to the "lazy commit" feature that Finalize is tied into. Signed-off-by: Erik Sipsma <erik@sipsma.dev>

Improve heredoc parsing to allow more generic shell-words

Improve progress and history messages for heredoc-related commands

Remove unneeded Finalize method from ImmutableRef.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>

Fix ref leak if fileop ref fails to mount.

Remove duplicate "not found" from some error messages

contenthash: use SeekLowerBound to seek radix tree

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

enable riscv64 build

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>

Update QEMU emulators

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

Instead of just assuming that the default branch is master, use ls-remote to find out. Also removed tests that didn't specifiy a branch but required authentication, because those will fail now that the repo is actually checked. Signed-off-by: Levi Harrison <levisamuelharrison@gmail.com>

It is my suspecion that the tests were failing on previous commits because of the lack of authentication and other stuff like that available in gitidentifier as compared to gitsource Signed-off-by: Levi Harrison <levisamuelharrison@gmail.com>

Unfortunately, further test cases will have to be removed because gitindentifier will now leave the branch blank instead of filling it in Signed-off-by: Levi Harrison <levisamuelharrison@gmail.com>

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

Signed-off-by: ktock <ktokunaga.mail@gmail.com>

dockerfile: move run network to stable channel

Signed-off-by: Yamazaki Masashi <masi19bw@gmail.com>

exporter: Enable to specify the compression type for all layers of the finally exported image

Add full timestamp to logs

[carry] Automatically detect default git branch

Signed-off-by: Yamazaki Masashi <masi19bw@gmail.com>

GetByBlob checks to see if there are any other blobs with the same (uncompressed) ChainID and, if so, reuses their unpacked snapshot if it exists. The problem is if this code finds a match, it was trying to get the matching record, but couldn't do so when the match is lazy because the caller doesn't necessarily have descriptor handlers setup for it. This commit changes the behavior to just ignore any match with the same ChainID that's also lazy as they just aren't usable for the snapshot-reuse optimization. Signed-off-by: Erik Sipsma <erik@sipsma.dev>

dockerfile: add suggestions to how to fix certain errors

Remove meaningless encode

Ignore missing providers for blobs w/ same chainid.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

Signed-off-by: Morlay <morlay.null@gmail.com>

log should with traceID and spanID

authprovider: handle eaccess on storing token seeds

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

cache backend for github actions

github: update CI buildkit to v0.9.0-rc1

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

The current default were even lower than stdlib defaults. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

Signed-off-by: Morlay <morlay.null@gmail.com>

remove tracetransform package

refactor to use `util/bklog.G(ctx)` instead `logrus.` directly

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>

When running a WORKDIR instruction, buildkit will create that folder and chown it to the currently set user. For this, it will try to read the /etc/passwd file to get the proper UID, and if that user is not found in the file, the root user will be considered as the owner. However, Windows image do not have that file, which will result in an error while building the image. We can consider not finding the /etc/passwd file as the same as not finding the user in the file, which would solve this issue. Signed-off-by: Claudiu Belu <cbelu@cloudbasesolutions.com>

Skips getting UID/GUID if passwd/group file is not found

This is a safer alternative until we figure out why http.Transport based limiting fails. Some connections like cache export/import do not have a domain key atm and these connections use global pool. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

Signed-off-by: coryb <cbennett@netflix.com>

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

update to github.com/containerd/containerd v1.5.3

add per domain semaphore to limit concurrent connections

vendor: update go-actions-cache with custom client support

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>

Enhance test matrix

fixes moby#2248 Signed-off-by: coryb <cbennett@netflix.com>

This can be a significant amount of time that isn't currently accounted for in traces. Signed-off-by: Aaron Lehmann <alehmann@netflix.com>

Add span for layer export

fix dropped pull progress output due to canceled context

The previous implementation had many issues. Eg. on fetch, even if the data already existed and no remote connections were needed the request would still be waiting in the queue. Or if two fetches of same blob happened together they would take up two places in queue although there was only one remote request. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

resolver: increase default idle conns reuse

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>

Exporter config digest typo

new implementation for limiting tcp connections

tracing: update to otelhttp roundtripper

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

daemonless: wait for daemon to finish before exit

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

add ktock and crazy-max to maintainers

docs: update images-readme to v0.9

Update Dockerfile references to use 1.3

GitHub Actions cache docs

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>

Bump to codecov/codecov-action v2

Update buildkit #48

Update buildkit #48

Commits on Mar 29, 2021

Commits on May 20, 2021

Commits on May 27, 2021

Commits on Jun 1, 2021

Commits on Jun 5, 2021

Commits on Jun 7, 2021

Commits on Jun 8, 2021

Commits on Jun 9, 2021

Commits on Jun 10, 2021

Commits on Jun 11, 2021

Commits on Jun 12, 2021

Commits on Jun 14, 2021

Commits on Jun 15, 2021

Commits on Jun 16, 2021

Commits on Jun 17, 2021

Commits on Jun 18, 2021

Commits on Jun 19, 2021

Commits on Jun 20, 2021

Commits on Jun 21, 2021

Commits on Jun 22, 2021

Commits on Jun 23, 2021

Commits on Jun 24, 2021

Commits on Jun 26, 2021

Commits on Jun 27, 2021

Commits on Jun 28, 2021

Commits on Jun 29, 2021

Commits on Jun 30, 2021

Commits on Jul 1, 2021

Commits on Jul 2, 2021

Commits on Jul 3, 2021

Commits on Jul 5, 2021

Commits on Jul 6, 2021

Commits on Jul 7, 2021

Commits on Jul 9, 2021

Commits on Jul 10, 2021

Commits on Jul 11, 2021

Commits on Jul 12, 2021

Commits on Jul 13, 2021

Commits on Jul 14, 2021

Commits on Jul 15, 2021

Commits on Jul 16, 2021

Commits on Jul 17, 2021

Commits on Jul 19, 2021

Commits on Jul 20, 2021

Commits on Jul 25, 2021

Commits on Jul 26, 2021

Commits on Jul 27, 2021