Local cache error/unused with Buildkit go client #2599
Comments
|
We have noticed an interesting
We are not seeing this error when running with
We don't know if this is related, but it does look suspicious. We are still investigating. |
|
The following change fixed the @@ -10,6 +10,8 @@ import (
_ "github.com/moby/buildkit/client/connhelper/dockercontainer" // import the container connection driver
"github.com/moby/buildkit/client/llb"
bkgw "github.com/moby/buildkit/frontend/gateway/client"
+ "github.com/moby/buildkit/session"
+ "github.com/moby/buildkit/session/auth/authprovider"
"golang.org/x/sync/errgroup"
)
@@ -28,6 +30,7 @@ func New(ctx context.Context) (*Client, error) {
func (c *Client) Do(ctx context.Context) error {
// Fill options
+ attachable := []session.Attachable{authprovider.NewDockerAuthProvider(os.Stderr)}
opts := bk.SolveOpt{
Exports: []bk.ExportEntry{{Type: "local", OutputDir: "result"}},
CacheImports: []bk.CacheOptionsEntry{{
@@ -43,6 +46,7 @@ func (c *Client) Do(ctx context.Context) error {
"dest": "store",
},
}},
+ Session: attachable,
}
return c.exec(ctx, opts)It did not fix the issue that we reported. |
|
We have noticed a difference in the &client.cacheOptions{options:moby_buildkit_v1.CacheOptions{ExportRefDeprecated:"", ImportRefsDeprecated:[]string(nil), ExportAttrsDeprecated:map[
string]string{}, Exports:[]*moby_buildkit_v1.CacheOptionsEntry{(*moby_buildkit_v1.CacheOptionsEntry)(0xc000148240)}, Imports:[]*moby_buildkit_v1.
CacheOptionsEntry{(*moby_buildkit_v1.CacheOptionsEntry)(0xc000148340)}, XXX_NoUnkeyedLiteral:struct {}{}, XXX_unrecognized:[]uint8(nil), XXX_size
cache:0}, contentStores:map[string]content.Store{"local:store":(*local.store)(0xc00007c100)}, indicesToUpdate:map[string]string{"store/index.json
":"latest"}, frontendAttrs:map[string]string{}}vs what &client.cacheOptions{options:moby_buildkit_v1.CacheOptions{ExportRefDeprecated:"",
ImportRefsDeprecated:[]string(nil), ExportAttrsDeprecated:map[string]string{},
Exports:[]*moby_buildkit_v1.CacheOptionsEntry{(*moby_buildkit_v1.CacheOptionsEntry)(0x14000690140)},
Imports:[]*moby_buildkit_v1.CacheOptionsEntry{(*moby_buildkit_v1.CacheOptionsEntry)(0x14000690240)},
XXX_NoUnkeyedLiteral:struct {}{}, XXX_unrecognized:[]uint8(nil),
XXX_sizecache:0},
contentStores:map[string]content.Store{"local:store":(*local.store)(0x140006a8120)},
indicesToUpdate:map[string]string{"store/index.json":"latest"},
frontendAttrs:map[string]string{"cache-imports":"[{\"Type\":\"local\",\"Attrs\":{\"digest\":\"sha256:fc046d2f4e63eb1ce6d32e7f1f68904d1531e29b09dda0b27d954f3d99ae6ea5\",\"src\":\"store\"}}]"}}The |
|
Based on this code in the buildkit client, the frontend must not be set: Lines 23 to 32 in a640b47 If that is the case, how do we pass cache instructions to buildkit? We understand that one way of doing this is via Lines 487 to 500 in a640b47 Are we on the right track? 🤔 https://gist.github.com/TomChv/ecf651b43553a2bd7da2bd5448c9dd13#file-main-go-L33-L45 |
|
If you are using If the local export doesn't work at all then check that the |
I tried to update // See the gist for complete code
res, err := c.Solve(ctx, bkgw.SolveRequest{
Definition: def.ToPB(),
CacheImports: []bkgw.CacheOptionsEntry{{
Type: "local",
Attrs: map[string]string{
"src": "store",
},
}},
})Result # Remove old daemon
docker container stop dagger-buildkitd && docker container rm dagger-buildkitd && docker volume rm dagger-buildkitd
dagger-buildkitd
dagger-buildkitd
dagger-buildkitd
# Start buildkit daemon
docker run --net=host -d --restart always -v dagger-buildkitd:/var/lib/buildkit --name dagger-buildkitd --privileged moby/buildkit:v0.9.3
7712e7878dafac5edefbbf1a3452e1142348842635c7e4e364f2fd4504e7fe07
# List files
ls
go.mod go.sum main.go
# Run example
time go run ./main.go
go run ./main.go 0.92s user 0.64s system 13% cpu 11.854 total
# Check files
l store/blobs/sha256
total 5328
-r--r--r-- 1 tomchauveau staff 894B Feb 11 15:09 11d64926da617f9a5ca9695d30b085de2d8ce7ddf938c4d98ea5478ce7822521
-r--r--r-- 1 tomchauveau staff 560B Feb 11 15:09 6293955c0b870a4d79d354f2fe6394c47f3906cd9c785ab2e08647e486ed883d
-r--r--r-- 1 tomchauveau staff 2.6M Feb 11 15:09 9b3977197b4f2147bdd31e1271f811319dcd5c2fc595f14e81f5351ab6275b99 // Cache is successfully exported
-r--r--r-- 1 tomchauveau staff 105B Feb 11 15:09 c37711adfbf5490df2a3ad0be475411e4e54f1829da8770e2a08367782806b96
# Stop and remove container
docker container stop dagger-buildkitd && docker container rm dagger-buildkitd && docker volume rm dagger-buildkitd
dagger-buildkitd
dagger-buildkitd
dagger-buildkitd
# Rerun buildkit daemon
docker run --net=host -d --restart always -v dagger-buildkitd:/var/lib/buildkit --name dagger-buildkitd --privileged moby/buildkit:v0.9.3
f789e7deb3d2ec1f5f5fe4eb7aca2b44bb642463ba78d6eb4adc5a8251447065
# Re execute
time go run ./main.go
go run ./main.go 0.92s user 0.68s system 13% cpu 11.725 total
Actually we thought that they were an issue with ERRO[0009] (*service).Write failed error="rpc error: code = Canceled desc = context canceled" expected="sha256:c4f68cac3b6805a14eef6e8812915bc64f81990df1667d1d97b720bfc05aaff7" ref="sha256:c4f68cac3b6805a14eef6e8812915bc64f81990df1667d1d97b720bfc05aaff7" total=894But this error do not always happens. The real problem is that the buildkit go client do not use the local cache directory when we are using raw LLB. |
|
Thank you @tonistiigi for the suggestions. As a recap, the Given our starting // fmt.Printf("contentStores: %#v\n", contentStores)
contentStores: map[string]content.Store{"local:store":(*local.store)(0xc0000a80e0)}
// fmt.Printf("contentStore: %#v\n", cs)
contentStore: &local.store{root:"store", ls:local.LabelStore(nil)}If we run the same with FROM alpine
RUN sleep 5 && echo testthis is how // fmt.Printf("contentStores: %#v\n", contentStores)
contentStores: map[string]content.Store{"local:store":(*local.store)(0x140002fafc0)}
// fmt.Printf("contentStore: %#v\n", cs)
contentStore: &local.store{root:"store", ls:local.LabelStore(nil)}&{{ [] map[] [0x14000331a00] [0x14000331b00] {} [] 0} map[local:store:0x140002fafc0] map[store/index.json:latest] map[cache-imports:[{"Type":"local","Attrs":{"digest":"sha256:a2c807e50263db527d8160cc1601503a6e4d218ec8be46d2a16d42305b8f96a9","src":"store"}}]]}This suggests to us that As a next step, we will try to understand why the local Lines 402 to 418 in 63eff24 By the way, we have confirmed that the {"schemaVersion":2,"manifests":[{"mediaType":"application/vnd.oci.image.index.v1+json","digest":"sha256:dc45a8a989c08248dc081675f96036c87360121b50a4caa854d4c4e1f104f43d","size":895,"annotations":{"org.opencontainers.image.ref.name":"latest"}}]}It is possible that our failing export is the reason why our import is not getting configured correctly, but we are not sure. |
|
We (+@TomChv) continued with this today and noticed that we made a mistake last time. The We have confirmed that
This confirms our assumption: the cache options are not propagated correctly when using the go client. The next step is to confirm our next assumption: if Lines 196 to 216 in 63eff24 I will skip next week, but @TomChv should be around to continue with this. |
|
Thank you for the PR, unfortunately, I tried with your changes and it didn't solves the cache issue. I also continue digging into the codebase and that's really strange because func (c *lLBBridgeClient) Solve(ctx context.Context, in *SolveRequest, opts ...grpc.CallOption) (*SolveResponse, error) {
out := new(SolveResponse)
fmt.Printf("Solve LLB client %#v\n", in)
err := c.cc.Invoke(ctx, "/moby.buildkit.v1.frontend.LLBBridge/Solve", in, out, opts...)
if err != nil {
return nil, err
}
return out, nil
}With this, we are sure that cache is forwarded to the daemon, do you think the problem come from the daemon? Click to see a demo with logs# List current files
ls
buildkit-repro go.mod go.sum main.go vendor
# Run buildkit daemon
docker run --net=host -d --restart always -v dagger-buildkitd:/var/lib/buildkit --name dagger-buildkitd --privileged moby/buildkit:v0.10.0-rc1
7fbf5df0e24b246bd4c1a57dd05a70e2c3bf50c46a65e382bb4c7972ad8cd697
# Export host
export BUILDKIT_HOST=docker-container://dagger-buildkitd
# Execute repro case => no cache exist for now
time ./buildkit-repro
WARN[0000] local cache import at store not found due to err: could not read store/index.json: open store/index.json: no such file or directory
Solve LLB client &moby_buildkit_v1_frontend.SolveRequest{Definition:(*pb.Definition)(0x140003b02a0), Frontend:"", FrontendOpt:map[string]string(nil), ImportCacheRefsDeprecated:[]string(nil), AllowResultReturn:true, AllowResultArrayRef:true, Final:false, ExporterAttr:[]uint8(nil), CacheImports:[]*moby_buildkit_v1_frontend.CacheOptionsEntry{(*moby_buildkit_v1_frontend.CacheOptionsEntry)(0x140006f2340)}, FrontendInputs:map[string]*pb.Definition(nil), Evaluate:false, XXX_NoUnkeyedLiteral:struct {}{}, XXX_unrecognized:[]uint8(nil), XXX_sizecache:0}
./buildkit-repro 0.21s user 0.20s system 2% cpu 14.816 total
# List files => Cache is correctly created
ls
buildkit-repro go.mod go.sum main.go result store vendor
# List store
l store/blobs/sha256/
total 5328
drwxr-xr-x 6 tomchauveau staff 192B Feb 24 17:42 .
drwxr-xr-x 3 tomchauveau staff 96B Feb 24 17:42 ..
-r--r--r-- 1 tomchauveau staff 895B Feb 24 17:42 7c1adf2606c30da26a4ed4ad887ee6c6c814183e315b2711cac4f49c0b135a9a
-r--r--r-- 1 tomchauveau staff 2.6M Feb 24 17:42 9b3977197b4f2147bdd31e1271f811319dcd5c2fc595f14e81f5351ab6275b99
-r--r--r-- 1 tomchauveau staff 105B Feb 24 17:42 9ebb20b96f11fb5412be45455f55eb110c787dd2cc0364fe48b233d58857bfa2
-r--r--r-- 1 tomchauveau staff 559B Feb 24 17:42 a5e548dc90fe39a789811dbd6a186eb8383b6f892568ee83aa843b07f64a49ba
# Down and restart daemon
docker container stop dagger-buildkitd && docker container rm dagger-buildkitd && docker volume rm dagger-buildkitd
dagger-buildkitd
dagger-buildkitd
dagger-buildkitd
docker run --net=host -d --restart always -v dagger-buildkitd:/var/lib/buildkit --name dagger-buildkitd --privileged moby/buildkit:v0.10.0-rc1
62b5ab5a52b7364a909601589f4e66784a3372866b36ac8c395b305f0a0af7e1
# Execute
time ./buildkit-repro
Solve LLB client &moby_buildkit_v1_frontend.SolveRequest{Definition:(*pb.Definition)(0x140000fea20), Frontend:"", FrontendOpt:map[string]string{"cache-imports":"[{\"Type\":\"local\",\"Attrs\":{\"digest\":\"sha256:7c1adf2606c30da26a4ed4ad887ee6c6c814183e315b2711cac4f49c0b135a9a\",\"src\":\"store\"}}]"}, ImportCacheRefsDeprecated:[]string(nil), AllowResultReturn:true, AllowResultArrayRef:true, Final:false, ExporterAttr:[]uint8(nil), CacheImports:[]*moby_buildkit_v1_frontend.CacheOptionsEntry{(*moby_buildkit_v1_frontend.CacheOptionsEntry)(0x140003c9780)}, FrontendInputs:map[string]*pb.Definition(nil), Evaluate:false, XXX_NoUnkeyedLiteral:struct {}{}, XXX_unrecognized:[]uint8(nil), XXX_sizecache:0}
./buildkit-repro 0.18s user 0.21s system 2% cpu 13.841 total # Cache do not works even it's sent :/ |
|
Okay, I figure out which piece of code is broken 🥳 Basically, the cache is not imported when Here's a new gist to reproduce the issue and understand the difference. Since both are calling func (c *Client) Build(ctx context.Context, opt SolveOpt, product string, buildFunc gateway.BuildFunc, statusChan chan *SolveStatus) (*SolveResponse, error) {
defer func() {
if statusChan != nil {
close(statusChan)
}
}()
if opt.Frontend != "" {
return nil, errors.New("invalid SolveOpt, Build interface cannot use Frontend")
}
if product == "" {
product = apicaps.ExportedProduct
}
feOpts := opt.FrontendAttrs
opt.FrontendAttrs = nil
workers, err := c.ListWorkers(ctx)
if err != nil {
return nil, errors.Wrap(err, "listing workers for Build")
}
var gworkers []gateway.WorkerInfo
for _, w := range workers {
gworkers = append(gworkers, gateway.WorkerInfo{
ID: w.ID,
Labels: w.Labels,
Platforms: w.Platforms,
})
}
cb := func(ref string, s *session.Session, opts map[string]string) error {
for k, v := range opts {
if feOpts == nil {
feOpts = map[string]string{}
}
feOpts[k] = v
}
gwClient := c.gatewayClientForBuild(ref)
g, err := grpcclient.New(ctx, feOpts, s.ID(), product, gwClient, gworkers)
if err != nil {
return err
}
gwClient.caps = g.BuildOpts().Caps
if err := g.Run(ctx, buildFunc); err != nil {
return errors.Wrap(err, "failed to run Build function")
}
return nil
}
return c.solve(ctx, nil, cb, opt, statusChan) |
|
Finally! The current issue come from a missing Here's an example # Store is already existing
➜ ls
go.mod go.sum main.go result store vendor
➜ docker run --net=host -d --restart always -v dagger-buildkitd:/var/lib/buildkit --name dagger-buildkitd --privileged moby/buildkit:v0.10.0-rc1
3006ad43256b01061322f2d51d019f7c4d017cd36d600b1867cafa10d03107ac
➜ time go run ./main.go
Support Import Caches: true
GrpcClient.Solve: cacheImports[0]: &moby_buildkit_v1_frontend.CacheOptionsEntry{Type:"local", Attrs:map[string]string{"src":"store"}, XXX_NoUnkeyedLiteral:struct {}{}, XXX_unrecognized:[]uint8(nil), XXX_sizecache:0}
go run ./main.go 0.89s user 0.67s system 13% cpu 11.705 totalAs you see in Attrs:map[string]string{"src":"store"}Now if I run and hardcode the # Clean and run docker
➜ docker container stop dagger-buildkitd && docker container rm dagger-buildkitd && docker volume rm dagger-buildkitd
➜ docker run --net=host -d --restart always -v dagger-buildkitd:/var/lib/buildkit --name dagger-buildkitd --privileged moby/buildkit:v0.10.0-rc1
f26162a4b48c007b66579f0c0fdb16685a25289084b14615cc06286bc075e2af
➜ time go run ./main.go
Support Import Caches: true
GrpcClient.Solve: cacheImports[0]: &moby_buildkit_v1_frontend.CacheOptionsEntry{Type:"local", Attrs:map[string]string{"digest":"sha256:8a2ca54e54fc09ed5a0c2293bbe0795b93fb5d06dae824a6d6cb7df174547403", "src":"store"}, XXX_NoUnkeyedLiteral:struct {}{}, XXX_unrecognized:[]uint8(nil), XXX_sizecache:0}
go run ./main.go 0.88s user 0.66s system 45% cpu 3.390 total # Cache is usedAs you see, with a Attrs:map[string]string{"digest":"sha256:8a2ca54e54fc09ed5a0c2293bbe0795b93fb5d06dae824a6d6cb7df174547403", "src":"store"}Now, I have to figure out why |
|
After more research, I spotted the problem. After comparing with When we use
But So here's possible fixes:
|
|
I implemented a possible fix there, but I'm not sure this is the best solution since there is a lot of code duplication with previous diff --git a/frontend/gateway/grpcclient/client.go b/frontend/gateway/grpcclient/client.go
index d8e2799f..7fca61b4 100644
--- a/frontend/gateway/grpcclient/client.go
+++ b/frontend/gateway/grpcclient/client.go
@@ -7,6 +7,7 @@ import (
"io"
"net"
"os"
+ "path/filepath"
"strings"
"sync"
"syscall"
@@ -16,6 +17,7 @@ import (
gogotypes "github.com/gogo/protobuf/types"
"github.com/golang/protobuf/ptypes/any"
"github.com/moby/buildkit/client/llb"
+ "github.com/moby/buildkit/client/ociindex"
"github.com/moby/buildkit/frontend/gateway/client"
pb "github.com/moby/buildkit/frontend/gateway/pb"
"github.com/moby/buildkit/identity"
@@ -25,6 +27,7 @@ import (
"github.com/moby/buildkit/util/grpcerrors"
"github.com/moby/sys/signal"
digest "github.com/opencontainers/go-digest"
+ ocispecs "github.com/opencontainers/image-spec/specs-go/v1"
"github.com/pkg/errors"
fstypes "github.com/tonistiigi/fsutil/types"
"golang.org/x/sync/errgroup"
@@ -313,24 +316,45 @@ func (c *grpcClient) Warn(ctx context.Context, dgst digest.Digest, msg string, o
return err
}
-func (c *grpcClient) Solve(ctx context.Context, creq client.SolveRequest) (res *client.Result, err error) {
- if creq.Definition != nil {
- for _, md := range creq.Definition.Metadata {
- for cap := range md.Caps {
- if err := c.llbCaps.Supports(cap); err != nil {
- return nil, err
- }
- }
- }
- }
+type cacheOptions struct {
+ legacyRegistryCacheImports []string
+ cacheImports []*pb.CacheOptionsEntry
+}
+
+func parseCacheOptions(ctx context.Context, reqCacheImports []client.CacheOptionsEntry, supportCapImportCaches bool) (*cacheOptions, error) {
var (
// old API
legacyRegistryCacheImports []string
// new API (CapImportCaches)
cacheImports []*pb.CacheOptionsEntry
)
- supportCapImportCaches := c.caps.Supports(pb.CapImportCaches) == nil
- for _, im := range creq.CacheImports {
+
+ for _, im := range reqCacheImports {
+ attrs := im.Attrs
+ if im.Type == "local" {
+ csDir := im.Attrs["src"]
+ if csDir == "" {
+ return nil, errors.New("local cache importer requires src")
+ }
+
+ // if digest is not specified, load from "latest" tag
+ if attrs["digest"] == "" {
+ idx, err := ociindex.ReadIndexJSONFileLocked(filepath.Join(csDir, "index.json"))
+ if err != nil {
+ bklog.G(ctx).Warning("local cache import at " + csDir + " not found due to err: " + err.Error())
+ continue
+ }
+ for _, m := range idx.Manifests {
+ if (m.Annotations[ocispecs.AnnotationRefName] == "latest" && attrs["tag"] == "") || (attrs["tag"] != "" && m.Annotations[ocispecs.AnnotationRefName] == attrs["tag"]) {
+ attrs["digest"] = string(m.Digest)
+ break
+ }
+ }
+ if attrs["digest"] == "" {
+ return nil, errors.New("local cache importer requires either explicit digest, \"latest\" tag or custom tag on index.json")
+ }
+ }
+ }
if !supportCapImportCaches && im.Type == "registry" {
legacyRegistryCacheImports = append(legacyRegistryCacheImports, im.Attrs["ref"])
} else {
@@ -340,6 +364,27 @@ func (c *grpcClient) Solve(ctx context.Context, creq client.SolveRequest) (res *
})
}
}
+ return &cacheOptions{
+ cacheImports: cacheImports,
+ legacyRegistryCacheImports: legacyRegistryCacheImports,
+ }, nil
+}
+
+func (c *grpcClient) Solve(ctx context.Context, creq client.SolveRequest) (res *client.Result, err error) {
+ if creq.Definition != nil {
+ for _, md := range creq.Definition.Metadata {
+ for cap := range md.Caps {
+ if err := c.llbCaps.Supports(cap); err != nil {
+ return nil, err
+ }
+ }
+ }
+ }
+ supportCapImportCaches := c.caps.Supports(pb.CapImportCaches) == nil
+ cacheOpts, err := parseCacheOptions(ctx, creq.CacheImports, supportCapImportCaches)
+ if err != nil {
+ return nil, err
+ }
// these options are added by go client in solve()
if _, ok := creq.FrontendOpt["cache-imports"]; !ok {
@@ -367,9 +412,9 @@ func (c *grpcClient) Solve(ctx context.Context, creq client.SolveRequest) (res *
AllowResultReturn: true,
AllowResultArrayRef: true,
// old API
- ImportCacheRefsDeprecated: legacyRegistryCacheImports,
+ ImportCacheRefsDeprecated: cacheOpts.legacyRegistryCacheImports,
// new API
- CacheImports: cacheImports,
+ CacheImports: cacheOpts.cacheImports,
}
// backwards compatibility with inline return |
|
Hey @tonistiigi -- TLDR: we have spotted an issue with local cache imports.
Repro case here. @TomChv managed to find the issue in the codebase, details above. |
Problem
Importing or exporting local cache directory using the Go buildkit client doesn't work.
The error that I'm always getting on the first run is :
This error sometimes happens on subsequent runs too (but not always), but not with buildctl.
In addition, even if the error is not thrown, the local cache directory is never used with the go client, it works as expected with buildctl but I do not understand why.
Buildkit version: 0.9.3
How to reproduce
This gist contains everything that you need to reproduce this issue.
Additional information
This started as part of dagger/dagger#1551 (the repository is private).
/cc @gerhard
The text was updated successfully, but these errors were encountered: