grpc: received message larger than max (4564294 vs. 4194304)

[nlg521]

use pod.privileged.yaml install in k8s

 kubectl apply -f pod.privileged.yaml
 buildctl \
  --addr kube-pod://buildkitd \
  build --frontend dockerfile.v0 --local context=/path/to/dir --local dockerfile=/path/to/dir

get some error

=> => # Progress (5): 0.6/1.7 MB | 216/453 kB | 328/528 kB | 320/749 kB | 27 kB
error: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4564294 vs. 4194304)
failed to receive status
github.com/moby/buildkit/client.(*Client).solve.func4
/src/client/solve.go:258
golang.org/x/sync/errgroup.(*Group).Go.func1
/src/vendor/golang.org/x/sync/errgroup/errgroup.go:57
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1357

how can i set grpc receive msssage size ?

[tonistiigi]

=> => # Progress (5): 0.6/1.7 MB | 216/453 kB | 328/528 kB | 320/749 kB | 27 kB

This output is not from buildkit afaics.

We can increase grpc message size but we would need to understand what request is hitting the limit and in what condition.

[nlg521]

this is a maven projecct ，the compile output is

 => CACHED [build 5/19] COPY config config                                                                                            0.0s
 => CACHED [build 6/19] COPY turbo-web/pom.xml turbo-web/pom.xml                                                                      0.0s
 => CACHED [build 7/19] COPY turbo-service/pom.xml turbo-service/pom.xml                                                              0.0s
 => CACHED [build 8/19] COPY turbo-dao/pom.xml turbo-dao/pom.xml                                                                      0.0s
 => CACHED [build 9/19] COPY turbo-starter/pom.xml turbo-starter/pom.xml                                                              0.0s
 => CACHED [build 10/19] COPY turbo-archetypes/pom.xml turbo-archetypes/pom.xml                                                       0.0s
 => CACHED [build 11/19] COPY generator/pom.xml generator/pom.xml                                                                     0.0s
 => CACHED [build 12/19] COPY turbo-web/src turbo-web/src                                                                             0.0s
 => [build 13/19] COPY turbo-service/src turbo-service/src                                                                            0.2s
 => [build 14/19] COPY turbo-dao/src turbo-dao/src                                                                                    0.5s
 => [build 15/19] COPY turbo-starter/src turbo-starter/src                                                                            0.4s
 => [build 16/19] COPY generator/src generator/src                                                                                    0.4s
 => [build 17/19] COPY turbo-archetypes/src  turbo-archetypes/src                                                                     0.4s
 => [build 18/19] RUN  --mount=type=cache,target=/root/.m2,id=rep,sharing=locked  mvn install -DskipTests  -s .mvn/settings.xml       9.7s
 => => # Downloading from central: http://artifactory.jd.com/libs-releases/com/googlecode/concurrentlinkedhashmap/concurrentlinkedhashmap-
 => => # lru/1.4.2/concurrentlinkedhashmap-lru-1.4.2.jar
 => => # Downloaded from central: http://artifactory.jd.com/libs-releases/com/googlecode/concurrentlinkedhashmap/concurrentlinkedhashmap-l
 => => # ru/1.4.2/concurrentlinkedhashmap-lru-1.4.2.jar (117 kB at 498 kB/s)
 => => # Downloading from central: http://artifactory.jd.com/libs-releases/com/carrotsearch/java-sizeof/0.0.4/java-sizeof-0.0.4.jar
 => => # Progress (5): 0.6/1.7 MB | 216/453 kB | 328/528 kB | 320/749 kB | 27 kB
error: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4564294 vs. 4194304)
failed to receive status
github.com/moby/buildkit/client.(*Client).solve.func4
        /src/client/solve.go:258
golang.org/x/sync/errgroup.(*Group).Go.func1
        /src/vendor/golang.org/x/sync/errgroup/errgroup.go:57

[nlg521]

edit this file /buildkit/client/client.go in line 33 with this , it is ok .

// New returns a new buildkit client. Address can be empty for the system-default address.
func New(ctx context.Context, address string, opts ...ClientOpt) (*Client, error) {
	gopts := []grpc.DialOption{
		grpc.WithDefaultCallOptions(grpc.MaxCallRecvMsgSize(defaults.DefaultMaxRecvMsgSize)),
		grpc.WithDefaultCallOptions(grpc.MaxCallSendMsgSize(defaults.DefaultMaxSendMsgSize)),
	}
   ...

Can it be set as a parameter ？

[tonistiigi]

Can you test with master buildkit/buildctl with --debug set in buildctl. You should get a better trace. I wonder if maven is producing so much logs that it overflows the object. And maybe the logs are not newline-separated causing them to pile up in a single object. In first case, we might need to drop logs as pulling 5MB of logs in this case unnecessarily slows down build. For the second one we should have a hard limit on log size.

If that is the case it looks partly as maven error as well. Buildkit is not enabling tty when calling maven so it shouldn't repeatedly split out the ansi codes.

[nlg521]

but ,Encountered another problem， It seems that the connection timed out ,how can i set ?

=> CACHED [build 11/19] COPY generator/pom.xml generator/pom.xml                                                                     0.0s
 => CACHED [build 12/19] COPY turbo-web/src turbo-web/src                                                                             0.0s
 => CACHED [build 13/19] COPY turbo-service/src turbo-service/src                                                                     0.0s
 => CACHED [build 14/19] COPY turbo-dao/src turbo-dao/src                                                                             0.0s
 => CACHED [build 15/19] COPY turbo-starter/src turbo-starter/src                                                                     0.0s
 => CACHED [build 16/19] COPY generator/src generator/src                                                                             0.0s
 => CACHED [build 17/19] COPY turbo-archetypes/src  turbo-archetypes/src                                                              0.0s
 => [build 18/19] RUN  --mount=type=cache,target=/root/.m2,id=rep,sharing=locked  mvn install -DskipTests  -s .mvn/settings.xml      69.8s
 => => # kB/s)
 => => # Downloading from central: http://artifactory.jd.com/plugins-releases/org/objenesis/objenesis-parent/2.2/objenesis-parent-2.2.pom
 => => # Downloaded from central: http://artifactory.jd.com/plugins-releases/org/objenesis/objenesis-parent/2.2/objenesis-parent-2.2.pom (
 => => # 17 kB at 787 kB/s)
 => => # Downloading from central: http://artifactory.jd.com/plugins-releases/org/apache/maven/maven-archiver/2.6/maven-archiver-2.6.pom
 => => # Progress (1): 4.3 kB
**error: failed to receive status: rpc error: code = Canceled desc = context canceled**
29832 v0.7.0-120-g48cb722f.m buildctl --debug --addr kube-pod://buildkitd-76d67f4686-cvjqc build --opt override-copy-image=j-hub.jd.com/turbo/dockerfile-copy:v0.1.9 --frontend dockerfile.v0 --local context=. --local dockerfile=.
github.com/moby/buildkit/client.(*Client).solve.func4
        /src/client/solve.go:257
golang.org/x/sync/errgroup.(*Group).Go.func1
        /src/vendor/golang.org/x/sync/errgroup/errgroup.go:57
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1357

[tonistiigi]

Can it be set as a parameter ？

Setting it as a parameter seems wrong solution. If it is too small we should set a bigger value. But bigger values are indicator that something else is wrong.

[nlg521]

yes it set with --debug ,but can't see any better trace.

$ buildctl --debug --addr kube-pod://buildkitd-76d67f4686-cvjqc build --opt override-copy-image=j-hub.jd.com/turbo/dockerfile-copy:v0.1.
9 --frontend dockerfile.v0 --local context=. --local dockerfile=.
DEBU[0000] commandconn: starting kubectl with [--context= --namespace= exec --container= -i buildkitd-76d67f4686-cvjqc -- buildctl dial-std
io]
[+] Building 0.0s (0/0)
DEBU[0000] unimplemented call: SetReadDeadline(0001-01-01 00:00:00 +0000 UTC)
[+] Building 1.3s (11/37)

[tonistiigi]

Using master buildctl and moby/buildkit:master image should show you a better stacktrace when the error happens.

[nlg521]

Can it be set as a parameter ？

Setting it as a parameter seems wrong solution. If it is too small we should set a bigger value. But bigger values are indicator that something else is wrong.

i kow ,i kow ,this is not a goog idea .

[nlg521]

moby/buildkit:master

using this moby/buildkit:master in k8s , stacktrace can‘t see
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: buildkitd
name: buildkitd
spec:
replicas: 1
selector:
matchLabels:
app: buildkitd
template:
metadata:
labels:
app: buildkitd
spec:
containers:
- name: buildkitd
image: moby/buildkit:master

[nlg521]

@tonistiigi

Can you test with master buildkit/buildctl with --debug set in buildctl. You should get a better trace. I wonder if maven is producing so much logs that it overflows the object. And maybe the logs are not newline-separated causing them to pile up in a single object. In first case, we might need to drop logs as pulling 5MB of logs in this case unnecessarily slows down build. For the second one we should have a hard limit on log size.

If that is the case it looks partly as maven error as well. Buildkit is not enabling tty when calling maven so it shouldn't repeatedly split out the ansi codes.

hi，i try build in my local pc, build without error ,Please help me ，what can i do ？

docker run -d --name buildkitd --privileged moby/buildkit:latest
export BUILDKIT_HOST=docker-container://buildkitd
buildctl   build   --frontend dockerfile.v0 --local context=. --local dockerfile=.