buildkitd.default.toml is not taken into account #2683

dvdksn · 2024-09-11T08:08:38Z

Contributing guidelines

I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

... the documentation does not mention anything about my problem
... there are no open or closed issues that are related to my problem

Description

By default, buildx should use a default buildkitd config if it exists in any of the following locations:

$BUILDX_CONFIG/buildkitd.default.toml
$DOCKER_CONFIG/buildx/buildkitd.default.toml
~/.docker/buildx/buildkitd.default.toml

But it does not seem like buildx is interpreting this config at all.

Expected behaviour

Settings in e.g. ~/.docker/buildx/buildkitd.default.toml should be automatically picked up with buildx create

Actual behaviour

buildkitd.default.toml is ignored

Buildx version

github.com/docker/buildx v0.17.0 78c8c28

Docker info

No response

Builders list

NAME/NODE              DRIVER/ENDPOINT     STATUS     BUILDKIT   PLATFORMS
container*             docker-container                          
 \_ container0          \_ desktop-linux   running    3a70550    linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
focused_cerf           docker-container                          
 \_ focused_cerf0       \_ desktop-linux   inactive              
default                docker                                    
 \_ default             \_ default         running    v0.15.2    linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64
desktop-linux          docker                                    
 \_ desktop-linux       \_ desktop-linux   running    v0.15.2    linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64
cloud-docker-default   cloud               error

Configuration

# docker-bake.hcl
target default {
  entitlements = ["security.insecure", "network.host"]
}

# ~/.docker/buildx/buildkitd.default.toml
insecure-entitlements = [ "network.host", "security.insecure" ]

$ docker buildx create --use
$ docker buildx bake --allow security.insecure --allow network.host
ERROR: failed to solve: granting entitlement security.insecure is not allowed by build daemon configuration

Build logs

No response

Additional info

No response

The text was updated successfully, but these errors were encountered:

crazy-max · 2024-09-11T08:33:06Z

It seems good on my side:

$ cat ~/.docker/buildx/buildkitd.default.toml 
# debug enables additional debug logging
debug = true
# trace enables additional trace logging (very verbose, with potential performance impacts)
trace = true
# insecure-entitlements allows insecure entitlements, disabled by default.
insecure-entitlements = [ "network.host", "security.insecure" ]

[log]
  # log formatter: json or text
  format = "text"

$ docker buildx create --name builder --driver-opt "image=moby/buildkit:v0.16.0"
builder
$ docker buildx inspect builder --bootstrap
#1 [internal] booting buildkit
#1 pulling image moby/buildkit:v0.16.0
#1 pulling image moby/buildkit:v0.16.0 1.1s done
#1 creating container buildx_buildkit_builder0
#1 creating container buildx_buildkit_builder0 0.5s done
#1 DONE 1.6s
Name:          builder
Driver:        docker-container
Last Activity: 2024-09-11 08:26:47 +0000 UTC

Nodes:
Name:                  builder0
Endpoint:              unix:///var/run/docker.sock
Driver Options:        image="moby/buildkit:v0.16.0"
Status:                running
BuildKit daemon flags: --allow-insecure-entitlement=network.host
BuildKit version:      v0.16.0
Platforms:             linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
Labels:
 org.mobyproject.buildkit.worker.executor:         oci
 org.mobyproject.buildkit.worker.hostname:         daa2f382ae75
 org.mobyproject.buildkit.worker.network:          host
 org.mobyproject.buildkit.worker.oci.process-mode: sandbox
 org.mobyproject.buildkit.worker.selinux.enabled:  false
 org.mobyproject.buildkit.worker.snapshotter:      overlayfs
GC Policy rule#0:
 All:           false
 Filters:       type==source.local,type==exec.cachemount,type==source.git.checkout
 Keep Duration: 48h0m0s
 Keep Bytes:    488.3MiB
GC Policy rule#1:
 All:           false
 Keep Duration: 1440h0m0s
 Keep Bytes:    94.06GiB
GC Policy rule#2:
 All:        false
 Keep Bytes: 94.06GiB
GC Policy rule#3:
 All:        true
 Keep Bytes: 94.06GiB

In the container I have the conf:

$ docker exec -it buildx_buildkit_builder0 cat /etc/buildkit/buildkitd.toml
debug = true
insecure-entitlements = ["network.host", "security.insecure"]
trace = true

[log]
  format = "text"

And buildkitd is using it:

$ docker exec -it buildx_buildkit_builder0 ps
PID   USER     TIME  COMMAND
    1 root      0:00 /sbin/docker-init -- buildkitd --config /etc/buildkit/buildkitd.toml --allow-insecure-entitlement=network.host
    7 root      0:00 buildkitd --config /etc/buildkit/buildkitd.toml --allow-insecure-entitlement=network.host
  100 root      0:00 buildctl dial-stdio
  120 root      0:00 ps

$ docker buildx create --use
$ docker buildx inspect | grep "BuildKit daemon flags"
BuildKit daemon flags: --allow-insecure-entitlement=network.host

You are seeing --allow-insecure-entitlement=network.host because we always allow network.host within the container, see #2266 and

buildx/builder/builder.go

Lines 670 to 672 in 40f444f

    
           // set network.host entitlement if user does not provide any as 
        
           // network is isolated for container drivers. 
        
           res = append(res, "--allow-insecure-entitlement=network.host")

Looking at this I think we should display buildkitd config file content when using buildx inspect in debug mode.

dvdksn · 2024-09-11T09:03:28Z

@crazy-max I updated the configuration section. You're right that the buildkitd flag shows network.host entitlement only since it doesn't display the buildkitd config. However I'm still seeing issues with security.insecure when set using buildkitd.default.toml, even though it's showing in the container

$ cat ~/.docker/buildx/buildkitd.default.toml 
# debug enables additional debug logging
debug = true
# trace enables additional trace logging (very verbose, with potential performance impacts)
trace = true
# insecure-entitlements allows insecure entitlements, disabled by default.
insecure-entitlements = [ "network.host", "security.insecure" ]

[log]
  # log formatter: json or text
  format = "text"
$ docker buildx create --use --driver-opt "image=moby/buildkit:v0.16.0"
infallible_ardinghelli
$ cat docker-bake.hcl 
target default {
  entitlements = ["security.insecure", "network.host"]
}
$ docker buildx bake --allow security.insecure --allow network.host
[+] Building 4.4s (2/2) FINISHED                                         docker-container:infallible_ardinghelli
[...]
ERROR: failed to solve: granting entitlement security.insecure is not allowed by build daemon configuration
$ docker ps
CONTAINER ID   IMAGE                   COMMAND                  CREATED          STATUS          PORTS     NAMES
b186701244a7   moby/buildkit:v0.16.0   "buildkitd --config …"   31 seconds ago   Up 30 seconds             buildx_buildkit_infallible_ardinghelli0
4d160548c56d   moby/buildkit:master    "buildkitd --config …"   42 minutes ago   Up 42 minutes             buildx_buildkit_container0
$ docker exec b186701244a7 cat /etc/buildkit/buildkitd.toml
debug = true
insecure-entitlements = ["network.host", "security.insecure"]
trace = true

[log]
  format = "text"
$ docker exec b186701244a7 ps
PID   USER     TIME  COMMAND
    1 root      0:00 /sbin/docker-init -- buildkitd --config /etc/buildkit/buildkitd.toml --allow-insecure-entitlement=network.host
    7 root      0:00 buildkitd --config /etc/buildkit/buildkitd.toml --allow-insecure-entitlement=network.host
   91 root      0:00 buildctl dial-stdio
  128 root      0:00 ps

crazy-max · 2024-09-11T09:05:06Z

$ docker buildx bake --allow security.insecure --allow network.host
[+] Building 4.4s (2/2) FINISHED                                         docker-container:infallible_ardinghelli
[...]
ERROR: failed to solve: granting entitlement security.insecure is not allowed by build daemon configuration

Yes this might be a bug with entitlements as setting --buildkitd-flags "--allow-insecure-entitlement security.insecure" on builder creation is working.

dvdksn · 2024-09-11T09:09:26Z

yes, you're right, --buildkitd-flags works, but both the default config lookup e.g. ~/.docker/buildx/buildkitd.default.toml and --buildkitd-config don't seem to work with entitelements.

crazy-max · 2024-09-11T09:13:54Z

Ok looking at buildkit code we are overriding whatever is set in configuration if a flag is set: https://github.com/moby/buildkit/blob/3a7055008a5e58a2abbe0e0c21c919d9e014e062/cmd/buildkitd/main.go#L583-L584

I think we should merge in this case otherwise it would never take into account insecure entitlements from config as we are always setting network.host with container driver: #2266

I'm afraid this is "broken" since Buildx v0.13.0 😨

crazy-max · 2024-09-11T09:16:15Z

I think we should merge

Or we could check if network.host entitlement is set in conf ans skip the flag in this case. That would avoid a buildkit change.

crazy-max · 2024-09-11T10:31:52Z

closing this one as this is working as intended but will open a PR related to #2683 (comment)

dvdksn added the status/triage label Sep 11, 2024

crazy-max self-assigned this Sep 11, 2024

crazy-max added status/needs-investigation area/builder and removed status/triage labels Sep 11, 2024

crazy-max mentioned this issue Sep 11, 2024

inspect: display buildkit daemon configuration file #2684

Merged

crazy-max closed this as completed Sep 11, 2024

crazy-max removed the status/needs-investigation label Sep 11, 2024

crazy-max mentioned this issue Sep 11, 2024

builder: do not set network.host entitlement flag if already set in buildkitd conf #2685

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

buildkitd.default.toml is not taken into account #2683

buildkitd.default.toml is not taken into account #2683

dvdksn commented Sep 11, 2024 •

edited

Loading

crazy-max commented Sep 11, 2024 •

edited

Loading

dvdksn commented Sep 11, 2024

crazy-max commented Sep 11, 2024

dvdksn commented Sep 11, 2024

crazy-max commented Sep 11, 2024 •

edited

Loading

crazy-max commented Sep 11, 2024

crazy-max commented Sep 11, 2024

buildkitd.default.toml is not taken into account #2683

buildkitd.default.toml is not taken into account #2683

Comments

dvdksn commented Sep 11, 2024 • edited Loading

Contributing guidelines

I've found a bug and checked that ...

Description

Expected behaviour

Actual behaviour

Buildx version

Docker info

Builders list

Configuration

Build logs

Additional info

crazy-max commented Sep 11, 2024 • edited Loading

dvdksn commented Sep 11, 2024

crazy-max commented Sep 11, 2024

dvdksn commented Sep 11, 2024

crazy-max commented Sep 11, 2024 • edited Loading

crazy-max commented Sep 11, 2024

crazy-max commented Sep 11, 2024

dvdksn commented Sep 11, 2024 •

edited

Loading

crazy-max commented Sep 11, 2024 •

edited

Loading

crazy-max commented Sep 11, 2024 •

edited

Loading