moby/buildkit:buildx-stable-1 not honoring custom network specification

[100kph]

Docker version 20.10.17, build 100c701
BuildX v0.9.1

Docker Desktop for Windows, integrated with WSL2, run under WSL2/Ubuntu 22.04.1 LTS

A custom network was created which has the properties listed below

[
    {
        "Name": "build-network",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.22.0.0/16",
                    "Gateway": "172.22.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Options": {},
        "Labels": {}
    }
]

A container is attached to this custom network. That container has an alias which is resolved by Docker networking. A new builder instance is created using docker buildx create --use --name mybuilder --driver docker-container --driver-opt network=build-network

Here are the containers shown in the custom network.

    "Containers": {
        "33e4ae21cfd1c757c0f02783a462bdda192acfd8ce3a8229e4dada6ca013a813": {
            "Name": "buildx_buildkit_mybuilder0",
            "EndpointID": "48781541f2008fe27731ae52a7eb7d5744d260e997f29672307fa15281f44a8e",
            "MacAddress": "02:42:ac:16:00:03",
            "IPv4Address": "172.22.0.3/16",
            "IPv6Address": ""
        },
        "3aa28c2a81c253b7e4516c5a72ba096a8dc9cc591e543a3c4f406b9d794fafa2": {
            "Name": "nuget-server",
            "EndpointID": "21dc8ba18d9cb253269e8ace17579a22b55efca634db2245dc36616c20f27602",
            "MacAddress": "02:42:ac:16:00:02",
            "IPv4Address": "172.22.0.2/16",
            "IPv6Address": ""
        }
    },

The DHCP assignment is working great, when I docker exec sh into to the builder instance, I can telnet/ping the network alias that is resolved by the containerd DNS. When I look into resolv.conf I see

# cat /etc/resolv.conf

nameserver 127.0.0.11
options ndots:0

BuildX/BuildKit

docker buildx use mybuilder
docker buildx inspect mybuilder --bootstrap
docker buildx bake --builder mybuilder -f docker-compose.yml -f docker-compose.build.yml --load --progress=plain --no-cache

During the buildx build, the buildkitsandbox cannot reach the network alias within the bridge network. The builder logs are saying

using host network as the default

found worker \"sogslmwr3a37nx3voaw492433\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:33e4ae21cfd1 org.mobyproject.buildkit.worker.network:host org.mobyproject.buildkit.worker.oci.process-mode:sandbox org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/amd64/v2 linux/amd64/v3 linux/amd64/v4 linux/arm64 linux/riscv64 linux/ppc64le linux/s390x linux/386 linux/mips64le linux/mips64 linux/arm/v7 linux/arm/v6]
skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist

No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]

The last one is coming from ResolveConf and then it's filling in default Google DNS, which shows up in the resolve.conf of the builder instance

 options ndots:0

 nameserver 8.8.8.8
 nameserver 8.8.4.4
 nameserver 2001:4860:4860::8888
 nameserver 2001:4860:4860::8844

Similar Challenges

Brandon's Observation

Issue #175 does not apply to me as I am going all in with buildkit the right way and using bake (see Tõnis's Stand, Tõnis's recommendation)

Conclusion

I really do not know where I am wrong. Could there be some undocumented way for buildx command line to instruct buildx to honor the network configuration set on the network attached to the builder?

This could be a game changer; we could have private package registries run on containers and we could use bridge to reach them from the build container context. All this without fear of reconfiguration of /etc/hosts when Network Changes/DHCP IP changes during assignment at the host. Or We could get configuration information at build time from a private vault in the bridge network

[tonistiigi]

Created moby/buildkit#3210 for tracking.