dockerd now requires ip6tables to startup

[cpuguy83]

Description

With the code recently merged to master dockerd now requires ip6tables in order to start.
This is a new requirement that is almost certainly a breaking change as it even broke my vanilla Ubuntu 22.04 setup (w/ Azure kernel) since the ip6_tables module was not being loaded at startup.

Reproduce

N/A

Expected behavior

No response

docker version

N/A

docker info

N/A

Additional Info

No response

[thaJeztah]

cc @robmry @akerouanton

[akerouanton]

This is a new requirement that is almost certainly a breaking change as it even broke my vanilla Ubuntu 22.04 setup (w/ Azure kernel) since the ip6_tables module was not being loaded at startup.

We discussed that during the last libnet maintainers call. While this is a breaking change, it might not be as bad as it looks.

man 8 ip6tables says:

If the kernel is configured with automatic module loading, an attempt will be made to load the appropriate module for that table if it is not already there.

'automatic module loading' is already required for iptables. In your specific case, ip6tables failed to load the appropriate kernel module because it was executed in a DinD environment.

Starting with v27.0, users doing DinD with a host Docker daemon < v27.0 or with its ip6tables parameter turned off will likely face the same issue -- they'll need to disable ip6tables on the containerized daemon or manually load the kernel module on the host. For other users, this shouldn't break, except if the kernel wasn't compiled with ip6tables support (either as a separate module or directly integrated into the kernel image). I think that's fine.

Anyway, we'll improve error handling here to allow the daemon to start and instead barf out on network creation if users want an IPv6 network, ip6tables wasn't disabled and the kernel module isn't available.

[thaJeztah]

Is ip6tables here a separate dependency, and is this something we need to change in our packaging (add as Recommends: dependency)?

Also does this require changes in the official DIND image?

cc @tianon

[akerouanton]

No, I don't think we need to change anything in our packaging:

• Debian 11: iptables (provides both nft and legacy variants)
• Debian 12: iptables (provides both nft and legacy variants)
• CentOS 9: iptables (nft variant) or iptables-legacy
• Fedora 39: iptables-nft or iptables-legacy
• Fedora 40: couldn't check
• RHEL: couldn't check
• SLES: iptables or iptables-backend-nft
• Ubuntu 24.04: iptables (provides both nft and legacy variants), already required by docker-ce
• Ubuntu 23.10: iptables (provides both nft and legacy variants), already required by docker-ce
• Ubuntu 22.04: iptables (provides both nft and legacy variants), already required by docker-ce
• Ubuntu 20.04: iptables (provides both nft and legacy variants), already required by docker-ce
• Raspberry Pi OS: couldn't check

For the docker image, it already has the ip6tables package installed: https://github.com/docker-library/docker/blob/6ce7fe78a5a66dadf37e9226d8485e94e894814f/26/dind/Dockerfile#L17.

[tianon]

We also use a fake shim for modprobe to encourage the kernel to load modules directly instead of from userspace: https://github.com/docker-library/docker/blob/6ce7fe78a5a66dadf37e9226d8485e94e894814f/modprobe.sh (which usually works correctly)

We have had a lot of fun with nftables vs xtables (because it's impossible to detect the state of the host accurately from within a container, but if you guess wrong things won't work at best and will break the host at worst; search for nftables/iptables/xtables in our recent issues/PRs if you're curious about the details), but generally this shouldn't be any more broken for any of our users than they might've been before; to put that another way, anyone who is working today, should still be working after this change. 👍

[hellt]

Hi all,
I think I am facing this exact issue. I am the maintainer of containerlab.dev project and one of our deployment environments is GitHub codespaces.
When I switched to docker v27.3.1 I faced an issue that when running DinD in GitHub codespaces with docker v27.3.1 I can not create ip6tables rules. At the same time v26.1.5 works fine.

Does it look similar to the OP's question?

[robmry]

Hi @hellt ... the issues here should be fixed - the DinD container will try to trigger the host to load the ip6_tables module, and the Docker daemon will start even if the load fails. But, it could be something related...

How are you trying to create the rules, and where? Or, do you mean the inner dockerd won't create IPv6 enabled networks, because it can't create its ip6tables rules? What errors are you seeing?

[hellt]

thanks for coming back @robmry

My app is a Go wrapper around docker/moby with a dep on github.com/docker/docker v27.3.1+incompatible

In my app I create a docker network before spawning containers, the network uses both v4 and v6 addresses - Client.NetworkCreate()

I see the error when I run my app in GitHub Codespaces VM. In particular this is the error that I see:

INFO[0000] Creating docker network: Name="clab", IPv4Subnet="172.20.20.0/24", IPv6Subnet="3fff:172:20:20::/64", MTU=1500 
Error: Error response from daemon: Failed to Setup IP tables: Unable to enable NAT rule:  (iptables failed: ip6tables --wait -t nat -I POSTROUTING -s 3fff:172:20:20::/64 ! -o br-90dbe6ab6a57 -j MASQUERADE: ip6tables v1.8.9 (legacy): can't initialize ip6tables table `nat': Table does not exist (do you need to insmod?)
Perhaps ip6tables or your kernel needs to be upgraded.
 (exit status 3))

Note, that I do not get this error, if I use docker-ce 26.1.4.

Below are the info and version outputs captured from the DinD installation in Github Codespaces VM

docker version:

Client: Docker Engine - Community
 Version:           27.3.1
 API version:       1.47
 Go version:        go1.22.7
 Git commit:        ce12230
 Built:             Fri Sep 20 11:41:11 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.3.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.7
  Git commit:       41ca978
  Built:            Fri Sep 20 11:41:11 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.23
  GitCommit:        57f17b0a6295a39009d861b89e3b3b87b005ca27
 runc:
  Version:          1.1.14
  GitCommit:        v1.1.14-0-g2c9f560
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info:

Client: Docker Engine - Community
 Version:    27.3.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.19.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.7
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 27.3.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 57f17b0a6295a39009d861b89e3b3b87b005ca27
 runc version: v1.1.14-0-g2c9f560
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.5.0-1025-azure
 Operating System: Debian GNU/Linux 12 (bookworm) (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 7.743GiB
 Name: codespaces-66d686
 ID: b8369378-d99f-4e63-963e-b3a6d28ea4b6
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: codespacesdev
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[robmry]

Thanks @hellt ... if you're not enabling or disabling ip6tables explicitly, docker will be trying to create ip6tables rules for IPv6 bridge networks in 27.0 where it wasn't before, because we changed the default. If the host hasn't loaded the ip6_tables module it'll fail, and that's probably what you're seeing.

In that case, if you explicitly disable ip6tables in 27.x you should get the old behaviour. That's --ip6tables=false on the dockerd command line, or "ip6tables":false in /etc/docker/daemon.json. Alternatively, a modprobe ip6_tables might help.

(That's all a bit general because I've not played with Codespaces, so don't know what it looks like. It sounds like you've got access to a VM, not a container, and that VM has docker installed ... is that the version you've shown above? Then you're running a DinD image in the VM, and creating an IPv6 network inside the DinD image fails? Does creating an IPv6 enabled bridge network directly in the VM work?)

[hellt]

@robmry yes, I think my problem is that ip6_tables kernel module is simply not available in the GitHub Codespaces VM... But I might clearly lack the expertise to verify that.

If you want to check it out firsthand, then this link - https://github.com/codespaces/new?hide_repo_select=true&ref=test-codespaces-ipv6&repo=894548369&skip_quickstart=true&machine=basicLinux32gb&devcontainer_path=.devcontainer%2Fdocker-in-docker%2Fdevcontainer.json&geo=EuropeWest - will open up the github codespaces launch menu where you can click the region you want to boot a VM in and this will start building a CodeSpaces VM with DinD 27.3.1 in it.

It is free for all GitHub account holders and limited to 120 cpu-hours/mo

I am not sure if this is a "proper" VM, maybe a micro-vm. It uses the same infra as GitHub actions hosted runners.

Most likely I won't be ablet to respin the dockerd with the new config, since this is done by devcontainers directly.

Is there a way to disable ip6 rule setup from the API perspective?

[robmry]

Interesting, thank you! I haven't quite figured out how it all fits together ... but running sudo ip link show ip6_tables does load the ip6_tables module (there's no modprobe, this is the trick the DinD image uses to try to get modules loaded).

So, if you can persuade that command to run before the docker daemon starts via your devcontainer config, it should be ok - running it after dockerd starts is no good, because dockerd doesn't initialise its ip6tables properly.

Also, the entrypoint script /usr/local/share/docker-init.sh has an environment variable DOCKER_DEFAULT_IP6_TABLES, but it's set to empty. If there's a way to set it to --ip6tables=false, I think that'd give you the 26.x behaviour.

[hellt]

Thanks a lot @robmry
your last comment about ip6tables disablement led me to have a look at the dind devcontainer options https://github.com/devcontainers/features/tree/main/src/docker-in-docker#options and indeed they have an option to disable ip6tables (an awful thing to say in 2024 after 25+ years of ipv6, but, hey, this is on system guys now, not network folks)

I will see if at first we could fix the moby azure packages #48987 so that I can pin to 26.1.4 for some more months.

[robmry]

Ah, yes - thank you ... that does look like an option that'll end up setting the env-var. Loading the ip6_tables module would be the best fix, but I guess that's down to devcontainers/codespaces.

I'll ask around in Docker too, others will know a lot more about this stuff than I do.