Podman and Docker network problem on Fedora 41 - no internet access from rootfull podman containers #24486

robobario · 2024-11-06T23:10:51Z

Issue Description

This isn't a podman issue (I think), but it'd be good to confirm what the problem is and make it visible. My machine has both docker and podman installed, which had been working fine in Fedora 40.

I upgraded from Fedora 40 to Fedora 41 and rootfull podman containers appeared to lose internet access:

rootless:

podman run --rm -it fedora curl --max-time 5 https://wikipedia.org/
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://www.wikipedia.org/">here</a>.</p>
</body></html>

rootfull:

sudo podman run --rm -it fedora curl --max-time 5 https://wikipedia.org/
curl: (28) Connection timed out after 5002 milliseconds

I found that in Fedora 41 the netavark default has moved to nftables so my theory is the docker iptables config is clobbering things.

Docker appears to set a policy of DROP on the iptables FORWARD chain. If I set the policy to ACCEPT, then rootfull podman has internet access.

sudo iptables -P FORWARD ACCEPT
sudo podman run --rm -it fedora curl --max-time 5 https://wikipedia.org/
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://www.wikipedia.org/">here</a>.</p>
</body></html>

Steps to reproduce the issue

I reproduced this in a fresh VM

Steps to reproduce the issue

Install latest Fedora 41 ISO (I used Fedora-Workstation-Live-x86_64-41-1.4.iso with Gnome Boxes)
Follow Fedora instructions to install docker
sudo dnf-3 config-manager --add-repo https://download.docker.com/linux/fedora/docker-ce.repo
sudo dnf install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo systemctl start docker

At this point docker should have configured iptables like:

sudo iptables --list
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (1 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere

now rootfull podman will fail to communicate with the internet:

sudo podman run --rm -it fedora curl -m 5 https://wikipedia.com
curl: (28) Resolving timed out after 5000 milliseconds

Change the iptables FORWARD policy to ACCEPT and rootfull podman has internet access

robeyoun@fedora:~$ sudo iptables -P FORWARD ACCEPT
robeyoun@fedora:~$ sudo podman run --rm -it fedora curl -m 5 https://wikipedia.com
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.22.1</center>
</body>
</html>

Describe the results you received

When docker is installed, rootfull podman doesn't have internet access

Describe the results you expected

The dream would be docker and podman can coexist together out of the box. But given it's about how docker configures things maybe it's just something that could go in docs?

podman info output

host:
  arch: amd64
  buildahVersion: 1.37.5
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.12-3.fc41.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: '
  cpuUtilization:
    idlePercent: 92.26
    systemPercent: 1.78
    userPercent: 5.96
  cpus: 16
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: workstation
    version: "41"
  eventLogger: journald
  freeLocks: 2048
  hostname: robeyoun-thinkpadp16vgen1.rmtnz.csb
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 4210516
      size: 1
    - container_id: 1
      host_id: 165536
      size: 165536
    uidmap:
    - container_id: 0
      host_id: 4210516
      size: 1
    - container_id: 1
      host_id: 165536
      size: 165536
  kernel: 6.11.5-300.fc41.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 2172317696
  memTotal: 65971240960
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.13.0-1.fc41.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.13.0
    package: netavark-1.13.0-1.fc41.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.13.0
  ociRuntime:
    name: crun
    package: crun-1.18.1-1.fc41.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.18.1
      commit: c41f034fdbb9742c395085fc98459c94ad1f9aae
      rundir: /run/user/4210516/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20241030.gee7d0b6-1.fc41.x86_64
    version: |
      pasta 0^20241030.gee7d0b6-1.fc41.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/4210516/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-3.fc41.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.8.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.5
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 1h 0m 9.00s (Approximately 0.04 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /home/robeyoun/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/robeyoun/.local/share/containers/storage
  graphRootAllocated: 1022488809472
  graphRootUsed: 221470416896
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/4210516/containers
  transientStore: false
  volumePath: /home/robeyoun/.local/share/containers/storage/volumes
version:
  APIVersion: 5.2.5
  Built: 1729209600
  BuiltTime: Fri Oct 18 13:00:00 2024
  GitCommit: ""
  GoVersion: go1.23.2
  Os: linux
  OsArch: linux/amd64
  Version: 5.2.5

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

rpm -qa | grep docker
docker-ce-cli-27.3.1-1.fc41.x86_64
docker-ce-27.3.1-1.fc41.x86_64
docker-ce-rootless-extras-27.3.1-1.fc41.x86_64
docker-compose-plugin-2.29.7-1.fc41.x86_64
docker-buildx-plugin-0.17.1-1.fc41.x86_64

Additional information

No response

The text was updated successfully, but these errors were encountered:

Luap99 · 2024-11-07T10:23:44Z

Yeah that is really a docker issue not really a podman one IMO: https://docs.docker.com/engine/network/packet-filtering-firewalls/#docker-on-a-router

Also similar to described here https://fedoraproject.org/wiki/Changes/NetavarkNftablesDefault#How_To_Test you could switch back the driver to iptables.

I am not sure what a good place to document this would be? Where would you look for such info?

robobario · 2024-11-07T18:03:05Z

Thanks Paul, maybe this issue is enough to help other users find the problem.

Luap99 · 2024-11-08T18:12:22Z

Added it as known issue to my change request: https://fedoraproject.org/wiki/Changes/NetavarkNftablesDefault#Known_Issue_with_docker

So I think this should hopefully raise enough awareness

robobario · 2024-11-08T18:14:47Z

Thanks Paul

ItalyPaleAle · 2024-12-05T17:40:31Z

For people who may be landing here from Google...

I've had this issue with Fedora CoreOS too after updating to 41, since FCOS includes Docker and Podman by default.

Because I only use Podman, I found it simpler to just disable Docker entirely:

systemctl disable docker.service
systemctl mask docker.service

Using Ignition:

systemd:
  units:
    - name: docker.service
      enabled: false
      mask: true

robobario added the kind/bug Categorizes issue or PR as related to a bug. label Nov 6, 2024

Luap99 added the network Networking related issue or feature label Nov 7, 2024

Luap99 closed this as completed Nov 8, 2024

pfeileon mentioned this issue Nov 11, 2024

Native support for nftables docker/for-linux#1472

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Podman and Docker network problem on Fedora 41 - no internet access from rootfull podman containers #24486

Podman and Docker network problem on Fedora 41 - no internet access from rootfull podman containers #24486

robobario commented Nov 6, 2024 •

edited

Loading

Luap99 commented Nov 7, 2024

robobario commented Nov 7, 2024

Luap99 commented Nov 8, 2024

robobario commented Nov 8, 2024

ItalyPaleAle commented Dec 5, 2024

Podman and Docker network problem on Fedora 41 - no internet access from rootfull podman containers #24486

Podman and Docker network problem on Fedora 41 - no internet access from rootfull podman containers #24486

Comments

robobario commented Nov 6, 2024 • edited Loading

Issue Description

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

podman info output

Podman in a container

Privileged Or Rootless

Upstream Latest Release

Additional environment details

Additional information

Luap99 commented Nov 7, 2024

robobario commented Nov 7, 2024

Luap99 commented Nov 8, 2024

robobario commented Nov 8, 2024

ItalyPaleAle commented Dec 5, 2024

robobario commented Nov 6, 2024 •

edited

Loading