ERROR: table `nat' is incompatible, use 'nft' tool.

[greenpau]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I use nftables; when starting a container I get:

ERRO[0000] Error adding network: failed to list chains: running [/usr/sbin/iptables -t nat -S --wait]: exit status 1: iptables v1.8.2 (nf_tables): table `nat' is incompatible, use 'nft' tool.

ERRO[0000] Error while adding pod to CNI network "podman": failed to list chains: running [/usr/sbin/iptables -t nat -S --wait]: exit status 1: iptables v1.8.2 (nf_tables): table `nat' is incompatible, use 'nft' tool.

Error: error configuring network namespace for container 51f6adbaed7d674fb4b48d501eb7ce0605d09e003ac09f6588b98dea7230ca9f: failed to list chains: running [/usr/sbin/iptables -t nat -S --wait]: exit status 1: iptables v1.8.2 (nf_tables): table `nat' is incompatible, use 'nft' tool.

Steps to reproduce the issue:

1. Create network configuration:

cat >/etc/cni/net.d/podman.conflist <<EOF
{
  "cniVersion": "0.4.0",
  "name": "podman",
  "plugins": [
    {
      "type": "bridge",
      "bridge": "cni-podman0",
      "isGateway": true,
      "ipMasq": true,
      "ipam": {
        "type": "host-local",
        "routes": [
          {
            "dst": "0.0.0.0/0"
          }
        ],
        "ranges": [
          [
            {
              "subnet": "192.168.124.0/24",
              "gateway": "192.168.124.1"
            }
          ]
        ]
      }
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      }
    },
    {
      "type": "firewall",
      "backend": "nftables"
    }
  ]
}
EOF

2. Pull fedora image and start a container:

podman pull fedora:latest
podman run -it fedora bash

Describe the results you received:

ERRO[0000] Error adding network: failed to list chains: running [/usr/sbin/iptables -t nat -S --wait]: exit status 1: iptables v1.8.2 (nf_tables): table `nat' is incompatible, use 'nft' tool.

ERRO[0000] Error while adding pod to CNI network "podman": failed to list chains: running [/usr/sbin/iptables -t nat -S --wait]: exit status 1: iptables v1.8.2 (nf_tables): table `nat' is incompatible, use 'nft' tool.

Error: error configuring network namespace for container 51f6adbaed7d674fb4b48d501eb7ce0605d09e003ac09f6588b98dea7230ca9f: failed to list chains: running [/usr/sbin/iptables -t nat -S --wait]: exit status 1: iptables v1.8.2 (nf_tables): table `nat' is incompatible, use 'nft' tool.

Describe the results you expected:

No errors.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman version 1.6.4

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.12.12
  podman version: 1.6.4
host:
  BuildahVersion: 1.12.0-dev
  CgroupVersion: v1
  Conmon:
    package: conmon-2.0.6-1.module+el8.1.1+5259+bcdd613a.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.6, commit: 6ffbb2ec70dbe5ba56e4bfde946fb04f19dd8bbf'
  Distribution:
    distribution: '"rhel"'
    version: "8.1"
  MemFree: 483997450240
  MemTotal: 540217061376
  OCIRuntime:
    name: runc
    package: runc-1.0.0-64.rc9.module+el8.1.1+5259+bcdd613a.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 10737414144
  SwapTotal: 10737414144
  arch: amd64
  cpus: 64
  eventlogger: journald
  kernel: 4.18.0-147.5.1.el8_1.x86_64
  os: linux
  rootless: false
  uptime: 662h 43m 16.45s (Approximately 27.58 days)
registries:
  blocked: null
  insecure: null
  search:
  - registry.redhat.io
  - registry.access.redhat.com
  - quay.io
  - docker.io
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 4
  GraphDriverName: overlay
  GraphOptions: {}
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 2
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.6.4-2.module+el8.1.1+5363+bf8ff1af.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):

Physical.

[mheon]

The nftables backend for iptables should work, even with nftables...
@mccv1r0 PTAL

[mccv1r0]

I can't recreate on fedora 30. The cni config uses backend nftables and nftables is being used by fedora 30.

$ sudo podman version 
Version:            1.7.0
RemoteAPI Version:  1
Go Version:         go1.12.14
OS/Arch:            linux/amd64
$ rpm -q podman 
podman-1.7.0-3.fc30.x86_64
$ 

What do you mean by The nftables backend for iptables should work, even with nftables...

The system should be running what the cni config specifies.

[greenpau]

The system should be running what the cni config specifies.

@mccv1r0 , why would the following clause trigger the execution of iptables and not nft?

      {
         "type": "firewall",
         "backend": "nftables"
      }

[mccv1r0]

CNI uses what the underlying OS is already running.

[greenpau]

CNI uses what the underlying OS is already running.

@mccv1r0 , in my case I don't run iptables. Only nftables. Should this issue belong to github.com/containernetworking/plugins?

[greenpau]

failed to list chains: running [/usr/sbin/iptables -t nat -S --wait]: exit status 1: iptables v1.8.2 (nf_tables): table `nat' is incompatible, use 'nft' tool.

The error is coming from here. Deep dive begins ...

[greenpau]

Opened a separate issue containernetworking/plugins#461.

[greenpau]

Upon research, it appears that cni is being implemented via vendor/github.com/cri-o/ocicni/pkg/ocicni/ocicni.go, which in turn uses github.com/containernetworking/cni/libcni

[greenpau]

@mheon , do you know why github.com/containernetworking/plugins/pkg/utils/iptables.go is not in libpod's vendor/ directory?

There is a series of dependencies when it comes to podman and CNI:

go get github.com/containernetworking/plugins@47a9fd80c825
go get github.com/cri-o/ocicni@d2881573038f
go get github.com/containernetworking/cni@4fae32b84921
go get github.com/coreos/go-iptables@f901d6c2a4f2

Unfortunately, I am getting when attempting to compile, the vendor/ directory gets only a subset of github.com/containernetworking/plugins files.

The files https://github.com/containers/libpod/tree/master/vendor/github.com/containernetworking/plugins/pkg/utils and https://github.com/containernetworking/plugins/tree/master/pkg/utils do not match.

[mheon]

Most likely because we're not invoking it directly?

My understanding is that CNI is packaged as a series of plugins - small binaries that are executed separately, each doing part of the job of network setup. The heavy lifting, including IPTables, likely occurs there. Podman sends along instructions for configuring the network, but we do not directly invoke the relevant code, but instead separate executables.

[greenpau]

My understanding is that CNI is packaged as a series of plugins - small binaries that are executed separately, each doing part of the job of network setup

@mheon , let me try recompiling the CNI plugins.

[greenpau]

@mheon , you are correct 👍 The plugins are being called indirectly.

Compiled github.com/containernetworking/plugins and added trace to pkg/utils/iptables.go

diff --git a/pkg/utils/iptables.go b/pkg/utils/iptables.go
index b38a2cd..3ec931f 100644
--- a/pkg/utils/iptables.go
+++ b/pkg/utils/iptables.go
@@ -19,6 +19,9 @@ import (
        "fmt"

        "github.com/coreos/go-iptables/iptables"
+
+       "github.com/davecgh/go-spew/spew"
+       "github.com/sirupsen/logrus"
 )

 const statusChainExists = 1
@@ -26,6 +29,11 @@ const statusChainExists = 1
 // EnsureChain idempotently creates the iptables chain. It does not
 // return an error if the chain already exists.
 func EnsureChain(ipt *iptables.IPTables, table, chain string) error {
+
+       //logrus.Errorf("EnsureChain() ipt: %s", spew.Sdump(ipt))
+       logrus.Errorf("EnsureChain() table: %s", spew.Sdump(table))
+       logrus.Errorf("EnsureChain() chain: %s", spew.Sdump(chain))
+
        if ipt == nil {
                return errors.New("failed to ensure iptable chain: IPTables was nil")
        }

As part of EnsureChain() function, there is ChainExists() check. The following arguments are being passed to it. It checks for the existence of the CNI-FORWARD chain.

ERRO[0000] EnsureChain() table: (string) (len=6) "filter"
ERRO[0000] EnsureChain() chain: (string) (len=11) "CNI-FORWARD"

That is where it fails.

[greenpau]

Another aspect of this issue. When I run iptables -t filter -S --wait it partially succeeds. However, when I do the same with root, then it fails.

$ iptables -t filter -S --wait
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
iptables: Permission denied (you must be root).

$ sudo iptables -t filter -S --wait
iptables v1.8.2 (nf_tables): table `filter' is incompatible, use 'nft' tool.

[greenpau]

@mccv1r0 , @mheon , this is not a bug. It is more of a feature request in the upstream repo, i.e. containernetworking/plugins. Closing this one. Thank you for being responsive! 👍 😃

[ratulb]

table `nft' is incompatible, use 'nft' tool - facing this issue while installing openstack via devstack on ubuntu20.04

[greenpau]

@ratulb , try using the plugins: https://github.com/greenpau/cni-plugins#getting-started

[elico]

@greenpau the simplest solution is to use another table name rather then nat or filter.
nftables actuall rules doesn't care about the table name and it uses a hook.
The main technical issue is that for compatibility it is allowed to create the filter/nat/raw/mangle tables directly via the nft tools.
If it's created by the nft tools instead of iptables I assume that there is some datastructure which is missing for iptables to run in parallel with nfttables.
(Again it's assumption rather then knowledge about a specific piece of code)
Simply naming/renaming the local nat table in nftables to main_nat or any other name will resolve the issue and will allow podman and probably also docker to run smoothly.

[egberts]

To get rid of that libvirt error, my permanent workaround in Debian 11 (as a host) with libvirtd daemon is to block the loading of iptables-related modules:

Create a file in /etc/modprobe.d/nft-only.conf:

#  Source: https://www.gaelanlloyd.com/blog/migrating-debian-buster-from-iptables-to-nftables/
#
blacklist x_tables
blacklist iptable_nat
blacklist iptable_raw
blacklist iptable_mangle
blacklist iptable_filter
blacklist ip_tables
blacklist ipt_MASQUERADE
blacklist ip6table_nat
blacklist ip6table_raw
blacklist ip6table_mangle
blacklist ip6table_filter
blacklist ip6_tables

libvirtd daemon now starts without any error.