CNI is trying to use iptables on CentOS 8

[ghost]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
Nearly the same issue as #5352. But that has been closed and I still do not know what was the cause or how I can prevent that.

I am using nftables (no firewalld) and I have a few nftables rules.
If I try to do podman run... it always fails with an iptables error message.

Steps to reproduce the issue:

1. Install Podman

2. Try to run any container using the default 'podman' network

3. Observe error message

4. Break down and cry :)

Describe the results you received:
Container should start.

Describe the results you expected:
Container does not start.

Additional information you deem important (e.g. issue happens only occasionally):
Error message:

ERRO[0000] Error adding network: failed to list iptables chains: running [/sbin/iptables -t filter -S --wait]: exit status 1: iptables v1.8.2 (nf_tables): table `filter' is incompatible, use 'nft' tool.

ERRO[0000] Error while adding pod to CNI network "podman": failed to list iptables chains: running [/sbin/iptables -t filter -S --wait]: exit status 1: iptables v1.8.2 (nf_tables): table `filter' is incompatible, use 'nft' tool.

Error: error configuring network namespace for container 93d79a257c409cbd53ca5d56c5cacfefadef5b4f8889dbe86e464e973940fcc4: failed to list iptables chains: running [/sbin/iptables -t filter -S --wait]: exit status 1: iptables v1.8.2 (nf_tables): table `filter' is incompatible, use 'nft' tool.

I have tried to change the /etc/cni/net.d/87-podman-bridge.conflist file as:

...
    {
      "type": "firewall",
      "backend": "nftables"
    },
...

I have also tried without the "backend" part and also with different "backend" entries: "iptables" or even "xxxxxxxxx". It always leads to the same error message. (Does this entry get ignored?)

Output of podman version:

podman version
Version:            1.8.2
RemoteAPI Version:  1
Go Version:         go1.12.12
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.12.12
  podman version: 1.8.2
host:
  BuildahVersion: 1.14.3
  CgroupVersion: v1
  Conmon:
    package: conmon-2.0.13-1.2.el8.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.13, commit: 68411cbc8d0a52079b13b9d3d1219d8205125965'
  Distribution:
    distribution: '"centos"'
    version: "8"
  MemFree: 7812005888
  MemTotal: 8359673856
  OCIRuntime:
    name: runc
    package: runc-1.0.0-15.2.el8.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc10
      commit: db2349efc4dc0001462089382d175ed38fdba742
      spec: 1.0.1-dev
  SwapFree: 2147479552
  SwapTotal: 2147479552
  arch: amd64
  cpus: 4
  eventlogger: journald
  hostname: < hidden >
  kernel: 4.18.0-147.5.1.el8_1.x86_64
  os: linux
  rootless: false
  uptime: 41m 32.45s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 2
  GraphDriverName: overlay
  GraphOptions:
    overlay.mountopt: nodev,metacopy=on
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  ImageStore:
    number: 24
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.8.2-1.1.el8.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):

[rhatdan]

This should be an issue with cni, not with podman.
https://github.com/containernetworking/plugins

[jamesdboone]

What's the fix for podman while waiting for the fix from CNI? Don't use on Red Hat 8?

[7underlines]

I still encounter this issue on RHEL 8.7 and can't figure out a workaround or how this was fixed from the mentioned links.
Should it be this one? containernetworking/plugins#461