Faulty ip rules after forwarder restart

[ljkiraly]

Expected Behavior

The ip rule recovery should not affect routes belonging to other rules.

Current Behavior

Running NSM v1.9.0, in case of forwarder restart (not graceful restart) the policy recovery sometimes flush routing tables which belongs to other rule (connection).

Failure Information

When the nsm interface is deleted the routes which belongs to the link are removed, but not the rules which refers to these routes. At forwarder restart the creation order can change and that leads to faulty ip rules. During recovery some rule in NSC refer to empty table or to a table which belongs to other rule (which was created meantime). The flush operation always delete the routes without any check.

Steps to Reproduce

1. Start the example spire and basic setup
2. Apply the following patch to the examples/features/mutually-aware-nses:

git diff .
diff --git a/examples/features/mutually-aware-nses/config-file-nse-1.yaml b/examples/features/mutually-aware-nses/config-file-nse-1.yaml
index 5876961dc50..9faea04d35e 100644
--- a/examples/features/mutually-aware-nses/config-file-nse-1.yaml
+++ b/examples/features/mutually-aware-nses/config-file-nse-1.yaml
@@ -10,3 +10,9 @@ data:
       dstport: 6666
       routes:
         - prefix: 172.16.1.100/32
+    - from: 214.14.131.114/32
+      routes:
+        - prefix: 172.16.1.1/32
+    - from: 214.14.131.113/32
+      routes:
+        - prefix: 172.16.1.1/32
\ No newline at end of file
diff --git a/examples/features/mutually-aware-nses/config-file-nse-2.yaml b/examples/features/mutually-aware-nses/config-file-nse-2.yaml
index 0f08ea1ddfe..a098545d808 100644
--- a/examples/features/mutually-aware-nses/config-file-nse-2.yaml
+++ b/examples/features/mutually-aware-nses/config-file-nse-2.yaml
@@ -10,3 +10,9 @@ data:
       dstport: 5555
       routes:
         - prefix: 172.16.1.100/32
+    - from: 214.14.132.65/32
+      routes:
+        - prefix: 172.16.16.1/32
+    - from: 214.14.132.66/32
+      routes:
+        - prefix: 172.16.16.1/32
\ No newline at end of file

3. Apply the kustomization manifest.
4. Check the ip rules in NSC
5. Restart the forwarder (e.g. by killing the vpp application)
6. Check the ip rules in NSC

Failure Logs

Connection 7de77836-9123-443c-b84c-7ac601a8375d has the following policy set in IPContext:

"policies":[{"from":"214.14.131.113/32","routes":[{"prefix":"0.0.0.0/0","nextHop":"172.16.1.1"}]},{"from":"214.14.131.114/32","routes":[{"prefix":"0.0.0.0/0","nextHop":"172.16.1.1"}]}]}

Connection cfb49ee2-3f61-4e5c-b7e1-0b4c17cf7b05 has the following policy set in IPContext:

"policies":[{"from":"214.14.132.65/32","routes":[{"prefix":"0.0.0.0/0","nextHop":"172.16.16.1"}]},{"from":"214.14.132.66/32","routes":[{"prefix":"0.0.0.0/0","nextHop":"172.16.16.1"}]}]}

Configuration set in NSC:

# ip rule
0: from all lookup local
32758: from 214.14.131.114 lookup 4
32759: from 214.14.131.113 lookup 3
32760: from 214.14.132.66 lookup 2
32761: from 214.14.132.65 lookup 1
32766: from all lookup main
32767: from all lookup default
# ip route show table all
default via 172.16.16.1 dev nsm-1 table 1 onlink 
default via 172.16.16.1 dev nsm-1 table 2 onlink 
default via 172.16.1.1 dev nsm-0 table 3 onlink 
default via 172.16.1.1 dev nsm-0 table 4 onlink 

After forwarder restart the configuration has changed to a faulty one:

# ip rule
0: from all lookup local
32762: from 214.14.132.66 lookup 3
32763: from 214.14.131.114 lookup 2
32764: from 214.14.132.65 lookup 2
32765: from 214.14.131.113 lookup 1
32766: from all lookup main
32767: from all lookup default
# ip route show table all
default via 172.16.1.1 dev nsm-0 table 1 onlink
default via 172.16.1.1 dev nsm-0 table 2 onlink
default via 172.16.16.1 dev nsm-1 table 3 onlink

Slice from forwarder log:

Jun 27 07:02:47.934[37m [DEBU] [id:7de77836-9123-443c-b84c-7ac601a8375d] [DstPort:] [From:214.14.131.113/32] [IPProto:] [SrcPort:] [Table:3] [type:networkService] [0m(22.1)                        policy recovered
Jun 27 07:02:47.935[37m [DEBU] [id:7de77836-9123-443c-b84c-7ac601a8375d] [netlink:flushTable] [tableID:3] [type:networkService] [0m(22.2)                        completed
Jun 27 07:02:47.935[37m [DEBU] [id:7de77836-9123-443c-b84c-7ac601a8375d] [DstPort:] [From:214.14.131.113/32] [IPProto:] [SrcPort:] [duration:85.02µs] [netlink:RuleDel] [type:networkService] [0m(22.3)                        completed
Jun 27 07:02:47.935[37m [DEBU] [id:7de77836-9123-443c-b84c-7ac601a8375d] [DstPort:] [From:214.14.131.114/32] [IPProto:] [SrcPort:] [Table:4] [type:networkService] [0m(22.4)                        policy recovered
Jun 27 07:02:47.935[37m [DEBU] [id:cfb49ee2-3f61-4e5c-b7e1-0b4c17cf7b05] [DstPort:] [From:214.14.132.65/32] [IPProto:] [SrcPort:] [Table:1] [type:networkService] [0m(22.1)                        policy recovered
Jun 27 07:02:47.935[37m [DEBU] [id:7de77836-9123-443c-b84c-7ac601a8375d] [netlink:flushTable] [tableID:4] [type:networkService] [0m(22.5)                        completed
Jun 27 07:02:47.935[37m [DEBU] [id:7de77836-9123-443c-b84c-7ac601a8375d] [DstPort:] [From:214.14.131.114/32] [IPProto:] [SrcPort:] [duration:36.561µs] [netlink:RuleDel] [type:networkService] [0m(22.6)                        completed
Jun 27 07:02:47.935[37m [DEBU] [id:cfb49ee2-3f61-4e5c-b7e1-0b4c17cf7b05] [netlink:flushTable] [tableID:1] [type:networkService] [0m(22.2)                        completed
Jun 27 07:02:47.935[37m [DEBU] [id:cfb49ee2-3f61-4e5c-b7e1-0b4c17cf7b05] [DstPort:] [From:214.14.132.65/32] [IPProto:] [SrcPort:] [duration:66.424µs] [netlink:RuleDel] [type:networkService] [0m(22.3)                        completed
Jun 27 07:02:47.935[37m [DEBU] [id:cfb49ee2-3f61-4e5c-b7e1-0b4c17cf7b05] [DstPort:] [From:214.14.132.66/32] [IPProto:] [SrcPort:] [Table:2] [type:networkService] [0m(22.4)                        policy recovered
Jun 27 07:02:47.936[37m [DEBU] [id:cfb49ee2-3f61-4e5c-b7e1-0b4c17cf7b05] [netlink:flushTable] [tableID:2] [type:networkService] [0m(22.5)                        completed
Jun 27 07:02:47.936[37m [DEBU] [id:cfb49ee2-3f61-4e5c-b7e1-0b4c17cf7b05] [DstPort:] [From:214.14.132.66/32] [IPProto:] [SrcPort:] [duration:33.839µs] [netlink:RuleDel] [type:networkService] [0m(22.6)                        completed
Jun 27 07:02:47.936[37m [DEBU] [id:7de77836-9123-443c-b84c-7ac601a8375d] [netlink:getFreeTableID] [tableID:1] [type:networkService] [0m(22.7)                        completed
Jun 27 07:02:47.936[37m [DEBU] [id:7de77836-9123-443c-b84c-7ac601a8375d] [Dst:0.0.0.0/0] [Flags:4] [Gw:172.16.1.1] [Scope:universe] [Table:1] [duration:43.195µs] [link.Name:nsm-0] [netlink:RouteReplace] [type:networkService] [0m(22.8)                        completed
Jun 27 07:02:47.936[37m [DEBU] [id:7de77836-9123-443c-b84c-7ac601a8375d] [DstPort:] [From:214.14.131.113/32] [IPProto:] [SrcPort:] [Table:1] [duration:37.144µs] [netlink:RuleAdd] [type:networkService] [0m(22.9)                        completed
Jun 27 07:02:47.936[37m [DEBU] [id:cfb49ee2-3f61-4e5c-b7e1-0b4c17cf7b05] [netlink:getFreeTableID] [tableID:2] [type:networkService] [0m(22.7)                        completed
Jun 27 07:02:47.936[37m [DEBU] [id:7de77836-9123-443c-b84c-7ac601a8375d] [netlink:getFreeTableID] [tableID:2] [type:networkService] [0m(22.10)                        completed
Jun 27 07:02:47.936[37m [DEBU] [id:cfb49ee2-3f61-4e5c-b7e1-0b4c17cf7b05] [Dst:0.0.0.0/0] [Flags:4] [Gw:172.16.16.1] [Scope:universe] [Table:2] [duration:34.952µs] [link.Name:nsm-1] [netlink:RouteReplace] [type:networkService] [0m(22.8)                        completed
Jun 27 07:02:47.936[37m [DEBU] [id:7de77836-9123-443c-b84c-7ac601a8375d] [Dst:0.0.0.0/0] [Flags:4] [Gw:172.16.1.1] [Scope:universe] [Table:2] [duration:29.231µs] [link.Name:nsm-0] [netlink:RouteReplace] [type:networkService] [0m(22.11)                        completed
Jun 27 07:02:47.937[37m [DEBU] [id:cfb49ee2-3f61-4e5c-b7e1-0b4c17cf7b05] [DstPort:] [From:214.14.132.65/32] [IPProto:] [SrcPort:] [Table:2] [duration:22.821µs] [netlink:RuleAdd] [type:networkService] [0m(22.9)                        completed
Jun 27 07:02:47.937[37m [DEBU] [id:7de77836-9123-443c-b84c-7ac601a8375d] [DstPort:] [From:214.14.131.114/32] [IPProto:] [SrcPort:] [Table:2] [duration:21.043µs] [netlink:RuleAdd] [type:networkService] [0m(22.12)                        completed
Jun 27 07:02:47.937[37m [DEBU] [id:cfb49ee2-3f61-4e5c-b7e1-0b4c17cf7b05] [netlink:getFreeTableID] [tableID:3] [type:networkService] [0m(22.10)                        completed
Jun 27 07:02:47.937[37m [DEBU] [id:cfb49ee2-3f61-4e5c-b7e1-0b4c17cf7b05] [Dst:0.0.0.0/0] [Flags:4] [Gw:172.16.16.1] [Scope:universe] [Table:3] [duration:27.663µs] [link.Name:nsm-1] [netlink:RouteReplace] [type:networkService] [0m(22.11)                        completed
Jun 27 07:02:47.937[37m [DEBU] [id:cfb49ee2-3f61-4e5c-b7e1-0b4c17cf7b05] [DstPort:] [From:214.14.132.66/32] [IPProto:] [SrcPort:] [Table:3] [duration:23.743µs] [netlink:RuleAdd] [type:networkService] [0m(22.12)                        completed