Unexpected routing policy added in EC2 instance with multiple interfaces

[xdxu]

Bug report

With the recent change in #4799 we will now add additional ip rule for each interfaces with the default gateway in VPC network. However this literally break at least two of our production services. Take one of the services for example:

--- Instance1 ---> Instance2 ---> Internet
       |
       X
       |
    VPC GW

In the above topo, both instance1 and instance2 are AWS EC2 instances, and both have multiple NICs. Both instances are using cloud-init and netplan for network configuration.

We installed some static routes on instance1, so that any traffic initiated from instance1 is supposed to be routed via instance2 (for inspection) regardless of its source IP. However since we have the policy routing rules which has a higher priority than main table lookup, it will be routed via VPC default gateway directly instead.

In my opinion this type of service chaining is very common in the cloud environment, and cloud-init and netplan are widely used across various cloud vendors. The routing between these different services should be defined by the service owner, rather than by the automatically generated PBR rules.

Steps to reproduce the problem

As described in the above section, the problem can be easily produced. This was working with 23.4.4 cloud-init package but recently it is upgraded to 24.1.3 in Ubuntu 20.04.

Environment details

• Cloud-init version: 24.1.3
• Operating System Distribution: Ubuntu 20.04
• Cloud provider, platform or installer type: AWS EC2

cloud-init logs

Since the issue has been illustrated and it is easy to reproduce, I assume no logs are needed. But please let me know in case any additional information is needed.