Fix return path of NodePort traffic when using Calico network policy.

Previously, commit 2cce7de fixed the return path
of NodePort traffic when pods were on secondary ENIs.

However when using aws-vpc-cni together with Calico network policy, the fix that was introduced
in 2cce7de does not work, as Calico terminates the mangle table
rule traversal early and the CONNMARK rules put by AWS VPC CNI are never reached.

This PR configures Felix (part of calico) to RETURN inside iptables mangle table instead of ACCEPT, so that
the rules that existed in the mangle table after the calico ones get a chance to be executed.

config/v1.2/calico.yaml

@@ -63,6 +63,10 @@ spec:
             # Disable IPV6 on Kubernetes.
             - name: FELIX_IPV6SUPPORT
               value: "false"
+            # This will make Felix honor AWS VPC CNI's mangle table
+            # rules.
+            - name: FELIX_IPTABLESMANGLEALLOWACTION
+              value: Return
             # Wait for the datastore.
             - name: WAIT_FOR_DATASTORE
               value: "true"
@@ -393,6 +397,10 @@ spec:
             value: "1"
           - name: TYPHA_HEALTHENABLED
             value: "true"
+          # This will make Felix honor AWS VPC CNI's mangle table
+          # rules.
+          - name: FELIX_IPTABLESMANGLEALLOWACTION
+            value: Return
         volumeMounts:
         - mountPath: /etc/calico
           name: etc-calico

config/v1.3/calico.yaml

@@ -60,6 +60,10 @@ spec:
             # Set Felix endpoint to host default action to ACCEPT.
             - name: FELIX_DEFAULTENDPOINTTOHOSTACTION
               value: "ACCEPT"
+            # This will make Felix honor AWS VPC CNI's mangle table
+            # rules.
+            - name: FELIX_IPTABLESMANGLEALLOWACTION
+              value: Return
             # Disable IPV6 on Kubernetes.
             - name: FELIX_IPV6SUPPORT
               value: "false"
@@ -393,6 +397,10 @@ spec:
             value: "1"
           - name: TYPHA_HEALTHENABLED
             value: "true"
+          # This will make Felix honor AWS VPC CNI's mangle table
+          # rules.
+          - name: FELIX_IPTABLESMANGLEALLOWACTION
+            value: Return
         volumeMounts:
         - mountPath: /etc/calico
           name: etc-calico