Calico should override bootstrap iptables rules

[invidian]

Currently, we add bootstrap itpables rules in Packet here: https://github.com/kinvolk/lokomotive/blob/master/assets/lokomotive-kubernetes/packet/flatcar-linux/kubernetes/cl/controller.yaml.tmpl#L267, which apply to all interfaces.

Then we create Calico rules here: https://github.com/kinvolk/lokomotive/blob/master/assets/lokomotive-kubernetes/packet/flatcar-linux/kubernetes/calico-host-protection/templates/host-protection.yaml#L37.

However, modify Calico rules is not sufficient to open some ports, as bootstrap iptables rules are still enforced. This is confusing (e.g. see #120).

I think we should investigate that and make sure, that Calico rules always take precedence over bootstrap rules.

[rata]

This seems very weird, because we rely on calico rules to open port 80 and 443, for example. Are you saying it does work for these ports but not for others? Or calico rules don't really open ports at all?

[invidian]

It looks like I'm able to reach kubelet on port 10250 on controller node too by default 🤦‍♂️

[invidian]

Same with kube-proxy on port 10256.

[invidian]

This is now fixed by #201, but we still need to have some tests to prevent this from happening in the future.

[invidian]

Well, #201 fixes it only partially. The bootstrap rules are still not overridden by Calico and as they are broader, they still have an effect. E.g.:

 1325  314K ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:22

bootstrap rule is broader than following GNP:

apiVersion: crd.projectcalico.org/v1
kind: GlobalNetworkPolicy
metadata:
  name: ssh
spec:
  applyOnForward: true
  ingress:
  - action: Allow
    destination:
      ports:
      - 22
    protocol: TCP
    source:
      nets:
      - 1.1.1.1/16
  order: 0
  preDNAT: true
  selector: has(host-endpoint)

[rata]

@invidian that should be a regression too. I know for a fact that the calico ssh rules take precedence over the bootstrap iptables rules before the helm refactors. We are either testing incorrectly or regressed, quite sure :)

[rata]

Well, #201 fixes it only partially. The bootstrap rules are still not overridden by Calico and as they are broader, they still have an effect. E.g.:

Well, what is shown there is that the rule still exist. That doesn't really matter in iptables, if other rules reject the traffic before this rule is matched. Have you tested that too? That is the test we want to do, IMHO (so, my first bet regarding #137 (comment) is that we didn't test, just looked at the iptables output?)

We need to check whether it makes sense to override the rules per se or just want that calico enforces it's policies (i.e. the rules might be present, but after a -j DROP or similar added by calico).

I think we want that calico enforces it's policies, not sure about failure scenarios to decide weather to keep the rules or not.

[invidian]

Well, what is shown there is that the rule still exist. That doesn't really matter in iptables, if other rules reject the traffic before this rule is matched. Have you tested that too? That is the test we want to do, IMHO (so, my first bet regarding #137 (comment) is that we didn't test, just looked at the iptables output?)

Yes, I tested it, with Calico rules, I'm still able to SSH to nodes from any public IP address, while it should only allow me to SSH from the management CIDRs.

[invidian]

The PR with tests is up: #210. I've tried to test it using Prometheus metrics (hence #205), but I couldn't get it to work, so I decided to implement it using Calico Go Client.

[invidian]

The PR with tests is now merged (#210). I consider it done, as the only related work is tracked in #190.