Question: How can we secure VXLAN on a bare metal provider?

[alexellis]

Is your feature request related to a problem? Please describe.

Question

Describe the solution you'd like

How can we secure VXLAN on a bare metal provider?

Describe alternatives you've considered

I tried ufw when using Docker (not k3s) and it didn't work because the iptables rules clashed with uff which also uses iptables, in affect I'd blocked all traffic, but for port 22, but this was ignored.

Additional context

Unable to safely run k3s without a proper set of working firewall rules on clouds where "security groups" are unavailable, think of Packet as an example of that.

[alexellis]

cc @ibuildthecloud @erikwilson

[erikwilson]

In general you will want to open up cni0 to the pod cidr (as found at #24 (comment)), so on each node ufw allow in on cni0 from 10.42.0.0/16

That will allow a single node install to work. Then you will want to allow communications between nodes for multi-node installs. This will depend on each system but for my vagrant setup it is something like ufw allow in on eth1 from 10.135.135.0/24 where we can use a node cidr

That would allow ufw in vagrant to work where we have an eth0 interface which is public facing and eth1 which the nodes communicate over (in that sense we could just open eth1 to everything, but as a practical example lock ufw down to only the ips that need it).

Now it is up to you to ufw allow the ports that you are explicitly exposing from your services.

Hope that helps as a starting point, this should probably be documented at https://rancher.com/docs/k3s/latest/en/running/ maybe with commands for firewalld

[erikwilson]

I am also working on setting up ipsec and wireguard backends for flannel, so encrypted networking will be provided by default in the not too distant future.

[alexellis]

I found that when using Docker, the ufw rules were ignored. There are some blog posts and StackOverflow issues about this.

[erikwilson]

I tried this setup without docker on alpine and it worked, not sure what docker would be doing that is special. We can see from #776 that ufw is preventing some communication over cni0 (and also previously reported with firewalld)

[alexellis]

Various articles like: https://blog.viktorpetersson.com/2014/11/03/the-dangers-of-ufw-docker.html

I will try this on Packet to try to confirm if it's an issue with CNI and containerd.

[erikwilson]

Yuck. I am not sure how effective k3s is with ufw, it may depend on the order of service being brought up, will need more testing. ufw should be able to prevent communication over UDP 8472 for VXLAN security though, since there are not any iptables rules related specifically to that port.

[TheNicholasNick]

iptables and docker interactions are a docker thing, as this isn't docker it'll come down to how k3s interacts with iptables. Docker has params to disable iptables interaction as per the docs https://docs.docker.com/network/iptables/

[alexellis]

@SoreGums What about the --docker option for k3s, which doesn't use containerd? Surely that does use Docker?

What happens on Kubernetes, when iptables is disabled? It doesn't sound like a good idea, but I don't have enough information to say for sure.

[erikwilson]

Yep. k3s is not docker but it does use containerd which docker uses also. There are other ways to route traffic besides iptables, but k3s uses an iptables based approach.

[alexellis]

but it does use containerd which docker uses also.

If that's the case, then what happens when I install with the --docker installation flag? My assumption was that instead of using containerd, docker is used.

https://rancher.com/docs/k3s/latest/en/installation/

[erikwilson]

Yeah, that is correct we connect to the docker socket instead of creating a containerd instance and connecting to that. Running docker without iptables essentially means it will not alter the iptables rules but provides no other mechanism for routing, so you are basically forced to create your own rules from what I understand. I am not sure if the KUBE- rules are enough for routing in that case. It does appear that starting ufw before k3s is important for the iptables chain tho.

[erikwilson]

It appears any of the ports which are exposed in kubernetes through the load balancer will bypass the firewall. This is because we insert into the beginning of the iptables list. https://github.com/rancher/klipper-lb/blob/master/entry

It seems like using iptables based firewalls is a bad idea for this reason, unfortunately I don't know of a good alternative tho.

[erikwilson]

Sorry, looking closer it appears klipper-lb doesn't really have anything to do with this, the iptables it is applying should only be used in that container. iptables-save on the host shows entries for ufw, k8s iptables proxier, cni, and flannel. k3s doesn't really get involved with iptables for the most part other than trying to make sure everything is available and configured correctly. But any of these subcomponents can easily clobber a firewall config.

Anyhow, most firewalls should only be concerned about filtering, and most programs should only care about routing, but with iptables we get to throw them together and watch them compete. A firewall definitely should not let stuff silently bypass it tho, sort of defeats the purpose.

[ibuildthecloud]

I would assume ufw deny 8472/udp would be sufficient if all you care about is a single node setup. Multi node is impossible to secure with iptables AFAIK.

[runningman84]

Would it be possible to use wireguard and tell k3s to only use the wireguard interface for internal stuff?

Would it be possible to use the lo interface for the same purpose in a single node setup?

[alexellis]

I would assume ufw deny 8472/udp would be sufficient if all you care about is a single node setup. Multi node is impossible to secure with iptables AFAIK.

Most of these bare-metal providers give a public and private IP range, so could ufw deny 8472/udp be run against the public interface, leaving the agents to communicate over the private IP with udp?

[alexellis]

See the note here on "Open ports / Network security"

IMPORTANT. The VXLAN port on nodes should not be exposed to the world, it opens up your cluster network to accessed by anyone. Run your nodes behind a firewall/security group that disables access to port 8472.

https://libraries.io/github/rancher/k3s

[runningman84]

Sometimes you do not have an external firewall in an edge case or on a cheap vps. It would be coold if k3s could just limit on which interfaces (like only the wireguard) ist listens...

[brandond]

Closing due to age. On Slack, there was discussion about using wireguard instead of vxlan in environments that don't offer external firewalling.