Automatically resolve Control Plane nodes that have changed IP adresses.

[gnur]

After maintenance on the underlying virtualisation our Talos Omni cluster couldn't automatically get itself in a healthy state anymore, after some investigation it was determined that the IP leases in DHCP expired and now the workers and control plane nodes had their IP addresses mixed in a random way.

To resolve this we have to patch the CP nodes to use a static IP address (that is in the DHCP lease pool) and hope that the DHCP server is smart enough to not re-lease the IP addresses that are statically assigned.

Possible long solutions from within Omni:

• bootstrap etcd with hostnames that are resolved internally in omni so it can automatically recover from changing IP addresses.
• use the internal kubespan ipv6 address for etcd communication so underlying network changes don't matter.
• Warn about using DHCP for control plane nodes in some way so you can resolve this before you have maintenance that triggers this.

Possible more?

[smira]

Warn about using DHCP for control plane nodes in some way so you can resolve this before you have maintenance that triggers this.

DHCP might have static reservation on the DHCP server side, so we can't say if it's safe or not.

use the internal kubespan ipv6 address for etcd communication so underlying network changes don't matter.

KubeSpan was designed specifically to be an overlay network (so that you can turn it on and off without losing everything), so KubeSpan addresses are not used as etcd member addresses, Kubernetes node addresses, and so on.

bootstrap etcd with hostnames that are resolved internally in omni so it can automatically recover from changing IP addresses.

This might be possible (?).

Talos specifically builds etcd cluster around IP addresses, allows to select subnets to be used to support multihoming.

Having hostnames for members is way harder for most users.

The technical limitation is that etcd member database is subject to quorum requirements, so updating member's endpoint require quorum nodes to be healthy and connected. In a 3 node cluster, this means that if more than a single member changes its endpoint, the change can't be committed, as there is no quorum.

---

I would suggest to use static network configuration, as it avoids lots of problems and hard to debug issues.