Creating a cluster with the same id multiple times with enabled discovery service

[michaelkebe]

I am playing around with talos.

1. Create config files (controlplane.yaml, ...) with talosctl gen config .... Be sure discovery is enabled.
2. Boot a node into maintenance mode.
3. Start installation with talosctl apply-config -n 192.168.178.1 -f controlplane.yaml --insecure`
4. talosctl bootstrap to get etcd ready.
5. wait until everything is up an running.
6. reset the node and boot it back into maintenance mode.
7. Start installation with talosctl apply-config -n 192.168.178.1 -f controlplane.yaml --insecure` (with the same controlplane.yaml)
8. talosctl bootstrap to get etcd ready.
9. wait until everything is up an running.
10. talosctl health will not succeed because it disovers the same IP multiple times. The output of the first line looks something like this discovered nodes: ["192.168.178.79" "192.168.178.79"]

This renders talosctl health useless.

Is this normal behavior?
Is this a problem with the discovery service?

I used https://discovery.talos.dev/inspect to list the affiliates. After step 7. or 8. there is one affiliate more. You can repeat it multiple times to get more in the list.

If the discovery is disabled all is fine.

[smira]

Yes, it's expected, never create clusters sharing same PKI (secrets), this doesn't make any sense in general.

Long version is that you're not properly removing old cluster (with talosctl reset), so the discovery service will hold stale data up to TTL (30 mins).

But the easiest way out is to use fresh secrets every time you create a new cluster. See documentation on proper ways to manage your machine configuration.