Unable to start secondary etcd nodes if initial cluster member is offline

[brandond]

Environmental Info:
K3s Version:
v1.21.7-rc2+k3s2

Node(s) CPU architecture, OS, and Version:
N/A

Cluster Configuration:
3 server managed etcd

Describe the bug:
Can't restart cluster nodes without the initial node being up

Steps To Reproduce:

1. Start 3 k3s servers
2. Wait 1-2 minutes for servers to finish initializing, then stop all nodes.
3. Start nodes 2 and 3 - note that they fail because the first node is not available:

Dec 15 00:29:51 ubuntu02.lan.khaus k3s[4614]: time="2021-12-15T00:29:51.457216141Z" level=info msg="Starting k3s v1.21.7-rc2+k3s2 (5260e4a6)"
Dec 15 00:29:51 ubuntu02.lan.khaus k3s[4614]: time="2021-12-15T00:29:51.460691299Z" level=info msg="Joining etcd cluster already initialized, forcing reconciliation"
Dec 15 00:29:51 ubuntu02.lan.khaus k3s[4614]: time="2021-12-15T00:29:51.463292745Z" level=fatal msg="starting kubernetes: preparing server: failed to get CA certs: Get \"https://ubuntu01.lan.khaus:6443/cacerts\": dial tcp 10.0.1.147:6443: connect: connection refused"

Expected behavior:
Once bootstrapped, additional nodes are able to start without needing to talk to the first server.

Actual behavior:
Servers depend on the first server being online and available in order to start.

Additional context / logs:

• We previously fixed this in After a complete power off: the server is currently unable to handle the request #2249 but it appears to have been regressed at some point. Confirmed that v1.21.7+k3s1 still does the right thing.

Backporting

• Needs backporting to older releases

[brandond]

Bisected this to the changes at 1055837#diff-4dcc46ade478259007245d87cf4eedb63ec9ebf0945746803e724cf1e8ee21a2R197-R198

This change forces the node to attempt to re-bootstrap from the 1st server, even if the node is already initialized. We need to ensure that already bootstrapped nodes have NO dependencies on other nodes when starting up.

[mdrahman-suse]

Verified in k3s on Release 1.23 using v1.23.1-rc1+k3s2 by running the steps mentioned and observed the behavior as expected