Nodes fail to join the cluster when external db is used as backend

[ShylajaDevadiga]

Environmental Info:
K3s Version:
v1.19.9-rc1+k3s1

Node(s) CPU architecture, OS, and Version:
Ubuntu 20.04, amd64

Cluster Configuration:
3 server nodes 1 worker node

Describe the bug:
When external db is used as backend, second node fails to join the cluster. For testing mysql backend was used.

Note: This issue is not seen when embedded etcd is used as backend.

Steps To Reproduce:
Create a 3 node cluster with mysql backend.

Expected behavior:
All nodes should successfully join the cluster

Actual behavior:
First node comes up in Ready state, additional nodes fail to join the cluster

Additional context / logs:

Mar 30 16:20:00 ip-172-31-13-217 k3s[1770]: time="2021-03-30T16:20:00.776025704Z" level=error msg="CA cert validation failed: Get \"https://127.0.0.1:6443/cacerts\": x509: certificate signed by unknown authority"
Mar 30 16:20:01 ip-172-31-13-217 k3s[1770]: time="2021-03-30T16:20:01.056116853Z" level=info msg="Cluster-Http-Server 2021/03/30 16:20:01 http: TLS handshake error from <redacted>:34482: remote error: tls: bad certificate"
Mar 30 16:20:01 ip-172-31-13-217 k3s[1770]: time="2021-03-30T16:20:01.188276212Z" level=error msg="Failed to authenticate request from <redacted>:45266: x509: certificate signed by unknown authority"
Mar 30 16:20:01 ip-172-31-13-217 k3s[1770]: time="2021-03-30T16:20:01.679291438Z" level=info msg="Waiting for master node  startup: resource name may not be empty"

[davidnuzik]

PR: #3129

[ShylajaDevadiga]

Validated using v1.19.9-rc2+k3s1 on ubuntu20.04 an centos8.2. Cluster is up and nodes are joined correctly.

[rudimk]

I'm still facing this. I think the biggest difference here is that I'm running v1.20.6+k3s1 on Ubuntu 18.04, while @ShylajaDevadiga tested this on 20.04. I guess I'll try this on 20.04.

[brandond]

@rudimk this was a regression only present in a specific QA build of k3s; if you're experiencing something similar with a GA release then something else is going on. I would recommend you open a new issue if you're unable to solve your problem with 20.04.

[rudimk]

@brandond Yeah, I'm facing this with a GA release. I'm running my playbooks on a bunch of fresh 20.04 machines, should know in a few minutes if something's up.

[rudimk]

Yep. Same results. Posting a new issue.