"kubeadm init" doesn't accommodate an external front proxy CA #918
Comments
|
/kind feature |
k8s-ci-robot
added
kind/feature
|
/area security |
k8s-ci-robot
added
area/security
sig/cluster-lifecycle
|
/assign @liztio |
timothysc
added
kind/bug
liztio
added
the
lifecycle/active
|
Verified. Quick and easy repro: kubeadm init
cp -r /etc/kubernetes/pki ~
kubeadm reset
cp -r ~/pki/* /etc/kubernetes/pki
rm /etc/kubernetes/pki/front-proxy-ca.key
kubeadm init |
…certs Automatic merge from submit-queue (batch tested with PRs 68087, 68256, 64621, 68299, 68296). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md. Fixes using externally managed certs for kubeadm **What this PR does / why we need it**: The certificates overhaul caused a regression when using external certificates. This fixes that issue so external CAs no longer require a key if all certificates exist. Walk the certificate tree, at each step checking for a CACert. If the CACert is found, try to use it to generate certificates. Otherwise, generate a new CA cert. **Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*: Fixes kubernetes/kubeadm#918 **Special notes for your reviewer**: **Release note**: ```release-note External CAs can now be used for kubeadm with only a certificate, as long as all required certificates already exist. ```
|
Thank you, @liztio! I look forward to this in the next release. |
|
Trying this with version "v1.13-beta.0," I see the following: Here are the set of files: Note that the front-proxy-ca.crt file is there, and the certificate in front-proxy-client.crt has its companion key in front-proxy-client.key. |
|
Here we can see that the client certificate is signed by the CA, and that the private key matches the certificate: What else would you suggest that I investigate here? |
|
Pertinent discussion in the "kubeadm" channel of the "Kubernetes" Slack team: |
|
Trying release v1.13.0-beta.2 this morning, I see this new problem, which may be related to the reworking of the certificate phases: I've never needed the etcd CA certificate at that location before, given that my etcd is hosted externally. The API server etcd client files are there, together with the etcd CA certificate in file etcd-ca.crt, as always: If I copy the etcd-ca.crt file to etcd/ca.crt, then kubeadm init succeeds. If I move that file, so that there's no etcd-ca.crt file in place, kubeadm init fails its preflight check: @liztio, this second copy of the etcd CA certificate shouldn't be required, right? |
|
Oh, I see what's going on: kubeadm init now wants to find the etcd CA certificate file at etcd/ca.crt, but in my ClusterConfiguration file supplied to kubeadm init, the "etcd.external.caFile" field still mentions the previous etcd-ca.crt file. It looks like I have to migrate both of these together. I'll try that now and report back on how I fare. |
|
Well, fixing up that path now leads to this problem: I confirmed that the etcd CA certificate file on this master matches the CA certificate on all the etcd servers, and that the API server etcd client certificate is indeed signed with this CA certificate. Using these files with etcdctl works just fine. |
|
Sorry for the noise; I realized that my configuration change didn't make it through properly, and I had mistakenly set the external etcd CA certificate path to that of the master's CA certificate file. The right files are in place, but my configuration file is still messed up. |
|
I think I see the problem here. In cmd/kubeadm/app/cmd/phases/certs.go's If we're using an externally hosted etcd, then when we go looking for its CA certificate, we should honor the configuration's Perhaps the |
|
I filed a fresh issue: #1276. |
|
Yes, we can close this one. Thank you again for the fix. |
|
/milestone clear |
What keywords did you search in kubeadm issues before filing this one?
Is this a BUG REPORT or FEATURE REQUEST?
BUG REPORT
Versions
kubeadm version (use
kubeadm version):Environment:
kubectl version):AWS EC2
uname -a):Linux ip-10-103-0-214.ec2.internal 4.14.48-coreos-r1 #1 SMP Fri Jun 8 08:51:52 UTC 2018 x86_64 Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz GenuineIntel GNU/LinuxWhat happened?
I created all of my private keys and certificates ahead of time, and made them available to kubeadm init as follows:
Note that the CA key—with file name ca.key—is present, but while there is a front-proxy-ca.crt file with the front proxy CA's certificate, the corresponding private key is not present. Even though I am acting as an "external CA" here, I supplied the ca.key file to kubeadm init in order to allow it to generate the various KUBECONFIG files, which are onerous to generate "by hand." However, kubeadm init has no need for my front proxy CA private key here, as I've already generated the client certificate and its private key.
I run kubeadm init, and it fails because it can't find the front-proxy-ca.crt file.
What you expected to happen?
kubeadm init should recognize that I am operating an "external front proxy CA" here, and do as it does for etcd and the other master-related keys and certificates: confirm that the key and certificate present are valid and signed by the provided CA certificate, but otherwise leave the files alone and not generate any files to replace them.
How to reproduce it (as minimally and precisely as possible)?
Unfortunately, this is easier said than done, but I'll lay out the steps here:
front-proxy-ca.key(absent)Anything else we need to know?
We touched on this topic before with #752 and kubernetes/kubernetes#62643, but the latter patch did not fix this problem, because it was addressing the case where we're operating in "external CA mode."
What's missing from kubeadm for now is the notion of an "external front proxy CA mode." We shouldn't roll handling of the front proxy-related PKI assets together with the rest of the control plane PKI assets, because they have separate CAs. It's possible that one CA is external while another is not.
Handling this with separate sub-phases would clarify what's going on. We could determine whether we're operating in "external control plane CA mode," and if not, generate the following files:
Next, determine whether we're operating in "external front proxy CA mode," and if not, generate the following files:
It should be possible to have kubeadm step in for CA management for either or both of these sets of control plane and front proxy files.
Without this tolerance in place, I can't use kubeadm init, and I instead have to drop down to the kubeadm alpha phase subcommands, mimicking everything kubeadm init does except the certificate generation phase. That's a brittle approach, given the lack of stability threatened by the very name of the subcommand: "alpha."
Peripherally related, but for etcd: #807 and kubernetes/kubernetes#63806, so perhaps @detiber has thought through this problem before.
The text was updated successfully, but these errors were encountered: