Skip to content

Commit

Permalink
Revert "kubeadm: ensure leaf certs properly signed in CreateTree"
Browse files Browse the repository at this point in the history 
PR kubernetes#68296 got ahead of this.

This reverts:
- commit 17b4c19
- commit 2fd01a0

----------[corresponding planned issue]---------
TITLE: kubeadm HA setup: silently broken certificate chains issue, leading to failed etcd clustering

<!-- This form is for bug reports and feature requests ONLY!

If you're looking for help check [Stack Overflow](https://stackoverflow.com/questions/tagged/kubernetes) and the [troubleshooting guide](https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/).

If the matter is security related, please disclose it privately via https://kubernetes.io/security/.
-->

**Is this a BUG REPORT or FEATURE REQUEST?**:
/kind bug

**What happened**:
Following the [HA setup documentation](https://kubernetes.io/docs/setup/independent/high-availability/) and having done the part `5. Run the kubeadm phase commands to bootstrap the kubelet`, etcd starts complaining about the certificate issue, with the cluster health never be reaching ok.

   - CP0 (CP0_IP: 192.168.122.27)
   ```
     ----(snip)----
     2018-09-08 01:50:36.892124 I | etcdserver/membership: added member 10309eb7b886aec3 [https://192.168.122.30:2380] to cluster 50244b6b5ec9586c
     2018-09-08 01:50:36.892205 I | rafthttp: starting peer 10309eb7b886aec3...
     2018-09-08 01:50:36.892446 I | rafthttp: started HTTP pipelining with peer 10309eb7b886aec3
     2018-09-08 01:50:36.894198 I | rafthttp: started peer 10309eb7b886aec3
     2018-09-08 01:50:36.894236 I | rafthttp: added peer 10309eb7b886aec3
     2018-09-08 01:50:36.894429 I | rafthttp: started streaming with peer 10309eb7b886aec3 (writer)
     2018-09-08 01:50:36.894443 I | rafthttp: started streaming with peer 10309eb7b886aec3 (writer)
     2018-09-08 01:50:36.894474 I | rafthttp: started streaming with peer 10309eb7b886aec3 (stream MsgApp v2 reader)
     2018-09-08 01:50:36.894663 I | rafthttp: started streaming with peer 10309eb7b886aec3 (stream Message reader)
     2018-09-08 01:50:37.845351 W | etcdserver: failed to reach the peerURL(https://192.168.122.30:2380) of member 10309eb7b886aec3 (Get https://192.168.122.30:2380/version: dial tcp 192.168.122.30:2380: getsockopt: connection refused)
     2018-09-08 01:50:37.845466 W | etcdserver: cannot get the version of member 10309eb7b886aec3 (Get https://192.168.122.30:2380/version: dial tcp 192.168.122.30:2380: getsockopt: connection refused)
     2018-09-08 01:50:37.900635 W | raft: d9e6643beae8a97f stepped down to follower since quorum is not active
     2018-09-08 01:50:37.900702 I | raft: d9e6643beae8a97f became follower at term 2
     2018-09-08 01:50:37.900741 I | raft: raft.node: d9e6643beae8a97f lost leader d9e6643beae8a97f at term 2
     2018-09-08 01:50:38.901030 I | raft: d9e6643beae8a97f is starting a new election at term 2
     2018-09-08 01:50:38.901193 I | raft: d9e6643beae8a97f became candidate at term 3
     2018-09-08 01:50:38.901261 I | raft: d9e6643beae8a97f received MsgVoteResp from d9e6643beae8a97f at term 3
     2018-09-08 01:50:38.901319 I | raft: d9e6643beae8a97f [logterm: 2, index: 553] sent MsgVote request to 10309eb7b886aec3 at term 3
     /*
      * etcd on CP1 boots up, but its client cert is invalid..
      */
> 2018-09-08 01:50:39.673266 I | etcdmain: rejected connection from "192.168.122.30:41466" (error "tls: failed to verify client's certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"etcd-ca\")", ServerName "")
     2018-09-08 01:50:40.500956 I | raft: d9e6643beae8a97f is starting a new election at term 3
     2018-09-08 01:50:40.501314 I | raft: d9e6643beae8a97f became candidate at term 4
     2018-09-08 01:50:40.501751 I | raft: d9e6643beae8a97f received MsgVoteResp from d9e6643beae8a97f at term 4
     2018-09-08 01:50:40.505545 I | raft: d9e6643beae8a97f [logterm: 2, index: 553] sent MsgVote request to 10309eb7b886aec3 at term 4
> 2018-09-08 01:50:40.938004 I | etcdmain: rejected connection from "192.168.122.30:41468" (error "tls: failed to verify client's certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"etcd-ca\")", ServerName "")
     2018-09-08 01:50:41.801044 I | raft: d9e6643beae8a97f is starting a new election at term 4
     2018-09-08 01:50:41.801112 I | raft: d9e6643beae8a97f became candidate at term 5
     2018-09-08 01:50:41.801137 I | raft: d9e6643beae8a97f received MsgVoteResp from d9e6643beae8a97f at term 5
     2018-09-08 01:50:41.801221 I | raft: d9e6643beae8a97f [logterm: 2, index: 553] sent MsgVote request to 10309eb7b886aec3 at term 5
     ----(snip)----
```

  - CP1 (CP1_IP: 192.168.122.30)
   ```
     ----(snip)----
     2018-09-08 14:53:32.763499 W | etcdserver: could not get cluster response from https://192.168.122.27:2380: Get https://192.168.122.27:2380/members: remote error: tls: bad certificate
     2018-09-08 14:53:32.768630 I | etcdmain: rejected connection from "192.168.122.27:33996" (error  remote error: tls: bad certificate", ServerName "")
     2018-09-08 14:53:32.771694 C | etcdmain: cannot fetch cluster info from peer urls: could not retrieve cluster information from the given urls
     /*
      * and k8s_etcd_etcd dies..
      */
     ----(snip)----
```

This is because the scp-ed CA root certificate for etcd clustering is not going to be used when generating the daughter certificate so etcd in the first stacked control plane cannot verify it.

This problem is not confined to etcd, but all the other PKI assets which are expected to be signed by corresponding scp-ed CAs as well. `kubeadm init` and `kubeadm join` paths seem not to be affected.

In my opinion, there are two options we can take:
- ensure the capability to utilize the pre-installed CA with kubeadm certs command, to be consistent with the current documentation.
- log some warning or error message saying that the certificate chains has been broken with the command.

and I think the former is reasonable, so I will post PR for that later.

**What you expected to happen**:
etcd clustering successfully starts with properly setup certificates chains.

**How to reproduce it (as minimally and precisely as possible)**:
As mentioned in **What happened:** section above.

**Anything else we need to know?**:
Nothing in particular.

**Environment**:
- Kubernetes version (use `kubectl version`): v1.13.0-alpha.0.1072+2c933695fa61d5
- Cloud provider or hardware configuration: hardware configuration (qemu-kvm)
- OS (e.g. from /etc/os-release): fedora 28
- Kernel (e.g. `uname -a`): 4.19.0-rc2
- Install tools: kubeadm
- Others:
  • Loading branch information
Koichiro Den committed Sep 8, 2018
1 parent 17b4c19 commit 7acc4e0
Show file tree
Hide file tree
Showing 2 changed files with 61 additions and 106 deletions.
32 changes: 17 additions & 15 deletions cmd/kubeadm/app/phases/certs/certlist.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -103,29 +103,31 @@ type CertificateTree map[*KubeadmCert]Certificates
// CreateTree creates the CAs, certs signed by the CAs, and writes them all to disk.
func (t CertificateTree) CreateTree(ic *kubeadmapi.InitConfiguration) error {
for ca, leaves := range t {
caCert, caKey, err := LoadCertificateAuthority(ic.CertificatesDir, ca.BaseName)
cfg, err := ca.GetConfig(ic)
if err != nil {
cfg, err := ca.GetConfig(ic)
caCert, caKey, err = NewCACertAndKey(cfg)
if err != nil {
return err
}
err = writeCertificateAuthorithyFilesIfNotExist(
ic.CertificatesDir,
ca.BaseName,
caCert,
caKey,
)
if err != nil {
return err
}
return err
}

caCert, caKey, err := NewCACertAndKey(cfg)
if err != nil {
return err
}

for _, leaf := range leaves {
if err := leaf.CreateFromCA(ic, caCert, caKey); err != nil {
return err
}
}

err = writeCertificateAuthorithyFilesIfNotExist(
ic.CertificatesDir,
ca.BaseName,
caCert,
caKey,
)
if err != nil {
return err
}
}
return nil
}
Expand Down
135 changes: 44 additions & 91 deletions cmd/kubeadm/app/phases/certs/certlist_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -109,114 +109,67 @@ func TestMakeCertTree(t *testing.T) {
}

func TestCreateCertificateChain(t *testing.T) {
rootCACert := &KubeadmCert{
config: certutil.Config{},
Name: "test-ca",
BaseName: "test-ca",
}
daughterCert := &KubeadmCert{
config: certutil.Config{
AltNames: certutil.AltNames{
DNSNames: []string{"test-domain.space"},
},
Usages: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth},
dir, err := ioutil.TempDir("", t.Name())
if err != nil {
t.Fatal(err)
}
defer os.RemoveAll(dir)

ic := &kubeadmapi.InitConfiguration{
NodeRegistration: kubeadmapi.NodeRegistrationOptions{
Name: "test-node",
},
configMutators: []configMutatorsFunc{
setCommonNameToNodeName(),
ClusterConfiguration: kubeadmapi.ClusterConfiguration{
CertificatesDir: dir,
},
CAName: "test-ca",
Name: "test-daughter",
BaseName: "test-daughter",
}

table := []struct {
name string
order []Certificates
expectedErrors bool
}{
{
name: "newly create both CA and signed cert all at once",
order: []Certificates{
Certificates{rootCACert, daughterCert},
},
},
caCfg := Certificates{
{
name: "create a certificate signed by existing CA",
order: []Certificates{
Certificates{rootCACert},
Certificates{rootCACert, daughterCert},
},
config: certutil.Config{},
Name: "test-ca",
BaseName: "test-ca",
},
{
name: "ensure CA would not be regenerated signing existing daughter",
order: []Certificates{
Certificates{rootCACert, daughterCert},
Certificates{rootCACert},
config: certutil.Config{
AltNames: certutil.AltNames{
DNSNames: []string{"test-domain.space"},
},
Usages: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth},
},
},
{
name: "missing CA",
order: []Certificates{
Certificates{daughterCert},
configMutators: []configMutatorsFunc{
setCommonNameToNodeName(),
},
expectedErrors: true,
CAName: "test-ca",
Name: "test-daughter",
BaseName: "test-daughter",
},
}

for _, item := range table {
errors := []error{}
t.Run(item.name, func(t *testing.T) {
dir, err := ioutil.TempDir("", "test-create-certificate-chain")
if err != nil {
t.Fatal(err)
}
defer os.RemoveAll(dir)

ic := &kubeadmapi.InitConfiguration{
NodeRegistration: kubeadmapi.NodeRegistrationOptions{
Name: "test-node",
},
ClusterConfiguration: kubeadmapi.ClusterConfiguration{
CertificatesDir: dir,
},
}

for _, caCfg := range item.order {
certTree, err := caCfg.AsMap().CertTree()
if err != nil {
errors = append(errors, err)
}

if certTree.CreateTree(ic); err != nil {
errors = append(errors, err)
}
}
certTree, err := caCfg.AsMap().CertTree()
if err != nil {
t.Fatalf("unexpected error getting tree: %v", err)
}

if len(errors) == 0 {
caCert, _ := parseCertAndKey(path.Join(dir, "test-ca"), t)
daughterCert, _ := parseCertAndKey(path.Join(dir, "test-daughter"), t)
if certTree.CreateTree(ic); err != nil {
t.Fatal(err)
}

pool := x509.NewCertPool()
pool.AddCert(caCert)
caCert, _ := parseCertAndKey(path.Join(dir, "test-ca"), t)
daughterCert, _ := parseCertAndKey(path.Join(dir, "test-daughter"), t)

_, err = daughterCert.Verify(x509.VerifyOptions{
DNSName: "test-domain.space",
Roots: pool,
KeyUsages: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth},
})
if err != nil {
errors = append(errors, err)
}
}
pool := x509.NewCertPool()
pool.AddCert(caCert)

if len(errors) == 0 && item.expectedErrors {
t.Errorf("Expected errors, got no error")
}
if len(errors) > 0 && !item.expectedErrors {
t.Errorf("Got unexpected errors: %v", errors)
}
})
_, err = daughterCert.Verify(x509.VerifyOptions{
DNSName: "test-domain.space",
Roots: pool,
KeyUsages: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth},
})
if err != nil {
t.Errorf("couldn't verify daughter cert: %v", err)
}

}

func parseCertAndKey(basePath string, t *testing.T) (*x509.Certificate, crypto.PrivateKey) {
Expand Down

0 comments on commit 7acc4e0

Please sign in to comment.