can etcdctl support https when connecting gRPC proxy?

[xjas]

etcd &etcdctl that I used is version 3.2.23.

AFAK, proxy mode is only supported in API2. In proxy mode, I can use etcdctl as a client to test https& http connection, and it does work well.

But in my sitution, only API3 can be used, so I try to use gRPC proxy.

I just test the gRPC by following the guide, and http can work well. etcdctl is my client.

when I try the https mode in gRPC proxy, it does't work. There is no more guide I can find. I don't known how to do it.

And also, listen-addr argument in the command below seems have no explanation.
$ etcd grpc-proxy start --endpoints=infra0.example.com,infra1.example.com,infra2.example.com --listen-addr=127.0.0.1:2379

Three nodes are started as follow:
etcd --name=node2 --listen-client-urls=https://localhost:1279 --advertise-client-urls=https://localhost:1279 --listen-peer-urls=http://localhost:1280 --initial-advertise-peer-urls=http://localhost:1280 --initial-cluster=node1=http://localhost:1180,node2=http://localhost:1280,node3=http://localhost:1380 --initial-cluster-token=etcd-cluster-token --initial-cluster-state=new --cert-file=./etcd-node.pem --key-file=./etcd-node-key.pem --peer-cert-file=./etcd-node.pem --peer-key-file=./etcd-node-key.pem --trusted-ca-file=./ca.pem --peer-trusted-ca-file=./ca.pem --data-dir=./nodes/node2 --peer-client-cert-auth=true

Thank you.

[gyuho]

@SIRChen Have you tried configuring TLS flags for grpc-proxy?

[xjas]

@gyuho Yeah, I think the TLS flags have been configured. The flags are --cert,--cacert,--key, is it right?

Command I used is this:
etcd grpc-proxy start --cacert=./ca.pem --cert=./etcd-node.pem --key=./etcd-node-key.pem --endpoints=https://localhost:1379,https://localhost:1279,https://localhost:1179 --listen-addr=127.0.0.1:2379

[gyuho]

@SIRChen Yes. Did it work?

[xjas]

Sorry, it didn’t work. In fact, I have already done that when I used gRPC proxy at the beginning.

[gyuho]

@SIRChen Oh, I misread. Those flags are for servers. Can you try specifying grpc-proxy start --key,trusted-ca,cert-file flags instead, if you want to secure connection "to" gRPC proxy?

[xjas]

@gyuho I’m pretty sure that the flags you mentioned were configured on server side.

Sorry that my computer is not at hand. I will do that again tomorrow. If possible, I will record the process.

If you have any suggestions, I’m glad to listen and have a try.

[gyuho]

@SIRChen Yes, please try those flags. And let us know if it still doesn't work.

Reproducible steps would help.

[xjas]

@gyuho Sorry, these days I was engaged in other things.
I have tried as you said, but it still did not work.

Maybe I used it in a wrong way, so I paste my usage and the process.

First, 3 etcd nodes were started, configure were as follows.

node1:

etcd
--name=node1
--listen-client-urls=https://localhost:1179
--advertise-client-urls=https://localhost:1179
--listen-peer-urls=https://localhost:1180
--initial-advertise-peer-urls=https://localhost:1180
--initial-cluster=node1=https://localhost:1180,node2=https://localhost:1280,node3=https://localhost:1380
--initial-cluster-token=etcd-cluster-token
--initial-cluster-state=new
--cert-file=./etcd-node.pem
--key-file=./etcd-node-key.pem
--peer-cert-file=./etcd-node.pem
--peer-key-file=./etcd-node-key.pem
--trusted-ca-file=./ca.pem
--peer-trusted-ca-file=./ca.pem
--data-dir=./nodes/node1
--peer-client-cert-auth=true

node2:

etcd
--name=node2
--listen-client-urls=https://localhost:1279
--advertise-client-urls=https://localhost:1279
--listen-peer-urls=https://localhost:1280
--initial-advertise-peer-urls=https://localhost:1280
--initial-cluster=node1=https://localhost:1180,node2=https://localhost:1280,node3=https://localhost:1380
--initial-cluster-token=etcd-cluster-token
--initial-cluster-state=new
--cert-file=./etcd-node.pem
--key-file=./etcd-node-key.pem
--peer-cert-file=./etcd-node.pem
--peer-key-file=./etcd-node-key.pem
--trusted-ca-file=./ca.pem
--peer-trusted-ca-file=./ca.pem
--data-dir=./nodes/node2
--peer-client-cert-auth=true

node3:

etcd
--name=node3
--listen-client-urls=https://localhost:1379
--advertise-client-urls=https://localhost:1379
--listen-peer-urls=https://localhost:1380
--initial-advertise-peer-urls=https://localhost:1380
--initial-cluster=node1=https://localhost:1180,node2=https://localhost:1280,node3=https://localhost:1380
--initial-cluster-token=etcd-cluster-token
--initial-cluster-state=new
--cert-file=./etcd-node.pem
--key-file=./etcd-node-key.pem
--peer-cert-file=./etcd-node.pem
--peer-key-file=./etcd-node-key.pem
--trusted-ca-file=./ca.pem
--peer-trusted-ca-file=./ca.pem
--data-dir=./nodes/node3
--peer-client-cert-auth=true

Then, gRPC-proxy was started.

gRPC-proxy:
etcd grpc-proxy start --cacert=./ca.pem --cert=./etcd-node.pem --key=./etcd-node-key.pem --endpoints=https://localhost:1379,https://localhost:1279,https://localhost:1179 --listen-addr=127.0.0.1:2379

Finally, the usage is as follows.
http way, works. etcdctl --cacert=./ca.pem --cert=./etcd-proxy.pem --key=./etcd-proxy-key.pem --endpoints=http://127.0.0.1:2379 member list

https way, did't work. etcdctl --cacert=./ca.pem --cert=./etcd-proxy.pem --key=./etcd-proxy-key.pem --endpoints=https://127.0.0.1:2379 member list

And my cert is generated by cfssl, I just do it according to what THIS ARTICLE says.

Besides, I am indeed confused. As the tutorial says, one argument is --listen-addr=127.0.0.1:2379, which protocol is used is not specified, http or https?
Like the snapshot I post, http protocol does work, https does not.

I am looking forward to your reply, thanks.

[xjas]

@gyuho I read another document, and I got a message from TLS termination.

Terminate TLS from a secure etcd cluster with the grpc proxy by serving an unencrypted local endpoint.

Does it main https connection between gRPC-proxy and client is not supported for now?

[xjas]

@gyuho I have read another simple proxy mode called etcd gateway. etcd gateway can support https well, but it can not improve performance as the tutorial says.
Dose the https way of gRPC-proxy mode have a bug?
So, Can anyone have a better suggestion? I have been stuck for many days.

[gyuho]

@SIRChen grpc-proxy with TLS works well for me.

When you run etcdctl commands, did you send requests to the endpoint that you specified with etcd grpc-proxy start --advertise-client-url?

[xjas]

@gyuho Sorry to interrupt again. I tried as you said, etcd nodes were started as last time.
gRPC-proxy with TLS was started with a tiny difference, with --advertise-client-url argument added.
etcd grpc-proxy start --cacert=./ca.pem --cert=./etcd-node.pem --key=./etcd-node-key.pem --endpoints=https://localhost:1379,https://localhost:1279,https://localhost:1179 --listen-addr=127.0.0.1:2379 --advertise-client-url=https://127.0.0.1:23790

Usage is as follows, did't work. Maybe the configure is not right.

gPRC-proxy：

etcdctl client：

Also, I tried another way. Difference is I changed --advertise-client-url=https://127.0.0.1:23790 to --advertise-client-url=https://127.0.0.1:2379, however, it still did not work.

In fact, I'm confused by the 2 arguments: --listen-addr and --advertise-client-url. what's the differnce between them?

To be honest, I didn't find further explanation about the 2 arguments except that I got a similar message from --listen-addr of etcd gateway.

--listen-addr
Interface and port to bind for accepting client requests.
Default: 127.0.0.1:23790

In the end, As you said:

grpc-proxy with TLS works well for me.

Will you please post the process that you use gRPC-proxy, so I can compare the difference to find the wrong configure. Maybe I think it's more efficient.

Thanks for your patience, I expect the reply.

[gyuho]

@SIRChen I am currently busy with something else, but in the meantime, you can try the following way that I used to confirm:

###########################################

rm -f /tmp/cfssl* && rm -rf /tmp/certs && mkdir -p /tmp/certs

curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /tmp/cfssl
chmod +x /tmp/cfssl
sudo mv /tmp/cfssl /usr/local/bin/cfssl

curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /tmp/cfssljson
chmod +x /tmp/cfssljson
sudo mv /tmp/cfssljson /usr/local/bin/cfssljson

/usr/local/bin/cfssl version
/usr/local/bin/cfssljson -h

###########################################

mkdir -p /tmp/certs

cat > /tmp/certs/etcd-root-ca-csr.json <<EOF
{
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "O": "etcd",
      "OU": "etcd Security",
      "L": "San Francisco",
      "ST": "California",
      "C": "USA"
    }
  ],
  "CN": "etcd-root-ca"
}
EOF
cfssl gencert --initca=true /tmp/certs/etcd-root-ca-csr.json | cfssljson --bare /tmp/certs/etcd-root-ca

# verify
openssl x509 -in /tmp/certs/etcd-root-ca.pem -text -noout


# cert-generation configuration
cat > /tmp/certs/etcd-gencert.json <<EOF
{
  "signing": {
    "default": {
        "usages": [
          "signing",
          "key encipherment",
          "server auth",
          "client auth"
        ],
        "expiry": "87600h"
    }
  }
}
EOF

###########################################

cat > /tmp/certs/s1-ca-csr.json <<EOF
{
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "O": "etcd",
      "OU": "etcd Security",
      "L": "San Francisco",
      "ST": "California",
      "C": "USA"
    }
  ],
  "CN": "s1",
  "hosts": [
    "127.0.0.1",
    "localhost"
  ]
}
EOF
cfssl gencert \
  --ca /tmp/certs/etcd-root-ca.pem \
  --ca-key /tmp/certs/etcd-root-ca-key.pem \
  --config /tmp/certs/etcd-gencert.json \
  /tmp/certs/s1-ca-csr.json | cfssljson --bare /tmp/certs/s1

# verify
openssl x509 -in /tmp/certs/s1.pem -text -noout

cp -rf /tmp/certs ${HOME}/certs
find ${HOME}/certs

###########################################

make build
./bin/etcd --version
ETCDCTL_API=3 ./bin/etcdctl version

###########################################

rm -rf /tmp/etcd/s1

./bin/etcd --name s1 \
  --logger zap \
  --log-outputs stderr \
  --data-dir /tmp/etcd/s1 \
  --listen-client-urls https://localhost:2379 \
  --advertise-client-urls https://localhost:2379 \
  --listen-peer-urls https://localhost:2380 \
  --initial-advertise-peer-urls https://localhost:2380 \
  --initial-cluster s1=https://localhost:2380 \
  --initial-cluster-token tkn \
  --initial-cluster-state new \
  --client-cert-auth \
  --trusted-ca-file ${HOME}/certs/etcd-root-ca.pem \
  --cert-file ${HOME}/certs/s1.pem \
  --key-file ${HOME}/certs/s1-key.pem \
  --peer-client-cert-auth \
  --peer-trusted-ca-file ${HOME}/certs/etcd-root-ca.pem \
  --peer-cert-file ${HOME}/certs/s1.pem \
  --peer-key-file ${HOME}/certs/s1-key.pem

ETCDCTL_API=3 ./bin/etcdctl \
  --endpoints localhost:2379 \
  --cacert ${HOME}/certs/etcd-root-ca.pem \
  --cert ${HOME}/certs/s1.pem \
  --key ${HOME}/certs/s1-key.pem \
  endpoint health

./bin/etcd grpc-proxy start \
  --endpoints 127.0.0.1:2379 \
  --advertise-client-url 127.0.0.1:23790 \
  --cacert ${HOME}/certs/etcd-root-ca.pem \
  --cert ${HOME}/certs/s1.pem \
  --key ${HOME}/certs/s1-key.pem \
  --trusted-ca-file ${HOME}/certs/etcd-root-ca.pem \
  --cert-file ${HOME}/certs/s1.pem \
  --key-file ${HOME}/certs/s1-key.pem

ETCDCTL_API=3 ./bin/etcdctl \
  --endpoints 127.0.0.1:23790 \
  --cacert ${HOME}/certs/etcd-root-ca.pem \
  --cert ${HOME}/certs/s1.pem \
  --key ${HOME}/certs/s1-key.pem \
  endpoint health
###########################################

Please let me know if it still doesn't work.

[xjas]

@gyuho Thanks, version 3.3.8 does work. But I'm confused by some questions.
Version 3.23 did not work, and it is version 3.3.8 that did work. Was the grpc-proxy with https supported after 3.2.23?

In fact, I don't want to change the etcd version radical, but version 3.2.24 can be accepted. I have seen that version 3.2.24 is TBD, which seems to be released in July. Does it have a certain time?

As the process you provide, I do the test. In version 3.3.8, args --logger zap and --log-outputs maybe doesn't exist. So I remove those flags.

I don't mean any offense, but I think the flags are chaotic and they are not explained clearly. Much time is costed on discussed how to use these flags. As a matter of fact, I'm still confused.

It seems that the flag --advertise-client-url is useless. No matter what value I give to it, It still use the default 127.0.0.1:23790, Is this a bug?

1. I think the etcd node should have 3 sets of cert, one for member to member, one for member to proxy, and one for member to proxy.
2. grpc-proxy should have 2 sets, one for proxy to client and one for proxy to client.
3. Also, on client side, 2 sets of cert is a necessary.

Am I right?

[gyuho]

@SIRChen Completely agree that it's confusing to set up...

And I just remembered that the health endpoint is only available in 3.3 for gRPC proxy #8322.

Please see https://github.com/coreos/etcd/blob/master/CHANGELOG-3.3.md for more detailed change logs.

[xjas]

@gyuho Thanks very much for your help of these days. I think I'm understanding the cert configure gradually.

I have not fully understand the meaning.
I got a message from CHANGELOG-3.2.

Add grpc-proxy start --cert-file, grpc-proxy start --key-file and grpc-proxy start --trusted-ca-file flags.

Does this means that gRPC-proxy with tls is already supported in 3.2.24. Could you tell me the release plan of 3.2.24?

You said, the health endpoint is only available in 3.3 for gRPC proxy, you mean I can't do the normal endpoint health check just like what I do in a non gRPC proxy mode? If so, version 3.2.24 may not work properly as I expected.

To put it simple, if I want to use grpc-proxy completely normal, upgrading etcd version to 3.3 is a must?

[xjas]

@gyuho I just do some tests as I understand, and the conclusion is etcd may support endpoint health check. Or maybe I missed the point.

Test1: grpc-proxy connects with server with http

Test2:grpc-proxy connects with server with https

And CHANGELOG-3.2 said:

Add grpc-proxy start --cert-file, grpc-proxy start --key-file and grpc-proxy start --trusted-ca-file flags.

So I think version 3.2.24 may support https between grpc-proxy and client.

[gyuho]

To put it simple, if I want to use grpc-proxy completely normal, upgrading etcd version to 3.3 is a must?

Yes, if you want to check the status of grpc proxy server. Otherwise, the endpoint health request will be forwarded to core cluster.