bug: APISIX start failed when one of ETCD nodes down

[chzhuo]

Issue description

I have three ETCD nodes

172.16.255.1
172.16.255.2
172.16.255.3

ETCD cluster work well when node 172.16.255.3 down

But APISIX start failed under the circumstances

Environment

apisix version: 2.8

Steps to reproduce

1

Actual result

2

Error log

3

Expected result

4

[tzssangglass]

if only configured 172.16.255.1 at APISIX, will it start properly?

[shuaijinchao]

When starting or restarting apisix through the cli tool, any node will exit the process directly when it is abnormal. Are you interested in fixing it ? @chzhuo

[shuaijinchao]

You can determine whether there is at least one healthy node after the etcd loop check is over to ensure that APISIX can run normally.

[spacewander]

The behavior is intended to make sure each configured node is working. So that we can avoid misconfigured nodes.

[spacewander]

I changed my mind. We can allow broken nodes to work around some unstable status in etcd.

[okaybase]

Production environment Case:
In our prod environment, we upgrade etcd cluster to v3.5, but we also met this case: etcd-io/etcd#12845 (comment), result to some etcd nodes down~
So, I also inclined to do some enhancement~

[shuaijinchao]

I'll fix this problem