deadlock when enabled cfn-signal on etcd nodes?

[redbaron]

It sees that there is no support of cfn-signal on etcd nodes or maybe I am doing something wrong?

deadlock goes like following:

• etcd-member is of type notify which is then fired when etcd server joins the cluster
• cfn-signal waits for etcd-member to become is-active which never happens until it reports readiness to systemd
• cloudformation waits for cfn-signal to fire before moving on to next etcd server

etcd-member can't join cluster as it first one and seems to wait for others to pop up, but it can't happen because cloudformation wont start next etcd until first one reports success

[camilb]

@redbaron It assign a EIP to your first node? Had something similar today, but it was because I forgot to add some parameters to VPC config. Do get this error in journald?

"Apr 10 13:05:30 ip-10-0-1-108.ec2.internal bash[1059]: run: discovery failed"

[mumoshu]

Good catch! It does support cfn-signal. etcdadm-reconfigure.service sets etcd-member's Type to simple or notify accordingly to the number of remaining nodes to be set up. If it fails to signal, maybe theres a bug in etcdadm 2017年4月11日(火) 7:18 Camil Blanaru <notifications@github.com>:

…

@redbaron <https://github.com/redbaron> It assign a EIP to your first node? Had something similar today, but it was because I forgot to add some parameters to VPC config. Do get this error in journald? "Apr 10 13:05:30 ip-10-0-1-108.ec2.internal bash[1059]: run: discovery failed" — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#525 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABV-YwDkUki-bB5KBz_Z6ZBVz69uI54ks5ruqq7gaJpZM4M5XbH> .

[redbaron]

@mumoshu , etcd.disasterRecovery.automated isn't enabled by default , therefore it disables etcdadm-reconfigure.service . I am porting our config to new version and we don't have this feature in existing one. Does it mean that this flag isn't really a flag and should always be enabled?

[mumoshu]

@redbaron As you've seen etcd.disasterRecovery.automated is disabled by default, but etcdadm-reconfigure.service. etcdadm-reconfigure.service is enabled as long as the specified etcd version is 3+, which is the default.

[redbaron]

@mumoshu , hmm, true.I'll investigate further

[redbaron]

@mumoshu , it sees that there is this line https://github.com/kubernetes-incubator/kube-aws/blob/dd345a7bacf74076883f4f8c21df90d18e6e1ab9/core/controlplane/config/templates/cloud-config-etcd#L154 which makes etcdadm-reconfigure.service to not start unless automation is enabled

[mumoshu]

@redbaron Wow!!! You are correct. It must be {{if .Etcd.DisasterRecovery.SupportsEtcdVersion .Etcd.Version -}}, too...

[mumoshu]

@redbaron It is now fixed in master! Would you mind confirming if it works for you?

[mumoshu]

Perhaps I couldn't notice the issue due to my insufficient test setup, which contained only one etcd node. To reproduce the issue, I believe I needed at least 3 or more, odd number of etcd nodes.