etcd operator

[hexfusion]

https://github.com/openshift/enhancements/blob/master/enhancements/etcd/cluster-etcd-operator.md

Perf

• problems bootstrapping cluster @alaypatel07

Metal

• ipv4 works on 3/4 (asked in slack channel)
• ipv6 works on 3/4 (asked in slack channel)

IPv6

• azure is broken on pivot @deads2k
  1. bug 1804913: make ipv6 support a little better cluster-etcd-operator#173
• azure bootstrap is broken @hexfusion
  • Bug 1804960: Revert: render: fix listening addresses in IPv6 case cluster-etcd-operator#154
  • pkg/cmd/render: populate bootstrapIP from local interfaces cluster-etcd-operator#175
  • stop using invalid bootstrapIP from installer bug 1807169: use localhost for bootstrap IP until bootkube is fixed installer#3175 @deads2k
  • use correct bootstrapIP in static pods bug 1806723: use correct IP for bootstrap host env vars cluster-etcd-operator#200 @deads2k

bootstrapping

• keep bootstrap kube-apiserver up @deads2k
  1. bug 1805421: allow bootstrap server to terminate before removing etcd-bootstrap fr… cluster-etcd-operator#179
• real kube-apiserver to only use real etcd @deads2k
  1. bug 1805421: skip using bootstrap etcd cluster-kube-apiserver-operator#772

shutdown clusters don't start back up

• need to bypass the member lookup when there is no cluster
  • create a golang command so we can follow the logic. @deads2k bug 1805807: create logic for golang ETCD_INITIAL_CLUSTER etcd#28
  • use golang command in our static pods @deads2k bug 1805807: call the discovery-etcd-initial-cluster cluster-etcd-operator#185
  • update golang command to have a memory. - this isn't needed because we can use presence of data-dir to skip. Thanks @alaypatel07 and @retroflexer
  • Bug 1807054: fix recovery of master nodes restarting at the same time cluster-etcd-operator#203 @alaypatel07

blocker bugs

• port conflict @p0lyn0mial https://bugzilla.redhat.com/show_bug.cgi?id=1806579 - Bug 1806579: waits for ports before starting etcd member cluster-etcd-operator#226
• this is 4.2 to 4.3. Not part of this feature. cert problem found by wking @sttts is finding someone
• oc rsh doesn't auto-start with work etcdctl. this is needed for maintenance. - @sanchezl https://bugzilla.redhat.com/show_bug.cgi?id=1805981 Bug 1805981: oc rsh to an etcd pod needs to have etcdctl "just work"  cluster-etcd-operator#207
• upgrade failures where etcd appears to be unavailable @sanchezl https://bugzilla.redhat.com/show_bug.cgi?id=1807194

Tests we think should work

• All nodes being shut off at the same time and restarted. - tested by QE and appears to be working.
• IP address change of a single member @hexfusion
• debugging and detection when DNS information for one member is lost @sanchezl Bug 1809732: cluster-etcd-operator: upgrades show "etcd member is unavailable" cluster-etcd-operator#225. It degrades nicely, but we don't like degrading during upgrade. May end up switching to IPs
• Addition of a new member when there is significant etcd data. - verified by @alaypatel07 by adding a fourth member.
• Upgrade, downgrade, re-upgrade @alaypatel07

Tests we think should fail

• If one master is lost, instructions for how to.
  • create a new master that joins the cluster
  • removal of the old master from the cluster
• Restoring etcd when no members start up correctly
  • Changes to the etcd-quorum recovery steps
  • restore etcd from backup @retroflexer
• Make etcd consistent when one member needs to be removed. Ideally this becomes "remove etcd member, rollout". with Bug 1806751: Archive data-dir if target member is unstarted etcd#29 it may become that because we can auto-remove a bad data-dir. We may need to find a way to mark a node as "don't include in etcd"
  • Removal of a member from the etcd cluster @retroflexer (probably should be @alaypatel07)
  • Recovery of a member with a bad data-dir. @retroflexer
  • replace an existing member @hexfusion
• restore etcd when 2 masters are replaced. @hexfusion
• IP address change of all members

[retroflexer]

I am working on:

1. Restoring a cluster to previous state
2. etcd-quorum recovery steps
3. Removal of a member from etcd cluster
4. Recovery of a member with a bad data-dir

[retroflexer]

@deads2k I am not suresh gaikwad. My handle here is @retroflexer.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci-robot]

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.