Multi-Node/Clustered ETCD

[amshuman-kr]

Stories

Please provide stories that we plan to tackle:

• As operator ...
• As provider ...

Motivation

We should support provisioning and management of multi-node etcd clusters via etcd-druid to serve the following goals:

• Generally, better robustness, resilience, and high availability (HA)
• Zero downtime maintenance (ZDM)
• Non-disruptive scaling
• Single-zone outage/fault tolerance
• Ephemeral (or in-memory) persistence for better performance

Acceptance Criteria

• First increment (please break-down into different increments rather than by component, if possible)

  Tasks

  • Documentation/Proposal

    • Proposal to support multi-node etcd clusters #115
    • [Feature] Enhance multi-node ETCD proposal to handle backup health more explicitly #152
    • [Documentation] Need a design doc of scale-up feature. #496

  • Etcd-Druid

    • [Feature] Enhance the Etcd resource status structure as proposed in the multi-node proposal #168
    • [Feature] Modularize and enhance status management according to multi-node ETCD proposal #151
    • [Feature] Deploy/maintain the correct PodDisruptionBudget configuration according to the Etcd resource status conditions #165
    • Enabled jobs instead of cronjobs to take care of running compaction. #235
    • [Feature] Etcd cluster scale-up #349
    • [Feature] Enhance reconciliation to handle multi-node scenario in etcd-druid #158
    • [Feature] Validate decision table for reconciliation as mentioned in ETCD design doc #354
    • [Feature] List down all the metrics necessary for maintaining multinode  #340
    • [Feature] Expose the metrics fro multinode ETCD #358
    • [Feature] Enhance gardener grafana dashboards for multi-node #221
    • [Feature] To discuss the impact of PR #383(Add owner checks and taking of final snapshots) on Multi-Node Etcd #242
    • [Feature] Recover ETCD multinode cluster from transient quorum loss #436
    • [Feature] Handle permanent quorum loss scenario for ETCD multinode cluster #437
    • [Feature]Restore a ETCD member when data directory / PVC is corrupted #368
    • [Feature] Design doc on restoration of single member of a ETCD cluster when PVC is corrupted #371
    • [Feature] Introduce BackupReady condition #251
    • [Feature] Failed backups to not block incoming traffic and trigger high prio alert instead #147
    • [Feature] e2e tests for multi-node etcd #286

  • Etcd-Backup-Restore

    • [Feature] Make etcd-backup-restore Kubernetes-aware keep member and backup status up-to-date etcd-backup-restore#333
    • [Feature] Add leader-election to the backup-restore sidecar according to the multi-node ETCD proposal. etcd-backup-restore#321
    • [Feature]To take out of sync full/delta snapshot, delegate the incoming http request from backup Follower to the backup Leader. etcd-backup-restore#354
    • [Feature] Enhance initialisation sequence according to the multi-node ETCD proposal etcd-backup-restore#322
    • [Feature] Restoration handling for a single node in a multinode cluster. etcd-backup-restore#489
    • [Feature] Validate and analysis restoration handling in etcd-backup-restore according to the multi-node ETCD proposal etcd-backup-restore#323
    • [Feature] Readiness Probe in multi-node etcd  etcd-wrapper#7

  • Additional Improvements

    • Fix etcd image for tests etcd-backup-restore#280

Enhancement/Implementation Proposal (optional)

• https://github.com/gardener/etcd-druid/tree/master/docs/proposals/multi-node

[shreyas-s-rao]

/close as all sub-tasks are done. Umbrella issue for compaction is tracked by #610