Deleting a pod while cluster is red results in pod not being replaced.

[jordansissel]

Bug Report

What did you do?

Deleted an elasticsearch pod.

What did you expect to see?

The loss of a pod should be reconciled by replacing the lost pod.

What did you see instead? Under which circumstances?

• No replacement arrived
• The PVC was still Bound (?)

Controller logged 503 Service Unavailable inside the reconciler.

Circumstances:

• The cluster was "red" according to kubectl get elasticsearch.

Environment

• Version information:

  https://github.com/elastic/cloud-on-k8s/blob/pre-release/docs/k8s-quickstart.asciidoc

• Kubernetes information:

GKE master 1.11.8-gke.6, nodes 1.11.7-gke.12

• Resource definition:

kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 7.0.1
  nodes:
  - nodeCount: 3
    config:
      node.master: false
      node.data: true
      node.ingest: true
      node.ml: true
      xpack.license.self_generated.type: trial
    volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1T
        storageClassName: ssd 

  - nodeCount: 3
    config:
      node.master: true
      node.data: false
      node.ingest: false
      node.ml: false
      xpack.license.self_generated.type: trial
    volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 30G
        storageClassName: ssd 

• Logs:

I don't have logs from the session, but we did observe a reconciler error message reporting 503 Service Unavailable at the time of the issue.

[jordansissel]

I don't know how this situation happened. All the data above is mostly correlating, not determining a cause.

The pod, when deleted, did not return after waiting for 10-15 minutes. I gave up waiting and deleted the elasticsearch resource and recreated it fully.

[jordansissel]

% kubectl delete pods  quickstart-es-72772cthj2

% kubectl -n elastic-system logs -f statefulset.apps/elastic-operator

{"level":"info","ts":1557515518.7931888,"logger":"elasticsearch-controller","msg":"Start reconcile iteration","iteration":103,"request":"default/quickstart"}                                      
{"level":"info","ts":1557515518.794385,"logger":"kubebuilder.nodecerts","msg":"Loaded trust relationships","clusterName":"quickstart","count":0}
{"level":"info","ts":1557515518.7947376,"logger":"kubebuilder.nodecerts","msg":"Reconciling node certificate secrets"}
{"level":"info","ts":1557515519.4884264,"logger":"driver","msg":"Calculated all required changes","to_create:":0,"to_keep:":6,"to_delete:":0}
{"level":"info","ts":1557515519.4885433,"logger":"driver","msg":"Calculated performable changes","schedule_for_creation_count":0,"schedule_for_deletion_count":0}                                  
{"level":"info","ts":1557515519.4921703,"logger":"version","msg":"Update minimum master nodes","how":"api","currentMasterCount":3,"nextMasterCount":3,"minimum_master_nodes":2}                    
{"level":"info","ts":1557515549.6412835,"logger":"elasticsearch-controller","msg":"Updating status","iteration":103}
{"level":"info","ts":1557515549.6522202,"logger":"generic-reconciler","msg":"Aggregate reconcile result is {Requeue:false RequeueAfter:8735h56m41.201660804s}"}                                    
{"level":"info","ts":1557515549.6522813,"logger":"elasticsearch-controller","msg":"End reconcile iteration","iteration":103,"took":30.859099394,"request":"default/quickstart"}                    
{"level":"error","ts":1557515549.6523283,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"elasticsearch-controller","request":"default/quickstart","error":"503 Service Unavailable: ","errorCauses":[{"error":"503 Service Unavailable: unknown"}],"stacktrace":"github.com/elastic/cloud-on-k8s/operators/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/elastic/cloud-on-k8s/operators/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/elastic/cloud-on-k8s/operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/elastic/cloud-on-k8s/operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/elastic/cloud-on-k8s/operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/elastic/cloud-on-k8s/operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/elastic/cloud-on-k8s/operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/elastic/cloud-on-k8s/operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/cloud-on-k8s/operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/elastic/cloud-on-k8s/operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/cloud-on-k8s/operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/elastic/cloud-on-k8s/operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}

Where there should be 6 pods, there are now only 5:

% kubectl get pods
NAME                       READY   STATUS        RESTARTS   AGE
quickstart-es-8qxxwmk8rv   1/1     Running       0          4m
quickstart-es-bggprwhw5p   1/1     Running       0          4m
quickstart-es-pkl97kg267   1/1     Running       0          4m
quickstart-es-rg4fcl46wn   1/1     Running       0          4m
quickstart-es-w475kdsn2w   1/1     Running       0          4m

[nkvoll]

I think we're seeing more of these errors now than before, since our readiness health check is too permissive for several of the write-operations we do at the moment.

Ready = true whenever ES returns a successful HTTP response to / (previously it required having a master node). In larger clusters, especially during bootstrapping I'd expect this to cause us to take much longer to reach the end of the reconciliation loop. Perhaps the right thing to do here is to not send requests through the service, but target a pod specifically (which also has the side benefit of making it easier to argue about expected ES Rest API versions during upgrades).

When it comes to the ready check implementation, I think we also have to consider it as something the user might want to configure, as the ready check itself could depend on the specific use case (and node type). To reap particularly good benefits of this, additional smarts would have to exist on the Service(s) layer though (e,g routing reads to data nodes that are ready, even if they are partitioned from master nodes?). In any case, there are other semantically valid ready checks that could work, such as the Helm-chart one (Node has seen cluster state = green/yellow/red once, then whether it is responding to /), and the (possibly) slightly more permissive (Node has seen a master node once, then whether it is responding to /).

[sebgl]

This will be fixed by moving to StatefulSets: the StatefulSet controller will automatically recreate deleted pods with the same PVCs.

[pebrc]

I am closing this as we are moving to StatefulSets