Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

k8s versions e2e test are failing since Oct 15th #2134

Closed
pebrc opened this issue Nov 19, 2019 · 3 comments
Closed

k8s versions e2e test are failing since Oct 15th #2134

pebrc opened this issue Nov 19, 2019 · 3 comments
Labels
:ci Things related to Continuous Integration, automation and releases >test Related to unit/integration/e2e tests

Comments

@pebrc
Copy link
Collaborator

pebrc commented Nov 19, 2019

https://devops-ci.elastic.co/view/cloud-on-k8s/job/cloud-on-k8s-versions-gke/
@pebrc pebrc added :ci Things related to Continuous Integration, automation and releases >test Related to unit/integration/e2e tests labels Nov 19, 2019
@pebrc pebrc mentioned this issue Nov 25, 2019
Merged
@thbkrkr
Copy link
Contributor

thbkrkr commented Nov 26, 2019

TL;DR I did not find the root cause but I think I spotted something strange.

I picked one build that produced a dump and was not related to an endpoint not yet removed (#1927): https://devops-ci.elastic.co/blue/organizations/jenkins/cloud-on-k8s-versions-gke/detail/cloud-on-k8s-versions-gke/129/pipeline.

The failure of this build was produced in the TestUpdateESSecureSettings test due because the pod pod test-es-keystore-drf6-es-masterdata-0 was not ready:

[2019-11-25T01:29:45.651Z]     --- FAIL: TestUpdateESSecureSettings/Elasticsearch_secure_settings_should_eventually_be_set_in_all_nodes_keystore#03 (300.00s)
[2019-11-25T01:29:45.651Z]         utils.go:80: 
[2019-11-25T01:29:45.651Z]             	Error Trace:	utils.go:80
[2019-11-25T01:29:45.651Z]             	Error:      	Received unexpected error:
[2019-11-25T01:29:45.651Z]             	            	pod test-es-keystore-drf6-es-masterdata-0 is not Ready.

The e2e test creates/updates/deletes some secure settings which restarts a 3-node ES cluster. It seems that the last restart took too much time (timeout is set to 5min).

I correlated the build logs with the e2e test steps, the k8s pods events and the ES logs (to try to) highlight what's going on for each restart.

(Commands and result.) 
curl -s "https://devops-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/cloud-on-k8s-versions-gke/runs/129/nodes/33/log/?start=0" \
	| grep TestUpdateESSecureSettings \
	| sed "s/.*{/{/" \
	| jq -c '{"t":.Time,"a":.Test}' > e2e-test-steps.full.json

cat e2e-m9cld/e2e-m9cld-mercury/events.json \
	| jq '.items[]
	 	| select(.involvedObject.name | contains("test-es-keystore-drf6-es-masterdata"))
	    | {"t":.firstTimestamp,"m":.message,"n":.involvedObject.name,"z":.involvedObject.fieldPath}' -c | sort > pods-events.json

grep -hEr '(usable_space|"started")' e2e-m9cld \
	| jq '{"t":.timestamp,"m":.message,"n":.["node.name"]}' -c > es-logs.json

cat e2e-test-steps.json pods-events.json es-logs.json \
	| sort \
	| sed '/^.*keystore#/a # -- Restart' > all.json

Result:

{"t":"2019-11-25T01:16:15.533227773Z","a":"TestUpdateESSecureSettings"}
{"t":"2019-11-25T01:16:15.985848466Z","a":"TestUpdateESSecureSettings/Create_secure_settings_secret"}
{"t":"2019-11-25T01:16:16.007497312Z","a":"TestUpdateESSecureSettings/K8S_should_be_accessible"}
{"t":"2019-11-25T01:16:16.01911516Z","a":"TestUpdateESSecureSettings/Elasticsearch_CRDs_should_exist"}
{"t":"2019-11-25T01:16:16.023444414Z","a":"TestUpdateESSecureSettings/Remove_Elasticsearch_if_it_already_exists"}
{"t":"2019-11-25T01:16:16.03768347Z","a":"TestUpdateESSecureSettings/Creating_an_Elasticsearch_cluster_should_succeed"}
{"t":"2019-11-25T01:16:16.059202613Z","a":"TestUpdateESSecureSettings/Elasticsearch_cluster_should_be_created"}
{"t":"2019-11-25T01:16:16.068839946Z","a":"TestUpdateESSecureSettings/ES_certificate_authority_should_be_set_and_deployed"}
{"t":"2019-11-25T01:16:17Z","m":"create Claim elasticsearch-data-test-es-keystore-drf6-es-masterdata-0 Pod test-es-keystore-drf6-es-masterdata-0 in StatefulSet test-es-keystore-drf6-es-masterdata success","n":"test-es-keystore-drf6-es-masterdata","z":null}
{"t":"2019-11-25T01:16:17Z","m":"create Claim elasticsearch-data-test-es-keystore-drf6-es-masterdata-1 Pod test-es-keystore-drf6-es-masterdata-1 in StatefulSet test-es-keystore-drf6-es-masterdata success","n":"test-es-keystore-drf6-es-masterdata","z":null}
{"t":"2019-11-25T01:16:17Z","m":"create Claim elasticsearch-data-test-es-keystore-drf6-es-masterdata-2 Pod test-es-keystore-drf6-es-masterdata-2 in StatefulSet test-es-keystore-drf6-es-masterdata success","n":"test-es-keystore-drf6-es-masterdata","z":null}
{"t":"2019-11-25T01:16:17Z","m":"create Pod test-es-keystore-drf6-es-masterdata-0 in StatefulSet test-es-keystore-drf6-es-masterdata successful","n":"test-es-keystore-drf6-es-masterdata","z":null}
{"t":"2019-11-25T01:16:17Z","m":"create Pod test-es-keystore-drf6-es-masterdata-1 in StatefulSet test-es-keystore-drf6-es-masterdata successful","n":"test-es-keystore-drf6-es-masterdata","z":null}
{"t":"2019-11-25T01:16:17Z","m":"create Pod test-es-keystore-drf6-es-masterdata-2 in StatefulSet test-es-keystore-drf6-es-masterdata successful","n":"test-es-keystore-drf6-es-masterdata","z":null}
{"t":"2019-11-25T01:16:17Z","m":"waiting for first consumer to be created before binding","n":"elasticsearch-data-test-es-keystore-drf6-es-masterdata-0","z":null}
{"t":"2019-11-25T01:16:17Z","m":"waiting for first consumer to be created before binding","n":"elasticsearch-data-test-es-keystore-drf6-es-masterdata-1","z":null}
{"t":"2019-11-25T01:16:17Z","m":"waiting for first consumer to be created before binding","n":"elasticsearch-data-test-es-keystore-drf6-es-masterdata-2","z":null}
{"t":"2019-11-25T01:16:19.094977577Z","a":"TestUpdateESSecureSettings/All_expected_Pods_should_eventually_be_ready"}
{"t":"2019-11-25T01:16:21Z","m":"Successfully provisioned volume pvc-2e68f7ee-0f21-11ea-90f0-42010a84012d using kubernetes.io/gce-pd","n":"elasticsearch-data-test-es-keystore-drf6-es-masterdata-0","z":null}
{"t":"2019-11-25T01:16:21Z","m":"Successfully provisioned volume pvc-2e8196bc-0f21-11ea-90f0-42010a84012d using kubernetes.io/gce-pd","n":"elasticsearch-data-test-es-keystore-drf6-es-masterdata-1","z":null}
{"t":"2019-11-25T01:16:22Z","m":"Successfully assigned e2e-m9cld-mercury/test-es-keystore-drf6-es-masterdata-0 to gke-eck-gke12-129-e2e-default-pool-d388c97d-d23r","n":"test-es-keystore-drf6-es-masterdata-0","z":null}
{"t":"2019-11-25T01:16:22Z","m":"Successfully assigned e2e-m9cld-mercury/test-es-keystore-drf6-es-masterdata-1 to gke-eck-gke12-129-e2e-default-pool-4cd67ad4-jq93","n":"test-es-keystore-drf6-es-masterdata-1","z":null}
{"t":"2019-11-25T01:16:22Z","m":"Successfully assigned e2e-m9cld-mercury/test-es-keystore-drf6-es-masterdata-2 to gke-eck-gke12-129-e2e-default-pool-b68d125e-f9xr","n":"test-es-keystore-drf6-es-masterdata-2","z":null}
{"t":"2019-11-25T01:16:22Z","m":"Successfully provisioned volume pvc-2e8c8b2c-0f21-11ea-90f0-42010a84012d using kubernetes.io/gce-pd","n":"elasticsearch-data-test-es-keystore-drf6-es-masterdata-2","z":null}
{"t":"2019-11-25T01:16:28Z","m":"AttachVolume.Attach succeeded for volume \"pvc-2e8196bc-0f21-11ea-90f0-42010a84012d\" ","n":"test-es-keystore-drf6-es-masterdata-1","z":null}
{"t":"2019-11-25T01:16:30Z","m":"AttachVolume.Attach succeeded for volume \"pvc-2e68f7ee-0f21-11ea-90f0-42010a84012d\" ","n":"test-es-keystore-drf6-es-masterdata-0","z":null}
{"t":"2019-11-25T01:16:30Z","m":"AttachVolume.Attach succeeded for volume \"pvc-2e8c8b2c-0f21-11ea-90f0-42010a84012d\" ","n":"test-es-keystore-drf6-es-masterdata-2","z":null}
{"t":"2019-11-25T01:16:32Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:16:32Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:16:32Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:16:32Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:16:32Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:16:32Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:16:34Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.initContainers{elastic-internal-init-keystore}"}
{"t":"2019-11-25T01:16:34Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.initContainers{elastic-internal-init-keystore}"}
{"t":"2019-11-25T01:16:34Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.initContainers{elastic-internal-init-keystore}"}
{"t":"2019-11-25T01:16:35Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.initContainers{elastic-internal-init-keystore}"}
{"t":"2019-11-25T01:16:35Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.initContainers{elastic-internal-init-keystore}"}
{"t":"2019-11-25T01:16:35Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.initContainers{elastic-internal-init-keystore}"}
{"t":"2019-11-25T01:16:39Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:16:39Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:16:39Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:16:40Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:16:40Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:16:40Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:16:40Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:16:40Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:16:40Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:16:42Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.initContainers{elastic-internal-init-keystore}"}
{"t":"2019-11-25T01:16:42Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.initContainers{elastic-internal-init-keystore}"}
{"t":"2019-11-25T01:16:43Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.initContainers{elastic-internal-init-keystore}"}
{"t":"2019-11-25T01:16:46Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:16:46Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:16:47Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:16:51Z","m":"Readiness probe failed: ","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:16:51Z","m":"Readiness probe failed: ","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:16:59Z","m":"Readiness probe failed: ","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:17:08.113681103Z","a":"TestUpdateESSecureSettings/ES_version_should_be_the_expected_one"}
{"t":"2019-11-25T01:17:08.123203549Z","a":"TestUpdateESSecureSettings/ES_services_should_be_created"}
{"t":"2019-11-25T01:17:08.131943417Z","a":"TestUpdateESSecureSettings/ES_pods_should_eventually_have_a_certificate"}
{"t":"2019-11-25T01:17:08.169539304Z","a":"TestUpdateESSecureSettings/ES_services_should_have_endpoints"}
{"t":"2019-11-25T01:17:08.175919375Z","a":"TestUpdateESSecureSettings/ES_cluster_health_should_eventually_be_green"}
{"t":"2019-11-25T01:17:32.244402603Z","a":"TestUpdateESSecureSettings/Elastic_password_should_be_available"}
{"t":"2019-11-25T01:17:32.247382046Z","a":"TestUpdateESSecureSettings/Elasticsearch_data_volumes_should_be_of_the_specified_type"}
{"t":"2019-11-25T01:17:32.252978995Z","a":"TestUpdateESSecureSettings/ES_nodes_topology_should_eventually_be_the_expected_one"}
{"t":"2019-11-25T01:17:32.404940348Z","a":"TestUpdateESSecureSettings/ES_version_should_be_the_expected_one#01"}
{"t":"2019-11-25T01:17:32.537346751Z","a":"TestUpdateESSecureSettings/ES_endpoint_should_eventually_be_reachable"}
{"t":"2019-11-25T01:17:32.564487194Z","a":"TestUpdateESSecureSettings/Elasticsearch_secure_settings_should_eventually_be_set_in_all_nodes_keystore"}
{"t":"2019-11-25T01:17:34.145542143Z","a":"TestUpdateESSecureSettings/Modify_secure_settings_secret"}
# -- Restart
{"t":"2019-11-25T01:17:34.152834101Z","a":"TestUpdateESSecureSettings/Elasticsearch_secure_settings_should_eventually_be_set_in_all_nodes_keystore#01"}
{"t":"2019-11-25T01:17:36Z","m":"Killing container with id docker://elasticsearch:Need to kill Pod","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:17:37Z","m":"Successfully assigned e2e-m9cld-mercury/test-es-keystore-drf6-es-masterdata-2 to gke-eck-gke12-129-e2e-default-pool-b68d125e-f9xr","n":"test-es-keystore-drf6-es-masterdata-2","z":null}
{"t":"2019-11-25T01:17:47Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:17:47Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.initContainers{elastic-internal-init-keystore}"}
{"t":"2019-11-25T01:17:47Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:17:47Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.initContainers{elastic-internal-init-keystore}"}
{"t":"2019-11-25T01:17:47Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:17:47Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.initContainers{elastic-internal-init-keystore}"}
{"t":"2019-11-25T01:17:52Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:17:52Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:17:52Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:18:05Z","m":"Readiness probe failed: ","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:18:17Z","m":"Killing container with id docker://elasticsearch:Need to kill Pod","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:18:19Z","m":"Successfully assigned e2e-m9cld-mercury/test-es-keystore-drf6-es-masterdata-1 to gke-eck-gke12-129-e2e-default-pool-4cd67ad4-jq93","n":"test-es-keystore-drf6-es-masterdata-1","z":null}
{"t":"2019-11-25T01:18:33Z","m":"AttachVolume.Attach succeeded for volume \"pvc-2e8196bc-0f21-11ea-90f0-42010a84012d\" ","n":"test-es-keystore-drf6-es-masterdata-1","z":null}
{"t":"2019-11-25T01:18:36Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:18:36Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:18:36Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:18:39Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.initContainers{elastic-internal-init-keystore}"}
{"t":"2019-11-25T01:18:39Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.initContainers{elastic-internal-init-keystore}"}
{"t":"2019-11-25T01:18:39Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.initContainers{elastic-internal-init-keystore}"}
{"t":"2019-11-25T01:18:44Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:18:44Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:18:44Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:18:57Z","m":"Readiness probe failed: ","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:19:09Z","m":"Killing container with id docker://elasticsearch:Need to kill Pod","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:19:11Z","m":"Successfully assigned e2e-m9cld-mercury/test-es-keystore-drf6-es-masterdata-0 to gke-eck-gke12-129-e2e-default-pool-d388c97d-d23r","n":"test-es-keystore-drf6-es-masterdata-0","z":null}
{"t":"2019-11-25T01:19:20Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:19:20Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:19:20Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:19:21Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.initContainers{elastic-internal-init-keystore}"}
{"t":"2019-11-25T01:19:21Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.initContainers{elastic-internal-init-keystore}"}
{"t":"2019-11-25T01:19:21Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.initContainers{elastic-internal-init-keystore}"}
{"t":"2019-11-25T01:19:25Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:19:25Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:19:25Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:19:35Z","m":"Readiness probe failed: ","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:19:48.098141339Z","a":"TestUpdateESSecureSettings/Remove_one_of_the_source_secrets"}
# -- Restart
{"t":"2019-11-25T01:19:48.109082792Z","a":"TestUpdateESSecureSettings/Elasticsearch_secure_settings_should_eventually_be_set_in_all_nodes_keystore#02"}
{"t":"2019-11-25T01:19:49Z","m":"Killing container with id docker://elasticsearch:Need to kill Pod","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:19:52Z","m":"Failed to update endpoint e2e-m9cld-mercury/test-es-keystore-drf6-es-masterdata: Operation cannot be fulfilled on endpoints \"test-es-keystore-drf6-es-masterdata\": the object has been modified; please apply your changes to the latest version and try again","n":"test-es-keystore-drf6-es-masterdata","z":null}
{"t":"2019-11-25T01:19:52Z","m":"Successfully assigned e2e-m9cld-mercury/test-es-keystore-drf6-es-masterdata-2 to gke-eck-gke12-129-e2e-default-pool-b68d125e-f9xr","n":"test-es-keystore-drf6-es-masterdata-2","z":null}
{"t":"2019-11-25T01:19:57Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:19:57Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:19:58Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.initContainers{elastic-internal-init-keystore}"}
{"t":"2019-11-25T01:19:58Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.initContainers{elastic-internal-init-keystore}"}
{"t":"2019-11-25T01:19:58Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:19:59Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.initContainers{elastic-internal-init-keystore}"}
{"t":"2019-11-25T01:20:02Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:20:02Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:20:03Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:20:17Z","m":"Readiness probe failed: ","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:20:29Z","m":"Killing container with id docker://elasticsearch:Need to kill Pod","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:20:40Z","m":"Successfully assigned e2e-m9cld-mercury/test-es-keystore-drf6-es-masterdata-1 to gke-eck-gke12-129-e2e-default-pool-4cd67ad4-jq93","n":"test-es-keystore-drf6-es-masterdata-1","z":null}
{"t":"2019-11-25T01:20:53Z","m":"AttachVolume.Attach succeeded for volume \"pvc-2e8196bc-0f21-11ea-90f0-42010a84012d\" ","n":"test-es-keystore-drf6-es-masterdata-1","z":null}
{"t":"2019-11-25T01:20:58Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:20:58Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.initContainers{elastic-internal-init-keystore}"}
{"t":"2019-11-25T01:20:58Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:20:58Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.initContainers{elastic-internal-init-keystore}"}
{"t":"2019-11-25T01:20:58Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:20:59Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.initContainers{elastic-internal-init-keystore}"}
{"t":"2019-11-25T01:21:02Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:21:03Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:21:03Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:21:15Z","m":"Readiness probe failed: ","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:21:27Z","m":"Killing container with id docker://elasticsearch:Need to kill Pod","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:21:29Z","m":"Successfully assigned e2e-m9cld-mercury/test-es-keystore-drf6-es-masterdata-0 to gke-eck-gke12-129-e2e-default-pool-d388c97d-d23r","n":"test-es-keystore-drf6-es-masterdata-0","z":null}
{"t":"2019-11-25T01:21:44Z","m":"AttachVolume.Attach succeeded for volume \"pvc-2e68f7ee-0f21-11ea-90f0-42010a84012d\" ","n":"test-es-keystore-drf6-es-masterdata-0","z":null}
{"t":"2019-11-25T01:21:46Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:21:46Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:21:46Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:21:49Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.initContainers{elastic-internal-init-keystore}"}
{"t":"2019-11-25T01:21:49Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.initContainers{elastic-internal-init-keystore}"}
{"t":"2019-11-25T01:21:49Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.initContainers{elastic-internal-init-keystore}"}
{"t":"2019-11-25T01:21:52Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:21:52Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:21:52Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:22:03Z","m":"Readiness probe failed: ","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:22:16.649161742Z","a":"TestUpdateESSecureSettings/Remove_secure_settings_from_the_spec"}
# -- Restart
{"t":"2019-11-25T01:22:16.662439747Z","a":"TestUpdateESSecureSettings/Elasticsearch_secure_settings_should_eventually_be_set_in_all_nodes_keystore#03"}
{"t":"2019-11-25T01:22:18Z","m":"Killing container with id docker://elasticsearch:Need to kill Pod","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:22:20Z","m":"MountVolume.SetUp failed for volume \"elastic-internal-secure-settings\" : secrets \"test-es-keystore-drf6-es-secure-settings\" not found","n":"test-es-keystore-drf6-es-masterdata-0","z":null}
{"t":"2019-11-25T01:22:20Z","m":"MountVolume.SetUp failed for volume \"elastic-internal-secure-settings\" : secrets \"test-es-keystore-drf6-es-secure-settings\" not found","n":"test-es-keystore-drf6-es-masterdata-1","z":null}
{"t":"2019-11-25T01:22:20Z","m":"Successfully assigned e2e-m9cld-mercury/test-es-keystore-drf6-es-masterdata-2 to gke-eck-gke12-129-e2e-default-pool-b68d125e-f9xr","n":"test-es-keystore-drf6-es-masterdata-2","z":null}
{"t":"2019-11-25T01:22:30Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:22:30Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:22:30Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:22:31Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:22:31Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:22:31Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:22:35,483Z","m":"using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sdc)]], net usable_space [957.3mb], net total_space [975.8mb], types [ext4]","n":"test-es-keystore-drf6-es-masterdata-2"}
{"t":"2019-11-25T01:22:45Z","m":"Readiness probe failed: ","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:22:49,984Z","m":"started","n":"test-es-keystore-drf6-es-masterdata-2"}
{"t":"2019-11-25T01:23:32Z","m":"Killing container with id docker://elasticsearch:Need to kill Pod","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:24:30Z","m":"Successfully assigned e2e-m9cld-mercury/test-es-keystore-drf6-es-masterdata-1 to gke-eck-gke12-129-e2e-default-pool-4cd67ad4-jq93","n":"test-es-keystore-drf6-es-masterdata-1","z":null}
{"t":"2019-11-25T01:24:36Z","m":"AttachVolume.Attach succeeded for volume \"pvc-2e8196bc-0f21-11ea-90f0-42010a84012d\" ","n":"test-es-keystore-drf6-es-masterdata-1","z":null}
{"t":"2019-11-25T01:24:48Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:24:48Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:24:48Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:24:50Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:24:50Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:24:50Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:24:54,324Z","m":"using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sdc)]], net usable_space [957.3mb], net total_space [975.8mb], types [ext4]","n":"test-es-keystore-drf6-es-masterdata-1"}
{"t":"2019-11-25T01:25:00Z","m":"Readiness probe failed: ","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:25:08,274Z","m":"started","n":"test-es-keystore-drf6-es-masterdata-1"}
{"t":"2019-11-25T01:25:12Z","m":"Killing container with id docker://elasticsearch:Need to kill Pod","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:26:41Z","m":"Successfully assigned e2e-m9cld-mercury/test-es-keystore-drf6-es-masterdata-0 to gke-eck-gke12-129-e2e-default-pool-d388c97d-d23r","n":"test-es-keystore-drf6-es-masterdata-0","z":null}
{"t":"2019-11-25T01:26:48Z","m":"AttachVolume.Attach succeeded for volume \"pvc-2e68f7ee-0f21-11ea-90f0-42010a84012d\" ","n":"test-es-keystore-drf6-es-masterdata-0","z":null}
{"t":"2019-11-25T01:26:58Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:26:58Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:26:58Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.initContainers{elastic-internal-init-filesystem}"}
{"t":"2019-11-25T01:26:59Z","m":"Container image \"docker.elastic.co/elasticsearch/elasticsearch:7.4.0\" already present on machine","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:26:59Z","m":"Created container","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:27:00Z","m":"Started container","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:27:03,614Z","m":"using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sdc)]], net usable_space [957.3mb], net total_space [975.8mb], types [ext4]","n":"test-es-keystore-drf6-es-masterdata-0"}
{"t":"2019-11-25T01:27:13Z","m":"Readiness probe failed: ","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:27:16,074Z","m":"started","n":"test-es-keystore-drf6-es-masterdata-0"}

We observe the abnormal duration of the 3th restart (>5min) compared to the first two (2min24 and 2min27).
I notice that some time is lost between the two events Killing container ... and Successfully assigned ....

# -- Restart
# 
{"t":"2019-11-25T01:17:36Z","m":"Killing container with id docker://elasticsearch:Need to kill Pod","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:17:37Z","m":"Successfully assigned e2e-m9cld-mercury/test-es-keystore-drf6-es-masterdata-2 to gke-eck-gke12-129-e2e-default-pool-b68d125e-f9xr","n":"test-es-keystore-drf6-es-masterdata-2","z":null}
# 1s
{"t":"2019-11-25T01:18:17Z","m":"Killing container with id docker://elasticsearch:Need to kill Pod","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:18:19Z","m":"Successfully assigned e2e-m9cld-mercury/test-es-keystore-drf6-es-masterdata-1 to gke-eck-gke12-129-e2e-default-pool-4cd67ad4-jq93","n":"test-es-keystore-drf6-es-masterdata-1","z":null}
# 2s
{"t":"2019-11-25T01:18:33Z","m":"AttachVolume.Attach succeeded for volume \"pvc-2e8196bc-0f21-11ea-90f0-42010a84012d\" ","n":"test-es-keystore-drf6-es-masterdata-1","z":null}
# 
{"t":"2019-11-25T01:19:09Z","m":"Killing container with id docker://elasticsearch:Need to kill Pod","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:19:11Z","m":"Successfully assigned e2e-m9cld-mercury/test-es-keystore-drf6-es-masterdata-0 to gke-eck-gke12-129-e2e-default-pool-d388c97d-d23r","n":"test-es-keystore-drf6-es-masterdata-0","z":null}
# 2s

# -- Restart
# 
{"t":"2019-11-25T01:19:49Z","m":"Killing container with id docker://elasticsearch:Need to kill Pod","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:19:52Z","m":"Successfully assigned e2e-m9cld-mercury/test-es-keystore-drf6-es-masterdata-2 to gke-eck-gke12-129-e2e-default-pool-b68d125e-f9xr","n":"test-es-keystore-drf6-es-masterdata-2","z":null}
# 3s
{"t":"2019-11-25T01:20:29Z","m":"Killing container with id docker://elasticsearch:Need to kill Pod","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:20:40Z","m":"Successfully assigned e2e-m9cld-mercury/test-es-keystore-drf6-es-masterdata-1 to gke-eck-gke12-129-e2e-default-pool-4cd67ad4-jq93","n":"test-es-keystore-drf6-es-masterdata-1","z":null}
# 11s
{"t":"2019-11-25T01:20:53Z","m":"AttachVolume.Attach succeeded for volume \"pvc-2e8196bc-0f21-11ea-90f0-42010a84012d\" ","n":"test-es-keystore-drf6-es-masterdata-1","z":null}
# 
{"t":"2019-11-25T01:21:27Z","m":"Killing container with id docker://elasticsearch:Need to kill Pod","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:21:29Z","m":"Successfully assigned e2e-m9cld-mercury/test-es-keystore-drf6-es-masterdata-0 to gke-eck-gke12-129-e2e-default-pool-d388c97d-d23r","n":"test-es-keystore-drf6-es-masterdata-0","z":null}
# 2s
{"t":"2019-11-25T01:21:44Z","m":"AttachVolume.Attach succeeded for volume \"pvc-2e68f7ee-0f21-11ea-90f0-42010a84012d\" ","n":"test-es-keystore-drf6-es-masterdata-0","z":null}

# -- Restart
# 
{"t":"2019-11-25T01:22:18Z","m":"Killing container with id docker://elasticsearch:Need to kill Pod","n":"test-es-keystore-drf6-es-masterdata-2","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:22:20Z","m":"Successfully assigned e2e-m9cld-mercury/test-es-keystore-drf6-es-masterdata-2 to gke-eck-gke12-129-e2e-default-pool-b68d125e-f9xr","n":"test-es-keystore-drf6-es-masterdata-2","z":null}
# 2s
{"t":"2019-11-25T01:23:32Z","m":"Killing container with id docker://elasticsearch:Need to kill Pod","n":"test-es-keystore-drf6-es-masterdata-1","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:24:30Z","m":"Successfully assigned e2e-m9cld-mercury/test-es-keystore-drf6-es-masterdata-1 to gke-eck-gke12-129-e2e-default-pool-4cd67ad4-jq93","n":"test-es-keystore-drf6-es-masterdata-1","z":null}
# >> 58s <<<<<<< !!
{"t":"2019-11-25T01:24:36Z","m":"AttachVolume.Attach succeeded for volume \"pvc-2e8196bc-0f21-11ea-90f0-42010a84012d\" ","n":"test-es-keystore-drf6-es-masterdata-1","z":null}
# 
{"t":"2019-11-25T01:25:12Z","m":"Killing container with id docker://elasticsearch:Need to kill Pod","n":"test-es-keystore-drf6-es-masterdata-0","z":"spec.containers{elasticsearch}"}
{"t":"2019-11-25T01:26:41Z","m":"Successfully assigned e2e-m9cld-mercury/test-es-keystore-drf6-es-masterdata-0 to gke-eck-gke12-129-e2e-default-pool-d388c97d-d23r","n":"test-es-keystore-drf6-es-masterdata-0","z":null}
# >> 1m29s <<<<< !!
{"t":"2019-11-25T01:26:48Z","m":"AttachVolume.Attach succeeded for volume \"pvc-2e68f7ee-0f21-11ea-90f0-42010a84012d\" ","n":"test-es-keystore-drf6-es-masterdata-0","z":null}
#

Why? Is it the time to schedule a pod? Is it related to the volume attachment?

@thbkrkr
Copy link
Contributor

thbkrkr commented Nov 28, 2019 •
edited
Loading

I went through the last 14 builds. 10 failed because we broke the job config so nothing to see there :(.

Summary of "interesting" failures:

  • 3x TestMutationHTTPToHTTPS/All_expected_Pods_should_eventually_be_ready probably because the fs init container waited for the certs too long (~2min)
  • 1x TestUpdateESSecureSettings/Elasticsearch_secure_settings_should_eventually_be_set_in_all_nodes_keystore (see the comment above)

Build https://devops-ci.elastic.co/job/cloud-on-k8s-versions-gke/137/

FAIL: TestMutationHTTPToHTTPS/All_expected_Pods_should_eventually_be_ready (300.00s)

Build https://devops-ci.elastic.co/job/cloud-on-k8s-versions-gke/136/

FAIL: TestMutationHTTPToHTTPS/All_expected_Pods_should_eventually_be_ready (300.00s)

Build https://devops-ci.elastic.co/job/cloud-on-k8s-versions-gke/135/ to 130

Job config was broken.

Build https://devops-ci.elastic.co/job/cloud-on-k8s-versions-gke/129/

FAIL: TestUpdateESSecureSettings/Elasticsearch_secure_settings_should_eventually_be_set_in_all_nodes_keystore#03 (300.00s)

Build https://devops-ci.elastic.co/job/cloud-on-k8s-versions-gke/128/

FAIL: TestMutationHTTPToHTTPS/All_expected_Pods_should_eventually_be_ready (300.00s)

Build https://devops-ci.elastic.co/job/cloud-on-k8s-versions-gke/127/ to 124

Already resolved issue that was related to a misconfiguration of the job for the E2E license test.

@thbkrkr
Copy link
Contributor

thbkrkr commented Dec 20, 2019

I close this because the job has been successful multiple times since the creation of this issue.
The next job failure should be created in a dedicated new issue.

@thbkrkr thbkrkr closed this as completed Dec 20, 2019
@sebgl sebgl mentioned this issue Jan 6, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
:ci Things related to Continuous Integration, automation and releases >test Related to unit/integration/e2e tests
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@thbkrkr @pebrc