Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TestFailoverPlayground flaky terratest #1345

Closed
ytsarev opened this issue Dec 12, 2023 · 0 comments Β· Fixed by #1374
Closed

TestFailoverPlayground flaky terratest #1345

ytsarev opened this issue Dec 12, 2023 · 0 comments Β· Fixed by #1374

Comments

@ytsarev
Copy link
Member

ytsarev commented Dec 12, 2023

Even after #1340 there TestFailoverPlayground if failing from time to time

Example run: https://github.com/k8gb-io/k8gb/actions/runs/7188544157

Terratests Results πŸš€
--- ❌ FAIL: TestFailoverPlayground (143.83s)
--- βœ“ PASS: TestFailoverPlayground/failover_on_two_concurrent_clusters_with_TestApp_running (0.75s)
--- βœ“ PASS: TestFailoverPlayground/stop_podinfo_on_eu_cluster (45.94s)
--- ❌ FAIL: TestFailoverPlayground/start_podinfo_again_on_eu_cluster (6.52s)

Detailed related logs from the dump:

estFailoverPlayground 2023-12-12T23:36:13Z logger.go:66: Running command kubectl with args [--context k3d-test-gslb1 -n k8gb-test-korpdu run -i --rm busybox --restart Never --image busybox --overrides {"spec":{"dnsConfig":{"nameservers":["10.43.52.104"]},"dnsPolicy": "None"}} -- sh -c wget -qO - playground-failover.cloud.example.com]
TestFailoverPlayground 2023-12-12T23:36:16Z logger.go:66: wget: server returned error: HTTP/1.1 503 Service Temporarily Unavailable
TestFailoverPlayground 2023-12-12T23:36:16Z logger.go:66: pod "busybox" deleted
TestFailoverPlayground 2023-12-12T23:36:16Z logger.go:66: pod k8gb-test-korpdu/busybox terminated (Error)
    extensions.go:437: 
        	Error Trace:	/home/runner/work/k8gb/k8gb/terratest/test/extensions.go:437
        	            				/home/runner/work/k8gb/k8gb/terratest/test/k8gb_failover_playground_test.go:60
        	            				/home/runner/work/k8gb/k8gb/terratest/test/k8gb_failover_playground_test.go:85
        	Error:      	Received unexpected error:
        	            	error while running command: exit status 1; pod k8gb-test-korpdu/busybox terminated (Error)
        	Test:       	TestFailoverPlayground
        	Messages:   	busybox%!(EXTRA []string=[sh -c wget -qO - playground-failover.cloud.example.com], string=wget: server returned error: HTTP/1.1 503 Service Temporarily Unavailable
        	            	pod "busybox" deleted
        	            	pod k8gb-test-korpdu/busybox terminated (Error))
=== CONT  TestFailoverPlayground
    extensions.go:245: killing Instance: k3d-test-gslb2:k8gb-test-109htg
TestFailoverPlayground 2023-12-12T23:36:16Z client.go:42: Configuring Kubernetes client using config file /home/runner/.kube/config with context k3d-test-gslb2
    extensions.go:245: killing Instance: k3d-test-gslb1:k8gb-test-korpdu
TestFailoverPlayground 2023-12-12T23:36:16Z client.go:42: Configuring Kubernetes client using config file /home/runner/.kube/config with context k3d-test-gslb1
--- FAIL: TestFailoverPlayground (143.83s)
    --- PASS: TestFailoverPlayground/failover_on_two_concurrent_clusters_with_TestApp_running (0.75s)
    --- PASS: TestFailoverPlayground/stop_podinfo_on_eu_cluster (45.94s)
    --- FAIL: TestFailoverPlayground/start_podinfo_again_on_eu_cluster (6.52s)

and

=== RUN   TestFailoverPlayground/start_podinfo_again_on_eu_cluster
=== CONT  TestFailoverPlayground/start_podinfo_again_on_eu_cluster
    testing.go:1343: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test
    --- FAIL: TestFailoverPlayground/start_podinfo_again_on_eu_cluster (6.52s)
ytsarev added a commit to ytsarev/k8gb that referenced this issue Dec 30, 2023
* Fixes k8gb-io#1345
* Avoid getting from testing.T context as in
  stretchr/testify#573
  and golang/go#21175

Signed-off-by: Yury Tsarev <yury@upbound.io>
ytsarev added a commit to ytsarev/k8gb that referenced this issue Dec 30, 2023
* Do not fail immediately on 503
* Fixes k8gb-io#1345

Signed-off-by: Yury Tsarev <yury@upbound.io>
ytsarev added a commit to ytsarev/k8gb that referenced this issue Dec 30, 2023
* Do not fail immediately on 503
* Fixes k8gb-io#1345

Signed-off-by: Yury Tsarev <yury@upbound.io>
ytsarev added a commit to ytsarev/k8gb that referenced this issue Dec 31, 2023
* Do not fail immediately on 503
* Fixes k8gb-io#1345

Signed-off-by: Yury Tsarev <yury@upbound.io>
ytsarev added a commit to ytsarev/k8gb that referenced this issue Dec 31, 2023
* Do not fail immediately on 503
* Fixes k8gb-io#1345

Signed-off-by: Yury Tsarev <yury@upbound.io>
ytsarev added a commit that referenced this issue Jan 1, 2024
* Fix last flaky terratest

* Do not fail immediately on 503
* Fixes #1345

Signed-off-by: Yury Tsarev <yury@upbound.io>

* Remove concurrency limits from the jobs

* Remove part of the config introduced by #1123 as the
  terratest suite is stable now

Signed-off-by: Yury Tsarev <yury@upbound.io>

* Change way to check for 503 and also sleep for 1 second in case of
detection

Signed-off-by: Yury Tsarev <yury@upbound.io>

* Add more verbose logging to 503 edge case

Signed-off-by: Yury Tsarev <yury@upbound.io>

---------

Signed-off-by: Yury Tsarev <yury@upbound.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant