Namespace deletion takes around 2 hrs sometimes during kuttl e2e tests

[sanjeevnewar2007]

KUTTL Version: version.Info{GitVersion:"0.6.1", GitCommit:"4908507", BuildDate:"2020-08-25T18:22:14Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}

Hello all,

I am facing an issue while running kuttl e2e test suite.
During any test cases failure, the namespace deletion takes around 2 hrs as below:

12:55:23 logger.go:42: 20:19:13 | abc-test | Deleting namespace: kudo-test-live-alpaca
14:56:34 === CONT kuttl/harness/tri-11-create-one-or-more-test
14:56:34 logger.go:42: 22:20:24 | tri-11-create-one-or-more-test/3-add-test | test step failed 3-add-test

kuttl-config.yaml values:
apiVersion: kudo.dev/v1alpha1
kind: TestSuite
testDirs:

• ./tests/e2e
  parallel: 6
  crdDir: ./config/crd/bases
  timeout: 300
  startKIND: true
  artifactsDir: kuttl-artifacts
  kindNodeCache: true

Is there any solution for this?
Please let me know if i missed any information.

Thanks,
Sanjeev

[porridge]

What is likely happening is that namespace deletion is blocked on deleting some contained resource, which in turn is waiting for some controller to clean up and remove a finalizer field. It is not possible to say anything more concrete than this without knowing the details the resources in your test.
Not sure if there is anything kuttl could do to help you debug this either...

[ANeumann82]

Hmmm, maybe kuttl shouldn't block on the deletion for 2 hours? It could time out and throw an error, that might be a better solution than to block forever

[sanjeevnewar2007]

Right. Thanks for the comment. Its true that namespace deletion is waiting for cleanup or resource waiting to be deleted. Do we have any force delete namespace option in kuttl? The timeout set for each test doesnt seems to be applied on namespace deletion.

[kensipe]

thanks for the awareness... I like the idea of a separate timeout on namespace del which will force delete if passed. I will look into options

[kensipe]

The bug label may not be justified... but it feels unintended (although not designed). Expect a solution to be in the next release (next week).

[sanjeevnewar2007]

Thanks kensipe.. Appreciate your help.

[kensipe]

@sanjeevnewar2007 the current solution ^^ #230 will apply current configured timeout to namespace create and delete. This doesn't solve the "why" does the namespace take so long.. but it will fail the test so you know there is an issue in whatever timeout you configure.

we could use your advice on additional preferred features around this if you have a moment.

[sanjeevnewar2007]

Thanks @kensipe . Just want to confirm, if the timeout for each teststep is 100 seconds, post failure of the teststep, does the namespace deletion will take another 100 seconds or it will be done at the same time?
Other thing I had to ask is about the assert for an item output in the array. I can explain more if you can let me know that i can do it in the same ticket.

[kensipe]

@sanjeevnewar2007 great question and points... yes on the first question... The solution provided to solve this issue was to use the same timeout for namespace deletes... and timeout is something I noted in another issue that will need some management. Currently the timeout is applied to each step along the way... which may not be clear. There is also currently not timeout for a full test. Just like asserts may have a different timeout that is assert specific... I would like to add a timeout control for namespace management. The solution currently is provide a timeout. Definitely more to come on this.

On the array... this has been reported for some time... and is a high priority to provide a solution for. Thanks for the separate issue... but I will close that as a duplicate. The issue to track is: #76