Allow for tests to run in a pre-created NS

[duglin]

Problem

In some environments it won't be possible for the testing infrastructure to easily create new namespaces at will, so it would be nice if there was a way to run the e2e tests all within a single, pre-created, namespace.

Persona:
Eventing contributors/devs

Exit Criteria
Ability to run all unit and e2e tests against a pre-created cluster and namespace.

[grantr]

This seems like a lot of work. How important is this to you @duglin?

[duglin]

Unfortunately, it's kind of important. We have a managed environment where people cant' just do kubectl create ns ... at will, they need to go through a custom process to ask for a new workspace/namespace. So, as of today I don't think we can run the community tests against our env to know if something is broken.

Aside from specific cross-ns testing, is it really required to run the tests in new/multiple namespaces?

[mgencur]

I've attached a possible solution. Instead of creating just one common namespace, make sure there are no random suffixes so the NS names are predictable.

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[mgencur]

/reopen
/remove-lifecycle stale

Hello, I'm re-opening this because this is a long-term issue that needs to be resolved. The issue description has it all: there are environments with restricted user privileges where people won't be able to run the tests.

Previously, there were some solutions tried in #3313:

• Having namespace names generated based on the test name (without random suffix) - this was problematic because tests often have sub-tests and these subtests would share the same namespace unless the individual objects have random suffixes
• Having just one common namespace and generate every object name for the tests with random suffixes so that all objects are unique even if they exist in the same namespace. This is a lot of work, and requires big changes in the test suite. Another problem is that some invidividual tests require label knative-eventing-injection": "enabled" and some require "disabled" which makes it hard to run those in parallel in the same namespace.

Recently, I ran across a different solution that is used in knative/client:
https://github.com/knative/client/blob/main/lib/test/integration.go#L47
It is pretty simple, each test has a unique namespace but the name of the namespace is predictable. Each test gets a namespace a numeric suffix that is increasing. The suffix is a simple number so it's possible to pre-create them, apply some additional logic or/and create additional objects in them. In a specific environment, this can be done by a user with increased privileges (admin). The limitation is that individual tests should not require specific settings that would have to be done by admin user because it is not known in advance which NS the test will run in.

Any thoughts on this solution? I'd like to try that.

[knative-prow-robot]

@mgencur: Reopened this issue.

In response to this:

/reopen
/remove-lifecycle stale

Hello, I'm re-opening this because this is a long-term issue that needs to be resolved. The issue description has it all: there are environments with restricted user privileges where people won't be able to run the tests.

Previously, there were some solutions tried in #3313:

• Having namespace names generated based on the test name (without random suffix) - this was problematic because tests often have sub-tests and these subtests would share the same namespace unless the individual objects have random suffixes
• Having just one common namespace and generate every object name for the tests with random suffixes so that all objects are unique even if they exist in the same namespace. This is a lot of work, and requires big changes in the test suite. Another problem is that some invidividual tests require label knative-eventing-injection": "enabled" and some require "disabled" which makes it hard to run those in parallel in the same namespace.

Recently, I ran across a different solution that is used in knative/client:
https://github.com/knative/client/blob/main/lib/test/integration.go#L47
It is pretty simple, each test has a unique namespace but the name of the namespace is predictable. Each test gets the next namespace from a predefined bucket. The number of available namespaces is limited so it's possible to pre-create them, apply some additional logic or/and create additional objects in them. In a specific environment, this can be done by a user with increased privileges (admin). The limitation is that the parallel flag can't be higher than the size of the bucket. And requires tests to NOT require specific settings that would have to be done by admin user because it is not known in advance which NS the test will run in.

Any thoughts on this solution? I'd like to try that.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.