Adding the ability to skip tests that require the AllowAll identity p… #11254
Conversation
…rovider Conformance tests are useful in some running clusters however admin can't always want to disable authentication. The main thing this commit does is add a new optional regex for excluding tests. The TEST_ONLY logic was flipped to make it a little easier to read (and I don't think it was actually working before). Running with these tests skipped against the latest Origin 1.4 codebase: ------- Ran 115 of 564 Specs in 2097.700 seconds SUCCESS! -- 115 Passed | 0 Failed | 0 Pending | 449 Skipped PASS Ran 7 of 564 Specs in 162.981 seconds SUCCESS! -- 7 Passed | 0 Failed | 0 Pending | 557 Skipped PASS ------- Another important difference is the fact that the conformance tests are now run with the extended.test binary instead of with ginkgo. Ginkgo doesn't ship with the official builds of Origin or OpenShift Container platform and it seems like the differences between the two ways of running the tests are arbitrary. This allows someone to run the tests as follows: PARALLEL_NODES=1 KUBECONFIG=/etc/origin/master/admin.kubeconfig EXTENDEDTEST="/usr/libexec/atomic-openshift/extended.test" TEST_ONLY=true OPENSHIFT_SKIP_ALLOWALL_AUTH_TESTS=true ./test/extended/conformance.sh
|
[test] |
|
rather than excluding lots of tests this way, can we work to fix the assumption that new credentials can be obtained within the test? |
|
Evaluated for origin test up to 9147bba |
|
continuous-integration/openshift-jenkins/test Running (https://ci.openshift.redhat.com/jenkins/job/test_pr_origin/9716/) |
|
@liggitt, how would you want it to work? I think it would be great if preconfigured users could be used. However, I'm worried many of the tests are relying on the fact that they get to start from a clean slate and we'd see even more problems with flakiness when the tests run in parallel. I don't really want to run the tests in a significantly different way than how most people are running them. My immediate goal is really just to have enough tests to exercise a running environment, clean up after themselves and finish in around 30 minutes. I'm already slightly over that time goal. I can definitely open an issue upstream. I won't likely have time to fix it though. |
|
The upstream tests all use the specified kubeconfig. Origin tests assume they can log in a new user with any password (https://github.com/openshift/origin/blob/master/test/util/client.go#L67) Instead of doing an anypassword login to accomplish that, we could provision the user and api token explicitly. That should allow those tests to run, regardless of auth config. Definitely open an issue to fix that. |
…rovider
Conformance tests are useful in some running clusters however admin can't
always want to disable authentication. The main thing this commit does is add
a new optional regex for excluding tests. The TEST_ONLY logic was flipped to
make it a little easier to read (and I don't think it was actually working
before).
Running with these tests skipped against the latest Origin 1.4 codebase:
Another important difference is the fact that the conformance tests are now run
with the extended.test binary instead of with ginkgo. Ginkgo doesn't ship with
the official builds of Origin or OpenShift Container platform and it seems like
the differences between the two ways of running the tests are arbitrary. This
allows someone to run the tests as follows: