Refactor E2E tests

[charith-elastic]

Based on the E2E testing tasks on our board, it appears that going forward we want to do the following:

• Run the e2e tests against multiple cloud providers, Kubernetes versions, and operator versions
• Test operators across multiple namespaces
• Run tests in parallel to save time and cost

We already have some nice abstractions in the source code such as the builders used for manipulating resource definitions. However, the bootstrap code and some of the prevailing assumptions present some issues going forward.

• Assumes a single namespace named e2e
  • Hard to test scenarios involving interactions between two or more namespaces (requires fundamental changes to the Makefile and duplication of config files)
  • Potential for test pollution because all tests run in the same namespace
  • Difficult to run unrelated tests in parallel
• Complicated bootstrap
  • Requires multiple nested calls to Makefile targets
  • Resource files have to be manipulated using sed/awk scripts
  • Awkward looping and control constructs available in Makefiles/shell scripts encourage unnecessary duplication or difficult to parse code
• Difficult to parameterize
  • Will need to introduce combinations of environment variables and more complicated scripts to setup the desired environment (eg. for testing against different cloud providers or operator versions)
• Different from other projects in the eco system
  • Most community projects including Kubernetes and Kubebuilder use Ginkgo for e2e tests and supports Kind for local tests

Proposal

In order to support more advanced testing scenarios such as the ones mentioned at the beginning of this issue, we should consider refactoring our E2E code base.

• Use Go code to setup tests as much a possible (Eg. see Kubebuilder e2e tests)
  • Provides flexibility to easily adapt to the environment (Eg. programmatic manipulation of YAML resource definitions)
  • Use random generated names for resources to avoid test pollution
  • Build abstractions to hide platform or version specific details from the testing logic
• Switch to ginkgo test framework
  • Standard choice of testing framework in the Kubernetes eco system
  • Ability to write table-driven test suites (useful for testing different cloud providers, operator versions etc.)
  • Fine-grained control over test lifecycle (BeforeSuite, BeforeEach, AfterEach, AfterSuite etc.)
  • Parallel execution of tests
  • Export results in JUnit format (See also Investigate possibility of using https://cloud-test-stats.elastic.co for collecting test data #1098 and CI builds: export test results to JUnit format #1229)
  • Ability to add custom matchers thus simplifying and keeping the codebase DRY

[thbkrkr]

I'm very favorable for this refactoring. This seems very legitimate.

It seems to me that Ginkgo works with Gomega.

@sebgl Did you remember why Gomega was replaced by Testify? #25

[sebgl]

@sebgl Did you remember why Gomega was replaced by Testify? #25

Only because most of the team at the time preferred the simplicity of testify, and the fact it is much closer to "simple" go testing.

I personally feel like gomega syntax is harder to read:

However if it makes sense for E2E tests then there's no reason not to use it (also it does not mean we would have to use it for non-E2E tests).
I'd love to see an example of what the tests would look like first and what benefits that would bring exactly.

[anyasabo]

+1 I'm definitely in favor of refactoring these to cover more use cases. A proof of concept PR replacing one of the tests would be welcome I think

[pebrc]

👍 on moving more test setup code into Go
👍 on doing a POC with a few tests as @anyasabo said
😐 on adopting Ginkgo. I am not opposed in principle but I think we should make sure we evaluate this properly before we commit to it:

• If Ginkgo is anything like the corresponding matcher library Gomega I would proceed with caution as we found that one a bit verbose and complex and replaced it as @sebgl has pointed out with testify
• running tests in parallel would be interesting but as I understand it Ginkgo just runs multiple go test invocations in parallel because in-process parallelisation is not feasible because of the closures it uses
• JUnit output would be great for our Jenkins integration, I wonder if we can get it via a dedicated library as well or write it ourselves, which might allow us to report 'failures' that are not go test related e.g. linter, license header checks or doc build failures

[david-kow]

We synced with @charith-elastic and while he works on adding more flexibility to running e2e tests I'll look into kubetest for provisioning clusters across different clouds.

[david-kow]

After looking at kubetest, I'd be against incorporating it for few reasons:

1. it's a facade, not an abstraction over providers - there is some provider specific configuration still involved,
2. it seems that authn/z with the provider has to be done outside of kubetest,
3. it requires running from kubernetes directory,
4. kubetest v2 is already planned, so the v1s path forward and support is unclear.

Initially I hoped we can drag&drop kubetest, but it seems there still would be some work involved and to me it doesn't look like it's worth it given that we do have the code to automate gke/eks/aks already.

[charith-elastic]

Created #1364 to lay the foundation.

• Moves test setup, execution and cleanup to Go code
• Generates unique names for test runs to support test isolation⁺ and multiple namespaces

[⁺] Some resources created by the operator such as the validating web hook cannot be effectively isolated yet

Ginkgo + Gomega does not seem to add much value at the present and can be shelved. This seems to be the opinion of most of the team as well.

Further things to consider are:

• Running different test suites in parallel (APM, ES and KB)
  • Cluster singletons such as the validating webhook might make this tricky
  • The batch job used to run the tests will need a lot more privileges in order to create namespaces and other resources from within the cluster
  • The test cluster would need more high-spec nodes (using preemptible GKE nodes should make this less expensive)

[charith-elastic]

Closing this as #1364 covers most of the points. Will raise different issues for further improvements.