Add acceptance tests for User CRD #226

andrewstucki · 2024-09-11T14:21:49Z

This adds acceptance tests based on #225 along with the start of an acceptance test github actions nightly workflow (may not actually fully work yet, since I haven't been able to test it locally). In order to leverage our client factory I had to move it from internal --> pkg and also found that the helmette client stuff I had used in the helm-charts repo just didn't work properly due to not setting up TLS trust properly when hitting the API client (makes sense since it's meant to be a chart installation helper), so I copied most of the code over to this repo and then modified it to use an already initialized controller-runtime client.

Additionally, while we may generally prefer k3d -- just so I can quickly throw something together we can potentially run in CI without too much setup, I threw together something with kind so I can follow some pretty simple setup for a nightly sort of run. I'd be happy to circle back on figuring out exactly how we want to do the acceptance test runs, how we want to setup said environments, etc. in a separate PR, but let me know initial thoughts for sure.

EDIT: This is a stacked PR, the parent PR should merged first, but opening this up so it's reviewable.

andrewstucki · 2024-09-11T14:36:16Z

go.work

@@ -1,6 +1,7 @@
 go 1.22.4

 use (
+	./acceptance


I have these as their own project just to really drive home the point that these are full environment-level tests that we consider black boxes -- we're not calling any internal controller code or "testing" sort of paths -- instead it's just "deploy a cluster, create some resources, and run expectations on their side-effects"

RafalKorepta

Interesting problem. The test suite failed for me and didn't clean cert-manager helm deployment. The next time I run it complained that cert-manager helm release with the same is already installed. It seems like AfterSuite is not called when test fail.

go test -test.v
=== RUN   TestSuite
2024/09/12 15:35:29 Executing: "helm repo add jetstack https://charts.jetstack.io"
2024/09/12 15:35:29 Executing: "helm install cert-manager jetstack/cert-manager --output=json --create-namespace=true --namespace=cert-manager --version=v1.14.2 --wait=true --wait-for-jobs=true --generate-name=false --values=/var/folders/_d/f98hxzln11s43hczs9_dx24m0000gn/T/go-helm-client2252713576/values-3764294398.yaml"
2024/09/12 15:35:59 Executing: "helm get metadata cert-manager --output=json --namespace cert-manager"
error setting up test suite: error validating "../src/go/k8s/config/crd/bases/cluster.redpanda.com_redpandas.yaml": error validating data: failed to download openapi: Get "https://127.0.0.1:54393/openapi/v2?timeout=32s": dial tcp 127.0.0.1:54393: connect: connection refused; if you choose to ignore these errors, turn validation off with --validate=false
error validating "../src/go/k8s/config/crd/bases/cluster.redpanda.com_topics.yaml": error validating data: failed to download openapi: Get "https://127.0.0.1:54393/openapi/v2?timeout=32s": dial tcp 127.0.0.1:54393: connect: connection refused; if you choose to ignore these errors, turn validation off with --validate=false
error validating "../src/go/k8s/config/crd/bases/cluster.redpanda.com_users.yaml": error validating data: failed to download openapi: Get "https://127.0.0.1:54393/openapi/v2?timeout=32s": dial tcp 127.0.0.1:54393: connect: connection refused; if you choose to ignore these errors, turn validation off with --validate=false
error validating "../src/go/k8s/config/crd/bases/redpanda.vectorized.io_clusters.yaml": error validating data: failed to download openapi: Get "https://127.0.0.1:54393/openapi/v2?timeout=32s": dial tcp 127.0.0.1:54393: connect: connection refused; if you choose to ignore these errors, turn validation off with --validate=false
error validating "../src/go/k8s/config/crd/bases/redpanda.vectorized.io_consoles.yaml": error validating data: failed to download openapi: Get "https://127.0.0.1:54393/openapi/v2?timeout=32s": dial tcp 127.0.0.1:54393: connect: connection refused; if you choose to ignore these errors, turn validation off with --validate=false
: exit status 1

Process finished with the exit code 1

go test -test.v
=== RUN   TestSuite
2024/09/12 15:38:39 Executing: "helm repo add jetstack https://charts.jetstack.io"
2024/09/12 15:38:39 Executing: "helm install cert-manager jetstack/cert-manager --output=json --create-namespace=true --namespace=cert-manager --version=v1.14.2 --wait=true --wait-for-jobs=true --generate-name=false --values=/var/folders/_d/f98hxzln11s43hczs9_dx24m0000gn/T/go-helm-client3946706215/values-2411661414.yaml"
error setting up test suite: stderr: Error: INSTALLATION FAILED: cannot re-use a name that is still in use: exit status 1

Process finished with the exit code 1

chrisseto

I love that we're getting this up and running but we're introducing a lot of duplication in this PR. experience tells me If we don't get this right the first time we're going to be dealing with it for months.

Let's:

Settle on buildkite or GH actions
Settle on kind or k3d (existing usages of one or the other is fine e.g. kuttl)
Not duplicate the clients package in the helm repo.

chrisseto · 2024-09-12T14:22:49Z

.github/workflows/nightly-acceptance-tests.yml

@@ -0,0 +1,60 @@
+name: Nightly Acceptance Tests


If possible I'd prefer to stick to one mechanism for CI. GH actions (reportedly) run into issues when running multiple k3d clusters due to the small/limited machine sizes. The buildkite machines are much beefier.

Hmmm, I see what you're doing with the build matrix and I'm not sure how to easily set that up in buildkite, especially with passing build artifacts around.

I think we need to draw a hard line and pick one or the other. Maintaining both is going to be a nightmare long term as we're already fighting to keep up with buildkite/task as is.

chrisseto · 2024-09-12T14:24:35Z

acceptance/main.go

+
+import "github.com/redpanda-data/redpanda-operator/harpoon/tablegenerator"
+
+var providers = []string{"eks", "aks", "gke", "kind"}


Ah, should have mentioned this in the other PR but can we start with k3d here instead of kind? We're going to need the ability to add and remove K8s nodes and kind does not have the ability to do so.

I saw the note about using kind to start but it should really just be a difference for changing a CLI invocation, no?

chrisseto · 2024-09-12T14:35:22Z

.github/workflows/nightly-acceptance-tests.yml

+    steps:
+      - uses: actions/checkout@v4.1.7
+
+      - uses: actions/setup-go@v5.0.2


No need to do this, the nix install already handles it.

chrisseto · 2024-09-12T14:37:17Z

.github/workflows/nightly-acceptance-tests.yml

+      - name: Install nix
+        uses: cachix/install-nix-action@v27
+
+      - name: Build


If you copy this from the helm-charts repo, you can omit nix develop -c bash everywhere.

.github/actions/acceptance-tests/action.yml

andrewstucki · 2024-09-12T16:54:47Z

@chrisseto:

Settle on buildkite or GH actions
Settle on kind or k3d (existing usages of one or the other is fine e.g. kuttl)

I'm fine with either of these. Do you think it's worth just dropping the build workflows for now and following up with a separate CI PR?

Not duplicate the clients package in the helm repo.

Yep, completely agree here. My initial thought was to keep the code in the helm-charts repo since that's where we're also defining the interplay between an installation spec and its dependent resources, but I also wanted to just make sure we were good on the end-to-end tests fully working before committing to one or the other. I can clean it up before we sign off on this. That said, do we still think that the code belongs in the helm-charts repo since we're planning on moving a good chunk of that code anyway, no?

andrewstucki · 2024-09-12T18:15:21Z

@RafalKorepta:

Interesting problem. The test suite failed for me and didn't clean cert-manager helm deployment. The next time I run it complained that cert-manager helm release with the same is already installed. It seems like AfterSuite is not called when test fail.

I just added a commit into the PR this depends on to attempt cleanup prior to calling os.Exit if something in the suite setup code fails, so we should be good now. The reason that this was happening is because the error was happening in the BeforeSuite hook where we have no context available to call cleanup on or the ability to return an error, so we were calling os.Exit via setupErrorCheck and immediately terminating without cleaning up, now we just call a cleanup closure before calling os.Exit, so we should be good.

RafalKorepta

=== RUN   TestSuite
2024/09/12 22:20:56 Executing: "helm repo add jetstack https://charts.jetstack.io"
2024/09/12 22:20:56 Executing: "helm install cert-manager jetstack/cert-manager --output=json --create-namespace=true --namespace=cert-manager --version=v1.14.2 --wait=true --wait-for-jobs=true --generate-name=false --values=/var/folders/_d/f98hxzln11s43hczs9_dx24m0000gn/T/go-helm-client2863089001/values-939991934.yaml"
2024/09/12 22:21:13 Executing: "helm get metadata cert-manager --output=json --namespace cert-manager"
Feature: User CRDs
=== RUN   TestSuite/Managing_Users
    testingt.go:111: Installing Redpanda operator chart
    testingt.go:120: Switiching namespace "default" --> "testing-a9kjo2yixi"
2024/09/12 22:21:14 Executing: "helm repo add redpanda https://charts.redpanda.com"
    testingt.go:120: installing chart "redpanda/operator"
2024/09/12 22:21:15 Executing: "helm install redpanda-operator redpanda/operator --output=json --create-namespace=true --namespace=testing-a9kjo2yixi --wait=true --wait-for-jobs=true --generate-name=false --values=/var/folders/_d/f98hxzln11s43hczs9_dx24m0000gn/T/go-helm-client1398218828/values-3695434477.yaml"
    testingt.go:120: 
        	Error Trace:	/Users/rafalkorepta/workspace/redpanda/redpanda-operator/harpoon/internal/testing/helm.go:33
        	            				/Users/rafalkorepta/workspace/redpanda/redpanda-operator/acceptance/main_test.go:61
        	            				/Users/rafalkorepta/workspace/redpanda/redpanda-operator/harpoon/suite.go:121
        	            				/Users/rafalkorepta/workspace/redpanda/redpanda-operator/harpoon/internal/tracking/features.go:85
        	            				/Users/rafalkorepta/workspace/redpanda/redpanda-operator/harpoon/suite.go:246
        	            				/Users/rafalkorepta/go/pkg/mod/github.com/cucumber/godog@v0.14.1/suite.go:269
        	            				/Users/rafalkorepta/go/pkg/mod/github.com/cucumber/godog@v0.14.1/suite.go:163
        	            				/Users/rafalkorepta/go/pkg/mod/github.com/cucumber/godog@v0.14.1/suite.go:476
        	            				/Users/rafalkorepta/go/pkg/mod/github.com/cucumber/godog@v0.14.1/suite.go:532
        	Error:      	Received unexpected error:
        	            	stderr: Error: INSTALLATION FAILED: context deadline exceeded: exit status 1
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/redpanda-data/helm-charts/pkg/helm.(*Client).runHelmInDir
        	            	  | 	/Users/rafalkorepta/go/pkg/mod/github.com/redpanda-data/helm-charts@v0.0.0-20240911060052-2bf9dd6f0996/pkg/helm/helm.go:501
        	            	  | github.com/redpanda-data/helm-charts/pkg/helm.(*Client).runHelm
        	            	  | 	/Users/rafalkorepta/go/pkg/mod/github.com/redpanda-data/helm-charts@v0.0.0-20240911060052-2bf9dd6f0996/pkg/helm/helm.go:485
        	            	  | github.com/redpanda-data/helm-charts/pkg/helm.(*Client).Install
        	            	  | 	/Users/rafalkorepta/go/pkg/mod/github.com/redpanda-data/helm-charts@v0.0.0-20240911060052-2bf9dd6f0996/pkg/helm/helm.go:242
        	            	  | github.com/redpanda-data/redpanda-operator/harpoon/internal/testing.(*TestingT).InstallHelmChart
        	            	  | 	/Users/rafalkorepta/workspace/redpanda/redpanda-operator/harpoon/internal/testing/helm.go:32
        	            	  | github.com/redpanda-data/redpanda-operator/acceptance.TestMain.func1
        	            	  | 	/Users/rafalkorepta/workspace/redpanda/redpanda-operator/acceptance/main_test.go:61
        	            	  | github.com/redpanda-data/redpanda-operator/acceptance.TestMain.(*SuiteBuilder).OnFeature.func2
        	            	  | 	/Users/rafalkorepta/workspace/redpanda/redpanda-operator/harpoon/suite.go:121
        	            	  | github.com/redpanda-data/redpanda-operator/harpoon/internal/tracking.(*FeatureHookTracker).Scenario
        	            	  | 	/Users/rafalkorepta/workspace/redpanda/redpanda-operator/harpoon/internal/tracking/features.go:85
        	            	  | github.com/redpanda-data/redpanda-operator/harpoon.(*SuiteBuilder).Build.func2.1
        	            	  | 	/Users/rafalkorepta/workspace/redpanda/redpanda-operator/harpoon/suite.go:246
        	            	  | github.com/cucumber/godog.(*suite).runBeforeScenarioHooks
        	            	  | 	/Users/rafalkorepta/go/pkg/mod/github.com/cucumber/godog@v0.14.1/suite.go:269
        	            	  | github.com/cucumber/godog.(*suite).runStep
        	            	  | 	/Users/rafalkorepta/go/pkg/mod/github.com/cucumber/godog@v0.14.1/suite.go:163
        	            	  | github.com/cucumber/godog.(*suite).runSteps
        	            	  | 	/Users/rafalkorepta/go/pkg/mod/github.com/cucumber/godog@v0.14.1/suite.go:476
        	            	  | github.com/cucumber/godog.(*suite).runPickle.func1
        	            	  | 	/Users/rafalkorepta/go/pkg/mod/github.com/cucumber/godog@v0.14.1/suite.go:532
        	            	  | testing.tRunner
        	            	  | 	/Users/rafalkorepta/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.6.darwin-arm64/src/testing/testing.go:1689
        	            	  | runtime.goexit
        	            	  | 	/Users/rafalkorepta/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.6.darwin-arm64/src/runtime/asm_arm64.s:1222
        	            	Wraps: (2) stderr: Error: INSTALLATION FAILED: context deadline exceeded
        	            	Wraps: (3) exit status 1
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *exec.ExitError
        	Test:       	TestSuite/Managing_Users

  Background: Cluster available
    Given cluster "sasl" is available
    
	Error Trace:	/Users/rafalkorepta/workspace/redpanda/redpanda-operator/harpoon/internal/testing/helm.go:33
	            				/Users/rafalkorepta/workspace/redpanda/redpanda-operator/acceptance/main_test.go:61
	            				/Users/rafalkorepta/workspace/redpanda/redpanda-operator/harpoon/suite.go:121
	            				/Users/rafalkorepta/workspace/redpanda/redpanda-operator/harpoon/internal/tracking/features.go:85
	            				/Users/rafalkorepta/workspace/redpanda/redpanda-operator/harpoon/suite.go:246
	            				/Users/rafalkorepta/go/pkg/mod/github.com/cucumber/godog@v0.14.1/suite.go:269
	            				/Users/rafalkorepta/go/pkg/mod/github.com/cucumber/godog@v0.14.1/suite.go:163
	            				/Users/rafalkorepta/go/pkg/mod/github.com/cucumber/godog@v0.14.1/suite.go:476
	            				/Users/rafalkorepta/go/pkg/mod/github.com/cucumber/godog@v0.14.1/suite.go:532
	Error:      	Received unexpected error:
	            	stderr: Error: INSTALLATION FAILED: context deadline exceeded: exit status 1
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/redpanda-data/helm-charts/pkg/helm.(*Client).runHelmInDir
	            	  | 	/Users/rafalkorepta/go/pkg/mod/github.com/redpanda-data/helm-charts@v0.0.0-20240911060052-2bf9dd6f0996/pkg/helm/helm.go:501
	            	  | github.com/redpanda-data/helm-charts/pkg/helm.(*Client).runHelm
	            	  | 	/Users/rafalkorepta/go/pkg/mod/github.com/redpanda-data/helm-charts@v0.0.0-20240911060052-2bf9dd6f0996/pkg/helm/helm.go:485
	            	  | github.com/redpanda-data/helm-charts/pkg/helm.(*Client).Install
	            	  | 	/Users/rafalkorepta/go/pkg/mod/github.com/redpanda-data/helm-charts@v0.0.0-20240911060052-2bf9dd6f0996/pkg/helm/helm.go:242
	            	  | github.com/redpanda-data/redpanda-operator/harpoon/internal/testing.(*TestingT).InstallHelmChart
	            	  | 	/Users/rafalkorepta/workspace/redpanda/redpanda-operator/harpoon/internal/testing/helm.go:32
	            	  | github.com/redpanda-data/redpanda-operator/acceptance.TestMain.func1
	            	  | 	/Users/rafalkorepta/workspace/redpanda/redpanda-operator/acceptance/main_test.go:61
	            	  | github.com/redpanda-data/redpanda-operator/acceptance.TestMain.(*SuiteBuilder).OnFeature.func2
	            	  | 	/Users/rafalkorepta/workspace/redpanda/redpanda-operator/harpoon/suite.go:121
	            	  | github.com/redpanda-data/redpanda-operator/harpoon/internal/tracking.(*FeatureHookTracker).Scenario
	            	  | 	/Users/rafalkorepta/workspace/redpanda/redpanda-operator/harpoon/internal/tracking/features.go:85
	            	  | github.com/redpanda-data/redpanda-operator/harpoon.(*SuiteBuilder).Build.func2.1
	            	  | 	/Users/rafalkorepta/workspace/redpanda/redpanda-operator/harpoon/suite.go:246
	            	  | github.com/cucumber/godog.(*suite).runBeforeScenarioHooks
	            	  | 	/Users/rafalkorepta/go/pkg/mod/github.com/cucumber/godog@v0.14.1/suite.go:269
	            	  | github.com/cucumber/godog.(*suite).runStep
	            	  | 	/Users/rafalkorepta/go/pkg/mod/github.com/cucumber/godog@v0.14.1/suite.go:163
	            	  | github.com/cucumber/godog.(*suite).runSteps
	            	  | 	/Users/rafalkorepta/go/pkg/mod/github.com/cucumber/godog@v0.14.1/suite.go:476
	            	  | github.com/cucumber/godog.(*suite).runPickle.func1
	            	  | 	/Users/rafalkorepta/go/pkg/mod/github.com/cucumber/godog@v0.14.1/suite.go:532
	            	  | testing.tRunner
	            	  | 	/Users/rafalkorepta/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.6.darwin-arm64/src/testing/testing.go:1689
	            	  | runtime.goexit
	            	  | 	/Users/rafalkorepta/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.6.darwin-arm64/src/runtime/asm_arm64.s:1222
	            	Wraps: (2) stderr: Error: INSTALLATION FAILED: context deadline exceeded
	            	Wraps: (3) exit status 1
	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *exec.ExitError
	Test:       	TestSuite/Managing_Users


  Scenario: Managing Users                                                     # features/user-crds.feature:7
    Given there is no user "bob" in cluster "sasl"                             # users.go:184 -> github.com/redpanda-data/redpanda-operator/acceptance/steps.thereIsNoUser
    And there is no user "james" in cluster "sasl"                             # users.go:184 -> github.com/redpanda-data/redpanda-operator/acceptance/steps.thereIsNoUser
    And there is no user "alice" in cluster "sasl"                             # users.go:184 -> github.com/redpanda-data/redpanda-operator/acceptance/steps.thereIsNoUser
    When I create CRD-based users for cluster "sasl":                          # users.go:43 -> github.com/redpanda-data/redpanda-operator/acceptance/steps.iCreateCRDbasedUsers
      | name  | password | mechanism     | acls |
      | bob   |          | SCRAM-SHA-256 |      |
      | james |          | SCRAM-SHA-512 |      |
      | alice | qwerty   | SCRAM-SHA-512 |      |
    Then "bob" should exist and be able to authenticate to the "sasl" cluster  # users.go:159 -> github.com/redpanda-data/redpanda-operator/acceptance/steps.shouldExistAndBeAbleToAuthenticateToTheCluster
    And "james" should exist and be able to authenticate to the "sasl" cluster # users.go:159 -> github.com/redpanda-data/redpanda-operator/acceptance/steps.shouldExistAndBeAbleToAuthenticateToTheCluster
    And "alice" should exist and be able to authenticate to the "sasl" cluster # users.go:159 -> github.com/redpanda-data/redpanda-operator/acceptance/steps.shouldExistAndBeAbleToAuthenticateToTheCluster
    suite.go:534: 
        	Error Trace:	/Users/rafalkorepta/workspace/redpanda/redpanda-operator/harpoon/internal/testing/helm.go:33
        	            				/Users/rafalkorepta/workspace/redpanda/redpanda-operator/acceptance/main_test.go:61
        	            				/Users/rafalkorepta/workspace/redpanda/redpanda-operator/harpoon/suite.go:121
        	            				/Users/rafalkorepta/workspace/redpanda/redpanda-operator/harpoon/internal/tracking/features.go:85
        	            				/Users/rafalkorepta/workspace/redpanda/redpanda-operator/harpoon/suite.go:246
        	            				/Users/rafalkorepta/go/pkg/mod/github.com/cucumber/godog@v0.14.1/suite.go:269
        	            				/Users/rafalkorepta/go/pkg/mod/github.com/cucumber/godog@v0.14.1/suite.go:163
        	            				/Users/rafalkorepta/go/pkg/mod/github.com/cucumber/godog@v0.14.1/suite.go:476
        	            				/Users/rafalkorepta/go/pkg/mod/github.com/cucumber/godog@v0.14.1/suite.go:532
        	Error:      	Received unexpected error:
        	            	stderr: Error: INSTALLATION FAILED: context deadline exceeded: exit status 1
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/redpanda-data/helm-charts/pkg/helm.(*Client).runHelmInDir
        	            	  | 	/Users/rafalkorepta/go/pkg/mod/github.com/redpanda-data/helm-charts@v0.0.0-20240911060052-2bf9dd6f0996/pkg/helm/helm.go:501
        	            	  | github.com/redpanda-data/helm-charts/pkg/helm.(*Client).runHelm
        	            	  | 	/Users/rafalkorepta/go/pkg/mod/github.com/redpanda-data/helm-charts@v0.0.0-20240911060052-2bf9dd6f0996/pkg/helm/helm.go:485
        	            	  | github.com/redpanda-data/helm-charts/pkg/helm.(*Client).Install
        	            	  | 	/Users/rafalkorepta/go/pkg/mod/github.com/redpanda-data/helm-charts@v0.0.0-20240911060052-2bf9dd6f0996/pkg/helm/helm.go:242
        	            	  | github.com/redpanda-data/redpanda-operator/harpoon/internal/testing.(*TestingT).InstallHelmChart
        	            	  | 	/Users/rafalkorepta/workspace/redpanda/redpanda-operator/harpoon/internal/testing/helm.go:32
        	            	  | github.com/redpanda-data/redpanda-operator/acceptance.TestMain.func1
        	            	  | 	/Users/rafalkorepta/workspace/redpanda/redpanda-operator/acceptance/main_test.go:61
        	            	  | github.com/redpanda-data/redpanda-operator/acceptance.TestMain.(*SuiteBuilder).OnFeature.func2
        	            	  | 	/Users/rafalkorepta/workspace/redpanda/redpanda-operator/harpoon/suite.go:121
        	            	  | github.com/redpanda-data/redpanda-operator/harpoon/internal/tracking.(*FeatureHookTracker).Scenario
        	            	  | 	/Users/rafalkorepta/workspace/redpanda/redpanda-operator/harpoon/internal/tracking/features.go:85
        	            	  | github.com/redpanda-data/redpanda-operator/harpoon.(*SuiteBuilder).Build.func2.1
        	            	  | 	/Users/rafalkorepta/workspace/redpanda/redpanda-operator/harpoon/suite.go:246
        	            	  | github.com/cucumber/godog.(*suite).runBeforeScenarioHooks
        	            	  | 	/Users/rafalkorepta/go/pkg/mod/github.com/cucumber/godog@v0.14.1/suite.go:269
        	            	  | github.com/cucumber/godog.(*suite).runStep
        	            	  | 	/Users/rafalkorepta/go/pkg/mod/github.com/cucumber/godog@v0.14.1/suite.go:163
        	            	  | github.com/cucumber/godog.(*suite).runSteps
        	            	  | 	/Users/rafalkorepta/go/pkg/mod/github.com/cucumber/godog@v0.14.1/suite.go:476
        	            	  | github.com/cucumber/godog.(*suite).runPickle.func1
        	            	  | 	/Users/rafalkorepta/go/pkg/mod/github.com/cucumber/godog@v0.14.1/suite.go:532
        	            	  | testing.tRunner
        	            	  | 	/Users/rafalkorepta/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.6.darwin-arm64/src/testing/testing.go:1689
        	            	  | runtime.goexit
        	            	  | 	/Users/rafalkorepta/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.6.darwin-arm64/src/runtime/asm_arm64.s:1222
        	            	Wraps: (2) stderr: Error: INSTALLATION FAILED: context deadline exceeded
        	            	Wraps: (3) exit status 1
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *exec.ExitError
        	Test:       	TestSuite/Managing_Users

Process finished with the exit code 1
=== RUN   TestSuite/Managing_Authentication-only_Users
    testingt.go:120: Checking cluster "sasl" is ready
    testingt.go:120: 
        	Error Trace:	/Users/rafalkorepta/workspace/redpanda/redpanda-operator/acceptance/steps/cluster.go:31
        	            				/Users/rafalkorepta/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.6.darwin-arm64/src/runtime/asm_arm64.s:1222
        	Error:      	Received unexpected error:
        	            	redpandas.cluster.redpanda.com "sasl" not found
        	Test:       	TestSuite/Managing_Authentication-only_Users
panic: FailNow or SkipNow called

goroutine 79 [running]:
github.com/cucumber/godog.(*testingT).FailNow(0x105046f20?)
	/Users/rafalkorepta/go/pkg/mod/github.com/cucumber/godog@v0.14.1/testingt.go:144 +0x44
github.com/redpanda-data/redpanda-operator/harpoon/internal/testing.(*TestingT).FailNow(0x12e7fe578?)
	/Users/rafalkorepta/workspace/redpanda/redpanda-operator/harpoon/internal/testing/testing.go:137 +0x34
github.com/stretchr/testify/require.NoError({0x1053c8070, 0x14000724e00}, {0x1053bce00, 0x140002e1720}, {0x0, 0x0, 0x0})
	/Users/rafalkorepta/go/pkg/mod/github.com/stretchr/testify@v1.9.0/require/require.go:1357 +0xcc
github.com/redpanda-data/redpanda-operator/acceptance/steps.checkClusterAvailability.func1()
	/Users/rafalkorepta/workspace/redpanda/redpanda-operator/acceptance/steps/cluster.go:31 +0xc8
github.com/stretchr/testify/assert.Eventually.func1()
	/Users/rafalkorepta/go/pkg/mod/github.com/stretchr/testify@v1.9.0/assert/assertions.go:1902 +0x30
created by github.com/stretchr/testify/assert.Eventually in goroutine 100
	/Users/rafalkorepta/go/pkg/mod/github.com/stretchr/testify@v1.9.0/assert/assertions.go:1902 +0x1c4

In kind cluster I didn't have container image localhost/redpanda-operator:dev. Operator Pod is currently in ImagePullBackOff phase. The helm uninstall nor kubectl delete ns ... was not performed.

andrewstucki · 2024-09-12T20:37:22Z

@RafalKorepta yeah, that's one more scenario I still need to handle. It looks like in the godog internals, when you call t.FailNow it calls panic -- I think I'll try and add a recover handler that exits gracefully and does as much cleanup as possible around the scenarios, just kind of wonky that they don't do that by default 🙄

chrisseto · 2024-09-13T13:46:12Z

I'm fine with either of these. Do you think it's worth just dropping the build workflows for now and following up with a separate CI PR?

Yeah that works for me! Smaller PRs are a lot easier to review 😅

Looks like the primary pain point of k3d is going to be the lack of the log export command.

As for GHA vs buildkite no strong feelings but thus far GHA has been a bit more pleasant to work with. On the other hand, the magic of actions makes it much more difficult to run the tasks that CI does locally for debugging and/or pre-checks.

andrewstucki · 2024-09-16T14:12:56Z

I'm going to close this and break this PR into 2-3 parts so we can get the core of this merged a bit quicker.

andrewstucki requested review from RafalKorepta and chrisseto as code owners September 11, 2024 14:21

andrewstucki force-pushed the acceptance-testing-split branch from f19cfd6 to c5ec10e Compare September 11, 2024 14:24

andrewstucki force-pushed the acceptance-testing-split-2 branch from fb986b1 to 26d7e3d Compare September 11, 2024 14:29

andrewstucki commented Sep 11, 2024

View reviewed changes

andrewstucki force-pushed the acceptance-testing-split branch from c5ec10e to 85441f9 Compare September 11, 2024 14:40

andrewstucki force-pushed the acceptance-testing-split-2 branch from 26d7e3d to bec2af9 Compare September 11, 2024 14:42

andrewstucki force-pushed the acceptance-testing-split branch from 85441f9 to c43ef96 Compare September 11, 2024 18:10

andrewstucki force-pushed the acceptance-testing-split-2 branch from bec2af9 to f859af4 Compare September 11, 2024 18:11

RafalKorepta reviewed Sep 12, 2024

View reviewed changes

RafalKorepta mentioned this pull request Sep 12, 2024

Add acceptance testing framework #225

Merged

chrisseto reviewed Sep 12, 2024

View reviewed changes

RafalKorepta reviewed Sep 12, 2024

View reviewed changes

Base automatically changed from acceptance-testing-split to main September 12, 2024 21:46

Add acceptance tests

45a980b

andrewstucki force-pushed the acceptance-testing-split-2 branch from f859af4 to 45a980b Compare September 13, 2024 19:23

andrewstucki added 2 commits September 13, 2024 23:43

Add panic handling for requires

de7481b

Fix up panic handling a bit more

df91796

andrewstucki mentioned this pull request Sep 16, 2024

Fix up use of helmette.SafeLookup calls redpanda-data/helm-charts#1526

Merged

andrewstucki closed this Sep 16, 2024

RafalKorepta deleted the acceptance-testing-split-2 branch December 2, 2024 13:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add acceptance tests for User CRD #226

Add acceptance tests for User CRD #226

andrewstucki commented Sep 11, 2024 •

edited

Loading

andrewstucki Sep 11, 2024

RafalKorepta left a comment •

edited

Loading

chrisseto left a comment

chrisseto Sep 12, 2024

chrisseto Sep 12, 2024

chrisseto Sep 12, 2024

chrisseto Sep 12, 2024

chrisseto Sep 12, 2024

andrewstucki commented Sep 12, 2024

andrewstucki commented Sep 12, 2024

RafalKorepta left a comment

andrewstucki commented Sep 12, 2024

chrisseto commented Sep 13, 2024

andrewstucki commented Sep 16, 2024


		import "github.com/redpanda-data/redpanda-operator/harpoon/tablegenerator"

		var providers = []string{"eks", "aks", "gke", "kind"}

Add acceptance tests for User CRD #226

Add acceptance tests for User CRD #226

Conversation

andrewstucki commented Sep 11, 2024 • edited Loading

andrewstucki Sep 11, 2024

Choose a reason for hiding this comment

RafalKorepta left a comment • edited Loading

Choose a reason for hiding this comment

chrisseto left a comment

Choose a reason for hiding this comment

chrisseto Sep 12, 2024

Choose a reason for hiding this comment

chrisseto Sep 12, 2024

Choose a reason for hiding this comment

chrisseto Sep 12, 2024

Choose a reason for hiding this comment

chrisseto Sep 12, 2024

Choose a reason for hiding this comment

chrisseto Sep 12, 2024

Choose a reason for hiding this comment

andrewstucki commented Sep 12, 2024

andrewstucki commented Sep 12, 2024

RafalKorepta left a comment

Choose a reason for hiding this comment

andrewstucki commented Sep 12, 2024

chrisseto commented Sep 13, 2024

andrewstucki commented Sep 16, 2024

andrewstucki commented Sep 11, 2024 •

edited

Loading

RafalKorepta left a comment •

edited

Loading