Add native registry login tests for EKS, AKS and GKE #275

darkowlzz · 2022-06-21T12:06:20Z

It introduces a new package tftestenv. It is similar to testenv but uses
terraform to create test environment. It provides helpers to run terraform
apply and destroy in the test context.

tftestenv is used to create tests similar to the controller tests but with
cloud provider managed cluster provisioned using terraform. The test
setup is written similar to the usual controller test suite_test.go,
providing flexibility in the test setup and execution. The structure tries
to keep the infrastructure code separate from the test code such that
the infrastructure provisioning tool can be easily replaced, without the
need to change the actual tests.

The go tests are also written similar to the usual controller tests,
with a kube client from the tftestenv environment global variable.

Terraform is automatically downloaded using the terraform-exec package.
The test container images are generated using the go-containerregistry
package and pushed to the test container image repository.

tests/integration/README.md contains more details about the test setup and workflow.

The terraform configurations for EKS is based on https://github.com/terraform-aws-modules/terraform-aws-eks/tree/6bb73bbdd469797cb067e9aa22ce2f31a7fb22be/examples/self_managed_node_group .

The terraform configurations for AKS is based on https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/kubernetes_cluster .

The terraform configurations for GKE is based on the official GKE terraform provider examples https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/container_cluster with Terraform Kubernetes Engine Auth Module to be able to easily retrieve the kubeconfig. Both GCR and Artifact Registry are tested.

Average test duration including provisioning and cleanup time:

EKS - 25-30 minutes
AKS - ~13 minutes
GKE - ~9 minutes

We may be able to optimize some configurations to bring down the time of EKS and AKS in the future.

Part of #264.

darkowlzz · 2022-06-22T22:24:30Z

Posting some results to provide an idea of what it looks like when run.

With .env

# AWS
export AWS_ACCESS_KEY_ID=<redacted>
export AWS_SECRET_ACCESS_KEY=<redacted>
export AWS_REGION=us-east-2

# Azure
export TF_VAR_azure_location=eastus

# GCP
export TF_VAR_gcp_project_id=<redacted>
# export TF_VAR_gcp_region=us-central1
# export TF_VAR_gcp_zone=us-central1-c
# Leave GCR region empty to use gcr.io. Else set it to `us`, `eu` or `asia`.
# export TF_VAR_gcr_region=

and az login and gcloud auth login being already run:

aws:

$ make test-aws 
make test PROVIDER_ARG="-provider aws"
make[1]: Entering directory '/home/go/src/github.com/fluxcd/image-reflector-controller/tests/integration'
cp kustomization.yaml build/flux
cd build/flux && kustomize edit set image fluxcd/image-reflector-controller=fluxcd/image-reflector-controller
kustomize build build/flux > build/flux.yaml
go test -timeout 30m -v ./...  -provider aws
2022/06/23 02:53:59 Terraform binary:  /usr/bin/terraform
2022/06/23 02:53:59 Init Terraform
2022/06/23 02:54:05 Checking for an empty Terraform state
2022/06/23 02:54:07 Applying Terraform
2022/06/23 03:08:34 pushing test image z.dkr.ecr.us-east-2.amazonaws.com/flux-test-repo-peaceful-ghost:v0.1.0
2022/06/23 03:08:38 pushing test image z.dkr.ecr.us-east-2.amazonaws.com/flux-test-repo-peaceful-ghost:v0.1.2
2022/06/23 03:08:41 pushing test image z.dkr.ecr.us-east-2.amazonaws.com/flux-test-repo-peaceful-ghost:v0.1.3
2022/06/23 03:08:44 pushing test image z.dkr.ecr.us-east-2.amazonaws.com/flux-test-repo-peaceful-ghost:v0.1.4
2022/06/23 03:08:47 Installing flux
=== RUN   TestImageRepositoryScan
=== RUN   TestImageRepositoryScan/ecr
--- PASS: TestImageRepositoryScan (2.01s)
    --- PASS: TestImageRepositoryScan/ecr (2.01s)
PASS
2022/06/23 03:09:20 Destroying environment...
ok      github.com/fluxcd/image-reflector-controller/tests/integration  1592.611s
make[1]: Leaving directory '/home/go/src/github.com/fluxcd/image-reflector-controller/tests/integration'

azure:

$ make test-azure 
make test PROVIDER_ARG="-provider azure"
make[1]: Entering directory '/home/go/src/github.com/fluxcd/image-reflector-controller/tests/integration'
cp kustomization.yaml build/flux
cd build/flux && kustomize edit set image fluxcd/image-reflector-controller=fluxcd/image-reflector-controller
kustomize build build/flux > build/flux.yaml
go test -timeout 30m -v ./...  -provider azure
2022/06/23 03:23:50 Terraform binary:  /usr/bin/terraform
2022/06/23 03:23:50 Init Terraform
2022/06/23 03:23:52 Checking for an empty Terraform state
2022/06/23 03:23:52 Applying Terraform
2022/06/23 03:29:34 pushing test image fluxtestrepomovingotick.azurecr.io/vynz8:v0.1.0
2022/06/23 03:29:38 pushing test image fluxtestrepomovingotick.azurecr.io/vynz8:v0.1.2
2022/06/23 03:29:40 pushing test image fluxtestrepomovingotick.azurecr.io/vynz8:v0.1.3
2022/06/23 03:29:42 pushing test image fluxtestrepomovingotick.azurecr.io/vynz8:v0.1.4
2022/06/23 03:29:45 Installing flux
=== RUN   TestImageRepositoryScan
=== RUN   TestImageRepositoryScan/acr
--- PASS: TestImageRepositoryScan (3.36s)
    --- PASS: TestImageRepositoryScan/acr (3.36s)
PASS
2022/06/23 03:30:16 Destroying environment...
ok      github.com/fluxcd/image-reflector-controller/tests/integration  749.433s
make[1]: Leaving directory '/home/go/src/github.com/fluxcd/image-reflector-controller/tests/integration'

gcp:

$ make test-gcp 
make test PROVIDER_ARG="-provider gcp"
make[1]: Entering directory '/home/go/src/github.com/fluxcd/image-reflector-controller/tests/integration'
cp kustomization.yaml build/flux
cd build/flux && kustomize edit set image fluxcd/image-reflector-controller=fluxcd/image-reflector-controller
kustomize build build/flux > build/flux.yaml
go test -timeout 30m -v ./...  -provider gcp
2022/06/23 03:39:18 Terraform binary:  /usr/bin/terraform
2022/06/23 03:39:18 Init Terraform
2022/06/23 03:39:22 Checking for an empty Terraform state
2022/06/23 03:39:23 Applying Terraform
2022/06/23 03:44:37 pushing test image gcr.io/z/ik9ar:v0.1.0
2022/06/23 03:44:46 pushing test image gcr.io/z/ik9ar:v0.1.2
2022/06/23 03:44:54 pushing test image gcr.io/z/ik9ar:v0.1.3
2022/06/23 03:45:02 pushing test image gcr.io/z/ik9ar:v0.1.4
2022/06/23 03:45:10 pushing test image us-central1-docker.pkg.dev/z/flux-test-repo-helpful-kangaroo/8y697:v0.1.0
2022/06/23 03:45:19 pushing test image us-central1-docker.pkg.dev/z/flux-test-repo-helpful-kangaroo/8y697:v0.1.2
2022/06/23 03:45:26 pushing test image us-central1-docker.pkg.dev/z/flux-test-repo-helpful-kangaroo/8y697:v0.1.3
2022/06/23 03:45:32 pushing test image us-central1-docker.pkg.dev/z/flux-test-repo-helpful-kangaroo/8y697:v0.1.4
2022/06/23 03:45:38 Installing flux
=== RUN   TestImageRepositoryScan
=== RUN   TestImageRepositoryScan/gcr
=== RUN   TestImageRepositoryScan/artifact_registry
--- PASS: TestImageRepositoryScan (9.11s)
    --- PASS: TestImageRepositoryScan/gcr (8.29s)
    --- PASS: TestImageRepositoryScan/artifact_registry (0.82s)
PASS
2022/06/23 03:46:17 Destroying environment...
ok      github.com/fluxcd/image-reflector-controller/tests/integration  568.476s
make[1]: Leaving directory '/home/go/src/github.com/fluxcd/image-reflector-controller/tests/integration'

tests/integration/.env.sample

tests/integration/azure_test.go

tests/integration/gcp_test.go

tests/integration/README.md

pjbgf

Small question, otherwise LGTM.

phillebaba · 2022-07-06T12:18:09Z

This looks really good but I think some of these things would be good to share with the Azure E2E tests. Maybe we should move some of the packages to pkg instead?

darkowlzz · 2022-07-06T12:41:31Z

This looks really good but I think some of these things would be good to share with the Azure E2E tests. Maybe we should move some of the packages to pkg instead?

@phillebaba yes, that's the plan after getting these changes reviewed, accepted and merged here initially.

pjbgf

Great job @darkowlzz!

LGTM!

phillebaba · 2022-07-06T13:56:16Z

tests/integration/terraform/azure/aks.tf

+  role_based_access_control_enabled = true
+  network_profile {
+    network_plugin = "kubenet"
+    network_policy = "calico"


Any reason for choosing kubenet and calico for Azure CNI? Don't think it really matters but the default value when creating and AKS cluster is to use Azure CNI.

Azure CNI does not work with Flux network policies, using Calico matches flux2 e2e tests.

phillebaba

Small NIT but otherwise great!

tftestenv is similar to testenv but uses terraform to create test environment. It provides helpers to run terraform apply and destroy in the test context and a few other helpers for executing commands. Signed-off-by: Sunny <darkowlzz@protonmail.com>

Use tftestenv to create tests similar to the controller tests but with EKS cluster provisioned using terraform. The test setup is written similar to the usual controller test suite_test.go, providing flexibility in the test setup and execution. The structure tries to keep the infrastructure code separate from the test code such that the infrastructure provisioning tool can be easily replaced, without the need to change the actual tests. The go tests are also written similar to the usual controller tests, with a kube client from the tftestenv environment global variable. Terraform is automatically downloaded using the terraform-exec package. The test container images are generated using the go-containerregistry package and pushed to the test container image repository. Co-authored-by: Somtochi Onyekwere <somtochionyekwere@gmail.com> Signed-off-by: Sunny <darkowlzz@protonmail.com>

Add azure as a new provider to provision infrastructure and test against. Unlike aws, azurerm terraform provider doesn't provide a way to natively configure the region/location. To work around that, a terraform variable `azure_location` is used, which can be configured by environment variable `TF_VAR_azure_location`, added in .env.sample. Unlike AWS ECR, Azure ACR requires creating a new container registry which creates dynamic repositories on push, similar to docker hub. The registry login code handles that and sets randomly generated repository names. Co-authored-by: Somtochi Onyekwere <somtochionyekwere@gmail.com> Signed-off-by: Sunny <darkowlzz@protonmail.com>

Add GCP as a new provider to provision infrastructure and test against. GCP has Google Container Registry and Artifact Registry for storing images. The test setup provisions and pushes images to both and tests them both. The GKE node pool is configured with OAuth Scope to provide full access to the GCP APIs. Workload identity is not used to avoid increasing the complixity of the setup. Co-authored-by: Somtochi Onyekwere <somtochionyekwere@gmail.com> Signed-off-by: Sunny <darkowlzz@protonmail.com>

darkowlzz · 2022-07-07T10:48:26Z

Rebased and verified that it works with go 1.18.

This was referenced Jun 21, 2022

Add native registry login e2e test for EKS #273

Closed

Extend E2E tests to accept provider flag #274

Closed

Introduce registry package #276

Merged

darkowlzz force-pushed the cloud-e2e-test branch 3 times, most recently from 7804f22 to ce83335 Compare June 22, 2022 21:24

darkowlzz changed the title ~~Add native registry login tests for EKS and AKS~~ Add native registry login tests for EKS, AKS and GKE Jun 22, 2022

darkowlzz mentioned this pull request Jun 22, 2022

Add e2e test for GCP Autologin #277

Closed

darkowlzz force-pushed the cloud-e2e-test branch from ce83335 to 640d2d9 Compare June 22, 2022 21:52

darkowlzz force-pushed the cloud-e2e-test branch from 640d2d9 to 780bf84 Compare June 24, 2022 12:21

pjbgf added this to the GA milestone Jun 27, 2022

pjbgf reviewed Jun 28, 2022

View reviewed changes

tests/integration/.env.sample Outdated Show resolved Hide resolved

pjbgf reviewed Jun 28, 2022

View reviewed changes

tests/integration/azure_test.go Outdated Show resolved Hide resolved

pjbgf reviewed Jun 28, 2022

View reviewed changes

tests/integration/gcp_test.go Outdated Show resolved Hide resolved

pjbgf reviewed Jun 28, 2022

View reviewed changes

tests/integration/gcp_test.go Outdated Show resolved Hide resolved

darkowlzz force-pushed the cloud-e2e-test branch from 780bf84 to 3a2b472 Compare June 28, 2022 10:39

darkowlzz mentioned this pull request Jun 28, 2022

[OCIRepository] GCP Artifact Registry Auth - Helm OCI 403 error fluxcd/source-controller#798

Closed

pjbgf reviewed Jun 29, 2022

View reviewed changes

tests/integration/README.md Outdated Show resolved Hide resolved

pjbgf reviewed Jun 29, 2022

View reviewed changes

darkowlzz force-pushed the cloud-e2e-test branch from 3a2b472 to de5d739 Compare July 1, 2022 12:17

pjbgf approved these changes Jul 6, 2022

View reviewed changes

phillebaba reviewed Jul 6, 2022

View reviewed changes

phillebaba self-requested a review July 6, 2022 13:56

phillebaba approved these changes Jul 6, 2022

View reviewed changes

darkowlzz and others added 3 commits July 7, 2022 15:09

Add tftestenv package

b90a80e

tftestenv is similar to testenv but uses terraform to create test environment. It provides helpers to run terraform apply and destroy in the test context and a few other helpers for executing commands. Signed-off-by: Sunny <darkowlzz@protonmail.com>

darkowlzz force-pushed the cloud-e2e-test branch from de5d739 to b921709 Compare July 7, 2022 10:47

hiddeco approved these changes Jul 7, 2022

View reviewed changes

darkowlzz merged commit e348e90 into main Jul 7, 2022

darkowlzz deleted the cloud-e2e-test branch July 7, 2022 12:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add native registry login tests for EKS, AKS and GKE #275

Add native registry login tests for EKS, AKS and GKE #275

darkowlzz commented Jun 21, 2022 •

edited

Loading

darkowlzz commented Jun 22, 2022

pjbgf left a comment

phillebaba commented Jul 6, 2022

darkowlzz commented Jul 6, 2022

pjbgf left a comment

phillebaba Jul 6, 2022

stefanprodan Jul 6, 2022

phillebaba left a comment

darkowlzz commented Jul 7, 2022

Add native registry login tests for EKS, AKS and GKE #275

Add native registry login tests for EKS, AKS and GKE #275

Conversation

darkowlzz commented Jun 21, 2022 • edited Loading

darkowlzz commented Jun 22, 2022

pjbgf left a comment

Choose a reason for hiding this comment

phillebaba commented Jul 6, 2022

darkowlzz commented Jul 6, 2022

pjbgf left a comment

Choose a reason for hiding this comment

phillebaba Jul 6, 2022

Choose a reason for hiding this comment

stefanprodan Jul 6, 2022

Choose a reason for hiding this comment

phillebaba left a comment

Choose a reason for hiding this comment

darkowlzz commented Jul 7, 2022

darkowlzz commented Jun 21, 2022 •

edited

Loading