Refactor ci #2137

davidcassany · 2024-07-09T23:28:26Z

Fixes #2132

Signed-off-by: David Cassany <dcassany@suse.com>

davidcassany

This PR allows to run the upgrade test locally too. In fact, that's the thing, if it is runnable locally without having to configure infraestructure, it should also be possible to run it in GHA without needing external services like a registry.

In my env the following sequences passes:

make build && make build-os
make test-upgrade

davidcassany · 2024-07-10T09:54:21Z

.github/workflows/build.yaml

    steps:
      - uses: actions/checkout@v4
        with:
          ref: "${{ github.event.pull_request.head.sha }}"
      - run: |
          git fetch --prune --unshallow
-      - name: Log in to ghcr.io
-        uses: docker/login-action@v3
+      - name: Define version


Use version identifier and tag base on a hash from certain paths. This allows to re-use cache in case of PRs that are only touching parts that do not affect the build (e.g. tests code, workflows, etc.)

davidcassany · 2024-07-10T10:01:05Z

scripts/run_registry.sh

@@ -0,0 +1,78 @@
+#!/bin/bash  


This is a script that handles the local registry instance with a variant of docker run registry:2. Challenges here were:

Need to compute a valid IP of the host that is visible from the qemu VM in "user mode" network setup.

Need to push to an insecure registry (I could not figure out how work around that, a self signed certificate would still be insecure unless the host is configured to trust it)

scripts/run_registry.sh

davidcassany · 2024-07-10T10:09:30Z

scripts/run_upgradetest.sh

+  pushd "${ROOT_PATH}" > /dev/null
+    go run ${ginkgo} ${ginkgo_args} ./tests/wait-active
+    go run ${ginkgo} ${ginkgo_args} ./tests/upgrade -- \
+      --toolkit-image=docker://${reg_url}/${toolkit_img} --upgrade-image=docker://${reg_url}/${upgrade_img} 


This scripts is essentially existing because it was so cumbersome to compose this arguments from variables in the Makefile directly

davidcassany · 2024-07-10T10:10:50Z

.github/workflows/build_and_test_x86.yaml

@@ -14,31 +17,47 @@ concurrency:
 jobs:
  build-os:
    permissions:
-      packages: write


No more packages: write in this workflow. Only contents: read.

davidcassany · 2024-07-10T14:20:28Z

.github/workflows/cache-cleanup.yaml

@@ -5,6 +5,7 @@ name: Cleanup caches by a branch
 on:
  pull_request_target:
    types:
+      - synchronize


To remove unused cache on pushing commits to an opened PR

scripts/run_registry.sh

anmazzotti

Great work. Left a couple of nitpicking comments, but nothing essential.
LGTM

Signed-off-by: David Cassany <dcassany@suse.com>

* Refactor CI to use a local registry * Cache image and fetch image tarballs * Composite fetch images * Push to insecure registry with podman * Simplify workflow runs * Improve cache cleanup * Improve ip detection Signed-off-by: David Cassany <dcassany@suse.com> (cherry picked from commit 644cdc7)

* Fix stackable sources for build-iso (#2061) * Fix stackable sources for build-iso * Move syncFunc as a parameter of DumpSource Signed-off-by: David Cassany <dcassany@suse.com> (cherry picked from commit 22baf14) * Fix error return Signed-off-by: David Cassany <dcassany@suse.com> (cherry picked from commit f9934d7) * Change default root password Signed-off-by: David Cassany <dcassany@suse.com> (cherry picked from commit 46a88e1) * Allow key pair of ssh connection in tests Signed-off-by: David Cassany <dcassany@suse.com> (cherry picked from commit a5f526d) * Set legacy password for upgrade test Signed-off-by: David Cassany <dcassany@suse.com> (cherry picked from commit 6f0b28e) * Allow insecure registries (#2131) * Allow skipping TLS verification on registries Signed-off-by: David Cassany <dcassany@suse.com> * Improve error logging Signed-off-by: David Cassany <dcassany@suse.com> * Fix flag default value Signed-off-by: David Cassany <dcassany@suse.com> * Fix image extractor mock and use default TLS verification for tests Signed-off-by: David Cassany <dcassany@suse.com> --------- Signed-off-by: David Cassany <dcassany@suse.com> (cherry picked from commit 22996ef) * Fix permissions * Set token to explicit content read permissions (cherry picked from commit 3d657c9) * Move permissions at top level (cherry picked from commit 8b8ae12) * Fix workflows, permissions at top level (cherry picked from commit d3da1cf) * Leftover from d3da1cf (cherry picked from commit 3206436) * Fix nested permissions definition Top level permissions can't be increased for specific jobs, hence setting the permission on each specific job on workflows that require more fine grain approach. In addition it removes the pull-request permission as this is mostly required for PR decorators, read/write labels, etc. (cherry picked from commit 1a0903a) * Refactor ci (#2137) * Refactor CI to use a local registry * Cache image and fetch image tarballs * Composite fetch images * Push to insecure registry with podman * Simplify workflow runs * Improve cache cleanup * Improve ip detection Signed-off-by: David Cassany <dcassany@suse.com> (cherry picked from commit 644cdc7) * Fix cache cleaning for PRs and main pushes (#2144) * Fix cache cleaning for PRs and main pushes * Update .github/workflows/cache-cleanup.yaml Co-authored-by: Francesco Giudici <francesco.giudici@gmail.com> Signed-off-by: David Cassany Viladomat <dcassany@suse.com> (cherry picked from commit f54f683) * Remove unused steps in workflow Signed-off-by: David Cassany <dcassany@suse.com> (cherry picked from commit 51a85a7) (cherry picked from commit 9f8d14fea58c2e458f8840d274422352f0b0b1c2) * Do not fetch upstream go dependencies Signed-off-by: David Cassany <dcassany@suse.com> * Update cmd/cloud-init.go Missing err Co-authored-by: Francesco Giudici <francesco.giudici@gmail.com> Signed-off-by: David Cassany Viladomat <dcassany@suse.com> --------- Signed-off-by: David Cassany <dcassany@suse.com> Signed-off-by: David Cassany Viladomat <dcassany@suse.com> Co-authored-by: Francesco Giudici <francesco.giudici@gmail.com>

davidcassany requested a review from a team as a code owner July 9, 2024 23:28

davidcassany added 5 commits July 10, 2024 01:39

Refactor CI to use a local registry

ef79c4c

Signed-off-by: David Cassany <dcassany@suse.com>

Cache image and fetch image tarballs

90eebf7

Signed-off-by: David Cassany <dcassany@suse.com>

Composite fetch images

93c2b5d

Signed-off-by: David Cassany <dcassany@suse.com>

Push to insecure registry with podman

6dc6518

Signed-off-by: David Cassany <dcassany@suse.com>

Simplify workflow runs

6d12f5a

Signed-off-by: David Cassany <dcassany@suse.com>

davidcassany force-pushed the refactor_ci branch 4 times, most recently from a32ef58 to 1215a59 Compare July 10, 2024 09:14

Improve cache cleanup

edf8382

Signed-off-by: David Cassany <dcassany@suse.com>

davidcassany force-pushed the refactor_ci branch from 1215a59 to edf8382 Compare July 10, 2024 09:47

davidcassany commented Jul 10, 2024

View reviewed changes

davidcassany self-assigned this Jul 10, 2024

davidcassany commented Jul 10, 2024

View reviewed changes

anmazzotti reviewed Jul 11, 2024

View reviewed changes

scripts/run_registry.sh Outdated Show resolved Hide resolved

anmazzotti approved these changes Jul 11, 2024

View reviewed changes

davidcassany force-pushed the refactor_ci branch 2 times, most recently from 67bbc24 to f7c4f7f Compare July 11, 2024 09:16

Improve ip detection

dbb8983

Signed-off-by: David Cassany <dcassany@suse.com>

davidcassany force-pushed the refactor_ci branch from f7c4f7f to dbb8983 Compare July 11, 2024 09:21

davidcassany enabled auto-merge (squash) July 11, 2024 09:25

davidcassany merged commit 644cdc7 into rancher:main Jul 11, 2024
29 of 32 checks passed

davidcassany deleted the refactor_ci branch July 11, 2024 14:55

davidcassany mentioned this pull request Jul 15, 2024

Fix cache cleaning for PRs and main pushes #2144

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor ci #2137

Refactor ci #2137

davidcassany commented Jul 9, 2024

davidcassany left a comment

davidcassany Jul 10, 2024

davidcassany Jul 10, 2024

davidcassany Jul 10, 2024

davidcassany Jul 10, 2024

davidcassany Jul 10, 2024

anmazzotti left a comment

Refactor ci #2137

Refactor ci #2137

Conversation

davidcassany commented Jul 9, 2024

davidcassany left a comment

Choose a reason for hiding this comment

davidcassany Jul 10, 2024

Choose a reason for hiding this comment

davidcassany Jul 10, 2024

Choose a reason for hiding this comment

davidcassany Jul 10, 2024

Choose a reason for hiding this comment

davidcassany Jul 10, 2024

Choose a reason for hiding this comment

davidcassany Jul 10, 2024

Choose a reason for hiding this comment

anmazzotti left a comment

Choose a reason for hiding this comment