[1.30] Refactor build process to enable auto-update of components (#419)

* Refactor build process to enable auto-update of components (#416)

* use release-1.30 branch for kubernetes and k8s-dqlite

.github/ISSUE_TEMPLATE/create_release_branch.md

@@ -20,6 +20,9 @@ Make sure to follow the steps below and ensure all actions are completed and sig
 
 <!-- Link to PR to initialize the release branch (see below) -->
 - **PR**:
+-
+<!-- Link to PR to initialize auto-update job for the release branch (see below) -->
+- **PR**:
 
 #### Actions
 
@@ -94,12 +97,16 @@ The steps are to be followed in-order, each task must be completed by the person
   - `rm -rf ~/tmp/release-1.xx`
 - [ ] **Reviewer**: Ensure `release-1.xx` branch is based on latest changes on `main` at the time of the release cut.
 - [ ] **Owner**: Create PR to initialize `release-1.xx` branch:
-  - [ ] Update `KUBE_TRACK` to `1.xx` in [/build-scripts/components/kubernetes/version.sh][]
-  - [ ] Update `master` to `release-1.xx` in [/build-scripts/components/k8s-dqlite/version.sh][]
+  - [ ] Update `KUBERNETES_RELEASE_MARKER` to `stable-1.xx` in [/build-scripts/hack/update-component-versions.py][]
+  - [ ] Update `master` to `release-1.xx` in [/build-scripts/components/k8s-dqlite/version][]
   - [ ] Update `"main"` to `"release-1.xx"` in [/build-scripts/hack/generate-sbom.py][]
   - [ ] `git commit -m 'Release 1.xx'`
-  - [ ] Create PR with the changes and request review from **Reviewer**. Make sure to update the issue `Information` section with a link to the PR.
+  - [ ] Create PR against `release-1.xx` with the changes and request review from **Reviewer**. Make sure to update the issue `Information` section with a link to the PR.
 - [ ] **Reviewer**: Review and merge PR to initialize branch.
+- [ ] **Owner**: Create PR to initialize `update-components.yaml` job for `release-1.xx` branch:
+  - [ ] Add `release-1.xx` in [.github/workflows/update-components.yaml][]
+  - [ ] Remove unsupported releases from the list (if applicable, consult with **Reviewer**)
+  - [ ] Create PR against `main` with the changes and request review from **Reviewer**. Make sure to update the issue information with a link to the PR.
 - [ ] **Reviewer**: On merge, confirm [Auto-update strict branch] action runs to completion and that the `autoupdate/release-1.xx-strict` branch is created.
 - [ ] **Owner**: Create launchpad builders for `release-1.xx`
   - [ ] Go to [lp:k8s][] and do **Import now** to pick up all latest changes.
@@ -148,9 +155,10 @@ The steps are to be followed in-order, each task must be completed by the person
 [.github/workflows/python.yaml]: ../workflows/python.yaml
 [.github/workflows/sbom.yaml]: ../workflows/sbom.yaml
 [.github/workflows/strict-integration.yaml]: ../workflows/strict-integration.yaml
-[.github/workflows/strict.yaml]: ..workflows/strict.yaml
-[/build-scripts/components/kubernetes/version.sh]: ../../build-scripts/components/kubernetes/version.sh
-[/build-scripts/components/k8s-dqlite/version.sh]: ../../build-scripts/components/k8s-dqlite/version.sh
+[.github/workflows/strict.yaml]: ../workflows/strict.yaml
+[.github/workflows/update-components.yaml]: ../workflows/update-components.yaml
+[/build-scripts/components/hack/update-component-versions.py]: ../../build-scripts/components/hack/update-component-versions.py
+[/build-scripts/components/k8s-dqlite/version]: ../../build-scripts/components/k8s-dqlite/version
 [/build-scripts/hack/generate-sbom.py]: ../..//build-scripts/hack/generate-sbom.py
 [lp:k8s]: https://code.launchpad.net/~cdk8s/k8s/+git/k8s-snap
 [lp:k8s/+snaps]: https://launchpad.net/k8s/+snaps

.github/workflows/go.yaml

@@ -7,6 +7,7 @@ on:
       - autoupdate/strict
       - 'release-[0-9]+.[0-9]+'
       - 'autoupdate/release-[0-9]+.[0-9]+-strict'
+      - 'autoupdate/sync/+'
   pull_request:
 
 jobs:

.github/workflows/integration.yaml

@@ -7,6 +7,7 @@ on:
       - autoupdate/strict
       - 'release-[0-9]+.[0-9]+'
       - 'autoupdate/release-[0-9]+.[0-9]+-strict'
+      - 'autoupdate/sync/+'
   pull_request:
 
 jobs:

.github/workflows/python.yaml

@@ -7,6 +7,7 @@ on:
       - autoupdate/strict
       - 'release-[0-9]+.[0-9]+'
       - 'autoupdate/release-[0-9]+.[0-9]+-strict'
+      - 'autoupdate/sync/+'
   pull_request:
 
 jobs:

.github/workflows/sbom.yaml

@@ -7,6 +7,7 @@ on:
       - autoupdate/strict
       - 'release-[0-9]+.[0-9]+'
       - 'autoupdate/release-[0-9]+.[0-9]+-strict'
+      - 'autoupdate/sync/+'
   pull_request:
 
 jobs:

.github/workflows/strict-integration.yaml

@@ -5,6 +5,7 @@ on:
     branches:
       - main
       - 'release-[0-9]+.[0-9]+'
+      - 'autoupdate/sync/+'
   pull_request:
 
 jobs:

.github/workflows/update-components.yaml

@@ -0,0 +1,49 @@
+name: Check for component upgrades
+
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: "0 10 * * *"
+
+jobs:
+  update:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        branch:
+          # Keep main branch up to date
+          - main
+          # Supported stable release branches
+          - release-1.30
+
+    steps:
+      - name: Checking out repo
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ matrix.branch }}
+          ssh-key: ${{ secrets.DEPLOY_KEY_TO_UPDATE_STRICT_BRANCH }}
+
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.10'
+
+      - name: Install dependencies
+        run: |
+          pip3 install -r ./build-scripts/hack/requirements.txt
+
+      - name: Check for new component versions
+        run: |
+          ./build-scripts/hack/update-component-versions.py
+
+      - name: Create pull request
+        uses: peter-evans/create-pull-request@v6
+        with:
+          token: ${{ secrets.DEPLOY_KEY_TO_UPDATE_STRICT_BRANCH }}
+          commit-message: "[${{ matrix.branch }}] Update component versions"
+          title: "[${{ matrix.branch }}] Update component versions"
+          body: "[${{ matrix.branch }}] Update component versions"
+          branch: "autoupdate/sync/${{ matrix.branch }}"
+          delete-branch: true
+          base: ${{ matrix.branch }}

build-scripts/README.md

@@ -14,7 +14,7 @@ build-scripts/
     components/
         $component_name/
             repository              <-- git repository to clone
-            version.sh              <-- prints the repository tag or commit to checkout
+            version                 <-- repository tag or commit to checkout
             build.sh                <-- runs as `build.sh $output $version`
                                         first argument is the output directory where
                                         binaries should be placed, second is the component version
@@ -26,7 +26,7 @@ build-scripts/
 
 ## Applying patches
 
-Most K8s components are retrieved from an upstream source (specified in the `repository`), with a specific tag (specified in `version.sh`), have some patches applied to them (from the `patches/` directory) and are then built (using `build.sh`).
+Most K8s components are retrieved from an upstream source (specified in the `repository`), with a specific tag (specified in `version`), have some patches applied to them (from the `patches/` directory) and are then built (using `build.sh`).
 
 This section explains the directory format for the `patches` directory.
 

build-scripts/build-component.sh

@@ -13,7 +13,7 @@ COMPONENT_NAME="${1}"
 COMPONENT_DIRECTORY="${DIR}/components/${COMPONENT_NAME}"
 
 GIT_REPOSITORY="$(cat "${COMPONENT_DIRECTORY}/repository")"
-GIT_TAG="$("${COMPONENT_DIRECTORY}/version.sh")"
+GIT_TAG="$(cat "${COMPONENT_DIRECTORY}/version")"
 
 COMPONENT_BUILD_DIRECTORY="${BUILD_DIRECTORY}/${COMPONENT_NAME}"
 

build-scripts/components/cni/version

@@ -0,0 +1 @@
+v1.4.0

build-scripts/components/containerd/version

@@ -0,0 +1 @@
+v1.6.28

build-scripts/components/helm/version

@@ -0,0 +1 @@
+v3.14.2

build-scripts/components/k8s-dqlite/version

@@ -0,0 +1 @@
+release-1.30

build-scripts/components/kubernetes/version

@@ -0,0 +1 @@
+v1.30.0

build-scripts/components/runc/version

@@ -0,0 +1 @@
+v1.1.12

build-scripts/generate-bom.py

@@ -51,7 +51,7 @@ def _read_file(path: Path) -> str:
         component_dir = DIR / "components" / component
 
         try:
-            version = _parse_output([component_dir / "version.sh"])
+            version = _read_file(component_dir / "version")
             patches = _parse_output([sys.executable, DIR / "print-patches-for.py", component, version])
             clean_patches = []
             if patches: