Skip to content

Commit

Permalink
Auto provisioning of PW clusters from GitHub CI added (#3051)
Browse files Browse the repository at this point in the history
# Description

This update to the GitHub dispatched CI pipeline to execute the
self-hosted GitHub Runner on Parallel Works now adds the feature that
starts up the virtual compute cluster automatically. We now have a
complete end-to-end automated process for running CI tests in Parallel
Works.

Next steps would be tear-down and adding more test to see if it scales.

It also has the update for getting a PR to load up when its originating
from a forked repo.

# Type of change
- [ ] Bug fix (fixes something broken)
- [x] New feature (adds functionality)
- [ ] Maintenance (code refactor, clean-up, new CI test, etc.)

# Change characteristics
<!-- Choose YES or NO from each of the following and delete the other
-->
- Is this a breaking change (a change in existing functionality)? NO
- Does this change require a documentation update? YES
- Does this change require an update to any of the following submodules?
NO (If YES, please add a link to any PRs that are pending.)
  - [ ] EMC verif-global <!-- NOAA-EMC/EMC_verif-global#1234 -->
  - [ ] GDAS <!-- NOAA-EMC/GDASApp#1234 -->
  - [ ] GFS-utils <!-- NOAA-EMC/gfs-utils#1234 -->
  - [ ] GSI <!-- NOAA-EMC/GSI#1234 -->
  - [ ] GSI-monitor <!-- NOAA-EMC/GSI-Monitor#1234 -->
  - [ ] GSI-utils <!-- NOAA-EMC/GSI-Utils#1234 -->
  - [ ] UFS-utils <!-- ufs-community/UFS_UTILS#1234 -->
  - [ ] UFS-weather-model <!-- ufs-community/ufs-weather-model#1234 -->
  - [ ] wxflow <!-- NOAA-EMC/wxflow#1234 -->

# How has this been tested?

The start up aspected has been tested from my forked repo but could not
test repos that are forked.
The test from forked repos has to be tested once the workflow pipeline
in the **develop** branch.

# Checklist
- [x] Any dependent changes have been merged and published
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have documented my code, including function, input, and output
descriptions
- [x] My changes generate no new warnings
- [x] New and existing tests pass with my changes
- [x] This change is covered by an existing CI test or a new one has
been added
- [ ] Any new scripts have been added to the .github/CODEOWNERS file
with owners
- [ ] I have made corresponding changes to the system documentation if
necessary

---------

Co-authored-by: tmcguinness <terry.mcguinness@noaa.gov>
Co-authored-by: tmcguinness <tmcguinness@129.qarestr.sub-172-16-82.myvzw.com>
  • Loading branch information
3 people authored Oct 31, 2024
1 parent 5a8a5aa commit ca8cd7a
Showing 1 changed file with 35 additions and 1 deletion.
36 changes: 35 additions & 1 deletion .github/workflows/pw_aws_ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -31,24 +31,57 @@ env:
MACHINE_ID: noaacloud

jobs:

run-start-clusters:
runs-on: ubuntu-latest
env:
PW_PLATFORM_HOST: noaa.parallel.works
steps:
- name: Checkout pw-cluster-automation repository
uses: actions/checkout@v4
with:
repository: TerrenceMcGuinness-NOAA/pw-cluster-automation
path: pw-cluster-automation
ref: pw_cluster_noaa

- name: Run startClusters
run: |
mkdir -p ~/.ssh
echo "${{ secrets.ID_RSA_AWS }}" > ~/.ssh/id_rsa
echo "${{ secrets.PW_API_KEY }}" > ~/.ssh/pw_api.key
chmod 700 ~/.ssh
chmod 600 ~/.ssh/id_rsa
chmod 600 ~/.ssh/pw_api.key
if [ "${{ github.event.inputs.os }}" == "rocky" ]; then
clustername="globalworkflowciplatformrocky8"
elif [ "${{ github.event.inputs.os }}" == "centos" ]; then
clustername="awsemctmcgc7i48xlargeciplatform"
fi
python3 pw-cluster-automation/startClusters.py $clustername
fetch-branch:
needs: run-start-clusters
runs-on: ubuntu-latest
env:
GH_TOKEN: ${{ secrets.GITHUBTOKEN }}
outputs:
branch: ${{ steps.get-branch.outputs.branch }}
repo: ${{ steps.get-branch.outputs.repo }}
steps:
- name: Fetch branch name for PR
- name: Fetch branch name and repo for PR
id: get-branch
run: |
pr_number=${{ github.event.inputs.pr_number }}
repo=${{ github.repository }}
if [ "$pr_number" -eq "0" ]; then
branch=${{ github.event.inputs.ref }}
repo_url="https://github.com/${{ github.repository_owner }}/${{ github.repository }}.git"
else
branch=$(gh pr view $pr_number --repo $repo --json headRefName --jq '.headRefName')
repo_url=$(gh pr view $pr_number --repo $repo --json headRepository --jq '.headRepository.url')
fi
echo "::set-output name=branch::$branch"
echo "::set-output name=repo::$repo_url"
checkout:
needs: fetch-branch
Expand All @@ -64,6 +97,7 @@ jobs:
with:
path: ${{ github.run_id }}/HOMEgfs
submodules: 'recursive'
repository: ${{ needs.fetch-branch.outputs.repo }}
ref: ${{ needs.fetch-branch.outputs.branch }}

build-link:
Expand Down

0 comments on commit ca8cd7a

Please sign in to comment.