Jenkins Declartive Pipeline for CI with gfs/gefs multibuilds#2246

Merged

WalterKolczynski-NOAA merged 472 commits intoNOAA-EMC:developfrom TerrenceMcGuinness-NOAA:jenkins-pipeline

Feb 8, 2024

+315-15

This pull request is big! We're only showing the most recent 250 commits

Commits on Jan 18, 2024

Update ci_utils.sh
TerrenceMcGuinness-NOAA
authored
working on state added BUILD semaphore file in sorc dir
tmcguinness
committed
reversed logic for BUILT file semaphore
tmcguinness
committed
added chmod x to sub scripts
tmcguinness
committed
needed to set HOMEgfs in run stub
tmcguinness
committed
needed to set HOMEgfs in run stub and looking at pullRequest.labels.contains
tmcguinness
committed
typoed pslot in run check stub script
tmcguinness
committed
hard coded node name
tmcguinness
committed
hard coded node names typo capped
tmcguinness
committed

Commits on Jan 19, 2024

returnStdout on cat BUIT semaphore didnt save
tmcguinness
committed
added node properties
tmcguinness
committed
restored configurable node
tmcguinness
committed
moved parameters into options block
tmcguinness
committed
added just parameters section
tmcguinness
committed
moved parameters into options
tmcguinness
committed
parameters can not be in options
tmcguinness
committed
removed class in parameters
tmcguinness
committed
added class in bracketrs
tmcguinness
committed
removed double bracket
tmcguinness
committed
not sure how to hadd class
tmcguinness
committed
still can not get syntax for parameters
tmcguinness
committed
still not finding class
tmcguinness
committed
just removed class line
tmcguinness
committed
moved properties into stage
tmcguinness
committed
moved properties into script in Run Seection
tmcguinness
committed
had machine instead of MACIHNE in node name
tmcguinness
committed
added properties for epic nodes
tmcguinness
committed
needed to remove RUNTESTS dir on rebuild
tmcguinness
committed
added exit 1 to run stub
tmcguinness
committed
removed returnstatus flag on run stub
tmcguinness
committed
Updated properties to my controler
tmcguinness
committed
Can not get parameteres to sork again
tmcguinness
committed
added defaut slave as running one
tmcguinness
committed
added prop in create
tmcguinness
committed
added prop in create still brokek
tmcguinness
committed
still broken
tmcguinness
committed
moved prop under get machine script
tmcguinness
committed
made machine global
tmcguinness
committed
Label issue
tmcguinness
committed
removed Configuiration class
tmcguinness
committed
added stub logic
tmcguinness
committed
remove exit 0 on create exper wrapper because once worked
tmcguinness
committed
two orion fake local adds
tmcguinness
committed

Commits on Jan 20, 2024

added my spak python 3
tmcguinness
committed
got =~ backwards
tmcguinness
committed
explitly remove existing experment before re-running create
tmcguinness
committed
hard coded HOMEgfs for testing
tmcguinness
committed
testing on node hera not orion
tmcguinness
committed
dump remove label constraint for now
tmcguinness
committed
RUNTESTS was wrong and yes works
tmcguinness
committed

Commits on Jan 22, 2024

Merge branch 'NOAA-EMC:develop' into jenkins-pipeline
TerrenceMcGuinness-NOAA
authored
removed hardcoded machine label on nodes
tmcguinness
committed
restore away from local build cludges
tmcguinness
committed
some linting
tmcguinness
committed
some eols
tmcguinness
committed
wrong MACHINE in node label
tmcguinness
committed
removed y pipe and added rm case in pipeline
tmcguinness
committed
removed DefaultSlave in parameters
tmcguinness
committed
added rm COMROOT line in pipeline
tmcguinness
committed
try var in param
tmcguinness
committed
trying to add more choices
tmcguinness
committed
add back real build run
tmcguinness
committed
revert
tmcguinness
committed
ready to start testing stub pipeline with system build
tmcguinness
committed
wrap a stages around build matrix stage
tmcguinness
committed
wrap matrix around build stage
tmcguinness
committed
needed axis not two axes
tmcguinness
committed
put matrix wrap in buid stage
tmcguinness
committed
still trying to get matrix hierachey to work
tmcguinness
committed
system literal literal
tmcguinness
committed
ws needed body
tmcguinness
committed
remove workspaces
tmcguinness
committed
WORKSPACE git just outside of build matrix in node
tmcguinness
committed
Added TESTDIR
tmcguinness
committed
deleted old script line
tmcguinness
committed
dir step needed bodey
tmcguinness
committed
created dedicated stage for getting global workspace for the PR
tmcguinness
committed

Commits on Jan 23, 2024

made HOMEgfs local in matrix and removed from global
tmcguinness
committed
global HOME TESTDIR
tmcguinness
committed
added local HOMEgfs and global HOME in Run Experiments
tmcguinness
committed
updated TESTDIR to global HOME
tmcguinness
committed
remved stubs to run on Orion
tmcguinness
committed
need updated buid all args for gefs
tmcguinness
committed
use old build all args for both systems
tmcguinness
committed
removed stubs add all nodes with restart selector and put back label rm
tmcguinness
committed
remove Running label without check
tmcguinness
committed
Merge branch 'NOAA-EMC:develop' into jenkins-pipeline
TerrenceMcGuinness-NOAA
authored
add dir into sorc
tmcguinness
committed
moved mkdir on RUNTESTS into top stage in Create Experments
tmcguinness
committed
added entire stage to mkdir TESTDIR and update labels
tmcguinness
committed
added full path for dir() into sorc
tmcguinness
committed
reduced save to three
tmcguinness
committed
can not figure out why dir() is not working
tmcguinness
committed
added a few more degubs and . for pwd path
tmcguinness
committed

Commits on Jan 25, 2024

added an explicit cd into the gfs/sorc dir to run build_all
tmcguinness
committed
fixed quote on echo PWD on script sh command line
tmcguinness
committed
added explicit git command for updating submoules recursivly, scm was apearing to skip gdas submodules
tmcguinness
committed

Commits on Jan 26, 2024

Merge branch 'NOAA-EMC:develop' into jenkins-pipeline
TerrenceMcGuinness-NOAA
authored
added max jobs on build all to 4
tmcguinness
committed
updated pipline with node on setup RUNDIR and removed system in run experment wrapper
tmcguinness
committed
updated path of HOMEgfs in run check scripts using gfs as default
tmcguinness
committed
use HOME to point to RUNDIRS for run-check
tmcguinness
committed

Commits on Jan 27, 2024

add label updates back move archive to post
tmcguinness
committed

Commits on Jan 29, 2024

added throttle to build
tmcguinness
committed
remove prop from throttle
tmcguinness
committed
moveed throttle inside options block of stage build
tmcguinness
committed
updated fixed label check code
tmcguinness
committed
ndded script block in post always
tmcguinness
committed
added confgired catigory to throttle
tmcguinness
committed
took out throttle option and added limit j to 2 on both concurrent builds
tmcguinness
committed
made remove all labels with machine in post alwasy a conventional loop
tmcguinness
committed
Update Jenkinsfile
TerrenceMcGuinness-NOAA
authored

Commits on Jan 30, 2024

Merge branch 'NOAA-EMC:develop' into jenkins-pipeline
TerrenceMcGuinness-NOAA
authored
added to matrix builds to make it sequential and added yaml reads
tmcguinness
committed
had extra values line in first matrix build
tmcguinness
committed
had an extra pasted GFS build
tmcguinness
committed
added build yaml file in cases yamls
tmcguinness
committed
variable key had to be imutable
tmcguinness
committed
testing imuable key
tmcguinness
committed
testing yaml output
tmcguinness
committed
traverse build args
tmcguinness
committed
missed } on iter
tmcguinness
committed
use get in yaml
tmcguinness
committed
try it key
tmcguinness
committed
just system
tmcguinness
committed
to string
tmcguinness
committed
to string
tmcguinness
committed
still trying
tmcguinness
committed
still foo
tmcguinness
committed
finally
tmcguinness
committed
finally then trim
tmcguinness
committed
finally then trim and nulls
tmcguinness
committed
got args to work
tmcguinness
committed

Commits on Jan 31, 2024

Commits on Feb 1, 2024

Update Jenkinsfile
TerrenceMcGuinness-NOAA
authored
added pre checkout in TESTDIR added source of gwsetup and changed first dir to ws
tmcguinness
committed
added git to hera gw setup module load
tmcguinness
committed
made shallow checkout a function call
tmcguinness
committed
misplaced bracket in source of gw_setup
tmcguinness
committed
cd into TESTDIR for shallow clone
tmcguinness
committed
moved git load on gwset hera because it is in the stack spac
tmcguinness
committed
use shallow checkout and git submoudle recussive update directly
tmcguinness
committed
configured scm shallow and using git modules recursive from host
tmcguinness
committed
still had shallow call and removed it
tmcguinness
committed
put dir back at attempt to use ws
tmcguinness
committed
take out pre checkout and just use submodule update directly
tmcguinness
committed

Commits on Feb 2, 2024

Merge branch 'NOAA-EMC:develop' into jenkins-pipeline
TerrenceMcGuinness-NOAA
authored
stub build all to just gdas
tmcguinness
committed
added git which and version for debugging
tmcguinness
committed
put buile lines back in build yaml
tmcguinness
committed
added a second Jenkins file to test dynamic parallel build stages
tmcguinness
committed
removed Jenkinsfile2 to put it into its own repo
tmcguinness
committed
adding deleteDir to get scm to checkout again
tmcguinness
committed
adding build stubs
tmcguinness
committed
adding build stubs again
tmcguinness
committed
forgot to add build.yaml updates
tmcguinness
committed
put throttle back and updated build args for gefs
tmcguinness
committed

Commits on Feb 3, 2024

use rsync to get clones before submodules
tmcguinness
committed
update rsync line
tmcguinness
committed
Merge branch 'NOAA-EMC:develop' into jenkins-pipeline
TerrenceMcGuinness-NOAA
authored
revert to scm and comment out disable current builds
tmcguinness
committed
Merge branch 'jenkins-pipeline' of github.com:TerrenceMcGuinness-NOAA/global-workflow into jenkins-pipeline
tmcguinness
committed

Commits on Feb 4, 2024

Update Jenkinsfile
TerrenceMcGuinness-NOAA
authored
added checkoutsubdir
tmcguinness
committed
added checkoutsubdir
tmcguinness
committed
extra bracket
tmcguinness
committed
moved options
tmcguinness
committed
add no scm in exp
tmcguinness
committed

Commits on Feb 5, 2024

devult no scm others
tmcguinness
committed
put throuttle back in
tmcguinness
committed
try chaning workspace to get scm to checkout on second parallel matrix build
tmcguinness
committed
added stubs for build all to test the Jenkins scm issue
tmcguinness
committed
removed throttle option for testing scm
tmcguinness
committed
Update ci/scripts/utils/ci_utils.sh

TerrenceMcGuinness-NOAA
and
WalterKolczynski-NOAA
authored
Update ci/scripts/utils/ci_utils.sh

TerrenceMcGuinness-NOAA
and
WalterKolczynski-NOAA
authored
Update ci/scripts/utils/ci_utils.sh

TerrenceMcGuinness-NOAA
and
WalterKolczynski-NOAA
authored
Update ci/scripts/utils/ci_utils.sh

TerrenceMcGuinness-NOAA
and
WalterKolczynski-NOAA
authored
Update modulefiles/module_gwsetup.hera.lua

TerrenceMcGuinness-NOAA
and
WalterKolczynski-NOAA
authored
took out throuttle and put checkout subdir back
tmcguinness
committed
Merge branch 'jenkins-pipeline' of github.com:TerrenceMcGuinness-NOAA/global-workflow into jenkins-pipeline
tmcguinness
committed
update build args for faster buld tests
tmcguinness
committed
took stub to create tests and put the action back
tmcguinness
committed
Merge branch 'NOAA-EMC:develop' into jenkins-pipeline
TerrenceMcGuinness-NOAA
authored
put throttle back and took out -j 2
tmcguinness
committed
Merge branch 'jenkins-pipeline' of github.com:TerrenceMcGuinness-NOAA/global-workflow into jenkins-pipeline
tmcguinness
committed
put concurrancy back in for the matrixed build stage because the scm non reset issue came back
tmcguinness
committed
moved mkdir out of Setup RUNTESTS because it the agent options was not using skepDefaultCheckout and erroring on scm GitHub
tmcguinness
committed
added explcit workspaces in create and using gfs as default for running
tmcguinness
committed
added explcit workspaces in create and using gfs as default for running
tmcguinness
committed
took out case and put in if else in cancel_batch_jobs (for now)
tmcguinness
committed
missed bracket at end of cancel_batch_jobs in ci utils
tmcguinness
committed
add true to return status on run-check because it through that peski Jobs SCM is not GitHub again and it is the only difference to the lines that did not
tmcguinness
committed
update all Jenkins script calls to returnStatus true and put full test list back
tmcguinness
committed
adding with credentials
tmcguinness
committed
stub create
tmcguinness
committed
add short list for exper
tmcguinness
committed
added withCredntials in all three post sections
tmcguinness
committed
took out create experments stub
tmcguinness
committed
put back in full test list for overnight run
tmcguinness
committed
Merge branch 'NOAA-EMC:develop' into jenkins-pipeline
TerrenceMcGuinness-NOAA
authored

Commits on Feb 6, 2024

Update Jenkinsfile
TerrenceMcGuinness-NOAA
authored
add agent to post block
TerrenceMcGuinness-NOAA
committed
move post into Run Tests block
TerrenceMcGuinness-NOAA
committed
stubbed out create expermentens again
TerrenceMcGuinness-NOAA
committed
took out sub for create and made case short list
TerrenceMcGuinness-NOAA
committed
removed setup block
TerrenceMcGuinness-NOAA
committed
just do one test
TerrenceMcGuinness-NOAA
committed
added back buildDiscarder
TerrenceMcGuinness-NOAA
committed

Commits on Feb 7, 2024

commented out remove label in build
terrance.mcguinness
committed
removed redundent checkout in subdir and commentout gitHub access after scm in build
terrance.mcguinness
committed
added ws with github creditials
terrance.mcguinness
committed
misspelled WORKSPACE
terrance.mcguinness
committed
missed a MACIHNE left in
terrance.mcguinness
committed
took timer out of Get Common
terrance.mcguinness
committed
updated machine names to Machine in labels missed
terrance.mcguinness
committed
added commonworkspace in build update labels
terrance.mcguinness
committed
updated workspace in post to be the same in the matrix that ran the scripts and should have a prevouse scm checkout profomed in it
terrance.mcguinness
committed
remove local varible scope on HOMEgfs in Run experments in hopes it works in post
terrance.mcguinness
committed
Merge branch 'NOAA-EMC:develop' into jenkins-pipeline
TerrenceMcGuinness-NOAA
authored
updated Machine in a label and in PR comments
terrance.mcguinness
committed
Merge branch 'jenkins-pipeline' of https://github.com/TerrenceMcGuinness-NOAA/global-workflow into jenkins-pipeline
terrance.mcguinness
committed
only replaced commonworkspace with HOMEgfs workspace in build matrix
terrance.mcguinness
committed
added info in Running message to use in GitHub PR
terrance.mcguinness
committed
put missing values line back in experments matrix
terrance.mcguinness
committed
added u to gfs build and put all tests back in
terrance.mcguinness
committed
added a stubbed version of the Jenkinsfiel for testing
terrance.mcguinness
committed

Commits on Feb 8, 2024

put a try around removelabel in all
terrance.mcguinness
committed
took pslot out for stubbing
terrance.mcguinness
committed
added back just build to stub
terrance.mcguinness
committed
removed build non stub and made stub again
terrance.mcguinness
committed
added only buils scripts but disabled otherwise scan poll
terrance.mcguinness
committed
remove nested ws in build
terrance.mcguinness
committed
added comments in Jenkinsfile
terrance.mcguinness
committed
added some markdown formatting in GitHub messages
terrance.mcguinness
committed
updated some markdown formatting in main GitHub PR message
terrance.mcguinness
committed
updated format with one more <br>
TerrenceMcGuinness-NOAA
authored