Code

CI JJOB Tests using CMake#3214

Code

Merged

WalterKolczynski-NOAA merged 160 commits intoNOAA-EMC:developNOAA-EMC/global-workflow:developfrom TerrenceMcGuinness-NOAA:ctest_jjobs_frameworkTerrenceMcGuinness-NOAA/global-workflow:ctest_jjobs_frameworkCopy head branch name to clipboardJan 17, 2025

+423

Commits

Commits on Dec 6, 2024

• added ctest dir with cmake to create an experment

  

  Terry McGUinness

  committed
  987ae5b

  Copy full SHA for 987ae5b

• added CMakeLists for creating initial case EXPDIR usint ctest

  

  Terry McGuinness

  committed
  61fcf83

  Copy full SHA for 61fcf83

• ready to test getting the job names

  

  Terry McGUinness

  committed
  f5f6595

  Copy full SHA for f5f6595

• move all operations of creating EXPDIR cases into configure phase

  

  Terry McGUinness

  committed
  cbe3bab

  Copy full SHA for cbe3bab

• got cmake configure to work for EXPDIR

  

  Terry.McGuinness

  authored and

  Terry.McGuinness

  committed
  6a50e14

  Copy full SHA for 6a50e14

• can not do both file and variable save for execute_process

  

  Terry McGUinness

  committed
  bcebeca

  Copy full SHA for bcebeca

• working cmake config up to output of job names

  

  Terry.McGuinness

  authored and

  Terry.McGuinness

  committed
  be7a890

  Copy full SHA for be7a890

Commits on Dec 9, 2024

• Merge branch 'NOAA-EMC:develop' into ctest_jjobs_framework

  TerrenceMcGuinness-NOAA

  authored
  2811629

  Copy full SHA for 2811629

• loop over jobnames and create an experment for each

  

  Terry McGuinness

  committed
  70a8183

  Copy full SHA for 70a8183

• update JOB name update in loop

  

  Terry McGUinness

  committed
  72e5a9c

  Copy full SHA for 72e5a9c

• addded debug outputs

  

  Terry McGUinness

  committed
  1f1fcd6

  Copy full SHA for 1f1fcd6

• modifying plslot list to be only on a per case

  

  Terry McGuinness

  committed
  ee27b02

  Copy full SHA for ee27b02

• filter pslot list for only relevent cases

  

  Terry McGUinness

  committed
  bc76ddf

  Copy full SHA for bc76ddf

• refining get_job_names

  

  Terry McGUinness

  committed
  38838a6

  Copy full SHA for 38838a6

• modigied get_jobs_names to task and metatask names

  

  Terry McGUinness

  committed
  d18ad01

  Copy full SHA for d18ad01

• got better jobname parcer working

  

  Terry McGuinness

  committed
  ef0e066

  Copy full SHA for ef0e066

• Merge branch 'NOAA-EMC:develop' into ctest_jjobs_framework

  TerrenceMcGuinness-NOAA

  authored
  519a919

  Copy full SHA for 519a919

Commits on Dec 10, 2024

• replaced generate_workflows with create_experment

  

  Terry McGUinness

  committed
  f813a1b

  Copy full SHA for f813a1b

• removed non-existing ENVIRONMENT setting to consolated COMMAND line

  

  Terry McGUinness

  committed
  8238228

  Copy full SHA for 8238228

• got cmake script to use creat_experiment

  

  Terry McGuinness

  committed
  bf579ee

  Copy full SHA for bf579ee

Commits on Dec 12, 2024

• Merge branch 'NOAA-EMC:develop' into ctest_jjobs_framework

  TerrenceMcGuinness-NOAA

  authored
  28ef1af

  Copy full SHA for 28ef1af

• just getting started with addeding a ctest

  

  Terry McGUinness

  committed
  374927f

  Copy full SHA for 374927f

• add in test for configured project

  

  Terry McGUinness

  committed
  8b4fc4f

  Copy full SHA for 8b4fc4f

• added ctest and testing

  

  Terry McGuinness

  committed
  ce37d86

  Copy full SHA for ce37d86

Commits on Dec 13, 2024

• Merge branch 'NOAA-EMC:develop' into ctest_jjobs_framework

  TerrenceMcGuinness-NOAA

  authored
  d4e3266

  Copy full SHA for d4e3266

• moved ctests dir to top HOMEgfs/ci/ctests with its own scripts dir

  

  Terry McGUinness

  committed
  bb12cda

  Copy full SHA for bb12cda

• added all jobs for tests for each case

  

  Terry McGUinness

  committed
  f78181c

  Copy full SHA for f78181c

Commits on Dec 14, 2024

• added data stagging dependances and started it as a stub script

  

  Terry McGUinness

  committed
  46cddb7

  Copy full SHA for 46cddb7

Commits on Dec 16, 2024

• Merge branch 'NOAA-EMC:develop' into ctest_jjobs_framework

  TerrenceMcGuinness-NOAA

  authored
  d6c3a49

  Copy full SHA for d6c3a49

Commits on Dec 18, 2024

• added driver for creating ctests from yaml files

  

  Terry McGUinness

  committed
  434f869

  Copy full SHA for 434f869

• added a driver to create ctests from defintions in yaml files

  

  Terry McGUinness

  committed
  99df88d

  Copy full SHA for 99df88d

• added final step to loop over yaml files for creating functiona ctests from configs

  

  Terry McGUinness

  committed
  e186173

  Copy full SHA for e186173

Commits on Dec 19, 2024

• Merge branch 'NOAA-EMC:develop' into ctest_jjobs_framework

  TerrenceMcGuinness-NOAA

  authored
  c4a2780

  Copy full SHA for c4a2780

• added second yaml case file for testing python driver

  

  Terry McGUinness

  committed
  0183667

  Copy full SHA for 0183667

• added a few comments for each section

  

  TerrenceMcGuinness-NOAA

  committed
  3088bff

  Copy full SHA for 3088bff

Commits on Dec 20, 2024

• Merge branch 'NOAA-EMC:develop' into ctest_jjobs_framework

  TerrenceMcGuinness-NOAA

  authored
  9d06df7

  Copy full SHA for 9d06df7

• made CMakeList.txt independent of python script

  

  Terry McGuinness

  authored and

  Terry McGuinness

  committed
  674f8d6

  Copy full SHA for 674f8d6

• simplified CMakeList to be compatible with manin build, consolated create exp into main test along with stage,run, and validate.

  

  Terry McGuinness

  committed
  68ca6ee

  Copy full SHA for 68ca6ee

• repurposed python script for generating args to man CMakeLists.txt

  

  Terry McGuinness

  committed
  f7bbd40

  Copy full SHA for f7bbd40

• added lables to run tests in groups and added stubbed scripts

  

  Terry McGuinness

  authored and

  Terry McGuinness

  committed
  36d497e

  Copy full SHA for 36d497e

• created script and cases dir to isolate CMakeLists.txt

  

  Terry McGuinness

  authored and

  Terry McGuinness

  committed
  808a4cf

  Copy full SHA for 808a4cf

• renamed test to create_experiment

  

  Terry McGuinness

  authored and

  Terry McGuinness

  committed
  81e640f

  Copy full SHA for 81e640f

• missed update to name of test from test_ to create_experiment_

  

  Terry McGuinness

  authored and

  Terry McGuinness

  committed
  a1d0f20

  Copy full SHA for a1d0f20

• forgot to save contents of the stubbed scripts

  

  Terry McGuinness

  authored and

  Terry McGuinness

  committed
  2cbf4ee

  Copy full SHA for 2cbf4ee

• spelling error on create_experiment

  

  Terry McGuinness

  authored and

  Terry McGuinness

  committed
  687217f

  Copy full SHA for 687217f

• got a few hints on using File Utils directly in case Yaml file

  

  Terry McGuinness

  authored and

  Terry McGuinness

  committed
  55766bd

  Copy full SHA for 55766bd

Commits on Dec 23, 2024

• updated adding functional test as line enteries in CMakeLists.txt directly that creats the three steps as tests

  

  Terry McGuinness

  authored and

  Terry McGuinness

  committed
  f69fb68

  Copy full SHA for f69fb68

• Cleaned up names to have Case and job name first

  

  Terry McGuinness

  authored and

  Terry McGuinness

  committed
  3cc83c6

  Copy full SHA for 3cc83c6

• added file utils copy commands in gfs_fcst for C48_ATM

  

  Terry McGuinness

  authored and

  Terry McGuinness

  committed
  e1e1858

  Copy full SHA for e1e1858

Commits on Dec 24, 2024

• started on stage data python script

  

  Terry McGuinness

  authored and

  Terry McGuinness

  committed
  5ce988f

  Copy full SHA for 5ce988f

• added exec line for python staging script

  

  Terry McGuinness

  authored and

  Terry McGuinness

  committed
  cefaccd

  Copy full SHA for cefaccd

• deleted space in yaml file for copy commands

  

  Terry McGuinness

  authored and

  Terry McGuinness

  committed
  bec03d9

  Copy full SHA for bec03d9

Commits on Dec 26, 2024

• Merge branch 'NOAA-EMC:develop' into ctest_jjobs_framework

  TerrenceMcGuinness-NOAA

  authored
  5f9451b

  Copy full SHA for 5f9451b

Commits on Jan 2, 2025

• removed extra $ on JOB var

  

  Terry McGuinness

  committed
  74fbf45

  Copy full SHA for 74fbf45

• idate needed for PDY

  

  Terry McGuinness

  committed
  19600f1

  Copy full SHA for 19600f1

• updated stage data with idate for PDY and HH and made sure paths where correct

  

  Terry McGuinness

  committed
  2caa229

  Copy full SHA for 2caa229

• added copy using FIleHandler

  

  Terry McGuinness

  committed
  749e54f

  Copy full SHA for 749e54f

• copy command needs quotes

  

  Terry McGuinness

  committed
  75450e1

  Copy full SHA for 75450e1

• added sync() on file handler

  

  Terry McGuinness

  committed
  0fc5d76

  Copy full SHA for 0fc5d76

• got FileUtil to work in stage data

  

  Terry McGuinness

  committed
  ea01737

  Copy full SHA for ea01737

• got started on execute

  

  Terry McGuinness

  committed
  e9c46f1

  Copy full SHA for e9c46f1

• execute needed shabang and exit 0

  

  Terry McGuinness

  committed
  2602166

  Copy full SHA for 2602166

• almost ready to launch execute.sh

  

  Terry McGuinness

  committed
  1950af4

  Copy full SHA for 1950af4

Commits on Jan 3, 2025

• Merge branch 'NOAA-EMC:develop' into ctest_jjobs_framework

  TerrenceMcGuinness-NOAA

  authored
  c691c0e

  Copy full SHA for c691c0e

• added working skeleton bash script for execute using rocotoboot with dryrun

  

  Terry McGuinness

  committed
  276a9bd

  Copy full SHA for 276a9bd

• added a few ICs that where missed

  

  Terry McGuinness

  committed
  283000e

  Copy full SHA for 283000e

• removed extra scripts no longer used

  

  Terry McGuinness

  committed
  b2a6fec

  Copy full SHA for b2a6fec

• added defulted values and over rides from command line

  

  Terry McGuinness

  committed
  9525c74

  Copy full SHA for 9525c74

• removed HOMEgfs for path to create expermient

  

  Terry McGuinness

  committed
  74f7268

  Copy full SHA for 74f7268

• removed another unused script

  

  Terry McGuinness

  committed
  5cd1c5f

  Copy full SHA for 5cd1c5f

• fixed a few path errors with HOMEgfs still hanging around

  

  Terry McGuinness

  committed
  c132786

  Copy full SHA for c132786

Commits on Jan 6, 2025

• Merge branch 'NOAA-EMC:develop' into ctest_jjobs_framework

  TerrenceMcGuinness-NOAA

  authored
  87674f2

  Copy full SHA for 87674f2

• added cmake_parse_arguments with extras and used inline template for setup.sh

  

  Terry McGuinness

  committed
  e873722

  Copy full SHA for e873722

• added template for cmake tests setup.sh

  

  Terry McGuinness

  committed
  32c5f87

  Copy full SHA for 32c5f87

• had to add ARG_ header to labels names for depenancies for setup worked

  

  Terry McGuinness

  committed
  128d0e6

  Copy full SHA for 128d0e6

• added bash wrapper to stage_data.py and made single job per yaml configuation

  

  Terry McGuinness

  committed
  abb2952

  Copy full SHA for abb2952

• added the part to drop in templated inline script for stage.sh

  

  Terry McGuinness

  committed
  1d54caf

  Copy full SHA for 1d54caf

• getting stage.sh.in args to work for now

  

  Terry McGuinness

  committed
  2532576

  Copy full SHA for 2532576

• still getting args straintened out with staging

  

  Terry McGuinness

  committed
  0967f10

  Copy full SHA for 0967f10

Commits on Jan 7, 2025

• added args in bash wrapper to stage.py to pick up ARG_CASE and ARG_JOB

  

  Terry McGuinness

  committed
  8231519

  Copy full SHA for 8231519

• updated args to stage.py to simply take yaml input

  

  Terry McGuinness

  committed
  30466b9

  Copy full SHA for 30466b9

• had extra ) for updated add_test

  

  Terry McGuinness

  committed
  3deeb41

  Copy full SHA for 3deeb41

• removed reading of pr case file for getting idate

  

  Terry McGuinness

  committed
  099a2cb

  Copy full SHA for 099a2cb

• replaced JOB_CASE in path for inputs to TEST_NAME

  

  Terry McGuinness

  committed
  94dd64e

  Copy full SHA for 94dd64e

• updated path to yaml input to stage

  

  Terry McGuinness

  committed
  a98d55d

  Copy full SHA for a98d55d

• ctest dir is ctests in path to YAML file for stage.py

  

  Terry McGuinness

  committed
  29517e3

  Copy full SHA for 29517e3

• finilized inputs to make stage.py to work without using CASE+JOB

  

  Terry McGuinness

  committed
  60a6136

  Copy full SHA for 60a6136

• mnissed a couple or ARGS_ again this time in execute

  

  Terry McGuinness

  committed
  b977e52

  Copy full SHA for b977e52

• ARG are prefixes in execute add test

  

  Terry McGuinness

  committed
  95be9f2

  Copy full SHA for 95be9f2

• ARG are prefixes in execute add test 2

  

  Terry McGuinness

  committed
  e9b0b6a

  Copy full SHA for e9b0b6a

• ARG are prefixes in execute add test 3

  

  Terry McGuinness

  committed
  a72827c

  Copy full SHA for a72827c

• added MAKE_CURRENT_BINARY_DIR as templated VAR in stage

  

  Terry McGuinness

  committed
  4f010a4

  Copy full SHA for 4f010a4

• made execute bash sript a template too

  

  Terry McGuinness

  committed
  b79d445

  Copy full SHA for b79d445

• forgot to save CMakeLists now has updated call to excute from script dir in build

  

  Terry McGuinness

  committed
  16cd19f

  Copy full SHA for 16cd19f

• last bug for working execute phase

  

  Terry McGuinness

  committed
  fffb13d

  Copy full SHA for fffb13d

• ready to test x coders add for waiting for slurm job to complete

  

  Terry McGuinness

  committed
  5c5ac0f

  Copy full SHA for 5c5ac0f

• do not need link to wxflow anymore, added it to PYTHONPATH in bash wrappers

  

  Terry McGuinness

  committed
  06c8e8f

  Copy full SHA for 06c8e8f

Commits on Jan 8, 2025

• Merge branch 'NOAA-EMC:develop' into ctest_jjobs_framework

  TerrenceMcGuinness-NOAA

  authored
  edde418

  Copy full SHA for edde418

• path to wxflow in PYTHONPATH was wrong in stage template

  

  Terry McGuinness

  committed
  d68b02e

  Copy full SHA for d68b02e

• generlized ARG over-ride and added check for exectution success from log file output for when slurm job id is no longer available after job completes

  

  Terry McGuinness

  committed
  b0602bc

  Copy full SHA for b0602bc

• should have checked for End not END for success

  

  Terry McGuinness

  committed
  30b3f3a

  Copy full SHA for 30b3f3a

• oops need done at then of loop in execute bash sript

  

  Terry McGuinness

  committed
  42084e7

  Copy full SHA for 42084e7

• removed C48_S2SW.yaml for functional test as it is not ready yet

  

  Terry McGuinness

  committed
  8d71151

  Copy full SHA for 8d71151

• working on shellnorms

  

  Terry McGuinness

  committed
  cf7be28

  Copy full SHA for cf7be28

• working on more shellnorms

  

  Terry McGuinness

  committed
  a49a642

  Copy full SHA for a49a642

• working on even more shellnorms

  

  Terry McGuinness

  committed
  466831d

  Copy full SHA for 466831d

• working couple more shellnorms

  

  Terry McGuinness

  committed
  2d18461

  Copy full SHA for 2d18461

Commits on Jan 10, 2025

• Merge branch 'NOAA-EMC:develop' into ctest_jjobs_framework

  TerrenceMcGuinness-NOAA

  authored
  53b13e3

  Copy full SHA for 53b13e3

• removed scripts used for getting system defaults and added top level CMakeLists.txt to use PROJECT_SOURCE_DIR for HOMEgfs in templiting bash wrapper sripts

  

  Terry McGuinness

  committed
  4ee8d81

  Copy full SHA for 4ee8d81

• added to yaml fun conf file to us jinja string modification TEST_DATE

  

  Terry McGuinness

  committed
  b3518ca

  Copy full SHA for b3518ca

• typos in CMakeLists and module file for adding CMake on Orion

  

  Terry McGuinness

  committed
  54f633b

  Copy full SHA for 54f633b

• added TEST_DATE as an agrment to adding a test in CMake List file

  

  Terry McGuinness

  committed
  a182c44

  Copy full SHA for a182c44

Commits on Jan 11, 2025

• needed idate in arg to exexcute

  

  Terry McGuinness

  committed
  fd8a8bc

  Copy full SHA for fd8a8bc

• just use par_yaml

  

  Terry McGuinness

  committed
  472d89c

  Copy full SHA for 472d89c

• trying to get args into yaml

  

  Terry McGuinness

  committed
  45d2901

  Copy full SHA for 45d2901

• added TEST_DATA via data to template

  

  Terry McGuinness

  committed
  8ca4d7a

  Copy full SHA for 8ca4d7a

• added TEST_DATE into arg to bash stage

  

  Terry McGuinness

  committed
  91d76e3

  Copy full SHA for 91d76e3

Commits on Jan 13, 2025

• changed test_date argument to datetime object so it can be used with Jinja functions with in the templating class

  

  Terry McGuinness

  committed
  61e7ce3

  Copy full SHA for 61e7ce3

• MACHINE_ID is no longer used (removed from messae)

  

  Terry McGuinness

  committed
  1860e05

  Copy full SHA for 1860e05

• updated arg to stagy.py for test_date to a str

  

  Terry McGuinness

  committed
  e979f75

  Copy full SHA for e979f75

• missed commit (adding test_date arg to stage.py)

  

  Terry McGuinness

  committed
  91f4331

  Copy full SHA for 91f4331

• added env vars to starge.py in wrapper via prefixed values to call

  

  Terry McGuinness

  committed
  17fd1da

  Copy full SHA for 17fd1da

• check if default is not used and needs to be set to valule for error message to occure

  

  Terry McGuinness

  committed
  48ec7c6

  Copy full SHA for 48ec7c6

• did not have updated var name for STAGED_TEST_DIR in yaml file

  

  Terry McGuinness

  committed
  243d47b

  Copy full SHA for 243d47b

• replaced scraping output file with status from sacct

  

  Terry McGuinness

  committed
  c092b10

  Copy full SHA for c092b10

• added extra step for making sure the submitted job_id is in the sacct system

  

  Terry McGuinness

  committed
  41b6aee

  Copy full SHA for 41b6aee

• remove white spaces from job_status

  

  Terry McGuinness

  committed
  42fbc2a

  Copy full SHA for 42fbc2a

• update path to validate stub and reduce first wait on sacct to 30 seconds

  

  Terry McGuinness

  committed
  b17ca77

  Copy full SHA for b17ca77

• update a few shellnorm suggestions

  

  Terry McGuinness

  committed
  bee8efc

  Copy full SHA for bee8efc

• one more shellnorm for true on pipe

  

  Terry McGuinness

  committed
  613d951

  Copy full SHA for 613d951

• innocuous extra line at end of toplevel CMake file

  

  Terry McGuinness

  committed
  53bfb62

  Copy full SHA for 53bfb62

• consolated GFS ROOT path for staged data dir on orion

  

  Terry McGuinness

  committed
  c6cfefb

  Copy full SHA for c6cfefb

• added some more notes for validate subs

  

  Terry McGuinness

  committed
  02a03f1

  Copy full SHA for 02a03f1

• remove extras from stage.py

  

  Terry McGuinness

  committed
  02e1a92

  Copy full SHA for 02e1a92

• still had path woring to valication stub (has no *.in file)

  

  Terry McGuinness

  committed
  3da4c54

  Copy full SHA for 3da4c54

• another shellnorm || true

  

  Terry McGuinness

  committed
  c416370

  Copy full SHA for c416370

• pynomrs spaces

  

  Terry McGuinness

  committed
  52c5081

  Copy full SHA for 52c5081

• pynomrs line space two

  

  Terry McGuinness

  committed
  1b74559

  Copy full SHA for 1b74559

Commits on Jan 14, 2025

• Merge branch 'NOAA-EMC:develop' into ctest_jjobs_framework

  TerrenceMcGuinness-NOAA

  authored
  99defb4

  Copy full SHA for 99defb4

• added back missing submodels gsi_enkf and gsi_utils

  

  Terry McGuinness

  committed
  e965602

  Copy full SHA for e965602

• updating to correct hash for submodule gsi_enkf

  

  Terry McGuinness

  committed
  241392d

  Copy full SHA for 241392d

• added inline documentation to the stage file python script

  

  Terry McGuinness

  committed
  4669e38

  Copy full SHA for 4669e38

• added simple README.md file in the ctest directory

  

  Terry McGuinness

  committed
  892a6bf

  Copy full SHA for 892a6bf

• Merge branch 'develop' into ctest_jjobs_framework

  TerrenceMcGuinness-NOAA

  authored
  3dff4d1

  Copy full SHA for 3dff4d1

• included instructions on how to add a new test in the README.md file

  

  Terry McGuinness

  committed
  b7f80cb

  Copy full SHA for b7f80cb

• Update README.md

  Show description for 898fcfe
  TerrenceMcGuinness-NOAA

  authored
  898fcfe

  Copy full SHA for 898fcfe

• Update README.md

  Show description for 12cf0cf
  TerrenceMcGuinness-NOAA

  authored
  12cf0cf

  Copy full SHA for 12cf0cf

• Update README.md

  Show description for eeb744f
  TerrenceMcGuinness-NOAA

  authored
  eeb744f

  Copy full SHA for eeb744f

• Update README.md

  Show description for 73211bc
  TerrenceMcGuinness-NOAA

  authored
  73211bc

  Copy full SHA for 73211bc

• Update ctests/scripts/validate.sh

  Show description for f18f1ee
  

  TerrenceMcGuinness-NOAA

  and

  aerorahul

  authored
  f18f1ee

  Copy full SHA for f18f1ee

• added validate.sh.in template bash wrapper script

  

  Terry McGuinness

  committed
  b1a9fd2

  Copy full SHA for b1a9fd2

• added cmake to Hercules gwsetup modulefile

  

  Terry McGuinness

  committed
  f058450

  Copy full SHA for f058450

• updated path to validate.sh

  

  Terry McGuinness

  committed
  eede574

  Copy full SHA for eede574

Commits on Jan 15, 2025

• needed to remove bash from the COMMAND line for validate.sh

  

  Terry McGuinness

  committed
  7da3828

  Copy full SHA for 7da3828

• Merge branch 'NOAA-EMC:develop' into ctest_jjobs_framework

  TerrenceMcGuinness-NOAA

  authored
  5f722e3

  Copy full SHA for 5f722e3

• allow the build to continue whe ctests can not be created without the needed env vars set

  

  Terry McGuinness

  committed
  e48ae52

  Copy full SHA for e48ae52

• added inpt_files in the path of stagged file to seperate them from output_files

  

  Terry McGuinness

  committed
  b691a59

  Copy full SHA for b691a59

• added ouput date into case yaml file

  

  Terry McGuinness

  committed
  5c8a176

  Copy full SHA for 5c8a176

• removed output_files from C48_ATM

  

  Terry McGuinness

  committed
  bfe10a2

  Copy full SHA for bfe10a2

Commits on Jan 16, 2025

• Merge branch 'NOAA-EMC:develop' into ctest_jjobs_framework

  TerrenceMcGuinness-NOAA

  authored
  2833514

  Copy full SHA for 2833514