CI JJOB Tests using CMake #3214

TerrenceMcGuinness-NOAA · 2025-01-08T18:57:38Z

Description

Adding CI tests at the JJOB level using CMake/ctest wrappers:
These individual JJOB tests have four distinct phases:

Setup: Creates a EXPDIR/COMROOT just for the individual JJOB
Stage: Moves the specific files into the COMROOT that are needed to run the JJOB specified in ${HOMEgfs}/ci/ctest/cases/{CASE}_{JJOB}.yaml
Execute: Run the JJOB in batch (batch card extracted from XML via Rocoto)
Validate: Check the outputs also specified in the above yaml configure file (currently stubbed)

Resolves #3204

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? YES
Does this change require an update to any of the following submodules? NO (If YES, please add a link to any PRs that are pending.)
- EMC verif-global
- GDAS
- GFS-utils
- GSI
- GSI-monitor
- GSI-utils
- UFS-utils
- UFS-weather-model
- wxflow

How has this been tested?

Ran with CMake/ctest:

mterry (orion-login-2) build (ctest_jjobs_framework) $ ctest -N
Test project /work2/noaa/global/mterry/global-workflow_forked/ci/ctests/build
  Test #1: test_C48_ATM_gfs_fcst_seg0_setup
  Test #2: test_C48_ATM_gfs_fcst_seg0_stage
  Test #3: test_C48_ATM_gfs_fcst_seg0_execute
  Test #4: test_C48_ATM_gfs_fcst_seg0_validate

Total Tests: 4
mterry (orion-login-2) build (ctest_jjobs_framework) $ ctest -L C48_ATM
Test project /work2/noaa/global/mterry/global-workflow_forked/ci/ctests/build
    Start 1: test_C48_ATM_gfs_fcst_seg0_setup
1/4 Test #1: test_C48_ATM_gfs_fcst_seg0_setup ......   Passed    3.25 sec
    Start 2: test_C48_ATM_gfs_fcst_seg0_stage
2/4 Test #2: test_C48_ATM_gfs_fcst_seg0_stage ......   Passed    1.90 sec
    Start 3: test_C48_ATM_gfs_fcst_seg0_execute
3/4 Test #3: test_C48_ATM_gfs_fcst_seg0_execute ....   Passed  2942.51 sec
    Start 4: test_C48_ATM_gfs_fcst_seg0_validate
4/4 Test #4: test_C48_ATM_gfs_fcst_seg0_validate ...   Passed    0.02 sec

100% tests passed, 0 tests failed out of 4

Label Time Summary:
C48_ATM          = 2947.68 sec*proc (4 tests)
gfs_fcst_seg0    = 2947.68 sec*proc (4 tests)

Total Test time (real) = 2947.68 sec

Checklist

github-advanced-security

ShellCheck found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

ci/ctests/scripts/execute.sh.in

ci/ctests/scripts/setup.sh.in

aerorahul · 2025-01-08T20:34:25Z

ci/ctests/cases/C48_ATM_gfs_fcst_seg0.yaml

+    mkdir:
+        - "{{RUNTESTS}}/COMROOT/{{ TEST_NAME }}//gfs.{{PDY}}/{{HH}}/model/atmos/input"
+    copy:
+        - ["{{ STAGED_FUNCTIONAL_TEST_DIR }}/{{ TEST_NAME }}/gfs.{{ PDY }}/{{ HH }}/model/atmos/input/gfs_ctrl.nc", "{{RUNTESTS}}/COMROOT/{{ TEST_NAME }}//gfs.{{PDY}}/{{HH}}/model/atmos/input/gfs_ctrl.nc"]


Suggested change

- ["{{ STAGED_FUNCTIONAL_TEST_DIR }}/{{ TEST_NAME }}/gfs.{{ PDY }}/{{ HH }}/model/atmos/input/gfs_ctrl.nc", "{{RUNTESTS}}/COMROOT/{{ TEST_NAME }}//gfs.{{PDY}}/{{HH}}/model/atmos/input/gfs_ctrl.nc"]

- ["{{ STAGED_FUNCTIONAL_TEST_DIR }}/{{ TEST_NAME }}/gfs.{{ TEST_DATE | to_YMD }}/{{ HH }}/model/atmos/input/gfs_ctrl.nc", "{{RUNTESTS}}/COMROOT/{{ TEST_NAME }}//gfs.{{PDY}}/{{HH}}/model/atmos/input/gfs_ctrl.nc"]

aerorahul · 2025-01-08T20:35:05Z

ci/ctests/cases/C48_ATM_gfs_fcst_seg0.yaml

+        - "{{RUNTESTS}}/COMROOT/{{ TEST_NAME }}//gfs.{{PDY}}/{{HH}}/model/atmos/input"
+    copy:
+        - ["{{ STAGED_FUNCTIONAL_TEST_DIR }}/{{ TEST_NAME }}/gfs.{{ PDY }}/{{ HH }}/model/atmos/input/gfs_ctrl.nc", "{{RUNTESTS}}/COMROOT/{{ TEST_NAME }}//gfs.{{PDY}}/{{HH}}/model/atmos/input/gfs_ctrl.nc"]
+        - ["{{ STAGED_FUNCTIONAL_TEST_DIR }}/{{ TEST_NAME }}/gfs.{{ PDY }}/{{ HH }}/model/atmos/input/gfs_data.tile1.nc", "{{RUNTESTS}}/COMROOT/{{ TEST_NAME }}//gfs.{{PDY}}/{{HH}}/model/atmos/input/gfs_data.tile1.nc"]


Suggested change

- ["{{ STAGED_FUNCTIONAL_TEST_DIR }}/{{ TEST_NAME }}/gfs.{{ PDY }}/{{ HH }}/model/atmos/input/gfs_data.tile1.nc", "{{RUNTESTS}}/COMROOT/{{ TEST_NAME }}//gfs.{{PDY}}/{{HH}}/model/atmos/input/gfs_data.tile1.nc"]

- ["{{ STAGED_FUNCTIONAL_TEST_DIR }}/{{ TEST_NAME }}/gfs.{{ PDY }}/{{ TEST_DATE | strftime('%H') }}/model/atmos/input/gfs_data.tile1.nc", "{{RUNTESTS}}/COMROOT/{{ TEST_NAME }}//gfs.{{PDY}}/{{HH}}/model/atmos/input/gfs_data.tile1.nc"]

aerorahul · 2025-01-08T20:40:50Z

ci/ctests/cases/C48_ATM_gfs_fcst_seg0.yaml

@@ -0,0 +1,17 @@
+input_files:


Suggested change

input_files:

{% set RUNTESTS = RUNTESTS | getenv %}

input_files:

something like this if one needs test specific local info. For e.g. in the DA jobs, we will need prior dates that are 6 hours behind TEST_DATE.

aerorahul · 2025-01-08T20:54:59Z

ci/ctests/CMakeLists.txt

+
+AddFunctionalTest(
+  CASE "C48_ATM"
+  JOB  "gfs_fcst_seg0"


Suggested change

JOB "gfs_fcst_seg0"

JOB "gfs_fcst_seg0"

DATE "2020011200" # Test date

…CMakeLists.txt to use PROJECT_SOURCE_DIR for HOMEgfs in templiting bash wrapper sripts

ctests/scripts/execute.sh.in

+rm -f ./*.db
+rm -f ./jobcard                                                                             
+
+job_card=$(yes | "${rocotoboot_dryrun}" -d "${TEST_NAME}.db" -w "${TEST_NAME}.xml" -v 10 -c "${idate}00" -t "${JOB}" 2> jobcard) || true


ctests/scripts/execute.sh.in

+rm -f ./jobcard                                                                             
+
+job_card=$(yes | "${rocotoboot_dryrun}" -d "${TEST_NAME}.db" -w "${TEST_NAME}.xml" -v 10 -c "${idate}00" -t "${JOB}" 2> jobcard) || true
+cat jobcard | sed '/^{{\|^}}/d' | sed '1d'  > "${TEST_NAME}.sub" || true


ctests/scripts/stage.sh.in

+TEST_NAME=${1:?"Name of the test is required"}
+
+# CMake to fill these variables
+STAGED_FUNCTIONAL_TEST_DIR="@STAGED_FUNCTIONAL_TEST_DIR@"


ctests/scripts/stage.sh.in

+
+# CMake to fill these variables
+STAGED_FUNCTIONAL_TEST_DIR="@STAGED_FUNCTIONAL_TEST_DIR@"
+RUNTESTS="@RUNTESTS@"


ctests/scripts/setup.sh.in

+
+TEST_NAME=${1:?"Name of the test is required"}
+YAML_FILE=${2:?"Name of the CI yaml file for the test"}
+TEST_DATE=${3:?"idate of the test is required"}


ctests/scripts/stage.sh.in

+
+# CMake to fill these variables
+STAGED_FUNCTIONAL_TEST_DIR="@STAGED_FUNCTIONAL_TEST_DIR@"
+RUNTESTS="@RUNTESTS@"


Terry McGUinness and others added 30 commits December 6, 2024 10:55

added ctest dir with cmake to create an experment

987ae5b

added CMakeLists for creating initial case EXPDIR usint ctest

61fcf83

ready to test getting the job names

f5f6595

move all operations of creating EXPDIR cases into configure phase

cbe3bab

got cmake configure to work for EXPDIR

6a50e14

can not do both file and variable save for execute_process

bcebeca

working cmake config up to output of job names

be7a890

Merge branch 'NOAA-EMC:develop' into ctest_jjobs_framework

2811629

loop over jobnames and create an experment for each

70a8183

update JOB name update in loop

72e5a9c

addded debug outputs

1f1fcd6

modifying plslot list to be only on a per case

ee27b02

filter pslot list for only relevent cases

bc76ddf

refining get_job_names

38838a6

modigied get_jobs_names to task and metatask names

d18ad01

got better jobname parcer working

ef0e066

Merge branch 'NOAA-EMC:develop' into ctest_jjobs_framework

519a919

replaced generate_workflows with create_experment

f813a1b

removed non-existing ENVIRONMENT setting to consolated COMMAND line

8238228

got cmake script to use creat_experiment

bf579ee

Merge branch 'NOAA-EMC:develop' into ctest_jjobs_framework

28ef1af

just getting started with addeding a ctest

374927f

add in test for configured project

8b4fc4f

added ctest and testing

ce37d86

Merge branch 'NOAA-EMC:develop' into ctest_jjobs_framework

d4e3266

moved ctests dir to top HOMEgfs/ci/ctests with its own scripts dir

bb12cda

added all jobs for tests for each case

f78181c

added data stagging dependances and started it as a stub script

46cddb7

Merge branch 'NOAA-EMC:develop' into ctest_jjobs_framework

d6c3a49

added driver for creating ctests from yaml files

434f869

Terry McGuinness added 2 commits January 8, 2025 12:17

oops need done at then of loop in execute bash sript

42084e7

removed C48_S2SW.yaml for functional test as it is not ready yet

8d71151

TerrenceMcGuinness-NOAA requested a review from WalterKolczynski-NOAA as a code owner January 8, 2025 18:57

github-advanced-security bot found potential problems Jan 8, 2025

View reviewed changes

TerrenceMcGuinness-NOAA marked this pull request as draft January 8, 2025 19:00

TerrenceMcGuinness-NOAA requested a review from aerorahul January 8, 2025 19:13

working on shellnorms

cf7be28

github-advanced-security bot found potential problems Jan 8, 2025

View reviewed changes

Terry McGuinness added 3 commits January 8, 2025 14:05

working on more shellnorms

a49a642

working on even more shellnorms

466831d

working couple more shellnorms

2d18461

aerorahul reviewed Jan 8, 2025

View reviewed changes

ci/ctests/scripts/setup.sh.in Outdated Show resolved Hide resolved

aerorahul reviewed Jan 8, 2025

View reviewed changes

TerrenceMcGuinness-NOAA and others added 2 commits January 10, 2025 10:26

Merge branch 'NOAA-EMC:develop' into ctest_jjobs_framework

53b13e3

removed scripts used for getting system defaults and added top level …

4ee8d81

…CMakeLists.txt to use PROJECT_SOURCE_DIR for HOMEgfs in templiting bash wrapper sripts

github-advanced-security bot found potential problems Jan 10, 2025

View reviewed changes

added to yaml fun conf file to us jinja string modification TEST_DATE

b3518ca

github-advanced-security bot found potential problems Jan 10, 2025

View reviewed changes

Terry McGuinness added 3 commits January 10, 2025 17:24

typos in CMakeLists and module file for adding CMake on Orion

54f633b

added TEST_DATE as an agrment to adding a test in CMake List file

a182c44

needed idate in arg to exexcute

fd8a8bc

github-advanced-security bot found potential problems Jan 11, 2025

View reviewed changes

Terry McGuinness added 4 commits January 10, 2025 18:35

just use par_yaml

472d89c

trying to get args into yaml

45d2901

added TEST_DATA via data to template

8ca4d7a

added TEST_DATE into arg to bash stage

91d76e3

github-advanced-security bot found potential problems Jan 11, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI JJOB Tests using CMake #3214

CI JJOB Tests using CMake #3214

TerrenceMcGuinness-NOAA commented Jan 8, 2025 •

edited

Loading

github-advanced-security bot left a comment

aerorahul Jan 8, 2025

aerorahul Jan 8, 2025

aerorahul Jan 8, 2025

aerorahul Jan 8, 2025

	- ["{{ STAGED_FUNCTIONAL_TEST_DIR }}/{{ TEST_NAME }}/gfs.{{ PDY }}/{{ HH }}/model/atmos/input/gfs_ctrl.nc", "{{RUNTESTS}}/COMROOT/{{ TEST_NAME }}//gfs.{{PDY}}/{{HH}}/model/atmos/input/gfs_ctrl.nc"]
	- ["{{ STAGED_FUNCTIONAL_TEST_DIR }}/{{ TEST_NAME }}/gfs.{{ TEST_DATE \| to_YMD }}/{{ HH }}/model/atmos/input/gfs_ctrl.nc", "{{RUNTESTS}}/COMROOT/{{ TEST_NAME }}//gfs.{{PDY}}/{{HH}}/model/atmos/input/gfs_ctrl.nc"]

	input_files:
	{% set RUNTESTS = RUNTESTS \| getenv %}
	input_files:

	JOB "gfs_fcst_seg0"
	JOB "gfs_fcst_seg0"
	DATE "2020011200" # Test date

CI JJOB Tests using CMake #3214

Are you sure you want to change the base?

CI JJOB Tests using CMake #3214

Conversation

TerrenceMcGuinness-NOAA commented Jan 8, 2025 • edited Loading

Description

Type of change

Change characteristics

How has this been tested?

Checklist

github-advanced-security bot left a comment

Choose a reason for hiding this comment

aerorahul Jan 8, 2025

Choose a reason for hiding this comment

aerorahul Jan 8, 2025

Choose a reason for hiding this comment

aerorahul Jan 8, 2025

Choose a reason for hiding this comment

aerorahul Jan 8, 2025

Choose a reason for hiding this comment

TerrenceMcGuinness-NOAA commented Jan 8, 2025 •

edited

Loading