Update hashes for select JEDI submodules #1298

RussTreadon-NOAA · 2024-09-26T10:29:49Z

Many of the JEDI hashes referenced in sorc are two or more months old. This is OK for some submodules since they do not frequently change (e.g, icepack @ 73136ee). Other submodules change more frequently. For example, oops @ e6485c0 is 18 commits behind the current head of oops develop.

Script ush/submodules/update_develop.sh updates the GDASApp hash for the following JEDI repos

oops
vader
saber
ioda
ufo
fv3-jedi
soca
iodaconv

to the current head of their respective develop.

This issue is opened to document the updating of the above GDASApp submodules using update_develop.sh

The text was updated successfully, but these errors were encountered:

RussTreadon-NOAA · 2024-09-26T10:46:39Z

@danholdaway , @emilyhcliu , @ADCollard , @CoryMartin-NOAA , @guillaumevernieres , @DavidNew-NOAA : When's a good time to update the submodule hashes in GDASApp sorc/?

I did so in a working copy of develop at c797d33 on 9/25/2024 using update_develop.sh. The updated GDASApp successfully built. All test_gdasapp tests pass apart from

        1949 - test_gdasapp_atm_jjob_var_run (Failed)
        1955 - test_gdasapp_atm_jjob_ens_obs (Failed)
        1956 - test_gdasapp_atm_jjob_ens_sol (Failed)

Each of these tests needed an updated reference file in test/atm/global-workflow.

Shall we update submodule hashes now or wait until we update to spack-stack/1.8 (#1283). It looks like spack-stack/1.8 will not be officially released and installed on all platforms until the second half of October 2024.

danholdaway · 2024-09-26T13:04:05Z

How much do the references need to change by? It's unfortunate that we cannot know what changes in JEDI causes the references to change. Perhaps we need a nightly mechanism to track things like that closer to real time.

RussTreadon-NOAA · 2024-09-26T13:43:43Z

test_gdasapp_atm_jjob_var_run increment rms differ in the 13th to 14th printed digit for wind components and temperature.
RMS increments differ in the 8th digit for specific humidity, 4th digit for cloud liquid ice, and 3rd digit for cloud liquid water.

test_gdasapp_atm_jjob_ens_obs H(x) RMS for members differ in the 7th digit for AMSUA_N19 and the first digit for sondes. The member RMS for sondes is 141.1993510066664 for develop. It drops to 66.47644691726953 when the submodule hashes are updated.

test_gdasapp_atm_jjob_ens_sol RMS for the analysis mean differs in the

12th digit for surface_pressure
7th digit for layer thickness
6th digit for air_pressure_thickness, ozone_mass_mixing_ratio
5th digit for eastward_wind, air_temprature
4th digit for northward_wind , specific_humidity
3rd digit for cloud_liquid_ice
2nd digit for cloud_liquid_water

danholdaway · 2024-09-26T14:26:19Z

That seems like a very large difference for the sondes h(x). Perhaps it's OK once we understand the nature of the changes. But when we update the hashes rarely it would be hard to know what JEDI changes occurred to cause the change. Some kind of nightly mechanism might be what we need to aim for.

RussTreadon-NOAA · 2024-09-26T16:37:03Z

ci/stable_driver.sh was designed to check and update, as needed, jedi hashes on a nightly basis. It ran under Cory's account. It wasn't working properly and we stopped the cron. We can revisit this script and see if we can get it running again.

RussTreadon-NOAA · 2024-09-26T16:52:13Z

Comparison of the sondes RMS values in the two test_gdasapp_atm_jjob_ens_obs suggests that different observations went into the calculations.

GDASApp develop has

0: Test     : sondes nobs= 8481 Min=-33.72958087992568, Max=309.9582199829497, RMS=141.2113994268562
0: Test     : sondes nobs= 8481 Min=-30.53144491152982, Max=310.2016372514533, RMS=141.1993510066664
0: Test     : sondes nobs= 8481 Min=-32.86095389873932, Max=310.3775309250524, RMS=141.1778363650817
0: Test     : sondes nobs= 8481 Min=-32.28721130380135, Max=310.1791293864851, RMS=141.1946022453276
0: Test     : sondes nobs= 3950 Min=-16.79891131775619, Max=16.07422994026274, RMS=4.060200040073279

whereas after we update JEDI hashes we have

0: Test     : sondes nobs= 2812 Min=-33.46923510113491, Max=237.4511515234925, RMS=66.49450460686083
0: Test     : sondes nobs= 2812 Min=-30.53144491152982, Max=237.5347783619015, RMS=66.47644691726953
0: Test     : sondes nobs= 2812 Min=-32.86095389873932, Max=237.9311435831063, RMS=66.51314984035037
0: Test     : sondes nobs= 8481 Min=-32.28721130380135, Max=310.1791293864851, RMS=141.1946022453276
0: Test     : sondes nobs= 3950 Min=-16.79891131775619, Max=16.07422994026274, RMS=4.060200040073279

Only the first three sondes nobs lines differ. The fourth and fifth lines are identical between to two runs. The first three lines correspond to H(x) for members 1, 2, 3. The fourth line is H(x) ensemble background mean. The last line is background y - H(x).

The number of nobs from develop is noticeably greater than when using the updated hashes: 8481 versus 2812. I wonder if the develop code computes these statistics using all observations whereas the run with the updated hashes only uses observations passing QC.

RussTreadon-NOAA · 2024-09-27T15:20:31Z

Attempts to manually execute ci/stable_driver.sh on Hera as user role.jedipara are running into batch queue problems.

      Start 1943: test_gdasapp_WCDA-3DVAR-C48mx500_gdasstage_ic_202103241200
24/46 Test #1943: test_gdasapp_WCDA-3DVAR-C48mx500_gdasstage_ic_202103241200 .........***Timeout 1500.10 sec
booting gdasstage_ic for cycle 202103241200
Can not boot task 'gdasstage_ic' for cycle '202103241200' because a job for it already exists in state QUEUED.  You need to rewind this task instead.

The queue wait time for batch jobs is hours!

RussTreadon-NOAA · 2024-09-27T20:54:40Z

test_gdasapp_WCDA-3DVAR-C48mx500

Need to consider turning off test_gdasapp_WCDA-3DVAR-C48mx500 tests. stable_driver.sh keeps failing on Hera due to non-zero return code from test_gdasapp_WCDA-3DVAR-C48mx500 jobs. For example,

23/34 Test #1942: test_gdasapp_WCDA-3DVAR-C48mx500 ...................................   Passed   11.97 sec
      Start 1943: test_gdasapp_WCDA-3DVAR-C48mx500_gdasstage_ic_202103241200
24/34 Test #1943: test_gdasapp_WCDA-3DVAR-C48mx500_gdasstage_ic_202103241200 .........***Timeout 1500.18 sec
booting gdasstage_ic for cycle 202103241200
Can not boot task 'gdasstage_ic' for cycle '202103241200' because a job for it already exists in state QUEUED.  You need to rewind this task instead.
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED

      Start 1944: test_gdasapp_WCDA-3DVAR-C48mx500_gdasfcst_202103241200
25/34 Test #1944: test_gdasapp_WCDA-3DVAR-C48mx500_gdasfcst_202103241200 .............   Passed  491.97 sec
      Start 1945: test_gdasapp_WCDA-3DVAR-C48mx500_gdasprepoceanobs_202103241800
26/34 Test #1945: test_gdasapp_WCDA-3DVAR-C48mx500_gdasprepoceanobs_202103241800 .....   Passed  457.43 sec
      Start 1946: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarinebmat_202103241800
27/34 Test #1946: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarinebmat_202103241800 .......***Timeout 1499.03 sec
booting gdasmarinebmat for cycle 202103241800
task 'gdasmarinebmat' for cycle '202103241800' has been booted
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: RUNNING
The task is in state: RUNNING
The task is in state: RUNNING
The task is in state: RUNNING
The task is in state: RUNNING
The task is in state: RUNNING
The task is in state: SUBMITTING
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED

      Start 1947: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlinit_202103241800

Another issue with test_gdasapp_WCDA-3DVAR-C48mx500. In order to run these tests we need g-w. If a developer clones GDASApp along and executes ./build.sh, the script fails during the configure step with

-- Adding bundle project gdas
-- ---------------------------------------------------------
-- ^[[32m[GDASApp] (1.0.0) [2be1fcd]^[[m
-- Feature TESTS enabled
-- Found Python3: /apps/spack/python/3.8.6/intel/19.1.3.304/pjn2nzkjvqgmjw4hmyz43v5x4jbxjzpk/bin/python3.8 (found version "3.8.6") found components: Interpreter
Traceback (most recent call last):
  File "/lfs/h2/emc/da/noscrub/russ.treadon/git/GDASApp/test2/bundle/gdas/test/gw-ci/get_cycles.py", line 38, in <module>
    main()
  File "/lfs/h2/emc/da/noscrub/russ.treadon/git/GDASApp/test2/bundle/gdas/test/gw-ci/get_cycles.py", line 32, in main
    idate_value = read_idate_from_yaml(args.yaml_file)
  File "/lfs/h2/emc/da/noscrub/russ.treadon/git/GDASApp/test2/bundle/gdas/test/gw-ci/get_cycles.py", line 8, in read_idate_from_yaml
    with open(file_path, 'r') as file:
FileNotFoundError: [Errno 2] No such file or directory: '/lfs/h2/emc/da/noscrub/russ.treadon/git/GDASApp/test2/bundle/../../../ci/cases/pr/C48mx500_3DVarAOWCDA.yaml'
CMake Error at gdas/test/gw-ci/CMakeLists.txt:17 (string):
  string sub-command REPLACE requires at least four arguments.
Call Stack (most recent call first):
  gdas/test/gw-ci/CMakeLists.txt:63 (add_cycling_tests)

...

-- Configuring incomplete, errors occurred!
See also "/lfs/h2/emc/da/noscrub/russ.treadon/git/GDASApp/test2/build/CMakeFiles/CMakeOutput.log".
See also "/lfs/h2/emc/da/noscrub/russ.treadon/git/GDASApp/test2/build/CMakeFiles/CMakeError.log".

Line 17 of test/gw-ci/CMakeLists.txt is the REPLACE line below

  # Get the 1/2 cycle and full cycle's dates
  execute_process(
    COMMAND ${CMAKE_COMMAND} -E env python ${PROJECT_SOURCE_DIR}/test/gw-ci/get_cycles.py ${YAML_PATH}
    OUTPUT_VARIABLE SCRIPT_OUTPUT
    RESULT_VARIABLE SCRIPT_RESULT
    OUTPUT_STRIP_TRAILING_WHITESPACE
  )
  string(REPLACE "," ";" DATES_LIST ${SCRIPT_OUTPUT})
  list(GET DATES_LIST 0 HALF_CYCLE)
  list(GET DATES_LIST 1 FULL_CYCLE)

Got around this by making the following modification to test/CMakeLists.txt

@@ -138,7 +138,9 @@ add_test(NAME test_gdasapp_jedi_increment_to_fv3
 if (${BUILD_GDASBUNDLE})
   add_subdirectory(fv3jedi)   # fv3jedi tests
   add_subdirectory(snow)      # snow da tests
-  add_subdirectory(gw-ci)   # subset of the global-workflow ci tests
+  if (WORKFLOW_TESTS)
+    add_subdirectory(gw-ci)   # subset of the global-workflow ci tests
+  endif()
 endif()
 
 # gdas atm tests

guillaumevernieres · 2024-09-28T00:44:30Z

I don't know why they would time out more than the other tests @RussTreadon-NOAA , most of these jobs only request 1 node.

RussTreadon-NOAA · 2024-09-28T00:57:52Z

@guillaumevernieres : Recently queue wait times have been extremely long on Hera. The queue is swamped. As of 8:56 pm, 9/27/2024, 1322 jobs are queued. 371 are running. 951 are pending. Even a 1 node job will sit in the batch queue a long time.

guillaumevernieres · 2024-09-28T14:26:53Z

I understand that @RussTreadon-NOAA , but why isolate the wcda tests? Aren't the other batch jobs also staying in the queue for ever?
We need to start testing on hercules ... or swap the job queue to debug.

RussTreadon-NOAA · 2024-09-28T15:03:15Z

@guillaumevernieres , not sure why WCDA jobs sit in the queue longer. We should compare how they are submitted with the submission of other test_gdasapp batch jobs. Yes, using the debug queue will certainly improve turn around. If we have multiple instances of the ctests running this will be problematic. debug only allows jobs in the queue per user at any given time.

Hercules, Orion, and WCOSS2 queue wait times are generally shorter but this will vary based on system load. On Hera, Hercules, and Orion I think queue wait time is also a function of how quickly we are consuming our monthly core hour allocation.

guillaumevernieres · 2024-09-28T15:55:03Z

@RussTreadon-NOAA , whatever the g-w does to submit jobs is what these tests are using. What I should do is add dependencies to the ctests to stop trying to run the entire sequence if one wcda ctest fails.

RussTreadon-NOAA · 2024-09-30T14:02:39Z

The current approach of using batch on Hera isn't working. stable_driver.sh fails because ctests do not succesfully complete due to sitting in the queue beyond our specified 1500 seconds.

Throughput is much better when I switch the test_atm_jjob suite to debug. Changing g-w jobs to debug require us to edit machine specific yamls in workflow/hosts. For example, hera.yaml sets

QUEUE: batch
QUEUE_SERVICE: batch

Work is needed in GDASApp to toggle to debug. One thing to remember with debug (at least on Hera, Hercules, and Orion) is that each user can only have two debug jobs in the queue at a time. We could potentially run into problems on Hera if role.jedipara is concurrently running stable_driver.sh and GW-RT using debug.

RussTreadon-NOAA · 2024-09-30T14:45:58Z

For the sake of completeness, here's the log.ctest from executing stable_driver.sh under role.jedipara on Hera.

Test project /scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/stable/20240930/global-workflow/sorc/gdas.cd/build
      Start 1570: test_gdasapp_util_coding_norms
 1/46 Test #1570: test_gdasapp_util_coding_norms .....................................   Passed    2.47 sec
      Start 1571: test_gdasapp_util_ioda_example
 2/46 Test #1571: test_gdasapp_util_ioda_example .....................................   Passed    2.67 sec
      Start 1572: test_gdasapp_util_prepdata
 3/46 Test #1572: test_gdasapp_util_prepdata .........................................   Passed    1.23 sec
      Start 1573: test_gdasapp_util_rads2ioda
 4/46 Test #1573: test_gdasapp_util_rads2ioda ........................................   Passed    0.15 sec
      Start 1574: test_gdasapp_util_ghrsst2ioda
 5/46 Test #1574: test_gdasapp_util_ghrsst2ioda ......................................   Passed    0.14 sec
      Start 1575: test_gdasapp_util_rtofstmp
 6/46 Test #1575: test_gdasapp_util_rtofstmp .........................................   Passed    1.05 sec
      Start 1576: test_gdasapp_util_rtofssal
 7/46 Test #1576: test_gdasapp_util_rtofssal .........................................   Passed    0.91 sec
      Start 1577: test_gdasapp_util_smap2ioda
 8/46 Test #1577: test_gdasapp_util_smap2ioda ........................................   Passed    0.14 sec
      Start 1578: test_gdasapp_util_smos2ioda
 9/46 Test #1578: test_gdasapp_util_smos2ioda ........................................   Passed    0.15 sec
      Start 1579: test_gdasapp_util_viirsaod2ioda
10/46 Test #1579: test_gdasapp_util_viirsaod2ioda ....................................   Passed    0.14 sec
      Start 1580: test_gdasapp_util_icecamsr2ioda
11/46 Test #1580: test_gdasapp_util_icecamsr2ioda ....................................   Passed    0.14 sec
      Start 1581: test_gdasapp_util_icecmirs2ioda
12/46 Test #1581: test_gdasapp_util_icecmirs2ioda ....................................   Passed    0.14 sec
      Start 1582: test_gdasapp_util_icecjpssrr2ioda
13/46 Test #1582: test_gdasapp_util_icecjpssrr2ioda ..................................   Passed    0.14 sec
      Start 1933: test_gdasapp_check_python_norms
14/46 Test #1933: test_gdasapp_check_python_norms ....................................   Passed    3.00 sec
      Start 1934: test_gdasapp_check_yaml_keys
15/46 Test #1934: test_gdasapp_check_yaml_keys .......................................   Passed    0.18 sec
      Start 1935: test_gdasapp_jedi_increment_to_fv3
16/46 Test #1935: test_gdasapp_jedi_increment_to_fv3 .................................   Passed    1.02 sec
      Start 1936: test_gdasapp_fv3jedi_fv3inc
17/46 Test #1936: test_gdasapp_fv3jedi_fv3inc ........................................   Passed   21.54 sec
      Start 1937: test_gdasapp_snow_create_ens
18/46 Test #1937: test_gdasapp_snow_create_ens .......................................   Passed    1.64 sec
      Start 1938: test_gdasapp_snow_imsproc
19/46 Test #1938: test_gdasapp_snow_imsproc ..........................................   Passed    3.36 sec
      Start 1939: test_gdasapp_snow_apply_jediincr
20/46 Test #1939: test_gdasapp_snow_apply_jediincr ...................................   Passed    6.14 sec
      Start 1940: test_gdasapp_snow_letkfoi_snowda
21/46 Test #1940: test_gdasapp_snow_letkfoi_snowda ...................................   Passed   14.37 sec
      Start 1941: test_gdasapp_convert_bufr_adpsfc_snow
22/46 Test #1941: test_gdasapp_convert_bufr_adpsfc_snow ..............................   Passed    3.10 sec
      Start 1942: test_gdasapp_WCDA-3DVAR-C48mx500
23/46 Test #1942: test_gdasapp_WCDA-3DVAR-C48mx500 ...................................   Passed    6.92 sec
      Start 1943: test_gdasapp_WCDA-3DVAR-C48mx500_gdasstage_ic_202103241200
24/46 Test #1943: test_gdasapp_WCDA-3DVAR-C48mx500_gdasstage_ic_202103241200 .........***Timeout 1500.22 sec
booting gdasstage_ic for cycle 202103241200
Can not boot task 'gdasstage_ic' for cycle '202103241200' because a job for it already exists in state QUEUED.  You need to rewind this task instead.
The task is in state: QUEUED
The task is in state: QUEUED
The task is in state: QUEUED

---

      Start 1965: test_gdasapp_aero_gen_3dvar_yaml
46/46 Test #1965: test_gdasapp_aero_gen_3dvar_yaml ...................................   Passed    0.85 sec

63% tests passed, 17 tests failed out of 46

Label Time Summary:
gdas-utils    =   9.48 sec*proc (13 tests)
manual        = 10014.68 sec*proc (9 tests)
script        =   9.48 sec*proc (13 tests)

Total Test time (real) = 11384.80 sec

The following tests FAILED:
        1943 - test_gdasapp_WCDA-3DVAR-C48mx500_gdasstage_ic_202103241200 (Timeout)
        1944 - test_gdasapp_WCDA-3DVAR-C48mx500_gdasfcst_202103241200 (Timeout)
        1946 - test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarinebmat_202103241800 (Timeout)
        1947 - test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlinit_202103241800 (Timeout)
        1948 - test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlvar_202103241800 (Timeout)
        1949 - test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlchkpt_202103241800 (Failed)
        1950 - test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlfinal_202103241800 (Timeout)
        1954 - test_gdasapp_atm_jjob_var_init (Failed)
        1955 - test_gdasapp_atm_jjob_var_run (Failed)
        1956 - test_gdasapp_atm_jjob_var_inc (Failed)
        1957 - test_gdasapp_atm_jjob_var_final (Failed)
        1958 - test_gdasapp_atm_jjob_ens_init (Failed)
        1959 - test_gdasapp_atm_jjob_ens_letkf (Failed)
        1961 - test_gdasapp_atm_jjob_ens_obs (Failed)
        1962 - test_gdasapp_atm_jjob_ens_sol (Failed)
        1963 - test_gdasapp_atm_jjob_ens_inc (Failed)
        1964 - test_gdasapp_atm_jjob_ens_final (Failed)
Errors while running CTest

For the full ctest listing view /scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/stable/20240930/global-workflow/sorc/gdas.cd/build/log.ctest

RussTreadon-NOAA · 2024-10-01T12:40:22Z

Work for this issue will be done in feature/update_hashes

RussTreadon-NOAA · 2024-10-01T16:22:50Z

test_gdasapp_snow_letkfoi_snowda failed with the update JEDI hashes committed at 52b5354.

Check sorc/fv3-jedi/test/testinput/letkf_snow.yaml and see that the .nc suffix is required for filename_orog. test/snow/letkfoi_snow.yaml had filename_orog: C48_oro_data. Add .nc. Rerun of test_gdasapp_snow_letkfoi_snowda with this change passed. Commit change at 7d92fa4.

RussTreadon-NOAA · 2024-10-01T19:37:57Z

test_gdasapp_WCDA-3DVAR-C48mx500 failures

After updating jedi hashes using ush/submodule/update_develop.sh in feature/update_hashes the following WCDA tests failed

        1949 - test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarinebmat_202103241800 (Failed)
        1950 - test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlinit_202103241800 (Failed)
        1951 - test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlvar_202103241800 (Failed)
        1952 - test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlchkpt_202103241800 (Failed)
        1953 - test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlfinal_202103241800 (Failed)

A check of test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarinebmat_202103241800 found that it failed with

 0: NOTE from PE     0: close_param_file: MOM_input has been closed successfully.
 7: Exception: EXPLICIT_DIFFUSION does not exist in saber::SaberCentralBlockFactory
 7: Exception: oops::ErrorCovarianceToolbox<SOCA> terminating...
 7: Exception: level: 0
 7: EXPLICIT_DIFFUSION does not exist in saber::SaberCentralBlockFactory
11: Exception: EXPLICIT_DIFFUSION does not exist in saber::SaberCentralBlockFactory
11: Exception: oops::ErrorCovarianceToolbox<SOCA> terminating...
11: Exception: level: 0
11: EXPLICIT_DIFFUSION does not exist in saber::SaberCentralBlockFactory

Looking at yamls in sorc/soca/test/testinput it looks like

saber block name: EXPLICIT_DIFFUSION

has been replaced by

saber block name: diffusion

This change was made in the following files

        modified:   parm/soca/berror/obsolete/soca_parameters_diffusion_hz.yaml
        modified:   parm/soca/berror/obsolete/soca_parameters_diffusion_vt.yaml
        modified:   parm/soca/berror/soca_hybrid_bmat.yaml
        modified:   parm/soca/berror/soca_parameters_diffusion_hz.yaml.j2
        modified:   parm/soca/berror/soca_parameters_diffusion_vt.yaml.j2
        modified:   parm/soca/berror/soca_static_bmat.yaml

After this change job gdasmarinebmat failed in a different place

0: ==================================================================================
 0:  saber::Diffusion calibration
 0: ----------------------------------------------------------------------------------
 0:
 0: group 1 of 2
 0: Generating horizontal scales
 0: Exception: createScales: either "fixed value" or "model file" should be set.
 6: Exception: createScales: either "fixed value" or "model file" should be set.
 1: Exception: createScales: either "fixed value" or "model file" should be set.
 2: Exception: createScales: either "fixed value" or "model file" should be set.
 7: Exception: createScales: either "fixed value" or "model file" should be set.
 9: Exception: createScales: either "fixed value" or "model file" should be set.
11: Exception: createScales: either "fixed value" or "model file" should be set.

Rather than keep guessing as to what to change and where to change, I'm reaching out to @guillaumevernieres . Do you know where / what to change given the above error message from gdasmarinebmat ?

My working copy of GDASApp uses soca at 4d7ef21 and saber at 1ca1596

test_gdasapp_atm_jjob failures

        1958 - test_gdasapp_atm_jjob_var_run (Failed)
        1964 - test_gdasapp_atm_jjob_ens_obs (Failed)
        1965 - test_gdasapp_atm_jjob_ens_sol (Failed)

The var_run, ens_obs, and ens_sol jobs failed the reference check (noted above). The following reference files were updated.

        modified:   test/atm/global-workflow/3dvar.ref
        modified:   test/atm/global-workflow/lgetkf_observer.ref
        modified:   test/atm/global-workflow/lgetkf_solver.ref

After this all test_gdasapp_atm_jjob passed

Test project /work/noaa/da/rtreadon/git/global-workflow/hashes/sorc/gdas.cd/build
      Start 1957: test_gdasapp_atm_jjob_var_init
 1/11 Test #1957: test_gdasapp_atm_jjob_var_init .........   Passed   45.90 sec
      Start 1958: test_gdasapp_atm_jjob_var_run
 2/11 Test #1958: test_gdasapp_atm_jjob_var_run ..........   Passed  106.25 sec
      Start 1959: test_gdasapp_atm_jjob_var_inc
 3/11 Test #1959: test_gdasapp_atm_jjob_var_inc ..........   Passed  298.29 sec
      Start 1960: test_gdasapp_atm_jjob_var_final
 4/11 Test #1960: test_gdasapp_atm_jjob_var_final ........   Passed   74.24 sec
      Start 1961: test_gdasapp_atm_jjob_ens_init
 5/11 Test #1961: test_gdasapp_atm_jjob_ens_init .........   Passed   45.98 sec
      Start 1962: test_gdasapp_atm_jjob_ens_letkf
 6/11 Test #1962: test_gdasapp_atm_jjob_ens_letkf ........   Passed  490.31 sec
      Start 1963: test_gdasapp_atm_jjob_ens_init_split
 7/11 Test #1963: test_gdasapp_atm_jjob_ens_init_split ...   Passed   45.79 sec
      Start 1964: test_gdasapp_atm_jjob_ens_obs
 8/11 Test #1964: test_gdasapp_atm_jjob_ens_obs ..........   Passed   42.28 sec
      Start 1965: test_gdasapp_atm_jjob_ens_sol
 9/11 Test #1965: test_gdasapp_atm_jjob_ens_sol ..........   Passed   42.26 sec
      Start 1966: test_gdasapp_atm_jjob_ens_inc
10/11 Test #1966: test_gdasapp_atm_jjob_ens_inc ..........   Passed   42.25 sec
      Start 1967: test_gdasapp_atm_jjob_ens_final
11/11 Test #1967: test_gdasapp_atm_jjob_ens_final ........   Passed   42.23 sec

100% tests passed, 0 tests failed out of 11

Total Test time (real) = 1276.12 sec

guillaumevernieres · 2024-10-01T19:46:55Z

Thanks for testing @RussTreadon-NOAA , I can take care of the soca update.

RussTreadon-NOAA · 2024-10-01T19:49:13Z

Thank you @guillaumevernieres for your help. Feel free to push commits to feature/update_hashes

…1298)

guillaumevernieres · 2024-10-02T12:46:01Z

Thank you @guillaumevernieres for your help. Feel free to push commits to feature/update_hashes

I'm planning to spend the morning on this @RussTreadon-NOAA .

RussTreadon-NOAA · 2024-10-02T13:06:17Z

Thank you @guillaumevernieres !

guillaumevernieres · 2024-10-02T23:38:43Z

It took "a bit" longer than expected ... Oops. I have 2 PR's for you to review @RussTreadon-NOAA , one in jcb-gdas and one in this repo into your branch.

RussTreadon-NOAA · 2024-10-03T11:44:39Z

feature/update_hashes at 3a4616e along with changes to

        modified:   ush/python/pygfs/task/marine_analysis.py
        modified:   ush/python/pygfs/utils/marine_da_utils.py

from @guillaumevernieres g-w feature/update_hashes allow all test_gdasapp ctests to pass

Test project /work/noaa/da/rtreadon/git/global-workflow/hashes/sorc/gdas.cd/build
      Start 1573: test_gdasapp_util_coding_norms
 1/46 Test #1573: test_gdasapp_util_coding_norms .....................................   Passed    1.56 sec
      Start 1574: test_gdasapp_util_ioda_example
 2/46 Test #1574: test_gdasapp_util_ioda_example .....................................   Passed    0.11 sec
      Start 1575: test_gdasapp_util_prepdata
 3/46 Test #1575: test_gdasapp_util_prepdata .........................................   Passed    1.02 sec
      Start 1576: test_gdasapp_util_rads2ioda
 4/46 Test #1576: test_gdasapp_util_rads2ioda ........................................   Passed    0.14 sec
      Start 1577: test_gdasapp_util_ghrsst2ioda
 5/46 Test #1577: test_gdasapp_util_ghrsst2ioda ......................................   Passed    0.12 sec
      Start 1578: test_gdasapp_util_rtofstmp
 6/46 Test #1578: test_gdasapp_util_rtofstmp .........................................   Passed    0.74 sec
      Start 1579: test_gdasapp_util_rtofssal
 7/46 Test #1579: test_gdasapp_util_rtofssal .........................................   Passed    0.61 sec
      Start 1580: test_gdasapp_util_smap2ioda
 8/46 Test #1580: test_gdasapp_util_smap2ioda ........................................   Passed    0.13 sec
      Start 1581: test_gdasapp_util_smos2ioda
 9/46 Test #1581: test_gdasapp_util_smos2ioda ........................................   Passed    0.11 sec
      Start 1582: test_gdasapp_util_viirsaod2ioda
10/46 Test #1582: test_gdasapp_util_viirsaod2ioda ....................................   Passed    0.12 sec
      Start 1583: test_gdasapp_util_icecamsr2ioda
11/46 Test #1583: test_gdasapp_util_icecamsr2ioda ....................................   Passed    0.11 sec
      Start 1584: test_gdasapp_util_icecmirs2ioda
12/46 Test #1584: test_gdasapp_util_icecmirs2ioda ....................................   Passed    0.12 sec
      Start 1585: test_gdasapp_util_icecjpssrr2ioda
13/46 Test #1585: test_gdasapp_util_icecjpssrr2ioda ..................................   Passed    0.11 sec
      Start 1936: test_gdasapp_check_python_norms
14/46 Test #1936: test_gdasapp_check_python_norms ....................................   Passed    1.90 sec
      Start 1937: test_gdasapp_check_yaml_keys
15/46 Test #1937: test_gdasapp_check_yaml_keys .......................................   Passed    0.09 sec
      Start 1938: test_gdasapp_jedi_increment_to_fv3
16/46 Test #1938: test_gdasapp_jedi_increment_to_fv3 .................................   Passed    0.49 sec
      Start 1939: test_gdasapp_fv3jedi_fv3inc
17/46 Test #1939: test_gdasapp_fv3jedi_fv3inc ........................................   Passed   32.17 sec
      Start 1940: test_gdasapp_snow_create_ens
18/46 Test #1940: test_gdasapp_snow_create_ens .......................................   Passed    1.00 sec
      Start 1941: test_gdasapp_snow_imsproc
19/46 Test #1941: test_gdasapp_snow_imsproc ..........................................   Passed    2.47 sec
      Start 1942: test_gdasapp_snow_apply_jediincr
20/46 Test #1942: test_gdasapp_snow_apply_jediincr ...................................   Passed    6.65 sec
      Start 1943: test_gdasapp_snow_letkfoi_snowda
21/46 Test #1943: test_gdasapp_snow_letkfoi_snowda ...................................   Passed   20.64 sec
      Start 1944: test_gdasapp_convert_bufr_adpsfc_snow
22/46 Test #1944: test_gdasapp_convert_bufr_adpsfc_snow ..............................   Passed    2.29 sec
      Start 1945: test_gdasapp_WCDA-3DVAR-C48mx500
23/46 Test #1945: test_gdasapp_WCDA-3DVAR-C48mx500 ...................................   Passed   12.06 sec
      Start 1946: test_gdasapp_WCDA-3DVAR-C48mx500_gdasstage_ic_202103241200
24/46 Test #1946: test_gdasapp_WCDA-3DVAR-C48mx500_gdasstage_ic_202103241200 .........   Passed  203.99 sec
      Start 1947: test_gdasapp_WCDA-3DVAR-C48mx500_gdasfcst_202103241200
25/46 Test #1947: test_gdasapp_WCDA-3DVAR-C48mx500_gdasfcst_202103241200 .............   Passed  326.39 sec
      Start 1948: test_gdasapp_WCDA-3DVAR-C48mx500_gdasprepoceanobs_202103241800
26/46 Test #1948: test_gdasapp_WCDA-3DVAR-C48mx500_gdasprepoceanobs_202103241800 .....   Passed  338.94 sec
      Start 1949: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarinebmat_202103241800
27/46 Test #1949: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarinebmat_202103241800 .......   Passed  117.55 sec
      Start 1950: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlinit_202103241800
28/46 Test #1950: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlinit_202103241800 ....   Passed   56.02 sec
      Start 1951: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlvar_202103241800
29/46 Test #1951: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlvar_202103241800 .....   Passed  241.07 sec
      Start 1952: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlchkpt_202103241800
30/46 Test #1952: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlchkpt_202103241800 ...   Passed  189.79 sec
      Start 1953: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlfinal_202103241800
31/46 Test #1953: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlfinal_202103241800 ...   Passed  388.58 sec
      Start 1954: test_gdasapp_convert_bufr_adpsfc
32/46 Test #1954: test_gdasapp_convert_bufr_adpsfc ...................................   Passed    2.90 sec
      Start 1955: test_gdasapp_convert_gsi_satbias
33/46 Test #1955: test_gdasapp_convert_gsi_satbias ...................................   Passed    7.23 sec
      Start 1956: test_gdasapp_setup_atm_cycled_exp
34/46 Test #1956: test_gdasapp_setup_atm_cycled_exp ..................................   Passed    2.48 sec
      Start 1957: test_gdasapp_atm_jjob_var_init
35/46 Test #1957: test_gdasapp_atm_jjob_var_init .....................................   Passed  113.16 sec
      Start 1958: test_gdasapp_atm_jjob_var_run
36/46 Test #1958: test_gdasapp_atm_jjob_var_run ......................................   Passed  106.33 sec
      Start 1959: test_gdasapp_atm_jjob_var_inc
37/46 Test #1959: test_gdasapp_atm_jjob_var_inc ......................................   Passed   42.34 sec
      Start 1960: test_gdasapp_atm_jjob_var_final
38/46 Test #1960: test_gdasapp_atm_jjob_var_final ....................................   Passed   42.33 sec
      Start 1961: test_gdasapp_atm_jjob_ens_init
39/46 Test #1961: test_gdasapp_atm_jjob_ens_init .....................................   Passed   48.79 sec
      Start 1962: test_gdasapp_atm_jjob_ens_letkf
40/46 Test #1962: test_gdasapp_atm_jjob_ens_letkf ....................................   Passed  522.46 sec
      Start 1963: test_gdasapp_atm_jjob_ens_init_split
41/46 Test #1963: test_gdasapp_atm_jjob_ens_init_split ...............................   Passed   80.92 sec
      Start 1964: test_gdasapp_atm_jjob_ens_obs
42/46 Test #1964: test_gdasapp_atm_jjob_ens_obs ......................................   Passed  106.35 sec
      Start 1965: test_gdasapp_atm_jjob_ens_sol
43/46 Test #1965: test_gdasapp_atm_jjob_ens_sol ......................................   Passed   42.48 sec
      Start 1966: test_gdasapp_atm_jjob_ens_inc
44/46 Test #1966: test_gdasapp_atm_jjob_ens_inc ......................................   Passed   42.35 sec
      Start 1967: test_gdasapp_atm_jjob_ens_final
45/46 Test #1967: test_gdasapp_atm_jjob_ens_final ....................................   Passed   74.33 sec
      Start 1968: test_gdasapp_aero_gen_3dvar_yaml
46/46 Test #1968: test_gdasapp_aero_gen_3dvar_yaml ...................................   Passed    0.67 sec

100% tests passed, 0 tests failed out of 46

Label Time Summary:
gdas-utils    =   5.01 sec*proc (13 tests)
manual        = 1874.40 sec*proc (9 tests)
script        =   5.01 sec*proc (13 tests)

Total Test time (real) = 3182.84 sec

RussTreadon-NOAA · 2024-10-03T16:15:01Z

@danholdaway , @CoryMartin-NOAA , @emilyhcliu , @ADCollard , @DavidNew-NOAA , @guillaumevernieres : are we OK with the fact that updating the hashes requires updates to

test/atm/global-workflow/3dvar.ref
test/atm/global-workflow/lgetkf_observer.ref
test/atm/global-workflow/lgetkf_solver.ref

If yes, I'll open a PR to get feature/update_hashes into develop.

If no, what's the next step?

danholdaway · 2024-10-03T17:48:52Z

I'm OK with it and I don't see that we have much choice. Luckily we're still early in the Atm development so our science work will catch issues. Moving forward we should find a way to monitor what changing the hashes means for our tests more frequently to we can track the changes against changes to JCSDA code.

RussTreadon-NOAA · 2024-10-04T15:55:30Z

10/4/2024 update

Rerun ush/submodules/update_develop.sh on features/update_hashes at 1c70c4a. The hashes for the following jedi repos were updated

        modified:   sorc/fv3-jedi (new commits)
        modified:   sorc/oops (new commits)
        modified:   sorc/saber (new commits)
        modified:   sorc/ufo (new commits)
        modified:   sorc/vader (new commits)

Build GDASApp with updated modules on Hercules. Rerun ctests inside g-w. All tests pass

Test project /work/noaa/da/rtreadon/git/global-workflow/hashes/sorc/gdas.cd/build
      Start 1579: test_gdasapp_util_coding_norms
 1/46 Test #1579: test_gdasapp_util_coding_norms .....................................   Passed    7.11 sec
      Start 1580: test_gdasapp_util_ioda_example
 2/46 Test #1580: test_gdasapp_util_ioda_example .....................................   Passed    6.87 sec
      Start 1581: test_gdasapp_util_prepdata
 3/46 Test #1581: test_gdasapp_util_prepdata .........................................   Passed    5.80 sec
      Start 1582: test_gdasapp_util_rads2ioda
 4/46 Test #1582: test_gdasapp_util_rads2ioda ........................................   Passed    0.14 sec
      Start 1583: test_gdasapp_util_ghrsst2ioda
 5/46 Test #1583: test_gdasapp_util_ghrsst2ioda ......................................   Passed    0.10 sec
      Start 1584: test_gdasapp_util_rtofstmp
 6/46 Test #1584: test_gdasapp_util_rtofstmp .........................................   Passed    1.79 sec
      Start 1585: test_gdasapp_util_rtofssal
 7/46 Test #1585: test_gdasapp_util_rtofssal .........................................   Passed    0.69 sec
      Start 1586: test_gdasapp_util_smap2ioda
 8/46 Test #1586: test_gdasapp_util_smap2ioda ........................................   Passed    0.13 sec
      Start 1587: test_gdasapp_util_smos2ioda
 9/46 Test #1587: test_gdasapp_util_smos2ioda ........................................   Passed    0.12 sec
      Start 1588: test_gdasapp_util_viirsaod2ioda
10/46 Test #1588: test_gdasapp_util_viirsaod2ioda ....................................   Passed    0.14 sec
      Start 1589: test_gdasapp_util_icecamsr2ioda
11/46 Test #1589: test_gdasapp_util_icecamsr2ioda ....................................   Passed    0.12 sec
      Start 1590: test_gdasapp_util_icecmirs2ioda
12/46 Test #1590: test_gdasapp_util_icecmirs2ioda ....................................   Passed    0.14 sec
      Start 1591: test_gdasapp_util_icecjpssrr2ioda
13/46 Test #1591: test_gdasapp_util_icecjpssrr2ioda ..................................   Passed    0.14 sec
      Start 1942: test_gdasapp_check_python_norms
14/46 Test #1942: test_gdasapp_check_python_norms ....................................   Passed    4.88 sec
      Start 1943: test_gdasapp_check_yaml_keys
15/46 Test #1943: test_gdasapp_check_yaml_keys .......................................   Passed    2.36 sec
      Start 1944: test_gdasapp_jedi_increment_to_fv3
16/46 Test #1944: test_gdasapp_jedi_increment_to_fv3 .................................   Passed   15.08 sec
      Start 1945: test_gdasapp_fv3jedi_fv3inc
17/46 Test #1945: test_gdasapp_fv3jedi_fv3inc ........................................   Passed   33.87 sec
      Start 1946: test_gdasapp_snow_create_ens
18/46 Test #1946: test_gdasapp_snow_create_ens .......................................   Passed    1.47 sec
      Start 1947: test_gdasapp_snow_imsproc
19/46 Test #1947: test_gdasapp_snow_imsproc ..........................................   Passed    3.96 sec
      Start 1948: test_gdasapp_snow_apply_jediincr
20/46 Test #1948: test_gdasapp_snow_apply_jediincr ...................................   Passed    2.18 sec
      Start 1949: test_gdasapp_snow_letkfoi_snowda
21/46 Test #1949: test_gdasapp_snow_letkfoi_snowda ...................................   Passed    8.61 sec
      Start 1950: test_gdasapp_convert_bufr_adpsfc_snow
22/46 Test #1950: test_gdasapp_convert_bufr_adpsfc_snow ..............................   Passed    4.14 sec
      Start 1951: test_gdasapp_WCDA-3DVAR-C48mx500
23/46 Test #1951: test_gdasapp_WCDA-3DVAR-C48mx500 ...................................   Passed   27.63 sec
      Start 1952: test_gdasapp_WCDA-3DVAR-C48mx500_gdasstage_ic_202103241200
24/46 Test #1952: test_gdasapp_WCDA-3DVAR-C48mx500_gdasstage_ic_202103241200 .........   Passed  215.64 sec
      Start 1953: test_gdasapp_WCDA-3DVAR-C48mx500_gdasfcst_202103241200
25/46 Test #1953: test_gdasapp_WCDA-3DVAR-C48mx500_gdasfcst_202103241200 .............   Passed  825.09 sec
      Start 1954: test_gdasapp_WCDA-3DVAR-C48mx500_gdasprepoceanobs_202103241800
26/46 Test #1954: test_gdasapp_WCDA-3DVAR-C48mx500_gdasprepoceanobs_202103241800 .....   Passed  282.83 sec
      Start 1955: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarinebmat_202103241800
27/46 Test #1955: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarinebmat_202103241800 .......   Passed  224.14 sec
      Start 1956: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlinit_202103241800
28/46 Test #1956: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlinit_202103241800 ....   Passed  131.10 sec
      Start 1957: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlvar_202103241800
29/46 Test #1957: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlvar_202103241800 .....   Passed  115.69 sec
      Start 1958: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlchkpt_202103241800
30/46 Test #1958: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlchkpt_202103241800 ...   Passed   55.83 sec
      Start 1959: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlfinal_202103241800
31/46 Test #1959: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlfinal_202103241800 ...   Passed  177.48 sec
      Start 1960: test_gdasapp_convert_bufr_adpsfc
32/46 Test #1960: test_gdasapp_convert_bufr_adpsfc ...................................   Passed   15.39 sec
      Start 1961: test_gdasapp_convert_gsi_satbias
33/46 Test #1961: test_gdasapp_convert_gsi_satbias ...................................   Passed    9.21 sec
      Start 1962: test_gdasapp_setup_atm_cycled_exp
34/46 Test #1962: test_gdasapp_setup_atm_cycled_exp ..................................   Passed    8.23 sec
      Start 1963: test_gdasapp_atm_jjob_var_init
35/46 Test #1963: test_gdasapp_atm_jjob_var_init .....................................   Passed   78.41 sec
      Start 1964: test_gdasapp_atm_jjob_var_run
36/46 Test #1964: test_gdasapp_atm_jjob_var_run ......................................   Passed  138.28 sec
      Start 1965: test_gdasapp_atm_jjob_var_inc
37/46 Test #1965: test_gdasapp_atm_jjob_var_inc ......................................   Passed   74.37 sec
      Start 1966: test_gdasapp_atm_jjob_var_final
38/46 Test #1966: test_gdasapp_atm_jjob_var_final ....................................   Passed   42.28 sec
      Start 1967: test_gdasapp_atm_jjob_ens_init
39/46 Test #1967: test_gdasapp_atm_jjob_ens_init .....................................   Passed   45.73 sec
      Start 1968: test_gdasapp_atm_jjob_ens_letkf
40/46 Test #1968: test_gdasapp_atm_jjob_ens_letkf ....................................   Passed  554.41 sec
      Start 1969: test_gdasapp_atm_jjob_ens_init_split
41/46 Test #1969: test_gdasapp_atm_jjob_ens_init_split ...............................   Passed  110.21 sec
      Start 1970: test_gdasapp_atm_jjob_ens_obs
42/46 Test #1970: test_gdasapp_atm_jjob_ens_obs ......................................   Passed   42.33 sec
      Start 1971: test_gdasapp_atm_jjob_ens_sol
43/46 Test #1971: test_gdasapp_atm_jjob_ens_sol ......................................   Passed  138.33 sec
      Start 1972: test_gdasapp_atm_jjob_ens_inc
44/46 Test #1972: test_gdasapp_atm_jjob_ens_inc ......................................   Passed   42.35 sec
      Start 1973: test_gdasapp_atm_jjob_ens_final
45/46 Test #1973: test_gdasapp_atm_jjob_ens_final ....................................   Passed   42.33 sec
      Start 1974: test_gdasapp_aero_gen_3dvar_yaml
46/46 Test #1974: test_gdasapp_aero_gen_3dvar_yaml ...................................   Passed    0.89 sec

100% tests passed, 0 tests failed out of 46

Label Time Summary:
gdas-utils    =  23.27 sec*proc (13 tests)
manual        = 2055.42 sec*proc (9 tests)
script        =  23.27 sec*proc (13 tests)

Total Test time (real) = 3504.23 sec

Given this, commit updated jedi hashes to feature/update_hashes. Done at f9ea5e4.

RussTreadon-NOAA self-assigned this Sep 26, 2024

RussTreadon-NOAA changed the title ~~Update hash for select JEDI submodules~~ Update hashes for select JEDI submodules Sep 26, 2024

RussTreadon-NOAA added a commit that referenced this issue Oct 1, 2024

update select jedi hashes (#1298)

52b5354

RussTreadon-NOAA added a commit that referenced this issue Oct 1, 2024

update yaml for test_gdasapp_snow_letkfoi_snowda (#1298)

7d92fa4

RussTreadon-NOAA added a commit that referenced this issue Oct 1, 2024

update reference files for atm var and lgetkf ctests (#1298)

6fa1cde

RussTreadon-NOAA added a commit that referenced this issue Oct 1, 2024

replace EXPLICIT_DIFFUSION with diffusion in parm/soca/berror yamls (#…

1d25325

…1298)

guillaumevernieres mentioned this issue Oct 3, 2024

Update gdas.cd NOAA-EMC/global-workflow#2978

Merged

7 tasks

RussTreadon-NOAA mentioned this issue Oct 4, 2024

Update select JEDI hashes #1310

Merged

RussTreadon-NOAA added a commit that referenced this issue Oct 4, 2024

update to more recent hashes for jedi repos (#1298)

f9ea5e4

RussTreadon-NOAA closed this as completed in #1310 Oct 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update hashes for select JEDI submodules #1298

Update hashes for select JEDI submodules #1298

RussTreadon-NOAA commented Sep 26, 2024

RussTreadon-NOAA commented Sep 26, 2024

danholdaway commented Sep 26, 2024

RussTreadon-NOAA commented Sep 26, 2024

danholdaway commented Sep 26, 2024

RussTreadon-NOAA commented Sep 26, 2024

RussTreadon-NOAA commented Sep 26, 2024

RussTreadon-NOAA commented Sep 27, 2024

RussTreadon-NOAA commented Sep 27, 2024

guillaumevernieres commented Sep 28, 2024

RussTreadon-NOAA commented Sep 28, 2024

guillaumevernieres commented Sep 28, 2024

RussTreadon-NOAA commented Sep 28, 2024

guillaumevernieres commented Sep 28, 2024

RussTreadon-NOAA commented Sep 30, 2024

RussTreadon-NOAA commented Sep 30, 2024

RussTreadon-NOAA commented Oct 1, 2024

RussTreadon-NOAA commented Oct 1, 2024

RussTreadon-NOAA commented Oct 1, 2024

guillaumevernieres commented Oct 1, 2024

RussTreadon-NOAA commented Oct 1, 2024

guillaumevernieres commented Oct 2, 2024

RussTreadon-NOAA commented Oct 2, 2024

guillaumevernieres commented Oct 2, 2024

RussTreadon-NOAA commented Oct 3, 2024

RussTreadon-NOAA commented Oct 3, 2024

danholdaway commented Oct 3, 2024

RussTreadon-NOAA commented Oct 4, 2024

Update hashes for select JEDI submodules #1298

Update hashes for select JEDI submodules #1298

Comments

RussTreadon-NOAA commented Sep 26, 2024

RussTreadon-NOAA commented Sep 26, 2024

danholdaway commented Sep 26, 2024

RussTreadon-NOAA commented Sep 26, 2024

danholdaway commented Sep 26, 2024

RussTreadon-NOAA commented Sep 26, 2024

RussTreadon-NOAA commented Sep 26, 2024

RussTreadon-NOAA commented Sep 27, 2024

RussTreadon-NOAA commented Sep 27, 2024

guillaumevernieres commented Sep 28, 2024

RussTreadon-NOAA commented Sep 28, 2024

guillaumevernieres commented Sep 28, 2024

RussTreadon-NOAA commented Sep 28, 2024

guillaumevernieres commented Sep 28, 2024

RussTreadon-NOAA commented Sep 30, 2024

RussTreadon-NOAA commented Sep 30, 2024

RussTreadon-NOAA commented Oct 1, 2024

RussTreadon-NOAA commented Oct 1, 2024

RussTreadon-NOAA commented Oct 1, 2024

guillaumevernieres commented Oct 1, 2024

RussTreadon-NOAA commented Oct 1, 2024

guillaumevernieres commented Oct 2, 2024

RussTreadon-NOAA commented Oct 2, 2024

guillaumevernieres commented Oct 2, 2024

RussTreadon-NOAA commented Oct 3, 2024

RussTreadon-NOAA commented Oct 3, 2024

danholdaway commented Oct 3, 2024

RussTreadon-NOAA commented Oct 4, 2024