Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v16 coupled tests can fail when output type is the tiled grid #566

Closed
DeniseWorthen opened this issue May 10, 2021 · 10 comments
Closed

v16 coupled tests can fail when output type is the tiled grid #566

DeniseWorthen opened this issue May 10, 2021 · 10 comments
Labels
bug Something isn't working

Comments

@DeniseWorthen
Copy link
Collaborator

Description

This issue was first noted on Gaea with the coupled V16 tests failing with a PIO error.

To Reproduce:

Run the cpld_bmarkfrac_v16 test on Gaea using tiled grid output.

Additional context

The initial issue was detailed in Issue #509. It was shown that:

  1. changing the number of write tasks allowed the test to run
  2. changing the output grid from tiled to gaussian allowed the test to run.

The issue appears to be related to the version and/or integration of PIO within ESMF.

@DeniseWorthen DeniseWorthen added the bug Something isn't working label May 10, 2021
@arunchawla-NOAA
Copy link

Is this still limited to Gaea or can occur in other systems as well ?

@DeniseWorthen
Copy link
Collaborator Author

We have only seen this issue on Gaea at this point.

@junwang-noaa
Copy link
Collaborator

@DeniseWorthen Do we still have issue on gaea?

@DeniseWorthen
Copy link
Collaborator Author

@junwang-noaa We turned off tiled output for this test on all platforms so it is not known whether the issue has been resolved. I suspect it is still an issue since the underlying problem was most likely due to the version of PIO we are using. I believe updating PIO is an active issue w/ the ESMF developers. Perhaps @rsdunlapiv could give an update.

@rsdunlapiv
Copy link

The 8.3 release of ESMF should have an updated PIO (to version 2.5.6). We should re-test at that point to see if the issue is resolved.

@arunchawla-NOAA
Copy link

moving to icebox till the 8.3 release of esmf is available for further testing

@junwang-noaa
Copy link
Collaborator

@DeniseWorthen The ESMF 830bs9 is now used in the UFS WM, can you check if the issue is resolved? Thanks

@rsdunlapiv
Copy link

@DeniseWorthen @junwang-noaa 830bs09 does not have the PIO update. I suggest waiting until the ESMF 8.3.0 release to test this. Alternatively, you can test now with 830bs14 if would like.

@DeniseWorthen
Copy link
Collaborator Author

I can confirm that 830bs09 does not resolve the tiled grid output issue for the bmark_p8 test. I still get the errror

304: fv3.exe            0000000001A80475  esmfpio_spmd_util         295  pio_spmd_utils.F90.in

@DeniseWorthen
Copy link
Collaborator Author

I've been able to run the cpld_bmark_p8 test using commit e051e0e from the develop branch on Gaea but switching to the cubed_sphere_grid output. I was able to run 3 consecutive times. I'm not sure what has changed, since we're still running 8.3.0b09 in UWM but I'm going to close the issue at this point.

pjpegion pushed a commit to NOAA-PSL/ufs-weather-model that referenced this issue Apr 4, 2023
…leanup (ufs-community#566)

* update submodule pointer to Man's ccpp-physics PR#944 branch: ccppv6 doc update
epic-cicd-jenkins pushed a commit that referenced this issue Apr 17, 2023
…ecast files (#566)

This PR enables running of only the SRW App's deterministic verification (vx) tasks on staged forecast files from previous runs of the App. It partially resolves Issue #565 (it resolves the issue for deterministic vx but not ensemble vx).

Specific changes:

* Update lua module file for vx tasks to suppress "Logging error" messages in vx task log files.
* Rename experiment variable MODEL to VX_FCST_MODEL_NAME to clarify that this is the name of the forecast model in the context of verification (and which will be used in the vx output files). This requires updates to most (all?) of the * METplus configuration files and the verification ex-scripts.
* Create the new variable VX_FCST_INPUT_BASEDIR to allow the user to specify a directory in which to look for staged forecast output (instead of running a forecast).
* Modify the rocoto template xml (FV3LAM_wflow.xml) to make dependencies of vx tasks on post-processing tasks appear only when the post tasks are enabled.
* Add a new WE2E test category subdirectory named verification in which to group all vx tests (since more vx tasks will be coming in future PRs). Move the two existing tests MET_verification and MET_ensemble_verification from wflow_features to verification, and add a new test named MET_verification_only_vxto test the capability that this PR introduces.
Note: The new WE2E test MET_verification_only_vx requires new data, specifically post-processed forecast output from the SRW App. This data needs to be staged on each platform; currently, it is located in a personal directory on Hera.
epic-cicd-jenkins pushed a commit that referenced this issue Apr 17, 2023
PR #566 changed the variable "MODEL" to a more descriptive name, but failed to make this change in config.community.yaml. The unit tests for generate_FV3LAM_wflow.py make use of this file as an input config.yaml, so they are now failing due to this incorrect variable name. This wasn't caught because prior to #558 the unit tests were broken for a different reason.

This change simply makes the appropriate rename, which should fix the failing unit test. Also created an f-string that was missed in a setup.py error message.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants