Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MYNN in parent, LES in a nest: sometimes the sim does not run. #1514

Closed
davegill opened this issue May 10, 2021 · 4 comments
Closed

MYNN in parent, LES in a nest: sometimes the sim does not run. #1514

davegill opened this issue May 10, 2021 · 4 comments
Assignees
@davegill

Comments

@davegill
Copy link
Contributor

Describe the bug
A clear and concise description of what the bug is.
This is an old issue we are having. When we try to use MYNN in one domain and LES in a nest sometimes the sim does not run. And the output provides no clue about what the problem could be. You pointed us a few years ago to the solution, copy the MYNN package definition in the registry and change to a nopbl package if the pbl is set to 0 in the namelist (LES).

The problem of this fix is that it is particular for MYNN. So it never made it into the official WRF, and it is consuming the time of many users trying to run MYNN with LES.

My idea for a generic fix was to put the fix above for MYNN (the most used PBL) and then in check a mundo put an check to stop any sim with a PBL with a package + LES and print the solution. This will stop consuming user's time and points to the fix.

I briefly talked to Jimy about it and he indicated that you may know a generic fix.

How do you think we should move forward with this to avoid this issue?

Thanks in advance,
Pedro.

To Reproduce
Steps to reproduce the behavior:

  1. Use compiler and version '....'
  2. Use namelist options '....'
  3. Output is '....'

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Attachments
If applicable, attach supporting files: namelist.input, rsl.* files, etc.

Additional context
Add any other context about the problem here, such as:

  • This used to work with a previous version.
  • The documentation is different from the exhibited behavior.
@davegill davegill self-assigned this May 10, 2021
@davegill
Copy link
Contributor Author

@dudhia @pedro-jm
Pedro,
Do you have a case on cheyenne that reproducibly fails with this MYNN + LES setup? Can you point us to it?

@mefrediani
Copy link
Contributor

Dave, there's a directory with the run files here: /glade/scratch/frediani/WRF-Fire/runs/cameron_peak_2020/no-fire/WRF/2dom
(Please note that the wrfinput and wrfbdy files are symlinks to subfolders)
Let me know if you have any questions.

Thanks
Maria

@dudhia
Copy link
Collaborator

dudhia commented Jul 30, 2021

Just to update on what this fix is.
When we use a PBL scheme on any domain, any advected scalars in its package list need to be applied to all domains. This is so that LES nests don't allocate less memory. What is not clear yet is if all state variables in the package list also have to be allocated. For Franciano, the quick fix that seemed to work was to add the qke_adv scalar to the microphysics package.

@dudhia
Copy link
Collaborator

dudhia commented Oct 28, 2021

Note that other PBL schemes and sub-options also have packaged arrays in a state: list. A workaround for those is to remove the state list from the relevant package statement.

@dudhia dudhia mentioned this issue Oct 29, 2021
Open
@dudhia dudhia mentioned this issue Nov 10, 2021
Closed
davegill added a commit to davegill/WRF that referenced this issue Nov 11, 2021
@davegill
Generalized method to handle different PBL options by domain
97132b5 
TYPE: bug fix

KEYWORDS: PBL, LES, proxy

SOURCE: internal

DESCRIPTION OF CHANGES:
Problem:
Only the YSU PBL scheme has no packaged variables. If a user wants to have any other PBL scheme on the CG,
and an LES PBL on the FG, the model will have memory corruption problems. This error is hard to detect.
Inside the model, this causes an inconsistent number of variables on the CG and FG, which then causes
segmentation faults when trying to do feedback or advection of unavailable fields.

Solution:
Introduce a derived namelist entry (MAX_DOM sized). This proxy namelist value is used for space allocation
due to packaging. The existing PBL physics definitions, and the associated packaged variables, have been
replicated. The proxy entries are used to define the space, and the original entries are used for the
automatically generated names.

1. There is no difference for the user facing namelist interface.
2. The developers see no differences inside of the WRF dynamics, or the physics schemes or drivers.
3. The only developer impact would be when adding additional packaged variables to an existing
PBL scheme, or when introducing a new scheme.

ISSUE:
Fixes wrf-model#1514 "MYNN in parent, LES in a nest: sometimes the sim does not run"

LIST OF MODIFIED FILES:
modified:   Registry/Registry.EM_COMMON
modified:   share/module_check_a_mundo.F

TESTS CONDUCTED:
1. A few test scenarios of the combinations of PBL options were tested. These results were printed out
from the real program (same part of check_a_mundo that the WRF model sees). In each of the examples,
the check_a_mundo logic translates the namelist entries into the required packages.

   . bl_pbl_physics = 5, 0,
The memory allocation on domain 2 is set to the same packaged variables as on domain 1.
```
           PBL      PBL
          PROXY   PHYSICS
          (mem)   (scheme)
         ------------------
           5           5
           5           0
         ------------------
```

   . bl_pbl_physics = 1, 0,
The memory allocation on domain 2 is set to the same packaged variables as on domain 1.
```
           PBL      PBL
          PROXY   PHYSICS
          (mem)   (scheme)
         ------------------
           1           1
           1           0
         ------------------
```

   . bl_pbl_physics = 1, 1,
The memory allocation remains the default, package definitions for domains 1 and 2 remain unchanged.
```
           PBL      PBL
          PROXY   PHYSICS
          (mem)   (scheme)
         ------------------
           1           1
           1           1
         ------------------
```

   . bl_pbl_physics = 0, 0,
The memory allocation remains the default, package definitions for domains 1 and 2 remain unchanged.
```
           PBL      PBL
          PROXY   PHYSICS
          (mem)   (scheme)
         ------------------
           0           0
           0           0
         ------------------
```
2. Hopefully, Jenkins tests are all PASS.

RELEASE NOTE: A generalized method to handle domain-wise inconsistent PBL options has been introduced. Previously, this problem caused an inconsistent number of variables on the CG and FG, which caused segmentation faults when trying to do feedback or advection of unavailable fields. For users that tried the LES PBL option and a non-YSU PBL scheme (AND it worked!), it is possible that the results will be somewhat different due to the CG FG interactions during feedback and the generation of lateral boundary conditions for the nest. Scheme developers are pointed to the new packaging for the PBL schemes (look for the string `_proxy`).
@davegill davegill mentioned this issue Nov 11, 2021
Closed
davegill added a commit to davegill/WRF that referenced this issue Dec 10, 2021
@davegill
Yet another
e0445f5 
TYPE: bug fix

KEYWORDS: PBL, LES

SOURCE: internal

DESCRIPTION OF CHANGES:
Problem:
A few PBL schemes include scalar components that are advected. These fields are
required on all domains. With an LES option on a finer domain, that grid does not
have the additional scalar component, which causes a segfault when trying to
access that field.

Solution:
Restrict what PBL schemes are eligible for use with the LES (no PBL) option.

ISSUE:
Fixes wrf-model#1514

LIST OF MODIFIED FILES:
modified:   Registry/Registry.EM_COMMON
modified:   share/module_check_a_mundo.F

TESTS CONDUCTED:
1. When an incompatible CG PBL and FG LES are selected, the code successfully stops.
```
 &physics
 bl_pbl_physics                      =  5,     0,

--- ERROR: LES PBL on fine grid does not work with CG PBL option  5
           Choose a CG PBL option without any scalar components
           Fix bl_pbl_physics in namelist.input
           Alternatively, remove all of the packaged variables from the CG PBL selection
-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE:  <stdin>  LINE:    2523
NOTE:       1 namelist settings are wrong. Please check and reset these options
```
```
 &physics
 bl_pbl_physics                      =  6,     0,

--- ERROR: LES PBL on fine grid does not work with CG PBL option  6
           Choose a CG PBL option without any scalar components
           Fix bl_pbl_physics in namelist.input
           Alternatively, remove all of the packaged variables from the CG PBL selection
-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE:  <stdin>  LINE:    2523
NOTE:       1 namelist settings are wrong. Please check and reset these options
```
```
 &physics
 bl_pbl_physics                      = 16,     0,

--- ERROR: LES PBL on fine grid does not work with CG PBL option  16
           Choose a CG PBL option without any scalar components
           Fix bl_pbl_physics in namelist.input
           Alternatively, remove all of the packaged variables from the CG PBL selection
-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE:  <stdin>  LINE:    2523
NOTE:       1 namelist settings are wrong. Please check and reset these options
```
2. Hopefully we say All jenkins tests are a PASS.

RELEASE NOTE: A test was introduced to stop the incorrect combination of particular PBL schemes on coarser grids with the LES PBL option selected on a finer grid. Previously, this problem caused an inconsistent number of variables on the CG and FG, which caused segmentation faults when trying to do feedback or advection of unavailable fields.
@davegill davegill mentioned this issue Dec 10, 2021
Merged
vlakshmanan-scala pushed a commit to scala-computing/WRF that referenced this issue Apr 4, 2024
@davegill @vlakshmanan-scala
PBL + LES restrictions (wrf-model#1596)
bbf8540 
TYPE: bug fix

KEYWORDS: PBL, LES

SOURCE: internal

DESCRIPTION OF CHANGES:
Problem:
A few PBL schemes include scalar components that are advected. These fields are
required on all domains. With an LES option on a finer domain, that grid does not
have the additional scalar component, which causes a segfault when trying to
access that field.

Solution:
Restrict what PBL schemes are eligible for use with the LES (no PBL) option.

ISSUE:
Fixes wrf-model#1514

LIST OF MODIFIED FILES:
modified:   Registry/Registry.EM_COMMON
modified:   share/module_check_a_mundo.F

TESTS CONDUCTED:
1. When an incompatible CG PBL and FG LES are selected, the code successfully stops.
```
 &physics
 bl_pbl_physics                      =  5,     0,

--- ERROR: LES PBL on fine grid does not work with CG PBL option  5
           Choose a CG PBL option without any scalar components
           Fix bl_pbl_physics in namelist.input
           Alternatively, remove all of the packaged variables from the CG PBL selection
-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE:  <stdin>  LINE:    2523
NOTE:       1 namelist settings are wrong. Please check and reset these options
```
```
 &physics
 bl_pbl_physics                      =  6,     0,

--- ERROR: LES PBL on fine grid does not work with CG PBL option  6
           Choose a CG PBL option without any scalar components
           Fix bl_pbl_physics in namelist.input
           Alternatively, remove all of the packaged variables from the CG PBL selection
-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE:  <stdin>  LINE:    2523
NOTE:       1 namelist settings are wrong. Please check and reset these options
```
```
 &physics
 bl_pbl_physics                      = 16,     0,

--- ERROR: LES PBL on fine grid does not work with CG PBL option  16
           Choose a CG PBL option without any scalar components
           Fix bl_pbl_physics in namelist.input
           Alternatively, remove all of the packaged variables from the CG PBL selection
-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE:  <stdin>  LINE:    2523
NOTE:       1 namelist settings are wrong. Please check and reset these options
```
2. All jenkins tests are a PASS.

RELEASE NOTE: A test was introduced to stop the incorrect combination of particular PBL schemes on coarser grids with the LES PBL option selected on a finer grid. Previously, this problem caused an inconsistent number of variables on the CG and FG, which caused segmentation faults when trying to do feedback or advection of unavailable fields.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@davegill davegill

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@davegill @dudhia @mefrediani