Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Generalized method to handle different PBL options by domain #1581

Closed
wants to merge 2 commits into from
Closed

Generalized method to handle different PBL options by domain #1581

wants to merge 2 commits into from

Conversation

davegill
Copy link
Contributor

@davegill davegill commented Nov 11, 2021 •
edited
Loading

TYPE: bug fix

KEYWORDS: PBL, LES, proxy

SOURCE: internal

DESCRIPTION OF CHANGES:
Problem:
Only the YSU PBL scheme has no packaged variables. If a user wants to have any other PBL scheme on the CG,
and an LES PBL on the FG, the model will have memory corruption problems. This error is hard to track down.
Inside the model, the having different PBL schemes on different domains (basically any non-YSU PBL with LES
PBL) causes an inconsistent number of variables on the CG and FG, which then causes segmentation faults when
trying to do feedback or advection of unavailable fields.

Solution:
Introduce a derived namelist entry (MAX_DOM sized). This proxy namelist value is used for space allocation
due to packaging. The existing PBL physics definitions, and the associated packaged variables, have been
replicated. The proxy entries are used to define the space, and the original entries are used for the
automatically generated names.

  1. There is no difference for the user facing namelist interface.
  2. The developers see no differences inside of the WRF dynamics, or the physics schemes or drivers.
  3. The only developer impact would be when adding additional packaged variables to an existing
    PBL scheme, or when introducing a new scheme.

ISSUE:
Fixes #1514 "MYNN in parent, LES in a nest: sometimes the sim does not run"

LIST OF MODIFIED FILES:
modified: Registry/Registry.EM_COMMON
modified: share/module_check_a_mundo.F

TESTS CONDUCTED:

  1. A few test scenarios of the combinations of PBL options were tested. These results were printed out
    from the real program (same part of check_a_mundo that the WRF model sees). In each of the examples,
    the check_a_mundo logic translates the namelist entries into the required packages. To demonstrate the
    availability of some variables, the Registry was modified to include h for some fields so that they would
    be eligible to be checked in the model output.
  • bl_pbl_physics = 5, 0,
    The memory allocation on domain 2 is set to the same packaged variables as on domain 1.
           PBL      PBL
          PROXY   PHYSICS
          (mem)   (scheme)
         ------------------
           5           5
           5           0
         ------------------
  • bl_pbl_physics = 1, 0,
    The memory allocation on domain 2 is set to the same packaged variables as on domain 1.
           PBL      PBL
          PROXY   PHYSICS
          (mem)   (scheme)
         ------------------
           1           1
           1           0
         ------------------
  • bl_pbl_physics = 1, 1,
    The memory allocation remains the default, package definitions for domains 1 and 2 remain unchanged.
           PBL      PBL
          PROXY   PHYSICS
          (mem)   (scheme)
         ------------------
           1           1
           1           1
         ------------------
  • bl_pbl_physics = 0, 0,
    The memory allocation remains the default, package definitions for domains 1 and 2 remain unchanged.
           PBL      PBL
          PROXY   PHYSICS
          (mem)   (scheme)
         ------------------
           0           0
           0           0
         ------------------
  1. More confirmation that the correct variables are being allocated is by looking at the output from the
    netcdf files.

With the standard code, with bl_pbl_physics = 5, 0,, there is qke_adv on d01, but not on d02:

$ ncdump -h wrfinput_d01 | grep -i qke_adv
	float qke_adv(Time, bottom_top, south_north, west_east) ;
		qke_adv:FieldType = 104 ;
		qke_adv:MemoryOrder = "XYZ" ;
		qke_adv:description = "twice TKE from MYNN" ;
		qke_adv:units = "m2 s-2" ;
		qke_adv:stagger = "" ;
		qke_adv:coordinates = "XLONG XLAT XTIME" ;

$ ncdump -h wrfinput_d02 | grep -i qke_adv

With the modified source, with bl_pbl_physics = 5, 0,, the qke_adv field is on both domains:

$ ncdump -h wrfinput_d01 | grep -i qke_adv
	float qke_adv(Time, bottom_top, south_north, west_east) ;
		qke_adv:FieldType = 104 ;
		qke_adv:MemoryOrder = "XYZ" ;
		qke_adv:description = "twice TKE from MYNN" ;
		qke_adv:units = "m2 s-2" ;
		qke_adv:stagger = "" ;
		qke_adv:coordinates = "XLONG XLAT XTIME" ;

$ ncdump -h wrfinput_d02 | grep -i qke_adv
	float qke_adv(Time, bottom_top, south_north, west_east) ;
		qke_adv:FieldType = 104 ;
		qke_adv:MemoryOrder = "XYZ" ;
		qke_adv:description = "twice TKE from MYNN" ;
		qke_adv:units = "m2 s-2" ;
		qke_adv:stagger = "" ;
		qke_adv:coordinates = "XLONG XLAT XTIME" ;

An h had to be added to the IO strings, but the qke_adv field with settings bl_pbl_physics = 5, 0, is in
the standard output file for both of the domains:

$ ncdump -h wrfout_d01_2000-01-24_12:00:00 | grep -i qke_adv
	float qke_adv(Time, bottom_top, south_north, west_east) ;
		qke_adv:FieldType = 104 ;
		qke_adv:MemoryOrder = "XYZ" ;
		qke_adv:description = "twice TKE from MYNN" ;
		qke_adv:units = "m2 s-2" ;
		qke_adv:stagger = "" ;
		qke_adv:coordinates = "XLONG XLAT XTIME" ;

$ ncdump -h wrfout_d02_2000-01-24_12:00:00 | grep -i qke_adv
	float qke_adv(Time, bottom_top, south_north, west_east) ;
		qke_adv:FieldType = 104 ;
		qke_adv:MemoryOrder = "XYZ" ;
		qke_adv:description = "twice TKE from MYNN" ;
		qke_adv:units = "m2 s-2" ;
		qke_adv:stagger = "" ;
		qke_adv:coordinates = "XLONG XLAT XTIME" ;
  1. Jenkins tests are all PASS.

RELEASE NOTE: A generalized method to handle domain-wise inconsistent PBL options has been introduced. Previously, this problem caused an inconsistent number of variables on the CG and FG, which caused segmentation faults when trying to do feedback or advection of unavailable fields. For users that tried the LES PBL option and a non-YSU PBL scheme (AND it actually worked!), it is possible that the results will be somewhat different due to the CG<->FG interactions during feedback and the generation of lateral boundary conditions for the nest. Scheme developers are pointed to the new packaging for the PBL schemes (look for the string _proxy in Registry.EM_COMMON in the package area).

@davegill
Generalized method to handle different PBL options by domain
97132b5 
TYPE: bug fix

KEYWORDS: PBL, LES, proxy

SOURCE: internal

DESCRIPTION OF CHANGES:
Problem:
Only the YSU PBL scheme has no packaged variables. If a user wants to have any other PBL scheme on the CG,
and an LES PBL on the FG, the model will have memory corruption problems. This error is hard to detect.
Inside the model, this causes an inconsistent number of variables on the CG and FG, which then causes
segmentation faults when trying to do feedback or advection of unavailable fields.

Solution:
Introduce a derived namelist entry (MAX_DOM sized). This proxy namelist value is used for space allocation
due to packaging. The existing PBL physics definitions, and the associated packaged variables, have been
replicated. The proxy entries are used to define the space, and the original entries are used for the
automatically generated names.

1. There is no difference for the user facing namelist interface.
2. The developers see no differences inside of the WRF dynamics, or the physics schemes or drivers.
3. The only developer impact would be when adding additional packaged variables to an existing
PBL scheme, or when introducing a new scheme.

ISSUE:
Fixes wrf-model#1514 "MYNN in parent, LES in a nest: sometimes the sim does not run"

LIST OF MODIFIED FILES:
modified:   Registry/Registry.EM_COMMON
modified:   share/module_check_a_mundo.F

TESTS CONDUCTED:
1. A few test scenarios of the combinations of PBL options were tested. These results were printed out
from the real program (same part of check_a_mundo that the WRF model sees). In each of the examples,
the check_a_mundo logic translates the namelist entries into the required packages.

   . bl_pbl_physics = 5, 0,
The memory allocation on domain 2 is set to the same packaged variables as on domain 1.
```
           PBL      PBL
          PROXY   PHYSICS
          (mem)   (scheme)
         ------------------
           5           5
           5           0
         ------------------
```

   . bl_pbl_physics = 1, 0,
The memory allocation on domain 2 is set to the same packaged variables as on domain 1.
```
           PBL      PBL
          PROXY   PHYSICS
          (mem)   (scheme)
         ------------------
           1           1
           1           0
         ------------------
```

   . bl_pbl_physics = 1, 1,
The memory allocation remains the default, package definitions for domains 1 and 2 remain unchanged.
```
           PBL      PBL
          PROXY   PHYSICS
          (mem)   (scheme)
         ------------------
           1           1
           1           1
         ------------------
```

   . bl_pbl_physics = 0, 0,
The memory allocation remains the default, package definitions for domains 1 and 2 remain unchanged.
```
           PBL      PBL
          PROXY   PHYSICS
          (mem)   (scheme)
         ------------------
           0           0
           0           0
         ------------------
```
2. Hopefully, Jenkins tests are all PASS.

RELEASE NOTE: A generalized method to handle domain-wise inconsistent PBL options has been introduced. Previously, this problem caused an inconsistent number of variables on the CG and FG, which caused segmentation faults when trying to do feedback or advection of unavailable fields. For users that tried the LES PBL option and a non-YSU PBL scheme (AND it worked!), it is possible that the results will be somewhat different due to the CG FG interactions during feedback and the generation of lateral boundary conditions for the nest. Scheme developers are pointed to the new packaging for the PBL schemes (look for the string `_proxy`).
@davegill
Zap the debug "stop" statement
07ab3cd 
modified:   share/module_check_a_mundo.F
@davegill davegill requested review from a team as code owners November 11, 2021 00:28
@davegill
Copy link
Contributor Author

@weiwangncar @dudhia @pedro-jm
Folks,
I need to do further testing, but this approach is probably as general as it gets.

  1. The nesting impact is likely a big deal. For example looking at qke_adv, there is an assumption of initial interpolation to the FG domain (not important), feedback at each CG time step (overwriting with junk or zeros), generating the lateral boundary forcing (not important).
  2. The packaged variables on the d01 PBL scheme are the packaged variables in use for every domain that has LES PBL. There won't be segfaults, but the specifically packaged PBL fields in the Registry file may need to have their nesting options reviewed.

We either do this PR (more general) or the more bandaid-oriented #1578 "Restrict which CG PBLs may use FG LES PBL"

@davegill
Copy link
Contributor Author

@smileMchen @weiwangncar
Ming,
Do you have a high-resolution test case, where it would be appropriate to select bl_pbl_physics =0 on the fine domain? I have looked at a number of infrastructure aspects of this PR, but we need to see if it makes physical sense. We have been suggesting that people use bl_pbl_physics = 0 on an inner-most domain, we should see what happens when the CG domain has a non-YSU PBL scheme (such as MYNN2).

@davegill
Copy link
Contributor Author

@weiwangncar @dudhia
Folks,
If we go this way with the PBL schemes, we should do the EXACT same with the CU schemes. Jimy pointed out that they are identical - the fine grids are permitted to have a "no scheme called" option.

@dudhia
Copy link
Collaborator

dudhia commented Nov 11, 2021

I am not a fan of this Registry change.
Alternative: Force all domains to use coarse-mesh bl_pbl_physics despite namelist settings. Activate LES in some domains with an overriding switch if pbl_option was zero in the namelist. This requires setting a switch based on namelist pbl options before overriding them. The PBL driver would skip the PBL based on this switch.

@davegill
Copy link
Contributor Author

@dudhia
Jimy,
So, let me see if I understand ...

  1. Have the user still choose something like this
bl_pbl_physics = 5, 5, 0, 0,
  1. In check_a_mundo, we set a LES PBL derived namelist option based on the PBL settings, something like this
LES_PBL = F, F, T, T,
  1. In check_a_mundo, we then set ALL of the domains' PBL options to the same as d01.
  2. The there needs to be a change somewhere in the first_rk_step to not do a PBL call if the LES_PBL option is true

Does this seem about right?

@dudhia
Copy link
Collaborator

dudhia commented Nov 11, 2021 via email

@davegill
Copy link
Contributor Author

@dudhia @weiwangncar
Jimy,
After a closer look, there are nearly 200 separate lines in 47 separate files that require logic changes if we change the meaning of bl_pbl_physics. That seems like a huge opportunity for bugs to creep in.

@dudhia
Copy link
Collaborator

dudhia commented Nov 17, 2021 via email

@davegill
Copy link
Contributor Author

@dudhia
Once we modify the namelist settings for bl_pbl_physics, so that columns that are not running any parameterized PBL scheme internally have a non-zero value for bl_pbl_physics, then we have changed the meaning of that variable.

Every place that there is a test for a specific scheme being used could possibly need to also test to see if the LES PBL flag is set, because we have changed the meaning of bl_pbl_physics.

Here is the list of impacted locations in the model:

      IF ( config_flags%bl_pbl_physics .GT. 0 ) CALL nl_set_bl_pbl_physics (head_grid%id, 98)
   IF ( config_flags%bl_pbl_physics .GT. 0 ) CALL nl_set_bl_pbl_physics (head_grid%id, 98)
   IF ( config_flags%bl_pbl_physics .GT. 0 ) CALL nl_set_bl_pbl_physics (head_grid%id, 98)
   IF ( model_config_rec%bl_pbl_physics(head_grid%id) .EQ. SURFDRAGSCHEME ) THEN
   IF ( config_flags%bl_pbl_physics .GT. 0 ) CALL nl_set_bl_pbl_physics (head_grid%id, 98)
   IF ( config_flags%bl_pbl_physics .GT. 0 ) CALL nl_set_bl_pbl_physics (head_grid%id, 98)
   IF ( model_config_rec%bl_pbl_physics(head_grid%id) .EQ. SURFDRAGSCHEME ) THEN
           model_config_rec % bl_pbl_physics_dfi(i) = -1
           model_config_rec % bl_pbl_physics_dfi(i) = model_config_rec % bl_pbl_physics(i)
./main/module_wrf_top.F
      if ( config_flags%bl_pbl_physics == BOULACSCHEME ) then
./dyn_nmm/module_initialize_tropical_cyclone.F
          IF ( CONFIG_FLAGS%BL_PBL_PHYSICS  == MYJPBLSCHEME) then
./dyn_nmm/module_DIFFUSION_NMM.F
      if(CONFIG_FLAGS%BL_PBL_PHYSICS == GFSSCHEME .OR. &
         CONFIG_FLAGS%BL_PBL_PHYSICS == GFSEDMFSCHEME) THEN
     &   CONFIG_FLAGS%BL_PBL_PHYSICS   ==MYJPBLSCHEME.AND.              &
./dyn_nmm/solve_nmm.F
      if ( config_flags%bl_pbl_physics == BOULACSCHEME ) then
     &          ,BL_PBL_PHYSICS=config_flags%bl_pbl_physics             &
     &               ,BL_PBL_PHYSICS=config_flags%bl_pbl_physics      &
      IF(CONFIG_FLAGS%BL_PBL_PHYSICS/=MYJPBLSCHEME)THEN
./dyn_nmm/module_PHYSICS_CALLS.F
      if ( config_flags%bl_pbl_physics == BOULACSCHEME ) then
./dyn_nmm/module_initialize_real.F
   if(       config_flags%bl_pbl_physics == CAMUWPBLSCHEME     .OR. config_flags%cu_physics == CAMZMSCHEME      &
          if(config_flags%bl_pbl_physics .ne. 1)CALL wrf_error_fatal &
   pbl_select: SELECT CASE(config_flags%bl_pbl_physics)
           SELECT CASE(config_flags%bl_pbl_physics)
           config_flags%bl_pbl_physics, param_first_scalar, restart, &
./phys/module_physics_init.F
       bl_pbl_physics, param_first_scalar, restart,              &
                                        bl_pbl_physics
  select case(bl_pbl_physics)
./phys/module_shcu_camuwshcu.F
                     ,cu_physics, bl_pbl_physics, sf_sfclay_physics   &
                   cugd_avedx,clos_choice,bl_pbl_physics,sf_sfclay_physics
               ,ITIMESTEP=itimestep, BL_PBL_PHYSICS=bl_pbl_physics  &
./phys/module_cumulus_driver.F
     &          ,bl_pbl_physics                                       &
                          ra_sw_physics, bl_pbl_physics
  myjpbl = ((bl_pbl_physics .EQ. MYJPBLSCHEME) .OR. &
            (bl_pbl_physics .EQ. QNSEPBLSCHEME) )
./phys/module_surface_driver.F
                 ,bl_pbl_physics, ra_lw_physics, dx, dy            &
   INTEGER,    INTENT(IN   )    ::     bl_pbl_physics, ra_lw_physics,sf_sfclay_physics,sf_urban_physics,windfarm_opt
  if (bl_pbl_physics .eq. 0) return
      IF (bl_pbl_physics .EQ. QNSEPBLSCHEME ) THEN
   pbl_select: SELECT CASE(bl_pbl_physics)
       WRITE( message , * ) 'The pbl option does not exist: bl_pbl_physics = ', bl_pbl_physics
./phys/module_pbl_driver.F
             ,itimestep, bl_pbl_physics, sf_sfclay_physics    &
                     bl_pbl_physics, & !pbl scheme
        call get_tpert(bl_pbl_physics, sf_sfclay_physics, dx, &
SUBROUTINE get_tpert(bl_pbl_physics, sf_sfclay_physics, dx, &
  integer, intent(in) :: bl_pbl_physics, kpbl, kte, sf_sfclay_physics
  if( bl_pbl_physics==CAMUWPBLSCHEME ) then
  elseif( bl_pbl_physics==MYJPBLSCHEME ) then
     if( bl_pbl_physics /= MYJPBLSCHEME ) &
./phys/module_cu_camzm_driver.F
   if (config_flags%bl_pbl_physics .gt. 0)                     &
   SELECT CASE(config_flags%bl_pbl_physics)
./phys/module_physics_addtendc.F
                     ,bl_pbl_physics                                  &
   INTEGER,   INTENT(IN   ) ::           bl_pbl_physics
     IF ( bl_pbl_physics == 2 ) THEN   ! MYJ
      ELSE IF ( bl_pbl_physics == 5 ) THEN   ! MYNN
                            bl_pbl_physics
./phys/module_shallowcu_driver.F
      IF ((config_flags%diff_opt .eq. 2) .and. (config_flags%bl_pbl_physics .eq. 0)) THEN
   IF(config_flags%bl_pbl_physics .GT. 0) THEN
./dyn_em/module_diffusion_em.F
       IF ( config_flags%bl_pbl_physics .ge. 1 ) THEN
         IF (config_flags%bl_pbl_physics .eq. 0) THEN
./dyn_em/module_first_rk_step_part2.F
       IF ( config_flags%bl_pbl_physics == MYNNPBLSCHEME2 .OR. &
            config_flags%bl_pbl_physics == MYNNPBLSCHEME3 ) THEN
      &        ,PBL=grid%bl_pbl_physics,EFCG=grid%EFCG,EFIG=grid%EFIG,EFSG=grid%EFSG &
./dyn_em/solve_em.F
   IF (config_flags%bl_pbl_physics .gt. 0) THEN
./dyn_em/module_big_step_utilities_em.F
     &        ,BL_PBL_PHYSICS=config_flags%bl_pbl_physics                              &
      IF ( config_flags%bl_pbl_physics .EQ. 98 ) THEN
     &        ,BL_PBL_PHYSICS=config_flags%bl_pbl_physics                           &
     IF ( num_chem >= PARAM_FIRST_SCALAR .AND. (config_flags%bl_pbl_physics == &
     & mynnpblscheme2 .OR. config_flags%bl_pbl_physics == mynnpblscheme3)  ) then
     &             ,BL_PBL_PHYSICS=config_flags%bl_pbl_physics            &
     &             ,BL_PBL_PHYSICS=config_flags%bl_pbl_physics            &
./dyn_em/module_first_rk_step_part1.F
        pbl_test : IF (config_flags%bl_pbl_physics .eq. 0) THEN
         pbl_test : IF (config_flags%bl_pbl_physics .eq. 0) THEN
   IF (config_flags%bl_pbl_physics .gt. 0) THEN
./dyn_em/module_em.F
&  fhh, gz1oz0, wspd, br, chklowq, bl_pbl_physics, ra_lw_physics, dx, &
  INTEGER, INTENT(IN) :: bl_pbl_physics, ra_lw_physics, &
  IF (bl_pbl_physics .NE. 0) THEN
        SELECT CASE  (bl_pbl_physics) 
./wrftladj/module_pbl_driver_ad.F
    SELECT CASE  (config_flags%bl_pbl_physics) 
./wrftladj/module_physics_init_ad.F
   IF(config_flags%bl_pbl_physics .GT. 0) THEN
         IF ((config_flags%diff_opt .eq. 2) .and. (config_flags%bl_pbl_physics .eq. 0)) THEN
./wrftladj/module_diffusion_em_ad.F
       IF ( config_flags%bl_pbl_physics == MYNNPBLSCHEME2 .OR. &
            config_flags%bl_pbl_physics == MYNNPBLSCHEME3 ) THEN
./wrftladj/solve_em_tl.F
  IF (config_flags%bl_pbl_physics .GT. 0) THEN
./wrftladj/module_big_step_utilities_em_ad.F
     &             ,BL_PBL_PHYSICS=config_flags%bl_pbl_physics            &
     &        ,BL_PBL_PHYSICS=config_flags%bl_pbl_physics                           &
./wrftladj/module_first_rk_step_part1_ad.F
&  cu_physics, bl_pbl_physics, sf_sfclay_physics, qv_curr, qv_currb, &
&  bl_pbl_physics, sf_sfclay_physics
./wrftladj/module_cumulus_driver_ad.F
 IF(config_flags%bl_pbl_physics .eq. 0) THEN
 IF(config_flags%bl_pbl_physics .eq. 0) THEN
  IF (config_flags%bl_pbl_physics .GT. 0) THEN
./wrftladj/module_em_tl.F
       IF ( config_flags%bl_pbl_physics .ge. 1 ) THEN
         IF (config_flags%bl_pbl_physics .eq. 0) THEN
./wrftladj/module_first_rk_step_part2_tl.F
&  fhh, gz1oz0, wspd, br, chklowq, bl_pbl_physics, ra_lw_physics, dx, &
  INTEGER, INTENT(IN) :: bl_pbl_physics, ra_lw_physics, &
  IF (bl_pbl_physics .EQ. 0) THEN
        SELECT CASE  (bl_pbl_physics) 
&          'The pbl option does not exist: bl_pbl_physics = ', &
&          bl_pbl_physics
./wrftladj/module_pbl_driver_tl.F
       IF ( config_flags%bl_pbl_physics == MYNNPBLSCHEME2 .OR. &
            config_flags%bl_pbl_physics == MYNNPBLSCHEME3 ) THEN
!      &        ,PBL=grid%bl_pbl_physics,EFCG=grid%EFCG,EFIG=grid%EFIG,EFSG=grid%EFSG &
./wrftladj/solve_em_ad.F
 IF(config_flags%bl_pbl_physics .GT. 0) THEN
   IF ((config_flags%diff_opt .eq. 2) .and. (config_flags%bl_pbl_physics .eq. 0)) THEN
./wrftladj/module_diffusion_em_tl.F
   if (config_flags%bl_pbl_physics .gt. 0)                     &
   SELECT CASE(config_flags%bl_pbl_physics)
./wrftladj/module_physics_addtendc_ad.F
&  cu_physics, bl_pbl_physics, sf_sfclay_physics, qv_curr, qv_currd, &
&  bl_pbl_physics, sf_sfclay_physics
./wrftladj/module_cumulus_driver_tl.F
!        pbl_test : IF (config_flags%bl_pbl_physics .eq. 0) THEN
!   IF(config_flags%bl_pbl_physics .eq. 0) THEN
   IF(config_flags%bl_pbl_physics .eq. 0) THEN
!   IF(config_flags%bl_pbl_physics .eq. 0) THEN
!       pbl_test : IF (config_flags%bl_pbl_physics .eq. 0) THEN
!   IF(config_flags%bl_pbl_physics .eq. 0) THEN
   IF(config_flags%bl_pbl_physics .eq. 0) THEN
  IF (config_flags%bl_pbl_physics .GT. 0) THEN
./wrftladj/module_em_ad.F
      IF ( config_flags%bl_pbl_physics .EQ. 98 ) THEN
     &        ,BL_PBL_PHYSICS=config_flags%bl_pbl_physics                           &
     &             ,BL_PBL_PHYSICS=config_flags%bl_pbl_physics            &
./wrftladj/module_first_rk_step_part1_tl.F
         IF (config_flags%bl_pbl_physics .eq. 0) THEN
       IF ( config_flags%bl_pbl_physics .ge. 1 ) THEN
./wrftladj/module_first_rk_step_part2_ad.F
  IF (config_flags%bl_pbl_physics .GT. 0) THEN
./wrftladj/module_big_step_utilities_em_tl.F
   if (config_flags%bl_pbl_physics .gt. 0)                     &
   SELECT CASE(config_flags%bl_pbl_physics)
./wrftladj/module_physics_addtendc_tl.F
      SELECT CASE  (config_flags%bl_pbl_physics) 
./wrftladj/module_physics_init_tl.F
    IF ( config_flags%bl_pbl_physics == ACMPBLSCHEME ) THEN
./chem/chemics_init.F
            sf_surface_physics, bl_pbl_physics, cu_physics
./share/wrf_bdyout.F
      CALL nl_set_bl_pbl_physics( grid%id, 0 )
      CALL nl_set_bl_pbl_physics( grid%id, grid%bl_pbl_physics)
      CALL nl_set_bl_pbl_physics( grid%id, 0 )
./share/dfi.F
            sf_surface_physics, bl_pbl_physics, cu_physics
./share/wrf_restartout.F
            sf_surface_physics, bl_pbl_physics, cu_physics, hypsometric_opt, sf_lake_physics, &
    call nl_get_bl_pbl_physics     ( grid%id,  bl_pbl_physics     )
    ibuf(1) = bl_pbl_physics
    CALL wrf_put_dom_ti_integer ( fid , 'BL_PBL_PHYSICS' ,  ibuf , 1 , ierr )
./share/output_wrf.F
! Check that SMS-3DTKE scheme (km_opt=5) Must work with bl_pbl_physics=0
              model_config_rec % bl_pbl_physics(i) .NE. 0  ) THEN
            wrf_err_message = '--- ERROR: SMS-3DTKE scheme can only work with bl_pbl_physics=0 '
            wrf_err_message = '--- ERROR: Fix km_opt or bl_pbl_physics in namelist.input.'
              (model_config_rec % bl_pbl_physics(i) /= myjpblscheme .AND. &
               model_config_rec % bl_pbl_physics(i) /= mynnpblscheme2 ) ) THEN
            wrf_err_message = '--- ERROR: Fix shcu_physics or bl_pbl_physics in namelist.input.'
! Check that all values of bl_pbl_physics are the same for all domains
         IF ( ( model_config_rec % bl_pbl_physics(i) .NE. model_config_rec % bl_pbl_physics(1) ) .AND. &
              ( model_config_rec % bl_pbl_physics(i) .NE. 0                                    ) ) THEN
            wrf_err_message = '--- ERROR: bl_pbl_physics must be equal for all domains (or = zero)'
            wrf_err_message = '--- Fix bl_pbl_physics in namelist.input '
         wrf_err_message = '--- Choose another bl_pbl_physics OR use another cu_physics option '
         IF ( ( model_config_rec%bl_pbl_physics(i) .NE. QNSEPBLSCHEME ) .AND. &
         wrf_err_message = '--- NOTE: bl_pbl_physics /= 4, implies mfshconv must be 0, resetting'
            IF ( (model_config_rec%bl_pbl_physics(i) .EQ. YSUSCHEME) .OR. &
                 (model_config_rec%bl_pbl_physics(i) .EQ. SHINHONGSCHEME) .OR. &
                 (model_config_rec%bl_pbl_physics(i) .EQ. MYNNPBLSCHEME2) .OR. &
                 (model_config_rec%bl_pbl_physics(i) .EQ. MYNNPBLSCHEME3) ) THEN
         wrf_err_message = '--- NOTE: bl_pbl_physics /= 1,5,6,11 implies shcu_physics cannot be 3, resetting'
         IF ( ( model_config_rec % bl_pbl_physics(i) .NE. MYNNPBLSCHEME2 ) .AND. &
              ( model_config_rec % bl_pbl_physics(i) .NE. MYNNPBLSCHEME3 ) ) THEN
           IF ( model_config_rec%bl_pbl_physics(i) .EQ. MYNNPBLSCHEME2 .OR. &
                model_config_rec%bl_pbl_physics(i) .EQ. MYNNPBLSCHEME3 ) THEN
         IF ( ( model_config_rec%bl_pbl_physics(i) .EQ. TEMFPBLSCHEME ) .AND. &
            wrf_err_message = '--- ERROR: Using bl_pbl_physics=10 requires sf_sfclay_physics=10 '
         ELSEIF ( ( model_config_rec%bl_pbl_physics(i) .NE. TEMFPBLSCHEME ) .AND. &
            wrf_err_message = '--- ERROR: Using sf_sfclay_physics=10 requires bl_pbl_physics=10 '
         IF ( ( model_config_rec%bl_pbl_physics(i) .EQ. TEMFPBLSCHEME ) .AND. &
            wrf_err_message = '--- ERROR: DFI not available for bl_pbl_physics=10 '
         IF ((model_config_rec%bl_pbl_physics(i) .EQ. MYNNPBLSCHEME2) .OR. &
             (model_config_rec%bl_pbl_physics(i) .EQ. MYNNPBLSCHEME3) ) THEN
! ra_sw_physics, bl_pbl_physics, sf_sfclay_physics, and sf_surface_physics)
                                           orig_bl_pbl_physics, orig_sf_sfclay_physics, orig_sf_surface_physics
      orig_bl_pbl_physics(1:max_dom) = model_config_rec % bl_pbl_physics(1:max_dom)
            IF ( model_config_rec % bl_pbl_physics(i) == -1 ) model_config_rec % bl_pbl_physics(i) = MYJPBLSCHEME        ! MYJ
            IF ( model_config_rec % bl_pbl_physics(i) == -1 ) model_config_rec % bl_pbl_physics(i) = YSUSCHEME           ! YSU
      WHERE (model_config_rec % bl_pbl_physics(1:max_dom) == orig_bl_pbl_physics(1:max_dom)) modified_bl_pbl_option(1:max_dom) = '*'
      WRITE (wrf_err_message, FMT=TRIM(formatstring)) 'bl_pbl_physics: ', &
                                                    (model_config_rec % bl_pbl_physics(i), modified_bl_pbl_option(i), i=1,max_dom)
              ( model_config_rec % bl_pbl_physics(i) .EQ. CAMUWPBLSCHEME  ) .OR. &
         IF ( ( model_config_rec % bl_pbl_physics(i) .NE. MYNNPBLSCHEME2 ) .AND. &
              ( model_config_rec % bl_pbl_physics(i) .NE. MYNNPBLSCHEME3 ) ) THEN
./share/module_check_a_mundo.F
      integer :: sf_sfclay_physics, sf_surface_physics, bl_pbl_physics, cu_physics, surface_input_source
      bl_pbl_physics 		=2
      call ncapt(cdfid, ncglobal, 'BL_PBL_PHYSICS', NCLONG, 1,bl_pbl_physics, rcode)
./var/convertor/nmm_interface_convertor/convert_e2c.F

@dudhia
Copy link
Collaborator

dudhia commented Nov 17, 2021 via email

@davegill
Copy link
Contributor Author

@dudhia @weiwangncar
Folks,

  1. These proposed changes are to two files: a registry file and check_a_mundo.
  2. Other than check_a_mundo, there are no changes to the Fortran source code in the typical user areas: dyn_em, phys, share. The mods to check_a_mundo follow a template, permitting easy understanding of what is going on.
  3. There is no change in the meaning of the switch bl_pbl_physics. When we say bl_pbl_physics=5, 0, then on d01 we run PBL option 5, and on d02 we run PBL option 0.
  4. There is no need for an additional T/F flag for LES processing for particular domains.
  5. These changes allow the model to perform correctly with multiple nests by enforcing that the same PBL variables are available on every domain - regardless of which scheme is selected, whether it is state or a member of a 4d array.
  6. These changes are simple enough that they can be reviewed for a release branch modification.
  7. Because of the structure of the changes, simple tests were able to be conducted to indicate that the code is properly behaving with multiple domains.
  8. Those tests were effectively able to mimic all combinations of PBL and LES options on multiple domains: PBL with PBL, LES with LES, PBL with LES.
  9. There is no concern related to possibly missing specific locations where additional logic is required.
  10. There is no need to explain different usage in the User Guide, as the standard run-time options remain the same.
  11. Should we want to do so, we can utilize the same "behind the scenes" structure to support cumulus options of coarser grids and no cumulus options on finer grids.

@dudhia
Copy link
Collaborator

dudhia commented Nov 29, 2021 via email

@davegill
Copy link
Contributor Author

@dudhia
Jimy,
It is the same registry mod that you did not prefer.

@dudhia
Copy link
Collaborator

dudhia commented Nov 29, 2021 via email

@davegill
Copy link
Contributor Author

Choosing #1596 "PBL + LES restrictions" as a temporary solution.

@davegill davegill closed this Dec 10, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
bug PHYSICS release-v4.3.2 This one or that one
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@davegill @dudhia