Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update thompson mp 20210213 (#567) for gsl/develop #80

Conversation

climbfuji
Copy link

@climbfuji climbfuji commented Feb 26, 2021

This PR contains the code changes from @gthompsnWRF in PR NCAR#567, but merged into the current gsl/develop ccpp-physics branch (and conflicts resolved). The originally git commit history is retained.

For a detailed description of the changes, see the original PR NCAR#567.

Additional changes:

  • updates to two tuning parameters for RUC LSM (from @tanyasmirnova)
  • bugfixes in MYNN PBL, Noah MP and UGWP v1 for uninitialized variables

Associated PRs:

#80 (contains NCAR#567)
NOAA-GSL/fv3atm#75
NOAA-GSL/ufs-weather-model#65

For regression testing, see NOAA-GSL/ufs-weather-model#65

@@ -3742,19 +3703,21 @@ subroutine mp_thompson (qv1d, qc1d, qi1d, qr1d, qs1d, qg1d, ni1d, &
qv1d(k) = MAX(1.E-10, qv1d(k) + qvten(k)*DT)
qc1d(k) = qc1d(k) + qcten(k)*DT
nc1d(k) = MAX(2./rho(k), MIN(nc1d(k) + ncten(k)*DT, Nt_c_max))
nwfa1d(k) = MAX(11.1E6/rho(k), MIN(9999.E6/rho(k), &
nwfa1d(k) = MAX(11.1E6, MIN(9999.E6, &
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@climbfuji I am not sure about the change on line 3745. I think the units of nwfa1d are modified, and later in the code nwfa1 is multiplied by rho. I already wrote about my concern on Greg's PR, but he seemed not to agree.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In lines 1788-1789 in the new/modified code, we have:

         nwfa(k) = MAX(11.1E6, MIN(9999.E6, nwfa1d(k)*rho(k)))
         nifa(k) = MAX(naIN1*0.01, MIN(9999.E6, nifa1d(k)*rho(k)))

In line 2990, we have:

         nwfa(k) = MAX(11.1E6, (nwfa1d(k) + nwfaten(k)*DT)*rho(k))

and then, in lines 3706-3709:

         nwfa1d(k) = MAX(11.1E6, MIN(9999.E6,                           &
                       (nwfa1d(k)+nwfaten(k)*DT)))
         nifa1d(k) = MAX(naIN1*0.01, MIN(9999.E6,                       &
                       (nifa1d(k)+nifaten(k)*DT)))

@tanyasmirnova and gthompsnWRF, the previous lines are assigning values of nwfa1d to nwfa by multiplying it with rho. But lines 3706-3709 are only updating nwfa1d itself, using the same lower/upper limits as for nwfa, and I think therefore @tanyasmirnova is correct that we need to divide by rho here?

But, isn't this an unnecessary operation anyway? After lines 3706-3709, these values of nwfa1d/ninfa1d are not used anymore, and since they are local variables there values are "forgotten"?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am hopeful we can put this issue to rest. The variable nwfa(k) is local inside the main code subroutine and represents aerosols in number per cubic meter. The variable nwfa1d is a vertical column passed in via gt_driver (where it went from 3D to 1D) and absolutely represents number per kg of air. So line#3706 is critical to eliminate the air density or else the number that goes back to the model for advection can be massive when rho is tiny such as up at 100-200km altitude. That is the bug fix for some models with ultra-high top.

I've decided to throw in the multiplication by rho for the Min/Max settings just to ensure no tendency gets created at the far ends of the range in the case that the model drifts off beyond one of the ranges. It shouldn't really be much chance of that, but it's best I don't take that chance.

But, let me emphasize how important line#3706 is that the values must not be divided by rho.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, @gthompsnWRF, all this sounds reasonable to me. You know this code best after all ... @tanyasmirnova are you also satisfied with this explanation?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DomHeinzeller I just pushed another commit to my branch for NCAR#567 dealing with this item, hopefully for final time.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I am fine with the current code.
Actually, I revisited our work with HRRR physics in the global FIM, and found that I did the same change myself in September 2017:
@@ -3547,10 +3553,14 @@
qv1d(k) = MAX(1.E-10, qv1d(k) + qvten(k)*DT)
qc1d(k) = qc1d(k) + qcten(k)*DT
nc1d(k) = MAX(2./rho(k), nc1d(k) + ncten(k)*DT)

  •     nwfa1d(k) = MAX(11.1E6/rho(k), MIN(9999.E6/rho(k),             &
    
  •                   (nwfa1d(k)+nwfaten(k)*DT/rho(k))))
    
  •     nifa1d(k) = MAX(naIN1*0.01/rho(k), MIN(9999.E6/rho(k),                &
    
  •                   (nifa1d(k)+nifaten(k)*DT)/rho(k)))
    

+! nwfa1d(k) = MAX(11.1E6/rho(k), MIN(9999.E6/rho(k), &
+!tgs (nwfa1d(k)+nwfaten(k)*DT/rho(k))))

  •     **nwfa1d(k) = MAX(11.1E6, MIN(9999.E6,             &
    
  •                   (nwfa1d(k)+nwfaten(k)*DT)))**
    

+! nifa1d(k) = MAX(naIN1*0.01/rho(k), MIN(9999.E6/rho(k), &
+!tgs (nifa1d(k)+nifaten(k)*DT)/rho(k)))

  •     **nifa1d(k) = MAX(naIN1*0.01, MIN(9999.E6,                &
    
  •                   (nifa1d(k)+nifaten(k)*DT)))**
    

Greg, sorry, that I haven't checked my FIM code earlier. I agree, let's put this issue to rest now.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yay.

(nwfa1d(k)+nwfaten(k)*DT)))
nifa1d(k) = MAX(naIN1*0.01, MIN(9999.E6/rho(k), &
nifa1d(k) = MAX(naIN1*0.01, MIN(9999.E6, &
(nifa1d(k)+nifaten(k)*DT)))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here.

nc = nc/(1.0_kind_phys-spechum)
nwfa = nwfa/(1.0_kind_phys-spechum)
nifa = nifa/(1.0_kind_phys-spechum)
end if
end if
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@climbfuji Is convert_dry_rho a new namelist parameter? Is it .true. for the GSD physics suite?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently convert_dry_rho is defined as a module variable in physics/mp_thompson.F90 at the top. The value is .False.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to consider covert_dry_rho in GFS_suite_interstitial and GFS_rrtmg_pre?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This variable is host-model specific, so I think yes. I should update this PR and move the convert_dry_rho to GFS_typedefs.F90 and give it a CCPP standard name. Let me do that later today ...

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I definitely like the idea of elevating a boolean variable for whether or not some subroutines should deal with the conversion between moist and dry air units of microphysics species. It will generalize the framework better for adoption of different models.

@DomHeinzeller
Copy link

@gthompsnWRF I pulled in your latest commit 2a2d750 into this PR just now.

Copy link
Collaborator

@joeolson42 joeolson42 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After a VERY quick glance, I have no objections.

@DomHeinzeller
Copy link

@gthompsnWRF I will need to make adjustments in module_mp_thompson.F90 to write messages like

ThompMP: computing qr_acr_qg

only for the master task. We cannot pollute stdout with ntasks number of these lines. Will do so now ...

@DomHeinzeller
Copy link

Quick update/progress report. My UFS debug tests crash with a floating point exception on cheyenne/gnu in line 1973 of radsw_main.F90.

            if ( cldliq <= f_zero ) then
              do ib = nblow, nbhgh
                tauliq(ib) = f_zero
                ssaliq(ib) = f_zero
                asyliq(ib) = f_zero
              enddo
            else
                factor = refliq - 1.5
                index  = max( 1, min( 57, int( factor ) ))
                fint   = factor - float(index)

It crashes in the second last line index =. Apparently an issue with refliq not being defined properly. I'll look into it later tonight when I have more time.

@gthompsnWRF
Copy link

@DomHeinzeller I ran a test with my mods and it didn't crash. But maybe we are running different cases. My test run is on Cheyenne at /glade/u/home/gthompsn/src/ccpp-scm/scm/bin

@DomHeinzeller
Copy link

@DomHeinzeller I ran a test with my mods and it didn't crash. But maybe we are running different cases. My test run is on Cheyenne at /glade/u/home/gthompsn/src/ccpp-scm/scm/bin

Mine is with the UFS. Will take a closer look tomorrow, getting too late today.

…aware version (nc not allocated), reduce noise written to stdout
@climbfuji climbfuji force-pushed the update_thompsonMP_20210213_merged_into_gsl_develop branch from d61764c to 3f2c4e5 Compare March 4, 2021 18:56
@DomHeinzeller
Copy link

@gthompsnWRF I know that freezeH2O.dat hasn't changed, but would it be better to also create a V2 file that goes with the updated files qr_acr_qgV2.dat, qr_acr_qsV2.dat ? Probably less confusing because you have one consistent set of files (filenames) for each version.

I just pushed commit 3f2c4e5 which

  • reduces the "noise" written to stdout
  • fixes a bug for the non-aerosol-aware version (nc is not allocated in this case)
  • makes convert_dry_rho an input argument

…it -> init_thompson, apply bounds after calls to calc_effectRad in radiation, set intent(out) variables in calc_effectRad
@DomHeinzeller
Copy link

DomHeinzeller commented Mar 5, 2021

We are getting there. Now the runs complete in PROD and DEBUG mode with GNU; some still crash with Intel. But I cannot get run-to-run reproducibility yet - probably uninitialized variables somewhere.

@DomHeinzeller
Copy link

Update: I am still getting floating point exceptions in at least one of the regression tests, this time in module_bl_mynn.F90:2950. I'll look into it today. Stay tuned.

@gthompsnWRF
Copy link

Update: I am still getting floating point exceptions in at least one of the regression tests, this time in module_bl_mynn.F90:2950. I'll look into it today. Stay tuned.

I might have to look at that exact line number in MYNN, because I saw messages like this all the time in WRF when the model was about to crash due to a variable going way out of physical bounds (like a pressure or a temperature). So the FPE might be happening there, but its cause is often well upstream.

@climbfuji
Copy link
Author

Update: I am still getting floating point exceptions in at least one of the regression tests, this time in module_bl_mynn.F90:2950. I'll look into it today. Stay tuned.

I might have to look at that exact line number in MYNN, because I saw messages like this all the time in WRF when the model was about to crash due to a variable going way out of physical bounds (like a pressure or a temperature). So the FPE might be happening there, but its cause is often well upstream.

That's what I think is happening. Unfortunately Cheyenne is down this week, I did the testing there. Moving to hera now.

@DomHeinzeller
Copy link

gthompsnWRF getting closer. One problem now is that the lower/upper limits for some of the radii conflict with the requirements in RRTMG. For instance, in RRTMG's radlw_main.F90:

                  if (radice .lt. 5.0_rb .or. radice .gt. 140.0_rb) then
                         write(errmsg,'(a,i5,i5,f8.2,f8.2)' )           &
     &         'ERROR: ICE GENERALIZED EFFECTIVE SIZE OUT OF BOUNDS'    &
     &          ,ig, lay, ciwpmc(ig,lay), radice

That means the lower limit for cloud effective radii is 5 micron, and I had adjusted the limits in Thompson MP previously to match those limits. Your recent changes re-introduce the original Thompson MP limits, in this example 2.5 micron for re_ice. What is your suggestion? Sanitizing those outside of Thompson MP means that we need to make changes in several places, but adjusting the value in Thompson MP may not be what you want for other radiation packages?

@gthompsnWRF
Copy link

Dom,

Yes, I remember this. At least in the method RRTMG was introduced within WRF, the min/max size of water and ice is already constrained by the particle size limits when the various extinction/scattering coefficients were created. I don't recall exactly what those limits are but roughly 2 microns (water or ice perhaps) up to 130 or 140 microns for ice. Somewhere, internally, RRTMG protects against going outside that range.

Those boundary/limits should probably be module-wide parameter constants at the start, defined per rad scheme, and while MP-Thompson or some other MP scheme may calculate beyond those limits for adaptability to other needs, they can get constrained by the radiation scheme's limits. Personally, I wanted the smallest and largest sizes possible for expansion and use of my radii variables to go to any radiation code (like CRTM which may have wider limits than RRTMG).

So my recommendation is each rad scheme should catch lower/higher values and peg those at their internal limits rather than limit what the MP scheme people might permit to be calculated for other usages.

@DomHeinzeller
Copy link

@joeolson42 I found a bug in module_bl_mynn.F90 while doing the tests of Greg's changes. In line 2950, it complains about an uninitialized variable. The line is

    khdz(kte+1)=rhoz(kte+1)*dfh(kte)

and the entire code block related to those variables is:

    !Prepare "constants" for diffusion equation.
    !khdz = rho*Kh/dz = rho*dfh
    dtz(kts)   =delt/dz(kts)
    rhoz(kts)  =rho(kts)
    rhoinv(kts)=1./rho(kts)
    khdz(kts)  =rhoz(kts)*dfh(kts)
    kmdz(kts)  =rhoz(kts)*dfm(kts)
    DO k=kts+1,kte
       dtz(k)   =delt/dz(k)
       rhoz(k)  =(rho(k)*dz(k-1) + rho(k-1)*dz(k))/(dz(k-1)+dz(k))
       rhoz(k)  =  MAX(rhoz(k),1E-4)
       rhoinv(k)=1./MAX(rho(k),1E-4)
       dzk      = 0.5  *( dz(k)+dz(k-1) )
       khdz(k)  = rhoz(k)*dfh(k)
       kmdz(k)  = rhoz(k)*dfm(k)
    ENDDO
    khdz(kte+1)=rhoz(kte+1)*dfh(kte)
    kmdz(kte+1)=rhoz(kte+1)*dfm(kte)

The culprit is rhoz(kte+1). How do you want this variable to be initialized? Thanks.

@DomHeinzeller
Copy link

Dom,

Yes, I remember this. At least in the method RRTMG was introduced within WRF, the min/max size of water and ice is already constrained by the particle size limits when the various extinction/scattering coefficients were created. I don't recall exactly what those limits are but roughly 2 microns (water or ice perhaps) up to 130 or 140 microns for ice. Somewhere, internally, RRTMG protects against going outside that range.

Those boundary/limits should probably be module-wide parameter constants at the start, defined per rad scheme, and while MP-Thompson or some other MP scheme may calculate beyond those limits for adaptability to other needs, they can get constrained by the radiation scheme's limits. Personally, I wanted the smallest and largest sizes possible for expansion and use of my radii variables to go to any radiation code (like CRTM which may have wider limits than RRTMG).

So my recommendation is each rad scheme should catch lower/higher values and peg those at their internal limits rather than limit what the MP scheme people might permit to be calculated for other usages.

Greg, I followed your suggestion, see here please: f573390

@DomHeinzeller
Copy link

@joeolson42 I found a bug in module_bl_mynn.F90 while doing the tests of Greg's changes. In line 2950, it complains about an uninitialized variable. The line is

    khdz(kte+1)=rhoz(kte+1)*dfh(kte)

and the entire code block related to those variables is:

    !Prepare "constants" for diffusion equation.
    !khdz = rho*Kh/dz = rho*dfh
    dtz(kts)   =delt/dz(kts)
    rhoz(kts)  =rho(kts)
    rhoinv(kts)=1./rho(kts)
    khdz(kts)  =rhoz(kts)*dfh(kts)
    kmdz(kts)  =rhoz(kts)*dfm(kts)
    DO k=kts+1,kte
       dtz(k)   =delt/dz(k)
       rhoz(k)  =(rho(k)*dz(k-1) + rho(k-1)*dz(k))/(dz(k-1)+dz(k))
       rhoz(k)  =  MAX(rhoz(k),1E-4)
       rhoinv(k)=1./MAX(rho(k),1E-4)
       dzk      = 0.5  *( dz(k)+dz(k-1) )
       khdz(k)  = rhoz(k)*dfh(k)
       kmdz(k)  = rhoz(k)*dfm(k)
    ENDDO
    khdz(kte+1)=rhoz(kte+1)*dfh(kte)
    kmdz(kte+1)=rhoz(kte+1)*dfm(kte)

The culprit is rhoz(kte+1). How do you want this variable to be initialized? Thanks.

I added an initialization for rhoz(kte+1) in commit c579871, @joeolson42 please review. Thanks!

@DomHeinzeller
Copy link

DomHeinzeller commented Mar 12, 2021

Looks good to me. Thanks.

Thank you! Hopefully this was the last change to make for these PRs.

@DomHeinzeller
Copy link

@gthompsnWRF The WRF changes themselves look good by now, I think. With the more strict compiler settings I am a few other places in various physics schemes with potentially uninitialized variables, and I am going to fix those as part of the commit.

@tanyasmirnova @joeolson42 @ligiabernardet @grantfirl @mdtoyNOAA FYI

@DomHeinzeller
Copy link

@barlage @HelinWei-NOAA I added a small bugfix for uninitialized variables in commit b23f06a. This will go to the NOAA-GSL fork, branch gsl/develop, first and then to the authoritative repositories in a week or two. Please check.

@ValeryYudin-NOAA @mdtoyNOAA I added a small bugfix for uninitialized variables in commit b67acad. This will go to the NOAA-GSL fork, branch gsl/develop, first and then to the authoritative repositories in a week or two. Please check.

Thanks!

@ValeryYudin-NOAA
Copy link

ValeryYudin-NOAA commented Mar 15, 2021 via email

@DomHeinzeller
Copy link

@tanyasmirnova @gthompsnWRF everything looks good now. Do you want to take another look and approve if ok?

@DomHeinzeller
Copy link

@DomHeinzeller not a big deal, but bgap and wgap are already initialized above; we can clean this up when we move to external repository later.

Good point, I fixed that and moved the ib-independent variables, too, see bbec192.

@barlage
Copy link

barlage commented Mar 15, 2021

OK, good. I was going to suggest removing that whole nband loop but didn't want to get too off topic...

Copy link
Collaborator

@tanyasmirnova tanyasmirnova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@climbfuji All changes, including the tuning parameters change in RUC LSM, look good to me. Thank you!

@DomHeinzeller DomHeinzeller merged commit 8507df6 into NOAA-GSL:gsl/develop Mar 16, 2021
@climbfuji climbfuji deleted the update_thompsonMP_20210213_merged_into_gsl_develop branch June 27, 2022 03:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants