Adding the WRF-Solar EPS model

[pedro-jm]

TYPE: New feature

KEYWORDS: WRF-Solar, ensemble, stochastic

SOURCE: Pedro A. Jimenez, Ju Hye Kim, Jimy Dudhia (NCAR) and Jaemo Yang (NREL)

DESCRIPTION OF CHANGES:
Problem:
The WRF-Solar model is a deterministic model tailored for solar energy applications. Having a probabilistic component tailored for solar energy application is missing.

Solution:
WRF-Solar Ensemble Prediction System (WRF-Solar EPS, Kim et al. 2021) was designed to provide probabilistic forecasts tailored for solar energy applications. We have selected the most relevant variables from 6 WRF packages using tangent linear methodologies (Yang et al. 2021). Then we have used the available stochastic infrastructure in WRF to perturb these variables at runtine. The scheme is activated setting multi_perturb = 1 in the &stoch block of namelist.input. The characteristics of the perturbations are specified in a new file in the run directory called STOCHPERT.TBL. Then the perturbations to each module and variable
are turned on/off via namelist settings (see run/README.namelist). More information can be found in this very preliminar website: https://ral.ucar.edu/solutions/products/wrf-solar-eps

Link to Yang et al. 2021 article: https://www-sciencedirect-com.cuucar.idm.oclc.org/science/article/pii/S0038092X21002322?via%3Dihub

Link to Kim et al. 2021 article: https://ieeexplore-ieee-org.cuucar.idm.oclc.org/document/9580552

LIST OF MODIFIED FILES:
M Registry/Registry.EM_COMMON
M Registry/registry.dimspec
M Registry/registry.stoch
M dyn_em/module_first_rk_step_part1.F
M dyn_em/module_first_rk_step_part2.F
M dyn_em/module_stoch.F
M dyn_em/solve_em.F
M phys/module_microphysics_driver.F
M phys/module_pbl_driver.F
M phys/module_ra_farms.F
M phys/module_radiation_driver.F
M phys/module_sf_noahdrv.F
M phys/module_shallowcu_driver.F
M phys/module_shcu_deng.F
M phys/module_surface_driver.F
M run/README.namelist
A run/STOCHPERT.TBL
M share/module_check_a_mundo.F

TESTS CONDUCTED:

1. We have run multiple years WRF-Solar EPS and analyzed the characteristics of the probabilistic forecasts as well as calibrated forecasts. Our analysis do not show any potential problems at this point.
2. We ran a simulation with 10 ensemble members and calculated the spread of the ensemble (standard deviation) as a function of the lead time. The spread (this PR compared to our other tests) is very similar.
   Below is plot with the ensemble spread at given lead time original (left) and current mods (right). The area covered is CONUS.

3. Jenkins testing is all PASS.

RELEASE NOTE: WRF-Solar was expanded to have a stochastic ensemble prediction system (WRF-Solar EPS) tailored for solar energy applications (Yang et al. 2021, Kim et al. 2022). The stochastic perturbations can be introduced into variables of six parameterizations controlling cloud and radiation processes. A more detailed description of the model is provided in the WRF-Solar EPS website:
https://ral.ucar.edu/solutions/products/wrf-solar-eps

Kim, J.-H., P.A. Jimenez, M. Sengupta, J. Yang, J. Dudhia, S. Alessandrini and Y. Xie, 2022: The WRF-Solar Ensemble Prediction System to provide solar irradiance probabilistic forecasts. IEEE J. of Photovoltaics (In press.)

Yang, J., J.-H. Kim, P. A. Jimenez, M. Sengupta, J. Dudhia, Y. Xie, A. Golnas, R. Giering: An Efficient method to identify uncertainties of WRF-Solar variables in forecasting solar irradiance using a tangent linear sensitivity analysis. Solar Energy, 220, 509-522.

[joeolson42]

I think we need to talk about the MYNN changes. This looks like it will be a significant snag in our code universalization efforts because these new stochastic variables/arrays are not in CCPP. Either this alternative stochastic approach needs to be added to CCPP or it needs to be merged with the preexisting stochastic approach in the MYNN. I'm open to compromises, but it would take some convincing to have to carry two separate approaches. It looks like there is considerable overlap in the targeted impact of both stochastic approaches.

[dudhia]

Joe, I see what you mean. Perhaps these could be made optional arguments and controlled within IF(PRESENT) sections of the code. Jimy

…

On Wed, Sep 1, 2021 at 3:06 PM Joseph Olson ***@***.***> wrote: I think we need to talk about the MYNN changes. This looks like it will be a significant snag in our code universalization efforts because these new stochastic variables/arrays are not in CCPP. Either this alternative stochastic approach needs to be added to CCPP or it needs to be merged with the preexisting stochastic approach in the MYNN. I'm open to compromises, but it would take some convincing to have to carry two separate approaches. It looks like there is considerable overlap in the targeted impact of both stochastic approaches. — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#1547 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEIZ77DGP62KF3W6V7C6UWLT72IXHANCNFSM5DHDTNKQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[joeolson42]

That may be a work around but it's ugly. I'm trying to eliminate dycore-specific code so the same features/configurations are possible in all dycores, not just identical code. Maybe we can do this in the short term but it should be called out as a bad practice if code universalization is truly our goal.

Is it possible to perturb these arrays (th, qv, qc, qi, etc) before they are input to the MYNN? It seems like these mods could be an interstitial routine.

[pedro-jm]

Joe, I think it may be possible to do the perturbations outside MYNN. Let me think about it a bit more and take a look at the code.
Pedro.

[dudhia]

Yes, we could consider doing all the schemes this way with a perturbation-adding routine ahead and subtracting it after. Haven't thought if there are any obstacles to this yet. In principle it may not even have to be scheme dependent, just in the driver, but tke may be a special case for some. Jimy

…

On Wed, Sep 1, 2021 at 5:21 PM pedro-jm ***@***.***> wrote: Joe, I think it may be possible to do the perturbations outside MYNN. Let me think about it a bit more and take a look at the code. Pedro. — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#1547 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEIZ77BFGNOVMJLK6X6FHODT72YQXANCNFSM5DHDTNKQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[pedro-jm]

Joe, @joeolson42
I was able to implement the perturbations outside the MYNN module. No changes in module_bl_mynn.F.
So we should be good to go with respect to MYNN.

[joeolson42]

Thanks Pedro. -joe

…

On Wed, Sep 8, 2021 at 4:58 PM pedro-jm ***@***.***> wrote: Joe, @joeolson42 <https://github.com/joeolson42> I was able to implement the perturbations outside the MYNN module. No changes in module_bl_mynn.F. So we should be good to go with respect to MYNN. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1547 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADLRR3SROAHQIAV46EOSOQ3UA7TAXANCNFSM5DHDTNKQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Joseph Olson Model Physics Branch Chief Environmental Prediction Advancement Division NOAA-Global Systems Laboratory Boulder, Colorado

[pedro-jm]

All the variables are now perturbed in the drivers (instead of the parameterizations modules).

[weiwangncar]

@pedro-jm Can you verify that a restart run would produce identical results as a continuous run? What about runs produced by different number of processors? Can you also show some plots that may illustrate the spread you can produce with this method?

[pedro-jm]

@davegill
I did a test and it appears to work fine. I run a sim with 10 ensemble members using the code before and after your mods. I then calculated the spread of the ensemble (standard deviation) as a function of the lead time. Spread is very similar. Below is plot with the ensemble spread at given lead time before (left) and after (right) your changes. The area covered is CONUS.

[pedro-jm]

The spread in the previous plots is for the global horizontal irradiance (GHI).

[davegill]

@pedro-jm
Great news!
Thank you much

[davegill]

@weiwangncar @dudhia @pedro-jm
Wei and Jimy,
This is ready for review.

Pedro,
Thank you so much for checking this out, especially considering the PLAGUE!

[weiwangncar]

@pedro-jm Can you verify that a restart run would produce identical results as a continuous run? We also like to know if the code would produce identical results running it with one and multiple processors. You can do this with dmpar-compiled code running it with one and then multiple processors.

[pedro-jm]

@davegill
Thanks to you. You added considerable devels. We probably would not be able to add all of that on time.

@weiwangncar
I should be able to do those test

[dudhia]

Does @pedro-jm need to turn off optimization for this test or can it work optimized?

…

On Wed, Jan 19, 2022 at 8:33 PM weiwangncar ***@***.***> wrote: @pedro-jm <https://github.com/pedro-jm> Can you verify that a restart run would produce identical results as a continuous run? We also like to know if the code would produce identical results running it with one and multiple processors. You can do this with dmpar-compiled code running it with one and then multiple processors. — Reply to this email directly, view it on GitHub <#1547 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEIZ77BZEE3RVKZXZ547LHDUW57BPANCNFSM5DHDTNKQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[weiwangncar]

@pedro-jm @dudhia For restart test, optimized code should be the one to use. For bit-for-bit test, all we require is to use code without optimization. But if using optimized code gives you bit-for-bit results, it is good too. Many of our options can produce bit-for-bit results when using different number of processors.

[davegill]

@pedro-jm @weiwangncar @dudhia

Does Pedro need to turn off optimization for this test or can it work optimized?

Yes, we tend to find that unoptimized builds produce bit-identical results more often.

1. use "configure -d", that will turn off all optimization
2. run about 10 - 20 time steps for the parallel test, you can use an MPI-only run and then vary the number of tasks
3. use a REALLY small domain, perhaps 70x70 up to 100x100, which means you could run "mpirun -np 1" and "mpirun -np 4", for example
4. for the restart test, compare 20 time steps to a 10+10 time step simulation

[pedro-jm]

I am able to get bit-identical results for the first 20 time steps I checked using 1 and 36 cores.

However, I needed to change the PBL setting from 5 to 0 to get bit-identical results. Turning off the perturbations of this PR and using PBL 5 does does not produce bit-identical results. There are no changes for PBL 5 in this PR so It seems we should be fine here.

[pedro-jm]

There was an oversight on my side with the perturbations and the bit-identical results for 1 and 36 cores.

I get bit-identical results when I perturb these modules:
pert_noah = .true.
pert_thom = .true.
pert_cld3 = .true.

I DO NOT get bit-identical results when I perturb these modules:
pert_farms = .true.
pert_deng = .true.

I cannot test this one:
pert_mynn = .true.

[pedro-jm]

I forgot to document that the random perturbations are OK.
The issue appears when we apply them. So it is clear where to look for the problems.

[pedro-jm]

Restarts currently do not provide bit for bit results. This is expected: we do not have the perturbations in the wrfrst files. And there might be issues with the initialization of the perturbations in the restart. We can put a check in checkamundo to stop if it is a restart run and we have this PR activated. Most of the users of this PR probably would not use restarts.

[davegill]

@weiwangncar @dudhia
What are opinions? If you deem that this is not ready for a review, I doubt anyone really has the time to make it ready for a review.

[dudhia]

I think it can go in and we sort out remaining issues later.

…

On Fri, Jan 21, 2022 at 10:51 AM Dave Gill ***@***.***> wrote: @weiwangncar <https://github.com/weiwangncar> @dudhia <https://github.com/dudhia> What are opinions? If you deem that this is not ready for a review, I doubt anyone really has the time to make it ready for a review. — Reply to this email directly, view it on GitHub <#1547 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEIZ77HRWUUKD7HBB2HEGOLUXGMLJANCNFSM5DHDTNKQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[weiwangncar]

@pedro-jm @dudhia Can you add a bit more info to the RELEASE NOTE? Since that is the bit we are going to describe the update in the release. Information on which physics that may be perturbed. You can site Solar physics suite, but it may be explicitly stated here for general info. Also does the paper describe how the STOCHPERT.TBL is created? Some words about that and paper reference in the RELEASE NOTE will be useful.

[pedro-jm]

@weiwangncar
I just updated the RELEASE NOTES info providing a link to the WRF-Solar EPS website. If you still miss any relevant info let me know.

[weiwangncar]

@pedro-jm Thanks for the added info. I'm ok with this PR. It is good that the test results are recorded in this PR (restart, bit-for-bit), so that we can refer to this later.

[davegill]

Jimy says it can go in AS-IS, ands we sort things out before the v4.4 release