Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HAFSv0.3 moving-nesting related coupling, vortex initialization, and data assimilation capabilities #132

Merged
merged 276 commits into from
Apr 29, 2022

Conversation

BinLiu-NOAA
Copy link
Collaborator

@BinLiu-NOAA BinLiu-NOAA commented Apr 15, 2022

Description of changes

The HAFSv0.3 phase2 configuration contains the latest HAFS moving nesting, coupling, vortex initialization, data assimilation capabilities, and it points to the latest develop branch (as of 04/12/2022) of ufs-weather-model. Besides, the hafs_utils.fd and hafs_gsi.fd were also synced with their corresponding develop/master branches recently.

  • Details and contributors regarding the hafs_forecast.fd level HAFS moving-nesting and coupling related developments can be seen from HAFS moving-nesting related developments ufs-community/ufs-weather-model#1104.
  • The HWRF/HMON type vortex initialization (VI) capability mainly mainly developed/ported to HAFS by @Qingfu-Liu, @BinLiu-NOAA, @JunghoonShin-NOAA, @ChuankaiWang-NOAA.
  • The HAFS datool developed by @yonghuiweng is used in HAFS VI pre- and post-processing procedures to interpolate FV3 restart files into the HAFS VI input data as well as HAFS VI analysis back to FV3 restart files. In addition, the HAFS datool is also used in atm_merge and analysis_merge jobs/tasks/steps for implementing various HAFS cold-start or warm-start cycling strategies.
  • Implemented the HAFS VI/DA application/workflow capability for both regional single-domain and moving-nesting configurations.
  • Added a HAFS application/workflow regression test in cronjob_hafs_rt.sh to test the regional storm-focused moving-nesting configuration with vortex initialization and domain 02 data assimilation.

Issues addressed (optional)

If this PR addresses one or more issues, please provide link(s) to the issue(s) here.

Tests conducted

This feature/hafsv0.3_phase2 branch has been tested on various HAFS supported platforms.

Application-level regression test status

Running the HAFS application-level regression tests is currently performed by code reviewers after the developer creates the initial PR. As regression tests are conducted, the testers should use the checklist below to indicate successful regression tests. You may add other tests as needed. If a test fails, do not check the box. Instead, describe the failure in the PR comments, noting the platform where the test failed.

  • Jet
  • Hera
  • Orion
  • WCOSS Dell P3
  • WCOSS Cray

BinLiu-NOAA and others added 30 commits February 10, 2021 14:51
…lobal

moving nest tests from Bill Ramstrom. Only for the prescribed nest moving currently.
…31' into feature/multi_nests (Bill and Bin).
*Update submodule sorc/hafs_utils.fd
*Configure two C96 globnest configurations:
 -hafs_globnest2_C96_static.conf: two same level static nests
 -hafs_globnest2tele_C96_static.conf: two telescopic static nests
hafs_globnest2tele_C96_static.conf (two telescopic nests) have run throught the
forecast jobs.
@panll
Copy link
Contributor

panll commented Apr 25, 2022

Regression tests passed on Hera after some minor modifications. @BinLiu-NOAA @evankalina The post process numbers need to be increased (e.g., changing from 2 nodes to 4 nodes) to pass the tests on Hera for HAFS_rt_regional_atm_only, HAFS_rt_regional_static_atm_ocn, HAFS_rt_regional_atm_ocn_wav, HAFS_rt_regional_mvnest_vida, HAFS_rt_regional_static_atm_wav

@BinLiu-NOAA
Copy link
Collaborator Author

Regression tests passed on Hera after some minor modifications. @BinLiu-NOAA @evankalina The post process numbers need to be increased (e.g., changing from 2 nodes to 4 nodes) to pass the tests on Hera for HAFS_rt_regional_atm_only, HAFS_rt_regional_static_atm_ocn, HAFS_rt_regional_atm_ocn_wav, HAFS_rt_regional_mvnest_vida, HAFS_rt_regional_static_atm_wav

Thanks, @panll! I am updating the ATM_POST_RESOURCES to use 4 nodes with 20 PEs per node as follows:

4:ppn=20:tpp=1TOTAL_TASKS80NCTSK20OMP_THREADS107:59:00">

Hopefully this will work fine.

@BijuThomas-NOAA
Copy link
Contributor

Again, as you helped previously, please help to conduct the HAFS application/workflow level regression tests for this PR. @evankalina, @panll, @mrinalbiswas (Hera and Orion) @JunghoonShin-NOAA (Jet) @LinZhu-NOAA (wcoss_dell_p3) @BijuThomas-NOAA (wcoss_cray)

atm_vi, atm_vi_fgat03, and atm_vi_fgat06 of hafsv0.3_phase2_rt_regional_mvnest_vida failed for the second cycle on wcoss-cray. The log file has several NaN values. @yonghuiweng: Can you take a look? Can you check if the interpolation in hafs_datool is working properly? /gpfs/hps3/ptmp/Biju.Thomas/hafsv0.3_phase2_rt_regional_mvnest_vida/2020082512/13L

Regression tests passed on WCOSS(Cray) after using classic NetCDF option(-n4) for the mppnccombine executable.

*Turn off delivering hafs.trak.atcfunix.orig file to NOSCRUB dir.
@JunghoonShin-NOAA
Copy link
Contributor

Hello Bin,
All regression tests for this branch passed successfully on Jet (Jet was slow again). No modification was needed to run all regression tests.
Thank you.

Copy link
Contributor

@mrinalbiswas mrinalbiswas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RTs passed on Orion.

…n ufs-weather-model.

*Fix the forecast job/task hanging/failure on wcoss_dell_p3.
@BinLiu-NOAA
Copy link
Collaborator Author

@BinLiu-NOAA The forecast job failed for the first cycle of da_C192s1n4_3denvar and regional_static_atm_ocn on wcoss-dell. The log file seems hang there.

@LinZhu-NOAA, I have just pushed a fix for the forecast job hanging/failure on wcoss_dell_p3. Please "git pull" the latest changes and rewind the failed tasks. And they should now go through successfully. The issue was the inconsistent modules loaded in modulefile.hafs.wcoss_dell_p3. Thanks!

@LinZhu-NOAA
Copy link
Contributor

@BinLiu-NOAA The regression tests completed successfully on WCOSS-Dell after updating the modules.

setting ll_bin size and max_points.
*Use the weighting function of exp(-r/r_scale) for the empirical linear
interpolation method, currently r_scale is hardcoded as 2 km, which can be
improved in the future so that it can be automatically determined/adjusted.
*These fixes/changes are based on discussions among Bin, Yonghui, and Zhan.
…sh so that

it is consitent with the value currently hardcoded in the source code.
@BinLiu-NOAA
Copy link
Collaborator Author

Thanks to @panll, @mrinalbiswas, @LinZhu-NOAA, @JunghoonShin-NOAA, @BijuThomas-NOAA for conducing the application level regression tests! And we have also fixed the hafsvi_pre-/postproc interpolation issues. So, I think this PR is ready merge now. Also, @evankalina, since this PR contains lots of developments and commits, I was thinking of merging this PR by keeping its commit log. Does is sound good to you? Thanks!

@BinLiu-NOAA BinLiu-NOAA added the Ready to Merge Review and RTs are all finished. The PR is ready to merge. label Apr 29, 2022
@evankalina
Copy link
Contributor

@BinLiu-NOAA Keeping the list of commits sounds good to me. Congratulations on getting this PR ready!

@BinLiu-NOAA
Copy link
Collaborator Author

Thanks All! Merging to close this PR now.

@BinLiu-NOAA BinLiu-NOAA merged commit 41b70b8 into develop Apr 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Ready for Review The PR is considered complete by the author(s) Ready for RTs The PR is ready to conduct Regression Tests Ready to Merge Review and RTs are all finished. The PR is ready to merge.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Merge HAFSv0.3 phase-2 configuration back to HAFS develop
8 participants