Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Single step oro & fix files generation procedure for atm and ocn res combo #830

Merged
merged 92 commits into from
Dec 12, 2023

Conversation

sanatcumar
Copy link
Collaborator

@sanatcumar sanatcumar commented Jun 15, 2023

DESCRIPTION OF CHANGES:

Consolidated the multi-step process for generating the oro and fix files for combinations of atmospheric and ocean grid resolutions.

TESTS CONDUCTED:

  • Compile branch on all Tier 1 machines using Intel (Orion, Jet, Hera and WCOSS2). Done using d0d1e8f.
  • Compile branch on Hera using GNU. Done using d0d1e8f.
  • Compile branch in 'Debug' mode on WCOSS2. Done using d0d1e8f on Dogwood.
  • Run unit tests locally on any Tier 1 machine. Done on Hera using d0d1e8f. All tests passed.
  • Run relevant consistency tests locally on all Tier 1 machine. Done using d0d1e8f. All tests other than grid_gen tests passed. This result was expected. For details on the grid_gen tests, see below.

Describe any additional tests performed.
Pasted below is a summary of the tests conducted.

New scripts and codes were written to reduce the multi step procedure of generating coupled grids into a single step process.

During the development of this new single step process, the original “orog” code was split into two parts that is sequentially activated by flags in the scripts to first generate a land mask and then use the land mask generated by the ocean merge procedure to populate the orography fields. The following information documents the effects of these code changes.

Five comparisons with baselines were made.

  1. Same grids, same data sets and different orography code

Ocean merged files (Identical)
Orography (Minor differences along spatial pattern at zero lat, 0, 90, 180 & 270 lon)
Vegetation type (Mostly identical)

  1. Different grids, same data sets, different orography code (Comparison with /scratch1/NCEPDEV/global/glopara/fix/orog/20230615)

Grids (Different but minor )
Orography (Minor differences along spatial pattern at zero lat, 0, 90, 180 & 270 lon)
Vegetation type (Mostly identical)

  1. Same grids, different data sets, vegfrac=False, different orography code

Vegetation type (Mostly identical)

  1. Same grids, different data sets, vegfrac=True, different orography code

Vegetation type (Mostly identical)

  1. Comparisons of orography files and select sfc fields for grid types other than uniform with files created using the latest develop branch

Orography (Identical)
Vegetation type (Identical)

DEPENDENCIES:

#843 must be merged first. #854 needs to be merged first

DOCUMENTATION:

The repository was compiled on Hera with Doxygen (d0d1e8f). The orog.fd and ocean_merge.fd had no Doxygen warnings.

ISSUE:

Fixes #825.

CONTRIBUTORS:

@GeorgeGayno-NOAA , @shansun6 , @aerorahul , @barlage

@sanatcumar sanatcumar marked this pull request as ready for review June 15, 2023 13:49
@sanatcumar sanatcumar changed the title Single step Single step oro & fix files generation procedure for atm and ocn res combo Jun 15, 2023
@sanatcumar
Copy link
Collaborator Author

sanatcumar commented Jun 23, 2023

New scripts and codes were written to reduce the multi step procedure of generating coupled grids into a single step process.

During the development of this new single step process, the original “orog” code was split into two parts that is sequentially activated by flags in the scripts to first generate a land mask and then use the land mask generated by the ocean merge procedure to populate the orography fields. The following information documents the effects of these code changes.

Detailed results are here

Five different comparisons were made (Inference)

  1. Same grids, same data sets and different orography code
  • Ocean merged files (Identical)
  • Orography (Minor differences along spatial pattern at zero lat, 0, 90, 180 & 270 lon)
  • Vegetation type (Mostly identical)
  1. Different grids, same data sets, different orography code (Comparison with /scratch1/NCEPDEV/global/glopara/fix/orog/20230615)
  • Grids (Different but minor )
  • Orography (Minor differences along spatial pattern at zero lat, 0, 90, 180 & 270 lon)
  • Vegetation type (Mostly identical)
  1. Same grids, different data sets, vegfrac=False, different orography code
  • Vegetation type (Mostly identical)
  1. Same grids, different data sets, vegfrac=,True, different orography code (comparison with
  • Vegetation type (Mostly identical)
  1. Comparisons of orography files and select sfc fields for grid types other than uniform with files created using the latest develop branch
  • Orography (Identical)
  • Vegetation type (Identical)

@GeorgeGayno-NOAA GeorgeGayno-NOAA self-requested a review June 26, 2023 17:18
input.nml Outdated Show resolved Hide resolved
ush/fort.15 Outdated Show resolved Hide resolved
ush/fort.235 Outdated Show resolved Hide resolved
ush/fort.25 Outdated Show resolved Hide resolved
@GeorgeGayno-NOAA
Copy link
Collaborator

I tried running 0b9a03e on Hera. I used the default settings. I got this error:

+ /scratch1/NCEPDEV/da/George.Gayno/ufs_utils.git/UFS_UTILS.sanath/driver_scripts/../ush/fv3gfs_ocean_merge.sh 96 500
ocean_merge  exe file is found in  /scratch1/NCEPDEV/da/George.Gayno/ufs_utils.git/UFS_UTILS.sanath/driver_scripts/../exec/
 flnm_ocn=
 /scratch1/NCEPDEV/stmp4/Sanath.Kumar/CPLD_GRIDGEN//500/C96.mx500.tile1.nc
                                           ncid=       65536  ndims=           2
 nvars=           3  natts=           0
 lat=          96 lon=          96
  flnm2=/scratch2/NCEPDEV/stmp1/George.Gayno/my_grids/C96/C96_oro_data.tile1.nc
 No such file or directory

999
error in fv3gfs_ocean_merge.sh

@sanatcumar
Copy link
Collaborator Author

Thanks for catching this bug. I just fixed and tested it . The issue was incorrect paths in the ocean merge routine. I just fixed and tested the default settings with empty directories.

@GeorgeGayno-NOAA
Copy link
Collaborator

To get the regression tests to work, I had to update these scripts.

  • c96.viirs.bnu.sh
  • esg.regional.pct.cat.sh
  • esg.regional.sh
  • gfdl.regional.sh
  • regional.gsl.gwd.sh

Also, since the default for uniform grids will now include the ocean merge step, I want to update this regression test to run that step:

  • c96.uniform.sh

Please find all updated scripts on Orion here: /work/noaa/da/ggayno/save/ufs_utils.git/UFS_UTILS.sanath/reg_tests/grid_gen:

@sanatcumar
Copy link
Collaborator Author

To get the regression tests to work, I had to update these scripts.

  • c96.viirs.bnu.sh
  • esg.regional.pct.cat.sh
  • esg.regional.sh
  • gfdl.regional.sh
  • regional.gsl.gwd.sh

Also, since the default for uniform grids will now include the ocean merge step, I want to update this regression test to run that step:

  • c96.uniform.sh

Please find all updated scripts on Orion here: /work/noaa/da/ggayno/save/ufs_utils.git/UFS_UTILS.sanath/reg_tests/grid_gen:

Hi George, although the "ocn " is defined in the driver, it is not declared as a default in the /ush/fv3gfs_driver*.sh. This way the grid gen test 1 will run and produce results that fails the original only because of the different orog .

I agree adding a test is the way forward. I will copy over these files and merge them in this. Thanks.

#-----------------------------------------------------------------------

export home_dir=$SLURM_SUBMIT_DIR/..
export TEMP_DIR=/scratch2/NCEPDEV/stmp1/$LOGNAME/fv3_grid.$gtype
export out_dir=/scratch2/NCEPDEV/stmp1/$LOGNAME/my_grids
export out_dir=/scratch2/NCEPDEV/stmp1/$LOGNAME/my_grids_ocean_fix/
#export ocean_mask_dir=/scratch1/NCEPDEV/stmp4/Sanath.Kumar/ocean_mask/CPLD_GRIDGEN/
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would set ocean_mask_dir in fv3gfs_driver_grid.sh using the $home_dir variable to set the path. Then, you don't need to declare it in every machine's driver script.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HI @GeorgeGayno-NOAA ,
I had to use dirname in /ush/fv3gfs_ocean_merge.sh as the fortran merge code could not handle the "../" relative paths.
ocean_mask_dir="$(dirname $(dirname $home_dir))/fix/orog/C${res}/ocean_mask/${ocn}/""
It works now. But is there a better solution

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Try this:

Update fv3gfs_ocean_merge.sh as follows:

     cat << EOF > input.nml
      &mask_nml
-     ocean_mask_dir="$(dirname $(dirname $home_dir))/fix/orog/C${res}/ocean_mask/${ocn}/"
+     ocean_mask_dir="${home_dir}/fix/orog/C${res}/ocean_mask/${ocn}/"
      ocnres="mx${ocn}"
      lake_mask_dir="${TEMP_DIR}/C${res}/orog/"
      atmres="C${res}"
@@ -22,6 +23,8 @@ EOF

and merge_lake_ocnmsk.f90 as follows

   integer :: binary_lake

-  character(len=120) :: flnm
+  character(len=250) :: flnm
   integer :: ncid,ndims,nvars,natts,lat,lon,v1id,v2id,v3id,v4id,start(2),count(2),i,j,latid,lonid,ncid4, dims(2),tile,nodp_pt

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This worked. Thanks @GeorgeGayno-NOAA


cat <<EOF > $readme_name
The following # was used
https://github.com/sanatcumar/UFS_UTILS/tree/single_step
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

References to your github branch won't work after the merge.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just thought of something. The logic in ./reg_tests/get_hash.sh is used to get the github commit hash. Could the hash be added as part of the readme file?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That worked great!

@GeorgeGayno-NOAA
Copy link
Collaborator

How are you pointing to the new fixed directory without updating ./fix/link_fixdirs.sh

@sanatcumar
Copy link
Collaborator Author

How are you pointing to the new fixed directory without updating ./fix/link_fixdirs.sh

I linked just the orog . I updated and and testing it now

@GeorgeGayno-NOAA
Copy link
Collaborator

Since the name of the fix_sfc subdirectory was changed to sfc, I had to make a minor change the grid_gen regression test scripts. Find them in /scratch1/NCEPDEV/da/George.Gayno/ufs_utils.git/UFS_UTILS.sanath/reg_tests/grid_gen

  • c96.uniform.sh
  • c96.viirs.bnu.sh
  • esg.regional.pct.cat.sh
  • esg.regional.sh
  • gfdl.regional.sh
  • regional.gsl.gwd.sh

@GeorgeGayno-NOAA
Copy link
Collaborator

Pointing to the 20231027 version of the 'fix' files results in some UFS_UTILS utility programs to break. These will be fixed under #867. However, one small fix that I would like to include in this PR is: /scratch1/NCEPDEV/da/George.Gayno/ufs_utils.git/UFS_UTILS.sanath/util/sfc_climo_gen/sfc_gen.sh

Thanks. I think this is the last change before we finally merge.

@GeorgeGayno-NOAA
Copy link
Collaborator

The regression tests were run on all Tier 1 machines using d0d1e8f. Only the grid_gen tests failed. This result was expected. The 'grid' and 'sfc climo' files were identical to the baseline. But the 'oro' showed differences. These differences were the result of updates to the orog code. (@sanatcumar can explain in more detail). The tile1 differences from the c96.uniform tests were:

Variable  Group Count      Sum  AbsSum         Min        Max   Range     Mean  StdDev
orog_raw  /       353  58512.3 58512.3 -0.00636026    979.795 979.801  165.757 167.988
orog_filt /      1008  58437.8 61182.7    -93.6286    978.194 1071.82   57.974 122.876
elvmax    /       352 -58088.7 58088.7    -979.795 0.00636101 979.801 -165.025 168.336

Other grid_gen tests showed similar differences.

@sanatcumar
Copy link
Collaborator Author

The regression tests were run on all Tier 1 machines using d0d1e8f. Only the grid_gen tests failed. This result was expected. The 'grid' and 'sfc climo' files were identical to the baseline. But the 'oro' showed differences. These differences were the result of updates to the orog code. (@sanatcumar can explain in more detail). The tile1 differences from the c96.uniform tests were:

Variable  Group Count      Sum  AbsSum         Min        Max   Range     Mean  StdDev
orog_raw  /       353  58512.3 58512.3 -0.00636026    979.795 979.801  165.757 167.988
orog_filt /      1008  58437.8 61182.7    -93.6286    978.194 1071.82   57.974 122.876
elvmax    /       352 -58088.7 58088.7    -979.795 0.00636101 979.801 -165.025 168.336

Other grid_gen tests showed similar differences.

Orography is set to zero when lake fraction and slmask is zero in the original codes. This causes some fractional land points near oceans to have zero orography with the inclusion of newer codes to generate atmosphere and ocean coupled grids. To avoid this error new logic was added to sorc/orog_mask_tools.fd/orog.fd/mtnlm7_oclsm.F to set orography to zero only when both lake and land frac is zero. Small differences are expected. @barlage please chime in if I have missed anything.

@GeorgeGayno-NOAA
Copy link
Collaborator

@sanatcumar - I have no more comments. Unless you object, I will merge tomorrow.

@sanatcumar
Copy link
Collaborator Author

@sanatcumar - I have no more comments. Unless you object, I will merge tomorrow.

I do not have any further comments/ suggestions. Thanks George

@GeorgeGayno-NOAA GeorgeGayno-NOAA merged commit a757eb0 into ufs-community:develop Dec 12, 2023
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Create oro from scratch for coupled model
3 participants