Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Point to new input orography directory. #758

Merged

Conversation

GeorgeGayno-NOAA
Copy link
Collaborator

DESCRIPTION OF CHANGES:

The 'fix' data required by the grid generation programs is moving to a new directory. This requires some minor script updates.

TESTS CONDUCTED:

The new 'fix' directory structure was created under my own space of Cactus. The grid generation driver script was successfully run. And all repository consistency tests were successfully run. See the comments for more details.

DEPENDENCIES:

Coordinate with @KateFriedman-NOAA, who maintains the 'fix' directories.

DOCUMENTATION:

N/A

ISSUE:

Fixes #757.

@GeorgeGayno-NOAA GeorgeGayno-NOAA self-assigned this Jan 18, 2023
@GeorgeGayno-NOAA
Copy link
Collaborator Author

On Cactus, before running the consistency tests, I set up the new 'fix' directory structure as follows:

lrwxrwxrwx 1 george.gayno global   54 Jan 18 14:11 am -> /lfs/h2/emc/global/save/emc.global/FIX/fix/am/20220805
-rwxr-xr-x 1 george.gayno global 1750 Jan 18 14:09 link_fixdirs.sh
drwxr-sr-x 2 george.gayno global 4096 Jan 18 14:29 orog
lrwxrwxrwx 1 george.gayno global   10 Jan 18 20:02 orog_raw -> ./raw/orog
drwxr-sr-x 3 george.gayno global 4096 Jan 18 14:13 raw
lrwxrwxrwx 1 george.gayno global   61 Jan 18 14:11 sfc_climo -> /lfs/h2/emc/global/save/emc.global/FIX/fix/sfc_climo/20221017

The ./orog directory (which no longer has the input files):

clogin05@ /lfs/h2/emc/global/noscrub/George.Gayno/ufs_utils.git/UFS_UTILS/fix\ $ ls -l orog
total 16
lrwxrwxrwx 1 george.gayno global 61 Jan 18 14:29 C192 -> /lfs/h2/emc/global/save/emc.global/FIX/fix/orog/20220805/C192
lrwxrwxrwx 1 george.gayno global 61 Jan 18 14:23 C384 -> /lfs/h2/emc/global/save/emc.global/FIX/fix/orog/20220805/C384
lrwxrwxrwx 1 george.gayno global 61 Jan 18 14:18 C768 -> /lfs/h2/emc/global/save/emc.global/FIX/fix/orog/20220805/C768
lrwxrwxrwx 1 george.gayno global 60 Jan 18 14:18 C96 -> /lfs/h2/emc/global/save/emc.global/FIX/fix/orog/20220805/C96

The new ./orog_raw directory that contains the input files:

clogin05@ /lfs/h2/emc/global/noscrub/George.Gayno/ufs_utils.git/UFS_UTILS/fix\ $ ls -l ./raw/orog
total 28
lrwxrwxrwx 1 george.gayno global 89 Jan 18 14:15 geo_em.d01.lat-lon.2.5m.HGT_M.nc -> /lfs/h2/emc/global/save/emc.global/FIX/fix/orog/20220805/geo_em.d01.lat-lon.2.5m.HGT_M.nc
lrwxrwxrwx 1 george.gayno global 76 Jan 18 14:15 GlobalLakeDepth.dat -> /lfs/h2/emc/global/save/emc.global/FIX/fix/orog/20220805/GlobalLakeDepth.dat
lrwxrwxrwx 1 george.gayno global 77 Jan 18 14:16 GlobalLakeStatus.dat -> /lfs/h2/emc/global/save/emc.global/FIX/fix/orog/20220805/GlobalLakeStatus.dat
lrwxrwxrwx 1 george.gayno global 76 Jan 18 14:16 gmted2010.30sec.int -> /lfs/h2/emc/global/save/emc.global/FIX/fix/orog/20220805/gmted2010.30sec.int
lrwxrwxrwx 1 george.gayno global 97 Jan 18 14:16 HGT.Beljaars_filtered.lat-lon.30s_res.nc -> /lfs/h2/emc/global/save/emc.global/FIX/fix/orog/20220805/HGT.Beljaars_filtered.lat-lon.30s_res.nc
lrwxrwxrwx 1 george.gayno global 74 Jan 18 14:17 landcover30.fixed -> /lfs/h2/emc/global/save/emc.global/FIX/fix/orog/20220805/landcover30.fixed
lrwxrwxrwx 1 george.gayno global 88 Jan 18 14:17 thirty.second.antarctic.new.bin -> /lfs/h2/emc/global/save/emc.global/FIX/fix/orog/20220805/thirty.second.antarctic.new.bin

@GeorgeGayno-NOAA GeorgeGayno-NOAA marked this pull request as ready for review January 18, 2023 20:53
@@ -52,13 +52,17 @@ am_ver=${am_ver:-20220805}
orog_ver=${orog_ver:-20220805}
sfc_climo_ver=${sfc_climo_ver:-20221017}

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@KateFriedman-NOAA Will this change work with the workflow?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In terms of clone, build, link it works fine. Did not try running with your branch since we haven't moved forward with UFS_UTILS hash yet. Will put output from clone, build, link test on WCOSS2 into comment in PR.

@KateFriedman-NOAA
Copy link
Collaborator

Tested cloning, building, and linking global-workflow develop with branch from this PR. Worked fine:

kate.friedman@clogin04:/lfs/h2/emc/global/noscrub/kate.friedman/git/develop_ufs_utils_758/sorc> ./checkout.sh -g
Received -g flag for optional checkout of GSI-based DA
Performing checkout of gfs_utils
|-- Cloning from https://github.com/NOAA-EMC/gfs-utils into gfs_utils.fd
|-- Checking out 0b8ff56
|-- Updating submodules (if any)

Performing checkout of ufs_model
|-- Cloning from https://github.com/ufs-community/ufs-weather-model into ufs_model.fd
|-- Checking out 7a1ce44
|-- Updating submodules (if any)

Performing checkout of ufs_utils
|-- Cloning from https://github.com/GeorgeGayno-NOAA/UFS_UTILS.git into ufs_utils.fd
|-- Checking out feature/fix_orog
|-- Updating submodules (if any)

Performing checkout of verif-global
|-- Cloning from https://github.com/NOAA-EMC/EMC_verif-global.git into verif-global.fd
|-- Checking out c267780
|-- Updating submodules (if any)

Performing checkout of gsi_enkf
|-- Cloning from https://github.com/NOAA-EMC/GSI.git into gsi_enkf.fd
|-- Checking out 48d8676

Performing checkout of gsi_utils
|-- Cloning from https://github.com/NOAA-EMC/GSI-Utils.git into gsi_utils.fd
|-- Checking out 322cc7b
|-- Updating submodules (if any)

Performing checkout of gsi_monitor
|-- Cloning from https://github.com/NOAA-EMC/GSI-Monitor.git into gsi_monitor.fd
|-- Checking out c64cc47
|-- Updating submodules (if any)

Performing checkout of gldas
|-- Cloning from https://github.com/NOAA-EMC/GLDAS.git into gldas.fd
|-- Checking out fd8ba62
|-- Updating submodules (if any)


kate.friedman@clogin04:/lfs/h2/emc/global/noscrub/kate.friedman/git/develop_ufs_utils_758/sorc> ./build_all.sh 
Creating ../exec folder
Resetting modules to system default. Reseting $MODULEPATH back to system default. All extra directories will be removed from $MODULEPATH.
 .... Building gfs_utils .... 
 .... Building WW3 pre and post execs .... 
 .... Building forecast model .... 
 .... Building gsi and enkf .... 
 .... Building gsi utilities .... 
 .... Skip building GDASApp  .... 
 .... Building gsi monitor .... 
 .... Building UPP .... 
 .... Building ufs_utils .... 
 .... Building gldas .... 

 .... Build system finished .... 
kate.friedman@clogin04:/lfs/h2/emc/global/noscrub/kate.friedman/git/develop_ufs_utils_758/sorc> ./link_workflow.sh 
./link_workflow.sh completed successfully
kate.friedman@clogin04:/lfs/h2/emc/global/noscrub/kate.friedman/git/develop_ufs_utils_758/sorc> ll ufs_utils.fd/fix/
total 8
lrwxrwxrwx 1 kate.friedman global   54 Jan 19 15:54 am -> /lfs/h2/emc/global/save/emc.global/FIX/fix/am/20220805
-rwxr-xr-x 1 kate.friedman global 1750 Jan 19 15:30 link_fixdirs.sh
lrwxrwxrwx 1 kate.friedman global   56 Jan 19 15:54 orog -> /lfs/h2/emc/global/save/emc.global/FIX/fix/orog/20220805
lrwxrwxrwx 1 kate.friedman global   51 Jan 19 15:54 orog_raw -> /lfs/h2/emc/global/save/emc.global/FIX/fix/raw/orog
lrwxrwxrwx 1 kate.friedman global   61 Jan 19 15:54 sfc_climo -> /lfs/h2/emc/global/save/emc.global/FIX/fix/sfc_climo/20220805

Copy link
Collaborator

@KateFriedman-NOAA KateFriedman-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Will run with these updates when we move global-workflow develop to a newer hash and report issues but changes don't look like they will cause any issues for GFS at runtime. Cloning, building, linking within global-workflow develop and this PR branch worked fine.

@GeorgeGayno-NOAA
Copy link
Collaborator Author

@JeffBeck-NOAA or @LarissaReames-NOAA can one of you please approve so I can merge?

@GeorgeGayno-NOAA
Copy link
Collaborator Author

@KateFriedman-NOAA I will merge this afternoon if you can update the links. (Orion is down today).

@KateFriedman-NOAA
Copy link
Collaborator

@KateFriedman-NOAA I will merge this afternoon if you can update the links. (Orion is down today).

@GeorgeGayno-NOAA Links from orog/20220805 to the new raw/orog are in place on all supported platforms. Cactus example:

emc.global@clogin02:/lfs/h2/emc/global/noscrub/emc.global/FIX/fix> ll orog/20220805/ | grep raw
lrwxrwxrwx 1 emc.global global   47 Jan 18 15:50 geo_em.d01.lat-lon.2.5m.HGT_M.nc -> ../../raw/orog/geo_em.d01.lat-lon.2.5m.HGT_M.nc
lrwxrwxrwx 1 emc.global global   34 Jan 18 15:51 GlobalLakeDepth.dat -> ../../raw/orog/GlobalLakeDepth.dat
lrwxrwxrwx 1 emc.global global   35 Jan 18 15:51 GlobalLakeStatus.dat -> ../../raw/orog/GlobalLakeStatus.dat
lrwxrwxrwx 1 emc.global global   34 Jan 18 15:51 gmted2010.30sec.int -> ../../raw/orog/gmted2010.30sec.int
lrwxrwxrwx 1 emc.global global   55 Jan 18 15:51 HGT.Beljaars_filtered.lat-lon.30s_res.nc -> ../../raw/orog/HGT.Beljaars_filtered.lat-lon.30s_res.nc
lrwxrwxrwx 1 emc.global global   32 Jan 18 15:51 landcover30.fixed -> ../../raw/orog/landcover30.fixed
lrwxrwxrwx 1 emc.global global   46 Jan 18 15:51 thirty.second.antarctic.new.bin -> ../../raw/orog/thirty.second.antarctic.new.bin
emc.global@clogin02:/lfs/h2/emc/global/noscrub/emc.global/FIX/fix> ll raw/orog/
total 31028236
-rw-r--r-- 1 emc.global global      616356 Mar 11  2017 clmgrb
-rw-r--r-- 1 emc.global global        5538 Mar 11  2017 clmgrb.index
-rw-r--r-- 1 emc.global global        6836 Aug 14  2017 convert.f90
-rw-r--r-- 1 emc.global global   447900364 Feb  6  2021 geo_em.d01.lat-lon.2.5m.HGT_M.nc
-rw-r--r-- 1 emc.global global  1866240000 May 31  2018 GlobalLakeDepth.dat
-rw-r--r-- 1 emc.global global  1866240000 Jul 11  2022 GlobalLakeDepth_GLDBv3release.dat
-rw-r--r-- 1 emc.global global  1866240000 Jul 11  2022 GlobalLakeDepth_GLOBathy.dat
-rw-r--r-- 1 emc.global global        2163 May 31  2018 GlobalLakeDepth.txt
-rw-r--r-- 1 emc.global global   933120000 May 31  2018 GlobalLakeStatus.dat
-rw-r--r-- 1 emc.global global   933120000 Jul 11  2022 GlobalLakeStatus_GLDBv3release.dat
-rw-r--r-- 1 emc.global global   933120000 Jul 11  2022 GlobalLakeStatus_MODISp.dat
-rw-r--r-- 1 emc.global global        2805 May 31  2018 GlobalLakeStatus.txt
-rw-r--r-- 1 emc.global global   933120000 Jul 11  2022 GlobalLakeStatus_VIIRS.dat
-rw-r--r-- 1 emc.global global  3608064000 Feb 18  2014 gmted2010.30sec.flt
-rw-r--r-- 1 emc.global global         263 May  5  2014 gmted2010.30sec.flt.ctl
-rw-r--r-- 1 emc.global global  1866240000 Aug 14  2017 gmted2010.30sec.int
-rw-r--r-- 1 emc.global global         264 May  5  2014 gmted2010.30sec.int.ctl
-rw-r--r-- 1 emc.global global  1866240008 Mar 11  2017 gtopo30_gg.fine
-rw-r--r-- 1 emc.global global  1866240000 Aug 16  2017 gtopo30_gg.fine.nh
-rw-r--r-- 1 emc.global global 11197441020 Jan 14  2021 HGT.Beljaars_filtered.lat-lon.30s_res.nc
-rw-r--r-- 1 emc.global global   933120000 Mar 11  2017 landcover30.fixed
-rw-r--r-- 1 emc.global global         240 May  5  2014 makefile
-rw-r--r-- 1 emc.global global         313 Aug 14  2017 run.lsf
-rw-r--r-- 1 emc.global global   622267212 Mar 11  2017 thirty.second.antarctic.new.bin
-rw-r--r-- 1 emc.global global    14762250 Mar 11  2017 TOP8M_avg.20I4.asc
-rw-r--r-- 1 emc.global global    14762250 Mar 11  2017 TOP8M_max.20I4.asc
-rw-r--r-- 1 emc.global global     3690563 Mar 11  2017 TOP8M_slm.80I1.asc

Let me know if you need additional ones.

Note, I moved the fix folder on both WCOSS2s a while back from save to noscrub. I have a symlink to redirect from save to noscrub still. You can update to the noscrub folder now if you wish, the fix set should live there moving forward now. Apologies if I failed to notify you of this move when I did it!

This /lfs/h2/emc/global/save/emc.global/FIX/fix points to this on WCOSS2 /lfs/h2/emc/global/noscrub/emc.global/FIX/fix.

@GeorgeGayno-NOAA
Copy link
Collaborator Author

Note, I moved the fix folder on both WCOSS2s a while back from save to noscrub. I have a symlink to redirect from save to noscrub still. You can update to the noscrub folder now if you wish, the fix set should live there moving forward now. Apologies if I failed to notify you of this move when I did it!

This /lfs/h2/emc/global/save/emc.global/FIX/fix points to this on WCOSS2 /lfs/h2/emc/global/noscrub/emc.global/FIX/fix.

Done at 638d14b.

@GeorgeGayno-NOAA GeorgeGayno-NOAA merged commit 56123a8 into ufs-community:develop Jan 25, 2023
@GeorgeGayno-NOAA GeorgeGayno-NOAA deleted the feature/fix_orog branch January 25, 2023 20:24
AnnetteGibbs-NOAA added a commit to AnnetteGibbs-NOAA/UFS_UTILS that referenced this pull request Jul 3, 2023
* origin/develop:
  Update ccpp submodule (ufs-community#827)
  Change length of character variable "mosaic_name" in "make_solo_mosaic.c" (ufs-community#824)
  Update GDAS INIT utility for GFS COM reorganization (ufs-community#820)
  Add backup calculation for orography and mask generation (ufs-community#713)
  Improve repository build script (ufs-community#819)
  Use latest wgrib2 executable on Jet. (ufs-community#816)
  chgres_cube - Minor update for high-resolution grids . (ufs-community#814)
  Update PR template (ufs-community#813)
  global_cycle - Add soil moisture nudging for NoahMP option (ufs-community#809)
  Release v1.10.0 (ufs-community#807)
  Run sfc_climo_gen utility on Hera (ufs-community#791)
  Fix bug in chgres_cube subroutine search_many (ufs-community#808)
  sfc_climo_gen - Output fractions of each vegetation/soil type category (ufs-community#748)
  Replace w3nco library with w3emc (ufs-community#802)
  Fix Debug compilation (ufs-community#800)
  Update WCOSS2 build module to use new ESMF v8.4.1 library. (ufs-community#799)
  Update chgres_cube documentation for duplicate grib2 records (ufs-community#795)
  Use latest Cray modules on WCOSS2 (ufs-community#792)
  Remove checksum attribute from global_cycle restart files (ufs-community#794)
  Remove goto statements from chgres_cube (ufs-community#775)
  FVCOM_TOOLS - Remove dependency on module_nwp_base.f90 (ufs-community#790)
  Remove "gcovr" step from the 'developer' workflow (ufs-community#785)
  Update build module for Cheyenne (ufs-community#783)
  GDAS Init utility - consolidate the copy of coldstart files (ufs-community#773)
  Remove support for ODIN machine (ufs-community#782)
  Update GDAS INIT utility to optionally use fracoro data (ufs-community#741)
  Option to build only application specific utilities (e.g. GFS) (ufs-community#777)
  Use combined IP/IP2 library (ufs-community#695)
  Initial updates to global_cycle for Noah-MP land model (ufs-community#774)
  New resolution options for the cpld_gridgen utility (ufs-community#769)
  Use new EPIC-maintained hpc-stack on Jet  (ufs-community#771)
  Update GDAS initialization scripts for the new ENKF directory (ufs-community#764)
  Reduce memory usage in chgres_cube (ufs-community#766)
  Detect duplicate vertical levels in chgres_cube. (ufs-community#767)
  Point to new input orography directory. (ufs-community#758)
  Release version 1.9 (ufs-community#754)
  HAFSv1 grid nesting updates (ufs-community#752)
  Remove compiler warnings from chgres_cube (ufs-community#747)
  chgres_cube: Split input_data module into to 3 separate modules (ufs-community#744)
  More work on CI, checking with different versions of ESMF (ufs-community#742)
  Add processing of soil color to sfc_climo_gen (ufs-community#721)
  more work on CI - added Linux_versions workflow (ufs-community#739)
  Add TEST_FILE_DIR option to CMake build, where test data files can be found instead of using FTP.  (ufs-community#732)
  adding developer workflow (ufs-community#724)
  Update GDAS INIT utility for v16.3. (ufs-community#723)
  OROG_GSL - Remove negative bias in orographic asymmetery (OA) fields (ufs-community#718)
  Incorporate BNU soil type data (ufs-community#717)
  High-resolution MODIS and STATSGO veg/soil data. (ufs-community#703)
  Run WCOSS2 consistency tests under role account (ufs-community#711)
  Fix macOS CI workflow (ufs-community#715)
  Add utility codes to create BNU soil texture data (ufs-community#707)
  Update chgres_cube to output netcdf4 file. (ufs-community#704)
  Incorporate high-res global VIIRS vegetation data. (ufs-community#699)
  Run grid_gen consistency tests in parallel. (ufs-community#697)
  Fix error handling in "link_fixdirs.sh". Add -L to the copy command. (ufs-community#701)
  Update for new fixed data directory structure (ufs-community#688)
  Add WCOSS2 support for tests (ufs-community#693)
  Increase test data pull timeout (ufs-community#692)
  Option to install binaries to any directory. (ufs-community#685)
  Update global_cycle to use the latest CCPP version of sfcsub.F (ufs-community#671)
  Incorporate weight_gen program. (ufs-community#677)
  Add install of jpeg-turbo to macos workflows (ufs-community#684)
  Improve logic in regression test driver script (ufs-community#681)
  Release v1.8 (ufs-community#679)
  Update link_fixdirs.sh for new data directories (ufs-community#672)
  Update default ice climatology in ./ush/global_cycle.sh. (ufs-community#664)
  Remove all references to WCOSS 1 from UFS_UTILS (ufs-community#667)
  Update workflows to use latest macOS and ubuntu (ufs-community#675)
  Update GDAS Initialization utility for recent HPSS tarball name change (ufs-community#666)
  Port UFS_UTILS to WCOSS2 (ufs-community#642)
  Fix chgres_cube to process GEFS GRIB2 data (ufs-community#658)
  Update global_cycle_driver.sh for GFS OPS directory convention  (ufs-community#655)
  Update documentation for SRW App (ufs-community#656)
  New coupled model utility (ufs-community#647)
  Update support for S4 and enable regression testing (ufs-community#654)
  global_cycle - Link to CCPP version of sfcsub.F (ufs-community#636)
  Add processing of new global AFWA snow data to emcsfc_snow2mdl. (ufs-community#648)
  Update build module on Cheyenne (ufs-community#646)
  Move to Intel 2022 on Jet, Hera and Orion (ufs-community#650)
  Host doxygen documentation for multiple releases (ufs-community#644)
  Download unit test data as part of the CMake build (ufs-community#630)
  chgres_cube - Complete removal of wgrib2 library (ufs-community#641)
  Eliminate circular dependency in chgres_cube
  Update workflow files to pull netcdf-c library from GitHub
  chgres_cube - Remove the wgrib2 library from the GRIB2 data read routines.
  Undefined symbols on macOS with Intel compiler (ufs-community#628)
  Update FVCOM code to handle sub-domain restart files using multiple cores. (ufs-community#624)
  chgres_cube - Run routine 'convert_omega' on all tasks. (ufs-community#627)
  Use ESMF 8.2.0 library
  Automate update of consistency test baseline data. (ufs-community#603)
  Update workflow files to use newer versions of ESMF and NCEPLIBS. (ufs-community#617)
  Update build modules to be lua compliant (ufs-community#614)
  Allow FVCOM tools to Update Ice Surface Roughness Length (ufs-community#604)
  Update the requested memory in the Orion chgres_cube consistency test script (ufs-community#611)
  Use copy of grib_util under Jet role account. (ufs-community#608)
  Run consistency tests on Orion using role account (ufs-community#606)
  Run consistency tests on Hera using role account (ufs-community#605)
  Run consistency tests on Jet using role account. (ufs-community#607)
  Update more documentation after move to ufs-community (ufs-community#597)
  fvcom_tools - Add option to process 'cold' or 'warm' restart files (ufs-community#595)
  Update documentation after move to ufs-community (ufs-community#594)
  chgres_cube - Eliminate segmentation fault in input_data.F90 (ufs-community#585)
  Update to language of unit test README to match that in unit test.
  chgres_cube - Simplify surface processing using field bundles (ufs-community#572)
  Add compiler flags for GNU Fortran v10 or newer compilers. (ufs-community#583)
  Move verbose output from example unit test to be commented to streamline test output.
  Updates to test README to add instructions for use of example unit test.
  Add Findwgrib2.cmake (ufs-community#578)
  Added unit test to be used as an instructional example for new users.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Point to new location for input orography data
3 participants