Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Time selection for fvcom_tools and unit test #595

Merged
merged 20 commits into from
Nov 5, 2021
Merged

Time selection for fvcom_tools and unit test #595

merged 20 commits into from
Nov 5, 2021

Conversation

dmwright526
Copy link
Collaborator

DESCRIPTION OF CHANGES:

Updates have been made to fvcom_tools to allow for the read in of either a cold start surface file from chgres_cube or a warm start surface file generated from a restart of the UFS-SRW. Updates have also been made to allow for the use of fvcom data files with multiple time indices. This requires two additional command line inputs:

  • "warm" or "cold" flag. "warm" will allow the tool to read in files generated from a restart. "cold" will read output files from chgres_cube.
  • String of time slice to be used in fvcom data file. For example, '2020-01-31T18:00:00.000000'

TESTS CONDUCTED:

Tests were conducted on Jet and Hera.

A unit test has also been added, which passes on Jet and Hera. It passes the "Intel" test on Github, but fails the "linux-mac-mpi-nceplibs", "netcdf-versions", and "debug-docs-test_coverage" for unknown reasons.

DEPENDENCIES:

None

DOCUMENTATION:

Updates have been made to fvcom_readme.md to include the required input variables.

ISSUE:

Fixing issues addressed in: #586

CONTRIBUTORS (optional):

@JeffBeck-NOAA

@dmwright526
Add command line value to select specific time from FVCOM files
074706b
@dmwright526
Merge branch 'NOAA-EMC:develop' into feature/TimeSelectingFVCOM
050752b
@dmwright526
Add ability to read restart files and choose specific time in FVCOM f…
a868ad0 
…iles
@dmwright526
Subroutine documentation/comments and updated error messaging
ab52eb6
@dmwright526
Merge branch 'NOAA-EMC:develop' into feature/TimeSelectingFVCOM
6e7d80b
@dmwright526
Add comments to finish_nwp to fix documentation build errors
07910f8
@dmwright526
Merge branch 'feature/TimeSelectingFVCOM' of https://github.com/dmwri…
40db8ee 
…ght526/UFS_UTILS into feature/TimeSelectingFVCOM

Updating to keep up with develop branch
@dmwright526
Removed wrong variable from get_time_ind_nwp documentation
f0e6caf
@dmwright526
Add documentation of wcstart and sfctl to read_nwp()
fff8d88
@dmwright526
Update docs to include new input variables
ba84ba7
@dmwright526
Merge branch 'ufs-community:develop' into feature/TimeSelectingFVCOM
4f3bb78
@dmwright526
Add unit test for fvcom_tools
2b6be39
@dmwright526
Merge branch 'feature/TimeSelectingFVCOM' of https://github.com/dmwri…
bd67390 
…ght526/UFS_UTILS into feature/TimeSelectingFVCOM
@dmwright526
Add LSanSuppress file
bd834c3
@dmwright526
Add LSanSuppress.supp to tests/fvcom_tools/CMakeFile.txt
17ca369
@dmwright526
Change to character length of FVCOM to fix compiler bug
aa6f91f
@dmwright526
Change to fvcom_tools unit test to try to pass netcdf build, matrix-b…
b593086 
…uild, and debug-doc tests
@dmwright526
TRY 2 Change to fvcom_tools unit test to try to pass netcdf build, ma…
1507f4a 
…trix-build, and debug-doc tests
@dmwright526 dmwright526 mentioned this pull request Oct 27, 2021
Merged
@LarissaReames-NOAA
Copy link
Collaborator

I was able to reproduce the error that the Github actions are experiencing by using the gnu compiler and the Cmake debug build type. I took a quick glance at the data type where the error is occurring (fcst_nwp) and there are a bunch of pointers in there, and the error is an invalid memory reference error, so I'm guessing that data type needs to be looked at more closely.

@JeffBeck-NOAA
Copy link
Collaborator

@GeorgeGayno-NOAA, @dmwright526 has two small NetCDF files that are used for the unit tests in this PR. Since we don't want to store any binary files in the repo, can we get these added to the EMC FTP server? If we can, how can David modify his unit test to download those files? Thanks.

@GeorgeGayno-NOAA
Copy link
Collaborator

@GeorgeGayno-NOAA, @dmwright526 has two small NetCDF files that are used for the unit tests in this PR. Since we don't want to store any binary files in the repo, can we get these added to the EMC FTP server? If we can, how can David modify his unit test to download those files? Thanks.

I can ask @KateFriedman-NOAA to host any files for us.

@GeorgeGayno-NOAA
Copy link
Collaborator

@GeorgeGayno-NOAA, @dmwright526 has two small NetCDF files that are used for the unit tests in this PR. Since we don't want to store any binary files in the repo, can we get these added to the EMC FTP server? If we can, how can David modify his unit test to download those files? Thanks.

To modify his unit test to download data from the server, he can follow this example:

First, add a 'files.txt' file, such as:
https://github.com/ufs-community/UFS_UTILS/blob/develop/tests/chgres_cube/data/files.txt

Then, invoke the file in the CMakeLists.txt file as follows:
https://github.com/ufs-community/UFS_UTILS/blob/develop/tests/chgres_cube/CMakeLists.txt#L19

@dmwright526
Copy link
Collaborator Author

Thanks @GeorgeGayno-NOAA . The files are currently located in the data folder for the test that need to be staged. I will go ahead and work on modify the code to download the data. Once that data is staged I will remove it from my fork.

Thanks @LarissaReames-NOAA for checking out the error. I will do some work on Cheyenne to see if I can fix the problem with the GNU compiler.

@KateFriedman-NOAA
Copy link
Collaborator

I can ask @KateFriedman-NOAA to host any files for us.

I'm happy to add files to our static "fix file" sets or push some files to the ftp server. The latter is not necessarily guaranteed to be official or long-term. The copy of the fix files on the ftp server is official and long-term though since we support those files on our NOAA HPC machines and host them on the ftp server for outside users. Do these two binary files qualify as fix-type files? See the fix file sets here on the ftp server for examples of what are contained within:

https://ftp.emc.ncep.noaa.gov/static_files/public/UFS/GFS/

The "fix" folder there is the developmental set. The other two "fix_nco_gfsv##" sets are the frozen fix sets from the GFSv15 and GFSv16 operational systems. I add the new CO2 files that we receive each fall to all sets when they arrive, including the frozen ops ones, but I otherwise leave the frozen sets frozen. Let me know what you need, thanks!

@GeorgeGayno-NOAA
Copy link
Collaborator

I can ask @KateFriedman-NOAA to host any files for us.

I'm happy to add files to our static "fix file" sets or push some files to the ftp server. The latter is not necessarily guaranteed to be official or long-term. The copy of the fix files on the ftp server is official and long-term though since we support those files on our NOAA HPC machines and host them on the ftp server for outside users. Do these two binary files qualify as fix-type files? See the fix file sets here on the ftp server for examples of what are contained within:

https://ftp.emc.ncep.noaa.gov/static_files/public/UFS/GFS/

The "fix" folder there is the developmental set. The other two "fix_nco_gfsv##" sets are the frozen fix sets from the GFSv15 and GFSv16 operational systems. I add the new CO2 files that we receive each fall to all sets when they arrive, including the frozen ops ones, but I otherwise leave the frozen sets frozen. Let me know what you need, thanks!

@KateFriedman-NOAA These files are only used by our Github unit tests. They would go somewhere here:
https://ftp.emc.ncep.noaa.gov/static_files/public/UFS/ufs_utils/unit_tests/ They don't need to be hosted on our supported machines.

@KateFriedman-NOAA
Copy link
Collaborator

@KateFriedman-NOAA These files are only used by our Github unit tests. They would go somewhere here: https://ftp.emc.ncep.noaa.gov/static_files/public/UFS/ufs_utils/unit_tests/ They don't need to be hosted on our supported machines.

Okie dokie, please point me to the files for pickup and I'll put them in that location on the ftp server. Thanks!

@dmwright526
Copy link
Collaborator Author

Thanks for the help, @KateFriedman-NOAA! The two netcdf files (fvcom_unittest.nc, sfcdata_unittest.nc) can be found here:

https://github.com/dmwright526/UFS_UTILS/tree/feature/TimeSelectingFVCOM/tests/fvcom_tools/data

@KateFriedman-NOAA
Copy link
Collaborator

@dmwright526 Done, see the files here:
https://ftp.emc.ncep.noaa.gov/static_files/public/UFS/ufs_utils/unit_tests/fvcom_unittest.nc
https://ftp.emc.ncep.noaa.gov/static_files/public/UFS/ufs_utils/unit_tests/sfcdata_unittest.nc

Let me know if you want a subfolder at that level to hold those two files, similar to the other two folders there. Thanks!

@GeorgeGayno-NOAA
Copy link
Collaborator

@dmwright526 Done, see the files here: https://ftp.emc.ncep.noaa.gov/static_files/public/UFS/ufs_utils/unit_tests/fvcom_unittest.nc https://ftp.emc.ncep.noaa.gov/static_files/public/UFS/ufs_utils/unit_tests/sfcdata_unittest.nc

Let me know if you want a subfolder at that level to hold those two files, similar to the other two folders there. Thanks!

@KateFriedman-NOAA Yes, please use sub-folders. The number of unit tests is increasing rapidly.

@KateFriedman-NOAA
Copy link
Collaborator

Yes, please use sub-folders. The number of unit tests is increasing rapidly.

Okie dokie...one for each file?

/fvcom/fvcom_unittest.nc
/sfcdata/sfcdata_unittest.nc

...or together and if together, what folder name would you like?

@GeorgeGayno-NOAA
Copy link
Collaborator

Yes, please use sub-folders. The number of unit tests is increasing rapidly.

Okie dokie...one for each file?

/fvcom/fvcom_unittest.nc
/sfcdata/sfcdata_unittest.nc

...or together and if together, what folder name would you like?

I would place both files in a subdirectory called fvcom_tools.

@KateFriedman-NOAA
Copy link
Collaborator

I would place both files in a subdirectory called fvcom_tools.

Done, see the new folder with the two files here:
https://ftp.emc.ncep.noaa.gov/static_files/public/UFS/ufs_utils/unit_tests/fvcom_tools/

Let me know if you need anything else related to this, thanks!

@GeorgeGayno-NOAA
Copy link
Collaborator

I was able to reproduce the error that the Github actions are experiencing by using the gnu compiler and the Cmake debug build type. I took a quick glance at the data type where the error is occurring (fcst_nwp) and there are a bunch of pointers in there, and the error is an invalid memory reference error, so I'm guessing that data type needs to be looked at more closely.

I was also able to reproduce the error on Hera. I was hoping the debug flags to point to the problem. But no such luck.

@dmwright526 and @JeffBeck-NOAA Do you still want to merge, then fix the GNU test later?

@JeffBeck-NOAA
Copy link
Collaborator

I was able to reproduce the error that the Github actions are experiencing by using the gnu compiler and the Cmake debug build type. I took a quick glance at the data type where the error is occurring (fcst_nwp) and there are a bunch of pointers in there, and the error is an invalid memory reference error, so I'm guessing that data type needs to be looked at more closely.

I was also able to reproduce the error on Hera. I was hoping the debug flags to point to the problem. But no such luck.

@dmwright526 and @JeffBeck-NOAA Do you still want to merge, then fix the GNU test later?

@GeorgeGayno-NOAA, due to the time-sensitive nature of this work, yes, we'd like to merge, then address the GNU unit test issue. The code changes in this PR are only related to the FVCOM tools and will impact only those people running that code (GSL and GLERL). @dmwright526 will create an issue to follow-up with the GNU unit test failure, and that will be fixed in the near future.

GeorgeGayno-NOAA
GeorgeGayno-NOAA reviewed Nov 4, 2021
View reviewed changes
sorc/fvcom_tools.fd/fvcom_readme.md Outdated
@@ -19,16 +19,23 @@
Installation depends on the netCDF library and cmake.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are placing all the 'readme' files in the ./docs directory. So rename this file as 'user_guide.md' and replace the version in ./docs.

@GeorgeGayno-NOAA
Copy link
Collaborator

@dmwright526 I will merge. But be sure to open an issue to fix the GNU test.

@GeorgeGayno-NOAA GeorgeGayno-NOAA merged commit 5f34ba1 into ufs-community:develop Nov 5, 2021
@JeffBeck-NOAA
Copy link
Collaborator

Thank you @GeorgeGayno-NOAA. @dmwright526 has opened an issue for the gnu unit test here.

AnnetteGibbs-NOAA added a commit to AnnetteGibbs-NOAA/UFS_UTILS that referenced this pull request Jul 3, 2023
@AnnetteGibbs-NOAA
Merge remote-tracking branch 'origin/develop' into feature/dyn_dng
2286a16 
* origin/develop:
  Update ccpp submodule (ufs-community#827)
  Change length of character variable "mosaic_name" in "make_solo_mosaic.c" (ufs-community#824)
  Update GDAS INIT utility for GFS COM reorganization (ufs-community#820)
  Add backup calculation for orography and mask generation (ufs-community#713)
  Improve repository build script (ufs-community#819)
  Use latest wgrib2 executable on Jet. (ufs-community#816)
  chgres_cube - Minor update for high-resolution grids . (ufs-community#814)
  Update PR template (ufs-community#813)
  global_cycle - Add soil moisture nudging for NoahMP option (ufs-community#809)
  Release v1.10.0 (ufs-community#807)
  Run sfc_climo_gen utility on Hera (ufs-community#791)
  Fix bug in chgres_cube subroutine search_many (ufs-community#808)
  sfc_climo_gen - Output fractions of each vegetation/soil type category (ufs-community#748)
  Replace w3nco library with w3emc (ufs-community#802)
  Fix Debug compilation (ufs-community#800)
  Update WCOSS2 build module to use new ESMF v8.4.1 library. (ufs-community#799)
  Update chgres_cube documentation for duplicate grib2 records (ufs-community#795)
  Use latest Cray modules on WCOSS2 (ufs-community#792)
  Remove checksum attribute from global_cycle restart files (ufs-community#794)
  Remove goto statements from chgres_cube (ufs-community#775)
  FVCOM_TOOLS - Remove dependency on module_nwp_base.f90 (ufs-community#790)
  Remove "gcovr" step from the 'developer' workflow (ufs-community#785)
  Update build module for Cheyenne (ufs-community#783)
  GDAS Init utility - consolidate the copy of coldstart files (ufs-community#773)
  Remove support for ODIN machine (ufs-community#782)
  Update GDAS INIT utility to optionally use fracoro data (ufs-community#741)
  Option to build only application specific utilities (e.g. GFS) (ufs-community#777)
  Use combined IP/IP2 library (ufs-community#695)
  Initial updates to global_cycle for Noah-MP land model (ufs-community#774)
  New resolution options for the cpld_gridgen utility (ufs-community#769)
  Use new EPIC-maintained hpc-stack on Jet  (ufs-community#771)
  Update GDAS initialization scripts for the new ENKF directory (ufs-community#764)
  Reduce memory usage in chgres_cube (ufs-community#766)
  Detect duplicate vertical levels in chgres_cube. (ufs-community#767)
  Point to new input orography directory. (ufs-community#758)
  Release version 1.9 (ufs-community#754)
  HAFSv1 grid nesting updates (ufs-community#752)
  Remove compiler warnings from chgres_cube (ufs-community#747)
  chgres_cube: Split input_data module into to 3 separate modules (ufs-community#744)
  More work on CI, checking with different versions of ESMF (ufs-community#742)
  Add processing of soil color to sfc_climo_gen (ufs-community#721)
  more work on CI - added Linux_versions workflow (ufs-community#739)
  Add TEST_FILE_DIR option to CMake build, where test data files can be found instead of using FTP.  (ufs-community#732)
  adding developer workflow (ufs-community#724)
  Update GDAS INIT utility for v16.3. (ufs-community#723)
  OROG_GSL - Remove negative bias in orographic asymmetery (OA) fields (ufs-community#718)
  Incorporate BNU soil type data (ufs-community#717)
  High-resolution MODIS and STATSGO veg/soil data. (ufs-community#703)
  Run WCOSS2 consistency tests under role account (ufs-community#711)
  Fix macOS CI workflow (ufs-community#715)
  Add utility codes to create BNU soil texture data (ufs-community#707)
  Update chgres_cube to output netcdf4 file. (ufs-community#704)
  Incorporate high-res global VIIRS vegetation data. (ufs-community#699)
  Run grid_gen consistency tests in parallel. (ufs-community#697)
  Fix error handling in "link_fixdirs.sh". Add -L to the copy command. (ufs-community#701)
  Update for new fixed data directory structure (ufs-community#688)
  Add WCOSS2 support for tests (ufs-community#693)
  Increase test data pull timeout (ufs-community#692)
  Option to install binaries to any directory. (ufs-community#685)
  Update global_cycle to use the latest CCPP version of sfcsub.F (ufs-community#671)
  Incorporate weight_gen program. (ufs-community#677)
  Add install of jpeg-turbo to macos workflows (ufs-community#684)
  Improve logic in regression test driver script (ufs-community#681)
  Release v1.8 (ufs-community#679)
  Update link_fixdirs.sh for new data directories (ufs-community#672)
  Update default ice climatology in ./ush/global_cycle.sh. (ufs-community#664)
  Remove all references to WCOSS 1 from UFS_UTILS (ufs-community#667)
  Update workflows to use latest macOS and ubuntu (ufs-community#675)
  Update GDAS Initialization utility for recent HPSS tarball name change (ufs-community#666)
  Port UFS_UTILS to WCOSS2 (ufs-community#642)
  Fix chgres_cube to process GEFS GRIB2 data (ufs-community#658)
  Update global_cycle_driver.sh for GFS OPS directory convention  (ufs-community#655)
  Update documentation for SRW App (ufs-community#656)
  New coupled model utility (ufs-community#647)
  Update support for S4 and enable regression testing (ufs-community#654)
  global_cycle - Link to CCPP version of sfcsub.F (ufs-community#636)
  Add processing of new global AFWA snow data to emcsfc_snow2mdl. (ufs-community#648)
  Update build module on Cheyenne (ufs-community#646)
  Move to Intel 2022 on Jet, Hera and Orion (ufs-community#650)
  Host doxygen documentation for multiple releases (ufs-community#644)
  Download unit test data as part of the CMake build (ufs-community#630)
  chgres_cube - Complete removal of wgrib2 library (ufs-community#641)
  Eliminate circular dependency in chgres_cube
  Update workflow files to pull netcdf-c library from GitHub
  chgres_cube - Remove the wgrib2 library from the GRIB2 data read routines.
  Undefined symbols on macOS with Intel compiler (ufs-community#628)
  Update FVCOM code to handle sub-domain restart files using multiple cores. (ufs-community#624)
  chgres_cube - Run routine 'convert_omega' on all tasks. (ufs-community#627)
  Use ESMF 8.2.0 library
  Automate update of consistency test baseline data. (ufs-community#603)
  Update workflow files to use newer versions of ESMF and NCEPLIBS. (ufs-community#617)
  Update build modules to be lua compliant (ufs-community#614)
  Allow FVCOM tools to Update Ice Surface Roughness Length (ufs-community#604)
  Update the requested memory in the Orion chgres_cube consistency test script (ufs-community#611)
  Use copy of grib_util under Jet role account. (ufs-community#608)
  Run consistency tests on Orion using role account (ufs-community#606)
  Run consistency tests on Hera using role account (ufs-community#605)
  Run consistency tests on Jet using role account. (ufs-community#607)
  Update more documentation after move to ufs-community (ufs-community#597)
  fvcom_tools - Add option to process 'cold' or 'warm' restart files (ufs-community#595)
  Update documentation after move to ufs-community (ufs-community#594)
  chgres_cube - Eliminate segmentation fault in input_data.F90 (ufs-community#585)
  Update to language of unit test README to match that in unit test.
  chgres_cube - Simplify surface processing using field bundles (ufs-community#572)
  Add compiler flags for GNU Fortran v10 or newer compilers. (ufs-community#583)
  Move verbose output from example unit test to be commented to streamline test output.
  Updates to test README to add instructions for use of example unit test.
  Add Findwgrib2.cmake (ufs-community#578)
  Added unit test to be used as an instructional example for new users.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@GeorgeGayno-NOAA GeorgeGayno-NOAA GeorgeGayno-NOAA approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@dmwright526 @LarissaReames-NOAA @JeffBeck-NOAA @GeorgeGayno-NOAA @KateFriedman-NOAA