Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The new function of chgres_cube to output the results in netcdf 4 format #689

Closed
TingLei-NOAA opened this issue Aug 29, 2022 · 31 comments · Fixed by #704
Closed

The new function of chgres_cube to output the results in netcdf 4 format #689

TingLei-NOAA opened this issue Aug 29, 2022 · 31 comments · Fixed by #704
Assignees
@GeorgeGayno-NOAA @TingLei-NOAA
Labels
enhancement New feature or request

Comments

@TingLei-NOAA
Copy link

Currently, the chgres_cube is generating results in netcdf 4 classic with hardwired setup.
This issue is opened for adding the function to generate netcdf 4 files. There are two reasons for this request: 1. The chgres_cube generated FV3-LAM cold start files would be read in and updated by GSI. Currently the GSI IO interface for fv3-lam (including the cold start files generated by chgres_cube ) is using the parallel IO of netcdf 4. When the netcdf 4 classic files are used for GSI, some minor differences would generated in the final analysis fields.
Hence, it is expected chgres_cube can create netcdf 4 files. It would be very helpful if this new function with chgres_cube could take care of the chunk size /shape in the generated netcdf 4 files, because this has an significant impact on the performance of GSI parallel IO for fv3-lam and it could also impact the IO for FV3 model.
2. For the fv3-lam forecast model , It was found the cold start files in netcdf 4 class would cause slower initialization part compared with runs using restart files in netcdf 4 (though the reasons are still to be identified).
Hence , that will be very helpful if chgres_cube would be implemented with the function to generate files in netcdf 4 format .
I also cc our colleagues directly involved in this issue Shun Liu , Ming Hu (@hu5970 ) and Eric Rodger . And they can clarify on this problem.
@GeorgeGayno-NOAA
Copy link
Collaborator

@TingLei-NOAA I was able to write netcdf 4 files. There is a way to adjust the cache size, but I just used the defaults for now. How can we test this?

@TingLei-NOAA
Copy link
Author

@GeorgeGayno-NOAA Great!. If you need me to test this new function, which branch should I use ?

GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Sep 27, 2022
@GeorgeGayno-NOAA
Update to write netcdf-4 files.
c774208 
Fixes ufs-community#689.
@GeorgeGayno-NOAA
Copy link
Collaborator

@TingLei-NOAA Use this branch: https://github.com/GeorgeGayno-NOAA/UFS_UTILS/tree/feature/netcdf4

@TingLei-NOAA
Copy link
Author

TingLei-NOAA commented Sep 27, 2022 via email

@TingLei-NOAA
Copy link
Author

TingLei-NOAA commented Sep 27, 2022 via email

@GeorgeGayno-NOAA
Copy link
Collaborator

Hi George, The changes look good to me. There is no code to control the trunk size, right? The trunk size will be decided by the system default, right? Thanks, Ming On Tue, Sep 27, 2022 at 9:59 AM Ting Lei - NOAA Affiliate @.***> wrote:
…
George, Thank you! I also cc this to Ming ,Shun and others. We will keep you posted through this issue. Ting

When using netcdf4 files, you can set the cache_size, cache_nelems and cach-preemption in the call to nf90_create. I am using the default values as I don't know how to set them for your process. Do you have an idea of how to set these arguments?

See https://docs.unidata.ucar.edu/netcdf-fortran/current/f90_datasets.html#f90-nf90_create

@TingLei-NOAA
Copy link
Author

TingLei-NOAA commented Sep 27, 2022 via email

GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Sep 28, 2022
@GeorgeGayno-NOAA
Add chunk sizes for atmospheric file write.
2038e2e 
Fixes ufs-community#689.
@GeorgeGayno-NOAA
Copy link
Collaborator

GeorgeGayno-NOAA commented Sep 28, 2022 •
edited
Loading

@TingLei-NOAA I added chunking to the atmospheric file at 2038e2e. Can you please test my branch and check for performance improvements.

Here is a check (using ncdump -h -s) of one of the wind records. The chunk sizes are set to the length of each dimension:

float v_s(lev, latp, lon) ;
      v_s:coordinates = "geolon_s geolat_s" ;
      v_s:_Storage = "chunked" ;
      v_s:_ChunkSizes = 64, 97, 96 ;
      v_s:_Endianness = "little" ;

@TingLei-NOAA
Copy link
Author

TingLei-NOAA commented Sep 29, 2022 •
edited
Loading

@GeorgeGayno-NOAA That is great! Just one question, how shall we define the parameters like i_target_out ? I didn't find out from your PR.

@GeorgeGayno-NOAA
Copy link
Collaborator

@GeorgeGayno-NOAA That is great! Just one question, how shall we define the parameters like i_target_out ? I didn't find out from your PR.

That is the 'i' dimension of the output grid. That is set by the user. My test used a C96 grid.

@TingLei-NOAA
Copy link
Author

TingLei-NOAA commented Sep 29, 2022 via email

@GeorgeGayno-NOAA
Copy link
Collaborator

@TingLei-NOAA We need to prove that these changes improve performance before merging. And I only added chunking to the atmospheric file so far. The surface and lateral boundary files do not have chunking.

Come up with a testing strategy and I can help you.

@TingLei-NOAA
Copy link
Author

TingLei-NOAA commented Sep 29, 2022 via email

@TingLei-NOAA
Copy link
Author

TingLei-NOAA commented Oct 11, 2022 via email

@TingLei-NOAA
Copy link
Author

TingLei-NOAA commented Oct 11, 2022 via email

@GeorgeGayno-NOAA
Copy link
Collaborator

GeorgeGayno-NOAA commented Oct 11, 2022 •
edited
Loading

Hi, George, I am trying to test the new chgres_cube in the GSL's RRFS 3km conus runs. But seems the namelist with it is not compatible with the new chgres_cube. Would you please point me to a one working for your branch? Thank you! Ting

What namelist error are you getting? Can I look at the log file and scripts?

@hu5970
Copy link
Contributor

hu5970 commented Oct 11, 2022 via email

@TingLei-NOAA
Copy link
Author

TingLei-NOAA commented Oct 11, 2022 via email

GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Oct 13, 2022
@GeorgeGayno-NOAA
Update chunking to write of surface file.
0c591fc 
Fixes ufs-community#689.
@TingLei-NOAA
Copy link
Author

@GeorgeGayno-NOAA
A update:
(@hu5970 's suggestion does work (the chgres_cube could be run smoothly in the workflow. Thanks).
I tested two versions of yours

  1. The first change (the output is changed to netcdf 4 from netcdf 4 classic , but no i_target_out parameters for for chunksize
  2. The second, on top of 1, set the chunk according to the variable's 3d dimension.
    The above is the findings:
    For 1, in the generated gfs_data* file, the layout is as
    ^^
    float t(lev, lat, lon) ;
    t:coordinates = "geolon geolat" ;
    t:_Storage = "contiguous" ;
    t:_Endianness = "little" ;
    VV
    When this file was used for GSI analysis, the wallclock time is as
    ^^

RESOURCE STATISTICS**************
The total amount of wall time = 308.359866
The total amount of time in user mode = 222.475745
The total amount of time in sys mode = 15.427660
The maximum resident set size (KB) = 1759124
Number of page faults without I/O activity = 682758
Number of page faults with I/O activity = 27
Number of times filesystem performed INPUT = 1719232
Number of times filesystem performed OUTPUT = 118512
Number of Voluntary Context Switches = 196078
Number of InVoluntary Context Switches = 379
END OF RESOURCE STATISTICS**
V
For the second change:
The layout in the gfs_data* file is as
^^
float t(lev, lat, lon) ;
t:coordinates = "geolon geolat" ;
t:_Storage = "chunked" ;
t:_ChunkSizes = 66, 1092, 1820 ;
t:_Endianness = "little" ;
V
The GSI using this gfs_data* file as the input gives the wall clock time as
^
RESOURCE STATISTICS**************
The total amount of wall time = 320.800557
The total amount of time in user mode = 225.983825
The total amount of time in sys mode = 15.965445
The maximum resident set size (KB) = 1757124
Number of page faults without I/O activity = 680563
Number of page faults with I/O activity = 19
Number of times filesystem performed INPUT = 1738576
Number of times filesystem performed OUTPUT = 118512
Number of Voluntary Context Switches = 195925
Number of InVoluntary Context Switches = 382
V
In summary, to my surprise, the specific chunk setup in the second according to the variable's 3d shape doesn't show benefits compared with the "continuous " storage in the first case in terms of the GSI 's performance using them as the background files.
Please let me know if any further tests you need.
Thank you so much for your developments demonstrated in this issue.
Ting

@GeorgeGayno-NOAA
Copy link
Collaborator

@TingLei-NOAA So, should we close this issue? Should we update chgres to output netcdf4 (without adding chunking)?

@edwardhartnett
Copy link
Collaborator

edwardhartnett commented Oct 14, 2022 via email

@TingLei-NOAA
Copy link
Author

TingLei-NOAA commented Oct 14, 2022 •
edited
Loading

@edwardhartnett Thanks a lot for your comments/explanation. We need netcdf 4 because the current GSI parallel IO for fv3-lam is the parallelization based on hdf 5, namely, only working for netcdf 4. But if we need compression in the near future, I have no idea. @hu5970 and Shun Liu could be a better position to give the answer.
@GeorgeGayno-NOAA , from the purpose of this issue, yes, this issue can be closed. The decision on whether to include the second change (chunk size setup ) is up to you with Ed's information.
Thanks!

GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Oct 17, 2022
@GeorgeGayno-NOAA
Merge branch 'develop' into feature/netcdf4
6ca174b 
Fixes ufs-community#689.
@GeorgeGayno-NOAA
Copy link
Collaborator

@edwardhartnett Thanks a lot for your comments/explanation. We need netcdf 4 because the current GSI parallel IO for fv3-lam is the parallelization based on hdf 5, namely, only working for netcdf 4. But if we need compression in the near future, I have no idea. @hu5970 and Shun Liu could be a better position to give the answer. @GeorgeGayno-NOAA , from the purpose of this issue, yes, this issue can be closed. The decision on whether to include the second change (chunk size setup ) is up to you with Ed's information. Thanks!

@TingLei-NOAA You said "we need netcdf 4" for the GSI. If I close this issue, chgres will continue to output "netcdf4 classic". Is that what you want?

@TingLei-NOAA
Copy link
Author

TingLei-NOAA commented Oct 17, 2022 via email •
edited
Loading

@GeorgeGayno-NOAA
Copy link
Collaborator

@TingLei-NOAA Ok. We can always revisit chunking later.

GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Oct 18, 2022
@GeorgeGayno-NOAA
Add chunking to the lateral boundary file.
9e662d7 
Fixes ufs-community#689.
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Oct 18, 2022
@GeorgeGayno-NOAA
Update file create to output netcdf4 file.
05b976e 
Fixes ufs-community#689.
@GeorgeGayno-NOAA
Copy link
Collaborator

Completed the updates to add chunking to all output files - atmosphere, surface and LBC - at 9e662d7. Will keep this branch in my fork in case we want to revisit chunking in the future.

Opened a new branch and modified the nf90_create calls to output netcdf4 - 05b976e.

@GeorgeGayno-NOAA
Copy link
Collaborator

Compiled the branch at 05b976e on Cactus and ran the consistency tests. All tests failed. None of the data values were different. Only the global attributes. Example:

DIFFER : FILE FORMATS : NC_FORMAT_NETCDF4 <> NC_FORMAT_NETCDF4_CLASSIC

@GeorgeGayno-NOAA
Copy link
Collaborator

Compiled the branch at 05b976e on Hera, Jet and Orion. All tests failed as they did on Cactus. This is the expected result. Will submit a PR.

@GeorgeGayno-NOAA GeorgeGayno-NOAA mentioned this issue Oct 18, 2022
Merged
@GeorgeGayno-NOAA
Copy link
Collaborator

@TingLei-NOAA Please review the PR - #704.

@GeorgeGayno-NOAA GeorgeGayno-NOAA added the enhancement New feature or request label Oct 18, 2022
@GeorgeGayno-NOAA GeorgeGayno-NOAA mentioned this issue Oct 19, 2022
Closed
@edwardhartnett
Copy link
Collaborator

edwardhartnett commented Oct 19, 2022 •
edited
Loading

OK, I'm not sure if this is relevant but there is some confusion here about netCDF-4 vs. netCDF-4 classic.

A netCDF-4 classic file is a netCDF-4 file that adheres to the netCDF classic data model. A netCDF-4 classic file is a netCDF-4 file and can easily be read by any netCDF-4 application.

So you do not need to turn off the NC_CLASSIC_MODEL flag in order to improve performance. If you take away that flag, netCDF will allow you to create elements of the enhanced model in the file. For example, in a file with NC_CLASSIC_MODEL, there can only be one unlimited dimension. If you try and create a second unlimited dimension, you will get an error.

But create a file without NC_CLASSIC_MODEL and you will be able to create as many unlimited dimensions as you want.

In both cases, a netCDF4/HDF5 file results, and both files can be read by any netCDF program.

So the difference between classic and not is simply that classic files restrict what you can add to the file, in a way that exactly matches the behavior of netCDF classic. Without the NC_CLASSIC_MODEL, netCDF allows you to use features that were not present in classic netCDF, including multiple dimensions, user-defined types, and unsigned integer types.

But there is no performance difference between netCDF-4 files created with or without NC_CLASSIC_MODEL.

@TingLei-NOAA
Copy link
Author

@edwardhartnett Thanks a lot for your information, very helpful!
As stated in the issue, the FV3-Lam GSI needs netcdf 4 because its' current parallel IO is based on parallel netcdf 4, which doesn't support treating of netcdf classic.
Thanks

GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Oct 19, 2022
@GeorgeGayno-NOAA
Merge branch 'develop' into feature/netcdf4_nochunk
43feadf 
Fixes ufs-community#689.
GeorgeGayno-NOAA added a commit that referenced this issue Oct 20, 2022
@GeorgeGayno-NOAA
Update chgres_cube to output netcdf4 file. (#704)
1817f1d 
Previously, the coldstart files were netcdf4-classic.

Fixes #689.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@GeorgeGayno-NOAA GeorgeGayno-NOAA

@TingLei-NOAA TingLei-NOAA

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Modify chgres_cube output netcdf4 files. GeorgeGayno-NOAA/UFS_UTILS
4 participants
@hu5970 @edwardhartnett @GeorgeGayno-NOAA @TingLei-NOAA