Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CIME update for chgres_cube "gaussian_netcdf" namelist option #162

Closed
arunchawla-NOAA opened this issue Aug 10, 2020 · 32 comments
Closed

CIME update for chgres_cube "gaussian_netcdf" namelist option #162

arunchawla-NOAA opened this issue Aug 10, 2020 · 32 comments
Assignees
@uturuncoglu
Labels
enhancement New feature or request release-1.1

Comments

@arunchawla-NOAA
Copy link
Collaborator

CIME needs to be updated to account for a new namelist option for chgres_cube. With the option of using netcdf files the input_type should now have a third option "gaussian_netcdf"
@arunchawla-NOAA arunchawla-NOAA added enhancement New feature or request release-1.1 labels Aug 10, 2020
@ligiabernardet
Copy link
Collaborator

CIME must also recognize the netCDF file names, which are of the form
gfs.tHHz.atmf000.nc
gfs.tHHz.sfcf000.nc

Examples on Hera at
/scratch1/NCEPDEV/da/George.Gayno/noscrub/chgres.gfsv16/gfs.20200202/00

@arunchawla-NOAA
Copy link
Collaborator Author

Also we need to update FV3 namelist options for damping coefficients. These were erroneously set for 127 level configuration. Need to be for 64 levels. See this thread for details

https://forums.ufscommunity.org/threads/strength-second-order-damping-top-layers

@uturuncoglu
Copy link
Collaborator

@arunchawla-NOAA sure. @rsdunlapiv could you move those files to Cheyenne or Orion? Thanks.

@rsdunlapiv
Copy link
Collaborator

@uturuncoglu Yes I can move the files. I have to have my hera certificate resigned, so it said it could be a day or two. Will try again tomorrow.

@climbfuji
Copy link
Collaborator

@uturuncoglu Yes I can move the files. I have to have my hera certificate resigned, so it said it could be a day or two. Will try again tomorrow.

Usually the certificate only takes an hour or so, even though they say 1-2 days.

@rsdunlapiv
Copy link
Collaborator

@climbfuji Execellent, I will try again shortly.

@rsdunlapiv
Copy link
Collaborator

@uturuncoglu I moved to files to Cheyenne: /glade/scratch/dunlap/ufs_xfer

@uturuncoglu
Copy link
Collaborator

Okay. Thanks @rsdunlapiv.

@uturuncoglu
Copy link
Collaborator

@climbfuji @ligiabernardet i think we need to update NCEPLIBS to handle netCDF files (especially CHGRES). If you don't mind could you point me the instructions as well as hashes that needs to be used to test the netCDF input? If you all provide a example CHGRES script that would be great. BTW, is there any change in the post-processing side?

@climbfuji
Copy link
Collaborator

I have NCEPLIBS ufs-v1.1.0 installed on almost all the platforms by now. For testing those with CIME, can you use hera or cheyenne? The new modulefiles are

# cheyenne.gnu
module use -a /glade/p/ral/jntp/GMTB/tools/modulefiles/gnu-8.3.0/mpt-2.19
module load NCEPlibs/1.1.0

# cheyenne.intel
module use -a /glade/p/ral/jntp/GMTB/tools/modulefiles/intel-19.0.5/mpt-2.19
module load NCEPlibs/1.1.0

# hera.intel
module use -a /scratch1/BMC/gmtb/software/modulefiles/intel-18.0.5.274/impi-2018.0.4
module load  NCEPlibs/1.1.0

I believe none of the prerequisite modules (compiler, ...) has changed for those, but I am not sure. For other systems such as gaea, the definitely changed. Can you please make sure for every system that you are testing/configuring for MRW v1.1 that the modules loaded in the CIME config match those in the modulefiles for the preconfigured platforms?

For the supported platforms, we need to check against the updated doc/README_*.txt files in the NCEPLIBS-external repo (but I am still working on the last documentation updates, hence this needs to wait for a bit).

If CIME aborts in case a module load fails due to wrong prerequisites, we should be on the safe side.

Thanks!

@uturuncoglu
Copy link
Collaborator

@climbfuji That is really great! Thanks. I could use cheyenne.intel for the tests. I'll keep you posted about the progress.

@ligiabernardet
Copy link
Collaborator

ligiabernardet commented Aug 12, 2020 via email

@rsdunlapiv rsdunlapiv changed the title CIME update CIME update for chgres_cube "gaussian_netcdf" namelist option Aug 14, 2020
@rsdunlapiv
Copy link
Collaborator

@uturuncoglu this is one of the main items that we need to take care of before the release. Please keep us posted as you try the modules provided by @climbfuji.

@uturuncoglu
Copy link
Collaborator

@ligiabernardet that is great! I'll look at them. Currently, I have problem related with the unification of UFSATM CIME interface and updating the model. The updated model can not be built with existing interface (more information in #169) and needs to be handled first. Then, I could start to the initial implementation of supporting new data format.

@rsdunlapiv Sure, i'll keep you posted about the progress.

@uturuncoglu
Copy link
Collaborator

@ligiabernardet what is the last three digit in the file - gfs.tHHz.atmf000.nc. I think it is forecast hour of input data but I am asking to be sure. Do we need only support 000? i.e. analysis. Is there any use case that user might need to start some other time, for example gfs.tHHz.atmf003.nc

@uturuncoglu
Copy link
Collaborator

@ligiabernardet BTW, tHHz part could change also?

@ligiabernardet
Copy link
Collaborator

ligiabernardet commented Aug 19, 2020 via email

@uturuncoglu
Copy link
Collaborator

@ligiabernardet Thanks for the information. That is really helpful and simply the CIME interface. I have already implemented gaussian_netcdf option and I am testing it now. I'll let you know when I have successful run with NetCDF files.

@uturuncoglu
Copy link
Collaborator

@ligiabernardet @climbfuji I run the CHGRES with netCDF input and it is failing without giving any error messages. I am using C96 resolution to test and grib2 works without any problem in same number of processor. I just wonder do we need to allocate more processor, if we are processing netCDF input in CHGRES? Here is my config.nml

&config
  atm_files_input_grid = "gfs.t00z.atmf000.nc"
  convert_atm = .true.
  convert_nst = .true.
  convert_sfc = .true.
  cycle_day = 2
  cycle_hour = 0
  cycle_mon = 2
  data_dir_input_grid = "/glade/p/cesmdata/cseg/ufs_inputdata/icfiles/202002/20200202"
  fix_dir_target_grid = "INPUT"
  input_type = "gaussian_netcdf"
  mosaic_file_target_grid = "INPUT/C96_mosaic.nc"
  orog_dir_target_grid = "INPUT"
  orog_files_target_grid = "oro_data.tile1.nc", "oro_data.tile2.nc",
      "oro_data.tile3.nc", "oro_data.tile4.nc", "oro_data.tile5.nc",
      "oro_data.tile6.nc"
  sfc_files_input_grid = "gfs.t00z.sfcf000.nc"
  tracers = "sphum", "liq_wat", "o3mr", "ice_wat", "rainwat", "snowwat",
      "graupel"
  tracers_input = "spfh", "clwmr", "o3mr", "icmr", "rwmr", "snmr", "grle"
  vcoord_file_target_grid = "INPUT/global_hyblev.l65.txt"
/

You could reach full CHGRES log from here,

/glade/scratch/turuncu/ufs-mrweather-app-workflow.c96v2/run/chgres_cube.200819-144825.log

In this case, I am using NCEPLIBS from /glade/p/ral/jntp/GMTB/tools/modulefiles/gnu-8.3.0/mpt-2.19, NCEPlibs/1.1.0. Is there any change in global_hyblev.l65.txt etc. after previous release. We are using global_hyblev.* files from https://ftp.emc.ncep.noaa.gov/EIB/UFS/global/fix/fix_am.v20191213/.

@ligiabernardet
Copy link
Collaborator

ligiabernardet commented Aug 19, 2020 via email

@uturuncoglu
Copy link
Collaborator

@ligiabernardet Thanks for the information. I doubled the number of processor from 36 to 72 for C96 and it seems it process the data and produce the inputs. I did not test the model yet with this input but it seems that "gaussian_netcdf" requires more resource in terms of memory. It would be nice to document it in somewhere. Maybe it could be optimized in CHGRES side. I am not sure.

@uturuncoglu
Copy link
Collaborator

@ligiabernardet varmap is not a requirement for netCDF if I look at the link. It is just for GRIB2.

@ligiabernardet
Copy link
Collaborator

ligiabernardet commented Aug 19, 2020 via email

@uturuncoglu
Copy link
Collaborator

@ligiabernardet @arunchawla-NOAA @rsdunlapiv @climbfuji i have successful run with netCDF input. I also updated app. So, if you want to test it here is the instructions:

git clone https://github.com/ufs-community/ufs-mrweather-app.git
cd ufs-mrweather-app
git checkout ufs-release-v1.1
./manage_externals/checkout_externals
cd cime/scripts
UFS_DRIVER=nems CIME_MODEL=ufs ./create_newcase --compset GFSv15p2 --res C96 --case ufs-mrweather-app-workflow.c96 --workflow ufs-mrweather
cd ufs-mrweather-app-workflow.c96
./case.setup
NOTE:  add input_type = "gaussian_netcdf" to user_nl_ufsatm
./xmlchange RUN_STARTDATE=2020-02-02
./xmlchange DOUT_S=FALSE
./xmlchange STOP_OPTION=nhours
./xmlchange STOP_N=36
./xmlchange JOB_WALLCLOCK_TIME=00:30:00
./xmlchange USER_REQUESTED_WALLTIME=00:30:00
./case.build
NOTE: you might need to issue ./case.build command again if you get error from radiation_aerosols.f, in the second one the model builds fine. I am not sure what is wrong with this file at this moment.
NOTE: you need to download files manually and place in your initial condition directory.
./case.submit 
NOTE: if CHGRES fails without any particular error. Just try to increase the number of core used by CHGRES using
./xmlchange task_count=144 --subgroup case.chgres and maybe more (for C96 i set 144 as default if you select gaussian_netcdf).

BTW, if you want to test gfs3 and gfs4 input, you just need to add resolution_for_grib2 = "1.0" or resolution_for_grib2 = "0.5" to user_nl_ufsatm. If you don't add anything gfs4 is the default unless if you have netcdf file in same directory. Then, you need to set input_type explicitly. The CIME interface also handles automatic retrieval of GRIB2 files.

@climbfuji We need to update the supported machines to use NCEPLIBS 1.1.0 also.

@ligiabernardet
Copy link
Collaborator

ligiabernardet commented Aug 21, 2020 via email

@uturuncoglu
Copy link
Collaborator

@ligiabernardet i am not sure. I did it intentionally becuase we were using ufs-release-v1.0 branch for development and then it became publicly available as release/public-v1. I did not want to push modifications that are not fully tested on all supported platforms to release/public-v1. We still need to run full test suite on Cheyenne, Stampede, Orion etc. Anyway, if you want me to update release/public-v1 branch that is fine but then the external users will get the updated code. Maybe I am missing something in here. If so, please clarify it.

@uturuncoglu
Copy link
Collaborator

@ligiabernardet i have no access to Hera and i don't know it works or not in there. We also need to update NCEPLIBS module for all platforms that are supported in release 1.0. So, my understanding is that we are not ready yet to go public.

@ligiabernardet
Copy link
Collaborator

ligiabernardet commented Aug 21, 2020 via email

@uturuncoglu
Copy link
Collaborator

@ligiabernardet That is great. I'll check the platforms and make modifications in the CIME side accordingly. I could test the model on Cheyenne, Stampede and maybe Orion but other platforms need to be tested also.

@ligiabernardet
Copy link
Collaborator

ligiabernardet commented Aug 21, 2020 via email

@uturuncoglu
Copy link
Collaborator

@ligiabernardet sure. I'll do it tomorrow first thing and let you know.

@ligiabernardet
Copy link
Collaborator

CIME can now process the netCDF data. Minimally tested so far, but it is working.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@uturuncoglu uturuncoglu

Labels
enhancement New feature or request release-1.1
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@uturuncoglu @rsdunlapiv @climbfuji @ligiabernardet @arunchawla-NOAA