Low wind speed and no diurnal cycle for temperature by SRW

[htan2013]

Dear experts,

I have used HRRR to initialize SRW for an extreme wind event. In the meantime, I also run WRF with HRRR.
The comparison between HRRR, SRW, and WRF shows that my SRW runs with HRRR clearly has a lower wind speed caused by a no-diurnal-cycle for temperature in SRW (see attached below).
For SRW, the ccpp suite that I used is FV3_HRRR. Right now I am testing with SRW with different ccpp suite (FV3_RRFS_v1beta). It looks like the issue comes from the surface layer scheme but I am not sure.

Another possibility is that the ICBC that I used is wrong. Because if there is no diurnal cycle for temperature, that means I "could" use the same file for all boundary conditions. The boundary conditions that I used is from https://noaa-hrrr-bdp-pds.s3.amazonaws.com/index.html#hrrr.20200306/conus/ where the boundary conditions files that I downloaded are: hrrr.t12z.wrfprsf00.grib2, hrrr.t12z.wrfprsf01.grib2, hrrr.t12z.wrfprsf02.grib2, hrrr.t12z.wrfprsf03.grib2......hrrr.t12z.wrfprsf36.grib2.

Sincerely appreciate any help on this.

Best regards,
Haochen

[ShawnCebulaNOAA]

Hello Haochen!

Thanks for including the data that came back that you weren't expecting, that should help in seeing what happened. I've let the team know and they'll take a look at it.

Shawn Cebula, EPIC End User Support

[htan2013]

Some updates:
I ran the same simulation with GFS data. It also shows similar results.

[natalie-perlin]

@htan2013
Haochen,
Could you please provide more information on the version of SRW you are using, and some setup files (build, logs) that could be useful to identify any potential issues? It does look like SRW may have configuration issue, as using either HRRR or GFS boundary show lack of diurnal cycle, not as would be expected. If you happen to use any of NOAA systems that our support team could access and look at, please provide paths and any custom log files.
Best,
Natalie Perlin, EPIC Advanced User Support (AUS) team

[htan2013]

@natalie-perlin
Hi Natalie,
I used SRW v2.0.0. The HPC that I used is Cheyenne and the path for UFS2.0 is in /glade/work/htan2013/UFS_2.0. In expt_dirs, you can find both HRRR_3km and GFS_3km experiments.

The ICBCs that I downloaded are from https://noaa-hrrr-bdp-pds.s3.amazonaws.com/ for HRRR and NCAR RDA for GFS. HRRR files are named hrrr.t12z.wrfprsf00.grib2, hrrr.t12z.wrfprsf01.grib2, hrrr.t12z.wrfprsf03.grib2.....

Thank you again for your help,
Haochen

[natalie-perlin]

Haochen,

Great, thanks. It might be a little less efficient to diagnose issues with SRW v2.0.0, as there were quite a few bug fixes and many changes introduced in SRW v2.1.0. Just just to be sure that the same issue with lack of the diurnal cycle persists in SRW v2.1.0, could you make one test run with the newer version?

I'm also attaching to this thread your config.*, var_defns.* from these two runs, as *.txt text files, just in case other EPIC team members of SRW users have some suggestions or may get a clue.

config_GFS_3km_20200306.txt
var_defns_GFS_3km_20200306.txt
config_HRRR_3km_20200306.txt
var_defns_HRRR_3km_20200306.txt

config_GFS_3km_20200306.txt:

MACHINE="CHEYENNE"
ACCOUNT="WYOM0111"

VERBOSE="TRUE"
COMPILER="intel"

RUN_ENVIR="community"
PREEXISTING_DIR_METHOD="delete"

# Option that using workflow
WORKFLOW_MANAGER="none"

## Custom Grid Settings ##
GRID_GEN_METHOD="ESGgrid"
EXPT_CONFIG_FN="config.sh"
PREDEF_GRID_NAME=""
ESGgrid_LON_CTR="-70.855"
ESGgrid_LAT_CTR="42.183"
ESGgrid_DELX="3000.0"
ESGgrid_DELY="3000.0"
ESGgrid_NX="286"
ESGgrid_NY="286"
ESGgrid_WIDE_HALO_WIDTH="6"
DT_ATMOS="40"
LAYOUT_X="12"
LAYOUT_Y="12"
BLOCKSIZE="40"

POST_OUTPUT_DOMAIN_NAME="East_Coast_grid"

# Directory Parameters
EXPT_BASEDIR="/glade/work/htan2013/UFS_2.0/expt_dirs"
EXPT_SUBDIR="GFS_3km_20200306"

# Parameters for Running Without a Workflow Manager
RUN_CMD_UTILS="mpirun -np 12"
RUN_CMD_FCST="mpirun -np 240"
RUN_CMD_POST="mpirun -np 12"

QUILTING="TRUE"
WRTCMP_write_groups="1"
WRTCMP_write_tasks_per_group="24"

WRTCMP_output_grid="regional_latlon"
WRTCMP_cen_lon="${ESGgrid_LON_CTR}"
WRTCMP_cen_lat="${ESGgrid_LAT_CTR}"

WRTCMP_lon_lwr_left="-74.26"
WRTCMP_lat_lwr_left="39.12"
WRTCMP_lon_upr_rght="-67.0"
WRTCMP_lat_upr_rght="45.1"

# WRTCMP_dlon = ESGgrid_DELX/(6371.2*cos(WRTCMP_cen_lat)) = [(3)/((6371.2)*cos(38.5*(pi/180)))]*(180/pi)
# WRTCMP_dlat = ESGgrid_DELY/rad_earth = [(3)/(6371.2)]*(180/pi) = 0.026979
# Use this website to calculate: https://www.mathpapa.com/algebra-calculator.html
WRTCMP_dlon="0.034473"
WRTCMP_dlat="0.026979"

## Physics Suite ##
CCPP_PHYS_SUITE="FV3_HRRR"
FCST_LEN_HRS="48"
LBC_SPEC_INTVL_HRS="3"

DO_ENSEMBLE="FALSE"
NUM_ENS_MEMBERS="2"

DATE_FIRST_CYCL="20200306"
DATE_LAST_CYCL="20200308"
CYCL_HRS=( "12" )

FIXgsm=/glade/work/htan2013/UFS_Static/fix_am
FIXaer=/glade/work/htan2013/UFS_Static/fix_aer
TOPO_DIR=/glade/work/htan2013/UFS_Static/fix_orog
SFC_CLIMO_INPUT_DIR=/glade/work/htan2013/UFS_Static/sfc_climo

EXTRN_MDL_NAME_ICS="FV3GFS"
EXTRN_MDL_NAME_LBCS="FV3GFS"

FV3GFS_FILE_FMT_ICS="grib2"
FV3GFS_FILE_FMT_LBCS="grib2"

WTIME_RUN_FCST="24:00:00"

RUN_TASK_MAKE_GRID="TRUE"
RUN_TASK_MAKE_OROG="TRUE"
RUN_TASK_MAKE_SFC_CLIMO="TRUE"
RUN_TASK_GET_OBS_CCPA="FALSE"
RUN_TASK_GET_OBS_MRMS="FALSE"
RUN_TASK_GET_OBS_NDAS="FALSE"
RUN_TASK_VX_GRIDSTAT="FALSE"
RUN_TASK_VX_POINTSTAT="FALSE"
RUN_TASK_VX_ENSGRID="FALSE"
RUN_TASK_VX_ENSPOINT="FALSE"

#
# Uncomment the following line in order to use user-staged external model 
# files with locations and names as specified by EXTRN_MDL_SOURCE_BASEDIR_ICS/
# LBCS and EXTRN_MDL_FILES_ICS/LBCS.
#
USE_USER_STAGED_EXTRN_FILES="TRUE"
#
# The following is specifically for Hera.  It will have to be modified
# if on another platform, using other dates, other external models, etc.
# Uncomment the following EXTRN_MDL_*_ICS/LBCS only when USE_USER_STAGED_EXTRN_FILES=TRUE
#
EXTRN_MDL_SOURCE_BASEDIR_ICS="/glade/work/htan2013/UFS_ICBC/GFS_2020030612"
EXTRN_MDL_FILES_ICS=( "gfs.t12z.pgrb2.0p25.f000")
EXTRN_MDL_SOURCE_BASEDIR_LBCS="/glade/work/htan2013/UFS_ICBC/GFS_2020030612"
EXTRN_MDL_FILES_LBCS=( "gfs.t12z.pgrb2.0p25.f003" "gfs.t12z.pgrb2.0p25.f006" "gfs.t12z.pgrb2.0p25.f009" "gfs.t12z.pgrb2.0p25.f012" \
                       "gfs.t12z.pgrb2.0p25.f015" "gfs.t12z.pgrb2.0p25.f018" "gfs.t12z.pgrb2.0p25.f021" "gfs.t12z.pgrb2.0p25.f024" \
		       "gfs.t12z.pgrb2.0p25.f027" "gfs.t12z.pgrb2.0p25.f030" "gfs.t12z.pgrb2.0p25.f033" "gfs.t12z.pgrb2.0p25.f036" \
 		       "gfs.t12z.pgrb2.0p25.f039" "gfs.t12z.pgrb2.0p25.f042" "gfs.t12z.pgrb2.0p25.f045" "gfs.t12z.pgrb2.0p25.f048" )

config_HRRR_3km_20200306.txt

ACCOUNT="WYOM0111"

VERBOSE="TRUE"
COMPILER="intel"

RUN_ENVIR="community"
PREEXISTING_DIR_METHOD="delete"

# Option that using workflow
WORKFLOW_MANAGER="none"

## Custom Grid Settings ##
GRID_GEN_METHOD="ESGgrid"
EXPT_CONFIG_FN="config.sh"
PREDEF_GRID_NAME=""
ESGgrid_LON_CTR="-71.1892"
ESGgrid_LAT_CTR="41.5231"
ESGgrid_DELX="3000.0"
ESGgrid_DELY="3000.0"
ESGgrid_NX="203"
ESGgrid_NY="203"
ESGgrid_WIDE_HALO_WIDTH="6"
DT_ATMOS="40"
LAYOUT_X="12"
LAYOUT_Y="12"
BLOCKSIZE="40"

POST_OUTPUT_DOMAIN_NAME="East_Coast_grid"

# Directory Parameters
EXPT_BASEDIR="/glade/work/htan2013/UFS_2.0/expt_dirs"
EXPT_SUBDIR="HRRR_3km_20200306"

# Parameters for Running Without a Workflow Manager
RUN_CMD_UTILS="mpirun -np 12"
RUN_CMD_FCST="mpirun -np 240"
RUN_CMD_POST="mpirun -np 12"

QUILTING="TRUE"
WRTCMP_write_groups="1"
WRTCMP_write_tasks_per_group="24"

WRTCMP_output_grid="regional_latlon"
WRTCMP_cen_lon="${ESGgrid_LON_CTR}"
WRTCMP_cen_lat="${ESGgrid_LAT_CTR}"

WRTCMP_lon_lwr_left="-74.7493"
WRTCMP_lat_lwr_left="38.6841"
WRTCMP_lon_upr_rght="-67.5412"
WRTCMP_lat_upr_rght="44.1189"

# WRTCMP_dlon = ESGgrid_DELX/(6371.2*cos(WRTCMP_cen_lat)) = [(3)/((6371.2)*cos(38.5*(pi/180)))]*(180/pi)
# WRTCMP_dlat = ESGgrid_DELY/rad_earth = [(3)/(6371.2)]*(180/pi) = 0.026979
# Use this website to calculate: https://www.mathpapa.com/algebra-calculator.html
WRTCMP_dlon="0.034473"
WRTCMP_dlat="0.026979"

## Physics Suite ##
CCPP_PHYS_SUITE="FV3_RRFS_v1beta"
FCST_LEN_HRS="48"
LBC_SPEC_INTVL_HRS="1"

DO_ENSEMBLE="FALSE"
NUM_ENS_MEMBERS="2"

DATE_FIRST_CYCL="20200306"
DATE_LAST_CYCL="20200308"
CYCL_HRS=( "12" )

FIXgsm=/glade/work/htan2013/UFS_Static/fix_am
FIXaer=/glade/work/htan2013/UFS_Static/fix_aer
TOPO_DIR=/glade/work/htan2013/UFS_Static/fix_orog
SFC_CLIMO_INPUT_DIR=/glade/work/htan2013/UFS_Static/sfc_climo

EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="HRRR"

WTIME_RUN_FCST="24:00:00"

RUN_TASK_MAKE_GRID="TRUE"
RUN_TASK_MAKE_OROG="TRUE"
RUN_TASK_MAKE_SFC_CLIMO="TRUE"
RUN_TASK_GET_OBS_CCPA="FALSE"
RUN_TASK_GET_OBS_MRMS="FALSE"
RUN_TASK_GET_OBS_NDAS="FALSE"
RUN_TASK_VX_GRIDSTAT="FALSE"
RUN_TASK_VX_POINTSTAT="FALSE"
RUN_TASK_VX_ENSGRID="FALSE"
RUN_TASK_VX_ENSPOINT="FALSE"

#
# Uncomment the following line in order to use user-staged external model 
# files with locations and names as specified by EXTRN_MDL_SOURCE_BASEDIR_ICS/
# LBCS and EXTRN_MDL_FILES_ICS/LBCS.
#
USE_USER_STAGED_EXTRN_FILES="TRUE"
#
# The following is specifically for Hera.  It will have to be modified
# if on another platform, using other dates, other external models, etc.
# Uncomment the following EXTRN_MDL_*_ICS/LBCS only when USE_USER_STAGED_EXTRN_FILES=TRUE
#
EXTRN_MDL_SOURCE_BASEDIR_ICS="/glade/work/htan2013/UFS_ICBC/HRRR_2020030612"
EXTRN_MDL_FILES_ICS=( "hrrr.t12z.wrfprsf00.grib2")
EXTRN_MDL_SOURCE_BASEDIR_LBCS="/glade/work/htan2013/UFS_ICBC/HRRR_2020030612"
EXTRN_MDL_FILES_LBCS=( "hrrr.t12z.wrfprsf01.grib2" "hrrr.t12z.wrfprsf02.grib2" "hrrr.t12z.wrfprsf03.grib2" "hrrr.t12z.wrfprsf04.grib2" \
                       "hrrr.t12z.wrfprsf05.grib2" "hrrr.t12z.wrfprsf06.grib2" "hrrr.t12z.wrfprsf07.grib2" "hrrr.t12z.wrfprsf08.grib2" \
		       "hrrr.t12z.wrfprsf09.grib2" "hrrr.t12z.wrfprsf10.grib2" "hrrr.t12z.wrfprsf11.grib2" "hrrr.t12z.wrfprsf12.grib2" \
 		       "hrrr.t12z.wrfprsf13.grib2" "hrrr.t12z.wrfprsf14.grib2" "hrrr.t12z.wrfprsf15.grib2" "hrrr.t12z.wrfprsf16.grib2" \
		       "hrrr.t12z.wrfprsf17.grib2" "hrrr.t12z.wrfprsf18.grib2" "hrrr.t12z.wrfprsf19.grib2" "hrrr.t12z.wrfprsf20.grib2" \
		       "hrrr.t12z.wrfprsf21.grib2" "hrrr.t12z.wrfprsf22.grib2" "hrrr.t12z.wrfprsf23.grib2" "hrrr.t12z.wrfprsf24.grib2" \
		       "hrrr.t12z.wrfprsf25.grib2" "hrrr.t12z.wrfprsf26.grib2" "hrrr.t12z.wrfprsf27.grib2" "hrrr.t12z.wrfprsf28.grib2" \
 		       "hrrr.t12z.wrfprsf29.grib2" "hrrr.t12z.wrfprsf30.grib2" "hrrr.t12z.wrfprsf31.grib2" "hrrr.t12z.wrfprsf32.grib2" \
		       "hrrr.t12z.wrfprsf33.grib2" "hrrr.t12z.wrfprsf34.grib2" "hrrr.t12z.wrfprsf35.grib2" "hrrr.t12z.wrfprsf36.grib2" )

[htan2013]

@natalie-perlin
Thank you for posting this issue.

I have started using 2.1 now but a new issue came for run_fcst step:

FATAL from PE 7: mpp_domains_define.inc: At least one pe in pelist is not used by any tile in the mosaic
FATAL from PE 14: mpp_domains_define.inc: At least one pe in pelist is not used by any tile in the mosaic

I have attached the run_fcst log and the path for exptdir is /glade/work/htan2013/UFS_2.1/expt_dirs/HRRR_3km_20200306.
Run_FCST.log

Really appreciate your time here.

Thanks,
Haochen

[natalie-perlin]

Haochen,
The error means that your job is lauched with more CPUs that the domain is configured for, i.e., the actual slurm job launched with "mpirun" need to correspond the domain decomposition defined for the grid (LAYOUT_X, LAYOUT_Y, PE_MEMBER01 computed).

[htan2013]

@natalie-perlin @ShawnCebulaNOAA

Hi Natalie and Shawn,
I have tested with SRW 2.1 and still have this "no diurnal cycle of temperature" issue.
I have tested SRW2.1 with different physics suites (FV3_HRRR, FV3_GFS, FV3_RRFS_v1beta). They showed very similar no-diurnal cycle results.

I used a customized predefined domain over the Cape Cod area (see predef_grid_params.yaml). I have attached config.yaml, predef_grid_params.yaml, valid_param_vals.yaml, and initial&boundary condition files from HRRR (google drive link). I would greatly appreciate any help with this.

Best regards,
Haochen

config.yaml.txt
predef_grid_params.yaml.txt
valid_param_vals.yaml.txt

HRRR ICBC in google drive: https://drive.google.com/file/d/13xbzFddXz_6_Z-vq_Vsx6eM_YAeMly3j/view?usp=sharing

[natalie-perlin]

Thank you, Haochen, for the updates, and providing configuration and input files.
What package do you use for plotting, and do you have the plotting scripts?

[htan2013]

@natalie-perlin
I used Matlab and Originlab to plot the data.
I have uploaded the outputs in here: https://drive.google.com/drive/folders/14KKoY0h3wLym0wPN6HHHlb2esB3cwRa8?usp=share_link
I have renamed them to the appropriate time.

[natalie-perlin]

Haochen, thank you for the outputs. My question was also about what type of diagnostics are you plotting in your figures, are these domain-averages, or land-only locations, etc.?

[htan2013]

It was domain averaged over the ocean only.
Lon covers from -74.46 to -68 and Lat from 39 to 44.1

[natalie-perlin]

Haochen,

Thank you, that helps to clarify things.
Please note that the SRW v2.1.0 is still atmosphere-only system. When you compile your code, you could see the following info:

   Configuring UFS app in Atmosphere Only mode

FMS .............. ON
FV3 .............. ON
AQM .............. OFF
GOCART ........... OFF
MOM6 ............. OFF
HYCOM ............ OFF
CICE6 ............ OFF
WW3 .............. OFF
STOCH_PHYS ....... ON
CDEPS ............ OFF
CMEPS ............ OFF
NOAHMP ........... OFF

Absence of diurnal cycle in temperature and winds over the ocean could possibly be explained by lack of SST variability, I think, in atmosphere-only simulation. Do still see lack of diurnal cycle for land points? What are vertical profiles of winds and temperature over the ocean (and their diurnal variability) vs. over the land? If the water is cooler that the atmosphere at ocean mesoscales, the stable marine boundary layer may form that inhibits downward momentum transfer - and as a result, low wind speed at lower levels.
Also note that CCPP physics suites are (Single-Column Model) (see notes: https://dtcenter.org/ccpp-scm-online-tutorial/exercise-1-run-scm-examine-results )
Let me know your findings about the land areas and vertical profiles, and whether the suggestions above could provide some explanation of the results you were getting.

[htan2013]

@natalie-perlin
Hi Natalie,
I have checked that the land points don't have such issue, although wind and temperature still deviate from HRRR and WRF. I know that SRW is currently a land-atmosphere simulation right now. I am wondering whether we have coupling options right now? I notice in config_defaults.yaml, there is an option called USE_FVCOM, which can update the surface conditions for lake/sea surface temperature. I am wondering where to get fvcom.nc file for option "FVCOM_FILE"?

Thanks,
Haochen

[natalie-perlin]

@htan2013
Haochen - yes, there is a lot of development happening towards coupled multi-component options! There could be some spaceholders for these future options or early tests, yet they are not fully supported. Users are still encouraged to explore the options and maybe tune them as they see fit (and even become code contributors?..), and may take a look at the current develop branch of the ufs-srweather-app, where the updates may bring more clarity into the options to choose.