Run FCST error: FATAL from PE 2: mpp_domains_define.inc: not all the pe_end are in the pelist

[htan2013]

Dear all experts,

I recently transferred my simulation from 2.0 to 2.1 in Cheyenne.
However, with my predefined domain in predef_grid_params.yaml, I met an error when running run_fcst.sh:

FATAL from PE 2: mpp_domains_define.inc: not all the pe_end are in the pelist
FATAL from PE 3: mpp_domains_define.inc: not all the pe_end are in the pelist
FATAL from PE 6: mpp_domains_define.inc: not all the pe_end are in the pelist
FATAL from PE 18: mpp_domains_define.inc: not all the pe_end are in the pelist

I feel like this is related to CPU number issue from previous posts but I have checked the layout_XY of my custom predefined domain by using get_layout.sh (suggested layout_x=6, layout_y=8). I also used 48 cpus to run.

The path for my simulation is in /glade/work/htan2013/UFS_2.1/expt_dirs/HRRR_3km_20200306. I have attached my config.yaml, predef_grid_params.yaml, logs for all steps including FCST. I would greatly appreciate any assistance in this matter.

Best regards,
Haochen

config.yaml.txt
Make_Grid.log.txt
Make_ICS.log.txt
Make_LBCS.log.txt
Make_Orog.log.txt
Make_SFC_CLIM.log.txt
predef_grid_params.yaml.txt
Run_FCST.log.txt
Run_Get_ICS.log.txt
Run_Get_LBCS.log.txt

[mkavulich]

@htan2013 It looks like you are over-subscribing your MPI tasks. You have the following settings per your log file:

++ QUILTING=TRUE
++ WRTCMP_write_groups=1
++ WRTCMP_write_tasks_per_group=24

With QUILTING=TRUE, the write component is activated and assigned MPI tasks based on the number of write tasks and groups. With your settings, you have 6×8=48 CPUs assigned to model integration, and 1×24=24 CPUs assigned to the write component (creating output files) for a total of 72 CPUs, which is more than the 48 specified in your mpirun command.

While you could set QUILTING=FALSE, this option is not extensively tested, so I would recommend changing your layout_x and layout_y to compensate for this info. 24 write component tasks is also quite large for your domain size, I'd recommend reducing WRTCMP_write_tasks_per_group to a much smaller number (2 or 4 should suffice).

[htan2013]

@mkavulich
Really appreciate your answer! Thank you!