Questions about LAYOUT_X, LAYOUT_Y, PE_MEMBER01, etc.

[psooner]

Hi,

I'm trying to use more computer resources to complete my jobs quicker.

I've already set up my experiment and I'm wondering what exactly I need to change to use more computer horsepower.

1. Do I need to edit var_defns.sh and make changes there? Only there or do I also need to make changes in other files?
2. Variables in var_defns.sh that seem important are: LAYOUT_X, LAYOUT_Y, NNODES_RUN_FCST, PPN_RUN_FCST, PE_MEMBER01. Is that correct? Are there other variables that also would need to be changed?
3. I did a run and got an error message: "FATAL from PE 1104: mpp_domains_define.inc: At least one pe in pelist is not used by any tile in the mosaic". What is the solution to this error?
4. How does one choose LAYOUT_X and LAYOUT_Y? Are there rules or constraints or can I just make up numbers (e.g., 31 and 27)?
5. I think PE_MEMBER01 is computed according to NNODES_RUN_FCST * PPN_RUN_FCST (right?). Are there any other rules or constraints that I'm violating to cause the error mentioned in step 3?

Thank you for your help.

Phillip S.

[mkavulich]

@psooner If you want to change the MPI layout, it's recommended you change this via modifying your config file and re-generating the experiment rather than attempting to modify your experiment variables manually. This is because not only do other variables get calculated based on these values (such as PE_MEMBER01 as you have identified), but other values (such as in the weather model namelist) are filled in based on this info.

Can you let us know what version of the App you are using? There have been a few changes to config format over time so the specific instructions will depend on this.

[psooner]

Hi.

I am using v2.1.0.

Since I've already started my experiment, I was hoping that I could change a few variables here and there within my experiment (after testing in a separate experiment) without having to recreate my experiment (which already is populated with some directories and model output).

Thank you.

Phillip S.

[DavidHuber-NOAA]

@psooner If you would like to go that route, then I would suggest following @mkavulich's suggestion and generate a new experiment. Afterwards, run a diff -q to determine which ascii files are different. Next modify your current experiment's files wherever processor counts, tiles, and other resource-related quantities differ. Use caution with this option. NOTE do not copy in any new binary files (e.g. the rocoto database file, which would effectively restart your experiment). I should also mention that I have never tried this before and thus cannot guarantee it will work, but in the worst case scenario, you would have a new experiment read yto go.

If you are working on one of the tier-1 machines, I'd be happy to take a look at the diffs.

[psooner]

That's a good idea. Thank you.

However, I did run an experiment with some different values and got the error message mentioned in 3) of my original message.
My 3), 4), and 5) in my original message possibly are related.

Would you please comment on those and perhaps provide some guidance? [I can't do the 'diff -q' approach until I get a successful run using more computer resources.]

Thank you.

Phillip S.

[EdwardSnyder-NOAA]

My apologies @psooner for the delayed reply. Below are the responses to the numbered questions from your initial message.

3. This error indicates that the tiling is not matching up with the number of cores requested (i.e. the slurm job has more or less CPUs than what the domain is configured for). It’s possible that by setting NNODES_RUN_FCST, more PEs than necessary were created. It is also possible that the number of processors in your mpirun command doesn't match your grid settings (discussed in points 4 and 5).

4. The LAYOUT_X and LAYOUT_Y variables are the number of MPI tasks to use in the horizontal x and y directions of the regional grid when running the forecast model. The larger your layout, the more MPI communication must occur, increasing overhead. Another option is to enable higher threading (OMP_NUM_THREADS=2 or 4; I wouldn’t recommend going higher than 4 or 5, though, and in any case always keep it to a multiple of the number of cores/CPU). This will use less memory and offers some speedup (typically not as much as doubling the PEs, but it’s a balancing act).

5. PE_MEMBER01 is the number of MPI processes required by the forecast and is calculated by LAYOUT_X * LAYOUT_Y + WRTCMP_write_groups * WRTCMP_write_tasks_per_group when QUILTING is true. Since these variables are connected, I'd recommend figuring out how many processors you want the forecast model to use (i.e. PE_MEMBER01) first and then work backwards from that. The WRTCMP_write_tasks_per_group value will depend on domain size. So larger domain would mean using a higher value, while a smaller domain would be something under 5. For simplicity, I'd keep WRTCMP_write_groups to 1. Take a look at our predefined grids to get a better idea of what these numbers are: ufs-srweather-app/ush/predef_grid_params.yaml. Finally, select numbers for the LAYOUT variables that will solve for the PE_MEMEBER01 variable. Out of curiosity, what grid are you using for this experiment?

Also feel free to take a look at our Computational Parameters section of our User Documentation for further guidance.

Thanks!

Eddie Snyder | EPIC Advanced User Support (AUS) team

[psooner]

Eddie,

Thank you very much for the detailed response.

It's possible that I had some inconsistencies between the variables that you mentioned. I will try again.

I'm running using the RRFS_CONUS_3km grid.

Thanks, again.

Phillip S.