ATMAQ Configuration Question

[jaberg7777]

Hello again - I was wondering if it is possible for anyone to provide direction for working with the ATM-CMAQ coupled configuration of the UFS weather model. I've built that configuration, but am unable to successfully run it. I believe that my problem might lie in an improperly populated export_fv3 function. Documentation describing the proper way to set this function up in the default_vars.sh file is not available in the user guide (yet?), and I was wondering if anybody might be able to offer some guidance. I'm aware that this might not be publicly available, but I was curious and hoping for guidance. Thanks very much!

[chan-hoo]

@jaberg7777, if you have access to the NOAA HPC tier-1 platforms such as Hera, Orion, Hercules, or Derecho, you can work with the UFS SRW App (SRW user's guide, Section 1.2.8.3; https://ufs-srweather-app.readthedocs.io/en/develop/UsersGuide/BuildingRunningTesting/AQM.html#we2e-test-for-aqm). If not, you can refer to the regression test of the ufs weather model (UFS weather model user's guide, Section 3.6.1; https://ufs-weather-model.readthedocs.io/en/develop/BuildingAndRunning.html#running-the-model). In the file tests/rt.conf, you can use regional_atmaq or regional_atmaq_debug. As you can see there, both UFS SRW App and the UW regression test are for regional modeling. As far as I know, there is no sample configuration script for global modeling yet.

[jaberg7777]

Understood! Thanks very much for your quick response.

When I've attempted to run the regional_atmaq test, the code compiles successfully but I then encounter a core dump/segmentation fault. I've tried running the _debug but have the same problem. I've been advised that that sticking point might be caused by running the model with insufficient memory/cpu. To that end, I was wondering if you have any suggestions for the minimum cores or mem per cores that I should be utilizing? I'm running this on Northwestern's Quest HPC, so not a level 1 system.

Thanks again!

[gspetro-NOAA]

Hi @jaberg7777 ,

A lot of the decisions about cores/members per core are system or configuration dependent. The lscpu command will output information about your system's CPU-related components. Depending on your job scheduler, you may be able to see additional information. For example, Slurm's sinfo provides some info about nodes, and pbsnodes -a does the same for PBS job schedulers. It's hard to say for certain how to proceed without knowing more about your system. As to the specifics for the regional_atmaq test, I've reached out to team members with AQM experience, and they are looking into whether there is a minimum for tasks and/or memory. Once we get you that information, if the test continues to fail, it would be helpful to see a stack trace.

Best,
Gillian

[jaberg7777]

Thanks very much @gspetro-NOAA! I really appreciate the help - getting that information would be very useful. I do know that the HPC we operate on has 3840 total cores with 4.8 GB memory/core- and we use Slurm. Up to this point, my failed runs have designated between 3.00 and 4.00 gb/core. I'm happy to give more information about the system specs if that can help.

Thanks again.
-Jamie

[BrianCurtis-NOAA]

@JianpingHuang-NOAA Do you know of a minimum number of tasks and/or memory in which the AQM can run with?

[JianpingHuang-NOAA]

@jaberg7777. If you run the UFS-AQM online system for the AQM_NA_13km domain, please refer to the table (here) to define the parameters required for your runs on your supercomputer platform. Please define the numbers for "Layout_x" and "Layout_y" in the /ush/config.yaml file, and numbers for "write_task_per_group" and "write_group" in the "predef_grid_params.yaml" (/ush). Please make sure that all the numbers that you defined follow the formula of
(LAYOUT_X * LAYOUT_Y + WRTCMP_write_groups * WRTCMP_write_tasks_per_group) * OMP_NUM_THREADS_RUN_FCST =
NNODES_RUN_FCST * PPN_RUN_FCST * OMP_NUM_THREADS_RUN_FCST. Hope this helps.

[jaberg7777]

@JianpingHuang-NOAA Thank you! I've verified that the numbers we defined in previous runs align with that formula, but unfortunately the segmentation fault is continuing to show up when I run the model. Because of this I'm led to believe that the error is not stemming from memory issues, but may be a more fundamental problem with how our dependencies/model are set up. I've tried diagnosing the problem, but cannot trace the error.

To that end - Could you let me know exactly what versions of each of the model dependencies and what commit/version of the coupled model we should be using to try and run AQM? We are currently using the most recent commit. Hopefully matching what you recommend down to the letter will help us better identify the source of the issue.

Thanks again - and apologies for the continued inquiries.

[JianpingHuang-NOAA]

@jaberg7777. Can you check out our latest package for AQMv7.1 from https://github.com/JianpingHuang-NOAA/ufs-srweather-app/tree/fire_voc_poc_fixed?

[jaberg7777]

Hi @JianpingHuang-NOAA - would these be the same packages for running AQM on the standard weather model - not the SRW app?

[JianpingHuang-NOAA]

@jaberg7777 The ufs-weather-model has not been updated since last Jan. Try to get this package running first.

[gspetro-NOAA]

@jaberg7777 Did you manage to get AQM v7.1 running, or did you need further assistance?