Test MAPL v2.53.0 in UFS weather model #2346

junwang-noaa · 2024-06-28T13:55:56Z

Description

MAPL 2.46.2 has fixes for issue #2162. UFS weather model needs to be tested and updated with this version.

20240830:
MAPL 2.46.2 has a bug. MAPL 2.46.3 should be installed and tested in UFS weather model. The issue title is updated.

Solution

Alternatives

Related to

junwang-noaa · 2024-06-28T16:08:07Z

@lipan-NOAA Can you confirm this version of MAPL works in GOCART for GEFSv13? Thanks

BrianCurtis-NOAA · 2024-07-01T12:05:45Z

@Hang-Lei-NOAA is MAPL 2.46.2 installed on Acorn/WCOSS2?

Hang-Lei-NOAA · 2024-07-01T12:08:03Z

I will add these today.

…

On Mon, Jul 1, 2024 at 8:06 AM Brian Curtis ***@***.***> wrote: @Hang-Lei-NOAA <https://github.com/Hang-Lei-NOAA> is MAPL 2.46.2 installed on Acorn/WCOSS2? — Reply to this email directly, view it on GitHub <#2346 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKWSMFE5DZSFI5WKMRDN65LZKFA25AVCNFSM6AAAAABKB6TLROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJZHE3DSNJQGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

lipan-NOAA · 2024-07-02T14:47:30Z

@Hang-Lei-NOAA Can you tell me where you installed it?

Hang-Lei-NOAA · 2024-07-02T14:55:02Z

@Li Pan - NOAA Affiliate ***@***.***> Please check here module use /lfs/h1/emc/nceplibs/noscrub/hpc-stack/libs/hpc-stack/modulefiles/mpi/intel/19.1.3.304/cray-mpich/8.1.9 module load esmf/8.6.1 module load mapl/4.6.2-esmf-8.6.1

…

On Tue, Jul 2, 2024 at 10:47 AM lipan-NOAA ***@***.***> wrote: @Hang-Lei-NOAA <https://github.com/Hang-Lei-NOAA> Can you tell me where you installed it? — Reply to this email directly, view it on GitHub <#2346 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKWSMFBJID7JN5QQCTIGLFDZKK4RPAVCNFSM6AAAAABKB6TLROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBTGQYTMMBYGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

ulmononian · 2024-07-02T15:05:16Z

do you want this installed in spack-stack/1.6.0? and on which machine?

jkbk2004 · 2024-09-24T14:05:00Z

@DusanJovic-NOAA @junwang-noaa new maple and esmf version are available on hercules and orion for the test and debug activities. @RatkoVasic-NOAA thanks for the installation!

Hercules: /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/ue-esmf-8.6.1-mapl-2.46.3/install/modulefiles/Core
Orion: /work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/envs/ue-esmf-8.6.1-mapl-2.46.3/install/modulefiles/Core

mathomp4 · 2024-09-24T14:17:37Z

As an FYI, the MAPL 2.46.3 fix was when using externally initialized MPI. (Which is something you all do but we don't do internally).

We also had this notice to users:

Notice to users: External code should initialize MPI with MPI_THREAD_MULTIPLE as that is what MAPL expects. Also, code might need to call:
call ESMF_InitializePreMPI()
for all features of MAPL to be supported, namely if ESMF Single System Image (SSI) code is enabled. If users do not use this (or know what it is), it most likely is not needed.

My guess is you do not need to call ESMF_InitializePreMPI() as I don't think even we internally quite use all the SSI code yet.

junwang-noaa · 2024-12-02T19:54:48Z

@weiyuan-jiang may I ask what compiler/mpi versions you want to test? Thanks

mathomp4 · 2024-12-02T21:53:32Z

@weiyuan-jiang may I ask what compiler/mpi versions you want to test? Thanks

@junwang-noaa Well for that version of MAPL, we would have been testing with:

Intel ifort 2021.6
Intel ifort 2021.12 (or .13)
GCC 13.2.0

We never had Intel ifort 2021.9 on any machines we have. Are any of these possible?

junwang-noaa · 2024-12-03T15:18:21Z

@AlexanderRichert-NOAA do we have these intel and GCC version on Hera or Hercules orGaea?

AlexanderRichert-NOAA · 2024-12-03T17:32:15Z

I can't get onto Hera at the moment but for the others:

Gaea (C5 & C6):
- ifort 2021.6 (module load intel-classic/2022.1.0)
- GCC 13.2 (module load gcc-native/13.2)
Hercules & Orion:
- ifort 2021.12.0 (module load intel-oneapi-compilers/2024.1.0)

mathomp4 · 2024-12-03T19:35:03Z

As we have access to Hercules/Orion, I guess ifort 2021.12 is our best bet at the moment (though I'd have to imagine there must be a recent-ish gcc on there).

I'm fairly certain we can run with 2021.12. There were some bugfixes needed in places of GEOS (I don't think MAPL, but I can find out), so by the time we got them all in, 2021.13 was out, but I think 2021.12 works.

Plus, we have 2021.12 on discover, so we can do some matching if need be.

junwang-noaa · 2024-12-04T16:11:45Z

@weiyuan-jiang @mathomp4 Which MAPL version would you like us to test in UFS? I assume the ESMF version is 8.6.1

weiyuan-jiang · 2024-12-04T17:37:43Z

@junwang-noaa It doesn't matter. I can always replace the MAPL as long as it can be built and run under new compiler on Hercules

DusanJovic-NOAA · 2025-01-02T14:18:59Z

@DusanJovic-NOAA I do not think we want to change the current (0,1) mask values for ATM because that will have downstream impacts in CMEPS.

I can't even find the MAPL code you reference, but if you don't want to mask the destination anywhere, then why not provide a special value? CMEPS uses ispval_mask = -987987 ! spval for RH mask values

MAPL code I reference is here:
https://github.com/GEOS-ESM/MAPL/blob/1c55973c244834ed0d78c39964d2738c3d8f8a8b/base/MAPL_EsmfRegridder.F90#L1519

I do not understand where that special value should be provided and how is MAPL going to use that value?

DeniseWorthen · 2025-01-02T14:29:29Z

I didn't mean that MAPL should somehow reach into CMEPS for the value. I was just pointing out that by setting a special value (one that is never encountered), you can map to all destination points.

DusanJovic-NOAA · 2025-01-02T14:44:29Z

I didn't mean that MAPL should somehow reach into CMEPS for the value. I was just pointing out that by setting a special value (one that is never encountered), you can map to all destination points.

Sorry, I still do not understand what should be set to a special value, a grid mask? The problem is that some of the fv3atm grid points have grid mask set to 0 and MAPL considers them as destination grid points that should be masked out.

The fv3atm grid mask is defined in this routine https://github.com/NOAA-EMC/fv3atm/blob/a7d46eee01a78f0915373ebc58c9b20ba14a6c36/atmos_model.F90#L3654

DeniseWorthen · 2025-01-02T14:53:00Z

The dstMaskValue can be set to a special value.

I'm probably misunderstanding, since I've never looked at MAPL. But the issue is that when the destination is ATM, you want all destination points mapped, right? Or is the destination in this case the aerosol "grid/mesh" ?

DusanJovic-NOAA · 2025-01-02T15:03:34Z

The dstMaskValue can be set to a special value.

Yes, it can be. But it is currently set to 0 (MAPL_MASK_OUT is 0). Here. That means any fv3atm destination point with grid mask 0 (all ocean points) will be masked out.

I'm probably misunderstanding, since I've never looked at MAPL. But the issue is that when the destination is ATM, you want all destination points mapped, right? Or is the destination in this case the aerosol "grid/mesh" ?

Yes. All fv3atm destination points should be mapped, ie. it should not have grid mask set to 0.

If we can not redefine fv3atm grid mask to not use 0 for any grid point, which it seems we can not do, then either we pass the information to MAPL to not use 0 to mask out destination points. Or maybe somewhere in fv3atm or maybe in gocart we redefine grid mask that is going to be passed to MAPL.

DeniseWorthen · 2025-01-02T15:10:55Z

But why does the dstMaskValue have to be set using the values that ATM uses to define it's mask? I realize that may be how the code is working, but I don't think it should be constrained like that.

junwang-noaa · 2025-01-02T15:13:29Z

@weiyuan-jiang @tclune UFS is using a land sea mask on FV3 cubed sphere grid (0 for ocean and 1 for land) for coupling purposes. May I ask if the following line (MAPL_MASK_OUT) can be updated so that mask value 0 in UFS won't be considered undefined?

          if (has_mask) dstMaskValues = [MAPL_MASK_OUT] ! otherwise unallocated

weiyuan-jiang · 2025-01-02T15:34:20Z

I think the constant value of MAPL_MASK_OUT should be updated

DusanJovic-NOAA · 2025-01-03T14:32:36Z

With this change in GOCART I was able to make a successful (no floating point exception) run.

diff --git a/ESMF/UFS/Aerosol_Cap.F90 b/ESMF/UFS/Aerosol_Cap.F90
index b753afa..e21ed11 100644
--- a/ESMF/UFS/Aerosol_Cap.F90
+++ b/ESMF/UFS/Aerosol_Cap.F90
@@ -243,6 +243,7 @@ contains
     type(MAPL_CapOptions)     :: maplCapOptions
     type(Aerosol_InternalState_T) :: is
     type(Aerosol_Tracer_T), pointer :: trp
+    integer(kind=ESMF_KIND_I4), pointer  :: maskPtr(:,:)
 
     ! begin
     rc = ESMF_SUCCESS
@@ -338,6 +339,10 @@ contains
       file=__FILE__)) &
       return  ! bail out
 
+    call ESMF_GridGetItem(grid, itemflag=ESMF_GRIDITEM_MASK,   &
+                          staggerloc=ESMF_STAGGERLOC_CENTER, farrayPtr=maskPtr, _RC)
+    maskPtr = 1
+
     ! provide model grid to MAPL
     call cap % cap_gc % set_grid(grid, lm=nlev, _RC)

This change explicitly sets all grid mask points to 1, before passing the grid to MAPL.

weiyuan-jiang · 2025-01-06T16:37:59Z

Ben has a new PR (GEOS-ESM/MAPL#3281) to solve the problem on MAPL. So no need to change on ufs

DusanJovic-NOAA · 2025-01-06T17:16:21Z

Ben has a new PR (GEOS-ESM/MAPL#3281) to solve the problem on MAPL. So no need to change on ufs

@weiyuan-jiang Thanks for the update. It's good to know that no changes in fv3atm nor gocart are necessary to solve the issues with masking in MAPL regridding.

Any progress with cubed sphere subgrids issue when threading is used.

weiyuan-jiang · 2025-01-06T18:37:25Z

@DusanJovic-NOAA I have opened an issue on MAPL. (GEOS-ESM/MAPL#3274) . It seems multi-threading is not working at this point

DusanJovic-NOAA · 2025-01-23T17:50:28Z

@weiyuan-jiang I see the MAPL team has released version 2.52.0, which includes the fix for grid masks. I tested it together with recently releases ESMF 8.8.0 on Hercules and all ufs-weather-model regression test passed (using intel compiler, I did not test gnu). I had to turn off OpenMP threading in GOCART due to the first issue with cubed-sphere grids and threading.

climbfuji · 2025-01-23T17:54:13Z

@DusanJovic-NOAA Do you want mapl to be updated to 2.52.0 for spack-stack-1.9.0 (we already have esmf 8.8.0)? We are hoping to cut the release branch tomorrow, thus this would be just in time. Thanks!

mathomp4 · 2025-01-23T18:30:55Z

@DusanJovic-NOAA Do you want mapl to be updated to 2.52.0 for spack-stack-1.9.0 (we already have esmf 8.8.0)? We are hoping to cut the release branch tomorrow, thus this would be just in time. Thanks!

Note: We do think we have the fix for the GOCART OpenMP issue here:

GEOS-ESM/MAPL#3350

I believe it worked for @weiyuan-jiang on Hercules, but if it works for you, I could get a MAPL version out tomorrow.

Also, MAPL 2.52 doesn't (yet) require ESMF 8.8 as we don't need any features from that yet in MAPL2. MAPL3 does currently require it.

DusanJovic-NOAA · 2025-01-23T19:26:44Z

@DusanJovic-NOAA Do you want mapl to be updated to 2.52.0 for spack-stack-1.9.0 (we already have esmf 8.8.0)? We are hoping to cut the release branch tomorrow, thus this would be just in time. Thanks!

Note: We do think we have the fix for the GOCART OpenMP issue here:

GEOS-ESM/MAPL#3350

I believe it worked for @weiyuan-jiang on Hercules, but if it works for you, I could get a MAPL version out tomorrow.

Also, MAPL 2.52 doesn't (yet) require ESMF 8.8 as we don't need any features from that yet in MAPL2. MAPL3 does currently require it.

Thanks for the update @mathomp4. I am using a shell script provided by @AlexanderRichert-NOAA to build chained spack environment with updated ESMF and MAPL to run test on Hercules. If you can make a temporary tag or branch or something that spack can point to (I'm not sure if it has to be release), I should be able to try that version that includes OpenMP fix from 3350 PR you mentioned above. And if we can get that tested before @climbfuji needs to cut release for 1.9.0 spack-stack that would be perfect.

climbfuji · 2025-01-23T19:27:53Z

Other components/models need esmf 8.8.0, might as well use that across the board if it works for Dusan. Thanks Matt!

mathomp4 · 2025-01-23T19:31:02Z

@DusanJovic-NOAA I've tagged with temp-tag-for-dusan if that's easy to use. It's also at commit GEOS-ESM/MAPL@260d563

DusanJovic-NOAA · 2025-01-23T19:47:30Z

@DusanJovic-NOAA I've tagged with temp-tag-for-dusan if that's easy to use. It's also at commit GEOS-ESM/MAPL@260d563

Thanks. @AlexanderRichert-NOAA is there a way to use this tag (specific commit hash) in your env chainer script to build MAPL.

AlexanderRichert-NOAA · 2025-01-24T00:11:20Z

Thanks. @AlexanderRichert-NOAA is there a way to use this tag (specific commit hash) in your env chainer script to build MAPL.

@DusanJovic-NOAA in theory yes-- I just made a small tweak in the env chainer script to allow it to use Spack's built-in support for using git references. For the mapl spec, it would be something like mapl@git.260d563d08ff0d3e8436ec33ed98a362110e731b=develop where 'develop' is the version that Spack is going to treat it as (you can set this to a numbered version for the purpose of determining dependencies etc.). The only trick is that in that commit, MAPL's CMakeLists.txt requires CMake 3.24 or higher, which is not what spack-stack-1.6.0 uses, so you have to work around that. mapl@git.260d563d08ff0d3e8436ec33ed98a362110e731b=develop cmake@3.24.4 for the specs worked for me on hercules just now (with another 'git pull' of the env chainer script, and not forgetting about the depends_on("udunits")).

DusanJovic-NOAA · 2025-01-24T12:42:20Z

Thanks. @AlexanderRichert-NOAA is there a way to use this tag (specific commit hash) in your env chainer script to build MAPL.

@DusanJovic-NOAA in theory yes-- I just made a small tweak in the env chainer script to allow it to use Spack's built-in support for using git references. For the mapl spec, it would be something like mapl@git.260d563d08ff0d3e8436ec33ed98a362110e731b=develop where 'develop' is the version that Spack is going to treat it as (you can set this to a numbered version for the purpose of determining dependencies etc.). The only trick is that in that commit, MAPL's CMakeLists.txt requires CMake 3.24 or higher, which is not what spack-stack-1.6.0 uses, so you have to work around that. mapl@git.260d563d08ff0d3e8436ec33ed98a362110e731b=develop cmake@3.24.4 for the specs worked for me on hercules just now (with another 'git pull' of the env chainer script, and not forgetting about the depends_on("udunits")).

Thanks. Let me try that.

mathomp4 · 2025-01-24T13:51:36Z

I just confirmed with @weiyuan-jiang that he didn't do anything else after using that new MAPL code. He kept OpenMP on in his test.

He did mention you'll need to make sure you have udunits loaded, but @AlexanderRichert-NOAA indicated that as well.

DusanJovic-NOAA · 2025-01-24T14:31:24Z

I successfully compiled MAPL tag (commit 260d563d08ff0d3e8436ec33ed98a362110e731b) using @AlexanderRichert-NOAA instructions, and I am running all intel ufs-wm regression tests. So far all tests passed (most importantly, cpld_control_p8_intel passed, which is the test that was failing previously). This is the configuration with:

$ grep use_threads parm/gocart/GOCART2G_GridComp.rc
use_threads: .TRUE.

So looks like we finally have a version of MAPL that fixes both issues, cubed_sphere grid with threading and masking of ESMF_grid due to mask value conflict between fv3atm and mapl.

In the meantime I will try to build spack-stack using GNU, and run gnu ufs-wm regression tests, just to be sure.

DusanJovic-NOAA · 2025-01-24T14:38:40Z

I should also mention that in order to use ufs-wm with ESMF v8.8.0 an update in GOCART was needed. In my tests, I used a branch @theurich created that has some updates for esmf 8.8.0. This one. So before we can actually switch to 8.8.0 this change will need to be merged in gocart. The only issue is that we (ufs-wm) are still using a gocart commit from Sep 2023. And Gerhard's changes are on top of that commit. Somebody will need to port Gerhard's changes to newer gocart and then update ufs-wm to use that version.

climbfuji · 2025-01-24T15:32:54Z

@mathomp4 or @DusanJovic-NOAA Would you mind creating an issue in github.com/jcsda/spack-stack to update MAPL to version x.y.z for spack-stack-1.9.0, and list changes to the build options (if applicable)? Thanks!

mathomp4 · 2025-01-24T15:40:24Z

I'm going to work on getting MAPL 2.53 out now. Shouldn't take me too long. Then you'll have a semver tag rather than a crazy git-hash.

ETA: Well, might take about 30-40 minutes. My final test is a full run on discover to make sure it is zero-diff, etc. and good ol' CMake has decided to rebuild the full model. I really hope ifx doesn't have this same CMake+ifort issue with recompiling!

mathomp4 · 2025-01-24T16:46:02Z

MAPL 2.53.0 has been released:

https://github.com/GEOS-ESM/MAPL/releases/tag/v2.53.0

And I have a PR to spack:

spack/spack#48712

and an issue for JCSDA/spack-stack:

JCSDA/spack-stack#1469

Though that last one might not be "complete" as I'm not sure if any updates are needed for how MAPL is built, etc.

climbfuji · 2025-01-24T17:18:17Z

Thanks @mathomp4 !

junwang-noaa added the enhancement New feature or request label Jun 28, 2024

junwang-noaa added this to Model infrastructure development FY2024Q4 Jun 28, 2024

jkbk2004 assigned RatkoVasic-NOAA, ulmononian and natalie-perlin Jul 2, 2024

ulmononian mentioned this issue Jul 2, 2024

Add mapl/2.46.2 and esmf/8.6.1 recipes to release/1.6.0 JCSDA/spack#446

Merged

junwang-noaa moved this to In Progress in Model infrastructure development FY2024Q4 Jul 22, 2024

junwang-noaa mentioned this issue Aug 12, 2024

Install ESMF 8.6.1 and MAPL 2.46.2 -> 2.46.3 NOAA-EMC/WCOSS2-requests#5

Open

junwang-noaa changed the title ~~Test MAPL v2.46.2 in UFS weather model~~ Test MAPL v2.46.3 in UFS weather model Aug 30, 2024

jkbk2004 mentioned this issue Sep 24, 2024

[INSTALL]: install ESMF 8.6.1 and MAPL 2.46.2 in spack-stack 1.6.0 JCSDA/spack-stack#1168

Open

14 tasks

junwang-noaa moved this to In Progress in Model infrastructure development FY25Q1 Sep 30, 2024

junwang-noaa added this to Model infrastructure development FY25Q1 Sep 30, 2024

DavidHuber-NOAA mentioned this issue Oct 4, 2024

Upgrade to Spack-Stack 1.9.0 NOAA-EMC/global-workflow#2984

Open

36 tasks

DavidHuber-NOAA mentioned this issue Nov 19, 2024

Update GDASApp to spack-stack/1.9.0 NOAA-EMC/GDASApp#1283

Open

junwang-noaa mentioned this issue Dec 2, 2024

[DRAFT] Remove local copies of FindESMF from all components, update ESMF to 8.6.1 and MAPL to 2.46.3 #2406

Draft

14 tasks

mathomp4 mentioned this issue Jan 24, 2025

[INSTALL]: MAPL 2.53.0 in spack-stack 1.9.0 JCSDA/spack-stack#1469

Open

DusanJovic-NOAA changed the title ~~Test MAPL v2.46.3 in UFS weather model~~ Test MAPL v2.53.0 in UFS weather model Jan 24, 2025

Test MAPL v2.53.0 in UFS weather model #2346

Test MAPL v2.53.0 in UFS weather model #2346

Comments

junwang-noaa commented Jun 28, 2024 • edited Loading

Description

Solution

Alternatives

Related to

junwang-noaa commented Jun 28, 2024

BrianCurtis-NOAA commented Jul 1, 2024

Hang-Lei-NOAA commented Jul 1, 2024 via email

lipan-NOAA commented Jul 2, 2024

Hang-Lei-NOAA commented Jul 2, 2024 via email

ulmononian commented Jul 2, 2024

jkbk2004 commented Sep 24, 2024

mathomp4 commented Sep 24, 2024

junwang-noaa commented Dec 2, 2024

mathomp4 commented Dec 2, 2024

junwang-noaa commented Dec 3, 2024

AlexanderRichert-NOAA commented Dec 3, 2024

mathomp4 commented Dec 3, 2024

junwang-noaa commented Dec 4, 2024

weiyuan-jiang commented Dec 4, 2024

DusanJovic-NOAA commented Jan 2, 2025

DeniseWorthen commented Jan 2, 2025

DusanJovic-NOAA commented Jan 2, 2025 • edited Loading

DeniseWorthen commented Jan 2, 2025

DusanJovic-NOAA commented Jan 2, 2025

DeniseWorthen commented Jan 2, 2025

junwang-noaa commented Jan 2, 2025

weiyuan-jiang commented Jan 2, 2025

DusanJovic-NOAA commented Jan 3, 2025

weiyuan-jiang commented Jan 6, 2025 • edited Loading

DusanJovic-NOAA commented Jan 6, 2025

weiyuan-jiang commented Jan 6, 2025

DusanJovic-NOAA commented Jan 23, 2025 • edited Loading

climbfuji commented Jan 23, 2025

mathomp4 commented Jan 23, 2025

DusanJovic-NOAA commented Jan 23, 2025

climbfuji commented Jan 23, 2025

mathomp4 commented Jan 23, 2025

DusanJovic-NOAA commented Jan 23, 2025

AlexanderRichert-NOAA commented Jan 24, 2025

DusanJovic-NOAA commented Jan 24, 2025

mathomp4 commented Jan 24, 2025

DusanJovic-NOAA commented Jan 24, 2025

DusanJovic-NOAA commented Jan 24, 2025 • edited Loading

climbfuji commented Jan 24, 2025

mathomp4 commented Jan 24, 2025 • edited Loading

mathomp4 commented Jan 24, 2025

climbfuji commented Jan 24, 2025

junwang-noaa commented Jun 28, 2024 •

edited

Loading

DusanJovic-NOAA commented Jan 2, 2025 •

edited

Loading

weiyuan-jiang commented Jan 6, 2025 •

edited

Loading

DusanJovic-NOAA commented Jan 23, 2025 •

edited

Loading

DusanJovic-NOAA commented Jan 24, 2025 •

edited

Loading

mathomp4 commented Jan 24, 2025 •

edited

Loading