Support IntelLLVM compiler

[DusanJovic-NOAA]

Commit Queue Requirements:

• Fill out all sections of this template.
• All sub component pull requests have been reviewed by their code managers.
• Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
• Commit 'test_changes.list' from previous step

---

Description:

Add preliminary support for Intel LLVM compiler. Added new top-level cmake/IntelLLVM.cmake configuration file. Update CMakeLists.txt in various submodules to recognize IntelLLVM as supported compiler and add Intel LLVM compiler flags.

Useful porting guide: https://www.intel.com/content/www/us/en/developer/articles/guide/porting-guide-for-ifort-to-ifx.html

At this moment I only tried to build the model on Hercules.

In order to use new compiler you need to set the following three environment variables:

export I_MPI_CC=icx
export I_MPI_CXX=icpx
export I_MPI_F90=ifx

Commit Message:

* UFSWM -  Add support for Intel LLVM compiler
  * CICE -  Add support for Intel LLVM compiler
  * FV3 -  Add support for Intel LLVM compiler
    * ccpp-physics -  Add support for Intel LLVM compiler
    * atmos_cubed_sphere -  Add support for Intel LLVM compiler 
    * upp -  Add support for Intel LLVM compiler 
  * WW3 - Add support for Intel LLVM compiler

Priority:

• Normal

Git Tracking

UFSWM:

• Closes Upgrade to Intel LLVM Compilers #2025

Sub component Pull Requests:

• FV3: Support IntelLLVM compiler NOAA-EMC/fv3atm#861
  • ccpp-physics: Support Intel LLVM compiler (ifx) ccpp-physics#220
  • atmos_cubed_sphere: Support IntelLLVM compiler NOAA-GFDL/GFDL_atmos_cubed_sphere#353
  • upp: Support IntelLLVM compiler NOAA-EMC/UPP#1001
• WW3: Add support for IntelLLVM compiler NOAA-EMC/WW3#1276
• CICE: CICE updates to support IntelLLVM compiler NOAA-EMC/CICE#91

UFSWM Blocking Dependencies:

• None

---

Changes

Regression Test Changes (Please commit test_changes.list):

• No Baseline Changes.

Input data Changes:

• None.

Library Changes/Upgrades:

• No Updates

---

Testing Log:

• RDHPCS
  • Hera
  • Orion
  • Hercules
  • Jet
  • Gaea
  • Derecho
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
• opnReqTest (complete task if unnecessary)

[BrianCurtis-NOAA]

Is this something for only Hercules, or will other systems be implementing IntelLLVM ?

[DusanJovic-NOAA]

Is this something for only Hercules, or will other systems be implementing IntelLLVM ?

Eventually we must switch to IntelLLVM on all platforms. When, I don't know. But classic Intel compilers are now deprecated, see:
https://www.intel.com/content/www/us/en/developer/articles/release-notes/oneapi-fortran-compiler-release-notes.html

NOTE: Intel® Fortran Compiler Classic (ifort) is now deprecated and will be discontinued in late 2024. Intel recommends that customers transition now to using the LLVM-based Intel® Fortran Compiler (ifx) for continued Windows* and Linux* support, new language support, new language features, and optimizations. For more information on ifx, see the Intel® Fortran Compiler Developer Guide and Reference and the Porting Guide for ifort Users to ifx.

[DusanJovic-NOAA]

Compilation with Intel LLVM on Hera fails with this error:

/scratch2/NCEPDEV/fv3-cam/Dusan.Jovic/ufs/support_intelllvm/ufs-weather-model/MOM6-interface/
MOM6/src/framework/MOM_unit_testing.F90(128): error #5533: Feature found on this line is not 
yet supported in ifx                                                                         
  test%proc => proc
--^
compilation aborted for /scratch2/NCEPDEV/fv3-cam/Dusan.Jovic/ufs/support_intelllvm/ufs-weath
er-model/MOM6-interface/MOM6/src/framework/MOM_unit_testing.F90 (code 3)
make[2]: *** [MOM6-interface/CMakeFiles/mom6_obj.dir/build.make:1765: MOM6-interface/CMakeFil
es/mom6_obj.dir/MOM6/src/framework/MOM_unit_testing.F90.o] Error 3

[DusanJovic-NOAA]

All coupled debug tests fail on Hercules with this error:

179: forrtl: severe (408): fort: (3): Subscript #3 of the array UVEL has value 0 which is less than the lower bound of 1
179: 
179: Image              PC                Routine            Line        Source             
179: ufs_model          000000000B5A1262  accum_hist               2318  ice_history.F90
179: ufs_model          000000000B4BFB6F  initializerealize         835  ice_comp_nuopc.F90
179: ufs_model          0000000000B94024  Unknown               Unknown  Unknown
179: ufs_model          0000000000B97F9F  Unknown               Unknown  Unknown
179: ufs_model          0000000000D0EBC7  Unknown               Unknown  Unknown
179: ufs_model          0000000000CFB022  Unknown               Unknown  Unknown

This is looks like a bug in the ifx.

[DeniseWorthen]

@NickSzapiro-NOAA Would you please take a look at what is happening here w/ CICE? You'll need to get from @DusanJovic-NOAA build instructions and maybe a sandbox run directory.

[NickSzapiro-NOAA]

Of course, @DeniseWorthen @DusanJovic-NOAA. Please let me know if there are any special instructions needed to reproduce. On first pass, it's curious that the ice speed may be the first call to accum_hist with an array that was not allocated:
https://github.com/NOAA-EMC/CICE/blob/emc/develop/cicecore/cicedyn/analysis/ice_history.F90#L2317-L2320

[DusanJovic-NOAA]

I was able to create a small test program that reproduces an error identical to the one we see in CICE. Dom (@climbfuji) confirmed that it works with the latest Intel LLVM compiler (ifx (IFX) 2024.1.0 20240308), so it is probably just a bug in the older versions.

Version on Hercules is (ifx (IFX) 2023.1.0 20230320). On Hera it's even older (ifx (IFORT) 2022.0.0 20211123), and that version can not even compile the code with MOM6. See above error.

We need to find a machine (NOAA R&D) on which we can get the latest version of Intel LLVM compiler (2024.1.0) installed.

[climbfuji]

I can install it on Hercules in the EPIC spack-stack space, but we still need to manage building all dependencies with it.

[junwang-noaa]

If it is confirmed that the latest Intel LLVM compiler fixes the issue, let's ask sysadmin to install the latest version.

[DusanJovic-NOAA]

I can install it on Hercules in the EPIC spack-stack space, but we still need to manage building all dependencies with it.

Eventually yes, but for now I think we can still build all libraries (spack-stack) with ifort (but the new version 2024) and only the model with ifx. Object files and module files should be binary compatible between ifort and ifx.

According to Intel:
... Binaries and libraries generated with ifort can be linked with binaries and libraries built with ifx, and .mod files generated with one compiler can be used by the other (64-bit targets only). Both compilers use the the same runtime libraries. ...

[climbfuji]

Except that there's no more icc anc icpc, so at least for those we have to switch to icx and icpx. It should be fairly straightforward to build just the dependencies for the ufs-weather-model with ifort and/or ifx and get all the other packages compiled in a second step.

[NickSzapiro-NOAA]

@DusanJovic-NOAA Does filling worka first (https://github.com/NOAA-EMC/CICE/blob/emc/develop/cicecore/cicedyn/analysis/ice_history.F90#L2321-L2331) give a temporary fix?

[DusanJovic-NOAA]

@DusanJovic-NOAA Does filling worka first (https://github.com/NOAA-EMC/CICE/blob/emc/develop/cicecore/cicedyn/analysis/ice_history.F90#L2321-L2331) give a temporary fix?

I think it should. Looks like having an array expression (that depends on iblk) as an actual subroutine argument is what fails.

[DusanJovic-NOAA]

Currently debug builds fail (on Hera) with the following error:

[100%] Linking Fortran executable ufs_model
ld: cannot find -lpioc
ld: attempted static link of dynamic object `/apps/oneapi/compiler/2023.2.0/linux/compiler/lib/intel64_lin/libiomp5.so'
make[2]: *** [CMakeFiles/ufs_model.dir/build.make:121: ufs_model] Error 1
make[1]: *** [CMakeFiles/Makefile2:328: CMakeFiles/ufs_model.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

That's why all debug tests are currently disabled in rt_intelllvm.conf.

All non-debug tests successfully compile, execute and reproduce it's own outputs, ie:

./rt.sh -l rt_intelllvm.conf -e -c
./rt.sh -l rt_intelllvm.conf -e -m  

is successful.

[DusanJovic-NOAA]

Regression test passed on Hera
RegressionTests_hera.log

[jkbk2004]

@DusanJovic-NOAA We can commit this pr today. Please, go ahead to sync up branch.

[DusanJovic-NOAA]

@jkbk2004 We would to update CICE submodule to use an updated PR and branch that includes both LLVM updates and some other updates from upstream. Results are expected to still be b4b.

[jkbk2004]

@jkbk2004 We would to update CICE submodule to use an updated PR and branch that includes both LLVM updates and some other updates from upstream. Results are expected to still be b4b.
@DusanJovic-NOAA I will confirm.

[DusanJovic-NOAA]

CICE submodule has been updated. Ready for regression testing.

[grantfirl]

@DusanJovic-NOAA We've been asked to add someone to our CODEOWNERS file in the ccpp-physics repo and we're trying to find a convenient PR to do this in so that we don't open a separate PR for this small task. Of course, it doesn't affect any testing, but you would need to merge a PR into ufs-community/ccpp-physics#220 and update submodule hashes. Would you be willing to do this? If not, no big deal. If yes, I'll prepare a PR into your branch for ufs-community/ccpp-physics#220.

[DusanJovic-NOAA]

@DusanJovic-NOAA We've been asked to add someone to our CODEOWNERS file in the ccpp-physics repo and we're trying to find a convenient PR to do this in so that we don't open a separate PR for this small task. Of course, it doesn't affect any testing, but you would need to merge a PR into ufs-community/ccpp-physics#220 and update submodule hashes. Would you be willing to do this? If not, no big deal. If yes, I'll prepare a PR into your branch for ufs-community/ccpp-physics#220.

Sure. I can do that. Please open PR to my ccpp-physics branch. @jkbk2004 Please hold on for few more minutes with starting RT.

[grantfirl]

@DusanJovic-NOAA We've been asked to add someone to our CODEOWNERS file in the ccpp-physics repo and we're trying to find a convenient PR to do this in so that we don't open a separate PR for this small task. Of course, it doesn't affect any testing, but you would need to merge a PR into ufs-community/ccpp-physics#220 and update submodule hashes. Would you be willing to do this? If not, no big deal. If yes, I'll prepare a PR into your branch for ufs-community/ccpp-physics#220.

Sure. I can do that. Please open PR to my ccpp-physics branch. @jkbk2004 Please hold on for few more minutes with starting RT.

@DusanJovic-NOAA Thanks. See DusanJovic-NOAA/ccpp-physics#1

[DusanJovic-NOAA]

Merged ccpp-physics PR, and updated submodules. Please double check and if everything looks good I think we can finally start RT

[grantfirl]

Merged ccpp-physics PR, and updated submodules. Please double check and if everything looks good I think we can finally start RT

@DusanJovic-NOAA Thanks for granting this last-minute request!

[NickSzapiro-NOAA]

The CICE #ifndef LLVM PR was merged at CICE-Consortium. Here is a new EMC/CICE branch (https://github.com/NickSzapiro-NOAA/CICE/tree/cice_llvm) and PR (NOAA-EMC/CICE#93) for use in this PR.

It is the same lines of code for us in CICE and cleaner git history. If can use, there is no need to repeat testing.