Porting UFS Coastal to custom platform

[uturuncoglu]

@pvelissariou1 @saeed-moghimi-noaa @hga007 I am creating this to continue conversation about porting ufs-coastal to new platform and also create documentation about it for the community.

[saeed-moghimi-noaa]

This sounds very good. We need more of https://github.com/oceanmodeling/ufs-coastal/wiki/Current-Status-of-UFS%E2%80%90Coastal-Implementation regular updates and adding more wiki's and howto's.

Thanks,

[uturuncoglu]

@saeed-moghimi-noaa I think we could use wiki for some quick update and maybe instructions but we need to go with Sphinx for more detailed documentation since it is commonly used by UFS. I could create the initial version of it and put instruction here for compiling and pushing documentation updates.

[uturuncoglu]

@pvelissariou1 I started looking to stack-stack and define the additional dependencies that is required for the ufs-coastal. There is my list,

• Metis / https://github.com/spack/spack/tree/develop/var/spack/repos/builtin/packages/metis
• Parmetis - https://github.com/spack/spack/tree/develop/var/spack/repos/builtin/packages/parmetis
• Proj - https://github.com/spack/spack/tree/develop/var/spack/repos/builtin/packages/proj
• Proj4-fortran (I think this is fortran interface for Proj) - this is not available through the spack and needs to be added. I think that the original location of this package is https://github.com/mhagdorn/proj4-fortran but I could not see version 1.1.0. I think we need to contact with the developer and add it to the spack.
• Julian - i am not sure about this. Who developed this library? Is there any GitHub end for it. This also needs to be added to spack if we want to manage those dependencies from central place.

Anyway, we need to make libraries consistent along the components. So, everybody could use same stack. @saeed-moghimi-noaa @pvelissariou1 Please let me know if you know any limitations in terms of used library in the components. I think that Metis is the common one (I am not sure about parmetis). We need to specify single version of each additional library.

[uturuncoglu]

Maybe we need to install Proj4-fortran and Julian externally outside of the spack environment and use as externals. Since they are not actively development and outside of the spack. This could be done in app level. Any suggestion?

[pvelissariou1]

@uturuncoglu ParMETIS has licensing issues and cannot be officially ported in NOAA applications. Newer versions of WW3 will use the scotch library. I don't know what the SCHISM developers will do. Julian code should be part of FVCOM source code or, they need to update their sources to account for the functionality of Julian. It is a very old library and will not be part of spack.

[pvelissariou1]

@uturuncoglu Proj4-fortran can also be compiled within the FVCOM source code (as is METIS in ADCIRC for example). It cannot be an external library. For both issues I have already talked with the FVCOM developers. They need to start working on their source code as soon as possible. I'll talk to them this week.

[pvelissariou1]

@uturuncoglu For the proj library, FVCOM only supports version 4* at this point (and proj4-fortran is not tested with versions of proj > 4). We might need to talk with the FVCOM developers about this as well.

[pvelissariou1]

@uturuncoglu , @saeed-moghimi-noaa As a first iteration we might need to stick with spack-stack but eventually we need to have a UFS-Coastal to be more portable than this.

[pvelissariou1]

@uturuncoglu Regarding the documentation, let's start with sphinx to create the basic documentation, but in the end we would like to have an extended documentation on NUOPC caps within UFS-Coastal with fortran calls, code trees etc. I am not sure if Sphinx can do that for Fortran (and in what level). We might need to use multiple documenters to create the complete end documentation (user and developer). Let's both of us work on this.

[uturuncoglu]

@pvelissariou1 I think we could go with spack-stack. It seems it is flexible enough. As I mentioned before we could install couple of libraries externally through the app. For parmetis, we could go with spack installation for now until models start to use scotch library. Since we are not plaining to use ufs-coastal in operational machine in short term going with the exiting libraries for now could be the solution. Switching scotch could take time for the components. Anyway, I am teasing stack-stack on Orion now and I'll update you about it. Once I have successful installation and also tested UFS Costal with it we could start populate the documentation with Sphinx.

[pvelissariou1]

@uturuncoglu Ok, then do your testing on Orion with spack-stack. On Hercules they are using spack. Soon we will check both implementations I guess.

[uturuncoglu]

@pvelissariou1 spack-stack is additional layer top of spack to make build easy. Even if you are seeing spack, the modules are built with spack-stack. So, spack-stack in the top level to install everything rather than installing through spack itself.

[pvelissariou1]

@uturuncoglu , @saeed-moghimi-noaa
Porting issues on Hercules/Paraller Works/Hera:

• Hercules: porting failed
  - Using "hpc-stack" is not feasible due to compiler versions mismatch (both Intel and GNU). So compiled libraries cannot be loaded.
  - We need to move to spack-stack as soon as possible (action item)
  - In general additional libraries like METIS/ParMETIS, for example, can be compiled and installed through the spack system
  - Very specific libraries (old/outdated) that required by a specific model (e.g., FVCOM) should be part of the model's source and automatically compiled/linked (statically) along with the compilation of the model. This approach should be the norm.
• ParallelWorks: porting failed
  - Using "hpc-stack" is not possible due to the fact that the /scratch1, /scratch2 filesystems are not shared
  - For other comments, see the "Hercules" section
• Hera: porting successful
  - The folder /scratch2/STI/coastal/save/UFS-Coastal contains all required files and it is readable and writable by all users
  - For other comments, see the "Hercules" section

[uturuncoglu]

@pvelissariou1 Hercules support with spack-stack was completed for both Intel and GNU last week (ufs-community#1733). We need to sync UFS-Coastal to bring those changes. Last week I tried to install spack-stack on Orion and try to run ufs wether model with the newly compiled stack but it was failing. I'll look at it more to find the issue. Once that works I'll try to add additional libraries like Metis/parmetis using chaining feature. In any case, the component needs to get rid of Metis to used under operational machine. I know this is a big transition but we might need to inform the component developers in advance.

[pvelissariou1]

@uturuncoglu Thank you. Let's work together on this to port UFS-Coastal to Hercules as soon as possible. It is going to be a usefule learning curve for me.

[uturuncoglu]

@pvelissariou1 Sure. I think once we sync the model, probably it will be run without any issue. Let's discuss it tomorrow.

[pvelissariou1]

@tufuk Yes, that is why I put my thoughts here

[uturuncoglu]

@pvelissariou1 @saeed-moghimi-noaa I sync ufs-coastal with ufs-weather-model as of today. Now it has Hercules support. At this point, I only tested ROMS RT with the synced model and everything is working as expected. ROMS now points its official repository (develop branch) too. I'll keep continue to test other components and sync the configurations with CoastalApp test suite and fix the issues if I have. In the meantime, please let me know if you have any issues. BTW, after sync now ufs-coastal is using ESMF 8.4.2 (the previous one was using 8.3.0b09).

[hga007]

It worked well using ESMF 8.4.2 on my computer with JEDI Spack-Stack. Gosh, you guys have so many computers to run. Is Hercules at MSU, too?

[pvelissariou1]

@hga007 Hercules is at MSU as well. Still not in full production mode though.

[uturuncoglu]

I was testing custom installation on Orion and hit minor problem. Here is the details of the issue: JCSDA/spack-stack#807. My plan is to install first to a known platform. Then, if everything goes fine, I am planning to chain extra dependencies like parmetis, Metis, proj etc. Some of the extra dependencies can be installed manually (it would be part of functionality provided by the ufs-coastal-app) since they don't have package under spack or spack-stack. After that we could create set of instructions for the user to port UFS Coastal to custom platforms. Eventually, these steps will be part of app.

[uturuncoglu]

@pvelissariou1 @saeed-moghimi-noaa I could able to run ufs-coastal with custom spack-stack build under Orion. At this point, I am trying to run it under Docker container and populate the instructions for it. When I have successful build and run (the model might need some minor fixes in terms of CMake interfaces of the ufs-coastal components but I think I could fix them), I'll push those information to app level documentation. So, you could also try to test.

[uturuncoglu]

@pvelissariou1 @saeed-moghimi-noaa I was working with spack-stack and about porting ufs-coastal. Since I have successful build on Orion, I decided to work on a Docker container (Ubuntu Linux) to see what happens if I go with totally unsupported platform. The good news is that I could able to install all the dependencies and build at leas SCHSIM case under container using gcc. I'll create a section about in the app level documentation for the users. I think this effort could be also extrapolated to any other linux cluster. Still following needs to be done,

• Chaining extra dependencies for UFS Coastal. Some additional libraries are not supported and also not exist under spack. So, we could create script to install them manually.
• The app level script to install all dependencies. Make required changes. There are some steps that requires manual intervention to spack-stack installation and those can be also part of our script. I think eventually spack-stack will be more mature.
• I did not run RT yet since that also requires some changes in regression testing system (i.e. rt.sh).
• I only build SCHSIM and fix couple of issues in its Cmake interface and plain to test also other components.
• I also did not try to run any configuration under container yet.

[saeed-moghimi-noaa]

@uturuncoglu @pvelissariou1

Thanks for the update. Good progress. I want to ask @SorooshMani-NOAA to chime in here in regards to using Singularity instead of Docker for UFS-Coastal, Also formed a list of a possible action items to check off:

• Chaining extra dependencies for UFS Coastal. Some additional libraries are not supported and also not exist under spack. So, we could create script to install them manually.
• The app level script to install all dependencies. Make required changes. There are some steps that requires manual intervention to spack-stack installation and those can be also part of our script. I think eventually spack-stack will be more mature.
• I did not run RT yet since that also requires some changes in regression testing system (i.e. rt.sh).
• I only build SCHSIM and fix couple of issues in its Cmake interface and plain to test also other components.
• I also did not try to run any configuration under container yet.

[SorooshMani-NOAA]

@saeed-moghimi-noaa reading the comments above my understanding is that Docker was tested just to see if spack-stack is successful or not in that environment. In that case it's a valid test, however if one wants to run tests regularly using container, especially on HPC platforms, then singularity (i.e. apptainer) containers is the way to go as far as I understand.

Also note that even if one wants to use Singularity on HPC there's a valid usecase for Docker, and that is to create a image with the dependencies to start from when building the Singularity image. This is similar to what @pvelissariou1 is doing for compilation tests of CoastalApp, but then instead of running GitHub it's going to run Singularity to start from the Docker image of interest.

All in all, both Docker and Singularity have their own merit depending on what is the use-case. That's my 2 cents.

[saeed-moghimi-noaa]

@saeed-moghimi-noaa reading the comments above my understanding is that Docker was tested just to see if spack-stack is successful or not in that environment. In that case it's a valid test, however if one wants to run tests regularly using container, especially on HPC platforms, then singularity (i.e. apptainer) containers is the way to go as far as I understand.

Also note that even if one wants to use Singularity on HPC there's a valid usecase for Docker, and that is to create a image with the dependencies to start from when building the Singularity image. This is similar to what @pvelissariou1 is doing for compilation tests of CoastalApp, but then instead of running GitHub it's going to run Singularity to start from the Docker image of interest.

All in all, both Docker and Singularity have their own merit depending on what is the use-case. That's my 2 cents.

Thanks @SorooshMani-NOAA .

[uturuncoglu]

@pvelissariou1 Okay. I know proj needs to be version 4. The latest version is 4.9.2 for version 4 (btw, the last version available with spack is 8.2.1). Does it work fine with the FVCOM? Do we have any restriction in terms of used Metis/parmetis libraries. What is the common version that can be used by all the components?

[pvelissariou1]

@uturuncoglu No it doesn't work with FVCOM. The model requires version 4 (4.9.2) is what I use in CoastalApp. Now, proj is not exactly required by FVCOM except special cases. I will again talk to them. Regarding Julian and proj-fortran libraries, these should be part of the FVCOM source tree and be compiled internally along with the FVCOM. These libraries can be in spack. They asked me to do this for them among other things but I don't have time at this point. I will again talk with them.

[uturuncoglu]

@pvelissariou1 Thanks. I think Julian and proj-fortran libraries needs to be handled internally since we have no packages in the spack side and I think we don't want to maintain those packages under spack. So, at this point just proj and Metis libraries can be handled by spack.

[pvelissariou1]

@uturuncoglu Correct, please let's focus on proj and METIS in spack.

[uturuncoglu]

@pvelissariou1 I could able to build SCHSIM against external parmetis installation. When I try to install ADCIRC I am getting error like following undefined reference to metis_estimatememory_'. The main reason is that ADCIRC is not compatible with Metis 5 and requires Metis 4.0.3 but parmetis spack-stack package can be only installed with Metis 5. At this point I am not sure what we need to do. I think that we need to update ADCIRC to use Metis 5 rater than Metis 4. The only function is metis_estimatememory_'` and it is used in the following sources,

../ADCIRC-interface/ADCIRC/prep/metis.F:        EXTERNAL metis_partgraphkway,metis_estimatememory
../ADCIRC-interface/ADCIRC/prep/metis.F:      CALL metis_estimatememory(MNP,XADJ,ADJNCY,NUMFLAG,
../ADCIRC-interface/ADCIRC/prep/metis.F:        EXTERNAL metis_estimatememory

I think this can be removed but not sure how it will affect. @dwirasae do you have any idea?

[uturuncoglu]

@pvelissariou1 BTW, I already push the changes for SCHSIM and now support both internal build and also use external parmetis. I have new version of ADCIRC build interface that supports both way (not working yet due to conflict in the Metis version but once we solve it I think it will build without any issue). I'll push that one also to repo.

[uturuncoglu]

@pvelissariou1 @dwirasae If i remove metis_estimatememory references from ADCIRC/prep/metis.F it builds but I did not try to run. I am not sure prep will run with Metis 5 or not.

[pvelissariou1]

@uturuncoglu @saeed-moghimi-noaa Ufuk, I think ParMETIS should not be part of spack as this will raise issues with NOAA. It is good to have the capability to build and use ParMETIS though in case we (or the user) need to do so. Regarding ADCIRC and METIS, my understanding is that @dwirasae is working on bringing METIS 5 into ADCIRC. The issue is that it will be a modified METIS to support Adaptive Message Passing Interface (AMPI). I don't know what kind of modifications @dwirasae or others are implementing and how to cope with these. @dwirasae Dam, could you please chime in?

[uturuncoglu]

@pvelissariou1 in the way that I used is not part of the spack-stack or model itself. We are just using low level spack commands to install it. So, it is not different from installing through the component CMakelist interface. Once we get rid of Métis dependency we could remove those extra commands.

[uturuncoglu]

@pvelissariou1 I found an issue in spack proj package and open an issue: spack/spack#40456 it will be fixed soon. I have a workaround for it until it appears in the spack-stack which allows me to modify FVCOM Cmake interface.

[pvelissariou1]

Great, thank you. What was the issue?

[uturuncoglu]

@pvelissariou1 @saeed-moghimi-noaa @yunfangsun I updated the documentation to include information about installing spack-stack and also using it under ufs-coastal. I still need to test the RT etc. but it is a good starting point. Anyway, if you have time please try those instructions and let me know if you have any issues/questions. The updated documentation is in following link: https://oceanmodeling.github.io/ufs-coastal-app/versions/main/html/porting.html#docker-container

[pvelissariou1]

@uturuncoglu Thank you Ufuk, I will follow on this as well.

[pvelissariou1]

Finished & presented tutorial # 1.
Preparations started for tutorial # 2 on Frontera (SCHISM & possibly an atm2fvc case)

[janahaddad]

Moving documentation issue to #57 so that this issue can be focused on porting to custom platform

[uturuncoglu]

@janahaddad @pvelissariou1 @saeed-moghimi-noaa I have just updated spack-stack installation on Frontera. spack-stack 1.6.0 version is installed without any issue and we have minor issue with develop. The draft PR is in here: JCSDA/spack-stack#1075 and the MAPL library issue is documented in JCSDA/spack-stack#1076. I still need to test in UFS Coastal side and update the module file.

[saeed-moghimi-noaa]

@uturuncoglu @pvelissariou1

Would it be possible to develop a how to on how to stand up a spack-stack for ufs-coastal?

Thanks

[uturuncoglu]

@saeed-moghimi-noaa We have a script in the app level but it is just tested on container. It requires more work. Also we have app level basic documentation. In Spack-stack side there is a plan to simplify the build process for university platforms etc. spack-stack has a script for Cloud based installation. So, there are planing to do the same for the clusters. I hope it would be easier in the near future since there are lots of pressure coming from community.