Move spack-stack repos into spack submodule, use two-level namespace …

…for openmpi and mpich, add package py-pyhdf, use only one macOS reference config (#131)

*Finalizes the move of the `spack-stack` repos into the `spack` submodule
    - This requires adding the projections/name translations for `ecmwf-atlas` to `atlas` in the common module config

* Enables use of two-level namespace for `openmpi` and `mpich` (the former required making changes to the package config in `spack`, for the latter this was already implemented)
    - Also need to export `OMPI_MCA_rmaps_base_oversubscribe=1` in the `openmpi` module so that one can use more tasks than CPU cores (for unit testing)

- Fixes several problems with CI builds on Linux and macOS
    - Use a combination of package-level build parallelism and install-level build parallelism (see https://spack.readthedocs.io/en/latest/packaging_guide.html?highlight=parallel#parallel-builds) to speed up the Ubuntu Intel build

* Adds the existing `hdf` and new `py-pyhdf` packages
    - Requires setting the default provider for `jpeg` to `libjpeg-turbo` (already part of our spack-stack)

* Updates one of the macOS reference configs to `macos-monterey-apple-clang-openmpi-reference` (use `apple-clang` instead of (llvm) `clang`) and removes the macOS reference config for easier maintenance
    - This reference config now builds openmpi-4.1.3 built by `spack` with two-level namespace enabled
    - Note that I tested the same config with `mpich-3.4.2` built by `spack` with two-level namespace enabled

* Adds the capability to create containers with `spack`, all built into the existing scripts
    - `create-env.py` is renamed to `create.py environment ...` for creating local environments; Github unfortunately doesn't see this and thinks it is a new file and the old one got deleted (use a graphical diff tool locally to compare)

* Containers are built using `create.py container ...`
- One example is added: `configs/containers/docker-ubuntu-gcc-openmpi.yaml`

* Comments out `git-lfs` in the common package config, because pinning a version there isn't needed (we don't build `git-lfs`, but require it due to the dependencies on `go`/`go-bootstrap`, which are difficult to build) and having it in there breaks the container builds

* Updates and consolidates the documentation in `README.md`, `config/sites/README.md` and `config/common/README.md` into a single (top-level) `README.md`
    - Note: In my next PR, I will add the capability to create HTML (and possibly LaTeX/PDF) documentation using `sphinx`

.github/actions/setup-spack-stack/action.yaml

@@ -34,12 +34,18 @@ runs:
     shell: bash
     run: |
       if [[ "$RUNNER_OS" == "Linux" ]]; then
+        # Needed for the following apt-get install calls to work
+        sudo apt-get update
+
         # Install Curl headers. Executable exists by default in spack external find.
         sudo apt-get install libcurl4-openssl-dev
 
         # Install git-lfs to avoid compilation errors of "go" with Intel
         sudo apt-get install git-lfs
 
+        # Install krb5 to avoid compilation errors of "hdf" on Ubuntu
+        sudo apt-get install krb5-user libkrb5-dev
+
         if [[ "${{ inputs.compiler }}" == "intel"* ]]; then
             cd /tmp
             wget https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB
@@ -49,6 +55,15 @@ runs:
             sudo apt-get update
             sudo apt-get install intel-oneapi-dev-utilities intel-oneapi-openmp intel-oneapi-compiler-fortran intel-oneapi-compiler-dpcpp-cpp-and-cpp-classic intel-oneapi-mpi-devel
             echo "source /opt/intel/oneapi/setvars.sh" >> ~/.bash_profile
+            #
+            # DH*
+            # For now, install qt5 only for Intel to shorten the build time
+            # without having to rerun all other CI tests. Later, should consider
+            # using https://github.com/jurplel/install-qt-action for all runners
+            # and get rid of the macOS homebrew installation/package config
+            #sudo apt-get install qt5-default qttools5-dev-tools
+            sudo apt-get install qt5-default
+            # *DH
         fi
       elif [[ "$RUNNER_OS" == "macOS" ]]; then
         # These are already installed
@@ -107,14 +122,34 @@ runs:
           tar -xzf openmpi-${OPENMPI_VERSION}.tar.gz
           cd openmpi-${OPENMPI_VERSION}
           # --with-hwloc=internal --with-libevent=internal : https://www.open-mpi.org/faq/?category=building#libevent-or-hwloc-errors-when-linking-fortran
-          ./configure --prefix=$HOME/mpi --enable-mpi-fortran --enable-mpi-cxx --with-hwloc=internal --with-libevent=internal
+          if [[ "$RUNNER_OS" == "macOS" ]]; then
+            ./configure --prefix=$HOME/mpi \
+                --enable-mpi-fortran --enable-mpi-cxx \
+                --with-hwloc=internal --with-libevent=internal \
+                --with-wrapper-ldflags="-Wl,-commons,use_dylibs" \
+                LIBS="-Wl,-commons,use_dylibs"
+          else
+            ./configure --prefix=$HOME/mpi \
+                --enable-mpi-fortran --enable-mpi-cxx \
+                --with-hwloc=internal --with-libevent=internal
+          fi
           make -j2
           make install
         elif [[ "${{ inputs.mpi }}" == "mpich"* ]]; then
           wget http://www.mpich.org/static/downloads/${MPICH_VERSION}/mpich-${MPICH_VERSION}.tar.gz
           tar -xzf mpich-${MPICH_VERSION}.tar.gz
           cd mpich-${MPICH_VERSION}
-          ./configure --prefix=$HOME/mpi --enable-fortran --enable-cxx  --with-device=ch4:ofi
+          if [[ "$RUNNER_OS" == "macOS" ]]; then
+            ./configure --prefix=$HOME/mpi \
+                --enable-fortran --enable-cxx \
+                --with-device=ch4:ofi \
+                --enable-two-level-namespace \
+                LIBS="-Wl,-commons,use_dylibs"
+          else
+            ./configure --prefix=$HOME/mpi \
+                --enable-fortran --enable-cxx \
+                --with-device=ch4:ofi
+          fi
           make -j2
           make install
         fi
@@ -128,7 +163,7 @@ runs:
       fi
 
       source setup.sh
-      ./create-env.py --site default --app ${{ inputs.app }} --name ${{ inputs.app }}
+      ./create.py environment --site default --app ${{ inputs.app }} --name ${{ inputs.app }}
       spack env activate envs/${{ inputs.app }}
 
       # LLVM Clang not in PATH, search for it specifically
@@ -149,7 +184,7 @@ runs:
       # So Spack can find external MPI
       export "PATH=$HOME/mpi/bin:${PATH}"
 
-      # No external find for intel-oneapi-mpi, and problems finding other MPI libraries as well
+      # No external find for intel-oneapi-mpi, 
       # And no way to add object entry to list using "spack config add"
       # Add this first so "spack config add packages:" will append to this entry
       if [[ "${{ inputs.mpi }}" == "intel-oneapi-mpi" ]]; then
@@ -163,6 +198,25 @@ runs:
         echo "        prefix: /opt/intel/oneapi" >> ${SPACK_ENV}/spack.yaml
       fi
 
+      # Same for qt@5 on macOS or on Linux w/ Intel
+      if [[ "$RUNNER_OS" == "macOS" ]]; then
+        echo "" >> ${SPACK_ENV}/spack.yaml
+        echo "  packages:" >> ${SPACK_ENV}/spack.yaml
+        echo "    qt:" >> ${SPACK_ENV}/spack.yaml
+        echo "      buildable: False" >> ${SPACK_ENV}/spack.yaml
+        echo "      externals:" >> ${SPACK_ENV}/spack.yaml
+        echo "      - spec: qt@5" >> ${SPACK_ENV}/spack.yaml
+        echo "        prefix: /usr/local/opt/qt@5" >> ${SPACK_ENV}/spack.yaml
+      #elif [[ "${{ inputs.compiler }}" == "intel"* ]]; then
+      #  echo "" >> ${SPACK_ENV}/spack.yaml
+      #  echo "  packages:" >> ${SPACK_ENV}/spack.yaml
+      #  echo "    qt:" >> ${SPACK_ENV}/spack.yaml
+      #  echo "      buildable: False" >> ${SPACK_ENV}/spack.yaml
+      #  echo "      externals:" >> ${SPACK_ENV}/spack.yaml
+      #  echo "      - spec: qt@5" >> ${SPACK_ENV}/spack.yaml
+      #  echo "        prefix: /usr" >> ${SPACK_ENV}/spack.yaml
+      fi
+
       # Spack external find is by default only looking for build-tools. Either
       # search for additional packages explicitly, or fall back to the old
       # behavior to find all external packages.
@@ -177,6 +231,10 @@ runs:
       source setup.sh
       spack env activate ${{ inputs.path }}/envs/${{ inputs.app }}
 
+      # Speed up builds using two build jobs per package and four
+      # jobs in parallel further down (spack install)
+      spack config add "config:build_jobs:2"
+
       # Use external MPI to save compilation time
       spack config add "packages:mpi:buildable:False"
       spack config add "packages:all:providers:mpi:[${{ inputs.mpi }}]"
@@ -193,6 +251,10 @@ runs:
       # Error when proj attempts to build using external on macOS
       spack config rm "packages:ncurses"
 
+      # Remove external m4 entry
+      # Error when cairo attempts to build using external on macOS
+      spack config rm "packages:m4"
+
       spack config add "packages:all:compiler:[${{ inputs.compiler }}]"
 
       # Currently, this is the case for all workflows - wrap in if statement
@@ -207,9 +269,6 @@ runs:
         # Clang has trouble finding -lomp for OpenMP because of non-standard location
         # Could be fixed by setting compiler environment LDFLAGS to -L$(brew --prefix llvm)/lib or wherever libomp is
         spack config add "packages:wgrib2:variants: ~openmp"
-        # DH* This should no longer be necessary
-        spack config add "packages:boost:version:[1.78.0]"
-        # *DH
         spack config add "packages:fms:variants: +64bit +enable_quad_precision +gfs_phys ~openmp +pic"
         spack config add "packages:fms-jcsda:variants: +64bit +enable_quad_precision +gfs_phys ~openmp +pic"
       fi
@@ -225,11 +284,26 @@ runs:
 
   - name: build-env
     shell: bash
+    env:
+      OMP_NUM_THREADS: 16
     run: |
       cd ${{ inputs.path }}
       source setup.sh
       spack env activate ${{ inputs.path }}/envs/${{ inputs.app }}
-      spack install --fail-fast
+      # DH* For now only with Intel to avoid having to rerun all CI tests
+      if [[ "${{ inputs.compiler }}" == "intel"* ]]; then
+        for i in {1..8}; do
+          nohup spack install --fail-fast >> spack_install.log 2>&1 &
+        done
+        #
+        while ps -ef | grep -ve 'grep' | grep 'spack install'; do
+          echo "Still running"
+          tail -n 10 spack_install.log
+          sleep 60
+        done
+      else
+        spack install --fail-fast
+      fi
 
   - name: create-meta-modules
     shell: bash

.github/workflows/create-spack-mirror.yaml

@@ -26,7 +26,7 @@ jobs:
       - name: create-env
         run: |
           source ./setup.sh
-          ./create-env.py --app ${{ github.event.inputs.app }}  --site ${{ github.event.inputs.site }}  --name mirror
+          ./create.py environment --app ${{ github.event.inputs.app }}  --site ${{ github.event.inputs.site }}  --name mirror
           spack env activate -d envs/mirror
           spack compiler find
           spack add re2c

README.md

@@ -5,8 +5,8 @@ spack-stack is a collaborative effort between the NOAA Environmental Modeling Ce
 [spack](https://github.com/spack/spack) is a community-supported, multi-platform, Python-based package manager originally developed by the Lawrence Livermore National Laboratory (LLNL; https://computing.llnl.gov/projects/spack-hpc-package-manager). It is provided as a submodule so that a stable version can be referenced. [See the Spack Documentation for more information](https://spack.readthedocs.io/en/latest/)
 
 spack-stack is mainly a collection of Spack configuration files, but provides a few Python scripts to simplify the installation process:
-- `create-env.py` is provided to copy common, site-specific, and application-specific configuration files into a coherent Spack environment
-- `meta_modules/setup_meta_modules.py` creates compiler, MPI and Python meta-modules for a convenient setup of a user environment using modules (currently lua only)
+- `create.py` is provided to copy common, site-specific, and application-specific configuration files into a coherent Spack environment and to create container recipes
+- `meta_modules/setup_meta_modules.py` creates compiler, MPI and Python meta-modules for a convenient setup of a user environment using modules (lua and tcl)
 
 spack-stack is maintained by:
 - Kyle Gerheiser (@kgerheiser), NOAA-EMC
@@ -32,7 +32,6 @@ For questions or problems, please consult the currently open [issues](https://gi
 **Note. spack-stack is in early development and not yet ready for use. Instructions may be incomplete or invalid.**
 
 ## Quickstart
-
 ```
 git clone https://github.com/NOAA-EMC/spack-stack.git
 cd spack-stack
@@ -42,12 +41,18 @@ cd spack-stack
 # Sources Spack from submodule and sets ${SPACK_STACK_DIR}
 source setup.sh
 
+# Basic usage of create.py
+./create.py -h
+```
+
+### Create local environment
+```
 # See a list of sites and apps
-./create-env.py -h
+./create.py environment -h
 
-# Creates a pre-configured Spack environment in envs/<app>.<site>
-# Copies site-specific, application-specific, and common config files into the environment directory
-./create-env.py --site hera --app jedi-fv3 --name jedi-fv3.hera
+# Create a pre-configured Spack environment in envs/<app>.<site>
+# (copies site-specific, application-specific, and common config files into the environment directory)
+./create.py environment --site hera --app jedi-fv3 --name jedi-fv3.hera
 
 # Activate the newly created environment
 # Optional: decorate the command line prompt using -p
@@ -57,10 +62,9 @@ source setup.sh
 spack env activate [-p] envs/jedi-fv3.hera
 
 # Optionally edit config files (spack.yaml, packages.yaml compilers.yaml, site.yaml)
-cd envs/jedi-fv3.hera
-emacs spack.yaml
-emacs common/*.yaml
-emacs site/*.yaml
+emacs envs/jedi-fv3.hera/spack.yaml
+emacs envs/jedi-fv3.hera/common/*.yaml
+emacs envs/jedi-fv3.hera/site/*.yaml
 
 # Process the specs and install
 # Note: both steps will take some time!
@@ -74,6 +78,53 @@ spack module lmod refresh
 ./meta_modules/setup_meta_modules.py
 ```
 
+### Create container
+```
+# See a list of preconfigured containers
+./create.py container -h
+
+# Create container spack definition (spack.yaml) in directory envs/<spec>.<config>
+./create.py container --config docker-ubuntu-gcc-openmpi --spec esmf
+
+# Descend into container environment directory
+cd envs/esmf.docker-ubuntu-gcc-openmpi
+
+# Optionally edit config file
+emacs spack.yaml
+
+# Docker: create Dockerfile and build container
+# See section "container" in spack.yaml for additional information
+spack containerize > Dockerfile
+docker build -t myimage .
+docker run -it myimage
+```
+
+## Generating new site config
+Recommended: Start with an empty (default) site config. Then run `spack external find` to locate common external packages such as git, Perl, CMake, etc., and run `spack compiler find` to locate compilers in your path. Compilers or external packages with modules need to be added manually.
+```
+./create.py environment --site default --app jedi-ufs --name jedi-ufs.mysite
+
+# Descend into site config directory
+cd envs/jedi-ufs.mysite/site
+
+# Find external packages and compilers, output the files here
+# (overwrites packages.yaml and compilers.yaml)
+SPACK_SYSTEM_CONFIG_PATH=`pwd` spack external find --all --scope system
+SPACK_SYSTEM_CONFIG_PATH=`pwd` spack compiler find --scope system
+
+# Optionally edit config files as above in the quickstart section
+
+# Optionally attempt to find additional packages by name,
+# for example: "spack external find wget"
+```
+It is also instructive to peruse the GitHub actions scripts in `.github/workflows` and `.github/actions` to see how automated spack-stack builds are configured for CI testing, as well as the existing site configs in `configs/sites`.
+
+## Known issues
+
+### General
+1. First call to `spack concretize` fails with `[Errno 2] No such file or directory: ... .json`
+This can happen when `spack concretize` is called the very first time in a new spack-stack clone, during which the boostrapping (installation of clingo) is done first. Simply rerunning the command should solve the problem.
+
 ## Disclaimer
 
 The United States Department of Commerce (DOC) GitHub project code is

configs/apps/nceplibs/spack.yaml

@@ -6,4 +6,4 @@ spack:
   @CONFIG_INCLUDES@
 
   specs:
-    - nceplibs-bundle
+    - nceplibs-bundle-env

configs/common/config.yaml

@@ -26,3 +26,6 @@ config:
   # Cache directory for miscellaneous files, like the package index.
   # This can be purged with `spack clean --misc-cache`
   misc_cache: ${SPACK_STACK_DIR}/cache/misc_cache
+
+  # Timeout in seconds for web requests (curl/urllib)
+  connect_timeout: 60

configs/common/modules.yaml

@@ -9,6 +9,7 @@ modules:
       projections:
         all: '{compiler.name}/{compiler.version}/{name}/{version}'
         ^mpi: '{^mpi.name}/{^mpi.version}/{compiler.name}/{compiler.version}/{name}/{version}'
+        ecmwf-atlas: '{compiler.name}/{compiler.version}/atlas/{version}'
         nlohmann-json: '{compiler.name}/{compiler.version}/json/{version}'
         nlohmann-json-schema-validator: '{compiler.name}/{compiler.version}/json-schema-validator/{version}'
       blacklist:
@@ -95,6 +96,10 @@ modules:
             '{name}_ROOT': '{prefix}'
         suffixes:
           +debug: 'debug'
+      ecmwf-atlas:
+        environment:
+          set:
+            'atlas_ROOT': '{prefix}'
       esmf:
         environment:
           set:
@@ -107,13 +112,18 @@ modules:
         suffixes:
           ^esmf@8.2.0~debug: 'esmf-8.2.0'
           ^esmf@8.2.0+debug: 'esmf-8.2.0-debug'
+      openmpi:
+        environment:
+          set:
+            'OMPI_MCA_rmaps_base_oversubscribe': '1'
 
     lmod:
       # Core compiler is a dummy that is not used to build the stack, do not change.
       # Hopefully nobody is trying to build software with gcc-4.6 in 2022 ...
       core_compilers::
       - gcc@4.6
       projections:
+        ecmwf-atlas: 'atlas/{version}'
         nlohmann-json: 'json/{version}'
         nlohmann-json-schema-validator: 'json-schema-validator/{version}'
       blacklist:
@@ -200,6 +210,10 @@ modules:
             '{name}_ROOT': '{prefix}'
         suffixes:
           +debug: 'debug'
+      ecmwf-atlas:
+        environment:
+          set:
+            'atlas_ROOT': '{prefix}'
       esmf:
         environment:
           set:
@@ -212,6 +226,10 @@ modules:
         suffixes:
           ^esmf@8.2.0~debug: 'esmf-8.2.0'
           ^esmf@8.2.0+debug: 'esmf-8.2.0-debug'
+      openmpi:
+        environment:
+          set:
+            'OMPI_MCA_rmaps_base_oversubscribe': '1'
       hierarchy:
       - mpi