Move spack-stack repos into spack submodule, use two-level namespace for openmpi and mpich, add package py-pyhdf, use only one macOS reference config #131

climbfuji · 2022-05-03T04:06:08Z

Description

This PR has once again grown bigger than intended ... it:

Finalizes the move of the spack-stack repos into the spack submodule
- This requires adding the projections/name translations for ecmwf-atlas to atlas in the common module config
Enables use of two-level namespace for openmpi and mpich (the former required making changes to the package config in spack, for the latter this was already implemented)
- Also need to export OMPI_MCA_rmaps_base_oversubscribe=1 in the openmpi module so that one can use more tasks than CPU cores (for unit testing)
Fixes several problems with CI builds on Linux and macOS
- Use a combination of package-level build parallelism and install-level build parallelism (see https://spack.readthedocs.io/en/latest/packaging_guide.html?highlight=parallel#parallel-builds) to speed up the Ubuntu Intel build
Adds the existing hdf and new py-pyhdf packages
- Requires setting the default provider for jpeg to libjpeg-turbo (already part of our spack-stack)
Updates one of the macOS reference configs to macos-monterey-apple-clang-openmpi-reference (use apple-clang instead of (llvm) clang) and removes the macOS reference config for easier maintenance
- This reference config now builds openmpi-4.1.3 built by spack with two-level namespace enabled
- Note that I tested the same config with mpich-3.4.2 built by spack with two-level namespace enabled
Adds the capability to create containers with spack, all built into the existing scripts
- create-env.py is renamed to create.py environment ... for creating local environments; Github unfortunately doesn't see this and thinks it is a new file and the old one got deleted (use a graphical diff tool locally to compare)
- Containers are built using create.py container ...
- One example is added: configs/containers/docker-ubuntu-gcc-openmpi.yaml
Comments out git-lfs in the common package config, because pinning a version there isn't needed (we don't build git-lfs, but require it due to the dependencies on go/go-bootstrap, which are difficult to build) and having it in there breaks the container builds
Updates and consolidates the documentation in README.md, config/sites/README.md and config/common/README.md into a single (top-level) README.md
- Note: In my next PR, I will add the capability to create HTML (and possibly LaTeX/PDF) documentation using sphinx

Dependencies

waiting on Feature/openmpi two level namespace spack#58
- this updates the submodule pointer to include the changes from the previously merged PR Move spack-stack repos into Spack builtin spack#57 (Move spack-stack repos into Spack builtin)

Issues

Testing

CI tests all pass - see https://github.com/climbfuji/spack-stack/actions (Note: currently some still running, but they all passed yesterday)
macOS Monterey with apple-clang@13.1.6 and openmpi-4.1.3 built by spack for several applications (including full ctests for the fv3-jedi bundle)
macOS Monterey with apple-clang@13.1.6 and mpich-3.4.2 built by spack for several applications (including full ctests for the fv3-jedi bundle)
Container build of docker-ubuntu-gcc-openmpi config on macOS

…d testing

…use two-level namespaces for openmpi and mpich

…d testing

…me create-env.py to create.py

climbfuji · 2022-05-06T07:42:56Z

This looks great.

Comments out git-lfs in the common package config, because it isn't needed (we don't build git-lfs but require it due to the dependencies on go/go-bootstrap, which are difficult to build) and having it in there breaks the container builds

JediBaseEnv has a dependency on git-lfs. Are you saying that it shouldn't have that?

Thanks for looking at this PR! Ready for "review" soon.

JediBaseEnv does have a dependency on git-lfs, yes, but we can't pin a version in packages.yaml (or have any entry in there), because of the way we piece together configs/common/packages.yaml and the docker config (config/container/*). It doesn't allow having two entries in the packages section (spack containerize ignores one of the two, which in the current order means the one from the container config).

We could fix that by stripping out the packages: section identifier and combine the common packages with the container packages in a different order, in which case the containerize command will ignore the entry in the common packages, i.e. it won't make a difference at all. Same for all regular sites, because we install it as an external package (go and go-bootstrap are really tricky to build).

climbfuji · 2022-05-06T14:05:57Z

.github/actions/setup-spack-stack/action.yaml

+            # without having to rerun all other CI tests. Later, should consider
+            # using https://github.com/jurplel/install-qt-action for all runners
+            # and get rid of the macOS homebrew installation/package config
+            #sudo apt-get install qt5-default qttools5-dev-tools


Note. These are not yet used in spack, there was a problem during the concretization step when I added it manually as an external spack. I worked around the long build times using package-level plus install-level parallelism, see further down.

climbfuji · 2022-05-06T14:08:45Z

meta_modules/setup_meta_modules.py

@@ -650,3 +650,5 @@ def substitute_config_vars(config_str):
                with open(python_module_file, 'w') as f:
                    f.write(module_content)
                logging.info("  ... writing {}".format(python_module_file))
+
+logging.info("Metamodule generation completed successfully in {}".format(meta_module_dir))


This is nice to have, because one can copy & paste the meta_module_dir for the module use statement.

climbfuji · 2022-05-06T14:09:28Z

@srherbener @mer-a-o FYI

* Add option to compile `openmpi` without flat namespace (aka two-level namespaces) - corresponding flags borrowed from `mpich` * Rename `nceplibs-bundle` to `nceplibs-bundle-env` * Add new package `py-pyhdf`, add it to `jedi-base-env` * Bug fix for building `hdf` on Ubuntu 20.04 aarch64 - see also spack#30522 * Get timeout for web requests with `urllib` from spack config, same as for `curl` (see corresponding PR for the authoritative spack repo spack#30468) See JCSDA/spack-stack#131 for the corresponding changes in `spack-stack` and for testing.

kgerheiser · 2022-05-06T15:44:55Z

.github/actions/setup-spack-stack/action.yaml

@@ -225,11 +284,26 @@ runs:

  - name: build-env
    shell: bash
+    env:
+      OMP_NUM_THREADS: 16


Does this help? The runner only has 2 vcpus, and what uses OpenMP? Does Spack use it internally?

OpenMP threads are used for the package-level build parallelism (config:build_jobs). It does help, though not a lot. At least it decreases the build time from 6hrs+ to 5hrs 25min with ubuntu/intel for jedi-ewok - good enough for the moment.

kgerheiser · 2022-05-06T15:46:16Z

configs/common/modules.yaml

+      openmpi:
+        environment:
+          set:
+            'OMPI_MCA_rmaps_base_oversubscribe': '1'


Nice, this was always one of the reasons I used MPICH. Having it encoded into the environment is good.

Problem with mpich on macOS is that once the ufs-weather-model update to use mpi_f08 is made, we can't use mpich on macOS anymore, because it doesn't build mpi_f08 on macOS. But openmpi does. So I am glad that there is a solution to (a) the oversubscribe problem and (b) the two-level namespace problem.

climbfuji · 2022-05-06T18:49:42Z

@kgerheiser Submodule pointer updated, .gitmodules reverted - ready for review and merging, if ok.

climbfuji added 5 commits May 3, 2022 05:55

Update .gitmodules and submodule pointer for spack for code review an…

573a952

…d testing

Remove configs/repos

f2ad0e1

Rename atlas to ecmwf-atlas, create module projection, add option to …

8518d24

…use two-level namespaces for openmpi and mpich

Update .gitmodules and submodule pointer for spack for code review an…

e1cc88e

…d testing

Remove repos from CONFIG_INCLUDES in create-env.py

37f77f4

climbfuji linked an issue May 3, 2022 that may be closed by this pull request

Move spack-stack repos to builtin repo within Spack #127

Closed

climbfuji added 2 commits May 3, 2022 09:33

Attempt to fix macos-apple-clang CI by removing external m4, and rena…

616878e

…me create-env.py to create.py

Initial import of configs/containers

8930a2b

climbfuji mentioned this pull request May 3, 2022

Broken: CI tests using apple-clang/llvm-clang on macOS #132

Closed

climbfuji added 18 commits May 3, 2022 16:08

Debug CI

1a831bf

Debug CI

26cd44a

Debug CI

93873bd

Debug CI

623af19

Final fix of jedi-ewok CI tests

d39278c

Improve logging in meta_modules/setup_meta_modules.py

1d685fb

Rename nceplibs-bundle to nceplibs-bundle-env

252eb2d

Try 'apt-get update' to fix 'apt-get install' errors on Ubuntu

6264223

Bump gcc version to 10 in .github/workflows/ubuntu-gcc.yaml

7b7c4f5

Debug CI tests

f37dfa0

Debug CI tests

2543778

Debug CI tests

50b96a1

Debug CI tests

84d3af3

Debug CI tests

8808a93

Update submodule pointer for spack

5cad206

Debug CI tests

cf52121

Use two-level namespace flags only on macOS

232a3bd

Install krb5 on Ubuntu via apt-get

ca884be

climbfuji force-pushed the feature/two_level_namespace_containers branch from 778b950 to ca884be Compare May 4, 2022 06:24

climbfuji added 2 commits May 4, 2022 08:48

Install krb5 on Ubuntu via apt-get

1bdadc4

Fix typo in create.py

a410820

climbfuji added 2 commits May 6, 2022 05:34

Try to speed up builds by using more build jobs (OpenMP threads)

bf89c55

Update submodule pointer for spack

ed9cc62

climbfuji force-pushed the feature/two_level_namespace_containers branch 5 times, most recently from bd23e62 to 35e8786 Compare May 6, 2022 06:07

Try installing four packages in parallel with Intel to speed up build

7bf7d24

climbfuji force-pushed the feature/two_level_namespace_containers branch from 35e8786 to 7bf7d24 Compare May 6, 2022 06:15

Final touches to docker config docker-ubuntu-gcc-openmpi

6b225d8

climbfuji mentioned this pull request May 6, 2022

Must build MPI with two-level namespaces on macOS to handle exceptions correctly #135

Closed

climbfuji linked an issue May 6, 2022 that may be closed by this pull request

Must build MPI with two-level namespaces on macOS to handle exceptions correctly #135

Closed

climbfuji force-pushed the feature/two_level_namespace_containers branch from 88ffdc3 to 8d5e566 Compare May 6, 2022 08:28

Next try to run spack install jobs in parallel

65bf01f

climbfuji force-pushed the feature/two_level_namespace_containers branch from 8d5e566 to 65bf01f Compare May 6, 2022 08:40

climbfuji commented May 6, 2022

View reviewed changes

climbfuji marked this pull request as ready for review May 6, 2022 14:09

climbfuji requested a review from kgerheiser May 6, 2022 14:09

climbfuji mentioned this pull request May 6, 2022

CI tests on Linux: use external qt@5 #137

Closed

kgerheiser approved these changes May 6, 2022

View reviewed changes

Revert .gitmodules and update submodule pointer for spack

ce03303

kgerheiser merged commit 928b1fc into JCSDA:develop May 6, 2022

jeromebarre mentioned this pull request May 9, 2022

Feature/modis aod from h4 JCSDA-internal/ioda-converters#942

Merged

climbfuji mentioned this pull request May 13, 2022

Move repo contents into spack for (builtin packages) #118

Closed

climbfuji deleted the feature/two_level_namespace_containers branch November 11, 2022 20:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move spack-stack repos into spack submodule, use two-level namespace for openmpi and mpich, add package py-pyhdf, use only one macOS reference config #131

Move spack-stack repos into spack submodule, use two-level namespace for openmpi and mpich, add package py-pyhdf, use only one macOS reference config #131

climbfuji commented May 3, 2022 •

edited

Loading

climbfuji commented May 6, 2022

climbfuji May 6, 2022

climbfuji May 6, 2022

climbfuji commented May 6, 2022

kgerheiser May 6, 2022

climbfuji May 6, 2022

kgerheiser May 6, 2022

climbfuji May 6, 2022

climbfuji commented May 6, 2022

Move spack-stack repos into spack submodule, use two-level namespace for openmpi and mpich, add package py-pyhdf, use only one macOS reference config #131

Move spack-stack repos into spack submodule, use two-level namespace for openmpi and mpich, add package py-pyhdf, use only one macOS reference config #131

Conversation

climbfuji commented May 3, 2022 • edited Loading

Description

Dependencies

Issues

Testing

climbfuji commented May 6, 2022

climbfuji May 6, 2022

Choose a reason for hiding this comment

climbfuji May 6, 2022

Choose a reason for hiding this comment

climbfuji commented May 6, 2022

kgerheiser May 6, 2022

Choose a reason for hiding this comment

climbfuji May 6, 2022

Choose a reason for hiding this comment

kgerheiser May 6, 2022

Choose a reason for hiding this comment

climbfuji May 6, 2022

Choose a reason for hiding this comment

climbfuji commented May 6, 2022

climbfuji commented May 3, 2022 •

edited

Loading