Update CHANGELOG.md for v0.23.0

CHANGELOG.md

@@ -1,3 +1,365 @@
+# v0.23.0 (2024-11-13)
+
+`v0.23.0` is a major feature release.
+
+We are planning to make this the last major release before Spack `v1.0`
+in June 2025. Alongside `v0.23`, we will be making pre-releases (alpha,
+beta, etc.)  of `v1.0`, and we encourage users to try them and send us
+feedback, either on GitHub or on Slack. You can track the road to
+`v1.0` here:
+
+  * https://github.com/spack/spack/releases
+  * https://github.com/spack/spack/discussions/30634
+
+## Features in this Release
+
+1. **Language virtuals**
+
+   Your packages can now explicitly depend on the languages they require.
+   Historically, Spack has considered C, C++, and Fortran compiler
+   dependencies to be implicit. In `v0.23`, you should ensure that
+   new packages add relevant C, C++, and Fortran dependencies like this:
+
+   ```python
+   depends_on("c", type="build")
+   depends_on("cxx", type="build")
+   depends_on("fortran", type="build")
+   ```
+
+   We encourage you to add these annotations to your packages now, to prepare
+   for Spack `v1.0.0`. In `v1.0.0`, these annotations will be necessary for
+   your package to use C, C++, and Fortran compilers. Note that you should
+   *not* add language dependencies to packages that don't need them, e.g.,
+   pure python packages.
+
+   We have already auto-generated these dependencies for packages in the
+   `builtin` repository (see #45217), based on the types of source files
+   present in each package's source code. We *may* have added too many or too
+   few language dependencies, so please submit pull requests to correct
+   packages if you find that the language dependencies are incorrect.
+
+   Note that we have also backported support for these dependencies to
+   `v0.21.3` and `v0.22.2`, to make all of them forward-compatible with
+   `v0.23`. This should allow you to move easily between older and newer Spack
+   releases without breaking your packages.
+
+2. **Spec splicing**
+
+   We are working to make binary installation more seamless in Spack. `v0.23`
+   introduces "splicing", which allows users to deploy binaries using local,
+   optimized versions of a binary interface, even if they were not built with
+   that interface. For example, this would allow you to build binaries in the
+   cloud using `mpich` and install them on a system using a local, optimized
+   version of `mvapich2` *without rebuilding*. Spack preserves full provenance
+   for the installed packages and knows that they were built one way but
+   deployed another.
+
+   Our intent is to leverage this across many key HPC binary packages,
+   e.g. MPI, CUDA, ROCm, and libfabric.
+
+   Fundamentally, splicing allows Spack to redeploy an existing spec with
+   different dependencies than how it was built. There are two interfaces to
+   splicing.
+
+   a. Explicit Splicing
+
+      #39136 introduced the explicit splicing interface. In the
+      concretizer config, you can specify a target spec and a replacement
+      by hash.
+
+      ```yaml
+      concretizer:
+        splice:
+          explicit:
+          - target: mpi
+            replacement: mpich/abcdef
+      ```
+
+      Here, every installation that would normally use the target spec will
+      instead use its replacement. Above, any spec using *any* `mpi` will be
+      spliced to depend on the specific `mpich` installation requested. This
+      *can* go wrong if you try to replace something built with, e.g.,
+      `openmpi` with `mpich`, and it is on the user to ensure ABI
+      compatibility between target and replacement specs. This currently
+      requires some expertise to use, but it will allow users to reuse the
+      binaries they create across more machines and environments.
+
+   b. Automatic Splicing (experimental)
+
+      #46729 introduced automatic splicing. In the concretizer config, enable
+      automatic splicing:
+
+      ```yaml
+      concretizer:
+        splice:
+          automatic: true
+      ```
+
+      or run:
+
+      ```console
+      spack config add concretizer:splice:automatic:true
+      ```
+
+      The concretizer will select splices for ABI compatibility to maximize
+      package reuse. Packages can denote ABI compatibility using the
+      `can_splice` directive. No packages in Spack yet use this directive, so
+      if you want to use this feature you will need to add `can_splice`
+      annotations to your packages. We are working on ways to add more ABI
+      compatibility information to the Spack package repository, and this
+      directive may change in the future.
+
+   See the documentation for more details:
+   * https://spack.readthedocs.io/en/latest/build_settings.html#splicing
+   * https://spack.readthedocs.io/en/latest/packaging_guide.html#specifying-abi-compatibility
+
+3. Broader variant propagation
+
+   Since #42931, you can specify propagated variants like `hdf5
+   build_type==RelWithDebInfo` or `trilinos ++openmp` to propagate a variant
+   to all dependencies for which it is relevant. This is valid *even* if the
+   variant does not exist on the package or its dependencies.
+
+   See https://spack.readthedocs.io/en/latest/basic_usage.html#variants.
+
+4. Query specs by namespace
+
+   #45416 allows a package's namespace (indicating the repository it came from)
+   to be treated like a variant. You can request packages from particular repos
+   like this:
+
+   ```console
+   spack find zlib namespace=builtin
+   spack find zlib namespace=myrepo
+   ```
+
+   Previously, the spec syntax only allowed namespaces to be prefixes of spec
+   names, e.g. `builtin.zlib`. The previous syntax still works.
+
+5. `spack spec` respects environment settings and `unify:true`
+
+   `spack spec` did not previously respect environment lockfiles or
+   unification settings, which made it difficult to see exactly how a spec
+   would concretize within an environment. Now it does, so the output you get
+   with `spack spec` will be *the same* as what your environment will
+   concretize to when you run `spack concretize`. Similarly, if you provide
+   multiple specs on the command line with `spack spec`, it will concretize
+   them together if `unify:true` is set.
+
+   See #47556 and #44843.
+
+6. Less noisy `spack spec` output
+
+   `spack spec` previously showed output like this:
+
+   ```console
+    > spack spec /v5fn6xo
+    Input spec
+    --------------------------------
+     -   /v5fn6xo
+
+    Concretized
+    --------------------------------
+    [+]  openssl@3.3.1%apple-clang@16.0.0~docs+shared arch=darwin-sequoia-m1
+    ...
+   ```
+
+   But the input spec is redundant, and we know we run `spack spec` to concretize
+   the input spec. `spack spec` now *only* shows the concretized spec. See #47574.
+
+7. Better output for `spack find -c`
+
+   In an environmnet, `spack find -c` lets you search the concretized, but not
+   yet installed, specs, just as you would the installed ones. As with `spack
+   spec`, this should make it easier for you to see what *will* be built
+   before building and installing it. See #44713.
+
+8. `spack -C <env>`: use an environment's configuration without activation
+
+   Spack environments allow you to associate:
+   1. a set of (possibly concretized) specs, and
+   2. configuration
+
+   When you activate an environment, you're using both of these. Previously, we
+   supported:
+   * `spack -e <env>` to run spack in the context of a specific environment, and
+   * `spack -C <directory>` to run spack using a directory with configuration files.
+
+   You can now also pass an environment to `spack -C` to use *only* the environment's
+   configuration, but not the specs or lockfile. See #45046.
+
+## New commands, options, and directives
+
+* The new `spack env track` command (#41897) takes a non-managed Spack
+  environment and adds a symlink to Spack's `$environments_root` directory, so
+  that it will be included for reference counting for commands like `spack
+  uninstall` and `spack gc`. If you use free-standing directory environments,
+  this is useful for preventing Spack from removing things required by your
+  environments. You can undo this tracking with the `spack env untrack`
+  command.
+
+* Add `-t` short option for `spack --backtrace` (#47227)
+
+  `spack -d / --debug` enables backtraces on error, but it can be very
+  verbose, and sometimes you just want the backtrace. `spack -t / --backtrace`
+  provides that option.
+
+* `gc`: restrict to specific specs (#46790)
+
+  If you only want to garbage-collect specific packages, you can now provide
+  them on the command line. This gives users finer-grained control over what
+  is uninstalled.
+
+* oci buildcaches now support `--only=package`. You can now push *just* a
+  package and not its dependencies to an OCI registry. This allows dependents
+  of non-redistributable specs to be stored in OCI registries without an
+  error. See #45775.
+
+## Notable refactors
+* Variants are now fully conditional
+
+  The `variants` dictionary on packages was previously keyed by variant name,
+  and allowed only one definition of any given variant. Spack is now smart
+  enough to understand that variants may have different values and defaults
+  for different versions. For example, `warpx` prior to `23.06` only supported
+  builds for one dimensionality, and newer `warpx` versions could be built
+  with support for many different dimensions:
+
+  ```python
+  variant(
+      "dims",
+      default="3",
+      values=("1", "2", "3", "rz"),
+      multi=False,
+      description="Number of spatial dimensions",
+      when="@:23.05",
+  )
+  variant(
+      "dims",
+      default="1,2,rz,3",
+      values=("1", "2", "3", "rz"),
+      multi=True,
+      description="Number of spatial dimensions",
+      when="@23.06:",
+  )
+  ```
+
+  Previously, the default for the old version of `warpx` was not respected and
+  had to be specified manually. Now, Spack will select the right variant
+  definition for each version at concretization time. This allows variants to
+  evolve more smoothly over time. See #44425 for details.
+
+## Highlighted bugfixes
+
+1. Externals no longer override the preferred provider (#45025).
+
+   External definitions could interfere with package preferences. Now, if
+   `openmpi` is the preferred `mpi`, and an external `mpich` is defined, a new
+   `openmpi` *will* be built if building it is possible. Previously we would
+   prefer `mpich` despite the preference.
+
+2. Composable `cflags` (#41049).
+
+   This release fixes a longstanding bug that concretization would fail if
+   there were different `cflags` specified in `packages.yaml`,
+   `compilers.yaml`, or on `the` CLI. Flags and their ordering are now tracked
+   in the concretizer and flags from multiple sources will be merged.
+
+3. Fix concretizer Unification for included environments (#45139).
+
+## Deprecations, removals, and syntax changes
+
+1. The old concretizer has been removed from Spack, along with the
+   `config:concretizer` config option. Spack will emit a warning if the option
+   is present in user configuration, since it now has no effect. Spack now
+   uses a simpler bootstrapping mechanism, where a JSON prototype is tweaked
+   slightly to get an initial concrete spec to download. See #45215.
+
+2. Best-effort expansion of spec matrices has been removed. This feature did
+   not work with the "new" ASP-based concretizer, and did not work with
+   `unify: True` or `unify: when_possible`. Use the
+   [exclude key](https://spack.readthedocs.io/en/latest/environments.html#spec-matrices)
+   for the environment to exclude invalid components, or use multiple spec
+   matrices to combine the list of specs for which the constraint is valid and
+   the list of specs for which it is not. See #40792.
+
+3. The old Cray `platform` (based on Cray PE modules) has been removed, and
+   `platform=cray` is no longer supported. Since `v0.19`, Spack has handled
+   Cray machines like Linux clusters with extra packages, and we have
+   encouraged using this option to support Cray. The new approach allows us to
+   correctly handle Cray machines with non-SLES operating systems, and it is
+   much more reliable than making assumptions about Cray modules. See the
+   `v0.19` release notes and #43796 for more details.
+
+4. The `config:install_missing_compilers` config option has been deprecated,
+   and it is a no-op when set in `v0.23`. Our new compiler dependency model
+   will replace it with a much more reliable and robust mechanism in `v1.0`.
+   See #46237.
+
+5. Config options that deprecated in `v0.21` have been removed in `v0.23`. You
+   can now only specify preferences for `compilers`, `targets`, and
+   `providers` globally via the `packages:all:` section. Similarly, you can
+   only specify `versions:` locally for a specific package. See #44061 and
+   #31261 for details.
+
+6. Spack's old test interface has been removed (#45752), having been
+   deprecated in `v0.22.0` (#34236). All `builtin` packages have been updated
+   to use the new interface. See the [stand-alone test documentation](
+   https://spack.readthedocs.io/en/latest/packaging_guide.html#stand-alone-tests)
+
+7. The `spack versions --safe-only` option, deprecated since `v0.21.0`, has
+   been removed. See #45765.
+
+* The `--dependencies` and `--optimize` arguments to `spack ci` have been
+  deprecated. See #45005.
+
+## Binary caches
+1. Public binary caches now include an ML stack for Linux/aarch64 (#39666)We
+   now build an ML stack for Linux/aarch64 for all pull requests and on
+   develop. The ML stack includes both CPU-only and CUDA builds for Horovod,
+   Hugging Face, JAX, Keras, PyTorch,scikit-learn, TensorBoard, and
+   TensorFlow, and related packages. The CPU-only stack also includes XGBoost.
+   See https://cache.spack.io/tag/develop/?stack=ml-linux-aarch64-cuda.
+
+2. There is also now an stack of developer tools for macOS (#46910), which is
+   analogous to the Linux devtools stack. You can use this to avoid building
+   many common build dependencies. See
+   https://cache.spack.io/tag/develop/?stack=developer-tools-darwin.
+
+## Architecture support
+* archspec has been updated to `v0.2.5`, with support for `zen5`
+* Spack's CUDA package now supports the Grace Hopper `9.0a` compute capability (#45540)
+
+## Windows
+* Windows bootstrapping: `file` and `gpg` (#41810)
+* `scripts` directory added to PATH on Windows for python extensions (#45427)
+* Fix `spack load --list` and `spack unload` on Windows (#35720)
+
+## Other notable changes
+* Bugfix: `spack find -x` in environments (#46798)
+* Spec splices are now robust to duplicate nodes with the same name in a spec (#46382)
+* Cache per-compiler libc calculations for performance (#47213)
+* Fixed a bug in external detection for openmpi (#47541)
+* Mirror configuration allows username/password as environment variables (#46549)
+* Default library search caps maximum depth (#41945)
+* Unify interface for `spack spec` and `spack solve` commands (#47182)
+* Spack no longer RPATHs directories in the default library search path (#44686)
+* Improved performance of Spack database (#46554)
+* Enable package reuse for packages with versions from git refs (#43859)
+* Improved handling for `uuid` virtual on macos (#43002)
+* Improved tracking of task queueing/requeueing in the installer (#46293)
+
+## Spack community stats
+
+* Over 2,000 pull requests updated package recipes
+* 8,307 total packages, 329 new since `v0.22.0`
+    * 140 new Python packages
+    * 14 new R packages
+* 373 people contributed to this release
+    * 357 committers to packages
+    * 60 committers to core
+
+
 # v0.22.2 (2024-09-21)
 
 ## Bugfixes
@@ -419,7 +781,7 @@
 - spack graph: fix coloring with environments (#41240)
 - spack info: sort variants in --variants-by-name (#41389)
 - Spec.format: error on old style format strings (#41934)
-- ASP-based solver: 
+- ASP-based solver:
   - fix infinite recursion when computing concretization errors (#41061)
   - don't error for type mismatch on preferences (#41138)
   - don't emit spurious debug output (#41218)