From 2cff82a806946f4d9f147c43baca03e7916d7f39 Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Sat, 22 Apr 2023 10:28:36 +1100 Subject: [PATCH 1/3] distinguish ABI-breaking migrations in infrastructure.rst/compilers --- src/maintainer/infrastructure.rst | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/src/maintainer/infrastructure.rst b/src/maintainer/infrastructure.rst index ce58ec6c85..ea24054c82 100644 --- a/src/maintainer/infrastructure.rst +++ b/src/maintainer/infrastructure.rst @@ -281,8 +281,14 @@ Compilers and Runtimes Conda-forge builds and maintains its own set of compilers for various languages and/or systems (e.g., ``C``, ``FORTRAN``, ``C++``, ``CUDA``, etc.). These are used -in all of our CI builds to build both core dependencies (e.g., ``Python``) and maintainer-contributed -packages. While we do not have any formal policies or promises of support for these +in all of our CI builds to build essentially all artefacts published by conda-forge. + +In the past, compiler upgrades often required full rebuilds of basically all of +conda-forge due to potential ABI breaks. These breaks have become much rarer in +frequency and scope, so it is not a current concern. However, we keep our policies +for such ABI breaks in place for the next time it should occur. + +While we do not have any promises of support for a generation of ABI-compatible compilers, we have historically maintained them according to the following (non-binding) principles. @@ -290,27 +296,22 @@ principles. and platforms is the `conda_build_config.yaml `_ in the `conda-forge/conda-forge-pinning-feedstock `_ as described in :ref:`globally_pinned_packages`. -* We provide no support of any kind in terms of the long-term stability of these pinnings. +* We provide no support of any kind in terms of the long-term stability / support of a given compiler generation. * We upgrade them in an ad-hoc manner on a periodic basis as we have the time and energy to do so. Note that because of the way we enforce runtime constraints, these compiler upgrades will not break existing packages. However, if you are using the compilers outside of ``conda``, then you may find issues. -* We generally provide notice in the form of an announcement when a compiler is going to be upgraded. +* We will provide notice in the form of an announcement when an ABI-incompatible compiler change is going to be happen. Note that these changes take a bit of time to complete, so you will generally have time to prepare should you need to. * Some of the criteria we think about when considering a compiler migration include 1) the degree of disruption to the ecosystem, 2) the amount of work for the ``core`` team, and 3) the amount of time it will cost our (volunteer) feedstock maintainers. -We do use some unofficial names for our compiler stack internally. Note however that -the existence of these names does not imply any level of support or stability for the compilers +These compiler generations may or may not have some unofficial names for our +internal use (e.g. ``comp7``). We note again that the existence of these names +does not imply any level of support or stability for the compilers that form the given stack. -* Our current compiler stack is referred to internally as ``comp7``. -* The previous compiler stack based in part on the various ``toolchain_*`` packages - was sometimes referred to as ``comp4``. On linux the ``toolchain_*`` compilers were - GCC 4.8.2 as packaged in the devtoolset-2 software collection. On osx, we use clang from - Apple's Xcode in the ``toolchain_*`` packages. - CentOS ``sysroot`` for ``linux-*`` Platforms --------------------------------------------- From 37c400164222d3a2f39a55e8b65b971608168821 Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Sat, 22 Apr 2023 13:12:49 +1100 Subject: [PATCH 2/3] update infrastructure.rst/compilers with current setup --- src/maintainer/infrastructure.rst | 76 +++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) diff --git a/src/maintainer/infrastructure.rst b/src/maintainer/infrastructure.rst index ea24054c82..10b5a17010 100644 --- a/src/maintainer/infrastructure.rst +++ b/src/maintainer/infrastructure.rst @@ -312,6 +312,82 @@ internal use (e.g. ``comp7``). We note again that the existence of these names does not imply any level of support or stability for the compilers that form the given stack. +The more recent status quo is that compilers can be upgraded in an ABI-compatible manner, +meaning that we can just increase the version in our global pinning, and it will slowly +roll out in a compatible manner to the ecosystem as feedstocks get rerendered. +For such ABI-compatible upgrades, similar but looser principles apply: + +* The pins are similarly in the global pinning, see :ref:`globally_pinned_packages`. +* We provide no support of any kind in terms of the long-term availability of a given compiler generation. +* We generally provide notice in the form of an announcement when a compiler is going to be upgraded. +* In general, our compilers on Linux and OSX are using very recent compilers, whereas + on windows, we generally use the last supported VS version. + +Despite the lack of explicit support, we try to keep the compilers in their various versions +working also outside of conda-forge, and even provide an easy way to install them +(through the `compilers `_ feedstock). + +In more detail, our default compiler stack is made up very differently on each platform. +More specifically, each compiler needs an _activation_ that makes the difference +between it being merely present in a build environment, and it being used by default. +These will be installed when using ``{{ compiler('xyz') }}`` in ``meta.yaml``, where +``'xyz'`` is one of ``'c', 'cxx', 'fortran', 'cuda', 'rust', 'go-cgo', 'go-nocgo'``. + +Linux (GCC): + +* [C, C++, Fortran] Activation: https://github.com/conda-forge/ctng-compiler-activation-feedstock/ +* [C, C++, Fortran] Implementation: https://github.com/conda-forge/ctng-compilers-feedstock +* Note that when used in conjunction with CUDA, compiler versions are restricted by the + maximum GCC version supported by nvcc (which is also reflected in the global pinning). + +OSX (Clang): + +* [C, C++] Activation: https://github.com/conda-forge/clang-compiler-activation-feedstock/ +* [C, C++] Required feedstocks: + `llvmdev `_, + `clangdev `_, + `compiler-rt `_, + `libcxx `_, + `openmp `_, + `lld `_, + `cctools `_ +* [Fortran] Activation: https://github.com/conda-forge/gfortran_osx-64-feedstock/ +* [Fortran] Implementation: https://github.com/conda-forge/gfortran_impl_osx-64-feedstock/ + +Windows (MSVC): + +* [C, C++] Activation: https://github.com/conda-forge/vc-feedstock + (we cannot redistribute the actual MSVC compilers due to licensing constraints) +* [Fortran] Activation & Implementation: https://github.com/conda-forge/flang-feedstock + +There exists an alternative, MinGW-based, compiler stack on windows, which is available +with a ``m2w64_`` prefix (e.g. ``{{ compiler('m2w64_c') }}``), which is however falling out +of use now that most projects will natively support compilation also with MSVC, in addition +to several complications arising from mixing compiler stacks. + +Additionally, there is a possibility to use clang as a compiler also on Linux & Windows: + +* Activation (Linux): https://github.com/conda-forge/clang-compiler-activation-feedstock/ +* Activation (Windows): https://github.com/conda-forge/clang-win-activation-feedstock/ + +Aside from the main C/C++/Fortran compilers, these are the feedstocks for the other compilers: + +* [CUDA] https://github.com/conda-forge/nvcc-feedstock (CUDA infra currently being overhauled) +* [Rust] `Activation `_ + and `Implementation `_ +* [Go] https://github.com/conda-forge/go-feedstock + +To upgrade the compiler version of our default compilers in the global pinning for +Linux or OSX, ensure that the respective above-mentioned feedstocks have been rebuilt +for the new major version, that all interrelated versions are lifted at the same time, +and obviously that the compilers work (e.g. by testing them on some feedstocks by +specifying the new version through the feedstock-local `conda_build_config.yaml`). + +For Windows, we stay on older compilers for longer, because using a newer toolchain +would force everyone wanting to locally develop with conda-forge artefacts to use +a toolchain that's at least as new. A more in-depth explanation can be found +`here `_. + CentOS ``sysroot`` for ``linux-*`` Platforms --------------------------------------------- From b75ffd1e05dc244ffbb452ae4141e526eb00efcd Mon Sep 17 00:00:00 2001 From: h-vetinari Date: Sun, 23 Apr 2023 14:23:18 +0200 Subject: [PATCH 3/3] Apply suggestions from code review Co-authored-by: Matthew R. Becker --- src/maintainer/infrastructure.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/maintainer/infrastructure.rst b/src/maintainer/infrastructure.rst index 10b5a17010..0b79311d49 100644 --- a/src/maintainer/infrastructure.rst +++ b/src/maintainer/infrastructure.rst @@ -300,7 +300,7 @@ principles. * We upgrade them in an ad-hoc manner on a periodic basis as we have the time and energy to do so. Note that because of the way we enforce runtime constraints, these compiler upgrades will not break existing packages. However, if you are using the compilers outside of ``conda``, then you may find issues. -* We will provide notice in the form of an announcement when an ABI-incompatible compiler change is going to be happen. +* We usually provide notice in the form of an announcement when an ABI-incompatible compiler change is going to happen. Note that these changes take a bit of time to complete, so you will generally have time to prepare should you need to. * Some of the criteria we think about when considering a compiler migration include @@ -328,7 +328,7 @@ working also outside of conda-forge, and even provide an easy way to install the (through the `compilers `_ feedstock). In more detail, our default compiler stack is made up very differently on each platform. -More specifically, each compiler needs an _activation_ that makes the difference +More specifically, each compiler uses an _activation_ package that makes the difference between it being merely present in a build environment, and it being used by default. These will be installed when using ``{{ compiler('xyz') }}`` in ``meta.yaml``, where ``'xyz'`` is one of ``'c', 'cxx', 'fortran', 'cuda', 'rust', 'go-cgo', 'go-nocgo'``. @@ -361,11 +361,11 @@ Windows (MSVC): * [Fortran] Activation & Implementation: https://github.com/conda-forge/flang-feedstock There exists an alternative, MinGW-based, compiler stack on windows, which is available -with a ``m2w64_`` prefix (e.g. ``{{ compiler('m2w64_c') }}``), which is however falling out +with a ``m2w64_`` prefix (e.g. ``{{ compiler('m2w64_c') }}``). However, it is falling out of use now that most projects will natively support compilation also with MSVC, in addition to several complications arising from mixing compiler stacks. -Additionally, there is a possibility to use clang as a compiler also on Linux & Windows: +Additionally, there is a possibility to use clang as a compiler on Linux & Windows: * Activation (Linux): https://github.com/conda-forge/clang-compiler-activation-feedstock/ * Activation (Windows): https://github.com/conda-forge/clang-win-activation-feedstock/