How to force @nixpkgs_foreign_cc//:make_nix_impl to be the preferred toolchain?

[attilaolah]

So I have @rules_nixpkgs_cc and @rules_nixpkgs_core in my MODULE.bazel, and in my WORKSPACE.bzlmod I am registering the toolchains as follows:

load("@rules_nixpkgs_cc//:cc.bzl", "nixpkgs_cc_configure")
load("@rules_nixpkgs_cc//:foreign_cc.bzl", "nixpkgs_foreign_cc_configure")

nixpkgs_cc_configure(
    attribute_path = "clang_16",
    repository = "@nixpkgs",
)

nixpkgs_foreign_cc_configure(
    repository = "@nixpkgs",
)

However, when I try to build something with Make or CMake, the toolchains from @rules_foreign_cc still have priority over the ones I registered here. Toolchain resolution looks like this:

INFO: ToolchainResolution:   Type @rules_foreign_cc~0.10.1//toolchains:make_toolchain: target platform @rules_nixpkgs_core~0.10.0//platforms:host: execution @rules_nixpkgs_core~0.10.0//platforms:host: Selected toolchain @rules_foreign_cc~0.10.1//toolchains:built_make
INFO: ToolchainResolution:     Type @rules_foreign_cc~0.10.1//toolchains:make_toolchain: target platform @rules_nixpkgs_core~0.10.0//platforms:host: execution platform @rules_nixpkgs_core~0.10.0//platforms:host: Skipping toolchain @nixpkgs_foreign_cc//:make_nix_impl; execution platform already has selected toolchain
INFO: ToolchainResolution: Target platform @rules_nixpkgs_core~0.10.0//platforms:host: Selected execution platform @rules_nixpkgs_core~0.10.0//platforms:host, type @rules_foreign_cc~0.10.1//toolchains:make_toolchain -> toolchain @rules_foreign_cc~0.10.1//toolchains:built_make, type @rules_foreign_cc~0.10.1//toolchains:automake_toolchain -> toolchain @rules_foreign_cc~0.10.1//toolchains:preinstalled_automake, type @bazel_tools//tools/cpp:toolchain_type -> toolchain @local_config_cc//:cc-compiler-k8, type @rules_foreign_cc~0.10.1//toolchains:m4_toolchain -> toolchain @rules_foreign_cc~0.10.1//toolchains:preinstalled_m4, type @rules_foreign_cc~0.10.1//toolchains:pkgconfig_toolchain -> toolchain @rules_foreign_cc~0.10.1//toolchains:built_pkgconfig, type @rules_foreign_cc~0.10.1//toolchains:autoconf_toolchain -> toolchain @rules_foreign_cc~0.10.1//toolchains:preinstalled_autoconf, type @rules_foreign_cc~0.10.1//foreign_cc/private/framework:shell_toolchain -> toolchain @rules_foreign_cc~0.10.1~tools~rules_foreign_cc_framework_toolchain_linux//:commands
INFO: ToolchainResolution:   Type @rules_foreign_cc~0.10.1//toolchains:make_toolchain: target platform @rules_nixpkgs_core~0.10.0//platforms:host: execution @rules_nixpkgs_core~0.10.0//platforms:host: Selected toolchain @rules_foreign_cc~0.10.1//toolchains:built_make
INFO: ToolchainResolution:     Type @rules_foreign_cc~0.10.1//toolchains:make_toolchain: target platform @rules_nixpkgs_core~0.10.0//platforms:host: execution platform @rules_nixpkgs_core~0.10.0//platforms:host: Skipping toolchain @nixpkgs_foreign_cc//:make_nix_impl; execution platform already has selected toolchain
INFO: ToolchainResolution: Target platform @rules_nixpkgs_core~0.10.0//platforms:host: Selected execution platform @rules_nixpkgs_core~0.10.0//platforms:host, type @rules_foreign_cc~0.10.1//toolchains:make_toolchain -> toolchain @rules_foreign_cc~0.10.1//toolchains:built_make, type @bazel_tools//tools/cpp:toolchain_type -> toolchain @local_config_cc//:cc-compiler-k8, type @rules_foreign_cc~0.10.1//foreign_cc/private/framework:shell_toolchain -> toolchain @rules_foreign_cc~0.10.1~tools~rules_foreign_cc_framework_toolchain_linux//:commands

I tried a combination of --host_platform and --platforms, nothing seems to help. The only way I found to avoid using the @rules_foreign_cc//toolchains:built_make toolchain from being picked up is to actually patch the @rules_foreign_cc module so that it wouldn't even register the toolchain in the first place.

This is obviously not the most elegant solution, so I'm trying to figure out if there is a way to somehow, maybe, attach a constraint to all my build targets, and have it so that only the nixpkgs_foreign_cc_configure rules would satisfy those constraints? I'm surprised even, why passing --platforms=@rules_nikxpkgs_core//platforms:host is not enough for that. But even if that worked, I wouldn't want to use the host platform since I'd want to cross-compile at some point, and I don't want the CC toolchain to be picked up for cross-compilation, but I still want the @nixpugs_foreign_cc toolchains, even for cross-compilation (builds that do not target the :host platform).

[attilaolah]

Apparently adding the toolchains via the command line helps, e.g. in .bazelrc:

build --extra_toolchains=@nixpkgs_foreign_cc_toolchain//:all

This is because, according to the documented toolchain resolution order:

The set of available toolchains, in priority order, is created from --extra_toolchains and register_toolchains:

1. Toolchains registered using --extra_toolchains are added first.
   a. Within these, the last toolchain has highest priority.
2. Toolchains registered using register_toolchains
   a. Within these, the first mentioned toolchain has highest priority.

This still does not disable the built toolchains, but the ones provided by the @nixpkgs_foreign_cc_toolchain repo will be preferred.

[aherrmann]

If you use MODULE.bazel, then you can also register it in the module file as shown in this example:

rules_nixpkgs/testing/cc/MODULE.bazel

Line 37 in 21c4ea4

register_toolchains("@nixpkgs_config_cc_toolchains//:all")

Once we have module extensions for toolchains (#183) this should become easier.

[avdv]

The problem with the nixpkgs_foreign_cc_configure though is that it always tries to register the toolchain and thus cannot be used from the MODULE.bazel currently.

[aherrmann]

Oh, I overlooked that this was about foreign. That should be fixed then. For bzlmod compatibility, it should provide a parameter to configure whether to register the toolchains or not, similar to here

rules_nixpkgs/testing/cc/tests/nixpkgs_repositories.bzl

Line 25 in 21c4ea4

register = not bzlmod,

[attilaolah]

To follow up on this, currently I'm using the .bazelrc flag approach, because of what @avdv mentioned above.

This works for targeting the host architecture, but doesn't seem to work when cross compiling, because the @nixpkgs_foreign_cc_toolchain is not compatible with the target (exec) env. It probably should be, though, since the foreign toolchains, like CMake, Configure/Make, etc. can work, usually, just fine with a different underlying CC toolchain that would target a different platform, in my case, a WebAssembly target.

It should be possible to e.g. use the Emscripten CC toolchain, or create a Wasi/Wasix toolchain, and use that as the CC toolchain, but keep using the same foreign CC toolchain.

In any case, the only downside is that I get a duplicate copy of the foreign CC toolchains, and the ones in rules_foreign_cc are built by default, so on first run my compiles are a bit slower as I need to bootstrap everything. Once the cache is warmed up, it is usually fine though.