Provide nix Toolchain

[midnightconman]

Is your feature request related to a problem? Please describe.
Not entirely, my request is related to a feature that would allow for more use-cases.

Describe the solution you'd like
I would like this repo to provide a nix binary toolchain. This would allow the execution of nix binary commands from inside bazel. I would like this toolchain to offer linux (amd64) and macOS (amd64 and arm64) binaries. At the moment it appears that rules_nixpkgs is meant to be run via the nix binary, which in turn will execute bazel -- Our group would like to run a bazel binary, which in turn would run a nix binary in a bazel sandbox.

Describe alternatives you've considered
Our group wouldn't like to require a nix binary in our developers local environments, this could be difficult to maintain version parity across our ecosystem.

Additional context
None at this time.

Thank you so much for this repo! 😸

[aherrmann]

Hi @midnightconman, thank you for raising this!

I would like this repo to provide a nix binary toolchain [...] to offer linux (amd64) and macOS (amd64 and arm64) binaries.

That Sounds quite doable, and could follow the pattern of the recent work on multi-platform toolchains for nodejs. Could you expand a bit more on the use-case you have in mind?

At the moment it appears that rules_nixpkgs is meant to be run via the nix binary, which in turn will execute bazel

I'll try to clarify this a bit. There are two layers:

1. Nix provides Bazel, e.g. nix-shell -p bazel or a shell.nix or flake.nix file to the same effect.
   The Bazel provided by Nix from nixpkgs includes a few patches that make it more hermetic and compatible with Nix. In theory it's possible to use a vanilla Bazel binary distribution with rules_nixpkgs. However, that combination is not well tested and not supported.
2. rules_nixpkgs provides repository rules which invoke nix-build when Bazel fetches the corresponding external workspaces.
   That means that Bazel invokes Nix, Nix builds or downloads Nix packages and symlinks them into Bazel's output base, rules_nixpkgs then generates Bazel targets to import the files provided by the Nix package into Bazel, e.g. as toolchains or just as files.
   Notably, this happens in Bazel repository rules, not regular rules. The reason is that, in some cases, we need to inspect the files provided by Nix and generate Bazel BUILD files based on the result, e.g. to generate proper toolchain definitions. This only possible in repository rules, not in regular rules.

You can also take a look at this talk from BazelCon 2022 which describes these layers as well.

Our group would like to run a bazel binary, which in turn would run a nix binary in a bazel sandbox.

This sounds like you would like to run Nix within a regular Bazel rule. That can be done, however, beware that, depending on the configuration, Nix will generate files outside of Bazel's control, e.g. under /nix/store/..., and this may clash with Bazel's assumptions around caching and target invalidation. Could you expand a bit on the use-case you have in mind?

Note, this is already possible without the need to define a dedicated toolchain. You can invoke nixpkgs_package with attribute_path = "nix" to import a Nix binary into Bazel. Without further configuration it will be compatible with the host machine architecture and OS. Assuming that you don't use remote execution, that will be sufficient. If you wish to use remote execution, please refer to #180.

---

Coming back to the feature request itself. I think the steps to implement this would be:

• Define a toolchain_type for this Nix toolchain, @rules_nixpkgs_core//nix:toolchain_type would seem like a reasonable spot.
• Define a nixpkgs_nix_configure_platforms repository macro similar to nixpkgs_nodejs_configure_platforms that iterates over the platforms and invokes `nixpkgs_nix_configure.
• Define a nixpkgs_nix_configure repository macro, similar to nixpkgs_nodejs_configure that invokes a nixpkgs_package repository rule to import nix and a dedicated _nixpkgs_nix_toolchain repository rule that defines the toolchain instance. (We always maintain that separation and indirection so that Bazel does not need to fetch and invoke nix-build just to perform toolchain resolution)
• Define the _nixpkgs_nix_toolchain repository rule similar to _nixpkgs_nodejs_toolchain.

[midnightconman]

Hello @aherrmann 👋 and thank you for the quick reply 😸

That Sounds quite doable, and could follow the pattern of the recent work on #309. Could you expand a bit more on the use-case you have in mind?

Excellent! Our developers are very familiar with bazel (not so much with nix), so we would prefer the main "entry" binary to be bazel. We build docker images from scratch for our build pipeline using rules_dpkg, but we have had some issues with maintainability (2,000+ lines of package dependencies)... so we are looking for alternatives to rules_dpkg and debian packages. Nix and nixpkgs seems like a very possible alternative.

... In theory it's possible to use a vanilla Bazel binary distribution with rules_nixpkgs. However, that combination is not well tested and not supported.

Ahh, good to know. Our hope was to use nix as a toolchain in our existing large bazel workspace, and then run nixpkgs from said toolchain binary. This way we don't have to ask our developers to install and maintain nix on their workstations, they just need to download bazelisk and go. We also have a few different workstation types (linux, macOS [amd and arm], and some winders)... so we assume trying to maintain nix version parity across our fleet might be difficult. Maybe that is not the case?

This sounds like you would like to run Nix within a regular Bazel rule. That can be done, however, beware that, depending on the configuration, Nix will generate files outside of Bazel's control, e.g. under /nix/store/..., and this may clash with Bazel's assumptions around caching and target invalidation. Could you expand a bit on the use-case you have in mind?

This is good to know. I assume we would be able to put these in the normal bazel-bin (or possibly bazel cache CAS). This is probably where your call out about "... beware that, depending on the configuration ..." comes into play?

Expanding more on the use-case... we would like to have our developers run bazel commands instead of nix commands. So their primary interface would be $ bazel build ..., $ bazel test ..., or $ bazel run //:nix -- --pure --command 'bazel --version'. Once that is possible, we would use nixpkgs for all system dependencies that we pull into our bazel workspace... which we can then build docker images predictably from said system dependencies.

Coming back to the feature request itself. I think the steps to implement this would be: ...

Let me look into this a bit further 😸 I downloaded the nix install tarball and it has many system dependencies and the nix related binaries are dynamically linked... so we would probably want to get statically linked binaries to ensure execution predictability.

I came across this thread about statically linked nix binaries (https://discourse.nixos.org/t/building-a-statically-linked-nix-for-hpc-environments/10865/4), so I'll look into this a bit more and familiarize myself more with nix in general.

Thank you so much again for the info!

[aherrmann]

@midnightconman Thanks for the swift reply and sharing more context about your use-case.

we would prefer the main "entry" binary to be bazel

You mean the "entry" binary for the users, as in what the user types in? That's usually still the case with a Nix provided Bazel. E.g. if you configure direnv to load a Nix shell, then developers wouldn't have to invoke nix-shell themselves, that would happen automatically when they enter the workspace. Then they just type bazel as usual to invoke a Nix provided Bazel.

This way we don't have to ask our developers to install and maintain nix on their workstations

Usually Nix still requires a one time installation on a workstation, because the Nix store is conventionally stored under /nix/store.

I assume we would be able to put these in the normal bazel-bin (or possibly bazel cache CAS).

Nix packages are built around the assumption that they are stored in a Nix store with a fixed absolute path, usually /nix/store. Bazel OTOH builds around the idea that outputs are relocatable in the sense that they only have a fixed relative path relative to the exec root.

Expanding more on the use-case... we would like to have our developers run bazel commands instead of nix commands.

That's indeed what a common rules_nixpkgs use-case looks like. Entering a Nix shell can be automated with direnv as described above, or other ways to expose a Nix provided Bazel could be considered. Then users will use Bazel as the entry point. They usually won't need to invoke Nix directly for a build, instead that is taken care of by the repository rules that rules_nixpkgs provides. For example, if you take the Nix + Bazel Codelab it uses Nix provided toolchains and packages, but apart from initial setup, all other steps only involve Bazel.

I'll look into this a bit more and familiarize myself more with nix in general.

If you're interested I could also offer a call to talk more about your use-case. If so, let me know how to get in touch, or feel free to contact me on the email on my Github profile.

[AttilaTheFun]

@aherrmann @midnightconman this is also very interesting to me. I would really like nix to just be an implementation detail, and developers would just interact with Bazel.

In my ideal world, I would add rules_nixpkgs from my WORKSPACE and use it to configure my toolchains + install nix packages. In BUILD files, developers could add dependencies on packages provided by nix like @nixpkgs//:ffmpeg.

[aherrmann]

@AttilaTheFun Thanks for reaching out.

In my ideal world, I would add rules_nixpkgs from my WORKSPACE and use it to configure my toolchains + install nix packages. In BUILD files, developers could add dependencies on packages provided by nix like @nixpkgs//:ffmpeg.

In terms of what the WORKSPACE and BUILD files look like (or the MODULE.bazel with bzlmod going forward), this is pretty much the reality already. Where it departs from what you describe, at least as I understand it, is that Nix needs to be installed outside of Bazel and that rules_nixpkgs only supports the use with a Nix provided Bazel. Is that what you had in mind or are there other aspects of rules_nixpkgs that don't fit with your intended use-case?

[AttilaTheFun]

Where it departs from what you describe, at least as I understand it, is that Nix needs to be installed outside of Bazel and that rules_nixpkgs only supports the use with a Nix provided Bazel.

Yes that's most of it -- ideally it would work from any shell and recent mainline Bazel version. rules_nixpkgs would be a repository rule and it would install nix as a toolchain inside the Bazel cache like rules_go / bazel_gazelle. It would use nix internally for fetching and configuring the packages. But then it would create symlinks to the packages inside the nix store so nix isn't required at runtime.

[aherrmann]

@AttilaTheFun That's understandable. As mentioned above there are some limitations to what's possible with Bazel and Nix at the moment.

rules_nixpkgs would be a repository rule

Just to clarify the terminology. rules_nixpkgs itself will be a Bazel module going forward. You'll depend on it with lines like the following in your MODULE.bazel file.

bazel_dep(name = "rules_nixpkgs_core")
bazel_dep(name = "rules_nixpkgs_cc")

It will provide module extensions (which under the covers will invoke repository rules) to import Nix provided packages and toolchains.

it would install nix as a toolchain inside the Bazel cache like rules_go / bazel_gazelle

The term toolchain has very specific meaning in Bazel and toolchains are not available in repository rules. However, rules_nixpkgs has to access Nix in repository rules to be able to generate toolchain configuration. So, it cannot be a toolchain for those use-cases.

That said, in theory rules_nixpkgs could install Nix in a repository rule. However, there is no single, straight-forward answer how to do this at the moment: A standard Nix installation uses a Nix store under /nix/store and typically requires root privileges to be installed, if only to create /nix/store. There are various approaches to non-root and user local Nix installations. However, AFAIK, there is no single standard approach at this point, and they all come with some caveats.

It would use nix internally for fetching and configuring the packages. But then it would create symlinks to the packages inside the nix store so nix isn't required at runtime.

This is already the case. Note, however, that using symlinks to the Nix store (as rules_nixpkgs already does right now) means that the target Nix store paths are required at runtime.