[Demo, RFC] Gentoo-like use-flags

[oxij]

What? Why?

With #55129 and following PRs merged nixpkgs can now finally have a working demo implementation of Gentoo-like use-flags of #12877. This is it (well, one of the two, see below).

(Note that I still have some leftovers to #55129, but the current state on master is 90% there already, so this can be demoed now while I have some time to write this.)

Before discussing the technical details, note that the whole point of use-flags is that they cover use cases not covered by overlays and overrides. For instance, it is very hard to maintain a working overlay of headless packages (as minimal.nix profile of NixOS dutifully demonstrates, see #29135 for more discussion), but it is almost trivial to make such a thing using use-flags.

#56109 implements "poor man's version" of use-flags by, pretty much, adding config.common option (named config.poorMansUseFlags there just for fun). It will work, but I feel that merging such an implementation would be unfortunate for several reasons:

• it clashes with use cases of normal config.* and overlays,
• there's no consistent naming of *Support, with*, etc options in nixpkgs,
• using such names produces use-flags with mangled package names.

The only proper way to implement use-flags I'm aware of is to write package expressions using the following idiom:

{ stdenv, ... }:
{ pulseaudio ? false }:
stdenv.mkDerivation { ... }

(See #39580 (comment) for an explanation in more words.)

(This idiom also has an advantage of providing a way to turn those use-flags-arguments into NixOS option modules using lib.setFunctionArgs if desired, though I don't think such thing is desirable, I think option module should be defined on the top-level similarly to what #39580 wanted to do.)

Anyway, this PR adds all the infra that is needed to implement the latter, better, kind of use-flags and changes everything what needs to be changed for it to work so that the result is a noop.

The main problem there is that calling callPackage over an expression of the form

{ stdenv, ... }: argument: ...

can't be distinguished from the above idiom for use-flags automatically, so all such uses have to be patched manually.

A cleaner but currently not very likely solution is to split all-packages.nix into trivial and non-trivial parts like #50643 tried to do.

Thoughts?

(Btw, SLNOS adds a second callPackage similar to the one in this PR and uses that in all-packages.nix where needed, but such an implementation is undesirable in the long run, IMHO.)

git log

• lib: generalize callPackageWith and callPackagesWith, add makeOverridableWithUseFlags

• splice.nix: demo working man's use flags

• treewide: change what needs to be changed to make this work

  In short, callPackage now can not be replaced with an import with
  some inherits, it's a bit weird w.r.t. functions.

• mpv: make an example use-flag

nix-instantiate environment

• Host OS: Linux 4.9, SLNOS 19.03
• Nix: nix-env (Nix) 2.1.3
• Multi-user: yes
• Sandbox: yes
• NIXPKGS_CONFIG:

{
  checkMeta = true;
  doCheckByDefault = true;
}

nix-env -qaP diffs

• On x86_64-linux: noop
• On aarch64-linux: noop
• On x86_64-darwin: noop

[nixos-discourse]

This pull request has been mentioned on Nix community. There might be relevant details there:

https://discourse.nixos.org/t/gentoo-like-use-flags/2210/1

[danbst]

How it should be used? Also, if "headless" use-flag is introduced, would it be possible to build "headless" world by Hydra? Also, please share a list of most popular (in your opinion) useflags, be it current SLNOS ones or potential future ones.

[oxij]

How it should be used?

This infra is equivalent to "Global USE flags" of Gentoo, see https://gentoo.org/support/use-flags/. Local flags are going to be pretty easy to add after #50643, but before that it would need yet another callPackage which would take package name as a parameter.

In NixOS the above should be used similarly to

nixpkgs.config.use = {
  pulseaudio = false;
  systemd = false;
};

Also, if "headless" use-flag is introduced, would it be possible to build "headless" world by Hydra?

Yes, with some more infra similar to that was discussed in #29135. I.e. I imagine another copy of pkgs as pkgs.pkgsHeadless with headless use-flag set by default.

most popular (in your opinion) useflags

See the above link to Gentoo's page. Things like headless, pulseaudio, systemd, pam and static are popular.

[oxij]

I bit the bullet and just changed everything that needs fixing for this to become a noop for nix-env -qaP diff. I.e. now this can be used as-is with current nixpkgs. I also added an example of one use flag for mpv here for illustrative purposes. The OP was updated to reflect all of this.

[danbst]

@oxij I think Hydra relationship is the most interesting question here. Because without Hydra, such useflags can be created with help of another method: #44196

With that PR applied I can define an overlay use.nix:

self: super: let

  inherit (super) lib;

  useDefinitions = {
    pulseAudio = value: if value then {} else {
      pulseaudio = null;
      libpulseaudio = null;
      pulseSupport = false;
      pulseaudioFull = null;
      pulseaudioSupport = false;
      libpulseaudio-vanilla = null;
    };
  };

  # actually, nixpkgs.config.use, but not sure how to get it correctly in overlay
  useFlags = {
    pulseAudio = false;
  };

  packages = let
    setOfUseSets = lib.mapAttrs (n: v: useDefinitions.${n} v) useFlags;
    useSet = lib.foldl' (acc: x: acc // x) {} (builtins.attrValues setOfUseSets);
  in lib.mapAttrs (n: v:
        if builtins.isAttrs v
            then v.overrideWithScope or (x: v) useSet
            else v
    ) super;

in packages

I still need a small patch in nixpkgs to make mpv example work:

diff --git a/pkgs/development/libraries/ffmpeg/4.nix b/pkgs/development/libraries/ffmpeg/4.nix
index 9821357de3b..03481cafb11 100644
--- a/pkgs/development/libraries/ffmpeg/4.nix
+++ b/pkgs/development/libraries/ffmpeg/4.nix
@@ -1,6 +1,7 @@
 { stdenv, callPackage, fetchpatch
 # Darwin frameworks
 , Cocoa, CoreMedia
+, libpulseaudio
 , ...
 }@args:

And, finally a demo:

$ NIX_PATH=nixpkgs=$PWD nix-instantiate -A mpv
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/xsly175q6rg9my9s5rj1zi3gdywjshs2-mpv-0.29.1.drv

$ nix-store -qR /nix/store/xsly175q6rg9my9s5rj1zi3gdywjshs2-mpv-0.29.1.drv | grep pulse

$

[7c6f434c]

Note that the minimal profile doesn't even use overlays. I think the problem with overlays is not so that they don't allow to achieve something, but that we don't even trust them to be good enough for use in mainline NixOS.

(Do they have an evaluation overhead or what?)

I would expect an overlay that overrides everything in xorg to null and probably overrides a bit more stuff (like gtk) to have a chance of doing the right thing, but haven't tried it.

[danbst]

@7c6f434c all stages in stages.nix are overlays. Even packageOverrides argument is massaged to become an overlay. So overlays are not insecure, rather autoloading from ~/.config/nixpkgs/overlays is questionable.

Yes, there is evaluation overhead. I'm comparing instantiating mpv without my overlay and with:

• totalBytes 58898944 -> 74932640 (16Mb increase)

For my system (without and with overlay and patch):

• totalBytes: 425760976 -> 473587824 (48Mb increase)

There are other problem with this overlay though: it doesn't work for gnome2 (I had to filter out all gnome2 packages), it doesn't work in some cases for nested attributes, #44196 has to be patched to not blow up on memory even more...

[7c6f434c]

So overlays are not insecure, rather autoloading from `~/.config/nixpkgs/overlays` is questionable.

I didn't really mean security, I meant efficiency/predictability combination.

There are other problem with this overlay though: it doesn't work for `gnome2` (I had to filter out all gnome2 packages), it doesn't work in some cases for nested attributes, #44196 has to be patched to not blow up on memory even more...

I wonder if there is any restructuring of Nixpkgs that could reduce the amount of pitfalls around overlays; waiting for Nix3 and full language support for extensible attribute sets doesn't sound like an attractive short-term plan… If not, maybe something like use flags is indeed the best option.

[spacefrogg]

Coming from a long-term Gentoo experience myself. I understand the wish for Gentoo-like use-flags. Especially in the context of nixpkgs, though, I don't see a particular use for the proposed idiom.

I totally agree, and I would like to stress this point, that we should strive for an agreed-upon set of use-flags that can be de-/activated on the client-side, so to speak. This alone, though, would require a rethinking of a lot of packages' build scripts to support headless/database-type-select/language-binding/whatever operation.

I don't see any use for the double function idiom. With a defined use-flags naming scheme, this could just run along callPackage supported by overlays. callPackage can do everything use-flags can, today, but it can do much more. And for the proposed approach to work you would need to have a naming scheme anyway, making the double function idiom obsolete.

What I do see as a major improvement would be to build some nixos module infrastructure that creates an ad hoc overlay that sets all relevant use-flags and the necessary library dependencies. I believe, this curated infrastructure would be necessary anyhow, because Gentoo use-flags can do something that we cannot express easily right now. Use-flags can create use-flag dependencies on other packages. I know there are assert statements, but they broke quite often in the past.

From a package implementors point of view, the main concern is, that one has to write the build scripts with the possible use flag combinations in mind. I implemented something like the following locally but never came around to polish it:

{ stdenv
, libA, libA-variantB
, pkgB, pkgB-variantB }:
stdenv.mkDerivation {
...
  buildInputs = [ libA pkgB ];
  meta = {
  ...
    validArguments = { libA = [ libA          libA-variantB libA ];
                       pkgB = [ pkgB-variantB pkgB          null ]; };
  };
}

When I implemented this scheme, I used it to automatically build a particular package in all interesting combinations (three in this case). This scheme could, of course, also be used to enforce particular dependencies.

I played with several other approaches but came to the conclusion that the information about reasonable package variants have to be kept in the package itself.

[danbst]

offtopic>

I wonder if there is any restructuring of Nixpkgs that could reduce the amount of pitfalls around overlays; waiting for Nix3 and full language support for extensible attribute sets doesn't sound like an attractive short-term plan…

One of the ways is described in #54266 (comment) - add a "guidance" function to each overlay, including all-packages, so it defines (guides) how various overlay components compose. It could be used here as well - override only leaves, recurse into sets.
/offtopic>

@oxij btw, #39580 is nice! Why didn't you want to continue in that way? I've tried it, it still had to have a fix, but small one:

diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix
index 997fbd7ed73..a2bf393c59e 100644
--- a/pkgs/top-level/all-packages.nix
+++ b/pkgs/top-level/all-packages.nix
@@ -18403,9 +18403,8 @@ in
   mpv = callPackage ../applications/video/mpv rec {
     inherit lua;
     waylandSupport     = stdenv.isLinux;
     alsaSupport        = !stdenv.isDarwin;
-    pulseSupport       = !stdenv.isDarwin;
     rubberbandSupport  = !stdenv.isDarwin;
     dvdreadSupport     = !stdenv.isDarwin;
     dvdnavSupport      = !stdenv.isDarwin;
     drmSupport         = !stdenv.isDarwin;

[grahamc]

Sounds like an actual RFC might be in order.

[oxij]

@danbst

btw, #39580 is nice! Why didn't you want to continue in that way? I've tried it, it

I like the NixOS module part, but I don't like the fact that it creates another set of attributes and mangles them with packages. IMHO that's worse than the current state of random `*Support` and `with*` options intermingled with `null`s.

With that PR applied I can define an overlay `use.nix`:

Actually, this implementation can be simplified to not require `overrideWithScope` and at the same time made pretty okay metric-wise by not using overlays. See #56158, where I married your code to #56109. (I especially like how `useDefinitions` is an attrset of functions that can do weird stuff, this is cool.) #56158 is now an equivalent to this UI-wise, but simpler. Cool, huh! Married with modules of #39580 that might be the ultimate solution, IMHO. @spacefrogg That's all good valid points.

From a package implementors point of view, the main concern is, that one has to write the build scripts with the possible use flag combinations in mind. I implemented something like the following locally but never came around to polish it:

I had a similar idea #36229 (comment) which is a bit more general.

I don't see any use for the double function idiom.

Well, over new #56158 it doesn't give much, I agree. But, generally, the point of this separation was typing. Separating use flags into a separate argument you "type" them, which means you can do weird stuff with them easily, like easily fuzzing them, for instance. In any case, looking at the big blob of changes in this PR again I now think that at the very least this implementation shows that `callPackage` needs to be split into `callPackage` (which actually makes a package) and `callWithScope` (or whatever) which just substitutes scope, as those are semantically different things.

[oxij]

@danbst

I've tried it, it still had to have a fix, but small one:

Yep, that's what I was talking about in #39580 (comment). See the second commit of #56088 :)

[LnL7]

I'm personally not a fan of this kind of global flags because it happens to the entire tree, resulting in large permutations and unpredictable behaviour. For example this pattern isn't usable for packages that are part of the stdenv without making our cache entirely useless for the user. That said, global config options, while not prevalent, are already a thing so not everybody shares this opinion.

What would be much more interesting to me is "enable flag to all runtime dependencies of a package". Think replaceSystemDependencies/replaceDependency but for flags/options.

Maybe it's still missing, but it's not clear to me how applying flags for just one package would work with the new format and compatibility is probably also something to look into. Giving out of tree usages a transition period seems given that a significant amount of packages changed.

[zimbatm]

{ pulseaudio ? false }:

haha @oxij, the crusade against pulseaudio continues :)

[samueldr]

{ pulseaudio ? false }:

haha @oxij, the crusade against pulseaudio continues :)

I'm not sure this is (entirely) related. I think this instead is to make all "use" flags logically work the same. All use are true for enabled and false for disabled. I don't see proof in the implementation, but I would assume that in the interest of making this a no-rebuild change it would default to true through flags instead of through tortuous package-dependent logic.

Even, here I'm doubting the "mpv: make an example use-flag" commit is right; the intermediary function has pulseaudio ? stdenv.isLinux, which I would rather see as pulseaudio ? false and the default use flags have pulseaudio = stdenv.isLinux or similar. Am I misinterpreting something in the demo?

[oxij]

@samueldr Yes, I think the ultimate solution is to have #56158 married with NixOS module like #39580 had. (Which I will do after I discover a good way to marry them, typing a part of config is not trivial, it turns out.) Plus split callPackage into callPackage and callWithScope, but that's of low priority.

[oxij]

Closed in favor of #56227.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/where-are-optional-deps-in-nixpkgs-standardised-as-a-prominent-reference-for-pkgs-authors/30397/1