Add pkgs.overrideWithScope

[nbp]

Add pkgs.overrideWithScope. This function is similar to pkgs.override, except that it filters the arguments like callPackage does.

While this might seems like a small addition, this is actually a really really big change for the future of Nixpkgs. This function can help us simplify a lot the manipulation of emacsPackages, pythonPackages, haskellPackages, linuxPackages, ... This can also help us rewrite the bootstrapping done before all-packages and move it within all-packages.nix. and on a simpler note, this will help solving the #43755 as explained in #43828.

The reason which makes this function is a revolution for the implementation of package sets is that it make it simple to ""recursively"" (dependencies of dependencies of dependencies) and conherently override the input of a set of packages with a simple function, which does not appear in future overlay.

To make it clearer, here is a set of overlay:

import <nixpkgs> { overlays = [
  # Add an interpreter and its package set.
  (self: super: 
  let interpreterPackages = withSet: {
      foo = super.lib.callPackageWith withSet ./pkgs/interpreter/foo { /* interpreter libxml */ };
      bar = super.lib.callPackageWith withSet ./pkgs/interpreter/bar { /* interpreter foo */ };
      baz = super.lib.callPackageWith withSet ./pkgs/interpreter/baz { /* interpreter foo bar qt */ };
      qux = super.lib.callPackageWith withSet ./pkgs/interpreter/qux { /* interpreter bar baz */ };
    };
  in
  rec {
    interpreter27 = super.callPackage ./pkgs/interpreter_27.nix {};
    interpreter35 = super.callPackage ./pkgs/interpreter_35.nix {};
    interpreter27Packages = let withSet = self // { interpreter = interpreter27; } // self.interpreter27Packages; in
      super.lib.mapAttrs (n: v: v.overrideWithScope withSet) (interpreterPackages withSet);
    interpreter35Packages = let withSet = self // { interpreter = interpreter35; } // self.interpreter35Packages; in
      super.lib.mapAttrs (n: v: v.overrideWithScope withSet) (interpreterPackages withSet);
  })
  # Change a package dependency. Like any ordinary package, as opposed as today.
  (self: super: {
    interpreter27Packages = super.interpreter27Packages // {
      # Will change foo within bar, baz, qux (as expected).
      foo = super.interpreter27Packages.foo.override { libxml = null; };
    };
  })
  # Duplicate a package set with a custom interpreter.
  (self: super: {
    interpreter27ProfilePackages = let withSet = self // self.interpreter27ProfilePackages; in
      super.lib.mapAttrs (n: v: v.overrideWithScope withSet) (super.interpreter27Packages // {
      interpreter = super.interpreter27.override { withProfiler = true; };
    });
  })
]; }

What is important to note, is that:

• Changing a package within an existing attribute set, except for the attribute set manipulation, is like any ordinary package.
• Creating a forked configuration (with a profiler for example), is as simple as changing the interpreter in a new set which is isolated with overrideWithScope.

To make it simple, pkgs.overrideWithScope make it possible to leverage the Nixpkgs fix-point without the burden of writing too many pkgs.override by hand (which need to have no-more than the expected set of arguments).

As it leverage the Nixpkgs fix-point, there is no need to unwrap the function to override packages within a given set, like is done with package sets today (Haskell being infamous due to the stacking of multiple fix-points).

Ideally, we would want to have the pkgs.overrideWithScope evaluated before the resolution of the arguments of a given package, which would not be possible until we add some post-processing of all packages at the end of Nixpkgs fix-point. This enforces that we have to provide the withSet argument to the interpreter set, and implies that we cannot have a raw set of packages which lives independently of a given interpreter version. (Note, one could cheat by providing a fake version of missing packages interpreter = null; libxml = null; qt = null; )

Another issue, which already exists but is made explicit here, is that self // self.interpreterPackages is inefficient in terms of memory because attribute sets are not lazy. This does not worry me much as this is a fixable issue in Nixpkgs, as long as the expression is strictly used as argument of callPackagesWith and overrideWithScope

Other suggestions:

• Renaming overrideWithScope to overrideWith.
• Replacing override by overrideWithScope.

[Ericson2314]

I'm not quite sure about this idiom. Say we had this function:

let
  # injectOverride
  # :  (name: String)
  # -> ({ a } -> { a } -> { a })
  # -> ({ name: {a}; ..b } -> { name: {a}; ..b } -> { name: {a}; ..b })
  injectOverride = name: f: self: super: { 
    "${name}" = super.${name} // f self.${name} super.${self} // {
      __overlay__ = lib.composeExtensions (super.__inner__ or (_: _: {})) f;
    };
  }; 

Then we can just do:

import <nixpkgs> { overlays = [
  # Add an interpreter and its package set.
  (self: super: 
  let 
    pkgs = self;
    interpreterPackages = self: {
      callPackage = pkgs.newScope self;
      foo = self.callPackage ./pkgs/interpreter/foo { /* interpreter libxml */ };
      bar = self.callPackage ./pkgs/interpreter/bar { /* interpreter foo */ };
      baz = self.callPackage ./pkgs/interpreter/baz { /* interpreter foo bar qt */ };
      qux = self.callPackage ./pkgs/interpreter/qux { /* interpreter bar baz */ };
    };
  in
  rec {
    interpreter27 = self.callPackage ./pkgs/interpreter_27.nix {};
    interpreter35 = self.callPackage ./pkgs/interpreter_35.nix {};
  } 
  // injectOverride "interpreter27Packages"
       (lib.extend (_: _: { interpreter = interpreter27; }) interpreterPackages)
  // injectOverride "interpreter35Packages"
       (lib.extend (_: _: { interpreter = interpreter35; }) interpreterPackages))
  # Change a package dependency. Like any ordinary package, as opposed as today.
  (injectOverride "interpreter27Packages" {
      # Will change foo within bar, baz, qux (as expected).
      foo = super.interpreter27Packages.foo.override { libxml = null; };
  })
  # Duplicate a package set with a custom interpreter.
  (self: super: injectOverride "interpreter27ProfilePackages"
    super.interpreter27Packages.__overlay__ self super)
]; }

[nbp]

@Ericson2314
The problem with the idiom you are suggesting is that users of a given sub-set of packages will have to know about the injectOverride function, and know when to use it. As opposed to overrideWithScope which would only be required at the creation/cloning of the sub-set.

(Also this should always be super.callPackage)

Honestly, if post-processing of Nixpkgs was an option, and every package would use callPackage (which is far from being the case), then we could make the callPackage function return a functor like object which is derived into a derivation only after going through the Nixpkgs fix-point, and thus we could go with just the following:

import <nixpkgs> { overlays = [
  # Add an interpreter and its package set.
  (self: super: 
  let interpreterPackages = {
      foo = super.callPackage ./pkgs/interpreter/foo { /* interpreter libxml */ };
      bar = super.callPackage ./pkgs/interpreter/bar { /* interpreter foo */ };
      baz = super.callPackage ./pkgs/interpreter/baz { /* interpreter foo bar qt */ };
      qux = super.callPackage ./pkgs/interpreter/qux { /* interpreter bar baz */ };
    };
  in
  rec {
    interpreter27 = super.callPackage ./pkgs/interpreter_27.nix {};
    interpreter35 = super.callPackage ./pkgs/interpreter_35.nix {};
    interpreter27Packages = { interpreter = interpreter27; } // interpreterPackages;
    interpreter35Packages = { interpreter = interpreter35; } // interpreterPackages;
  })
  # Change a package dependency. Like any ordinary package, as opposed as today.
  (self: super: {
    interpreter27Packages = super.interpreter27Packages // {
      # Will change foo within bar, baz, qux (as expected).
      foo = super.interpreter27Packages.foo.override { libxml = null; };
    };
  })
  # Duplicate a package set with a custom interpreter.
  (self: super: {
    interpreter27ProfilePackages = super.interpreter27Packages // {
      interpreter = super.interpreter27.override { withProfiler = true; };
    });
  })
]; }

As the scope would be determine by the location of the package within Nixpkgs. But this future is not yet there, because not every package uses callPackage, and not every dependency comes from self.

[aszlig]

Very cool, now I can finally get rid of a bit of churn in some of my Nix expressions, thanks :-)

One minor nitpick however:

If my assumption is correct that newScope should always be an attribute set (intersectAttrs would bail out if not), there is no need to go through isFunction in overrideWith above again, so removing intersectArgs and changing overrideWithScope to the following would avoid that:

newScope: origArgs // (if fnArgs != {} then builtins.intersectAttrs fnArgs newScope else newScope)

[aszlig]

Also, just for the sake of completeness (this nitpick is so minor that it isn't even one), maybe add overrideWithScope in the output for the nix repl examples below.

[oxij]

Bear with me for a minute. - Q: What is `nixpkgs`? A: Roughly, a fixed point over an attrset of packages (functions of type `deps -> derivation` applied to `nixpkgs` itself). - Q: What is `pkg.override`? A: A way to override `deps` with an attrset. I would call it `overrideArgs`, but whatever. - Q: What are `pkg.overrideAttrs` and `pkg.overrideDerivation`? A: Ways to override `derivation` in the above. I have no idea why we need both, I think `overrideAttrs` covers all the cases. - Q: What is `config.packageOverrides`? A: A way to override `nixpkgs` by changing the base of the fixed point by merging `nixpkgs` with another attrset. - Q: What are `overlays`? A: A Way to override `nixpkgs` by introducing arbitrary number of additional steps into the fixed point. - Q: What `overrideWithScope` of this PR adds? A: A way to override `deps` with an attrset, duplicating `callPackage` filtering. - Q: Why? A: #43828 (review) I'm 👍 on adding this. Btw, `override*` functions have two dimensions now: they override either deps or the derivation, and either apply raw attrs or filter them out. - `override`: deps, raw - `overrideWithScope`: deps, filter - `overrideDerivation`: should not exist - `overrideAttrs`: derivation, raw I have no idea if `overrideAttrs` that filters has any use, though. Btw btw, if we treat `nixpkgs` as a fixed point over `pkgs -> pkgs` applied to `all-packages.nix` (think `metanixpkgs` with `nixpkgs` as a single package called `nixpkgs` that is made by `callPackage` over itself) with `overlays` introducing arbitrary steps into the fixed point there, then `overlays` are just repeated `override`s of `metanixpkgs` without filtering. Which bears two questions: - Wouldn't it be nice to expose the whole `nixpkgs` as a package that can be `override`n? For instance, this way use-flags will be just such `override`s. - What is the equivalent of `overlays` for packages?

[twhitehead]

For a bit now, I've been thinking it would be nice if the callPackage mechanism could add an override function that would do the equivalent of adding another overlay and then re-evaluate the package.

The idea would be then be that you could also effect changes in all the dependencies of a package as well. Typical examples would be where the dependent packages must also be changed or else you windup with the final executable sucking in two different versions of the same shared library (you see this in some existing expressions with the version of boost or cuda libraries used in its dependencies).

I originally thought this might be what this pull request was, but, upon closer looking, I don't believe so.
I thought I would mention it here though as it does seem there is some significant overlap.

Thanks! -Tyson

PS: Technically, I expect, it could make sense to only apply the overlay to runtime dependencies (i.e., not to the build time ones).

[timokau]

Can we move forward with this?

[tomberek]

Does this need review? testers? redesign?

[Ericson2314]

I now think #51213 is a better approach than this and my first comment's suggestion.

[danbst]

In #54266 I make it even simpler to extend existing package sets, so instead of

  (self: super: {
    interpreter27Packages = super.interpreter27Packages // {
      # Will change foo within bar, baz, qux (as expected).
      foo = super.interpreter27Packages.foo.override { libxml = null; };
    };
  })

you can write

  (self: super: { _merge_interpreter27Packages = true; })
  (self: super: {
    interpreter27Packages.foo = super.interpreter27Packages.foo.override { libxml = null; };
  })

[nixos-discourse]

This pull request has been mentioned on Nix community. There might be relevant details there:

https://discourse.nixos.org/t/wip-rfc-a-new-nixpkgs-frontend-for-language-infrastructures/3447/9

[FRidh]

A year later and I had another look at this, because I need a way to override the Nix packages set from within itself.

@nbp, @Ericson2314 and I had a brief discussion last year on IRC as well about this PR
https://logs.nix.samueldr.com/nixos-dev/2018-07-31#1433188;

The main issue I still have is not having a good way to refer to packages in the main set pkgs. We need a solution there. @nbp proposed renaming e.g. these packages by prepending _ or suffixing '.

[InLaw]

Is there a target when the pending review will take place?

[stale]

I marked this as stale due to inactivity. → More info

[aviallon]

Still relevant to me.

[AndersonTorres]

Relevant to no one else. I will close this.
Feel free to restart.

[collares]

This is relevant for #43755.

[AndersonTorres]

This is relevant for #43755.

A five-year-old PR stalled for two years - with merge conflicts! Hum...

Edit: I am not doubting on the technical merits of those huge modifications.

However, they are very old: five years, ten releases, lots of new packages inbetween.

A cleaner slate is imperative (pun not intended) in order to get such code in shape.

Also, anyone is free to open this PR or restart and link it.