WIP lib: Better extensible attrsets

[Ericson2314]

Motivation for this change

This is a WIP implementation of something like @edolstra's {{...}} recursive attrsets (from older https://gist.github.com/edolstra/29ce9d8ea399b703a7023073b0dbc00d), but with the self-super pattern. I'm all for the more ambitious goals (sharing abstractions for nixos/nixpkgs, no callPackage, prioritizes or something finer-grained over super sledge-hammer, etc), but fear without incremental steps like this, they won't happen.

• Like it or not, self super is today's semantics for Nixpkgs
• Doesn't try to fight callPackage, works very well with self.callPackage in the manner of Haskell package sets.
• Recursion on merge solves the problem of deep overrides in overlays / recursive package sets

In particular, having both the inner and outer fixed points solves the problems with #44196 and my injectOverride alternative:

• "Copying" (rebinding) an inner extensible attribute set just works, future overrides of the original won't affect the copy
• overriding inside works per recursive merge
• no overrideScope needed as one can just use makeScope to bind a local callPackage which will be updated automatically by extensible attrset merge.

The last commit is an example rushed example of how it simplifies llvmPackages_5 and darwin inner attribute set overriding in the Darwin bootstrapping. Further simplification would come from the unrelated change of having just llvmPackages_*.{tools,libraries}, rather than that + the legacy smushing of everything inside those alongside.

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Assured whether relevant documentation is up to date
• Fits CONTRIBUTING.md.

---

CC @ElvishJerricco

[edolstra]

No, this is a wrong direction for Nix. I don't want to have to explain to users what a "monoidal extension" is. The Nix language (as a DSL) should provide the proper concepts, they should not be provided in a library. Also, I suspect that this will make Nixpkgs evaluation even slower and memory-hungry.

[Ericson2314]

@edolstra don't miss the forest for the trees :)

1. All the monoid stuff is just an implemention detail. It can be inlined away.

2. If we make zipAttrsWith a built-in there should be little performance impact. (Typically the overlay atteset is vastly smaller, and the overriding vs new attrs vastly smaller than that.)

[Ericson2314]

Also, there's a separate issue that it makes little sense to implement a DSL and try it out in one step: One cannot know a priori if the abstraction works, and changing C++ Nix everytime the abstraction needs tweaking is too slow a debug cycle.

Changes like this raise the bar, slowly kneading nixpkgs into shape towards the final DSL. Both the refactoring of existing packages and fleshing out of the DSL can take place incrementally. Only once we've converged on a stable interface that's used by a wide range of package does it make sense to freeze and enshrine it with new syntax.

[ElvishJerricco]

I agree with @Ericson2314 about trying to do things incrementally; we're not going to be able to convert all of nixpkgs to the new system at once whenever we do get the language level feature. But @Ericson2314, I would recommend finding more friendly names for this stuff. IMO this has little to do with the term "monoid", because it's not actually abstract over the monoid in use. It'd be like calling || "monoidal choice".

[edolstra]

@Ericson2314 That's what we've been doing for the last few years: we have stuff like the module system as a library feature. Now it's time to do it properly as a language feature.

[Ericson2314]

@ElvishJerricco no it actually is abstract over all monoids in this implementation, just as we talked about long ago. The user doesn't need to worry about arbitrary monoids, but by the same token it is bad and a sign of the WIP-ness of this PR that they currently use any of the monad-polymorphic stuff.

@edolstra Yes, NixOS is ready, but Nixpkgs-the-package-set isn't. The most widely used things in nixpkgs like stdenv and cc-wrapepr are known-bad, and the langauge-specific/nested package stuff is not at all standardized. If your proposal only talked about NixOS, it would be one thing, but as it aims for the unified abstraction for packages and configurations, we have no choice but to first force Nixpkgs-the-package-set into good condition.

[Ericson2314]

@edolstra @ElvishJerricco and @angerman in particular are doing some cool work with next-gen Haskell infra using the NixOS module system. A wonderful yet realistic sequence of events to me looks like:

1. Merge cleaned up version of this
2. Get everything in nixpkgs using nested self: super: { .. } package sets / overriding
3. New Haskell stuff done with NixOS modules, possible extended modules
4. Use @Profpatsch's type checker work to separate merging / checking aspects of module system
5. Use Module system instead of self: super: { .. } package sets.
6. Deal with any remaining callPackage and friends (e.g. make that compat layer) (Your older stuff / @nbp SOS).
7. NixOS + Nixpkgs all uses improved module system (maybe with some deprecated layers)
8. Some static typing stuff is finally ready?
9. Enshrine everything with new syntax.

[mmahut]

Are there any updates on this pull request, please?

[stale]

Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the
   related code and @ mention them in a comment.
2. Ask on the NixOS Discourse. 3. Ask on the #nixos channel on
   irc.freenode.net.

[infinisil]

Relevant rfc: NixOS/rfcs#67

[stale]

I marked this as stale due to inactivity. → More info