lib.extendMkDerivation: init

doc/build-helpers.md

@@ -17,6 +17,7 @@ There is no uniform interface for build helpers.
 [Language- or framework-specific build helpers](#chap-language-support) usually follow the style of `stdenv.mkDerivation`, which accepts an attribute set or a fixed-point function taking an attribute set.
 
 ```{=include=} chapters
+build-helpers/fixed-point-arguments.chapter.md
 build-helpers/fetchers.chapter.md
 build-helpers/trivial-build-helpers.chapter.md
 build-helpers/testers.chapter.md

doc/build-helpers/fixed-point-arguments.chapter.md

@@ -0,0 +1,202 @@
+# Fixed-point arguments of build helpers {#chap-build-helpers-finalAttrs}
+
+As mentioned in the beginning of this part, `stdenv.mkDerivation` could alternatively accept a fixed-point function. The input of such function, typically named `finalAttrs`, is expected to be the final state of the attribute set.
+A build helper like this is said to accept **fixed-point arguments**.
+
+Build helpers don't always support fixed-point arguments yet, as support in [`stdenv.mkDerivation`](#mkderivation-recursive-attributes) was first included in Nixpkgs 22.05.
+
+## Defining a build helper with `lib.extendMkDerivation` {#sec-build-helper-extendMkDerivation}
+
+Developers can use the Nixpkgs library function [`lib.customisation.extendMkDerivation`](#function-library-lib.customisation.extendMkDerivation) to define a build helper supporting fixed-point arguments from an existing one with such support, with an attribute overlay similar to the one taken by [`<pkg>.overrideAttrs`](#sec-pkg-overrideAttrs).
+
+:::{.example #ex-build-helpers-extendMkDerivation-simple}
+
+# Example definition of `mkLocalDerivation` extended from `stdenv.mkDerivation` with `lib.extendMkDerivation`
+
+We want to define a build helper named `mkLocalDerivation` that builds locally without using substitutes by default.
+
+Instead of taking a plain attribute set,
+
+```nix
+{
+  preferLocalBuild ? true,
+  allowSubstitute ? false,
+  ...
+}@args:
+
+stdenv.mkDerivation (
+  args
+  // {
+    # Attributes to update
+    inherit
+      preferLocalBuild
+      allowSubstitute
+      ;
+  }
+)
+```
+
+we could define with `lib.extendMkDerivation` an attribute overlay to make the result build helper also accepts the the attribute set's fixed point passing to the underlying `stdenv.mkDerivation`, named `finalAttrs` here:
+
+```nix
+lib.extendMkDerivation {
+  constructDrv = stdenv.mkDerivation;
+  extendArgs =
+    finalAttrs:
+    {
+      preferLocalBuild ? true,
+      allowSubstitute ? false,
+      ...
+    }@args:
+
+    # No need of `args //` here.
+    # The whole set of input arguments is passed in advance.
+    {
+      # Attributes to update
+      inherit
+        preferLocalBuild
+        allowSubstitute
+        ;
+    };
+}
+```
+:::
+
+Should there be a need to apply extra changes to the result derivation, pass the derivation transformation function to `lib.extendMkDerivation` as `lib.customisation.extendMkDerivation { transformDrv = drv: ...; }`.
+
+### Optionally remove specified arguments with `lib.extendMkDerivation` {#sec-build-helper-extendMkDerivation-removedArgNames}
+
+Many existing build helpers only pass part of their arguments down to their base build helper, and developers may also remove some arguments when implementing compatibility layers. To address that, `lib.extendMkDerivation` optionally takes a list of argument names to remove (`removedArgNames`).
+
+:::{.example #ex-build-helpers-extendMkDerivation-removedArgNames}
+
+# Example definition of `mkLocalDerivation` with `lib.extendMkDerivation` and `removedArgNames` specified
+
+Let's take as our example a build helper `mkLocalDerivation`, which requires an argument `specialArg` that cannot be passed to `stdenv.mkDerivation`.
+
+```nix
+{
+  preferLocalBuild ? true,
+  allowSubstitute ? false,
+  specialArg ? (_: false),
+  ...
+}@args:
+
+stdenv.mkDerivation (
+  removeAttrs [
+    # Don't pass specialArg into mkDerivation.
+    "specialArg"
+  ] args
+  // {
+    # Arguments to pass
+    inherit preferLocalBuild allowSubstitute;
+    # Some expressions involving specialArg
+    greeting = if specialArg "hi" then "hi" else "hello";
+  }
+)
+```
+
+Specify `removedArgNames` attribute so that `specialArg` won't be passed to `stdenv.mkDerivation`.
+
+```nix
+lib.extendMkDerivation {
+  constructDrv = stdenv.mkDerivation;
+  extendArgs =
+    finalAttrs:
+    {
+      preferLocalBuild ? true,
+      allowSubstitute ? false,
+      specialArg ? (_: false),
+      ...
+    }@args:
+    {
+      # Arguments to pass
+      inherit
+        preferLocalBuild
+        allowSubstitute
+        ;
+      # Some expressions involving specialArg
+      greeting = if specialArg "hi" then "hi" else "hello";
+    };
+  removedArgNames = [
+    # Don't pass specialArg into mkDerivation.
+    "specialArg"
+  ];
+}
+```
+:::
+
+Nevertheless, those removed arguments are often hard to override via `<pkg>.overrideAttrs`, leading to the use of custom overriders (such as `overridePythonPackage`) and extra complexity when overriding. In the long run, we would like to refactor build helpers to pass every argument down to `stdenv.mkDerivation`, so that they can all be overridden by [`overrideAttrs`](#sec-pkg-overrideAttrs), eliminating the use of custom overriders (e.g., `overridePythonAttrs`).
+
+The following example shows a smooth migration that empties the `removedArgNames` by guiding users to pass `specialArg` via `passthru`:
+
+:::{.example #ex-build-helpers-extendMkDerivation-removedArgNames-migration}
+
+# Getting rid of `removedArgNames` in a backport-compatible way
+
+Refactor the definition to pass `specialArg` properly while keeping some backward compatibility.
+
+```nix
+lib.extendMkDerivation {
+  constructDrv = stdenv.mkDerivation;
+  extendArgs =
+    finalAttrs:
+    {
+      preferLocalBuild ? true,
+      allowSubstitute ? false,
+      ...
+    }@args:
+    {
+      # Arguments to pass
+      inherit
+        preferLocalBuild
+        allowSubstitute
+        ;
+      passthru = {
+        specialArg =
+          if (args ? specialArg) then
+            # For backward compatibility only.
+            # TODO: Convert to throw after XX.XX branch-off.
+            lib.warn "mkLocalDerivation: Expect specialArg under passthru." args.specialArg
+          else
+            (_: false);
+      } // args.passthru or { };
+      # Some expressions involving specialArg
+      greeting = if finalAttrs.passthru.specialArg "hi" then "hi" else "hello";
+    };
+  removedArgNames = [
+    # Don't pass specialArg into mkDerivation.
+    "specialArg"
+  ];
+}
+```
+
+Remove the `removedArgNames` specification after deprecating `specialArg`.
+
+```nix
+lib.extendMkDerivation {
+  constructDrv = stdenv.mkDerivation;
+  extendArgs =
+    finalAttrs:
+    {
+      preferLocalBuild ? true,
+      allowSubstitute ? false,
+      ...
+    }@args:
+    # The arguments to pass
+    {
+      inherit
+        preferLocalBuild
+        allowSubstitute
+        ;
+      passthru = {
+        specialArg =
+          (lib.throwIf (args ? specialArg) "mkLocalDerivation: Expect specialArg under passthru.")
+            (_: false);
+      } // args.passthru or { };
+      # Some expressions involving specialArg
+      greeting = if finalAttrs.passthru.specialArg "hi" then "hi" else "hello";
+    };
+}
+```
+:::

doc/redirects.json

@@ -1,7 +1,19 @@
 {
+  "chap-build-helpers-finalAttrs": [
+    "index.html#chap-build-helpers-finalAttrs"
+  ],
   "chap-release-notes": [
     "release-notes.html#chap-release-notes"
   ],
+  "ex-build-helpers-extendMkDerivation-removedArgNames": [
+    "index.html#ex-build-helpers-extendMkDerivation-removedArgNames"
+  ],
+  "ex-build-helpers-extendMkDerivation-removedArgNames-migration": [
+    "index.html#ex-build-helpers-extendMkDerivation-removedArgNames-migration"
+  ],
+  "ex-build-helpers-extendMkDerivation-simple": [
+    "index.html#ex-build-helpers-extendMkDerivation-simple"
+  ],
   "neovim": [
     "index.html#neovim"
   ],
@@ -38,6 +50,12 @@
   "sec-allow-insecure": [
     "index.html#sec-allow-insecure"
   ],
+  "sec-build-helper-extendMkDerivation": [
+    "index.html#sec-build-helper-extendMkDerivation"
+  ],
+  "sec-build-helper-extendMkDerivation-removedArgNames": [
+    "index.html#sec-build-helper-extendMkDerivation-removedArgNames"
+  ],
   "sec-modify-via-packageOverrides": [
     "index.html#sec-modify-via-packageOverrides"
   ],

lib/customisation.nix

@@ -31,6 +31,8 @@ let
     flatten
     deepSeq
     extends
+    toFunction
+    id
     ;
   inherit (lib.strings) levenshtein levenshteinAtMost;
 
@@ -730,4 +732,124 @@ rec {
     in
     self;
 
+  /**
+    Define a `mkDerivation`-like function based on another `mkDerivation`-like function.
+
+    [`stdenv.mkDerivation`](#part-stdenv) gives access to
+    its final set of derivation attributes when it is passed a function,
+    or when it is passed an overlay-style function in `overrideAttrs`.
+
+    Instead of composing new `stdenv.mkDerivation`-like build helpers
+    using normal function composition,
+    `extendMkDerivation` makes sure that the returned build helper
+    supports such first class recursion like `mkDerivation` does.
+
+    `extendMkDerivation` takes an extra attribute set to configure its behaviour.
+    One can optionally specify
+    `transformDrv` to specify a function to apply to the result derivation,
+    or `inheritFunctionArgs` to decide whether to inherit the `__functionArgs`
+    from the base build helper.
+
+    # Inputs
+
+    `extendMkDerivation`-specific configurations
+    : `constructDrv`: Base build helper, the `mkDerivation`-like build helper to extend
+    : `extendArgs` : An extension (overlay) of argument set, like the one taken by [overrideAttrs](#sec-pkg-overrideAttrs).
+    : `inheritFunctionArgs`: Whether to inherit `__functionArgs` from the base build helper (default to `true`)
+    : `removedArgNames`: Argument names to remove before passing down to `constructDrv`. Note that it doesn't remove the ones returned by `extendArgs`.
+    : `transformDrv`: Function to apply to the result derivation (default to `lib.id`)
+
+    # Type
+
+    ```
+    extendMkDerivation ::
+      {
+        constructDrv :: ((FixedPointArgs | AttrSet) -> a)
+        extendArgs :: (AttrSet -> AttrSet -> AttrSet)
+        inheritFunctionArgs :: Bool,
+        removedArgNames :: [ String ],
+        transformDrv :: a -> a,
+      }
+      -> (FixedPointArgs | AttrSet) -> a
+
+    FixedPointArgs = AttrSet -> AttrSet
+    a = Derivation when defining a build helper
+    ```
+
+    # Examples
+
+    :::{.example}
+    ## `lib.customisation.extendMkDerivation` usage example
+    ```nix-repl
+    mkLocalDerivation = lib.extendMkDerivation {
+      constructDrv = pkgs.stdenv.mkDerivation;
+      extendArgs =
+        finalAttrs: args@{ preferLocalBuild ? true, allowSubstitute ? false, specialArg ? (_: false), ... }:
+        { inherit preferLocalBuild allowSubstitute; passthru = { inherit specialArg; } // args.passthru or { }; };
+      removedArgNames = [ "specialArg" ];
+    }
+
+    mkLocalDerivation.__functionArgs
+    => { allowSubstitute = true; preferLocalBuild = true; specialArg = true; }
+
+    mkLocalDerivation { inherit (pkgs.hello) pname version src; specialArg = _: false; }
+    => «derivation /nix/store/xirl67m60ahg6jmzicx43a81g635g8z8-hello-2.12.1.drv»
+
+    mkLocalDerivation (finalAttrs: { inherit (pkgs.hello) pname version src; specialArg = _: false; })
+    => «derivation /nix/store/xirl67m60ahg6jmzicx43a81g635g8z8-hello-2.12.1.drv»
+
+    (mkLocalDerivation (finalAttrs: { inherit (pkgs.hello) pname version src; passthru = { foo = "a"; bar = "${finalAttrs.passthru.foo}b"; }; })).bar
+    => "ab"
+    ```
+    :::
+
+    :::{.note}
+    If `transformDrv` is specified,
+    it should take care of existing attributes that perform overriding
+    (e.g., [`overrideAttrs`](#sec-pkg-overrideAttrs))
+    to ensure that the overriding functionality of the result derivation
+    work as expected.
+    Modifications that breaks the overriding include
+    direct [attribute set update](https://nixos.org/manual/nix/stable/language/operators#update)
+    and [`lib.extendDerivation`](#function-library-lib.customisation.extendDerivation).
+    :::
+  */
+  extendMkDerivation =
+    let
+      extendsWithRemovedArgs =
+        removedNames: g: f: final:
+        let
+          previous = f final;
+        in
+        removeAttrs previous removedNames // g final previous;
+    in
+    {
+      constructDrv,
+      extendArgs,
+      removedArgNames ? [ ],
+      transformDrv ? id,
+      inheritFunctionArgs ? true,
+    }:
+    setFunctionArgs
+      # Adds the fixed-point style support
+      (
+        fpargs:
+        transformDrv (constructDrv (extendsWithRemovedArgs removedArgNames extendArgs (toFunction fpargs)))
+      )
+      # Add __functionArgs
+      (
+        # Inherit the __functionArgs from the base build helper
+        optionalAttrs inheritFunctionArgs (removeAttrs (functionArgs constructDrv) removedArgNames)
+        # Recover the __functionArgs from the derived build helper
+        // functionArgs (extendArgs { })
+      )
+    // {
+      inherit
+        # Expose to the result build helper.
+        constructDrv
+        extendArgs
+        removedArgNames
+        transformDrv
+        ;
+    };
 }

lib/default.nix

@@ -121,7 +121,8 @@ let
       noDepEntry fullDepEntry packEntry stringAfter;
     inherit (self.customisation) overrideDerivation makeOverridable
       callPackageWith callPackagesWith extendDerivation hydraJob
-      makeScope makeScopeWithSplicing makeScopeWithSplicing';
+      makeScope makeScopeWithSplicing makeScopeWithSplicing'
+      extendMkDerivation;
     inherit (self.derivations) lazyDerivation optionalDrvAttr warnOnInstantiate;
     inherit (self.meta) addMetaAttrs dontDistribute setName updateName
       appendToName mapDerivationAttrset setPrio lowPrio lowPrioSet hiPrio