mkDerivation: Expose pname & version in meta #68620

jtojnar · 2019-09-12T22:21:23Z

Nix only knows of name attribute and considers pname what parseDrvName spits out. Unfortunately, that fails to account for many real-life packages that have variants or dashes followed by number in their name. Since many of packages in nixpkgs already have pname corresponding to project name, we will export it here. We will fallback to the parseDrvName algorithm when pname is not present.
We handle the version similarly.

This will be useful for Repology as well as any other scripted processing of Nixpkgs JSON dump.

See: https://github.com/NixOS/nixos-homepage/issues/306

Nix only knows of `name` attribute and considers `pname` what parseDrvName spits out. Unfortunately, that fails to account for many real-life packages that have variants or dashes followed by number in their name. Since many of packages in nixpkgs already have pname corresponding to project name, we will export it here. We will fallback to the parseDrvName algorithm when pname is not present. We handle the version similarly. This will be useful for Repology as well as any other scripted processing of Nixpkgs JSON dump. See: https://github.com/NixOS/nixos-homepage/issues/306

jtojnar · 2019-09-12T23:04:48Z

Well, we need to get the project name somehow, even when pname attribute is not set. We could promote it to meta only when it is in attrs as you suggest but then scripts would still need fall back to parseDrvName themselves (or use the pname attribute introduced in NixOS/nix@cd933b2). This will at least provide an uniform interface.

On the other hand, going your proposed route route would allow us to consider meta.pname vetted, which might be beneficial. Repology could decide to consider packages without meta.pname untrusted or drop them altogether. cc @AMDmi3

jtojnar · 2019-09-13T00:26:09Z

If pname is not set then it is not a project (in repology's sense and also with the meaning that name does not contain project name and/or version)

As fair as I know, we do not have formally defined semantics for pname.

Would it be correct to extract pname and version from names like "CVE-2017-6827+CVE-2017-6828+CVE-2017-6832+CVE-2017-6835+CVE-2017-6837.patch", "unit-user-.service", "ntfs-3g_ntfsprogs-2017.3.23.tgz" or "daemon.conf" ?

Those derivations would not be listed by nix-env. Which makes me wonder how does nix-env which attributes are packages and which are not.

jtojnar · 2019-09-13T02:11:04Z

Those derivations would not be listed by nix-env. Which makes me wonder how does nix-env which attributes are packages and which are not.

It looks like recurseForDerivations is used by nix-env to traverse nested atribute sets:

https://github.com/NixOS/nixpkgs/blob/83aa6f5d7f540a088e79941e103bd85a2f0114afP/pkgs/top-level/all-packages.nix#L67

And things like auto-patchelf-hook are not filtered out. Weirdly, Repology ignores it, even though I did not see any relevant ignore rule.

vcunat · 2019-09-13T06:17:53Z

nix-env anyway will stay with Nix heuristic split (so it will split "package-unstable-20190101" as "package-unstable" and "20190101" irregardless of whether pname is "package", "package-unstable" or absent).

That's one of the reasons I really dislike this "unstable-" versions naming. It confuses things, and I can see almost no advantages of that scheme. For example, I use nix-env -i /some/nix/store/path and there it's not even possible to detect where version starts.

7c6f434c · 2019-09-13T06:20:28Z

And things like `auto-patchelf-hook` are not filtered out. Weirdly, Repology [ignores it](https://repology.org/projects/?search=auto-patchelf-hook), even though I did not see any relevant ignore rule.

Given the attention to version availability, a completely unique name without version might be enough to ignore by default.

ryantm · 2019-11-07T05:44:07Z

pkgs/stdenv/generic/make-derivation.nix

+          # algorithm when pname is not present.
+          # We handle the version similarly.
+          pname = attrs.pname or (builtins.parseDrvName attrs.name).name;
+          version = attrs.version or ((x: if x != "" then x else null) (builtins.parseDrvName attrs.name).version);


I agree with @volth that we should not have these fallbacks to parseDrvName because it is surprising to me that it would work like that. pname typically goes into name, so it is weird for name to go into pname in some cases.

Without these fallbacks the change seems quite uncontroversial. You are just saying make the attrs available in meta if they were passed in to mkDerivation.

If we want to make something like nameForRepology in the website export code with this logic, we could do that.

edolstra · 2019-11-07T10:35:05Z

I don't think this is a good idea because it creates the possibility of the attributes in the derivation being inconsistent with the ones in meta. (In other words, it denormalizes the derivation, providing possibly conflicting sources of truth.)

jtojnar · 2019-11-07T12:16:10Z

Yeah, this is definitely a concern but we do need some way to find the version. Since is not extractable from name in general case (for projects containing a -[0-9] or versions starting with [^0-9]), we either need to pass it through some channel that nix-env recognizes (like done using meta here), or teach Nix about version and pname.

ryantm · 2019-11-07T15:30:20Z

Can we add pname and version to derivationArg instead? Is there still a concern they can be desynced by overrides?

danbst · 2020-03-15T14:12:28Z

related #75813

FRidh · 2020-03-29T07:51:42Z

I think that, when pname is explicitly passed, we should check that parseDrvName pname return an empty version.

The `pname` passed to `stdenv.mkDerivation` consists of the name prefix as well as the `pname` passed to `stdenv.mkDerivation`. This is needed in case we want to test for valid `pname` in `stdenv.mkDerivation`. NixOS#68620 (comment)

stale · 2020-11-28T08:28:22Z

I marked this as stale due to inactivity. → More info

davidak · 2020-11-28T21:27:02Z

What is the status? What need to happen that this can get merged?

stale · 2021-06-03T02:36:17Z

I marked this as stale due to inactivity. → More info

AndersonTorres · 2024-11-02T10:44:31Z

What is the status? What need to happen that this can get merged? [2]

jtojnar requested review from Ericson2314 and matthewbauer as code owners September 12, 2019 22:21

jtojnar requested review from edolstra and volth September 12, 2019 22:23

ofborg bot added the 6.topic: stdenv Standard environment label Sep 12, 2019

jtojnar force-pushed the meta-pname branch from 2ad12bf to b08f56c Compare September 12, 2019 22:46

jtojnar added the 6.topic: repology https://repology.org/ label Sep 12, 2019

ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux labels Sep 12, 2019

ryantm reviewed Nov 7, 2019

View reviewed changes

FRidh mentioned this pull request Mar 29, 2020

buildPythonPackage: add prefix to pname #83656

Closed

10 tasks

jtojnar marked this pull request as draft June 1, 2020 05:20

stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Nov 28, 2020

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Nov 28, 2020

jtojnar mentioned this pull request May 26, 2021

stdenv/mkDerivation: derive pname and version when not given #124520

Closed

9 tasks

stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 3, 2021

wegank added the 2.status: merge conflict This PR has merge conflicts with the target branch label Mar 20, 2024

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Mar 20, 2024

wegank added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jul 4, 2024

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Nov 2, 2024

AndersonTorres added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Nov 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

mkDerivation: Expose pname & version in meta #68620

mkDerivation: Expose pname & version in meta #68620

jtojnar commented Sep 12, 2019

jtojnar commented Sep 12, 2019

jtojnar commented Sep 13, 2019

jtojnar commented Sep 13, 2019

vcunat commented Sep 13, 2019

7c6f434c commented Sep 13, 2019 via email

ryantm Nov 7, 2019

ryantm Nov 7, 2019

edolstra commented Nov 7, 2019

jtojnar commented Nov 7, 2019 •

edited

Loading

ryantm commented Nov 7, 2019

danbst commented Mar 15, 2020

FRidh commented Mar 29, 2020

stale bot commented Nov 28, 2020

davidak commented Nov 28, 2020

stale bot commented Jun 3, 2021

AndersonTorres commented Nov 2, 2024

mkDerivation: Expose pname & version in meta #68620

Are you sure you want to change the base?

mkDerivation: Expose pname & version in meta #68620

Conversation

jtojnar commented Sep 12, 2019

jtojnar commented Sep 12, 2019

jtojnar commented Sep 13, 2019

jtojnar commented Sep 13, 2019

vcunat commented Sep 13, 2019

7c6f434c commented Sep 13, 2019 via email

ryantm Nov 7, 2019

Choose a reason for hiding this comment

ryantm Nov 7, 2019

Choose a reason for hiding this comment

edolstra commented Nov 7, 2019

jtojnar commented Nov 7, 2019 • edited Loading

ryantm commented Nov 7, 2019

danbst commented Mar 15, 2020

FRidh commented Mar 29, 2020

stale bot commented Nov 28, 2020

davidak commented Nov 28, 2020

stale bot commented Jun 3, 2021

AndersonTorres commented Nov 2, 2024

jtojnar commented Nov 7, 2019 •

edited

Loading