Important
This file is automatically generated -- do not make edits to it. Instead, edit README_template.md and run nix run .#regen-readme > README.md to regenerate this file.
- Improve dependency resolution by being less strict with versions.
- Further documentation.
- Test cases.
This package provides a script which finds the "features" of the redistributable packages NVIDIA provides for CUDA. It does this by using redistrib_*.json manifests from https://developer.download.nvidia.com/compute/cuda/redist/ (and similar) and doing the following for each package:
-
Use
nix store prefetch-fileto download a package archive to the Nix store.- The manifest provides SHA256 hashes for each package, so we can verify the download.
- Additionally, providing the expected hash allows us to avoid re-downloading archives.
-
Use
nix flake prefetchon the store path of the archive to unpack it.- Though an abuse of the command, it does effectively serve as a way to unpack archives into the nix store.
- It also allows us to avoid unpacking archives multiple times by short-circuiting to the store path if it already exists.
-
Evaluate the contents of the unpacked archive to decide what "features" it provides.
- Implemented with heuristics. For example, if a directory contains a
libdirectory with alibfoo.afile, we assume that the package should have an output namedstaticcontaining all its static libraries.
- Implemented with heuristics. For example, if a directory contains a
-
Write a complementary JSON file containing the "features" each package should have next to the manifest passed as argument to the script.
These live in detectors.
cuda_architectures.py: Runscuobjdumpon the unpacked archive to find the CUDA architectures it supports.dynamic_library.py: Checks if the unpacked archive contains alibdirectory with dynamic libraries.executable.py: Checks if the unpacked archive contains executables inbin.header.py: Checks if the unpacked archive contains aincludedirectory with headers.needed_libs.py: Runspatchelf --print-neededon the unpacked archive to find the libraries it needs.provided_libs.py: Runspatchelf --print-sonameon the unpacked archive to find the libraries it provides.python_module.py: Checks if the unpacked archive contains asite-packagesdirectory with Python modules.static_library.py: Checks if the unpacked archive contains alibdirectory with static libraries.
The script is meant to be used as part of the process of updating the manifests or supported CUDA versions in Nixpkgs. It is not meant to be used directly by users.
There are two commands:
download-manifests: Download manifests from NVIDIA's website.process-manifests: Process manifests and write JSON files containing "features" each package should have.print-feature-schema: Print the JSON schema a "feature" manifest will have.print-manifest-schema: Print the JSON schema used to parse NVIDIA manifests.
$ nix run .# -- --help
Usage: cuda-redist-find-features [OPTIONS] COMMAND [ARGS]...
Options:
--help Show this message and exit.
Commands:
download-manifests
print-feature-schema
print-manifest-schema
process-manifests$ nix run .# -- download-manifests --help
Usage: cuda-redist-find-features download-manifests [OPTIONS] URL MANIFEST_DIR
Options:
--log-level [DEBUG|INFO|WARNING|ERROR|CRITICAL]
Set the logging level. [default: WARNING]
--no-parallel / --parallel Disable parallel processing. [default:
parallel]
--min-version VERSION Minimum version to accept. Exclusive with
--version.
--max-version VERSION Maximum version to accept. Exclusive with
--version.
--version VERSION Version to accept. If not specified,
operates on all versions. Exclusive with
--min-version and --max-version.
--help Show this message and exit.$ nix run .# -- process-manifests --help
Usage: cuda-redist-find-features process-manifests [OPTIONS] URL MANIFEST_DIR
Options:
--log-level [DEBUG|INFO|WARNING|ERROR|CRITICAL]
Set the logging level. [default: WARNING]
--cleanup / --no-cleanup Remove files after use. [default: no-
cleanup]
--no-parallel / --parallel Disable parallel processing. [default:
parallel]
--min-version VERSION Minimum version to accept. Exclusive with
--version.
--max-version VERSION Maximum version to accept. Exclusive with
--version.
--version VERSION Version to accept. If not specified,
operates on all versions. Exclusive with
--min-version and --max-version.
--help Show this message and exit.$ nix run .# -- print-feature-schema
{
"$defs": {
"CudaArch": {
"pattern": "^sm_\\d+[a-z]?$",
"type": "string"
},
"FeatureOutputs": {
"description": "Describes the different outputs a release can have.\n\nA number of these checks are taken from\nhttps://github.com/NixOS/nixpkgs/blob/d4d822f526f1f72a450da88bf35abe132181170f/pkgs/build-support/setup-hooks/multiple-outputs.sh.",
"properties": {
"bin": {
"title": "Bin",
"type": "boolean"
},
"dev": {
"title": "Dev",
"type": "boolean"
},
"doc": {
"title": "Doc",
"type": "boolean"
},
"lib": {
"title": "Lib",
"type": "boolean"
},
"static": {
"title": "Static",
"type": "boolean"
},
"sample": {
"title": "Sample",
"type": "boolean"
}
},
"required": [
"bin",
"dev",
"doc",
"lib",
"static",
"sample"
],
"title": "FeatureOutputs",
"type": "object"
},
"FeaturePackage": {
"description": "Describes the different features a package can have.\n\nA package is a release for a specific architecture.",
"properties": {
"outputs": {
"allOf": [
{
"$ref": "#/$defs/FeatureOutputs"
}
],
"description": "The Nix outputs of the package.",
"examples": [
"lib",
"dev",
"static"
]
},
"cudaArchitectures": {
"anyOf": [
{
"items": {
"$ref": "#/$defs/CudaArch"
},
"type": "array"
},
{
"additionalProperties": {
"items": {
"$ref": "#/$defs/CudaArch"
},
"type": "array"
},
"type": "object"
}
],
"description": "\n The CUDA architectures supported by the package.\n This is either a list of architectures or a mapping from subdirectory name to list of architectures.\n ",
"title": "Cudaarchitectures"
},
"providedLibs": {
"anyOf": [
{
"items": {
"$ref": "#/$defs/LibSoName"
},
"type": "array"
},
{
"additionalProperties": {
"items": {
"$ref": "#/$defs/LibSoName"
},
"type": "array"
},
"type": "object"
}
],
"description": "\n The libraries provided by the package.\n This is either a list of libraries or a mapping from subdirectory name to list of libraries.\n ",
"title": "Providedlibs"
},
"neededLibs": {
"anyOf": [
{
"items": {
"$ref": "#/$defs/LibSoName"
},
"type": "array"
},
{
"additionalProperties": {
"items": {
"$ref": "#/$defs/LibSoName"
},
"type": "array"
},
"type": "object"
}
],
"description": "\n The libraries needed by the package.\n This is either a list of libraries or a mapping from subdirectory name to list of libraries.\n ",
"title": "Neededlibs"
}
},
"required": [
"outputs",
"cudaArchitectures",
"providedLibs",
"neededLibs"
],
"title": "FeaturePackage",
"type": "object"
},
"FeatureRelease": {
"additionalProperties": {
"$ref": "#/$defs/FeaturePackage"
},
"description": "Represents a release in the manifest.\n\nA release is a collection of packages of the same library for different architectures.",
"title": "FeatureRelease",
"type": "object"
},
"LibSoName": {
"pattern": "\\.so(?:\\.\\d+)*$",
"type": "string"
}
},
"additionalProperties": {
"$ref": "#/$defs/FeatureRelease"
},
"description": "Represents the manifest file containing releases.",
"title": "FeatureManifest",
"type": "object"
}$ nix run .# -- print-manifest-schema
{
"$defs": {
"Md5": {
"pattern": "[0-9a-f]{32}",
"type": "string"
},
"NvidiaPackage": {
"description": "Represents a single package in the manifest.\n\nA package is a release for a specific architecture.",
"properties": {
"relative_path": {
"description": "The path to the package relative to the release.",
"format": "path",
"title": "Relative Path",
"type": "string"
},
"sha256": {
"allOf": [
{
"$ref": "#/$defs/Sha256"
}
],
"description": "The SHA256 hash of the package."
},
"md5": {
"allOf": [
{
"$ref": "#/$defs/Md5"
}
],
"description": "The MD5 hash of the package."
},
"size": {
"description": "The size of the package in bytes, as a string.",
"title": "Size",
"type": "string"
}
},
"required": [
"relative_path",
"sha256",
"md5",
"size"
],
"title": "NvidiaPackage",
"type": "object"
},
"NvidiaRelease": {
"additionalProperties": {
"$ref": "#/$defs/NvidiaPackage"
},
"description": "Represents a release in the manifest.\n\nA release is a collection of packages of the same library for different architectures.",
"properties": {
"name": {
"description": "The name of the release.",
"title": "Name",
"type": "string"
},
"license": {
"description": "The license of the release.",
"title": "License",
"type": "string"
},
"version": {
"description": "The version of the release.",
"title": "Version",
"type": "string"
},
"license_path": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "The path to the license file.",
"title": "License Path"
}
},
"required": [
"name",
"license",
"version"
],
"title": "NvidiaRelease",
"type": "object"
},
"Sha256": {
"pattern": "[0-9a-f]{64}",
"type": "string"
}
},
"additionalProperties": {
"$ref": "#/$defs/NvidiaRelease"
},
"description": "Represents the manifest file containing releases.",
"properties": {
"release_date": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "The date of the release.",
"title": "Release Date"
},
"release_label": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "The label of the release.",
"title": "Release Label"
},
"release_product": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "The product of the release.",
"title": "Release Product"
}
},
"title": "NvidiaManifest",
"type": "object"
}download-manifests
$ nix run .# -- download-manifests https://developer.download.nvidia.com/compute/cutensor/redist cutensor_manifests --log-level INFO
Set logging level to INFO.
Using URL https://developer.download.nvidia.com/compute/cutensor/redist.
Using dir cutensor_manifests
2023-10-31T03:43:32 INFO Manifest cutensor_manifests/redistrib_1.4.0.json already exists, overwriting... _manifest_ref.py:70
INFO Reading manifest from https://developer.download.nvidia.com/compute/cutensor/redist/redistrib_1.4.0.json... _manifest_ref.py:44
INFO Manifest cutensor_manifests/redistrib_1.6.1.json already exists, overwriting... _manifest_ref.py:70
INFO Reading manifest from https://developer.download.nvidia.com/compute/cutensor/redist/redistrib_1.6.1.json... _manifest_ref.py:44
INFO Manifest cutensor_manifests/redistrib_1.6.2.json already exists, overwriting... _manifest_ref.py:70
INFO Reading manifest from https://developer.download.nvidia.com/compute/cutensor/redist/redistrib_1.6.2.json... _manifest_ref.py:44
INFO Manifest cutensor_manifests/redistrib_1.3.2.json already exists, overwriting... _manifest_ref.py:70
INFO Manifest cutensor_manifests/redistrib_1.6.0.json already exists, overwriting... _manifest_ref.py:70
INFO Manifest cutensor_manifests/redistrib_1.7.0.json already exists, overwriting... _manifest_ref.py:70
INFO Manifest cutensor_manifests/redistrib_1.3.3.json already exists, overwriting... _manifest_ref.py:70
INFO Manifest cutensor_manifests/redistrib_1.5.0.json already exists, overwriting... _manifest_ref.py:70
INFO Reading manifest from https://developer.download.nvidia.com/compute/cutensor/redist/redistrib_1.3.2.json... _manifest_ref.py:44
INFO Reading manifest from https://developer.download.nvidia.com/compute/cutensor/redist/redistrib_1.6.0.json... _manifest_ref.py:44
INFO Reading manifest from https://developer.download.nvidia.com/compute/cutensor/redist/redistrib_1.7.0.json... _manifest_ref.py:44
INFO Reading manifest from https://developer.download.nvidia.com/compute/cutensor/redist/redistrib_1.3.3.json... _manifest_ref.py:44
INFO Reading manifest from https://developer.download.nvidia.com/compute/cutensor/redist/redistrib_1.5.0.json... _manifest_ref.py:44
INFO Wrote manifest to cutensor_manifests/redistrib_1.5.0.json. _manifest_ref.py:73
INFO Wrote manifest to cutensor_manifests/redistrib_1.7.0.json. _manifest_ref.py:73
INFO Wrote manifest to cutensor_manifests/redistrib_1.6.2.json. _manifest_ref.py:73
INFO Wrote manifest to cutensor_manifests/redistrib_1.4.0.json. _manifest_ref.py:73
INFO Wrote manifest to cutensor_manifests/redistrib_1.3.2.json. _manifest_ref.py:73
INFO Wrote manifest to cutensor_manifests/redistrib_1.6.1.json. _manifest_ref.py:73
INFO Wrote manifest to cutensor_manifests/redistrib_1.6.0.json. _manifest_ref.py:73
INFO Wrote manifest to cutensor_manifests/redistrib_1.3.3.json. _manifest_ref.py:73download-manifests with --version
$ nix run .# -- download-manifests https://developer.download.nvidia.com/compute/cutensor/redist cutensor_manifests --log-level INFO --version 1.4.0
Set logging level to INFO.
Version set to 1.4.0.
Using URL https://developer.download.nvidia.com/compute/cutensor/redist.
Using dir cutensor_manifests
2023-10-31T03:43:33 INFO Manifest cutensor_manifests/redistrib_1.4.0.json already exists, overwriting... _manifest_ref.py:70
INFO Reading manifest from https://developer.download.nvidia.com/compute/cutensor/redist/redistrib_1.4.0.json... _manifest_ref.py:44
INFO Wrote manifest to cutensor_manifests/redistrib_1.4.0.json. _manifest_ref.py:73download-manifests with --min-version and --max-version
$ nix run .# -- download-manifests https://developer.download.nvidia.com/compute/cutensor/redist cutensor_manifests --log-level INFO --min-version 1.4.0 --max-version 1.6.2
Set logging level to INFO.
Minimum version set to 1.4.0.
Maximum version set to 1.6.2.
Using URL https://developer.download.nvidia.com/compute/cutensor/redist.
Using dir cutensor_manifests
2023-10-31T03:43:35 INFO Manifest cutensor_manifests/redistrib_1.4.0.json already exists, overwriting... _manifest_ref.py:70
INFO Manifest cutensor_manifests/redistrib_1.5.0.json already exists, overwriting... _manifest_ref.py:70
INFO Reading manifest from https://developer.download.nvidia.com/compute/cutensor/redist/redistrib_1.4.0.json... _manifest_ref.py:44
INFO Manifest cutensor_manifests/redistrib_1.6.1.json already exists, overwriting... _manifest_ref.py:70
INFO Manifest cutensor_manifests/redistrib_1.6.0.json already exists, overwriting... _manifest_ref.py:70
INFO Reading manifest from https://developer.download.nvidia.com/compute/cutensor/redist/redistrib_1.5.0.json... _manifest_ref.py:44
INFO Manifest cutensor_manifests/redistrib_1.6.2.json already exists, overwriting... _manifest_ref.py:70
INFO Reading manifest from https://developer.download.nvidia.com/compute/cutensor/redist/redistrib_1.6.1.json... _manifest_ref.py:44
INFO Reading manifest from https://developer.download.nvidia.com/compute/cutensor/redist/redistrib_1.6.0.json... _manifest_ref.py:44
INFO Reading manifest from https://developer.download.nvidia.com/compute/cutensor/redist/redistrib_1.6.2.json... _manifest_ref.py:44
INFO Wrote manifest to cutensor_manifests/redistrib_1.4.0.json. _manifest_ref.py:73
INFO Wrote manifest to cutensor_manifests/redistrib_1.6.1.json. _manifest_ref.py:73
INFO Wrote manifest to cutensor_manifests/redistrib_1.5.0.json. _manifest_ref.py:73
INFO Wrote manifest to cutensor_manifests/redistrib_1.6.2.json. _manifest_ref.py:73
INFO Wrote manifest to cutensor_manifests/redistrib_1.6.0.json. _manifest_ref.py:73process-manifests
Assuming
nix run .# -- download-manifests https://developer.download.nvidia.com/compute/cutensor/redist cutensor_manifests --log-level INFO --min-version 1.4.0 --max-version 1.6.2was run previously,
$ nix run .# -- process-manifests https://developer.download.nvidia.com/compute/cutensor/redist cutensor_manifests --log-level INFO
Set logging level to INFO.
Using URL https://developer.download.nvidia.com/compute/cutensor/redist.
Using dir cutensor_manifests
2023-10-31T03:43:36 INFO Reading manifest from cutensor_manifests/redistrib_1.3.2.json... _manifest_ref.py:44
INFO Manifest version: 1.3.2 _manifest_ref.py:57
INFO Manifest date: 2021-09-22 _manifest_ref.py:58
INFO Manifest label: unknown _manifest_ref.py:59
INFO Manifest product: unknown _manifest_ref.py:60
INFO Reading manifest from cutensor_manifests/redistrib_1.3.3.json... _manifest_ref.py:44
INFO Manifest version: 1.3.3 _manifest_ref.py:57
INFO Manifest date: 2021-09-22 _manifest_ref.py:58
INFO Manifest label: unknown _manifest_ref.py:59
INFO Manifest product: unknown _manifest_ref.py:60
INFO Reading manifest from cutensor_manifests/redistrib_1.6.2.json... _manifest_ref.py:44
INFO Manifest version: 1.6.2 _manifest_ref.py:57
INFO Manifest date: 2022-12-12 _manifest_ref.py:58
INFO Manifest label: unknown _manifest_ref.py:59
INFO Manifest product: unknown _manifest_ref.py:60
INFO Reading manifest from cutensor_manifests/redistrib_1.4.0.json... _manifest_ref.py:44
INFO Manifest version: 1.4.0 _manifest_ref.py:57
INFO Manifest date: 2021-11-19 _manifest_ref.py:58
INFO Manifest label: unknown _manifest_ref.py:59
INFO Manifest product: unknown _manifest_ref.py:60
INFO Reading manifest from cutensor_manifests/redistrib_1.6.0.json... _manifest_ref.py:44
INFO Manifest version: 1.6.0 _manifest_ref.py:57
INFO Manifest date: 2022-06-24 _manifest_ref.py:58
INFO Manifest label: unknown _manifest_ref.py:59
INFO Manifest product: unknown _manifest_ref.py:60
INFO Reading manifest from cutensor_manifests/redistrib_1.5.0.json... _manifest_ref.py:44
INFO Manifest version: 1.5.0 _manifest_ref.py:57
INFO Manifest date: 2022-03-08 _manifest_ref.py:58
INFO Manifest label: unknown _manifest_ref.py:59
INFO Manifest product: unknown _manifest_ref.py:60
INFO Reading manifest from cutensor_manifests/redistrib_1.6.1.json... _manifest_ref.py:44
INFO Manifest version: 1.6.1 _manifest_ref.py:57
INFO Manifest date: 2022-10-05 _manifest_ref.py:58
INFO Manifest label: unknown _manifest_ref.py:59
INFO Manifest product: unknown _manifest_ref.py:60
INFO Reading manifest from cutensor_manifests/redistrib_1.7.0.json... _manifest_ref.py:44
INFO Manifest version: 1.7.0 _manifest_ref.py:57
INFO Manifest date: 2023-03-16 _manifest_ref.py:58
INFO Manifest label: unknown _manifest_ref.py:59
INFO Manifest product: unknown _manifest_ref.py:60