Add builtins.fetchGitArchive (WIP)

[bergkvist]

Motivation

When working with private repos, it is desirable to interface with the users local git config to avoid checking secrets into the nix store. builtins.fetchGit currently allows for this, but can be slow for large repositories.

Thanks to git-archive - specific versions of these large private repositories can be checked out much more quickly (20s instead of 1h in my case)

There are some quirks though:

• Doesn't work with git submodules
• Not every git hosting provider supports it: Support git-archive protocol isaacs/github#554

Context

See #8134

Checklist for maintainers

Maintainers: tick if completed or explain if not relevant

• agreed on idea
• agreed on implementation strategy
• tests, as appropriate
  • functional tests - tests/**.sh
  • unit tests - src/*/tests
  • integration tests - tests/nixos/*
• documentation in the manual
• code and comments are self-explanatory
• commit message explains why the change was made
• new feature or incompatible change: updated release notes

Priorities

Add 👍 to pull requests you find important.

[bjornfor]

The 1h runtime, was that with shallow/depth=1?

[bergkvist]

The 1h runtime, was that with shallow/depth=1?

I just tested with shallow = true; now, and it took ~30 minutes (28.02s user 10.33s system 2% cpu 30:39.64 total).

time nix-instantiate -E --eval 'fetchGit { url="git@gitlab.example.com:private/repo"; ref = "refs/tags/v1.2.3"; shallow = true; }'

[edolstra]

This should probably be a libfetcher input type, or maybe just an attribute for the git fetcher (e.g. archive = true). That way these archives can be used as flake inputs.

[roberth]

How does this relate to the archive for example the archive endpoint semantics on github? Specifically, it distinguishes between commits and trees.

• commit hash: apply export-ignore
• tree hash: serialize a tree without much processing

By understanding the semantics, we can have a better cache utilization (fewer unique keys), and we could switch between the three methods transparently. (regular git protocol, archive over git protocol, archive over forge api)