Tracking Issue for -Zgitoxide

[Byron]

Summary

RFC: None
Documentation: https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#gitoxide
Issues: Z-gitoxide Nightly: gitoxide integration
Original issue:

• shallow clones Allow specifying shallow clones in .config #1171
• slow on file:// Cargo-update: exceedingly slow progress on update when local registry/index is used and git-fetch-with-cli is false #9167
• slow on fetch Resolving deltas takes long time  #11014
• high CPU usage High CPU usage when cargo updates its index #9215
• high memory usage Cargo eating all RAM during aarch64 build under Docker on x86-64 #10583
• publish failure due to git-ignored .DS_Store cargo-package: list of dirty files contains ignored files #8407

With the ‘gitoxide’ unstable feature, all of the specified git operations will be performed by the gitoxide crate instead of git2. This allows git2 to be replaced in full.

Steps

Unstable feature sub-categories:

• fetch: all fetches are done with gitxide (gitoxide integration: fetch  #11448)
• shallow_index: the crates index will be a shallow clone (support for shallow clones and fetches with gitoxide #11840)
• shallow_deps: git dependencies will be a shallow clone (support for shallow clones and fetches with gitoxide #11840)
• checkout: plain checkouts with gitoxide (including submodule handling)
• package: status checks, untracked files, excluded files

Further tasks

• short-hash handling can be done by gix_hash::Prefix.
• optimize dependencies to only include those that cargo actually uses.
• When specifying a dependency with rev but nothing found, cargo with shallow fetch should fetch full history as a fallback.

Unresolved Issues

• work with https://rsproxy.cn (the proxy seems to work different making naive negotiations fail due to multi-round negotiation)
• all features as listed in this gitoxide tracking issue to be able to replace git2 in full
• support for the file:// protocol without relying on the git binary
• support of the ssh:// protocol without relying on the ssh binary
• double-check and validate that authentication works similarly to git2 or if unfeasible, assure users have a migration path. How to deal with this correctly is still to be determined.
• transitive dependency tempfile is pinned to ~3.3.0 temporarily (see gix-tempfile@4.1.1)
• A-git Area: anything dealing with git might contain more potential blockers
  • take a look at all of these and either fix them, resolve them, or store them in gitoxide itself

Future Extensions

Some features not in libgit2 are good candidates for future extensions

• sparse checkout Cargo build crash in project with git Split Index #10150
• split index Add support for sparse git checkouts #11165
• git-lfs integration Git dependency with LFS objects #9692

About tracking issues

Tracking issues are used to record the overall progress of implementation.
They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions.
A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature.
Instead, open a dedicated issue for the specific matter and add the relevant feature gate label.

Implementation history

• gitoxide integration: fetch  #11448
• gitoxide progress bar fixes #11800
• Disable flaky auth tests when gitoxide runs them #11830
• [BLOCKED] upgrade gix to v0.41 to fix flaky auth tests. #11831
• support for shallow clones and fetches with gitoxide #11840

[Nemo157]

Is it possible to enable this through environment variables somehow? I wasn't able to figure out any syntax that didn't run into this error:

> CARGO_UNSTABLE_GITOXIDE=true cargo build
error: missing field `fetch`

(it may also be useful for this usecase to support an explicit CARGO_UNSTABLE_GITOXIDE=all feature alias that acts like -Zgitoxide)

[Byron]

I think I did that with the following, on stable:

export CARGO_REGISTRIES_CRATES_IO_PROTOCOL=git
export __CARGO_USE_GITOXIDE_INSTEAD_OF_GIT2=1 # nobody should be using this on stable, but it's there if truly needed.

CARGO_UNSTABLE_GITOXIDE=fetch cargo build should work, and so should CARGO_UNSTABLE_GITOXIDE=fetch,shallow cargo build for shallow clones. Please note that I wasn't aware of these environment variables before reading this, so I just assume whatever -Z gitoxide=<value> supports can be used as CARGO_UNSTABLE_GITOXIDE=<value>.

[Nemo157]

CARGO_UNSTABLE_GITOXIDE=fetch cargo build should work

It doesn't, it gives the same error message. The CARGO_UNSTABLE_* environment variables correspond to the [unstable] table in .cargo/config.toml not the -Z flag.

By actually trying to set it in .cargo/config.toml it gave some more useful error messages, and I managed to get it working with

[unstable.gitoxide]
fetch = true
shallow_index = true
shallow_deps = true
checkout = true
internal_use_git2 = false

and

> CARGO_UNSTABLE_GITOXIDE_FETCH=true CARGO_UNSTABLE_GITOXIDE_SHALLOW_INDEX=true CARGO_UNSTABLE_GITOXIDE_SHALLOW_DEPS=true CARGO_UNSTABLE_GITOXIDE_CHECKOUT=true CARGO_UNSTABLE_GITOXIDE_INTERNAL_USE_GIT2=false cargo build

I do think it would be good to override the toml parser for this variable too so that you don't have to specify every single field.

[ibraheemdev]

How is shallow cloning dependencies different than specifying the refspec (e.g. +HEAD:refs/remotes/origin/HEAD) that cargo already does? I think it only really helps if when fetching a specific branch or tag?

[Nemo157]

A refspec reduces the heads that you're requesting, so you don't get other unneeded branches being pulled down. But you still request the entire history from that head back to initial commit(s) of the repo. A shallow clone allows you to limit how much of the history you request, presumably for cargo's usecase it only needs to pull down the current commit of the requested head and can ignore the history entirely (though technically this is visible to build.rs scripts running from git dependencies).