[Upstream Issue] Libgit2 and transitively Stacked Git do not support sparse checkouts #195

NonLogicalDev · 2022-07-29T22:00:19Z

Upstream Issues:

Adding this issue to track upstream issues of LibGit2 related to sparse checkout support.

Our company is using a huge monorepo, and we have recently onboarded onto Git Sparse checkouts, which seem to be unsupported in LibGit2 which Rust version is using.

Despite my attempts at doing custom builds with some PRs I found online for LibGit2, although I could get the Read Only workflow to succeed like stg series.

I could not get the stg push to work as it seems like sparse index is throwing LibGit2 way off.

The text was updated successfully, but these errors were encountered:

jpgrayson · 2022-08-15T17:33:34Z

Thanks for writing up this issue, @NonLogicalDev.

It would be nice if libgit2 supported sparse checkout natively. That said, it doesn't seem like that project has the will to take on the risk of sparse checkout any time soon.

When reimplementing StGit in Rust, a key design decision was how to interact with the git repository. Initially, I tried orienting almost entirely to libgit2--it is fast and its abstractions are pretty nice. However, as I implemented more and more of StGit, I ran into a bunch of compatibility problems; tests would mostly pass, but various edge cases would fail. Patch application and merge resolution (in various contexts) just don't work correctly.

So where libgit2 didn't get the job done, I reverted to using git commands.

Coming back to sparse checkouts, it might be that there is yet another subset of operations that StGit is currently doing with libgit2 that could be converted to git commands that would enable StGit to just work with sparse checkouts. I think this is a more likely path to success versus waiting for libgit2 to gain sparse checkout support.

Perhaps the line we can draw for libgit2's use in StGit is to only use it as an object and configuration database. I.e. looking up refs, commits, trees, blobs, and config, but any significant mutations of repository or working tree state should be done by git proper. At one point, I had done the work to entirely eliminate libgit2 as a dependency (I wrote an entire replacement stupid2 library), but the runtime performance was significantly worse than the current hybrid approach.

I can think of a couple of concrete next steps for this issue:

Define some sparse checkout test cases that we want to work, covering at least new, refresh, push, and pop.
Audit for the remaining places where StGit uses libgit2 for checkout and tree comparisons.

arxanas · 2022-08-16T17:14:15Z

If you do not use worktrees for your sparse checkout, the documentation implies that it should be fine to delete that configuration option (git config --unset extensions.worktreeConfig). See https://git-scm.com/docs/git-worktree#_configuration_file

A coworker of mine has a fork of Rust's git2 which implements (ignores? not sure) the extensions.worktreeConfig option: https://github.com/bierbaum/git2-rs

jpgrayson · 2022-08-16T20:05:16Z

@arxanas, you seem to be pointing out a detail related to how the sparse checkouts feature intersects with the multiple worktrees feature, which is interesting and potentially useful toward the intersection of those two features in StGit, but does not provide a solution path for StGit support of sparse checkouts. I just want to make sure I'm not missing something here.

arxanas · 2022-08-16T20:34:31Z

@jpgrayson For the end user, you can restore StGit support for sparse checkouts, assuming you don't also use multiple worktrees for that same checkout, by deleting the problematic extensions.worktreeConfig configuration key. If you don't happen to use both features together, then you can get StGit to work on your work monorepo's sparse checkout again.

Or you as the developer can update StGit to use the forked git2, which would at least let you ignore the unsupported extensions.worktreeConfig option. Then you can move forward with writing the tests and auditing the checkout/diff sites.

jpgrayson · 2022-08-16T20:51:38Z

My assumption is that StGit has plethora of problems with sparse checkouts independent of its intersection with multiple worktrees.

I had performed a quick test scenario where I enabled sparse checkouts on a small subset of a linux kernel repo using a vanilla/default worktree. The patch resulting from stg new --refresh contained deletions for all the thousands of files that were outside my sparse checkout cones. This is an obvious indicator that StGit does not work properly on a sparse checkout, even without using multiple worktrees.

Performing that same scenario again, but with extensions.worktreeConfig unset yields the same problem. AFAICT, StGit cannot do even the most basic operations in the context of a sparse checkout, regardless of whether multiple worktrees is in use.

From my point of view, there are mostly orthogonal paths toward solving sparse checkouts with StGit and multiple worktrees with StGit, and solving the major sparse checkouts issues does not require using a libgit2 patched to comprehend per-worktree configs. I still feel like I may be missing a point you're trying to make, @arxanas, so apologies if I'm not understanding your point.

NonLogicalDev · 2022-08-16T22:15:03Z

@arxanas I had the same experience as @jpgrayson when testing on my repo, I tried both building the PR that ignored per Worktree Config extension, and just un-setting the WorkTree Config Extension enable switch. In read only the experience was butter smooth, like it ought to be, the problems start when you try to interact with the actual index. I.e. refresh commands.

At that point LibGit really goes off the rails, as it thinks that all of the excluded files from the sparse index are deleted. I wish the fix was that simple. I almost believed myself that it was that simple, but then the reality rudely hit me in the face. =]

It is a difficult tradeoff to make, either support partial functionality (i.e. do not support the sparse checkouts) and keep the blazing speed of LibGit.
Or take the hit of doing execs...
Or shudders have a hybrid approach that can switch on the fly based on some signals, like if the sparse checkout is configured. (but that nearly double the work to support 1.5 codebases instead of just 1)

I really like the rust version, it is so darn fast! Big props @jpgrayson. I just wish I could use it for work haha.

arxanas · 2022-08-17T02:47:59Z

Sorry, I misunderstood the scope of this issue to be about crashing only. My own project git-branchless more or less worked after deleting extensions.worktreeConfig, so I was expecting the same here.

I use the hybrid approach where libgit2 is mostly an object database, and I shell out to git for checkout/status/commit/etc. operations. (There are other reasons to do this too. For example, libgit2 doesn't run Git hooks, so its implementation of status won't make use of the fsmonitor, which is a significant performance hit on large repos.)

Feel free to copy any code from there which you find useful. Maybe of note:

and solving the major sparse checkouts issues does not require using a libgit2 patched to comprehend per-worktree configs

Well, I think you will still need to deal with the fact that git sparse-checkout set automatically sets the problematic extensions.worktreeConfig config variable, which will cause your calls to git2 to failing, even if you only use it as an object database.

jpgrayson · 2022-08-17T16:01:25Z

Thanks for this additional context, @arxanas. And thanks for the links to git-branchless. I've started looking at some of the code. Looks like a very cool project and it's helpful to see your approaches to similar problems.

This relates to sparse checkout support in StGit (#195). Since libgit2 does not support git's sparse checkout feature, when getting worktree file status via the git2 API in a sparsely checked-out worktree will indicate that all known files outside the sparse checkout cone(s) are deleted. This is the first of several problems preventing `stg refresh` from working correctly in the context of a sparsely checked-out worktree. The next problem is that when building the index for the refresh, the git2 API is further tripped-up and "adds" deletion entries for all the out-of-cone files. By using `git` (via the stupid module) for status and index operations, `stg refresh` is made to work with sparse checkout. The stupid module is converted from a single file into a directory (stupid/mod.rs) in order to make room for some higher level abstractions for status and changed files. The new stupid::status module provides abstractions for handling the output of `git status --porcelain=v2`. Care is taken to minimize allocations and defer processing. Allocations are minimized by Statuses holding onto the status output buffer (the only allocation) and providing references into that buffer for its various methods. A similar approach is used in stupid::diff, which provides an abstraction over the modified file list coming from `git diff-tree --name-only`. To make `stg pop` work, worktree and index status checks are now performed using `git status` instead of libgit2. Eventually, all StGit commands will need to use this approach to avoid problems in sparse checkout worktrees.

This relates to sparse checkout support (#195). Conflict checks and index/worktree cleanness checks are converted to using Stupid::statuses() for all StGit commands. This eliminates a common class of problem that can prevent StGit from working in a sparse checkout worktree.

This affects transaction-time checkout and the various checkouts performed by `stg branch`. This further improves StGit support for sparse checkout worktrees (#195).

The new Stupid::with_temp_index() method replaces the Index::with_temp_index() and Index::with_temp_index_file() extension methods. This is motivated by sparse checkouts (#195) since git2::Index does not comprehend sparse checkouts.

This relates to sparse checkout support (#195). git2 does not comprehend sparse checkouts and thus using any index <=> worktree operations is likely to break sparse checkouts. Most important index operations have already been converted to using git commands, this gets all the remaining instances. N.B. in run_pre_commit_hook(), GIT_INDEX_FILE is no longer being supplied explicitly. In all of StGit's use cases either using the default index or inheriting GIT_INDEX_FILE from the calling environment should be okay.

The new t7000-sparse-checkout.sh test script exercises various StGit commands in the context of a sparse checkout worktree. Relates-to: #195

jpgrayson · 2022-08-24T23:21:56Z

I've done some work on this problem. See above commits. For all the use cases I've tried, StGit now seems to work in a sparse checkout worktree. I'd love to know if this works for your use cases, @NonLogicalDev. Maybe checkout the new t7000-sparse-checkout.sh tests as well.

Some additional details:

The strategy was to use git instead of libgit2 for more operations, namely:

Status checks
Checkouts
All index operations

These changes do seem to affect runtime performance, at least as measured by how long the test suite takes. Perhaps on the order of 10%. This seems like a small price to pay for correctness. Also, these changes shouldn't have much/any affect on the most latency-sensitive StiGit commands, such as stg status, stg id, stg top. It's the commands that mutate repository state that now have a bit more overhead. I'll also note that for regular/large repos (as opposed to the tiny repos used in the test suite), the overhead of running more subordinate git processes will be less relevant and the git versions of the various operations may very well be faster at-scale than what we were doing with libgit2.

NonLogicalDev · 2022-08-26T21:06:58Z

@jpgrayson ❤️ , let me get to testing this right away!

NonLogicalDev · 2022-08-26T21:15:58Z

I think I may have uncovered another issue:

> stg lg
error: Malformed version 4 meta: missing Previous

> stg version
Stacked Git 2.0.0-beta.2 (2f0a3980b)
Copyright (C) 2005-2022 StGit authors
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
SPDX-License-Identifier: GPL-2.0-only
git version 2.36.1

NonLogicalDev · 2022-08-26T21:33:34Z

Strange one:

> git cat-file -p refs/stacks/master:stack.json
{
  "version": 5,
  "prev": "78ca357f4edf94549b0e91146076ba0f6b305f3c",
  "head": "2f0a3980b9b10721cd1d6c1e32c078438b894dae",
  "applied": [],
  "unapplied": [
    "notes-copy",
    "silly",
    "update-index-error-fix",
    "feature-edit-patch-update"
  ],
  "hidden": [
    "branch-rename-tests"
  ],
  "patches": {
    "notes-copy": {
      "oid": "a3c04fdcb7135f6b7f74e5b8747d04e6c28acd1e"
    },
    "silly": {
      "oid": "7c04ba132f1b7083670a89bcc61060cebc5f80aa"
    },
    "update-index-error-fix": {
      "oid": "d7a2b12e477e3558736cd197552b6e85162b7d82"
    },
    "feature-edit-patch-update": {
      "oid": "7077272df01e0cb9dc4dde015b469b98b93f61fb"
    },
    "branch-rename-tests": {
      "oid": "1e19a8c6bf879986ee9bacd07eba72102467d39b"
    }
  }
}

NonLogicalDev · 2022-08-26T21:41:12Z

I will cut a separate issue with that, I think I just happened to have old master.stgit ref sticking around.

jpgrayson · 2022-08-26T21:41:44Z

The stack upgrade path (which is where the "Malformed version 4 meta" error originates) is triggered by the presence of the refs/heads/master.stgit ref. So although you seem to have version 5 metadata (i.e. because refs/stacks/master exists and contains stack.json, your master branch also seems to have version 4 metadata. It is unclear to me how a branch would end up in this state via normal StGit operations.

I'll note that stack metadata version 4 was introduced in StGit 1.0 and stack metadata version 5 was introduced in StGit 1.2.

Also, I believe this metadata issue is orthogonal to sparse checkout. You should be able to run StGit on a new branch to test sparse checkout. And you can probably recover from the metadata problem by deleting refs/heads/master.stgit.

NonLogicalDev · 2022-08-26T21:46:44Z

@jpgrayson I played around with new stg version on our monorepo with sparse checkouts. So far so good. push / pop seem to be working as expected.

But I had to disable git config --unset extensions.worktreeConfig to have it work otherwise I get our good old friend:

> stg series
error: unsupported extension name extensions.worktreeconfig; class=Repository (6)

I will need to do further testing if sparse checkout related commands still work with that setting un-set.

jpgrayson · 2022-08-26T23:56:01Z

Thanks for the note about extensions.worktreeConfig. I guess this is yet another corner of the libgit2 API that StGit should perhaps avoid.

Since 0.13, a few changes have been made to git2, including rust-lang/git2-rs#791 which should address the extensions.worktreeConfig issue in #195. The upgrade required fixing uses of [ConfigEntries][1] because it's no longer an iterator. [1]: https://docs.rs/git2/0.15.0/git2/struct.ConfigEntries.html However, it provides an API that lends itself well to `while let` to provide a for loop like experience, so this was an easy fix.

jpgrayson · 2022-11-01T19:11:22Z

AFAIK StGit is doing okay with sparse checkouts at this point. Closing this issue. We can reopen as necessary.

NonLogicalDev · 2022-11-15T23:45:43Z

The vanilla STG still does not work:

> stg series
error: unsupported extension name extensions.worktreeconfig; class=Repository (6)

I have created a PR to fix this with a workaround. It has been working without issues for us at my company for the past few months.

`libgit2` library by default fails any operation if it encounters any unrecognised extensions configured in a repo, as per Git documentation. ref: https://git-scm.com/docs/repository-version > If a version-1 repository specifies any extensions.* keys that the > running git has not implemented, the operation MUST NOT proceed. > Similarly, if the value of any known key is not understood by the > implementation, the operation MUST NOT proceed. If a repository is using a sparse checkout mode it force enables the `worktreeconfig` extensions which allows setting configuration in repo on a per worktree basis, to enable some of the sparse checkout functionality. Since `stg` is using `libgit2` library only as an interface to object database and for looking up alias lookup from git configuration (i.e. Read Only operation) and uses git cli for everything else, it is relatively safe to ignore per worktree git configuration. This patch adds `worktreeconfig` extension to the `libgit2` extension whitelist. CLOSES stacked-git#195

`libgit2` library by default fails any operation if it encounters any unrecognised extensions configured in a repo, as per Git documentation. ref: https://git-scm.com/docs/repository-version > If a version-1 repository specifies any extensions.* keys that the > running git has not implemented, the operation MUST NOT proceed. > Similarly, if the value of any known key is not understood by the > implementation, the operation MUST NOT proceed. If a repository is using a sparse checkout mode it force enables the `worktreeconfig` extensions which allows setting configuration in repo on a per worktree basis, to enable some of the sparse checkout functionality. Since `stg` is using `libgit2` library only as an interface to object database and for looking up alias lookup from git configuration (i.e. Read Only operation) and uses git cli for everything else, it is relatively safe to ignore per worktree git configuration. This patch adds `worktreeconfig` extension to the `libgit2` extension whitelist. CLOSES stacked-git#195 Signed-off-by: Oleg Utkin <oleg@nonlogical.io>

`libgit2` library by default fails any operation if it encounters any unrecognised extensions configured in a repo, as per Git documentation. ref: https://git-scm.com/docs/repository-version > If a version-1 repository specifies any extensions.* keys that the > running git has not implemented, the operation MUST NOT proceed. > Similarly, if the value of any known key is not understood by the > implementation, the operation MUST NOT proceed. If a repository is using a sparse checkout mode it force enables the `worktreeconfig` extensions which allows setting configuration in repo on a per worktree basis, to enable some of the sparse checkout functionality. Since `stg` is using `libgit2` library only as an interface to object database and for looking up alias lookup from git configuration (i.e. Read Only operation) and uses git cli for everything else, it is relatively safe to ignore per worktree git configuration. This patch adds `worktreeconfig` extension to the `libgit2` extension whitelist. CLOSES #195 Signed-off-by: Oleg Utkin <oleg@nonlogical.io>

When core.repositoryformatversion=1 and extensions.worktreeconfig=true, libgit2 will fail unless StGit initializes libgit2 to allow the worktreeconfig extension. This test exersizes StGit in this scenario. Refs: #195, #234

jpgrayson added a commit that referenced this issue Aug 24, 2022

Replace checkout_tree_ex() with stupid checkout

0cb47b3

This affects transaction-time checkout and the various checkouts performed by `stg branch`. This further improves StGit support for sparse checkout worktrees (#195).

jpgrayson added a commit that referenced this issue Aug 24, 2022

Test cases for sparse checkout

af04cf8

The new t7000-sparse-checkout.sh test script exercises various StGit commands in the context of a sparse checkout worktree. Relates-to: #195

abhinav mentioned this issue Sep 9, 2022

Upgrade to git2 0.15 #206

Merged

jpgrayson closed this as completed Nov 1, 2022

NonLogicalDev mentioned this issue Nov 15, 2022

Enable basic support for extension.worktreeconfig with core.repositoryformatversion=1 to unblock sparse checkout in partial clones. #234

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Upstream Issue] Libgit2 and transitively Stacked Git do not support sparse checkouts #195

[Upstream Issue] Libgit2 and transitively Stacked Git do not support sparse checkouts #195

NonLogicalDev commented Jul 29, 2022 •

edited

Loading

jpgrayson commented Aug 15, 2022

arxanas commented Aug 16, 2022

jpgrayson commented Aug 16, 2022

arxanas commented Aug 16, 2022

jpgrayson commented Aug 16, 2022

NonLogicalDev commented Aug 16, 2022 •

edited

Loading

arxanas commented Aug 17, 2022

jpgrayson commented Aug 17, 2022

jpgrayson commented Aug 24, 2022

NonLogicalDev commented Aug 26, 2022

NonLogicalDev commented Aug 26, 2022

NonLogicalDev commented Aug 26, 2022

NonLogicalDev commented Aug 26, 2022

jpgrayson commented Aug 26, 2022

NonLogicalDev commented Aug 26, 2022

jpgrayson commented Aug 26, 2022

jpgrayson commented Nov 1, 2022

NonLogicalDev commented Nov 15, 2022

[Upstream Issue] Libgit2 and transitively Stacked Git do not support sparse checkouts #195

[Upstream Issue] Libgit2 and transitively Stacked Git do not support sparse checkouts #195

Comments

NonLogicalDev commented Jul 29, 2022 • edited Loading

jpgrayson commented Aug 15, 2022

arxanas commented Aug 16, 2022

jpgrayson commented Aug 16, 2022

arxanas commented Aug 16, 2022

jpgrayson commented Aug 16, 2022

NonLogicalDev commented Aug 16, 2022 • edited Loading

arxanas commented Aug 17, 2022

jpgrayson commented Aug 17, 2022

jpgrayson commented Aug 24, 2022

NonLogicalDev commented Aug 26, 2022

NonLogicalDev commented Aug 26, 2022

NonLogicalDev commented Aug 26, 2022

NonLogicalDev commented Aug 26, 2022

jpgrayson commented Aug 26, 2022

NonLogicalDev commented Aug 26, 2022

jpgrayson commented Aug 26, 2022

jpgrayson commented Nov 1, 2022

NonLogicalDev commented Nov 15, 2022

NonLogicalDev commented Jul 29, 2022 •

edited

Loading

NonLogicalDev commented Aug 16, 2022 •

edited

Loading