A proposal on treating git refs as versions

[becker33]

1. Motivation

Many users pin packages to particular git commits, tags, or branches, rather than released versions. This is particularly relevant to developers, who often track several packages through their release processes simultaneously, with versions constraints that propagate through their entire DAG. Therefore, Spack should allow users to seamlessly treat git refs, and other Version Control System (VCS) references, as versions in a Spack spec, while maintaining appropriate comparisons between those versions and the versions, version ranges, and version lists used as control-flow in the Spack package.

2. Approach

Fetching git versions is a fundamentally easy problem -- the only difficulty there is determining to use git for packages that have both git and url download options. The difficulty is how to build from versions that are not directly comparable to the (usually semver) version constraints in the build recipe.

2.A Comparing Versions

For version comparisons, we need to compare a git commit to a semver version. Assuming the release versions are tagged in git, it is feasible to compare them to a git version. To compare semver S to git version G, first we compute the merge-base of S (as tagged in git) with the default branch, MS, and the merge-base of G with the default branch, MG. If MG is an ancestor of MS, G is "less than" S, and vice-versa. If MG and MS are equal, then G is "less than" S if it is an ancestor of S, and vice-versa. If MG = MS and G and S are not directly comparable, then G and S are not comparable versions, and neither is "less than" the other, but they are not equal.

If the release versions are not tagged in git, there is no automatic way to compare them to git versions. However, the package file may contain a translation from release versions to git commits, which would allow the prior algorithm to proceed for such a package.

2.B When to resolve comparisons

There are two primary options for when to resolve comparisons between git versions and release versions.

2.B.i Compare as necessary

The easiest way to check comparisons between git versions and release versions would be to compute the comparison as necessary. This involves the least complex caching (although we may cache individual comparisons with this method).

2.B.ii Compare once, use range

Initial planning for this effort focused on the possibility of computing a "range" for each git version of the closest release
versions that could be ascertained to be "less than" and "greater than" the git commit. While this method initially seemed to create fewer comparisons (and more obvious caching) than the method to compare as necessary, caching individual comparisons in the ad-hoc method should result in similar performance with less complexity of design.

2.C When to clone

While git versioning will be most useful to developers, it is also of use to large-scale coordination projects like the XSDK. For that reason, we will avoid relying on developer packages (specified by the develop key in a Spack environment) for version comparisons.

However, cloning for each comparison would be prohibitive from a performance perspective, in addition to being problematic on systems without internet access.

To resolve this, we propose to modify the structure of Spack mirrors to better support VCS softare. For each type
of VCS understood by Spack (currently git, hg, and svn) we propose a top-level directory under the Spack mirror which contains a single instance of the VCS repo per-package. The builtin Spack mirror (the "source cache") will attempt to pull this repo any time it encounters a git ref it cannot find, as well as any time it checks out a branch. User-made mirrors will be static, and will throw errors on git refs they cannot find, as will the builtin mirror when fetching does not resolve the issue. Spack will attempt to pull from the internet when such an error is raised, but fail gracefully if it cannot be reached.

3. Project Breakdown

We propose 3 separate pull requests to Spack

3.A Mirror VCS software separately

This PR will establish the mirror subdirectories for each VCS system, and manage when we pull/fail for unfound git refs for each type of mirror. It will leave for future work the details for non-git VCS systems, while reserving their subdirectory names for that future use.

3.B Automatic interpolation of git versions by FetchStrategy

This PR will allow Spack to (a) determine whether an unknown version is a git reference or an unknown release version, and (b) fetch a package based on a git reference when it is not known to the package file, even when the type of the reference (branch, tag, commit) is unknown. Other VCS systems will be left for future work.

3.C Automatic comparison of git versions to release versions

Implement the algorithm described in section 2.A, along with appropriate caching of results for performance.

3.D Future PRs implied by this work, but not included

3.D.i Mirror HG software separately

3.D.ii Mirror SVN software separately

3.D.iii Automatic interpolation of HG versions

3.D.iv Automatic interpolation of SVN versions

3.D.v Automatic comparison of HG versions to release versions

This will require independent research into the appropriate comparison algorithm

3.D.vi Automatic comparison of SVN versions to release versions

This will require independent research into the appropriate comparison algorithm

[becker33]

@tgamblin This is the proposal for the work I want to lock myself in a room and get a jump on next week when there won't be any meetings

@rblake-llnl I think you will find this interesting.

[adamjstewart]

Should this be a discussion instead of an issue?

[becker33]

@adamjstewart I'm not sure. Do we want to push all feature requests to discussions? If so then definitely yes.

[tgamblin]

Should we have a separate repo for proposals like this? e.g., like k8s KEPs: https://github.com/kubernetes/enhancements.

It might be useful to add this written step to the design process for larger features, not just for discussions but to give people a chance to see the full idea. I think Spack's getting large enough that the writing will help -- there are a lot of interconnected issues for many of the features.

[cosmicexplorer]

Multiple Repos: Unsure?

@tgamblin:

Should we have a separate repo for proposals like this? e.g., like k8s KEPs: https://github.com/kubernetes/enhancements.

My Response:

• A separate repo seems like a workaround (maybe), and worries me personally that people are going to have issues with managing notifications from that separate repository (where notifications are pretty important).
• While k8s and rust have a large number of bots (see above) helping them, we have fewer such automation for scaffolding new repos or checking PRs.
  • Finally, I'm not sure yet what a separate repo solves for them or us, though.

A Flexible and Canonical Design Process: Possible Improvements!

In general, it seems like there's some consensus on some form of "Spack Enhancement Proposals"!! I don't want to block that! But some questions remain about the specifics:

• What are the rules for creating those?
• What timeline do we want to have this process by?
• What are we trying to get out of such a proposal?
• Does this induce additional burden?

That's what the rest of this comment tries to help expand!!

@tgamblin:

It might be useful to add this written step to the design process for larger features, not just for discussions but to give people a chance to see the full idea. I think Spack's getting large enough that the writing will help -- there are a lot of interconnected issues for many of the features.

Rust Strategies

I have wanted to mention this really interesting retrospective on two separate processes the rust team tried to adopt in 2020 (https://smallcultfollowing.com/babysteps/blog/2020/12/18/looking-back-on-2020/#the-major-change-process-helped-compiler-team-spend-more-time-on-design). The second one was less successful and seems to refer to a pretty large number of very active participants, so it is only included here for contrast, but I believe the first process was a good idea to consider. I have quoted each below:

1. Major Change Process: Specific Strategies

This one was a success!

One of the things I am most happy with is the compiler team’s Major Change Process. For those not familiar with it, the idea is simple: if you would like to make a Major Change to the compiler (defined loosely as “something that would change documentation in the rustc-dev-guide”), then you first open an issue (called a Major Change Proposal, or MCP) on the compiler-team repository. In that issue, you describe roughly the idea. This also automatically opens a Zulip thread in #t-compiler/major changes for discussion. If somebody on the compiler team likes the idea, they “second” the proposal. This automatically starts off a Final Comment Period of 10 days. At the end of that, the MCP is approved.

I thought this process was notable by itself in that a team identified this specific process being so specifically effective in addressing this (imo) very difficult problem, especially at scale. But I would classify it maybe as a source of useful specific strategies that we can consider adopting later -- the rest of the apparatus may be totally inappropriate. Just thought this was a surprisingly positive response to hear from anyone about this topic!

2. Project Proposals: Failure Mode to Avoid

This one appeared to be less of a success, and the discussion around it highlighted a few failure modes I think we should look out for:

I would say that the “project” system has been a mixed success. We’ve had a number of successful project groups, but we’ve also had some that are slow to start. We’ve not done a great job of recruiting fresh liaisons and I think the role could use more definition.

I think the second half of this quote is a really well-stated description of a failure mode to avoid:

Finally, we need to have much clearer messaging, and a more finalized “decision” around the RFC process – I’m also concerned if the RFC process starts to diverge too much between teams. I think it’s quite confusing for people right now to know how they’re supposed to “pitch” an idea (and people are often unclear which team is the best fit for an idea).

Focusing on avoiding this failure mode would be one way to "check our work" on the process I've quoted you on beneath.

Analysis

Let's get back to @tgamblin's quote to recall the problem landscape:

It might be useful to add this written step to the design process for larger features, not just for discussions but to give people a chance to see the full idea. I think Spack's getting large enough that the writing will help -- there are a lot of interconnected issues for many of the features.

My Response:
To confirm: interconnected sounds like an implication that we can reasonably expect to keep this writing up-to-date? Because that would be awesome!!!

Here are a few suggestions I have which may help us towards that goal:

1. Declare file X as "ground truth"!

Declare file X as "ground truth describing any form of design process we may institute".
- Since the website is sourced from files in the repo, I nominate the Contribution Guide.

2. Put a flowchart at the top of file X!

At the very top of file X, describe either a literal or figurative flowchart for contributors. The intent of the flowchart would be very similar to our current issue templates, focusing on triage and documentability.

This is one flowchart I made just now representing the current state of file X:
- Unclear Fix => ???
- Clear Fix => Make a fix to spack using all of the instructions on this page.
- Large Change => ???

Contribution Guide Mismatches

There are a couple things I noticed in the contribution guide that made me want to prioritize the "flowchart":
(a) In particular, I wanted to point out that it made me laugh when I found the only existing docs on "writing a big PR" in all of Spack are at the end of a long section on git cherry-pick:

Note:
It is important that whenever you change something that might be of importance upstream, create a pull request as soon as possible. Do not wait for weeks/months to do this, because:
- you might forget why you modified certain files
- it could get difficult to isolate this change into a stand-alone clean PR.

It seems more efficient to be able to address that situation before our users are rebasing constantly!

(b) And more directly, we have all of the content we would need for the "Clear Fix" section of the flowchart -- this is already at the top of the Contribution Guide

Important is completed feature. The changes one proposes in a PR should correspond to one feature/bugfix/extension/etc. One can create PRs with changes relevant to different ideas, however reviewing such PRs becomes tedious and error prone. If possible, try to follow the one-PR-one-package/feature rule.

• (a) might be rewritten as a failure mode to avoid from the start for the "Large Change" section of the flowchart.
• (b) could replace the "Clear Fix" section of the flowchart entirely.

3. Propose workflow for the "Unclear Fix"

I propose a workflow specifically for the "Unclear Fix" section of the flowchart from (2).

It is not always clear whether it's preferred to ping the slack Y times vs create an issues. In line with the explicit flowchart described in (2), I would like the section for our customer segment "Unclear Fix" to be something like:

1. Search for a similar issue.
2. Check Slack, then
3. Match labels.

, while modifying the priority of those tasks as seems fit.

[cosmicexplorer]

So the above^ was a little of:

• we need some doc fixes. (made into update contribution guide with a flowchart for triage #20567)
• Do we need to have a discussion about how to have a discussion (i.e. is there consensus on how to structure submissions for a Spack Enhancement Proposal?)?
  • If not, I would strongly recommend we read the Major Change Proposal text from earlier and decide whether we can copy it.
    • Google docs often help with this.

[cosmicexplorer]

I have reached out to the spack maintainers group to determine our next steps on the Spack Enhancement Proposal.

[adamjstewart]

Personally I think issues are fine for minor feature requests, but big changes that require discussion and planning might be better as discussions. I think a formal system for SEPs (Spack Enhancement Proposals) sounds like overkill.

Anyway, sorry for derailing this conversation. Feel free to hide these comments as off-topic once we make a decision.

[cosmicexplorer]

I didn't realize you had in mind the entire PEP process for some process. I actually 100% agree with you on this point -- python PEPs are a great example of open source working in general, but are much more of a goal to aspire to for other projects, imho.

I gave another example of something we might adopt from a programming language, but perhaps more immediately, here: #20536 (reply in thread). I believe the rust community's protocols are really nice, but as I mention in the comment above, I absolutely agree that Spack should try to avoid introducing multiple "sign-off" phases and more focusing on making sure proposal constantly make forward motion.

[becker33]

We could call them "Spack Upgrades" just to confuse anyone who has ever used another package manager...

[cosmicexplorer]

This is totally true. On pants, users would choose the term "pants upgrade" or "pants release" indiscriminately. Congruently, a "Spack Release" would then refer to either the process of fixing spack to make a new release, or the process of fixing a spack project to be compatible with a new release.

[becker33]

Converting this to a discussion; I agree large feature requests should be discussions.

[cosmicexplorer]

1. The git ref / version congruency assumption is false, but I think it still works!

Spack should allow users to seamlessly treat git refs, and other Version Control System (VCS) references, as versions in a Spack spec, while maintaining appropriate comparisons between those versions and the versions, version ranges, and version lists used as control-flow in the Spack package.

It seems that clingo is already a counterexample to "we can make git into just a version" conjecture. In particular, its dependencies would differ (requiring bison and re2c).

However, I think that something like adding a variant +git would rescue this immediately!

To be horribly pedantic because this word is used a lot: I think interchangeably might be a more appropriate replacement than seamlessly. Could be wrong!

2. [nonblocking] Pedantry

Fetching git versions is a fundamentally easy problem -- the only difficulty there is determining to use git for packages that have both git and url download options. The difficulty is how to build from versions that are not directly comparable to the (usually semver) version constraints in the build recipe.

Was the intention here to say "Fetching git versions is a previously-solved problem"? The performance concerns described in 2C seem to contradict the "fundamentally easy" part, imho. I could be missing something here.

2.A Comparing Versions

Seems great here, but I might also recommend looking for prior art on "tracking git versions through other version systems". I'm not sure if svn -> git will have to perform some of this interspersing of versions, for example. But this section looks good.

2.B.ii Compare once, use range

imho keep this as a followup issue even if we can't do it for v1 of this feature!

While git versioning will be most useful to developers, it is also of use to large-scale coordination projects like the XSDK. For that reason, we will avoid relying on developer packages (specified by the develop key in a Spack environment) for version comparisons.

One possible way to avoid this type of edge case is to adopt a deterministic strategy like sbt's dynver plugin which records more than just git info (e.g. the string can be used to determine e.g. whether the working tree is dirty, and a timestamp).

3. Counterproposal to "Clone Management" via Fetching Tags Only

2.C When to clone
However, cloning for each comparison would be prohibitive from a performance perspective, in addition to being problematic on systems without internet access.

Why would cloning for each comparison be necessary? Do we need to have their source code at some point to infer the version? I thought that the list of tags and their mapping to git checksums was defining the version comparison process. Have we investigated methods to avoid performing a clone each time?

Proposed v1/v2 tag fetching

I believe there is a v1 and a v2 standard that git servers perform for "ref advertisement", which is a process we can latch onto specifically to avoid reading any other git data. That version is controlled by the git server administrator, but a v2 request can gracefully degrade into a v1 version. Most git servers open HTTP and GIT interfaces, as well as SSH for places like github.

• v2 (with ls-refs): https://www.git-scm.com/docs/protocol-v2

• v1 (monolithic): The git-fetch(1) on my osx git 2.20.1 has --tags, along with --negotiation-tip:

        --negotiation-tip=<commit|glob>
           By default, Git will report, to the server, commits reachable from
           all local refs to find common commits in an attempt to reduce the
           size of the to-be-received packfile. If specified, Git will only
           report commits reachable from the given tips. This is useful to
           speed up fetches when the user knows which local ref is likely to
           have commits in common with the upstream ref being fetched.
       -t, --tags
           Fetch all tags from the remote (i.e., fetch remote tags refs/tags/*
           into local tags with the same name), in addition to whatever else
           would otherwise be fetched. Using this option alone does not
           subject tags to pruning, even if --prune is used (though tags may
           be pruned anyway if they are also the destination of an explicit
           refspec; see --prune).

Example v1

> git init
> git fetch --tags https://github.com/spack/spack
> git tag
NCSA-v2
efischer/docs-v1
old-install-layout
releases/latest
v0.10.0
...

---

4. Sync Up Downstream

The reason I am familiar with git ref checkouts is same reason that led me to create #20359 and #20407 -- because I was extremely familiar with a git FUSE project which made checkouts instantaneous and removed the need for explicit push/pull by syncing new refs remotely as they were created. This article is also useful for describing the aspects of git that can contribute to performance issues (as in the precise sequence of internal changes, not just the individual commands): https://www.atlassian.com/git/tutorials/monorepos.

Completely separately from the above, I wanted to use this opportunity to introduce some situations those two PRs could be become useful in, and I think some of your proposal elements were ideal for this.

Many users pin packages to particular git commits, tags, or branches, rather than released versions. This is particularly relevant to developers, who often track several packages through their release processes simultaneously, with versions constraints that propagate through their entire DAG.

To resolve this, we propose to modify the structure of Spack mirrors to better support VCS softare. For each type
of VCS understood by Spack (currently git, hg, and svn) we propose a top-level directory under the Spack mirror which contains a single instance of the VCS repo per-package. The builtin Spack mirror (the "source cache") will attempt to pull this repo any time it encounters a git ref it cannot find, as well as any time it checks out a branch.
User-made mirrors will be static, and will throw errors on git refs they cannot find,
as will the builtin mirror when fetching does not resolve the issue.
Spack will attempt to pull from the internet when such an error is raised, but fail gracefully if it cannot be reached.

This seems like a lot of extra error-handling and user behavior we'd need to define (a whole new service protocol of sorts) in order to support a feature that works purely by comparing specs. It reminds me of how java URIs used to contact DNS.

5. #20407 Mirror requires no modification to support VCS, or any other feature.

• Rewrite of entire codebase not necessary.
• The main result of hermetic/remoteable/cacheable process executions #20407 is that a mirror's data model is reduced to a Merkle tree (or a forest).
  • Spack users don't need to upgrade mirrors after spack changes versions.
  • Spack users don't need to know anything about the mirror format or exchange mirror packfiles.
• The second result of this PR is that Merkle tree operations are backed by rust code and an LMDB database (with a bunch of build experts relying on it daily).
  • This async rust code can very effectively traverse a directory recursively while inserting entries into the database.
• The third result of this PR is that like git fetch's --negotiate-tip option, fs_util has very fast logic to recursively request merkle tree elements it doesn't have yet.
  • This makes syncing small subsets of very large and deeply nested filesystem trees computationally tractable.
  • Including synchronizing spack build results across machines immediately after builds complete!
• Also, all build steps could be performed on a remote cluster using a standardized API.
  • scoot is my favorite implementor, and also implements the API using fs_util as hermetic/remoteable/cacheable process executions #20407 describes.

However, if we used #20407, we would have a few more constraints:
On the Spack dev side:

• Would need to parse a git packfile or something similar to quickly extract corresponding CAS entries to put into our DB.
  • This is different than having to change the mirror format -- instead we just need an adapter when entering it.

On the Spack user side:

• A mirror backed by fs_util must be configured with:
  • 1 Local LMDB instance (auto-configured), by directory path. This requires special installation -- see pants install page.
  • Any number (including 0) of remote servers implementing the CAS API (this may be over a LAN).
• A user can digest any file or directory into a checksum with fs_util, ensure it syncs remotely, then ask another to retrieve it by the same checksum.

6. #20359 Avoid any Unnecessary I/O

• Files are never copied to the real filesystem, avoiding I/O performed by faulty caches.
• Unnecessary or unchanged output files from a build can be ignored and/or duplicated.
• Necessarily blocks rationale for process sandboxing on linux, then a proposal for a how to emulate sandboxing on all the other platforms #20260 due to performance concerns reconstructing an entire filesystem.
• This would have been especially helpful in the specific case where we still needed to pull off a full git checkout in order to compare revisions. In that case, it would have been possible to even perform operations in parallel.
  • This is the first place (with hermetic/remoteable/cacheable process executions #20407) I would look if we ever needed to suddenly run a lot more spack builds than usual at once.

[alalazo]

This PR will establish the mirror subdirectories for each VCS system, and manage when we pull/fail for unfound git refs for each type of mirror.

Should we be concerned about tags being removed or replaced in a git repo?

[cosmicexplorer]

I think personally the behavior I might want for that would be, whenever we re-fetch for tags from a repo, to:

• tag removed: retain the tag in spack
• tag replaced: change the tag in spack

I see the concern about propagating those behaviors to users, and I realize that it might be nice to turn these two tag refresh behaviors into yaml config options.

Question: would we also then want to add a spack config explain command which prints out help for config options? Or am I missing an existing method to do that?