Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not recommend shallow_since for git_repository #17356

Closed
wants to merge 1 commit into from

Conversation

fmeum
Copy link
Collaborator

@fmeum fmeum commented Jan 30, 2023

Git's --shallow-since feature can lead to broken checkouts and a fix isn't likely to happen:
https://public-inbox.org/git/CAPSG9dZV2EPpVKkOMcjv5z+NF7rUu=V-ZkZNx47rCv122HsiKg@mail.gmail.com/T/#u

This commit removes shallow_since from the canonical reproducible form of git_repository unless the attribute is provided explicitly. In cases where Bazel's attempt at fetching with --depth=1 fail, e.g. if there is no server support for shallow fetches of arbitrary commits, the progress message is now used to inform the user about this potential for fetch time optimizations.

Fixes #10292
Fixes #12857

Git's `--shallow-since` feature can lead to broken checkouts and a fix
isn't likely to happen:
https://public-inbox.org/git/CAPSG9dZV2EPpVKkOMcjv5z+NF7rUu=V-ZkZNx47rCv122HsiKg@mail.gmail.com/T/#u

This commit removes `shallow_since` from the canonical reproducible form
of `git_repository` unless the attribute is provided explicitly. In
cases where Bazel's attempt at fetching with `--depth=1` fail, e.g. if
there is no server support for shallow fetches of arbitrary commits, the
progress message is now used to inform the user about this potential for
fetch time optimizations.

Fixes bazelbuild#10292
Fixes bazelbuild#12857
@fmeum
Copy link
Collaborator Author

fmeum commented Jan 30, 2023

@meteorcloudy Not sure who owns the git rules. Would you be available to review it?

@nelhage @asuffield Given that you contributed extensively to the linked issues, it would be highly appreciated if you could take a look at this proposed solution.

@fmeum fmeum marked this pull request as ready for review January 30, 2023 12:54
@sgowroji sgowroji added team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. awaiting-review PR is awaiting review from an assigned reviewer labels Jan 30, 2023
@asuffield
Copy link
Contributor

It's certainly an improvement over the status quo. To fully unbreak it we should also retry without shallow when reset fails - that's when we discover that shallow has "succeeded" and fetched junk.

@fmeum
Copy link
Collaborator Author

fmeum commented Feb 2, 2023

It's certainly an improvement over the status quo. To fully unbreak it we should also retry without shallow when reset fails - that's when we discover that shallow has "succeeded" and fetched junk.

I was thinking of proposing something more radical as a follow-up (but not cherry-pickable into Bazel 6): Deprecating shallow_since and always removing it from the canonical representation. Assuming the user has control over the Git server, they can always enable lookups by SHA if a full fetch is too costly. If they don't control it, it's very likely that the server is GitHub/GitLab/... which all seem to have this feature enabled.

@asuffield Do you think shallow_since is worth keeping around and "fixing"? If so, are there use cases in which it is clearly better than the other options?

@asuffield
Copy link
Contributor

Yes, for some future bazel version we should drop it entirely and always pull commits by hash at depth 1, when an exact commit hash is given. This does require a relatively recent version of git, since depth 1 by hash was only added a couple of years ago, but that doesn't seem unreasonable. Of course that needs the usual deprecation dance, since it'll break any current users; the best cherry pick I can think of is to retry when reset fails and just pull a full clone, since that prevents any existing builds from being surprisingly broken in the future.

@meteorcloudy
Copy link
Member

I was thinking of proposing something more radical as a follow-up

Thanks for the improvement! +1 if we can do the follow up change in a backwards compatible way.

@meteorcloudy meteorcloudy added awaiting-PR-merge PR has been approved by a reviewer and is ready to be merge internally and removed awaiting-review PR is awaiting review from an assigned reviewer labels Feb 7, 2023
@fmeum
Copy link
Collaborator Author

fmeum commented Feb 7, 2023

@meteorcloudy Do you think the current PR should be cherry-picked?

@meteorcloudy
Copy link
Member

Sure, I'm fine with cherry-picking it.

@fmeum
Copy link
Collaborator Author

fmeum commented Feb 8, 2023

@bazel-io flag

@bazel-io bazel-io added the potential release blocker Flagged by community members using "@bazel-io flag". Should be added to a release blocker milestone label Feb 8, 2023
@meteorcloudy
Copy link
Member

@bazel-io fork 6.1.0

@bazel-io bazel-io removed the potential release blocker Flagged by community members using "@bazel-io flag". Should be added to a release blocker milestone label Feb 9, 2023
@sgowroji sgowroji removed the awaiting-PR-merge PR has been approved by a reviewer and is ready to be merge internally label Feb 10, 2023
@fmeum fmeum deleted the 12857-shallow-since branch February 10, 2023 11:20
hvadehra pushed a commit that referenced this pull request Feb 14, 2023
Git's `--shallow-since` feature can lead to broken checkouts and a fix isn't likely to happen:
https://public-inbox.org/git/CAPSG9dZV2EPpVKkOMcjv5z+NF7rUu=V-ZkZNx47rCv122HsiKg@mail.gmail.com/T/#u

This commit removes `shallow_since` from the canonical reproducible form of `git_repository` unless the attribute is provided explicitly. In cases where Bazel's attempt at fetching with `--depth=1` fail, e.g. if there is no server support for shallow fetches of arbitrary commits, the progress message is now used to inform the user about this potential for fetch time optimizations.

Fixes #10292
Fixes #12857

Closes #17356.

PiperOrigin-RevId: 508569071
Change-Id: I01e6662e38c236d1800d427f66d48a05df818800
ShreeM01 added a commit that referenced this pull request Feb 16, 2023
Git's `--shallow-since` feature can lead to broken checkouts and a fix isn't likely to happen:
https://public-inbox.org/git/CAPSG9dZV2EPpVKkOMcjv5z+NF7rUu=V-ZkZNx47rCv122HsiKg@mail.gmail.com/T/#u

This commit removes `shallow_since` from the canonical reproducible form of `git_repository` unless the attribute is provided explicitly. In cases where Bazel's attempt at fetching with `--depth=1` fail, e.g. if there is no server support for shallow fetches of arbitrary commits, the progress message is now used to inform the user about this potential for fetch time optimizations.

Fixes #10292
Fixes #12857

Closes #17356.

PiperOrigin-RevId: 508569071
Change-Id: I01e6662e38c236d1800d427f66d48a05df818800

Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
copybara-service bot pushed a commit to google/pigweed that referenced this pull request Feb 21, 2024
Using shallow_since is not recommended, and the attribute is expected
to be removed in a future Bazel version:
bazelbuild/bazel#17356,
bazelbuild/bazel#12857

Change-Id: I3d6daeefad609fc34566b69de67eee2a49f7d5e4
Reviewed-on: https://pigweed-review.googlesource.com/c/pigweed/pigweed/+/193473
Presubmit-Verified: CQ Bot Account <pigweed-scoped@luci-project-accounts.iam.gserviceaccount.com>
Commit-Queue: Auto-Submit <auto-submit@pigweed-service-accounts.iam.gserviceaccount.com>
Pigweed-Auto-Submit: Ted Pudlik <tpudlik@google.com>
Reviewed-by: Armando Montanez <amontanez@google.com>
yperess pushed a commit to yperess/zephyr-bazel that referenced this pull request Mar 22, 2024
Using shallow_since is not recommended, and the attribute is expected
to be removed in a future Bazel version:
bazelbuild/bazel#17356,
bazelbuild/bazel#12857

Original-Reviewed-on: https://pigweed-review.googlesource.com/c/pigweed/pigweed/+/193473

https://pigweed.googlesource.com/pigweed/pigweed
third_party/pigweed Rolled-Commits: 371015e36501c36..9a3bb012f2c6dc4
Roller-URL: https://ci.chromium.org/b/8755460054149076049
GitWatcher: ignore
CQ-Do-Not-Cancel-Tryjobs: true
Change-Id: I8952a6aa062170a57fca6a2f0420e40d7242d403
Reviewed-on: https://pigweed-review.googlesource.com/c/example/echo/+/193492
Bot-Commit: Pigweed Roller <pigweed-roller@pigweed-service-accounts.iam.gserviceaccount.com>
Commit-Queue: Pigweed Roller <pigweed-roller@pigweed-service-accounts.iam.gserviceaccount.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file.
Projects
None yet
5 participants