Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cp18269 #3

Open
wants to merge 19 commits into
base: cp18450
Choose a base branch
from
Open

Cp18269 #3

wants to merge 19 commits into from

Conversation

iancha1992
Copy link
Owner

@iancha1992 iancha1992 commented Jun 9, 2023 •
edited
Loading

Closes #362

sluongng and others added 19 commits May 17, 2023 11:42
@sluongng @iancha1992
Add ServerCapabilities into RemoteExecutionClient
dbb9feb 
In bazelbuild#18202, we discussed the possibility of conditionally using the new
field exclusively based on the Remote Execution Server's capabilities.

Capture Remote Execution Server's capabilities and store it in
RemoteExecutor implementations for furture usage.

Closes bazelbuild#18269.

PiperOrigin-RevId: 531999688
Change-Id: I370869a45c804af1ec499b9c1654c6977c7ab7d0
@iancha1992 @fmeum
Report percentual download progress in repository rules (bazelbuild#1…
92cc903 
…8471)

If the HTTP response to a download request contains a `Content-Length` header, download progress is now reported as `10.1 MiB (20.2%)` instead of `10.1 MiB (10,590,000B)`.

Closes bazelbuild#18450.

PiperOrigin-RevId: 534035444
Change-Id: I1c5144555eda1890652b4d3f62b414292ba909d5

Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
@tjgq
[6.3.0] Support remote symlink outputs when building without the byte…
c422565 
…s. (bazelbuild#18476)

Fixes bazelbuild#13355.

PiperOrigin-RevId: 534008963
Change-Id: Ia4f22f16960bcdae86c1a8820e3d47a3689876d8
@iancha1992 @benjaminp
Enrich local BEP upload errors with file path and digest possible. (b…
53d785b 
…azelbuild#18481)

Closes bazelbuild#18461.

PiperOrigin-RevId: 534006853
Change-Id: I256297fe494393f13e178bf74f967b9b691db281

Co-authored-by: Benjamin Peterson <benjamin@engflow.com>
@iancha1992 @fmeum
Set GTEST_SHARD_STATUS_FILE in test setup (bazelbuild#18482)
f148571 
googletest only reads `GTEST_SHARD_STATUS_FILE`, not `TEST_SHARD_STATUS_FILE`, so this variable has to be set to keep sharded `cc_test`s relying on the test framework working with `--incompatible_check_sharding_support`.

Closes bazelbuild#18469.

PiperOrigin-RevId: 534375567
Change-Id: I0ca909cc7eb0b0f28f756e90e333e6bf39a0954d

Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
@keertk
Fix relnotes script (bazelbuild#18491)
96124a0 
- Update GitHub token retrieval
- Allow checking out tags (final releases) in addition to branches (RCs)

PiperOrigin-RevId: 534851155
Change-Id: I8b92898a8f5dd20139ca2467bb8604c6a8eabd67
@iancha1992 @keith
Fix Xcode 14.3 compatibility (bazelbuild#18490)
15fc292 
With Xcode 14.3+ on x86_64 machines there is an opaque error when linking binaries on macOS because Apple removed an old support library. That library is only linked if the macOS target is < 10.11, so this bumps the default versions past that. This macOS version was released in September 2015.

Fixes bazelbuild#18278

Closes bazelbuild#18460.

PiperOrigin-RevId: 534743568
Change-Id: I131880096c941df0097fe3b1faabd5a6afada4f3

Co-authored-by: Keith Smiley <keithbsmiley@gmail.com>
@Pavank1992 @meisterT
Fix bazelbuild#18493. (bazelbuild#18514)
a93c460 
The use of the pipe in
bazelbuild@b8e92cc made it swallow the exit code, so we need to set pipefail.

Fix bazelbuild#18493

Closes bazelbuild#18498.

PiperOrigin-RevId: 535520323
Change-Id: Idf1a5c39bf5b7deec29b76c10ece2825b568ebf2

Co-authored-by: Tobias Werth <twerth@google.com>
@tjgq
[6.3.0] Extend the credential helper default timeout to 10s. (bazelbu…
df283f2 
…ild#18527)

The default timeout makes the integration tests 1% flaky on RBE.

PiperOrigin-RevId: 535255555
Change-Id: I8270ffbfcbd00ec7c38a92a546d5726f1b10b68d
@keertk @iancha1992
Fix formatting of release notes (bazelbuild#18534)
e4814dc 
PiperOrigin-RevId: 536324450
Change-Id: I62c0aa7c91f5b38ddae9fae0127ea7b3fb601862

Co-authored-by: Ian (Hee) Cha <heec@google.com>
@iancha1992 @fmeum
Use extension rather than local names in ModuleExtensionMetadata (baz…
53d1d23 
…elbuild#18536)

ModuleExtensionMetadata incorrectly identified repos by their local names rather than the names used by the generating extension, which resulted in incorrect fixup warnings when supplying keyword arguments to `use_repo`.

Closes bazelbuild#18528.

PiperOrigin-RevId: 536386347
Change-Id: I4368b0dcdc28d6b2827b74b3b0b73c31a0215c0f

Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
@iancha1992 @Yannic
[credentialhelper] Ignore all errors when writing stdin (bazelbuild#1…
4afed73 
…8540)

It's totally fine for helpers to not care about the request on stdin and close stdin early.

```
ERROR: java.io.IOException: Broken pipe
	at java.base/java.io.FileOutputStream.writeBytes(Native Method)
	at java.base/java.io.FileOutputStream.write(Unknown Source)
	at java.base/java.io.BufferedOutputStream.flushBuffer(Unknown Source)
	at java.base/java.io.BufferedOutputStream.flush(Unknown Source)
	at java.base/java.io.FilterOutputStream.close(Unknown Source)
	at java.base/sun.nio.cs.StreamEncoder.implClose(Unknown Source)
	at java.base/sun.nio.cs.StreamEncoder.close(Unknown Source)
	at java.base/java.io.OutputStreamWriter.close(Unknown Source)
	at com.google.devtools.build.lib.authandtls.credentialhelper.CredentialHelper.getCredentials(CredentialHelper.java:83)
	at com.google.devtools.build.lib.authandtls.credentialhelper.CredentialHelperCredentials.getCredentialsFromHelper(CredentialHelperCredentials.java:108)
	at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$13(BoundedLocalCache.java:2451)
	at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
	at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2449)
	at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2432)
	at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:107)
	at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
	at com.google.devtools.build.lib.authandtls.credentialhelper.CredentialHelperCredentials.getRequestMetadata(CredentialHelperCredentials.java:80)
	at com.google.auth.Credentials.blockingGetToCallback(Credentials.java:112)
	at com.google.auth.Credentials$1.run(Credentials.java:98)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
```

Follow-up on bazelbuild@7c235ff

Closes bazelbuild#18501.

PiperOrigin-RevId: 536403312
Change-Id: I837bae70978e29f748c0fcc398c8f64de2b7b7c6

Co-authored-by: Yannic <yannic@engflow.com>
@iancha1992 @fmeum
Improve error on invalid -//foo and -@repo//foo options (bazelbui…
28ebcdc 
…ld#18516)

`-//foo` and `-@repo//foo` are always invalid Bazel options, but are usually meant to be either negative target patterns (which have to come after the `--` separator) or Starlark options (which always start with `--`).

With this commit, the error shown to the user mentions these two situations and how to fix the issue in each case.

Closes bazelbuild#16563.

PiperOrigin-RevId: 486920951
Change-Id: I9390d3859aa62c2b67eac05cb96a06209a9b7c36

Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
@amishra-u
[6.3.0] Implement failure circuit breaker (bazelbuild#18541)
31f07cc 
* feat: Implement failure circuit breaker

Copy of bazelbuild#18120: I accidentally closed bazelbuild#18120 during rebase and doesn't have permission to reopen.

We have noticed that any problems with the remote cache have a detrimental effect on build times. On investigation we found that the interface for the circuit breaker was left unimplemented.

To address this issue, implemented a failure circuit breaker, which includes three new Bazel flags: 1) experimental_circuitbreaker_strategy, 2) experimental_remote_failure_threshold, and 3) experimental_emote_failure_window.

In this implementation, I have implemented failure strategy for circuit breaker and used failure count to trip the circuit.

Reasoning behind using failure count instead of failure rate : To measure failure rate I also need the success count. While both the failure and success count need to be an AtomicInteger as both will be modified concurrently by multiple threads. Even though getAndIncrement is very light weight operation, at very high request it might contribute to latency.

Reasoning behind using failure circuit breaker : A new instance of Retrier.CircuitBreaker is created for each build. Therefore, if the circuit breaker trips during a build, the remote cache will be disabled for that build. However, it will be enabled again
for the next build as a new instance of Retrier.CircuitBreaker will be created. If needed in the future we may add cool down strategy also. e.g. failure_and_cool_down_startegy.

closes bazelbuild#18136

Closes bazelbuild#18359.

PiperOrigin-RevId: 536349954
Change-Id: I5e1c57d4ad0ce07ddc4808bf1f327bc5df6ce704

* remove target included in cherry-pick by mistake
@fmeum @keertk @meteorcloudy
Actually check TEST_SHARD_STATUS_FILE has been touched (bazelbuild#…
40d3998 
…18418)

Adds the missing check that a test runner running a sharded test actually declares support for sharding by generating the `TEST_SHARD_STATUS_FILE`. Previously, if it didn't, each shard would silently run all tests. This new behavior is guarded by the new `--incompatible_check_sharding_support` flag.

Closes bazelbuild#18236.

PiperOrigin-RevId: 530259354
Change-Id: If3b01b2c796e66ad7a77e542145efe3ab412eae9

Co-authored-by: keertk <keerthanakumar@google.com>
Co-authored-by: Yun Peng <pcloudy@google.com>
@iancha1992 @fmeum
Ignore hash string casing (bazelbuild#18414)
7bdd173 
Fixes this crash when e.g. the `sha256` attribute is passed an upper case string:

```
Caused by: java.lang.IllegalArgumentException: Illegal hexadecimal character: D
        at com.google.common.hash.HashCode.decode(HashCode.java:360)
        at com.google.common.hash.HashCode.fromString(HashCode.java:346)
        at com.google.devtools.build.lib.bazel.repository.downloader.Checksum.fromString(Checksum.java:47)
        at com.google.devtools.build.lib.bazel.repository.starlark.StarlarkBaseExternalContext.validateChecksum(StarlarkBaseExternalContext.java:302)
        at com.google.devtools.build.lib.bazel.repository.starlark.StarlarkBaseExternalContext.downloadAndExtract(StarlarkBaseExternalContext.java:650)
        ...
```

Fixes bazelbuild#18291

Closes bazelbuild#18305.

PiperOrigin-RevId: 529601921
Change-Id: I75deee47bcac80ee81603591c22f43d013ba0c29

Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
@iancha1992 @illicitonion
Error if repository name isn't supplied (bazelbuild#18425)
caf00d5 
Currently, this results in a server crash along the lines of:
```
FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.RuntimeException: Unrecoverable error while evaluating node '[/redacted]/[WORKSPACE.bazel], 1' (requested by nodes 'com.google.devtools.build.lib.skyframe.ExternalPackageFunction$$Lambda$231/0x0000000800df7998@34f133c')
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:642)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:382)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: net.starlark.java.eval.Starlark$UncheckedEvalException: NullPointerException thrown during Starlark evaluation (WORKSPACE)
	at <starlark>.repository_rule(<builtin>:0)
	at <starlark>.<toplevel>(/redacted/WORKSPACE.bazel:3)
Caused by: java.lang.NullPointerException: Cannot invoke "String.isEmpty()" because "name" is null
	at com.google.devtools.build.lib.cmdline.RepositoryName.create(RepositoryName.java:71)
	at com.google.devtools.build.lib.packages.WorkspaceFactoryHelper.addMainRepoEntry(WorkspaceFactoryHelper.java:106)
	at com.google.devtools.build.lib.bazel.repository.starlark.StarlarkRepositoryModule$RepositoryRuleFunction.createRuleLegacy(StarlarkRepositoryModule.java:228)
	at com.google.devtools.build.lib.bazel.repository.starlark.StarlarkRepositoryModule$RepositoryRuleFunction.call(StarlarkRepositoryModule.java:185)
```

Closes bazelbuild#18335.

PiperOrigin-RevId: 530577557
Change-Id: Ic402e8fabac180aaa2da19cd3201b9fb2671dccb

Co-authored-by: Daniel Wagner-Hall <dwagnerhall@apple.com>
@iancha1992 @fmeum @keertk
Track repo rule label attributes after the first non-existent one (ba…
2a026e6 
…zelbuild#18412)

Even with this commit, the fact that a particular label in a repository rule's label attributes does not resolve to a regular file is not tracked, which means that there is still a potential for incorrect incremental fetches.

Work towards bazelbuild#13441

Closes bazelbuild#18352.

PiperOrigin-RevId: 531150549
Change-Id: I086e4813ca88a3f49cde77d9be20adaaf954834b

Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
Co-authored-by: keertk <keerthanakumar@google.com>
@iancha1992
Merge branch 'release-6.3.0' into cp18269
39f9b38
@iancha1992
Copy link
Owner Author

@bazel-io fork 6.3.0

This was referenced Jun 9, 2023
Closed
Closed
@iancha1992
Copy link
Owner Author

@bazel-io fork 6.3.0

@iancha1992 iancha1992 mentioned this pull request Jun 9, 2023
Closed
@iancha1992
Copy link
Owner Author

@bazel-io fork 6.3.0

@iancha1992 iancha1992 mentioned this pull request Jun 9, 2023
Closed
@iancha1992
Copy link
Owner Author

@bazel-io fork 6.3.0

@iancha1992 iancha1992 mentioned this pull request Jun 9, 2023
Closed
@iancha1992
Copy link
Owner Author

@bazel-io fork 6.3.0

@iancha1992 iancha1992 mentioned this pull request Jun 9, 2023
Closed
@iancha1992
Copy link
Owner Author

@bazel-io fork 6.3.0

@iancha1992 iancha1992 mentioned this pull request Jun 11, 2023
Closed
@iancha1992
Copy link
Owner Author

@bazel-io fork 6.3.0

@iancha1992 iancha1992 mentioned this pull request Jun 11, 2023
Closed
@iancha1992
Copy link
Owner Author

@bazel-io fork 6.3.0

@iancha1992 iancha1992 mentioned this pull request Jun 11, 2023
Closed
@iancha1992 iancha1992 reopened this Jun 12, 2023
@iancha1992
Copy link
Owner Author

@bazel-io fork 6.3.0

@iancha1992 iancha1992 mentioned this pull request Jun 12, 2023
Closed
@iancha1992
Copy link
Owner Author

@bazel-io fork 6.3.0

@iancha1992 iancha1992 mentioned this pull request Jun 12, 2023
Closed
@iancha1992
Copy link
Owner Author

@bazel-io fork 6.3.0

@iancha1992 iancha1992 mentioned this pull request Jun 13, 2023
Closed
@iancha1992 iancha1992 closed this Jun 13, 2023
@iancha1992 iancha1992 reopened this Jun 13, 2023
@iancha1992 iancha1992 closed this Jun 13, 2023
@iancha1992 iancha1992 reopened this Jun 13, 2023
@iancha1992 iancha1992 closed this Jun 13, 2023
@iancha1992 iancha1992 reopened this Jun 13, 2023
@iancha1992
Copy link
Owner Author

@bazel-io fork 6.3.0

@iancha1992 iancha1992 mentioned this pull request Jun 13, 2023
Closed
@iancha1992
Copy link
Owner Author

@bazel-io fork 6.3.0

@iancha1992 iancha1992 mentioned this pull request Jun 13, 2023
Closed
@iancha1992
Copy link
Owner Author

@bazel-io fork 6.3.0

@iancha1992 iancha1992 mentioned this pull request Jun 13, 2023
Closed
@iancha1992
Copy link
Owner Author

@bazel-io fork 6.3.0

@iancha1992 iancha1992 mentioned this pull request Jun 13, 2023
Closed
@iancha1992
Copy link
Owner Author

@bazel-io fork 6.3.0

@iancha1992 iancha1992 mentioned this pull request Jun 21, 2023
Open
@iancha1992 iancha1992 linked an issue Nov 21, 2023 that may be closed by this pull request
[6.3.0] testing_phase2 Please ignore. #387
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[6.3.0] testing_phase2 Please ignore.
7 participants
@iancha1992 @sluongng @tjgq @keertk @Pavank1992 @amishra-u @fmeum