-
Notifications
You must be signed in to change notification settings - Fork 13k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
rustc: document the jobserver #121564
Merged
+90
−1
Merged
rustc: document the jobserver #121564
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,86 @@ | ||
# Jobserver | ||
|
||
Internally, `rustc` may take advantage of parallelism. `rustc` will coordinate | ||
with the build system calling it if a [GNU Make jobserver] is passed in the | ||
`MAKEFLAGS` environment variable. Other flags may have an effect as well, such | ||
as [`CARGO_MAKEFLAGS`]. If a jobserver is not passed, then `rustc` will choose | ||
the number of jobs to use. | ||
|
||
Starting with Rust 1.76.0, `rustc` will warn if a jobserver appears to be | ||
available but is not accessible, e.g.: | ||
|
||
```console | ||
$ echo 'fn main() {}' | MAKEFLAGS=--jobserver-auth=3,4 rustc - | ||
warning: failed to connect to jobserver from environment variable `MAKEFLAGS="--jobserver-auth=3,4"`: cannot open file descriptor 3 from the jobserver environment variable value: Bad file descriptor (os error 9) | ||
| | ||
= note: the build environment is likely misconfigured | ||
``` | ||
|
||
## Integration with build systems | ||
|
||
The following subsections contain recommendations on how to integrate `rustc` | ||
with build systems so that the jobserver is handled appropriately. | ||
|
||
### GNU Make | ||
|
||
When calling `rustc` from GNU Make, it is recommended that all `rustc` | ||
invocations are marked as recursive in the `Makefile` (by prefixing the command | ||
line with the `+` indicator), so that GNU Make enables the jobserver for them. | ||
For instance: | ||
|
||
<!-- ignore-tidy-tab --> | ||
|
||
```make | ||
x: | ||
+@echo 'fn main() {}' | rustc - | ||
petrochenkov marked this conversation as resolved.
Show resolved
Hide resolved
|
||
``` | ||
|
||
In particular, GNU Make 4.3 (a widely used version as of 2024) passes a simple | ||
pipe jobserver in `MAKEFLAGS` even when it was not made available for the child | ||
process, which in turn means `rustc` will warn about it. For instance, if the | ||
`+` indicator is removed from the example above and GNU Make is called with e.g. | ||
`make -j2`, then the aforementioned warning will trigger. | ||
|
||
For calls to `rustc` inside `$(shell ...)` inside a recursive Make, one can | ||
disable the jobserver manually by clearing the `MAKEFLAGS` variable, e.g.: | ||
petrochenkov marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
```make | ||
S := $(shell MAKEFLAGS= rustc --print sysroot) | ||
|
||
x: | ||
@$(MAKE) y | ||
|
||
y: | ||
@echo $(S) | ||
``` | ||
|
||
### CMake | ||
|
||
CMake 3.28 supports the `JOB_SERVER_AWARE` option in its [`add_custom_target`] | ||
command, e.g.: | ||
|
||
```cmake | ||
cmake_minimum_required(VERSION 3.28) | ||
project(x) | ||
add_custom_target(x | ||
JOB_SERVER_AWARE TRUE | ||
COMMAND echo 'fn main() {}' | rustc - | ||
) | ||
``` | ||
|
||
For earlier versions, when using CMake with the Makefile generator, one | ||
workaround is to have [`$(MAKE)`] somewhere in the command so that GNU Make | ||
treats it as a recursive Make call, e.g.: | ||
|
||
```cmake | ||
cmake_minimum_required(VERSION 3.22) | ||
project(x) | ||
add_custom_target(x | ||
COMMAND DUMMY_VARIABLE=$(MAKE) echo 'fn main() {}' | rustc - | ||
) | ||
``` | ||
|
||
[GNU Make jobserver]: https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html | ||
[`CARGO_MAKEFLAGS`]: https://doc.rust-lang.org/cargo/reference/environment-variables.html | ||
[`add_custom_target`]: https://cmake.org/cmake/help/latest/command/add_custom_target.html | ||
[`$(MAKE)`]: https://www.gnu.org/software/make/manual/html_node/MAKE-Variable.html |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seemed a little unusual to call out near the beginning of the chapter. Perhaps a different way to flow this would be to move this down below after the paragraph that says "if the
+
indicator is removed", and show that as an illustration of what happens when+
is removed.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The reason I put it at the top is that the warning applies to all build systems, i.e. the intention was that we would have subsections for other build systems too if needed, but it isn't clear since we only have one, as you mention (and indeed I did have to do a bit of a dance to make the text work without having to repeat the warning block, which is what I originally had, but it felt too heavy).
On the other hand, it may be true that other build systems do not have this issue (i.e. either they don't use a jobserver, or if they do, they enable it "properly", so perhaps nobody else will actually hit that warning, but it is hard to say). Still, it seemed like conceptually it was something outside GNU Make in particular.
Perhaps it could help adding a subsection called "Integrations" or similar with a small text like "These are recommendations for integration of
rustc
with different build systems." and then make the GNU Make one a subsubsection of that.I also thought about adding a trivial one about Cargo, saying something like "Cargo already handles it".
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pushed a new version with what I meant here so that it is simpler to see.