Support #[global_allocator] without the allocator shim #86844

bjorn3 · 2021-07-03T16:07:54Z

This makes it possible to use liballoc/libstd in combination with --emit obj if you use #[global_allocator]. This is what rust-for-linux uses right now and systemd may use in the future. Currently they have to depend on the exact implementation of the allocator shim to create one themself as --emit obj doesn't create an allocator shim.

Note that currently the allocator shim also defines the oom error handler, which is normally required too. Once #![feature(default_alloc_error_handler)] becomes the only option, this can be avoided. In addition when using only fallible allocator methods and either --cfg no_global_oom_handling for liballoc (like rust-for-linux) or --gc-sections no references to the oom error handler will exist.

To avoid this feature being insta-stable, you will have to define __rust_no_alloc_shim_is_unstable to avoid linker errors.

(Labeling this with both T-compiler and T-lang as it originally involved both an implementation detail and had an insta-stable user facing change. As noted above, the __rust_no_alloc_shim_is_unstable symbol requirement should prevent unintended dependence on this unstable feature.)

rust-highfive · 2021-07-03T16:07:56Z

Some changes occured to rustc_codegen_cranelift

cc @bjorn3

rust-highfive · 2021-07-03T16:07:57Z

r? @jackh726

(rust-highfive has picked a reviewer for you, use r? to override)

jyn514 · 2021-07-03T17:10:17Z

Hmm, r? @scottmcm maybe?

compiler/rustc_codegen_cranelift/src/allocator.rs

bors · 2021-08-06T21:39:16Z

☔ The latest upstream changes (presumably #87822) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2021-08-07T23:40:20Z

☔ The latest upstream changes (presumably #87743) made this pull request unmergeable. Please resolve the merge conflicts.

nbdd0121 · 2021-08-08T15:33:01Z

It took me a few while to understand this, so essentially instead of generating __rust_alloc that forwards to __rg_alloc or __rdl_alloc based on allocator kind, this PR makes #[global_allocator] generate __rust_alloc directly, while in case global allocator is absent, generate a shim that forwards __rust_alloc to __rdl_alloc.

Would it make sense to (in addition to this PR), just generate a

#[global_allocator]
static GLOBAL: System = System;

at somewhere in higher level when global allocator is absent, instead of having the logic duplicated in codegen? This would allow __rdl_alloc etc to be removed completely (and is more consistent with what document describes in https://doc.rust-lang.org/std/alloc/struct.System.html).

bjorn3 · 2021-08-08T15:37:18Z

Would it make sense to just generate a
#[global_allocator]
static GLOBAL: System = System;
at somewhere in higher level instead of having the logic duplicated in codegen? This would allow __rdl_alloc etc to be removed completely (and is more consistent with what document describes in https://doc.rust-lang.org/std/alloc/struct.System.html).

You can use say --crate-type cdylib --crate-type lib, in which case the cdylib would need the allocator shim, but the lib must not get the allocator shim. Both use the same object files, except for the allocator shim object file that only the bin gets.

compiler/rustc_builtin_macros/src/global_allocator.rs

wesleywiser · 2021-08-26T23:33:21Z

I posted a message in the wg-allocators Zulip stream to see if anyone there has opinions on this PR before it's merged.

This is the second upgrade to the Rust toolchain, from 1.68.2 to 1.71.0 (i.e. the latest). See the upgrade policy [1] and the comments on the first upgrade in commit 3ed03f4 ("rust: upgrade to Rust 1.68.2"). No unstable features (that we use) were stabilized. Therefore, the only unstable feature allowed to be used outside the `kernel` crate is still `new_uninit`, though other code to be upstreamed may increase the list. Please see [2] for details. For the upgrade, this patch requires the following changes: - Removal of the `__rust_*` allocator functions, together with the addition of the `__rust_no_alloc_shim_is_unstable` static. See [3] for details. - Some more compiler builtins added due to `<f{32,64}>::midpoint()` that got added in Rust 1.71 [4]. The vast majority of changes are due to our `alloc` fork being upgraded at once. There are two kinds of changes to be aware of: the ones coming from upstream, which we should follow as closely as possible, and the updates needed in our added fallible APIs to keep them matching the newer infallible APIs coming from upstream. Instead of taking a look at the diff of this patch, an alternative approach is reviewing a diff of the changes between upstream `alloc` and the kernel's. This allows to easily inspect the kernel additions only, especially to check if the fallible methods we already have still match the infallible ones in the new version coming from upstream. Another approach is reviewing the changes introduced in the additions in the kernel fork between the two versions. This is useful to spot potentially unintended changes to our additions. To apply these approaches, one may follow steps similar to the following to generate a pair of patches that show the differences between upstream Rust and the kernel (for the subset of `alloc` we use) before and after applying this patch: # Get the difference with respect to the old version. git -C rust checkout $(linux/scripts/min-tool-version.sh rustc) git -C linux ls-tree -r --name-only HEAD -- rust/alloc | cut -d/ -f3- | grep -Fv README.md | xargs -IPATH cp rust/library/alloc/src/PATH linux/rust/alloc/PATH git -C linux diff --patch-with-stat --summary -R > old.patch git -C linux restore rust/alloc # Apply this patch. git -C linux am rust-upgrade.patch # Get the difference with respect to the new version. git -C rust checkout $(linux/scripts/min-tool-version.sh rustc) git -C linux ls-tree -r --name-only HEAD -- rust/alloc | cut -d/ -f3- | grep -Fv README.md | xargs -IPATH cp rust/library/alloc/src/PATH linux/rust/alloc/PATH git -C linux diff --patch-with-stat --summary -R > new.patch git -C linux restore rust/alloc Now one may check the `new.patch` to take a look at the additions (first approach) or at the difference between those two patches (second approach). For the latter, a side-by-side tool is recommended. Link: https://rust-for-linux.com/rust-version-policy [1] Link: Rust-for-Linux/linux#2 [2] Link: rust-lang/rust#86844 [3] Link: rust-lang/rust#92048 [4] Signed-off-by: Miguel Ojeda <ojeda@kernel.org>

This is the second upgrade to the Rust toolchain, from 1.68.2 to 1.71.0 (i.e. the latest). See the upgrade policy [1] and the comments on the first upgrade in commit 3ed03f4 ("rust: upgrade to Rust 1.68.2"). No unstable features (that we use) were stabilized. Therefore, the only unstable feature allowed to be used outside the `kernel` crate is still `new_uninit`, though other code to be upstreamed may increase the list. Please see [2] for details. For the upgrade, this patch requires the following changes: - Removal of the `__rust_*` allocator functions, together with the addition of the `__rust_no_alloc_shim_is_unstable` static. See [3] for details. - Some more compiler builtins added due to `<f{32,64}>::midpoint()` that got added in Rust 1.71 [4]. The vast majority of changes are due to our `alloc` fork being upgraded at once. There are two kinds of changes to be aware of: the ones coming from upstream, which we should follow as closely as possible, and the updates needed in our added fallible APIs to keep them matching the newer infallible APIs coming from upstream. Instead of taking a look at the diff of this patch, an alternative approach is reviewing a diff of the changes between upstream `alloc` and the kernel's. This allows to easily inspect the kernel additions only, especially to check if the fallible methods we already have still match the infallible ones in the new version coming from upstream. Another approach is reviewing the changes introduced in the additions in the kernel fork between the two versions. This is useful to spot potentially unintended changes to our additions. To apply these approaches, one may follow steps similar to the following to generate a pair of patches that show the differences between upstream Rust and the kernel (for the subset of `alloc` we use) before and after applying this patch: # Get the difference with respect to the old version. git -C rust checkout $(linux/scripts/min-tool-version.sh rustc) git -C linux ls-tree -r --name-only HEAD -- rust/alloc | cut -d/ -f3- | grep -Fv README.md | xargs -IPATH cp rust/library/alloc/src/PATH linux/rust/alloc/PATH git -C linux diff --patch-with-stat --summary -R > old.patch git -C linux restore rust/alloc # Apply this patch. git -C linux am rust-upgrade.patch # Get the difference with respect to the new version. git -C rust checkout $(linux/scripts/min-tool-version.sh rustc) git -C linux ls-tree -r --name-only HEAD -- rust/alloc | cut -d/ -f3- | grep -Fv README.md | xargs -IPATH cp rust/library/alloc/src/PATH linux/rust/alloc/PATH git -C linux diff --patch-with-stat --summary -R > new.patch git -C linux restore rust/alloc Now one may check the `new.patch` to take a look at the additions (first approach) or at the difference between those two patches (second approach). For the latter, a side-by-side tool is recommended. Link: https://rust-for-linux.com/rust-version-policy [1] Link: Rust-for-Linux#2 [2] Link: rust-lang/rust#86844 [3] Link: rust-lang/rust#92048 [4] Signed-off-by: Miguel Ojeda <ojeda@kernel.org> Reviewed-by: Martin Rodriguez Reboredo <yakoyoku@gmail.com> Link: https://lore.kernel.org/r/20230729220317.416771-1-ojeda@kernel.org [boqun: Resolve conflicts with bindgen 0.65.1 patch] [boqun: Resolve conflicts with __rust_alloc* fix ]

This is the second upgrade to the Rust toolchain, from 1.68.2 to 1.71.0 (i.e. the latest). See the upgrade policy [1] and the comments on the first upgrade in commit 3ed03f4 ("rust: upgrade to Rust 1.68.2"). No unstable features (that we use) were stabilized. Therefore, the only unstable feature allowed to be used outside the `kernel` crate is still `new_uninit`, though other code to be upstreamed may increase the list. Please see [2] for details. For the upgrade, this patch requires the following changes: - Removal of the `__rust_*` allocator functions, together with the addition of the `__rust_no_alloc_shim_is_unstable` static. See [3] for details. - Some more compiler builtins added due to `<f{32,64}>::midpoint()` that got added in Rust 1.71 [4]. The vast majority of changes are due to our `alloc` fork being upgraded at once. There are two kinds of changes to be aware of: the ones coming from upstream, which we should follow as closely as possible, and the updates needed in our added fallible APIs to keep them matching the newer infallible APIs coming from upstream. Instead of taking a look at the diff of this patch, an alternative approach is reviewing a diff of the changes between upstream `alloc` and the kernel's. This allows to easily inspect the kernel additions only, especially to check if the fallible methods we already have still match the infallible ones in the new version coming from upstream. Another approach is reviewing the changes introduced in the additions in the kernel fork between the two versions. This is useful to spot potentially unintended changes to our additions. To apply these approaches, one may follow steps similar to the following to generate a pair of patches that show the differences between upstream Rust and the kernel (for the subset of `alloc` we use) before and after applying this patch: # Get the difference with respect to the old version. git -C rust checkout $(linux/scripts/min-tool-version.sh rustc) git -C linux ls-tree -r --name-only HEAD -- rust/alloc | cut -d/ -f3- | grep -Fv README.md | xargs -IPATH cp rust/library/alloc/src/PATH linux/rust/alloc/PATH git -C linux diff --patch-with-stat --summary -R > old.patch git -C linux restore rust/alloc # Apply this patch. git -C linux am rust-upgrade.patch # Get the difference with respect to the new version. git -C rust checkout $(linux/scripts/min-tool-version.sh rustc) git -C linux ls-tree -r --name-only HEAD -- rust/alloc | cut -d/ -f3- | grep -Fv README.md | xargs -IPATH cp rust/library/alloc/src/PATH linux/rust/alloc/PATH git -C linux diff --patch-with-stat --summary -R > new.patch git -C linux restore rust/alloc Now one may check the `new.patch` to take a look at the additions (first approach) or at the difference between those two patches (second approach). For the latter, a side-by-side tool is recommended. Link: https://rust-for-linux.com/rust-version-policy [1] Link: Rust-for-Linux/linux#2 [2] Link: rust-lang/rust#86844 [3] Link: rust-lang/rust#92048 [4] Signed-off-by: Miguel Ojeda <ojeda@kernel.org>

This is the second upgrade to the Rust toolchain, from 1.68.2 to 1.71.1 (i.e. the latest). See the upgrade policy [1] and the comments on the first upgrade in commit 3ed03f4 ("rust: upgrade to Rust 1.68.2"). # Unstable features No unstable features (that we use) were stabilized. Therefore, the only unstable feature allowed to be used outside the `kernel` crate is still `new_uninit`, though other code to be upstreamed may increase the list. Please see [2] for details. # Required changes For the upgrade, this patch requires the following changes: - Removal of the `__rust_*` allocator functions, together with the addition of the `__rust_no_alloc_shim_is_unstable` static. See [3] for details. - Some more compiler builtins added due to `<f{32,64}>::midpoint()` that got added in Rust 1.71 [4]. # `alloc` upgrade and reviewing The vast majority of changes are due to our `alloc` fork being upgraded at once. There are two kinds of changes to be aware of: the ones coming from upstream, which we should follow as closely as possible, and the updates needed in our added fallible APIs to keep them matching the newer infallible APIs coming from upstream. Instead of taking a look at the diff of this patch, an alternative approach is reviewing a diff of the changes between upstream `alloc` and the kernel's. This allows to easily inspect the kernel additions only, especially to check if the fallible methods we already have still match the infallible ones in the new version coming from upstream. Another approach is reviewing the changes introduced in the additions in the kernel fork between the two versions. This is useful to spot potentially unintended changes to our additions. To apply these approaches, one may follow steps similar to the following to generate a pair of patches that show the differences between upstream Rust and the kernel (for the subset of `alloc` we use) before and after applying this patch: # Get the difference with respect to the old version. git -C rust checkout $(linux/scripts/min-tool-version.sh rustc) git -C linux ls-tree -r --name-only HEAD -- rust/alloc | cut -d/ -f3- | grep -Fv README.md | xargs -IPATH cp rust/library/alloc/src/PATH linux/rust/alloc/PATH git -C linux diff --patch-with-stat --summary -R > old.patch git -C linux restore rust/alloc # Apply this patch. git -C linux am rust-upgrade.patch # Get the difference with respect to the new version. git -C rust checkout $(linux/scripts/min-tool-version.sh rustc) git -C linux ls-tree -r --name-only HEAD -- rust/alloc | cut -d/ -f3- | grep -Fv README.md | xargs -IPATH cp rust/library/alloc/src/PATH linux/rust/alloc/PATH git -C linux diff --patch-with-stat --summary -R > new.patch git -C linux restore rust/alloc Now one may check the `new.patch` to take a look at the additions (first approach) or at the difference between those two patches (second approach). For the latter, a side-by-side tool is recommended. Link: https://rust-for-linux.com/rust-version-policy [1] Link: Rust-for-Linux#2 [2] Link: rust-lang/rust#86844 [3] Link: rust-lang/rust#92048 [4] Closes: Rust-for-Linux#68 Reviewed-by: Martin Rodriguez Reboredo <yakoyoku@gmail.com> Reviewed-by: Trevor Gross <tmgross@umich.edu> Link: https://lore.kernel.org/r/20230729220317.416771-1-ojeda@kernel.org Signed-off-by: Miguel Ojeda <ojeda@kernel.org>

This is the second upgrade to the Rust toolchain, from 1.68.2 to 1.71.0 (i.e. the latest). See the upgrade policy [1] and the comments on the first upgrade in commit 3ed03f4 ("rust: upgrade to Rust 1.68.2"). No unstable features (that we use) were stabilized. Therefore, the only unstable feature allowed to be used outside the `kernel` crate is still `new_uninit`, though other code to be upstreamed may increase the list. Please see [2] for details. For the upgrade, this patch requires the following changes: - Removal of the `__rust_*` allocator functions, together with the addition of the `__rust_no_alloc_shim_is_unstable` static. See [3] for details. - Some more compiler builtins added due to `<f{32,64}>::midpoint()` that got added in Rust 1.71 [4]. The vast majority of changes are due to our `alloc` fork being upgraded at once. There are two kinds of changes to be aware of: the ones coming from upstream, which we should follow as closely as possible, and the updates needed in our added fallible APIs to keep them matching the newer infallible APIs coming from upstream. Instead of taking a look at the diff of this patch, an alternative approach is reviewing a diff of the changes between upstream `alloc` and the kernel's. This allows to easily inspect the kernel additions only, especially to check if the fallible methods we already have still match the infallible ones in the new version coming from upstream. Another approach is reviewing the changes introduced in the additions in the kernel fork between the two versions. This is useful to spot potentially unintended changes to our additions. To apply these approaches, one may follow steps similar to the following to generate a pair of patches that show the differences between upstream Rust and the kernel (for the subset of `alloc` we use) before and after applying this patch: # Get the difference with respect to the old version. git -C rust checkout $(linux/scripts/min-tool-version.sh rustc) git -C linux ls-tree -r --name-only HEAD -- rust/alloc | cut -d/ -f3- | grep -Fv README.md | xargs -IPATH cp rust/library/alloc/src/PATH linux/rust/alloc/PATH git -C linux diff --patch-with-stat --summary -R > old.patch git -C linux restore rust/alloc # Apply this patch. git -C linux am rust-upgrade.patch # Get the difference with respect to the new version. git -C rust checkout $(linux/scripts/min-tool-version.sh rustc) git -C linux ls-tree -r --name-only HEAD -- rust/alloc | cut -d/ -f3- | grep -Fv README.md | xargs -IPATH cp rust/library/alloc/src/PATH linux/rust/alloc/PATH git -C linux diff --patch-with-stat --summary -R > new.patch git -C linux restore rust/alloc Now one may check the `new.patch` to take a look at the additions (first approach) or at the difference between those two patches (second approach). For the latter, a side-by-side tool is recommended. Link: https://rust-for-linux.com/rust-version-policy [1] Link: Rust-for-Linux/linux#2 [2] Link: rust-lang/rust#86844 [3] Link: rust-lang/rust#92048 [4] Signed-off-by: Miguel Ojeda <ojeda@kernel.org>

…=bjorn3 read_volatile __rust_no_alloc_shim_is_unstable in alloc_zeroed It was pointed out in rust-lang#128854 (comment) that the magic volatile read was probably missing from `alloc_zeroed`. I can't find any mention of `alloc_zeroed` on rust-lang#86844, so it looks like this was just missed initially.

rust-highfive assigned jackh726 Jul 3, 2021

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jul 3, 2021

This comment has been minimized.

Sign in to view

bjorn3 force-pushed the global_alloc_improvements branch from ba27e9f to 51f054c Compare July 3, 2021 16:11

This comment has been minimized.

Sign in to view

bjorn3 force-pushed the global_alloc_improvements branch from e4a996a to b2b1a59 Compare July 3, 2021 16:40

rust-highfive assigned scottmcm and unassigned jackh726 Jul 3, 2021

bjorn3 marked this pull request as ready for review July 3, 2021 17:16

bjorn3 mentioned this pull request Jul 3, 2021

Implement "default_alloc_error_handler" feature rust-lang/rustc_codegen_cranelift#1182

Closed

ojeda mentioned this pull request Jul 4, 2021

Rust wanted features Rust-for-Linux/linux#354

Open

37 tasks

RalfJung reviewed Jul 6, 2021

View reviewed changes

compiler/rustc_codegen_cranelift/src/allocator.rs Outdated Show resolved Hide resolved

bjorn3 mentioned this pull request Jul 12, 2021

Start working on proof of concept for exposing Backtrace in core #77384

Closed

camelid added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 23, 2021

bjorn3 force-pushed the global_alloc_improvements branch from 355b470 to 4b56c58 Compare August 7, 2021 09:38

bjorn3 force-pushed the global_alloc_improvements branch from 4b56c58 to 74b52ae Compare August 8, 2021 09:47

nbdd0121 reviewed Aug 9, 2021

View reviewed changes

compiler/rustc_builtin_macros/src/global_allocator.rs Outdated Show resolved Hide resolved

Mark-Simulacrum mentioned this pull request Mar 24, 2024

Tracking Issue for __rust_no_alloc_shim_is_unstable #123015

Open

3 tasks

saethlin mentioned this pull request Sep 18, 2024

read_volatile __rust_no_alloc_shim_is_unstable in alloc_zeroed #130497

Merged

saethlin added the A-rust-for-linux Relevant for the Rust-for-Linux project label Sep 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support #[global_allocator] without the allocator shim #86844

Support #[global_allocator] without the allocator shim #86844

bjorn3 commented Jul 3, 2021 •

edited by pnkfelix

Loading

rust-highfive commented Jul 3, 2021

rust-highfive commented Jul 3, 2021

This comment has been minimized.

This comment has been minimized.

jyn514 commented Jul 3, 2021

bors commented Aug 6, 2021

bors commented Aug 7, 2021

nbdd0121 commented Aug 8, 2021 •

edited

Loading

bjorn3 commented Aug 8, 2021

wesleywiser commented Aug 26, 2021

Support #[global_allocator] without the allocator shim #86844

Support #[global_allocator] without the allocator shim #86844

Conversation

bjorn3 commented Jul 3, 2021 • edited by pnkfelix Loading

rust-highfive commented Jul 3, 2021

rust-highfive commented Jul 3, 2021

This comment has been minimized.

This comment has been minimized.

jyn514 commented Jul 3, 2021

bors commented Aug 6, 2021

bors commented Aug 7, 2021

nbdd0121 commented Aug 8, 2021 • edited Loading

bjorn3 commented Aug 8, 2021

wesleywiser commented Aug 26, 2021

bjorn3 commented Jul 3, 2021 •

edited by pnkfelix

Loading

nbdd0121 commented Aug 8, 2021 •

edited

Loading