Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

avoid duplicating TLS state between test std and realstd #110946

Merged
merged 1 commit into from
May 5, 2023

Conversation

RalfJung
Copy link
Member

This basically re-lands #100201 and #106638, which got reverted by #110861. This works around 2 Miri limitations:

  • Miri doesn't support the magic linker section that our Windows TLS support relies on, and instead knows where in std to find the symbol that stores the thread callback.
  • For macOS, Miri only supports at most one destructor to be registered per thread.

The 2nd would not be very hard to fix (though the intended destructor order is unclear); the first would be a lot of work to fix. Neither of these is a problem for regular Rust code, but in the std test suite we have essentially 2 copies of the std code and then these both become issues. To avoid that we have the std test crate import the TLS code from the real std instead of having its own copy.

r? @m-ou-se

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-libs Relevant to the library team, which will review and decide on the PR/issue. labels Apr 28, 2023
@rustbot
Copy link
Collaborator

rustbot commented Apr 28, 2023

Hey! It looks like you've submitted a new PR for the library teams!

If this PR contains changes to any rust-lang/rust public library APIs then please comment with @rustbot label +T-libs-api -T-libs to tag it appropriately. If this PR contains changes to any unstable APIs please edit the PR description to add a link to the relevant API Change Proposal or create one if you haven't already. If you're unsure where your change falls no worries, just leave it as is and the reviewer will take a look and make a decision to forward on if necessary.

Examples of T-libs-api changes:

  • Stabilizing library features
  • Introducing insta-stable changes such as new implementations of existing stable traits on existing stable types
  • Introducing new or changing existing unstable library APIs (excluding permanently unstable features / features without a tracking issue)
  • Changing public documentation in ways that create new stability guarantees
  • Changing observable runtime behavior of library APIs

@ChrisDenton
Copy link
Member

(tbh it would be nice if the "magic linker section" bit was implemented with the help of a little bit of compiler magic as it is hacky even in std, but I digress)

@m-ou-se
Copy link
Member

m-ou-se commented May 1, 2023

If it's only for miri, can this be cfg(miri) or cfg(all(miri, test)) rather than cfg(test)?

@m-ou-se
Copy link
Member

m-ou-se commented May 1, 2023

Removes a seemingly unnecessary realstd re-export on cfg(test).

This was necessary, see #100201 and #106638. I guess I need to leave more detailed comments in the code.

We do run miri's tests in rust-lang/rust's CI, right? Is there no (miri) test that relies on thread locals?

@m-ou-se
Copy link
Member

m-ou-se commented May 1, 2023

Oh, this is only for the situation where miri is used to test std itself. I suppose we don't run that in CI.

@RalfJung
Copy link
Member Author

RalfJung commented May 1, 2023 via email

@m-ou-se
Copy link
Member

m-ou-se commented May 1, 2023

We could make this 'all(test, miri)' if you prefer. I used cfg(test) since realstd is usually used to avoid duplicate global state and having 2 rust-managed lists of tls dtors seems like that kind of duplicated global state.

Yeah that's true. It might make sense to run the regular std tests with 'duplicated' global state on purpose, because almost the same thing would happen when (dynamically) loading a Rust library into a Rust binary if both the library and the binary have their own copy of std. But I don't think we support that situation well yet, so maybe that's something for later.

@bors r+

@bors
Copy link
Contributor

bors commented May 1, 2023

📌 Commit d5e7ac5 has been approved by m-ou-se

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 1, 2023
@bjorn3
Copy link
Member

bjorn3 commented May 1, 2023

Yeah that's true. It might make sense to run the regular std tests with 'duplicated' global state on purpose, because almost the same thing would happen when (dynamically) loading a Rust library into a Rust binary if both the library and the binary have their own copy of std.

Either both need to dynamically link the same libstd or they need to communicate only through extern "C". For testing libstd however the libstd to be tested borrows the lang items from realstd and communication happens through extern "Rust". Because of those things we need to prevent duplication of global state. When only communicating through extern "C" and having separate sets of lang items the duplicate state doesn't matter as panics can't cross between contexts with separate panic counts as global state.

@m-ou-se
Copy link
Member

m-ou-se commented May 1, 2023

For panics and the panic counter, sure. But for thread_local destructors, it should work "just fine" to keep two independent lists, I think. (Note that no tests failed other than miri.)

matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request May 1, 2023
avoid duplicating TLS state between test std and realstd

This basically re-lands rust-lang#100201 and rust-lang#106638, which got reverted by rust-lang#110861. This works around 2 Miri limitations:
- Miri doesn't support the magic linker section that our Windows TLS support relies on, and instead knows where in std to find the symbol that stores the thread callback.
- For macOS, Miri only supports at most one destructor to be registered per thread.

The 2nd would not be very hard to fix (though the intended destructor order is unclear); the first would be a lot of work to fix. Neither of these is a problem for regular Rust code, but in the std test suite we have essentially 2 copies of the std code and then these both become issues. To avoid that we have the std test crate import the TLS code from the real std instead of having its own copy.

r? `@m-ou-se`
Dylan-DPC added a commit to Dylan-DPC/rust that referenced this pull request May 2, 2023
avoid duplicating TLS state between test std and realstd

This basically re-lands rust-lang#100201 and rust-lang#106638, which got reverted by rust-lang#110861. This works around 2 Miri limitations:
- Miri doesn't support the magic linker section that our Windows TLS support relies on, and instead knows where in std to find the symbol that stores the thread callback.
- For macOS, Miri only supports at most one destructor to be registered per thread.

The 2nd would not be very hard to fix (though the intended destructor order is unclear); the first would be a lot of work to fix. Neither of these is a problem for regular Rust code, but in the std test suite we have essentially 2 copies of the std code and then these both become issues. To avoid that we have the std test crate import the TLS code from the real std instead of having its own copy.

r? ``@m-ou-se``
Dylan-DPC added a commit to Dylan-DPC/rust that referenced this pull request May 3, 2023
avoid duplicating TLS state between test std and realstd

This basically re-lands rust-lang#100201 and rust-lang#106638, which got reverted by rust-lang#110861. This works around 2 Miri limitations:
- Miri doesn't support the magic linker section that our Windows TLS support relies on, and instead knows where in std to find the symbol that stores the thread callback.
- For macOS, Miri only supports at most one destructor to be registered per thread.

The 2nd would not be very hard to fix (though the intended destructor order is unclear); the first would be a lot of work to fix. Neither of these is a problem for regular Rust code, but in the std test suite we have essentially 2 copies of the std code and then these both become issues. To avoid that we have the std test crate import the TLS code from the real std instead of having its own copy.

r? ```@m-ou-se```
Dylan-DPC added a commit to Dylan-DPC/rust that referenced this pull request May 3, 2023
avoid duplicating TLS state between test std and realstd

This basically re-lands rust-lang#100201 and rust-lang#106638, which got reverted by rust-lang#110861. This works around 2 Miri limitations:
- Miri doesn't support the magic linker section that our Windows TLS support relies on, and instead knows where in std to find the symbol that stores the thread callback.
- For macOS, Miri only supports at most one destructor to be registered per thread.

The 2nd would not be very hard to fix (though the intended destructor order is unclear); the first would be a lot of work to fix. Neither of these is a problem for regular Rust code, but in the std test suite we have essentially 2 copies of the std code and then these both become issues. To avoid that we have the std test crate import the TLS code from the real std instead of having its own copy.

r? ````@m-ou-se````
Dylan-DPC added a commit to Dylan-DPC/rust that referenced this pull request May 3, 2023
avoid duplicating TLS state between test std and realstd

This basically re-lands rust-lang#100201 and rust-lang#106638, which got reverted by rust-lang#110861. This works around 2 Miri limitations:
- Miri doesn't support the magic linker section that our Windows TLS support relies on, and instead knows where in std to find the symbol that stores the thread callback.
- For macOS, Miri only supports at most one destructor to be registered per thread.

The 2nd would not be very hard to fix (though the intended destructor order is unclear); the first would be a lot of work to fix. Neither of these is a problem for regular Rust code, but in the std test suite we have essentially 2 copies of the std code and then these both become issues. To avoid that we have the std test crate import the TLS code from the real std instead of having its own copy.

r? `````@m-ou-se`````
bors added a commit to rust-lang-ci/rust that referenced this pull request May 5, 2023
Rollup of 8 pull requests

Successful merges:

 - rust-lang#110946 (avoid duplicating TLS state between test std and realstd)
 - rust-lang#110954 (Reject borrows of projections in ConstProp.)
 - rust-lang#111052 (Fix problems with backtraces in two ui tests.)
 - rust-lang#111132 (cleanup nll generalizer)
 - rust-lang#111173 (Still more encoder cleanups)
 - rust-lang#111187 (bootstrap: add llvm-project/runtimes to the sources)
 - rust-lang#111213 (Fixup "since" dates for `array_tuple_conv` feature)
 - rust-lang#111223 (Use `free-args` consistently in bootstrap)

Failed merges:

r? `@ghost`
`@rustbot` modify labels: rollup
@bors bors merged commit d98e174 into rust-lang:master May 5, 2023
@rustbot rustbot added this to the 1.71.0 milestone May 5, 2023
@RalfJung RalfJung deleted the tls-realstd branch May 11, 2023 06:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-libs Relevant to the library team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants