Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rust binding to LFS64 ABI-compat symbols rather than public names on musl libc #2934

Closed
richfelker opened this issue Sep 30, 2022 · 8 comments
Labels
C-bug Category: bug O-musl
Milestone

Comments

@richfelker
Copy link

It's come to our attention as part of removing the legacy LFS64 glibc-ABI-compat symbols that Rust's libc crate is apparently binding to the LFS64 ("large file support") symbol names like fopen64, etc. rather than the standard symbol names. In the musl API/ABI (defined by our public headers) the LFS64 interfaces exist only as macros remapping to the standard function names, and the nonstandard LFS64 names are purely a glibc-ABI-compat mechanism that will not be linkable in the future (i.e. only available to already-dynamic-linked binaries referencing these symbols). This is part of an effort to get rid of the macros (which cause a lot of havok on various C++ projects), which is blocked on having the symbols present, because the prevalence of autoconf link-only tests makes it unsafe to have the symbols linkable without declarations in the headers.

What needs to happen here in order for Rust to reference only the standard libc interface names on musl targets?

@richfelker richfelker added the C-bug Category: bug label Sep 30, 2022
@bossmc
Copy link
Contributor

bossmc commented Sep 30, 2022

Is the position that the xxx64 calls are simply aliases to xxx guaranteed to be true going forwards (i.e. are the xxx functions already effectively the xxx64 versions)? The libc crate can perform the same aliasing when targeting -musl fairly easily. An alternative would be to remove these symbols from the libc crate on the musl targets, but that feels like a bigger breaking change in the rust-world and would be a problem if the ABI ever changed such that xxx64 actually did something other than be aliased to xxx.

Similar question around off64_t and friends, musl currently defines these to their non-64 counterparts, is that also a position we can rely on going forwards?

I'm guessing the answers to the above are all "yes" since I'm assuming that musl's offset/size types have always been 64-bit and the methods have always used the 64-bit underlying syscalls. Hence there being no need to have separate names for compatibility reasons.

Lastly is there a list of affected symbols or is it simply "all the ones that end with 64"?

@richfelker
Copy link
Author

Is the position that the xxx64 calls are simply aliases to xxx guaranteed to be true going forwards

The intent going forward is that the xxx64 ones don't exist. These symbols never existed as part of a programming interface (if you used the xxx64 identifiers in the programming interface, they were macros that evaluated to the non-64 ones to prevent actually emitting unwanted references to those symbols) and were only there for ABI-compat purposes. I wasn't aware until somebody tested removal that the rust libc crate was emitting references to them. Presumably this was not intentional but just fallout of doing what it was doing on glibc.

(i.e. are the xxx functions already effectively the xxx64 versions)?

Yes. off_t is, always has been, and always will be 64-bit in musl. There are no 32-bit-off_t interfaces.

Lastly is there a list of affected symbols or is it simply "all the ones that end with 64"?

That's basically it (modulo readdir64_r where the 64 isn't at the end). But you can find detailed lists below, as:

@bossmc
Copy link
Contributor

bossmc commented Sep 30, 2022

Excellent, I've pushed a proposed change in #2935 that leaves the xxx64 symbols in the API of the libc crate (so this isn't a platform-specific breaking change) but aliases them to the xxx symbol in the libc library.

There's code in the stdlib (and likely elsewhere in the wider world) that explicitly calls the xxx64 functions hence my preference not to break the crate API. Does mean that rust will pretend forever than musl has LFS64 symbols, but it won't actually use them.

@richfelker
Copy link
Author

richfelker commented Sep 30, 2022

Great, that seems like the correct fix. I don't think there's any reason to object to having the aliases in a FFI context like this.

One (I think the main) reason we didn't do it in a pure C context was that the types were wrong: without #define stat64 stat, the LFS64 stat functions that, on the interface level, are supposed to take struct stat64, would have been aliased to functions expecting struct stat, and the stores through the argument pointers would have been violations of type-based aliasing rules.

@bossmc
Copy link
Contributor

bossmc commented Sep 30, 2022

In that (those?) cases is there an actual (ABI) difference between struct stat64 and struct stat (beyond the name)? If there is then the aliasing at FFI level is not going to fly, if they're structurally identical then I think we're good (I'm not going to open the can of worms that is "is unsafe rust allowed to break C's type-based aliasing rules?").

@richfelker
Copy link
Author

No, there's no ABI difference. At an ABI/FFI level everything is fine. It's only not-okay if you consider the C sources/IR cross-translation-unit as having to follow the C TBA rules and you defined struct stat and struct stat64 as two different sructure types with identical members (which we specifically didn't do) rather than making stat64 a macro expanding to stat. Anyway this all ended up being a mess because C++ programs sometimes (rightly) don't like hitting macros like that where they didn't expect them, and that's why we're trying to bury all this and just keep the ABI-compat part. But yeah, that's all C stuff that doesn't affect the rust libc crate or rust API/ABI in any way, IMO. It's just context for why we're trying to do this change on the musl side.

@workingjubilee
Copy link
Member

In that (those?) cases is there an actual (ABI) difference between struct stat64 and struct stat (beyond the name)? If there is then the aliasing at FFI level is not going to fly, if they're structurally identical then I think we're good (I'm not going to open the can of worms that is "is unsafe rust allowed to break C's type-based aliasing rules?").

The answer is "yes, actually", because lifetime analysis replaces it functionally. As a result, Unsafe Rust has historically been allowed to rely on pointer-casting in a way that would make a seasoned C developer blanch, so it would be very hard to change, even amongst things people say are "very hard to change".

orbea added a commit to orbea/gentoo that referenced this issue May 28, 2023
This fixes the build with musl-1.2.4, for those that are still
experiencing issues these steps should resolve them:

1. Downgrade to musl-1.2.3
2. Rebuild dev-lang/rust with these patches
3. Upgrade to musl-1.2.4 again
4. Rebuild rust with USE=system-bootstrap

At this point USE=system-boostrap will be required with >= musl-1.2.4
until uptream merges these patches and updates their boostrap.

This was tested with musl-1.2.3, musl-1.2.4 and glibc-2.37-r3.

Closes: https://bugs.gentoo.org/903607
Upstream-PR: rust-lang/rust#106246
Upstream-Issue: rust-lang/libc#2934
Upstream-PR: rust-lang/libc#2935
Upstream-PR: rust-random/getrandom#326
Upstream-Commit: rust-random/getrandom@7f73e3c
Signed-off-by: orbea <orbea@riseup.net>
bors added a commit that referenced this issue May 29, 2023
Alias all LFS64 symbols to their non-LFS64 counterparts on musl

As per #2934 the LFS64 symbols on musl-libc are simply aliases to the non-LFS64 symbols.  Currently this is done both in the header files (as `#define` entries) and in the library (as aliasing symbols).  There is a desire in musl to drop the ABI compatibility shims (the symbol aliases) - currently the `libc` crate exports the LFS64 symbols by `extern`-ing the compatibility shims which will fail if musl removes them.

This changes the musl build of libc to replicate the aliasing that's in the C header files (with `pub use xxx as xxx64`) so the API from the `libc` crate is unchanged, but the crate is now compatible with the upcoming musl release.

I've also checked and all the LFS64 types (e.g. `off64_t`) are already the same as their non-LFS64 equivalents.

This is an annoying change to test, `libc-test` seems expect to build against `musl 1.1.24` (in fact, does Rust even support `musl 1.2`?  It's not obvious that it does... e.g. see https://github.com/rust-lang/libc/blob/d99c37d97c7a075ffc7f4260f2dd134d1ee3daaa/src/unix/linux_like/linux/musl/mod.rs#L288-L292)
bors added a commit that referenced this issue May 31, 2023
Alias all LFS64 symbols to their non-LFS64 counterparts on musl

As per #2934 the LFS64 symbols on musl-libc are simply aliases to the non-LFS64 symbols.  Currently this is done both in the header files (as `#define` entries) and in the library (as aliasing symbols).  There is a desire in musl to drop the ABI compatibility shims (the symbol aliases) - currently the `libc` crate exports the LFS64 symbols by `extern`-ing the compatibility shims which will fail if musl removes them.

This changes the musl build of libc to replicate the aliasing that's in the C header files (with `pub use xxx as xxx64`) so the API from the `libc` crate is unchanged, but the crate is now compatible with the upcoming musl release.

I've also checked and all the LFS64 types (e.g. `off64_t`) are already the same as their non-LFS64 equivalents.

This is an annoying change to test, `libc-test` seems expect to build against `musl 1.1.24` (in fact, does Rust even support `musl 1.2`?  It's not obvious that it does... e.g. see https://github.com/rust-lang/libc/blob/d99c37d97c7a075ffc7f4260f2dd134d1ee3daaa/src/unix/linux_like/linux/musl/mod.rs#L288-L292)
bors added a commit that referenced this issue May 31, 2023
Alias all LFS64 symbols to their non-LFS64 counterparts on musl

As per #2934 the LFS64 symbols on musl-libc are simply aliases to the non-LFS64 symbols.  Currently this is done both in the header files (as `#define` entries) and in the library (as aliasing symbols).  There is a desire in musl to drop the ABI compatibility shims (the symbol aliases) - currently the `libc` crate exports the LFS64 symbols by `extern`-ing the compatibility shims which will fail if musl removes them.

This changes the musl build of libc to replicate the aliasing that's in the C header files (with `pub use xxx as xxx64`) so the API from the `libc` crate is unchanged, but the crate is now compatible with the upcoming musl release.

I've also checked and all the LFS64 types (e.g. `off64_t`) are already the same as their non-LFS64 equivalents.

This is an annoying change to test, `libc-test` seems expect to build against `musl 1.1.24` (in fact, does Rust even support `musl 1.2`?  It's not obvious that it does... e.g. see https://github.com/rust-lang/libc/blob/d99c37d97c7a075ffc7f4260f2dd134d1ee3daaa/src/unix/linux_like/linux/musl/mod.rs#L288-L292)
@bossmc
Copy link
Contributor

bossmc commented May 31, 2023

Fixed by #2935

orbea added a commit to orbea/gentoo that referenced this issue Jun 1, 2023
This fixes the build with musl-1.2.4, for those that are still
experiencing issues these steps should resolve them:

1. Downgrade to musl-1.2.3
2. Rebuild dev-lang/rust with these patches
3. Upgrade to musl-1.2.4 again
4. Rebuild rust with USE=system-bootstrap

At this point USE=system-boostrap will be required with >= musl-1.2.4
until uptream merges these patches and updates their boostrap.

This was tested with musl-1.2.3, musl-1.2.4 and glibc-2.37-r3.

Closes: https://bugs.gentoo.org/903607
Upstream-PR: rust-lang/rust#106246
Upstream-Issue: rust-lang/libc#2934
Upstream-PR: rust-lang/libc#2935
Upstream-PR: rust-random/getrandom#326
Upstream-Commit: rust-random/getrandom@7f73e3c
Signed-off-by: orbea <orbea@riseup.net>
orbea added a commit to orbea/gentoo that referenced this issue Jun 5, 2023
This fixes the build with musl-1.2.4, for those that are still
experiencing issues these steps should resolve them:

1. Downgrade to musl-1.2.3
2. Rebuild dev-lang/rust with these patches
3. Upgrade to musl-1.2.4 again
4. Rebuild rust with USE=system-bootstrap

At this point USE=system-boostrap will be required with >= musl-1.2.4
until uptream merges these patches and updates their boostrap.

This was tested with musl-1.2.3, musl-1.2.4 and glibc-2.37-r3.

Closes: https://bugs.gentoo.org/903607
Upstream-PR: rust-lang/rust#106246
Upstream-Issue: rust-lang/libc#2934
Upstream-PR: rust-lang/libc#2935
Upstream-PR: rust-random/getrandom#326
Upstream-Commit: rust-random/getrandom@7f73e3c
Signed-off-by: orbea <orbea@riseup.net>
orbea added a commit to orbea/gentoo that referenced this issue Jun 5, 2023
This fixes the build with musl-1.2.4, for those that are still
experiencing issues these steps should resolve them:

1. Downgrade to musl-1.2.3
2. Rebuild dev-lang/rust with these patches
3. Upgrade to musl-1.2.4 again
4. Rebuild rust with USE=system-bootstrap

At this point USE=system-boostrap will be required with >= musl-1.2.4
until uptream merges these patches and updates their boostrap.

This was tested with musl-1.2.3, musl-1.2.4 and glibc-2.37-r3.

Closes: https://bugs.gentoo.org/903607
Upstream-PR: rust-lang/rust#106246
Upstream-Issue: rust-lang/libc#2934
Upstream-PR: rust-lang/libc#2935
Upstream-PR: rust-random/getrandom#326
Upstream-Commit: rust-random/getrandom@7f73e3c
Signed-off-by: orbea <orbea@riseup.net>
orbea added a commit to orbea/gentoo that referenced this issue Jun 8, 2023
This fixes the build with musl-1.2.4, for those that are still
experiencing issues these steps should resolve them:

1. Downgrade to musl-1.2.3
2. Rebuild dev-lang/rust with these patches
3. Upgrade to musl-1.2.4 again
4. Rebuild rust with USE=system-bootstrap

At this point USE=system-boostrap will be required with >= musl-1.2.4
until uptream merges these patches and updates their boostrap.

This was tested with musl-1.2.3, musl-1.2.4 and glibc-2.37-r3.

Closes: https://bugs.gentoo.org/903607
Upstream-PR: rust-lang/rust#106246
Upstream-Issue: rust-lang/libc#2934
Upstream-PR: rust-lang/libc#2935
Upstream-PR: rust-random/getrandom#326
Upstream-Commit: rust-random/getrandom@7f73e3c
Signed-off-by: orbea <orbea@riseup.net>
orbea added a commit to orbea/gentoo that referenced this issue Jun 8, 2023
This fixes the build with musl-1.2.4, for those that are still
experiencing issues these steps should resolve them:

1. Downgrade to musl-1.2.3
2. Rebuild dev-lang/rust with these patches
3. Upgrade to musl-1.2.4 again
4. Rebuild rust with USE=system-bootstrap

At this point USE=system-boostrap will be required with >= musl-1.2.4
until uptream merges these patches and updates their boostrap.

This was tested with musl-1.2.3, musl-1.2.4 and glibc-2.37-r3.

Closes: https://bugs.gentoo.org/903607
Upstream-PR: rust-lang/rust#106246
Upstream-Issue: rust-lang/libc#2934
Upstream-PR: rust-lang/libc#2935
Upstream-PR: rust-random/getrandom#326
Upstream-Commit: rust-random/getrandom@7f73e3c
Signed-off-by: orbea <orbea@riseup.net>
orbea added a commit to orbea/gentoo that referenced this issue Jun 9, 2023
This fixes the build with musl-1.2.4, for those that are still
experiencing issues these steps should resolve them:

1. Downgrade to musl-1.2.3
2. Rebuild dev-lang/rust with these patches
3. Upgrade to musl-1.2.4 again
4. Rebuild rust with USE=system-bootstrap

At this point USE=system-boostrap will be required with >= musl-1.2.4
until uptream merges these patches and updates their boostrap.

This was tested with musl-1.2.3, musl-1.2.4 and glibc-2.37-r3.

Closes: https://bugs.gentoo.org/903607
Upstream-PR: rust-lang/rust#106246
Upstream-Issue: rust-lang/libc#2934
Upstream-PR: rust-lang/libc#2935
Upstream-PR: rust-random/getrandom#326
Upstream-Commit: rust-random/getrandom@7f73e3c
Signed-off-by: orbea <orbea@riseup.net>
orbea added a commit to orbea/gentoo that referenced this issue Jun 29, 2023
This fixes the build with musl-1.2.4, for those that are still
experiencing issues these steps should resolve them:

1. Downgrade to musl-1.2.3
2. Rebuild dev-lang/rust with these patches
3. Upgrade to musl-1.2.4 again
4. Rebuild rust with USE=system-bootstrap

At this point USE=system-boostrap will be required with >= musl-1.2.4
until uptream merges these patches and updates their boostrap.

This was tested with musl-1.2.3, musl-1.2.4 and glibc-2.37-r3.

Closes: https://bugs.gentoo.org/903607
Upstream-PR: rust-lang/rust#106246
Upstream-Issue: rust-lang/libc#2934
Upstream-PR: rust-lang/libc#2935
Upstream-PR: rust-random/getrandom#326
Upstream-Commit: rust-random/getrandom@7f73e3c
Signed-off-by: orbea <orbea@riseup.net>
orbea added a commit to orbea/gentoo that referenced this issue Jul 16, 2023
This fixes the build with musl-1.2.4, for those that are still
experiencing issues these steps should resolve them:

1. Downgrade to musl-1.2.3
2. Rebuild dev-lang/rust with these patches
3. Upgrade to musl-1.2.4 again
4. Rebuild rust with USE=system-bootstrap

This was tested with musl-1.2.3, musl-1.2.4 and glibc-2.37-r3.

Closes: https://bugs.gentoo.org/903607
Upstream-PR: rust-lang/rust#106246
Upstream-Issue: rust-lang/libc#2934
Upstream-PR: rust-lang/libc#2935
Upstream-PR: rust-random/getrandom#326
Upstream-Commit: rust-random/getrandom@7f73e3c
Signed-off-by: orbea <orbea@riseup.net>
orbea added a commit to orbea/gentoo that referenced this issue Jul 17, 2023
This fixes the build with musl-1.2.4, for those that are still
experiencing issues these steps should resolve them:

1. Downgrade to musl-1.2.3
2. Rebuild dev-lang/rust with these patches
3. Upgrade to musl-1.2.4 again
4. Rebuild rust with USE=system-bootstrap

This was tested with musl-1.2.3, musl-1.2.4 and glibc-2.37-r3.

Closes: https://bugs.gentoo.org/903607
Upstream-PR: rust-lang/rust#106246
Upstream-Issue: rust-lang/libc#2934
Upstream-PR: rust-lang/libc#2935
Upstream-PR: rust-random/getrandom#326
Upstream-Commit: rust-random/getrandom@7f73e3c
Signed-off-by: orbea <orbea@riseup.net>
orbea added a commit to orbea/gentoo that referenced this issue Jul 20, 2023
This fixes the build with musl-1.2.4, may need -system-bootstrap after
updating musl.

Closes: https://bugs.gentoo.org/903607
Upstream-PR: rust-lang/rust#106246
Upstream-Issue: rust-lang/libc#2934
Upstream-PR: rust-lang/libc#2935
Upstream-Commit: rust-lang/libc@1e8c55c
Upstream-PR: rust-random/getrandom#326
Upstream-Commit: rust-random/getrandom@7f73e3c
Signed-off-by: orbea <orbea@riseup.net>
orbea added a commit to orbea/gentoo that referenced this issue Jul 21, 2023
This fixes the build with musl-1.2.4, may need -system-bootstrap after
updating musl.

Closes: https://bugs.gentoo.org/903607
Upstream-PR: rust-lang/rust#106246
Upstream-Issue: rust-lang/libc#2934
Upstream-PR: rust-lang/libc#2935
Upstream-Commit: rust-lang/libc@1e8c55c
Upstream-PR: rust-random/getrandom#326
Upstream-Commit: rust-random/getrandom@7f73e3c
Signed-off-by: orbea <orbea@riseup.net>
orbea added a commit to orbea/gentoo that referenced this issue Aug 7, 2023
This fixes the build with musl-1.2.4, may need -system-bootstrap after
updating musl.

Closes: https://bugs.gentoo.org/903607
Upstream-PR: rust-lang/rust#106246
Upstream-Issue: rust-lang/libc#2934
Upstream-PR: rust-lang/libc#2935
Upstream-Commit: rust-lang/libc@1e8c55c
Upstream-PR: rust-random/getrandom#326
Upstream-Commit: rust-random/getrandom@7f73e3c
Signed-off-by: orbea <orbea@riseup.net>
@Amanieu Amanieu closed this as completed Mar 20, 2024
@tgross35 tgross35 added this to the 1.0 milestone Aug 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-bug Category: bug O-musl
Projects
None yet
Development

No branches or pull requests

5 participants