Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-95855: Refactor platform triplet detection code, add detection for MIPS soft float and musl libc #107221

Merged
merged 12 commits into from
Aug 24, 2023

Conversation

jefferyto
Copy link
Contributor

@jefferyto jefferyto commented Jul 25, 2023

Most of the ideas in this come from #96001 so @tiran deserves the credit. But since the platform triplet detection code in configure.ac has changed since #96001, and because that PR is still in an unfinished state, I thought it would make more sense to reimplement it (as well as split it into commits that explain the major steps).

@erlend-aasland
Copy link
Contributor

Thanks! I liked Christian's initial efforts in #96001; thanks for picking this up.

Instead of the grep exercises; what do you think of the printf approach taken in the CMake-for-CPython project: https://github.com/python-cmake-buildsystem/python-cmake-buildsystem/blob/master/cmake/platform.c

@jefferyto
Copy link
Contributor Author

Instead of the grep exercises; what do you think of the printf approach taken in the CMake-for-CPython project: https://github.com/python-cmake-buildsystem/python-cmake-buildsystem/blob/master/cmake/platform.c

That would require fully compiling the program (the current approach only requires the preprocessing step) and running it to get the output, which would not be feasible when cross-compiling.

My preference would be something similar to config.guess, where we output the triplet with a prefix like PLATFORM_TRIPLET= then grep for the prefix specifically (but without evaling the output). Would this be acceptable?

@erlend-aasland
Copy link
Contributor

That would require fully compiling the program (the current approach only requires the preprocessing step) and running it to get the output, which would not be feasible when cross-compiling.

That's a good point.

My preference would be something similar to config.guess, where we output the triplet with a prefix like PLATFORM_TRIPLET= then grep for the prefix specifically (but without evaling the output). Would this be acceptable?

Yeah, I would prefer something like that.

@jefferyto
Copy link
Contributor Author

Yeah, I would prefer something like that.

Updated 👍

configure.ac Outdated Show resolved Hide resolved
@erlend-aasland erlend-aasland enabled auto-merge (squash) July 27, 2023 21:05
@erlend-aasland erlend-aasland disabled auto-merge July 27, 2023 21:07
@erlend-aasland erlend-aasland added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jul 27, 2023
@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @erlend-aasland for commit 7fb142c 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jul 27, 2023
@jefferyto
Copy link
Contributor Author

Not sure if the buildbot errors are related to this change?

@erlend-aasland
Copy link
Contributor

Not sure if the buildbot errors are related to this change?

Yeah, unfortunately we'll have to examine each bot. I'll see if I can find time later. Feel free to take a look yourself :)

@jefferyto
Copy link
Contributor Author

I have looked at every error log - while the errors do not look related to this change to me, I'm not familiar enough with the tests being run to say for certain.

@jefferyto
Copy link
Contributor Author

Is there anything else I can do to help resolve the buildbot errors (perhaps update/rebase the branch)?

@erlend-aasland
Copy link
Contributor

erlend-aasland commented Aug 9, 2023

I did a fairly quick inspection of the buildbot run. Feel free to go through them as well (or even better a different subset).

Buildbot old triplet new triplet
AMD64 Arch Linux TraceRefs PR x86_64-linux-gnu  x86_64-linux-gnu
AMD64 Debian root PR  x86_64-linux-gnu x86_64-linux-gnu
AMD64 Fedora Stable Clang Installed PR x86_64-linux-gnu x86_64-linux-gnu
AMD64 RHEL7 PR x86_64-linux-gnu x86_64-linux-gnu
AMD64 RHEL8 PR x86_64-linux-gnu x86_64-linux-gnu
AMD64 Ubuntu Shared PR x86_64-linux-gnu x86_64-linux-gnu
ARM Raspbian PR arm-linux-gnueabihf arm-linux-gnueabihf
ARM64 macOS PR aarch64-apple-darwin22.5.0/clang aarch64-apple-darwin22.5.0/clang
PPC64 Fedora PR powerpc-linux-gnu ❌ powerpc64-linux-gnu
PPC64LE Fedora Stable Clang Installed PR powerpc64le-linux-gnu ❌ powerpc-linux-gnu
PPC64LE RHEL7 PR powerpc64le-linux-gnu ❌ powerpc-linux-gnu
PPC64LE RHEL8 PR  powerpc64le-linux-gnu ❌ powerpc-linux-gnu
aarch64 Debian Clang LTO + PGO PR aarch64-linux-gnu aarch64-linux-gnu
aarch64 Fedora Stable Clang Installed PR aarch64-linux-gnu aarch64-linux-gnu
aarch64 RHEL8 PR aarch64-linux-gnu aarch64-linux-gnu
s390x Debian PR s390x-linux-gnu s390x-linux-gnu
s390x Fedora Clang Installed PR s390x-linux-gnu s390x-linux-gnu
s390x RHEL7 PR s390x-linux-gnu  s390x-linux-gnu
s390x RHEL8 PR s390x-linux-gnu s390x-linux-gnu
s390x SLES PR s390x-linux-gnu s390x-linux-gnu
wasm32-emscripten browser (dynamic linking, no tests) PR wasm32-emscripten wasm32-emscripten
wasm32-emscripten node (dynamic linking) PR wasm32-emscripten wasm32-emscripten
wasm32-emscripten node (pthreads) PR wasm32-emscripten wasm32-emscripten
wasm32-wasi PR wasm32-wasi wasm32-wasi
x86 Gentoo Installed with X PR i686-pc-linux-gnu ❌ i386-linux-gnu
x86 Gentoo Non-Debug with X PR i386-linux-gnu i386-linux-gnu
x86-64 macOS PR x86_64-apple-darwin21.6.0/clang x86_64-apple-darwin21.6.0/clang

As you see, there are some changes.

Also, I noticed this configure warning in the Gentoo Non-Debug buildbot (despite it getting the i386-linux-gnu triplet):

configure: WARNING: i686-pc-linux-gnu/gcc is not supported

@jefferyto
Copy link
Contributor Author

  • PPC64LE * (was powerpc64le-linux-gnu, now powerpc-linux-gnu)
    This should be fixed
  • PPC64 Fedora PR (was powerpc-linux-gnu, now powerpc64-linux-gnu) (build)
    I think the results are switched (was powerpc64-linux-gnu, now powerpc-linux-gnu) - the change is the same issue as with PPC64LE (and should also be fixed)
  • x86 Gentoo Installed with X PR (was i686-pc-linux-gnu, now i386-linux-gnu) (build)
    I checked several previous builds and they all found i386-linux-gnu as the platform triplet (with i686-pc-linux-gnu as the build/host system type)

@AA-Turner AA-Turner added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 11, 2023
@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @AA-Turner for commit cbaf416 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 11, 2023
@AA-Turner
Copy link
Member

@jefferyto I've kicked off another buildbot run for your latest changes.

A

@jefferyto
Copy link
Contributor Author

@AA-Turner thanks!

@erlend-aasland
Copy link
Contributor

Thanks, Adam! @jefferyto, can you do an audit of the buildbot run when it completes? I won't be able to return to this until probably late next week.

@jefferyto
Copy link
Contributor Author

Here are all the results:

Buildbot Old triplet New triplet
AMD64 Arch Linux Asan Debug PR x86_64-linux-gnu x86_64-linux-gnu
AMD64 Arch Linux Asan PR x86_64-linux-gnu x86_64-linux-gnu
AMD64 Arch Linux Perf PR x86_64-linux-gnu x86_64-linux-gnu
AMD64 Arch Linux TraceRefs PR x86_64-linux-gnu x86_64-linux-gnu
AMD64 Arch Linux Usan PR x86_64-linux-gnu x86_64-linux-gnu
AMD64 Debian PGO PR x86_64-linux-gnu x86_64-linux-gnu
AMD64 Debian root PR x86_64-linux-gnu x86_64-linux-gnu
AMD64 Fedora Stable Clang Installed PR x86_64-linux-gnu x86_64-linux-gnu
AMD64 Fedora Stable Clang PR x86_64-linux-gnu x86_64-linux-gnu
AMD64 Fedora Stable LTO + PGO PR x86_64-linux-gnu x86_64-linux-gnu
AMD64 Fedora Stable LTO PR x86_64-linux-gnu x86_64-linux-gnu
AMD64 Fedora Stable PR x86_64-linux-gnu x86_64-linux-gnu
AMD64 Fedora Stable Refleaks PR x86_64-linux-gnu x86_64-linux-gnu
AMD64 RHEL7 LTO + PGO PR x86_64-linux-gnu x86_64-linux-gnu
AMD64 RHEL7 LTO PR x86_64-linux-gnu x86_64-linux-gnu
AMD64 RHEL7 PR x86_64-linux-gnu x86_64-linux-gnu
AMD64 RHEL7 Refleaks PR x86_64-linux-gnu x86_64-linux-gnu
AMD64 RHEL8 FIPS Only Blake2 Builtin Hash PR x86_64-linux-gnu x86_64-linux-gnu
AMD64 RHEL8 LTO + PGO PR x86_64-linux-gnu x86_64-linux-gnu
AMD64 RHEL8 LTO PR x86_64-linux-gnu x86_64-linux-gnu
AMD64 RHEL8 PR x86_64-linux-gnu x86_64-linux-gnu
AMD64 RHEL8 Refleaks PR x86_64-linux-gnu x86_64-linux-gnu
AMD64 Ubuntu Shared PR x86_64-linux-gnu x86_64-linux-gnu
ARM Raspbian PR arm-linux-gnueabihf arm-linux-gnueabihf
ARM64 macOS PR darwin darwin
PPC64 Fedora PR powerpc64-linux-gnu powerpc64-linux-gnu
PPC64LE Fedora Stable Clang Installed PR powerpc64le-linux-gnu powerpc64le-linux-gnu
PPC64LE Fedora Stable Clang PR powerpc64le-linux-gnu powerpc64le-linux-gnu
PPC64LE Fedora Stable LTO + PGO PR powerpc64le-linux-gnu powerpc64le-linux-gnu
PPC64LE Fedora Stable LTO PR powerpc64le-linux-gnu powerpc64le-linux-gnu
PPC64LE Fedora Stable PR powerpc64le-linux-gnu powerpc64le-linux-gnu
PPC64LE Fedora Stable Refleaks PR powerpc64le-linux-gnu powerpc64le-linux-gnu
PPC64LE RHEL7 LTO + PGO PR powerpc64le-linux-gnu powerpc64le-linux-gnu
PPC64LE RHEL7 LTO PR powerpc64le-linux-gnu powerpc64le-linux-gnu
PPC64LE RHEL7 PR powerpc64le-linux-gnu powerpc64le-linux-gnu
PPC64LE RHEL7 Refleaks PR powerpc64le-linux-gnu powerpc64le-linux-gnu
PPC64LE RHEL8 LTO + PGO PR powerpc64le-linux-gnu powerpc64le-linux-gnu
PPC64LE RHEL8 LTO PR powerpc64le-linux-gnu powerpc64le-linux-gnu
PPC64LE RHEL8 PR powerpc64le-linux-gnu powerpc64le-linux-gnu
PPC64LE RHEL8 Refleaks PR powerpc64le-linux-gnu powerpc64le-linux-gnu
aarch64 Debian Clang LTO + PGO PR aarch64-linux-gnu aarch64-linux-gnu
aarch64 Fedora Stable Clang Installed PR aarch64-linux-gnu aarch64-linux-gnu
aarch64 Fedora Stable Clang PR aarch64-linux-gnu aarch64-linux-gnu
aarch64 Fedora Stable LTO + PGO PR aarch64-linux-gnu aarch64-linux-gnu
aarch64 Fedora Stable LTO PR aarch64-linux-gnu aarch64-linux-gnu
aarch64 Fedora Stable PR aarch64-linux-gnu aarch64-linux-gnu
aarch64 Fedora Stable Refleaks PR aarch64-linux-gnu aarch64-linux-gnu
aarch64 RHEL8 LTO + PGO PR aarch64-linux-gnu aarch64-linux-gnu
aarch64 RHEL8 LTO PR aarch64-linux-gnu aarch64-linux-gnu
aarch64 RHEL8 PR aarch64-linux-gnu aarch64-linux-gnu
aarch64 RHEL8 Refleaks PR aarch64-linux-gnu aarch64-linux-gnu
s390x Debian PR s390x-linux-gnu s390x-linux-gnu
s390x Fedora Clang Installed PR s390x-linux-gnu s390x-linux-gnu
s390x Fedora Clang PR s390x-linux-gnu s390x-linux-gnu
s390x Fedora LTO + PGO PR s390x-linux-gnu s390x-linux-gnu
s390x Fedora LTO PR s390x-linux-gnu s390x-linux-gnu
s390x Fedora PR s390x-linux-gnu s390x-linux-gnu
s390x Fedora Refleaks PR s390x-linux-gnu s390x-linux-gnu
s390x RHEL7 LTO + PGO PR s390x-linux-gnu s390x-linux-gnu
s390x RHEL7 LTO PR s390x-linux-gnu s390x-linux-gnu
s390x RHEL7 PR s390x-linux-gnu s390x-linux-gnu
s390x RHEL7 Refleaks PR s390x-linux-gnu s390x-linux-gnu
s390x RHEL8 LTO + PGO PR s390x-linux-gnu s390x-linux-gnu
s390x RHEL8 LTO PR s390x-linux-gnu s390x-linux-gnu
s390x RHEL8 PR s390x-linux-gnu s390x-linux-gnu
s390x RHEL8 Refleaks PR s390x-linux-gnu s390x-linux-gnu
s390x SLES PR s390x-linux-gnu s390x-linux-gnu
wasm32-emscripten browser (dynamic linking, no tests) PR wasm32-emscripten wasm32-emscripten
wasm32-emscripten node (dynamic linking) PR wasm32-emscripten wasm32-emscripten
wasm32-emscripten node (pthreads) PR wasm32-emscripten wasm32-emscripten
wasm32-wasi PR wasm32-wasi wasm32-wasi
x86 Gentoo Installed with X PR i386-linux-gnu i386-linux-gnu
x86 Gentoo Non-Debug with X PR i386-linux-gnu i386-linux-gnu
x86-64 macOS PR darwin darwin

I found no regressions, though I suppose this will need to be independently verified.

@erlend-aasland
Copy link
Contributor

Wow, good job! Thank you so much. I'll try to find time to verify it the coming week.

@jefferyto
Copy link
Contributor Author

@erlend-aasland ping 🙏

@erlend-aasland
Copy link
Contributor

@corona10, do you want to have a look?

@erlend-aasland
Copy link
Contributor

@tiran, if you are around; feel free to have a look :)

@corona10
Copy link
Member

Okay I will take a look but please wait until this weekend ;)

@erlend-aasland
Copy link
Contributor

Okay I will take a look but please wait until this weekend ;)

Sure, there is no hurry :)

@erlend-aasland
Copy link
Contributor

cc. @indygreg

Copy link
Member

@corona10 corona10 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@erlend-aasland erlend-aasland merged commit c163d7f into python:main Aug 24, 2023
@erlend-aasland
Copy link
Contributor

Thanks @jefferyto, Christian, and Dong-hee!

@jefferyto
Copy link
Contributor Author

Thanks @erlend-aasland for your help as well!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants