Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Basic GraalPy Support #3247

Merged
merged 25 commits into from
Mar 25, 2024
Merged

Conversation

timfel
Copy link
Contributor

@timfel timfel commented Jun 16, 2023

With these changes, a GraalPy master build, and using maturin master, I can run the pytests and the examples. I have 3 failures in the pytests still. Please let me know if this is going in the right direction for you.

@timfel timfel force-pushed the timfel/graalpy-support-mk2 branch from 9e69f0e to 9b9ddd6 Compare June 16, 2023 09:01
@adamreichold
Copy link
Member

I think the most important question to clarify here is the status of this integration in the CI. If we are planning on proper support we should include it both for checking PR and for full target checks for the merge queues.

Is this possible and could you sketch an approach here?

@timfel
Copy link
Contributor Author

timfel commented Jun 16, 2023

I think the most important question to clarify here is the status of this integration in the CI. If we are planning on proper support we should include it both for checking PR and for full target checks for the merge queues.
Is this possible and could you sketch an approach here?

There are some changes for this that are only in graalpy master. Our releases go into pyenv, and our next release that will include the relevant changes will be out in September. So one option is to then setup the CI action to add graalpy to the matrix, install it using pyenv and simply run nox -s test-py with it. We can also at any time do nightly builds and we start by adding an action that just uses a well-known nightly build in CI, and we switch to a release build via pyenv once that is out.

@adamreichold
Copy link
Member

install it using pyenv and simply run nox -s test-py with it

Ideally, this would use exactly the same CI jobs we use for CPython and PyPy or is there a technical limitation which makes this unreasonable?

We can also at any time do nightly builds and we start by adding an action that just uses a well-known nightly build in CI, and we switch to a release build via pyenv once that is out.

I think this mainly depends on how urgent this is for you, if you'd like to have this in our main branch before your release, then integrating a nightly build would be a prerequisite. Otherwise, waiting for the next release appears preferable.

@timfel
Copy link
Contributor Author

timfel commented Jun 19, 2023

install it using pyenv and simply run nox -s test-py with it

Ideally, this would use exactly the same CI jobs we use for CPython and PyPy or is there a technical limitation which makes this unreasonable?

We would first have to get graalpy into github actions.

We can also at any time do nightly builds and we start by adding an action that just uses a well-known nightly build in CI, and we switch to a release build via pyenv once that is out.

I think this mainly depends on how urgent this is for you, if you'd like to have this in our main branch before your release, then integrating a nightly build would be a prerequisite. Otherwise, waiting for the next release appears preferable.

We would of course prefer to have this in before our next release, so that we can start running the tests of actual packages that use PyO3 in our CI and fix any additional issues that occur before the September release.

@timfel
Copy link
Contributor Author

timfel commented Jun 27, 2023

We are adding support for graalpy to the setup-python action, then running it in the PyO3 CI should need minimal changes

@timfel
Copy link
Contributor Author

timfel commented Jul 18, 2023

The setup-python action maintainers had a question about GraalPy's licence, I hope they are happy with it soon and will merge, at which point I will rebase this PR and add GraalPy pytest to your CI (actions/setup-python#694)

@timfel timfel force-pushed the timfel/graalpy-support-mk2 branch 2 times, most recently from 70deae9 to b34fef4 Compare September 15, 2023 16:16
@timfel
Copy link
Contributor Author

timfel commented Sep 15, 2023

This now passes all pytests on GraalPy 23.1 with the latest maturin release. Once PyO3/maturin#1773 and actions/setup-python#694 are merged, I can add GraalPy to the CI config.

@davidhewitt
Copy link
Member

Great! Do you have a sense of how close the setup-python PR is to merging? I think we're going to push an 0.20 release imminently, so if this is extremely close we could wait for it. (Otherwise this could land in 0.20.1 later.)

@timfel
Copy link
Contributor Author

timfel commented Sep 16, 2023

Great! Do you have a sense of how close the setup-python PR is to merging? I think we're going to push an 0.20 release imminently, so if this is extremely close we could wait for it. (Otherwise this could land in 0.20.1 later.)

I couldn't say. We've tried to be very quick to do any requested changes, but I assume the github folks are busy and they end up not getting back to us for a week or two each time they ask for something. So don't wait for us.

@davidhewitt
Copy link
Member

Ah okay, frustrating!

@davidhewitt
Copy link
Member

@timfel I added CI-build-full label, so if you fix the fit issues and push again you should find we do a full CI run including the new graalpy job.

@davidhewitt davidhewitt added the CI-no-fail-fast If one job fails, allow the rest to keep testing label Oct 12, 2023
@timfel timfel force-pushed the timfel/graalpy-support-mk2 branch 7 times, most recently from c54d795 to ee0d6ed Compare October 13, 2023 15:22
@timfel
Copy link
Contributor Author

timfel commented Oct 13, 2023

Ok, it's basically working now. The failure on python3.9 is a 500 server error downloading the manylinux image, so I suspect it's transient.

For GraalPy, we need a new release of maturin with this PR: PyO3/maturin#1802. I ran the CI config locally and with an updated maturin it passes that one step that's not running.

@davidhewitt
Copy link
Member

Great, I will try to give this a full review in the coming days! I just got back from EuroRust with a ton of threads to follow up before they go cold as well as some time due for my family, so please forgive me if it takes a little longer than usual 🙏

.github/workflows/build.yml Outdated Show resolved Hide resolved
Copy link
Member

@davidhewitt davidhewitt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! Overall it looks very reasonable to me. Just a couple of style nits and a few questions to understand the future support and testing strategies for this.

@messense, when might you plan to do the next maturin release?

.github/workflows/build.yml Show resolved Hide resolved
.github/workflows/build.yml Show resolved Hide resolved
.github/workflows/build.yml Show resolved Hide resolved
.github/workflows/build.yml Show resolved Hide resolved
README.md Show resolved Hide resolved
pyo3-build-config/src/impl_.rs Show resolved Hide resolved
pyo3-ffi/src/cpython/abstract_.rs Show resolved Hide resolved
pyo3-ffi/src/object.rs Outdated Show resolved Hide resolved
pytests/tests/test_awaitable.py Outdated Show resolved Hide resolved
.github/workflows/build.yml Show resolved Hide resolved
@timfel
Copy link
Contributor Author

timfel commented Mar 25, 2024

Do GraalPy wheels get uploaded to PyPI? Does the GraalPy version (presumably 24.0) go in the tag?

Yes, the graalpy and implemented cpython version go in the tag. pip debug gives wheel tags like graalpy310-graalpy240_310_native-manylinux_2_35_x86_64

@timfel timfel force-pushed the timfel/graalpy-support-mk2 branch from 06e1b63 to 765ae0a Compare March 25, 2024 08:35
@timfel timfel force-pushed the timfel/graalpy-support-mk2 branch 2 times, most recently from bca1c5a to 1eb6b06 Compare March 25, 2024 09:27
@timfel timfel force-pushed the timfel/graalpy-support-mk2 branch from 1eb6b06 to c5ae4c7 Compare March 25, 2024 09:57
@timfel
Copy link
Contributor Author

timfel commented Mar 25, 2024

I'm not entirely sure how the performance differences (both regressions and improvements) are caused by my changes. Maybe those benchmarks are so micro that there is jitter?

@davidhewitt
Copy link
Member

Yep there's a few benchmarks which need to have volatility ignored after I re-enabled them in #3986

I cleared those now 👍

@timfel
Copy link
Contributor Author

timfel commented Mar 25, 2024

Is the failure in "CI / python3.12-x64 ubuntu-latest rust-nightly" something I could have done? It fails in pyo3-macros-backend, I haven't touched that, but I fear I don't understand enough to rule out some interaction with my changes.

@davidhewitt
Copy link
Member

Nightly probably has improved dead code detection. I don't think you need to worry about it here.

Copy link
Member

@davidhewitt davidhewitt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My view here is that we're now in agreement that this is suitable to merge in the current form, and there will be follow ups to tidy up various pieces subject to future work in graalpy.

I think given the size of the effort put in here it makes sense to see this merge now and release in 0.21, both to unblock the ecosystem and so that we can start learning about user experiences with this.

Thanks so much @timfel and @adamreichold for all the time invested here 👍

@davidhewitt davidhewitt added this pull request to the merge queue Mar 25, 2024
Merged via the queue into PyO3:main with commit 9a38e70 Mar 25, 2024
76 of 77 checks passed
@timfel timfel deleted the timfel/graalpy-support-mk2 branch March 26, 2024 08:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI-build-full CI-no-fail-fast If one job fails, allow the rest to keep testing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants