[Python][Packaging] Support Python 3.13 and upload wheels

[jorisvandenbossche]

Python 3.13 is currently in a prerelease status (a first release candidate was released today: https://discuss.python.org/t/python-3-13-0-release-candidate-1-released/59703) and is expected to have a final release in October this year (https://devguide.python.org/versions).

We should ensure we are fully ready to support Python 3.13 for the PyArrow 18 release that will happen around the same time (or potentially even sooner if we would have a 17.0.x bug fix release).

#42034 already fixed a build issue with Python 3.13, and on the short term I think we should try to have 1) a CI build using Python 3.13 to ensure this keeps working, and 2) set up a wheel build so we can upload nightly wheels for Python 3.13 for downstream testing.

(equivalent PR for Python 3.12: #37901)

---

• Start building nightly wheels for Python 3.13: GH-43519: [Python] Set up wheel building for Python 3.13 #43539
• Also add py 3.13 conda build once that is supported in conda-forge (

  arrow/dev/tasks/tasks.yml

  Lines 1200 to 1201 in 712cfe6

  	{% for python_version in ["3.8", "3.9", "3.10", "3.11", "3.12"] %}
  	test-conda-python-{{ python_version }}:

  )
• Update requirements-wheel-build.txt and requirements-wheel-test.txt to install released versions
• Update the places where we currently use "3.13-rc" or "rc1" or "3.13.0-rc1" (dev/tasks/python-wheels/github.linux.yml, ci/scripts/install_python.sh, the windows docker files)
• Enable GCS testing once google-testbench (grpcio) can be installed (update install_gcs_testbench.sh) (or address GH-38183 and install testbench with independent python version)
  • This has been resolved by GH-38183: [CI][Python] Use pipx to install GCS testbench #43852 and GH-43883: [CI] Remove Python version guard when installing GCS testbench #43884

[raulcd]

I was planning, if necessary, to delay the feature freeze for 18.0.0 for a week to have Python 3.13 wheels.

[jorisvandenbossche]

We can actually make wheels right now (and we should ideally produce nightly wheels well before the scheduled release of 18.0), because Python 3.13 is ABI stable now, and wheels built with the current Python 3.13 will work with the final 3.13 releases.

(so there is certainly no need to delay 18.0.0 AFAIK)

[jorisvandenbossche]

Follow-ups after #43539:

• Also add py 3.13 conda build once that is supported in conda-forge (

  arrow/dev/tasks/tasks.yml

  Lines 1200 to 1201 in 712cfe6

  	{% for python_version in ["3.8", "3.9", "3.10", "3.11", "3.12"] %}
  	test-conda-python-{{ python_version }}:

  )
• Update requirements-wheel-build.txt and requirements-wheel-test.txt to install released versions
• Update the places where we currently use "3.13-rc" or "rc1" or "3.13.0-rc1" (dev/tasks/python-wheels/github.linux.yml, ci/scripts/install_python.sh, the windows docker files)
• Enable GCS testing once google-testbench (grpcio) can be installed (update install_gcs_testbench.sh)

EDIT: added those to the top post

[jarrodmillman]

Python 3.13 is scheduled for Monday (10/7). Would it be possible to make a 18.0.0b0 or 18.0.0rc0 release on PyPI in the near future? I know you have wheels on https://anaconda.org/scientific-python-nightly-wheels/pyarrow/files, but having a pre-release on PyPI would make it easier for projects that aren't aware of the "nightly" wheels.

[wizard86pz]

I've installed today the v3.13 of Python, and it's affected by this pyarrow issue.
Should I wait for a new release of python 3.13 to have the fix?

[pitrou]

According to https://discuss.python.org/t/launch-day-availability-of-python-3-13-in-conda-forge/67124/1, 3.13 is now available in conda-forge.

[pitrou]

I've installed today the v3.13 of Python, and it's affected by this pyarrow issue.
Should I wait for a new release of python 3.13 to have the fix?

Yes, please wait for 18.0.0 which will provide binary wheels for 3.13.

[raulcd]

The only missing task if to update to the final released version of Python (we are just waiting for chocolatey or to migrate out of chocolatey).
I am closing this as done as I am close to create the first RC for 18.0.0.
Feel free to open a subtask for:

• Update the places where we currently use "3.13-rc" or "rc1" or "3.13.0-rc1" (dev/tasks/python-wheels/github.linux.yml, ci/scripts/install_python.sh, the windows docker files)

[bersbersbers]

we are just waiting for chocolatey or to migrate out of chocolatey

Wow - I was not aware it takes chocolatey more than a week to update the latest stable Python version. Considering that there are packages whose 3.13 support depends on arrow's, maybe you want to consider scoop in the future, which updated a week ago: ScoopInstaller/Main@ec70b27

[pllim]

Given this issue is closed, where should I subscribe to make sure I am aware when pyarrow that is compatible is Python 3.13 is actually released? Thanks!

[edgarrmondragon]

Given this issue is closed, where should I subscribe to make sure I am aware when pyarrow that is compatible is Python 3.13 is actually released? Thanks!

@pllim you can subscribe to the PyPI feed: https://pypi.org/rss/project/pyarrow/releases.xml