Consider cibuildwheel for building wheels?

[jorisvandenbossche]

Disclaimer: I haven't been involved recently in maintaining our wheel building infrastructure, so I don't know how well it is working or not. But, I recently had a good experience with cibuildwheel with a relatively complex package, so therefore thought to bring it up.

Cibuildwheel (https://cibuildwheel.readthedocs.io/en/stable/) is a package helping to build wheels for all different platforms and architectures / python versions on your CI of choice (they support most of the common CI services). And if using GitHub Actions, you can build all wheels with a single CI service.

Scikit-learn started to use it last year (scikit-learn/scikit-learn#17921). You can see their github workflow as an example: https://github.com/scikit-learn/scikit-learn/blob/main/.github/workflows/wheels.yml (this single workflow config includes building all the wheels, building the sdist, and uploading them to the anaconda wheelhouse for nightly builds).
(cc @ogrisel @thomasjpfan in case you have feedback on how the switch worked out)

And it seems numpy also started considering this: numpy/numpy#20102
Another example mentioned in the dev meeting: https://github.com/googleapis/python-crc32c/blob/b1a8d14bf0a1b4f1faa0f9fdd4307e15699daeff/.github/workflows/python-publish.yml#L245

Some potential advantages:

• Easier to maintain (no issues with Travis credits, be able to do everything with a single CI, easier to support all architectures and new python versions, no need to synchronize with a different repo, ..)
• Having it in the repo here also allows triggering the builds on a PR on demand, if a PR would influence wheel building (I wouldn't enable the workflow by default for every PR).

cc @simonjayhawkins @lithomas1 since you are the ones mostly working on https://github.com/MacPython/pandas-wheels, it's of course mostly up to you to decide (and even if the end result might be better, every migration also has a cost).

[thomasjpfan]

I think migrating to cibuildwheel was a smooth experience. At scikit-learn, we still need to keep travis-ci for the arm builds (also using cibuildwheel). Triggering wheel builds in PRs is very convenient when cutting a release. There is no need to sync between two repos when debugging issues with wheel building.

[lithomas1]

+1. Our nightly builds just silently failed on the other repo, and I think moving builds over here would increase visibility of build issues. I would be fine doing this if @simonjayhawkins is also fine with it.

[ogrisel]

Le mar. 19 oct. 2021 à 15:40, Thomas J. Fan ***@***.***> a écrit :

I think migrating to cibuildwheel was a smooth experience. At scikit-learn, we still need to keep travis-ci for the arm builds (also using cibuildwheel). Triggering wheel builds in PRs is very convenient when cutting a release. There is no need to sync between two repos when debugging issues with wheel building.

100% agree.

…

-- Olivier

[tgross35]

Has there been any further thought on this? I opened MacPython/pandas-wheels#176 (asking for musl wheels) before realizing this issue exists. Cibuildwheel makes it dead easy.

If the goal is to more or less copy numpy's cibuildwheel organization (they are basically fully moved over to my knowledge) then this change should be pretty straightforward.

[jreback]

@tgross35 we would certainly be happy to try this approach

if u r willing please push up a PR

[lithomas1]

Hi all. I'm actually currently helping out numpy with their cibuildwheel migration.

I would wait until numpy releases successfully with their new cibuildwheel config before switching us to cibuildwheel.

PR is welcome, but just be aware that it may be a while before it gets merged.

[jorisvandenbossche]

@lithomas1 can you clarify a bit? (I was assuming that numpy is already using cibuildwheel, but it's not yet fully working on their side?)

[lithomas1]

It's basically done on the numpy side, and numpy nightlies are currently being built with cibuildwheel. The one last thing we're still working on is building the sdist on CI(numpy/numpy#21207).

I was just suggesting waiting to port to our side because sometimes there are issues related to the wheel builders that aren't caught until a release/pre-release.

[jorisvandenbossche]

Sounds good, thanks for the additional details @lithomas1 !

[sanmai-NL]

@lithomas1 How come there still aren't musllinux wheels on PyPI as yet?

[lithomas1]

We need testing for musl first.

[sanmai-NL]

@lithomas1 Is

pandas/.github/workflows/unit-tests.yml

Lines 243 to 283 in 97c4ef6

	Linux-Musl:
	runs-on: ubuntu-22.04
	container:
	image: quay.io/pypa/musllinux_1_1_x86_64
	steps:
	- name: Checkout pandas Repo
	# actions/checkout does not work since it requires node
	run: |
	git config --global --add safe.directory $PWD
	if [ $GITHUB_EVENT_NAME != pull_request ]; then
	git clone --recursive --branch=$GITHUB_REF_NAME https://github.com/${GITHUB_REPOSITORY}.git $GITHUB_WORKSPACE
	git reset --hard $GITHUB_SHA
	else
	git clone --recursive https://github.com/${GITHUB_REPOSITORY}.git $GITHUB_WORKSPACE
	git fetch origin $GITHUB_REF:my_ref_name
	git checkout $GITHUB_BASE_REF
	git -c user.email="you@example.com" merge --no-commit my_ref_name
	fi
	- name: Configure System Packages
	run: |
	apk update
	apk add musl-locales
	- name: Build environment
	run: |
	/opt/python/cp39-cp39/bin/python -m venv ~/virtualenvs/pandas-dev
	. ~/virtualenvs/pandas-dev/bin/activate
	python -m pip install -U pip wheel setuptools meson-python==0.13.1 meson[ninja]==1.0.1
	python -m pip install --no-cache-dir versioneer[toml] cython numpy python-dateutil pytz pytest>=7.0.0 pytest-xdist>=2.2.0 pytest-asyncio>=0.17 hypothesis>=6.46.1
	python -m pip install --no-cache-dir --no-build-isolation -e .
	python -m pip list --no-cache-dir
	- name: Run Tests
	run: |
	. ~/virtualenvs/pandas-dev/bin/activate
	export PANDAS_CI=1
	python -m pytest -m 'not slow and not network and not clipboard and not single_cpu' pandas --junitxml=test-data.xml
	concurrency:
	# https://github.community/t/concurrecy-not-work-for-push/183068/7
	group: ${{ github.event_name == 'push' && github.run_number || github.ref }}-musl
	cancel-in-progress: true

incomplete? Any pointers, existing issues, PRs on this?

[mroeschke]

musllinux wheel generation was recently added so they should be uploaded starting with 2.0.3