Use cibuildwheel #7552

radarhere · 2023-11-14T12:16:56Z

It has been suggested that we move from multibuild to cibuildwheel.

Looking at cibuildwheel though, it doesn't have multibuild's library builders to build our dependencies.

So this PR employs a hybrid approach, using multibuild to build the dependencies first, and then cibuildwheel to build the wheels.

It seems to me that the primary advantage of cibuildwheel is speed, in that the dependencies are built less, and re-used by different Python environments.

.github/workflows/wheels.yml

Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>

.github/workflows/wheels.yml

hugovk · 2023-11-15T16:33:28Z

This is looking good, thanks!

Comparing the zip generated before with this PR:

The sdist covered above.

We're also building PyPy 3.8 wheels in addition to PyPy 3.9-3.10, I think it's because cibuildwheel by default looks at our requires-python = ">=3.8". We should be able to skip with something like this:

CIBW_SKIP: pp38-*

The macOS Intel wheels are built for 10.9 here instead of 10.10 before. I don't think this matters?

.github/workflows/wheels.yml

hugovk · 2023-11-15T16:45:43Z

Hmm, installing a wheel for this Apple Silicon Mac: pillow-10.2.0.dev0-cp311-cp311-macosx_11_0_arm64.whl

Gives an error on import:

Python 3.11.5 (v3.11.5:cce6ba91b3, Aug 24 2023, 10:50:31) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from PIL import Image
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ in <module>:1                                                                                    │
│                                                                                                  │
│ /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/PIL/Image.py:82  │
│ in <module>                                                                                      │
│                                                                                                  │
│     79 │   # import Image and use the Image.core variable instead.                               │
│     80 │   # Also note that Image.core is not a publicly documented interface,                   │
│     81 │   # and should be considered private and subject to change.                             │
│ ❱   82 │   from . import _imaging as core                                                        │
│     83 │                                                                                         │
│     84 │   if __version__ != getattr(core, "PILLOW_VERSION", None):                              │
│     85 │   │   msg = (                                                                           │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/PIL/_imaging.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace
'_jpeg_resync_to_restart'

Renaming this wheel and the equivalent Pillow-10.1.0-cp311-cp311-macosx_11_0_arm64.whl from https://pypi.org/project/Pillow/10.1.0/#files to add .zip, and comparing, shows we're now missing the .dylibs directory, and some of the .so files:

Details

hugovk · 2023-11-15T16:50:18Z

A quick inspection of Intel Mac, manylinux, and musllinux wheels, they look like they have everything.

nulano · 2023-11-15T17:14:50Z

.github/workflows/wheels.yml

+          CIBW_BUILD: ${{ matrix.build }}
+          CIBW_CONFIG_SETTINGS: raqm=enable raqm=vendor fribidi=vendor
+          CIBW_MANYLINUX_PYPY_X86_64_IMAGE: ${{ matrix.manylinux }}
+          CIBW_MANYLINUX_X86_64_IMAGE: ${{ matrix.manylinux }}


While arm64 wheels can be built on x86_64, they cannot be tested. The ability to test the arm64 wheels will be added in a future release of cibuildwheel, once Apple Silicon CI runners are widely available. To silence this warning, set CIBW_TEST_SKIP: *-macosx_arm64.

Suggested change

CIBW_MANYLINUX_X86_64_IMAGE: ${{ matrix.manylinux }}

CIBW_MANYLINUX_X86_64_IMAGE: ${{ matrix.manylinux }}

CIBW_TEST_SKIP: *-macosx_arm64

Sure, I've added a commit for this. Testing, I found that it required quotes, as per https://cibuildwheel.readthedocs.io/en/stable/options/#test-skip

I've created a PR to fix this upstream: pypa/cibuildwheel#1665

radarhere · 2023-11-15T22:31:40Z

The macOS Intel wheels are built for 10.9 here instead of 10.10 before. I don't think this matters?

Taking a look, I found that #6179 had a problem when I accidentally reverted it to 10.9 once before, so I've pushed a commit to restore 10.10.

Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>

hugovk · 2023-11-16T21:50:13Z

Thanks for the update. Now we get the same filenames before and after: also the sdist, and 10_10 files instead of 10_9. 👍

However, I'm still getting this for pillow-10.2.0.dev0-cp311-cp311-macosx_11_0_arm64.whl:

ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/PIL/_imaging.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace
'_jpeg_resync_to_restart'

We still seem to be missing .dylibs:

Details

hugovk · 2023-11-16T21:57:45Z

On an Intel Mac, pillow-10.2.0.dev0-cp311-cp311-macosx_10_10_x86_64.whl does install, and a quick Image.new() and .show() work.

radarhere · 2023-11-17T08:47:10Z

Ok, the macOS arm64 wheels should be better now.

hugovk · 2023-11-17T09:34:19Z

Thanks! I tested the cp38 to cp312 macOS arm64 wheels and they all passed a simple install/im=Image.new("RGB", (20, 20), "red")/im.show() test.

The wheels contain the same files as before:

Details

I'll test the Intel ones later. I note they now contain libjpeg and libzstd dylib files that they didn't have before:

Details

It would be good if anyone can help test out some manylinux and musllinux wheels: go to https://github.com/python-pillow/Pillow/actions/runs/6901285566?pr=7552 and download the "dist" artifact, and pip install <filename>.whl.

nulano · 2023-11-17T16:28:49Z

I can see that the libzstd dylib has different permissions (-r--r--r--) than most of the other files (-rwxr-xr-x). Don't know if that is an issue, but it usually means it was copied from the system and not built during the release.

It would be good if anyone can help test out some manylinux and musllinux wheels

/pillow-10.2.0.dev0-cp38-cp38-manylinux_2_28_x86_64.whl passes tests on my Ubuntu Server system.

All of the musllinux wheels pass tests in the corresponding python:3.X-alpine Docker containers.

hugovk · 2023-11-17T19:15:53Z

✅ Working on Intel macOS with Python 3.8-3.12 with:

#!/usr/bin/env bash

for VARIABLE in 8 9 10 11 12
do
	python3.$VARIABLE -m pip install -U pip
	python3.$VARIABLE -m pip uninstall -y pillow
	python3.$VARIABLE -m pip install pillow-10.2.0.dev0-cp3$VARIABLE-cp3$VARIABLE-macosx_10_10_x86_64.whl
	python3.$VARIABLE 1.py
done

from PIL import Image
im = Image.new("RGB", (10, 20), "red")
im.show()

radarhere · 2023-11-17T19:44:56Z

I can see that the libzstd dylib has different permissions (-r--r--r--) than most of the other files (-rwxr-xr-x). Don't know if that is an issue, but it usually means it was copied from the system and not built during the release.

Ok, I've pushed a commit to remove it again, like it was before this PR.

hugovk · 2023-11-18T10:52:46Z

✅ Looking good testing with #7552 (comment) on both Intel and Apple Silicon for all versions.

hugovk

Thank you!

@hugovk

pillow has started using [cibuildwheel](https://github.com/pypa/cibuildwheel) - python-pillow/Pillow#7552 This moved the script for building wheel dependencies from `wheels/config.sh`, a more natural location for [multibuild](https://github.com/multi-build/multibuild), to just under GitHub Actions instead, at [.github/workflows/wheel-dependencies.sh](https://github.com/python-pillow/Pillow/blob/main/.github/workflows/wheels-dependencies.sh) It also now includes `wheels/multibuild/library_builders.sh` [by itself](https://github.com/python-pillow/Pillow/blob/e2ddd27500cdd4c8156bc0d5fe0bfa050408c707/.github/workflows/wheels-dependencies.sh#L10), and rearranged the code to run immediately, instead of needing `pre_build` to be called separately. cc @hugovk Co-authored-by: Andrew Murray <radarhere@users.noreply.github.com>

Use cibuildwheel

ad7fee5

radarhere force-pushed the cibuildwheel branch from 859aded to ad7fee5 Compare November 14, 2023 12:25

hugovk reviewed Nov 14, 2023

View reviewed changes

.github/workflows/wheels.yml Outdated Show resolved Hide resolved

.github/workflows/wheels.yml Outdated Show resolved Hide resolved

hugovk added the Build label Nov 14, 2023

radarhere and others added 2 commits November 15, 2023 11:48

Use action

9be181c

Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>

Removed --std=c99 setting

0a12f6c

radarhere force-pushed the cibuildwheel branch from 9102c1d to 3bf2414 Compare November 15, 2023 02:38

radarhere added 2 commits November 15, 2023 15:47

Pin cibuildwheel version

6672663

Removed openblas-devel and pkgconfig on manylinux

0204e70

radarhere force-pushed the cibuildwheel branch from 3bf2414 to 0204e70 Compare November 15, 2023 04:48

radarhere marked this pull request as ready for review November 15, 2023 08:34

hugovk reviewed Nov 15, 2023

View reviewed changes

.github/workflows/wheels.yml Show resolved Hide resolved

hugovk reviewed Nov 15, 2023

View reviewed changes

.github/workflows/wheels.yml Show resolved Hide resolved

nulano reviewed Nov 15, 2023

View reviewed changes

radarhere added 4 commits November 16, 2023 08:13

Fixed xcb setup

9f1a1d3

Skip tests on macOS arm64

d077190

Move wheels into dist.zip

3b1e46e

Restored MACOSX_DEPLOYMENT_TARGET on x86_64

e013ad9

radarhere force-pushed the cibuildwheel branch from e487a0f to e013ad9 Compare November 15, 2023 22:23

nulano mentioned this pull request Nov 15, 2023

Automate build and release to PyPI #7390

Closed

radarhere and others added 2 commits November 16, 2023 09:32

Skip PyPy 3.8

c42ebf7

Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>

Skip PyPy 3.8

d4a7016

Fixed bundling dylibs into arm64 wheels

72ee8c5

Removed zstd again

07c216c

radarhere force-pushed the cibuildwheel branch from fdf18f5 to 07c216c Compare November 17, 2023 19:44

hugovk mentioned this pull request Nov 18, 2023

Use quotes when silencing warning pypa/cibuildwheel#1665

Merged

hugovk approved these changes Nov 23, 2023

View reviewed changes

hugovk merged commit e2ddd27 into python-pillow:main Nov 25, 2023
61 of 62 checks passed

radarhere deleted the cibuildwheel branch November 25, 2023 08:39

This was referenced Nov 25, 2023

pillow: Updated script path google/oss-fuzz#11267

Merged

Updated wheels README after cibuildwheel integration #7572

Merged

Updated wheels script path python-pillow/docker-images#194

Merged

radarhere mentioned this pull request Dec 2, 2023

Release Pillow 10.1.0 on October 15, 2023 #7348

Closed

21 tasks

This was referenced Jan 2, 2024

Downgrade TravisCI to Focal and Python 3.11 #7680

Closed

Package name is now lowercase in wheel filenames #7681

Merged

Skip PyPy3.8 Windows wheel #7682

Merged

radarhere mentioned this pull request Oct 21, 2024

Add support for macOS 11.0 as a target multi-build/multibuild#528

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use cibuildwheel #7552

Use cibuildwheel #7552

radarhere commented Nov 14, 2023 •

edited by hugovk

Loading

hugovk commented Nov 15, 2023

hugovk commented Nov 15, 2023 •

edited

Loading

hugovk commented Nov 15, 2023

nulano Nov 15, 2023

radarhere Nov 15, 2023

hugovk Nov 18, 2023

radarhere commented Nov 15, 2023

hugovk commented Nov 16, 2023

hugovk commented Nov 16, 2023

radarhere commented Nov 17, 2023

hugovk commented Nov 17, 2023

nulano commented Nov 17, 2023 •

edited

Loading

hugovk commented Nov 17, 2023 •

edited

Loading

radarhere commented Nov 17, 2023

hugovk commented Nov 18, 2023 •

edited

Loading

hugovk left a comment

	CIBW_MANYLINUX_X86_64_IMAGE: ${{ matrix.manylinux }}
	CIBW_MANYLINUX_X86_64_IMAGE: ${{ matrix.manylinux }}
	CIBW_TEST_SKIP: *-macosx_arm64

Use cibuildwheel #7552

Use cibuildwheel #7552

Conversation

radarhere commented Nov 14, 2023 • edited by hugovk Loading

hugovk commented Nov 15, 2023

hugovk commented Nov 15, 2023 • edited Loading

hugovk commented Nov 15, 2023

nulano Nov 15, 2023

Choose a reason for hiding this comment

radarhere Nov 15, 2023

Choose a reason for hiding this comment

hugovk Nov 18, 2023

Choose a reason for hiding this comment

radarhere commented Nov 15, 2023

hugovk commented Nov 16, 2023

hugovk commented Nov 16, 2023

radarhere commented Nov 17, 2023

hugovk commented Nov 17, 2023

nulano commented Nov 17, 2023 • edited Loading

hugovk commented Nov 17, 2023 • edited Loading

radarhere commented Nov 17, 2023

hugovk commented Nov 18, 2023 • edited Loading

hugovk left a comment

Choose a reason for hiding this comment

radarhere commented Nov 14, 2023 •

edited by hugovk

Loading

hugovk commented Nov 15, 2023 •

edited

Loading

nulano commented Nov 17, 2023 •

edited

Loading

hugovk commented Nov 17, 2023 •

edited

Loading

hugovk commented Nov 18, 2023 •

edited

Loading