Building from source does not support freetype and raqm #7040

ccoulombe · 2023-03-28T14:35:59Z

OS: Centos
Python: 3.10
Pillow: 9.4.0
freetype : 6.16.1

Building Pillow 9.4.0 from source does not seem to properly provide RAQM support.
Building with the bundled raqm (v0.9.0) or a system raqm (v0.9.0) fails to provide freetype or raqm support.

Using bundled raqm

Build commands

cd Pillow-9.4.0
export CPATH=$EBROOTGENTOO/include:$CPATH
export LIBRARY_PATH=$EBROOTGENTOO/lib64:$LIBRARY_PATH
sed -i -e 's;/sbin/ldconfig;ldconfig;g' setup.py
python setup.py build_ext --debug --vendor-raqm bdist_wheel |& tee bundled.out

The build setup summary is ok:

--------------------------------------------------------------------
PIL SETUP SUMMARY
--------------------------------------------------------------------
version      Pillow 9.4.0
platform     linux 3.10.2 (main, Feb  4 2022, 19:10:35)
             [GCC 9.3.0]
--------------------------------------------------------------------
--- JPEG support available
--- OPENJPEG (JPEG2000) support available (2.3)
--- ZLIB (PNG/ZIP) support available
*** LIBIMAGEQUANT support not available
--- LIBTIFF support available
--- FREETYPE2 support available
--- RAQM (Text shaping) support available (bundled)
--- LITTLECMS2 support available
--- WEBP support available
--- WEBPMUX support available
--- XCB (X protocol) support available
--------------------------------------------------------------------

But once installed in a virtual environment, freetype and raqm are mentioned as not installed but the libraries are correctly linked:

$ python -m PIL | rg installed
*** FREETYPE2 support not installed
*** RAQM (Bidirectional Text) support not installed
*** LIBIMAGEQUANT (Quantization method) support not installed

$ ldd $VIRTUAL_ENV/lib/python3.10/site-packages/PIL/_imagingft.cpython-310-x86_64-linux-gnu.so:
	linux-vdso.so.1 (0x00007ffceb9fd000)
	libfreetype.so.6 => /cvmfs/soft.computecanada.ca/gentoo/2020/usr/lib64/libfreetype.so.6 (0x00002b4085e78000)
	libharfbuzz.so.0 => /cvmfs/soft.computecanada.ca/gentoo/2020/usr/lib64/libharfbuzz.so.0 (0x00002b4085f38000)
	libfribidi.so.0 => /cvmfs/soft.computecanada.ca/gentoo/2020/usr/lib64/libfribidi.so.0 (0x00002b4086041000)

And running a short example results in the following error:

ImportError: The _imagingft C module is not installed

but the library exists.

Using system raqm

Building a system RAQM results in the same as above.

Build commands:

cd Pillow-9.4.0
export CPATH=$HOME/libraqm/INST/usr/local/include:$EBROOTGENTOO/include:$CPATH
export LIBRARY_PATH=$HOME/libraqm/INST/usr/local/lib64:$EBROOTGENTOO/lib64:$LIBRARY_PATH
sed -i -e 's;/sbin/ldconfig;ldconfig;g' setup.py
python setup.py build_ext --debug --enable-raqm bdist_wheel |& tee system.out

Setup from the build output:

PIL SETUP SUMMARY
--------------------------------------------------------------------
version      Pillow 9.4.0
platform     linux 3.10.2 (main, Feb  4 2022, 19:10:35)
             [GCC 9.3.0]
--------------------------------------------------------------------
--- JPEG support available
--- OPENJPEG (JPEG2000) support available
--- ZLIB (PNG/ZIP) support available
*** LIBIMAGEQUANT support not available
--- LIBTIFF support available
--- FREETYPE2 support available
--- RAQM (Text shaping) support available
--- LITTLECMS2 support available
--- WEBP support available
--- WEBPMUX support available
--- XCB (X protocol) support available

But once installed in a virtual environment, freetype and raqm are mentioned as not installed but the libraries are correctly linked:

(495) Pillow-9.4.0 (main *) $ python -m PIL | rg installed
*** FREETYPE2 support not installed
*** RAQM (Bidirectional Text) support not installed
*** LIBIMAGEQUANT (Quantization method) support not installed

(495) Pillow-9.4.0 (main *) $ ldd $VIRTUAL_ENV/lib/python3.10/site-packages/PIL/_imagingft.cpython-310-x86_64-linux-gnu.so:
	linux-vdso.so.1 (0x00007ffe3f9b6000)
	libfreetype.so.6 => /home/coulombc/libraqm/INST/usr/local/lib64/libfreetype.so.6 (0x00002aaeeba95000)
	libraqm.so.0 => /home/coulombc/libraqm/INST/usr/local/lib64/libraqm.so.0 (0x00002aaeebb88000)
	libc.so.6 => /cvmfs/soft.computecanada.ca/gentoo/2020/lib64/libc.so.6 (0x00002aaeebbc6000)
	libharfbuzz.so.0 => /home/coulombc/libraqm/INST/usr/local/lib64/libharfbuzz.so.0 (0x00002aaeebd80000)
	libfribidi.so.0 => /cvmfs/soft.computecanada.ca/gentoo/2020/usr/lib64/libfribidi.so.0 (0x00002aaeec045000)

This looks like a bug to me but could a wrong detection from Pillow. Anyone successfully built Pillow with RAQM? What am I missing here?

The text was updated successfully, but these errors were encountered:

radarhere · 2023-03-28T21:28:34Z

What version of CentOS are you using?

I wouldn't have thought that you needed to run sed -i -e 's;/sbin/ldconfig;ldconfig;g' setup.py. If you'd like to talk about why that is, maybe we can fix that within Pillow itself.

once installed in a virtual environment

So Pillow works without a problem in your normal environment?

ccoulombe · 2023-03-29T13:55:03Z

It is a multi-layer system, the centos layer is 7.9.

I wouldn't have thought that you needed to run sed -i -e 's;/sbin/ldconfig;ldconfig;g' setup.py. If you'd like to talk about why that is, maybe we can fix that within Pillow itself.

This is needed since the path of ldconfig is hardcoded in the setup, relying on the ldconfig from $PATH would be flexible to different systems.

No, Pillow does not work properly. In summary, the setup after the build says everything is ok, but once installed in a virtual environment, support for freetype or raqm is not available.

Here's a fresh build that reproduce the issue:

~ $ module load python/3.10
~ $ git clone https://github.com/python-pillow/Pillow.git -b 9.4.0
~ $ makevenv 
(8666) ~ $ pip install oldest-supported-numpy cython
(8666) ~ $ cd Pillow/
(8666) Pillow ((9.4.0) ) $ sed -i -e 's;/sbin/ldconfig;ldconfig;g' setup.py
(8666) Pillow ((9.4.0) *) $ export CPATH=$EBROOTGENTOO/include:$CPATH
(8666) Pillow ((9.4.0) *) $ export LIBRARY_PATH=$EBROOTGENTOO/lib64:$LIBRARY_PATH
(8666) Pillow ((9.4.0) *) $ python setup.py build_ext --inplace --debug --vendor-raqm bdist_wheel |& tee out.out
running build_ext
...
Using vendored version of raqm
...
--------------------------------------------------------------------
PIL SETUP SUMMARY
--------------------------------------------------------------------
version      Pillow 9.4.0
platform     linux 3.10.2 (main, Feb  4 2022, 19:10:35)
             [GCC 9.3.0]
--------------------------------------------------------------------
--- JPEG support available
--- OPENJPEG (JPEG2000) support available (2.3)
--- ZLIB (PNG/ZIP) support available
*** LIBIMAGEQUANT support not available
--- LIBTIFF support available
--- FREETYPE2 support available
--- RAQM (Text shaping) support available (bundled)
--- LITTLECMS2 support available
--- WEBP support available
--- WEBPMUX support available
--- XCB (X protocol) support available
--------------------------------------------------------------------

--------------------------------------------------------------------
PIL SETUP SUMMARY
--------------------------------------------------------------------
version      Pillow 9.4.0
platform     linux 3.10.2 (main, Feb  4 2022, 19:10:35)
             [GCC 9.3.0]
--------------------------------------------------------------------
--- JPEG support available
--- OPENJPEG (JPEG2000) support available
--- ZLIB (PNG/ZIP) support available
*** LIBIMAGEQUANT support not available
--- LIBTIFF support available
--- FREETYPE2 support available
--- RAQM (Text shaping) support available (bundled)
--- LITTLECMS2 support available
--- WEBP support available
--- WEBPMUX support available
--- XCB (X protocol) support available
--------------------------------------------------------------------

(8666) Pillow ((9.4.0) *) $ pip install dist/Pillow-9.4.0-cp310-cp310-linux_x86_64.whl --force-reinstall
(8666) Pillow ((9.4.0) *) $ cd
(8666) ~ $ python -c 'from PIL import features; print(features.check("raqm"))'
None
(8666) ~ $ python -c 'from PIL import features; print(features.check("freetype2"))'
False

ccoulombe · 2023-03-29T14:54:00Z

Importing the library _imagingft raises a missing symbol:

(8666) Pillow ((9.4.0) *) $ cd src/PIL
(8666) PIL ((9.4.0) *) $ python -c 'import _imagingft'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: /home/coulombc/wheels_builder/Pillow/src/PIL/_imagingft.cpython-310-x86_64-linux-gnu.so: undefined symbol: FT_Get_Transform
(8666) PIL ((9.4.0) *) $ nm -u _imagingft.cpython-310-x86_64-linux-gnu.so |& rg Transform
                 U FT_Get_Transform
                 U FT_Vector_Transform

freetype version is: libfreetype.so.6.16.1
harfbuzz version is : 2.6.4

Ah, those versions are too old for raqm.

Forcing the path to the libfreetype and libharfbuzz built as part of raqm fixed importing _imagingft.

ccoulombe · 2023-03-29T15:43:55Z

The issue was the version of freetype and harfbuzz that were too old for raqm.
Using libfreetype.so.6.18.0 and libharfbuzz.so.0.30000.0, I can now successfully:

(8666) ~ $ python -c 'from PIL import features; print(features.check("freetype2"))'
True
(8666) ~ $ python -c 'from PIL import features; print(features.check("raqm"))'
True

In light of this, and in order to help wheels builders/maintainers, I'd suggest to be more explicit when :

the exception is raised about import _imagingft
checking module
checking feature module
If an another exception is not acceptable, maybe raise a user warning?

Also, I think the setup should verify that the version of dependencies are met.

radarhere · 2023-03-30T06:04:14Z

freetype version is: libfreetype.so.6.16.1

From what I can see, this is actually freetype 2.9.1

We should support it, according to https://pillow.readthedocs.io/en/stable/deprecations.html#freetype-2-7

Support for FreeType 2.7 has been removed.

We recommend upgrading to at least FreeType 2.10.4, which fixed a severe vulnerability introduced in FreeType 2.6 (CVE-2020-15999).

However, libraqm raised the minimum version of FreeType required in HOST-Oman/libraqm#151.

ccoulombe · 2023-03-30T12:47:37Z

The main culprit was the version of harfbuzz (2.6.4).
According to raqm v0.9.0, freetype should be >= 2.11 and harfbuzz >= 3.0.0

radarhere · 2023-03-30T19:36:36Z

PR #7044 has replaced "/sbin/ldconfig" with "ldconfig". This should be available in Pillow 9.5.0 in a few days.

ccoulombe · 2023-03-30T19:47:58Z

Great, thanks!

nulano · 2023-03-30T19:56:15Z

It might be worth checking the version numbers of used libraries before starting a build, but setup.py is already quite complex, and other than the bundled raqm (which is not the default option), Pillow usually supports quite old versions of dependencies. It is expected that most users will build using their system raqm library, which should import correctly.

I imagine you got an "implicit function declaration" warning (which really should be an error IMO) during the build. I'd suggest using CFLAGS="-Werror=implicit-function-declaration" to treat it as an error as is done for CI in #4963. Perhaps a compile-time version check could be added to the bundled raqm sources to make it more obvious?

Forcing the path to the libfreetype and libharfbuzz built as part of raqm fixed importing _imagingft.

I could be misunderstanding here, but is (part of) the issue here that your system raqm was not correctly importing freetype and harfbuzz?

ccoulombe · 2023-03-30T20:09:54Z

Agreed, then the question : How can one normal user easily verify the version when they are not documented in the readme nor the installation docs? raises.
I am not a normal user and I can dig around to found the answers.

But! A setup or any package should verify that its requirements are met. So many do with very complex setups, think of numpy or scipy. An idea could be to change the setup for meson or cmake ? Or simply verify them in the current setup script.

Sure, I could turn the warning into an error.

Yes, both, the bundled and system raqm did not linked to a recent version of freetype and harfbuzz, causing the symbol not found. We updated the version of both freetype and hafbuzz and installed a system raqm to satisfy all requirements.

If the exception raised would have been redirected to stderr or a warning issued, I would have found the culprit sooner.

ccoulombe · 2023-03-30T20:15:53Z

From a previous comment:

In light of all this, and in order to help wheels builders/maintainers, I'd suggest to be more explicit when :

the exception is raised about import _imagingft
checking module
checking feature module
If an another exception is not acceptable, maybe raise a user warning?

With the above, I'd also suggest:

documenting the minimal version for all requirements, some are documented, but others are not
adding requirements verification to the setup/build

nulano · 2023-03-30T21:26:04Z

How can one normal user easily verify the version when they are not documented in the readme nor the installation docs?

I'd also suggest:

documenting the minimal version for all requirements, some are documented, but others are not

adding requirements verification to the setup/build

AFAIK there are no explicit minimum versions (except for FreeType, as noted above), simply a requirement that the tested systems' default library versions are supported. IIRC, support for some library versions was unintentionally dropped in the past simply due to lack of testing.

Looking at the latest CI run, the following are the minimum versions currently tested:

--- TKINTER support ok, loaded 8.5                                 # centos-7
--- FREETYPE2 support ok, loaded 2.8.0                             # centos-7
--- LITTLECMS2 support ok, compiled for 2.6                        # centos-7
--- WEBP support ok, loaded 0.3.0                                  # centos-7
--- WEBP Transparency support ok
*** WEBPMUX support not installed
*** WEBP Animation support not installed
--- JPEG support ok, compiled for libjpeg-turbo 1.2.90             # centos-7
--- OPENJPEG (JPEG2000) support ok, loaded 2.3.1                   # centos-7
--- ZLIB (PNG/ZIP) support ok, loaded 1.2.7                        # centos-7
--- LIBTIFF support ok, loaded 4.0.3                               # centos-7
--- RAQM (Bidirectional Text) support ok, loaded 0.7.0             # centos-7
--- LIBIMAGEQUANT (Quantization method) support ok, loaded 2.16.0  # fedora-36
--- XCB (X protocol) support ok     # no idea, XCB does not expose its version number

However, much older versions are likely supported, although it is unclear which ones due to lack of CI testing.
I agree that the documentation should probably be updated - it might be worth discussing adding explicit minimum version requirements.

nulano · 2023-03-31T01:13:00Z

Yes, both, the bundled and system raqm did not linked to a recent version of freetype and harfbuzz, causing the symbol not found. We updated the version of both freetype and hafbuzz and installed a system raqm to satisfy all requirements.

If the exception raised would have been redirected to stderr or a warning issued, I would have found the culprit sooner.

I've created #7047 to avoid discarding the error message when importing ImageFont, and to raise a warning in PIL.features if the ImportError is not a ModuleNotFoundError (i.e. the import failed but the module was enabled during compilation).

radarhere · 2023-03-31T08:58:53Z

I attempted to install a newer raqm on CentOS 7 to test it in combination with freetype 2.8.0 - but the installation failed because raqm detected that freetype2 wasn't recent enough.

ccoulombe · 2023-03-31T13:48:36Z

@nulano Right, I am mixing up requirements of a sub-dependency (raqm) and general requirements.
Documenting the minimal known version tested will be beneficial.

Thanks both of you for the responsiveness and making improvements!

radarhere · 2023-04-01T10:09:52Z

#7044 and #7047 have been released as part of Pillow 9.5.0.

ccoulombe · 2023-04-03T11:23:40Z

@radarhere Works! Thanks!

radarhere added the Build label Mar 28, 2023

radarhere mentioned this issue Mar 30, 2023

Removed absolute path to ldconfig #7044

Merged

nulano mentioned this issue Mar 31, 2023

Do not discard error message if _imagingft fails to import #7047

Merged

radarhere changed the title ~~Pillow 9.4.0 built from source results freetype and raqm support not available~~ Building from source does not support freetype and raqm Mar 31, 2023

ccoulombe closed this as completed Apr 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Building from source does not support freetype and raqm #7040

Building from source does not support freetype and raqm #7040

ccoulombe commented Mar 28, 2023

radarhere commented Mar 28, 2023

ccoulombe commented Mar 29, 2023 •

edited

Loading

ccoulombe commented Mar 29, 2023 •

edited by radarhere

Loading

ccoulombe commented Mar 29, 2023

radarhere commented Mar 30, 2023 •

edited

Loading

ccoulombe commented Mar 30, 2023

radarhere commented Mar 30, 2023

ccoulombe commented Mar 30, 2023

nulano commented Mar 30, 2023

ccoulombe commented Mar 30, 2023 •

edited

Loading

ccoulombe commented Mar 30, 2023 •

edited

Loading

nulano commented Mar 30, 2023 •

edited

Loading

nulano commented Mar 31, 2023 •

edited

Loading

radarhere commented Mar 31, 2023

ccoulombe commented Mar 31, 2023 •

edited

Loading

radarhere commented Apr 1, 2023

ccoulombe commented Apr 3, 2023

Building from source does not support freetype and raqm #7040

Building from source does not support freetype and raqm #7040

Comments

ccoulombe commented Mar 28, 2023

Using bundled raqm

Build commands

Using system raqm

Build commands:

radarhere commented Mar 28, 2023

ccoulombe commented Mar 29, 2023 • edited Loading

ccoulombe commented Mar 29, 2023 • edited by radarhere Loading

ccoulombe commented Mar 29, 2023

radarhere commented Mar 30, 2023 • edited Loading

ccoulombe commented Mar 30, 2023

radarhere commented Mar 30, 2023

ccoulombe commented Mar 30, 2023

nulano commented Mar 30, 2023

ccoulombe commented Mar 30, 2023 • edited Loading

ccoulombe commented Mar 30, 2023 • edited Loading

nulano commented Mar 30, 2023 • edited Loading

nulano commented Mar 31, 2023 • edited Loading

radarhere commented Mar 31, 2023

ccoulombe commented Mar 31, 2023 • edited Loading

radarhere commented Apr 1, 2023

ccoulombe commented Apr 3, 2023

ccoulombe commented Mar 29, 2023 •

edited

Loading

ccoulombe commented Mar 29, 2023 •

edited by radarhere

Loading

radarhere commented Mar 30, 2023 •

edited

Loading

ccoulombe commented Mar 30, 2023 •

edited

Loading

ccoulombe commented Mar 30, 2023 •

edited

Loading

nulano commented Mar 30, 2023 •

edited

Loading

nulano commented Mar 31, 2023 •

edited

Loading

ccoulombe commented Mar 31, 2023 •

edited

Loading