feat: add `libz.so.1` to whitelisted libraries #334

mayeut · 2021-08-29T18:18:01Z

This also requires to blacklist some symbols from libz.so.1.
There was some breakage in ABI compatibility for this library.
It's seems unlikely that those symbols are used in the wild.

Fix #152
Fix #161

auditwheel/policy/manylinux-policy.json

codecov · 2021-08-29T18:32:41Z

Codecov Report

Merging #334 (8a03a19) into master (b2d597c) will increase coverage by 0.18%.
The diff coverage is 93.61%.

@@            Coverage Diff             @@
##           master     #334      +/-   ##
==========================================
+ Coverage   89.35%   89.54%   +0.18%     
==========================================
  Files          23       23              
  Lines        1203     1244      +41     
  Branches      255      270      +15     
==========================================
+ Hits         1075     1114      +39     
- Misses         78       79       +1     
- Partials       50       51       +1

Impacted Files	Coverage Δ
auditwheel/wheel_abi.py	`96.99% <75.00%> (-1.43%)`	⬇️
auditwheel/elfutils.py	`94.05% <95.23%> (+0.23%)`	⬆️
auditwheel/main_repair.py	`90.47% <100.00%> (+0.47%)`	⬆️
auditwheel/main_show.py	`75.86% <100.00%> (+5.27%)`	⬆️
auditwheel/policy/external_references.py	`97.82% <100.00%> (+0.20%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b2d597c...8a03a19. Read the comment docs.

This also requires to blacklist some symbols from `libz.so.1`. There was some breakage in ABI compatibility for this library. It's seems unlikely that those symbols are used in the wild.

lkollar

Where is the list of blacklisted symbols coming from? This is something we should definitely document to make sure we can keep the list up to date.

lkollar · 2021-09-08T20:02:42Z

auditwheel/elfutils.py

+        section = elf.get_section_by_name(".dynsym")
+        if section is not None:
+            # look for all undef symbols
+            for sym in section.iter_symbols():
+                if sym["st_shndx"] == "SHN_UNDEF":
+                    undef_symbols.add(sym.name)


nit: This could be made into a utility function to find undefined symbols. That would make this function more readable as well.

auditwheel/elfutils.py

lkollar · 2021-09-08T20:20:01Z

auditwheel/wheel_abi.py

@@ -285,12 +293,19 @@ def analyze_wheel_abi(wheel_fn: str) -> WheelAbIInfo:
        sym_tag,
        ucs_tag,
        pyfpe_tag,
+        blacklist_tag,
    )


 def update(d, u):


Unrelated to this PR, but we should refactor this function.

mayeut · 2021-09-09T05:54:48Z

Where is the list of blacklisted symbols coming from? This is something we should definitely document to make sure we can keep the list up to date.

Everything comes from mayeut/pep600_compliance#88 (The same repo was used to define policies for manylinux_2_24 & manylinux_2_27).
I crossed referenced this information with https://abi-laboratory.pro/index.php?view=timeline&l=zlib as well.

Co-authored-by: László Kiss Kollár <lkollar@users.noreply.github.com>

vgod-dbx · 2022-03-08T18:07:20Z

Hi,

According to the manylinux policies, libz.so.1 should not be whitelisted. Can we revert this change or at least make it as an opt-in argument so all libraries that use auditwheel can still comply with the manylinux policies by default?

A bit more context on why this is causing problems. Our company runs a minimal Linux runtime to ensure the portability of our code across Linux versions/distros. We have been replying on the manylinux policies to ensure that all pip libraries run on our runtime without missing dependencies. However, this change introduced an external dependency on libz.so.1 (which does not exist in our runtime) into some libraries (huggingface/tokenizers#946, for one example) and prevented us to upgrade to newer versions.

mayeut · 2022-03-12T10:54:07Z

@vgod-dbx,

According to the manylinux policies, libz.so.1 should not be whitelisted.

You're probably talking about the previous PEP 513, PEP 571 & PEP 599. Those 3 PEP have been superseded by the much more generic PEP 600 which does not list allowed / disallowed libraries & which states:

manylinux_${GLIBCMAJOR}_${GLIBCMINOR}_${ARCH}
This tag is a promise: the wheel’s creator promises that the wheel will work on any mainstream Linux distro that uses glibc version ${GLIBCMAJOR}.${GLIBCMINOR} or later, and where the ${ARCH} matches the return value from distutils.util.get_platform(). (For more detail about architecture tags, see PEP 425.)

libz.so.1 is available on all mainstream Linux distros & not whitelisting it caused some issues (ranging from "not working" to "security issue" on unmaintained grafted library).
libz.so.1 is also part of the LSB Core specification which more or less means it should also be available on all Linux distros (i.e. not just the mainstream ones).

CPython itself relies on zlib for multiple modules (although, if you're using CPython and linking zlib statically, it does not depend on libz.so.1).

Your company should probably update its minimal runtime package with libz.so.1.

mayeut requested a review from lkollar August 29, 2021 18:18

mayeut mentioned this pull request Aug 29, 2021

Release 5.0.0 #327

Closed

mayeut commented Aug 29, 2021

View reviewed changes

auditwheel/policy/manylinux-policy.json Show resolved Hide resolved

mayeut force-pushed the zlib branch 2 times, most recently from 3d9e720 to 3845ebc Compare September 3, 2021 19:09

feat: add libz.so.1 to whitelisted libraries

619af6d

This also requires to blacklist some symbols from `libz.so.1`. There was some breakage in ABI compatibility for this library. It's seems unlikely that those symbols are used in the wild.

mayeut force-pushed the zlib branch from 3845ebc to 619af6d Compare September 4, 2021 08:59

lkollar reviewed Sep 8, 2021

View reviewed changes

mayeut and others added 3 commits September 9, 2021 07:56

chore: refactor iterating over symbol dict

fed00a8

Co-authored-by: László Kiss Kollár <lkollar@users.noreply.github.com>

fix: refactor iterating over symbol dict

195681a

refactor: add elfutils.get_undefined_symbols

8a03a19

lkollar approved these changes Sep 14, 2021

View reviewed changes

messense mentioned this pull request Sep 15, 2021

auditwheel: add libz.so.1 to whitelisted libraries PyO3/maturin#622

Closed

mayeut merged commit 16b2e80 into pypa:master Sep 15, 2021

mayeut deleted the zlib branch September 15, 2021 05:38

radarhere mentioned this pull request Oct 9, 2021

Fix failing Linux jobs python-pillow/pillow-wheels#225

Merged

messense mentioned this pull request Dec 15, 2021

Implement auditwheel repair with patchelf PyO3/maturin#742

Merged

tylerjereddy mentioned this pull request Feb 15, 2022

pydarshan can't load libdarshan-util-*.so from pypi on Cori&Perlmutter (NERSC/LBL) darshan-hpc/darshan#656

Closed

radarhere mentioned this pull request Feb 16, 2022

Downgrade zlib on Linux instead of auditwheel python-pillow/pillow-wheels#254

Merged

vgod-dbx mentioned this pull request Mar 8, 2022

0.11.5 and 0.11.6 packages not compatible with manylinux2010 huggingface/tokenizers#946

Closed

jorisvandenbossche mentioned this pull request Mar 15, 2022

Build wheels using vcpkg for Linux/MacOS geopandas/pyogrio#55

Merged

takluyver mentioned this pull request Apr 2, 2022

Upgrade bundled zlib to 1.2.12 wheels once HDF5 is compatible h5py/h5py#2081

Closed

bbayles mentioned this pull request Apr 25, 2023

Wheels for Linux sbraz/pymediainfo#128

Open

henryiii mentioned this pull request Aug 1, 2023

Embedded libz.so.1 missing psycopg/psycopg#615

Closed

alopezz mentioned this pull request Feb 7, 2024

Include zlib in wheels that need it DataDog/integrations-core#16814

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add `libz.so.1` to whitelisted libraries #334

feat: add `libz.so.1` to whitelisted libraries #334

mayeut commented Aug 29, 2021 •

edited

Loading

codecov bot commented Aug 29, 2021 •

edited

Loading

lkollar left a comment

lkollar Sep 8, 2021

mayeut Sep 9, 2021

mayeut Sep 14, 2021

lkollar Sep 8, 2021

mayeut commented Sep 9, 2021

vgod-dbx commented Mar 8, 2022

mayeut commented Mar 12, 2022

feat: add libz.so.1 to whitelisted libraries #334

feat: add libz.so.1 to whitelisted libraries #334

Conversation

mayeut commented Aug 29, 2021 • edited Loading

codecov bot commented Aug 29, 2021 • edited Loading

Codecov Report

lkollar left a comment

Choose a reason for hiding this comment

lkollar Sep 8, 2021

Choose a reason for hiding this comment

mayeut Sep 9, 2021

Choose a reason for hiding this comment

mayeut Sep 14, 2021

Choose a reason for hiding this comment

lkollar Sep 8, 2021

Choose a reason for hiding this comment

mayeut commented Sep 9, 2021

vgod-dbx commented Mar 8, 2022

mayeut commented Mar 12, 2022

feat: add `libz.so.1` to whitelisted libraries #334

feat: add `libz.so.1` to whitelisted libraries #334

mayeut commented Aug 29, 2021 •

edited

Loading

codecov bot commented Aug 29, 2021 •

edited

Loading