Make order of files in repaired wheel deterministic #507
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
August 5, 2024 20:53
In order to make the output zip file reproducible (independent of the underlying filesystem's directory traversal order), sort each list of subdirectories and each list of files before adding them to the zip file. (Note that we want to sort the dirs list in place, causing os.walk to traverse the subdirectories in order.)
In order to make the output zip file reproducible (independent of the underlying filesystem's directory traversal order), sort each list of subdirectories and each list of files while we are generating the RECORD file. (Note that we want to sort the dirs list in place, causing os.walk to traverse the subdirectories in order.)
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #507 +/- ##
==========================================
+ Coverage 92.25% 92.28% +0.02%
==========================================
Files 20 20
Lines 1266 1270 +4
Branches 305 305
==========================================
+ Hits 1168 1172 +4
Misses 56 56
Partials 42 42 ☔ View full report in Codecov by Sentry. |
mayeut
requested changes
Aug 24, 2024
There was a problem hiding this comment.
Thanks for the PR
According to https://peps.python.org/pep-0427/#recommended-archiver-features, it is recommended to place the .dist-info folder at the end of the archive.
If we're ensuring the order for build reproducibility, can this be taken into account please ?
If the wheel metadata files are physically located at the end of the zip file, this allows other tools to modify the metadata without rewriting the entire archive.
|
Sure, that makes sense and is easy to do. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, when running
auditwheel repair, the contents of the output whl file are unpredictable:In both cases, the order is dependent on the order of entries returned by
os.walk.This is a problem for build reproducibility - provided that the build process is sufficiently well defined, different people should be able to run the same process on different machines and get identical outputs.
Note that when
setuptoolsorwheelgenerates a whl file, it does something similar (seeWheelFile.write_filesinwheel.wheelfile.) The code here won't do quite the same as what setuptools does, but that shouldn't be a problem.