archive: Generate crc32 over 16MiB chunks #6274
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Running crc32 over the whole content of the compressed file potentially requires a lot of RAM. The crc32 function in zlib allows for calculating the checksum in chunks. This changes the code to calculate the checksum over 16 MiB chunks instead. 16 MiB is the value also used by shutil.copyfileobj().
ansibullbot
added
bug
felixfontein
reviewed
Apr 2, 2023
changelogs/fragments/6199-archive-generate-checksum-in-chunks.yml
Outdated
Show resolved
Hide resolved
felixfontein
added
check-before-release
felixfontein
reviewed
Apr 2, 2023
changelogs/fragments/6199-archive-generate-checksum-in-chunks.yml
Outdated
Show resolved
Hide resolved
Change the type of change to bugfix Co-authored-by: Felix Fontein <felix@fontein.de>
Co-authored-by: Felix Fontein <felix@fontein.de>
Docs Build 📝Thank you for contribution!✨ This PR has been merged and your docs changes will be incorporated when they are next published. |
felixfontein
approved these changes
Apr 5, 2023
felixfontein
removed
the
check-before-release
Backport to stable-5: 💚 backport PR created✅ Backport PR branch: Backported as #6325 🤖 @patchback |
patchback bot
pushed a commit
that referenced
this pull request
Apr 13, 2023
* archive: Generate crc32 over 16MiB chunks Running crc32 over the whole content of the compressed file potentially requires a lot of RAM. The crc32 function in zlib allows for calculating the checksum in chunks. This changes the code to calculate the checksum over 16 MiB chunks instead. 16 MiB is the value also used by shutil.copyfileobj(). * Update changelogs/fragments/6199-archive-generate-checksum-in-chunks.yml Change the type of change to bugfix Co-authored-by: Felix Fontein <felix@fontein.de> * Update changelogs/fragments/6199-archive-generate-checksum-in-chunks.yml Co-authored-by: Felix Fontein <felix@fontein.de> --------- Co-authored-by: Felix Fontein <felix@fontein.de> (cherry picked from commit 14b19af)
Backport to stable-6: 💚 backport PR created✅ Backport PR branch: Backported as #6326 🤖 @patchback |
|
@nilsmeyer thanks for your contribution! |
patchback bot
pushed a commit
that referenced
this pull request
Apr 13, 2023
* archive: Generate crc32 over 16MiB chunks Running crc32 over the whole content of the compressed file potentially requires a lot of RAM. The crc32 function in zlib allows for calculating the checksum in chunks. This changes the code to calculate the checksum over 16 MiB chunks instead. 16 MiB is the value also used by shutil.copyfileobj(). * Update changelogs/fragments/6199-archive-generate-checksum-in-chunks.yml Change the type of change to bugfix Co-authored-by: Felix Fontein <felix@fontein.de> * Update changelogs/fragments/6199-archive-generate-checksum-in-chunks.yml Co-authored-by: Felix Fontein <felix@fontein.de> --------- Co-authored-by: Felix Fontein <felix@fontein.de> (cherry picked from commit 14b19af)
|
@felixfontein Thanks for your valuable input and review. 10/10, would pull request again ;) |
felixfontein
pushed a commit
that referenced
this pull request
Apr 13, 2023
…6MiB chunks (#6325) archive: Generate crc32 over 16MiB chunks (#6274) * archive: Generate crc32 over 16MiB chunks Running crc32 over the whole content of the compressed file potentially requires a lot of RAM. The crc32 function in zlib allows for calculating the checksum in chunks. This changes the code to calculate the checksum over 16 MiB chunks instead. 16 MiB is the value also used by shutil.copyfileobj(). * Update changelogs/fragments/6199-archive-generate-checksum-in-chunks.yml Change the type of change to bugfix Co-authored-by: Felix Fontein <felix@fontein.de> * Update changelogs/fragments/6199-archive-generate-checksum-in-chunks.yml Co-authored-by: Felix Fontein <felix@fontein.de> --------- Co-authored-by: Felix Fontein <felix@fontein.de> (cherry picked from commit 14b19af) Co-authored-by: Nils Meyer <nils@nm.cx>
felixfontein
pushed a commit
that referenced
this pull request
Apr 13, 2023
…6MiB chunks (#6326) archive: Generate crc32 over 16MiB chunks (#6274) * archive: Generate crc32 over 16MiB chunks Running crc32 over the whole content of the compressed file potentially requires a lot of RAM. The crc32 function in zlib allows for calculating the checksum in chunks. This changes the code to calculate the checksum over 16 MiB chunks instead. 16 MiB is the value also used by shutil.copyfileobj(). * Update changelogs/fragments/6199-archive-generate-checksum-in-chunks.yml Change the type of change to bugfix Co-authored-by: Felix Fontein <felix@fontein.de> * Update changelogs/fragments/6199-archive-generate-checksum-in-chunks.yml Co-authored-by: Felix Fontein <felix@fontein.de> --------- Co-authored-by: Felix Fontein <felix@fontein.de> (cherry picked from commit 14b19af) Co-authored-by: Nils Meyer <nils@nm.cx>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
SUMMARY
Running crc32 over the whole content of the compressed file potentially requires a lot of RAM. The crc32 function in zlib allows for calculating the checksum in chunks. This changes the code to calculate the checksum over 16 MiB chunks instead. 16 MiB is the value also used by shutil.copyfileobj().
Fixes: #6272
ISSUE TYPE
COMPONENT NAME
archive
ADDITIONAL INFORMATION