Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Force ZIP64 streams #71

Closed
wants to merge 1 commit into from
Closed

Force ZIP64 streams #71

wants to merge 1 commit into from

Conversation

scw
Copy link
Contributor

@scw scw commented Jan 18, 2024

If the archive stream is >2GB, currently the conversion will fail because while the conda_file object will support ZIP64 by default, open still requires an explicit call when the input file size is unknown.

Description

Prior to this change, if trying to convert an archive which exceeds 2GB (see a few packagesfrom the deep learning space), the conversion will fail with a message like:

failed files:
{'input.tar.bz2': 'File size unexpectedly exceeded ZIP64 limit'}

This ensures that the output stream is written as ZIP64 and the 2GB barrier isn't encountered for the archive contents.

If the archive stream is >2GB, currently the conversion will fail because while the `conda_file` object will support ZIP64 by default, `open` still requires an explicit call when the input file size is unknown.
@dholth
Copy link
Contributor

dholth commented May 21, 2024

Nice idea.
Should we check https://github.com/conda/conda-package-streaming/pull/71/files#diff-dc2e8c2bcb91b561ad39dc1a7bc7571592ce4fd273eb8678cb45cafbe651a583L88-L89
(oh, that would be larger than the zstd-compressed data inside the zip)

Do we avoid this problem when creating a .conda directly in conda-package-handling?

@dholth dholth self-requested a review May 21, 2024 16:10
@dholth
Copy link
Contributor

dholth commented Jun 5, 2024

Replaced with #79 , #80

@dholth dholth closed this Jun 5, 2024
@dholth
Copy link
Contributor

dholth commented Jun 6, 2024

Need to check with direct .conda creation also conda/conda-package-handling#248

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

2 participants