Add streamed file upload #75
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
frostyfab
pushed a commit
to frostyfab/proxmox-api-go
that referenced
this pull request
Nov 17, 2023
Add streamed file upload
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When uploading files, this library copies first the whole file in memory in order to build the POST body for the upload request.
Problem: large files may cause issues like the API consumer running out of memory. I have noticed this, for instance, when trying to upload some Windows ISOs using the Proxmox builder for Packer.
Solution in this PR: when possible, use MultiReader to stream the file directly from the reader into the request, without previously storing it in memory. In order to do so, the ContentLength header needs to be manually calculated and specified. To avoid breaking the API, this is only performed in case the Reader passed as the argument to Upload is a File: this way the file size can be obtained from the OS.
Of course, this PR open to any suggestions.