-
Notifications
You must be signed in to change notification settings - Fork 634
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Will there be serious memory improvements for the S3 Upload Manager? #343
Comments
Thanks for reaching out to us @Rsm10. We plan to investigate performance improvements for both the S3 upload and download managers. We've not made these refactor changes yet. A high level of the current changes to the v2 SDK can be found on the v2 AWS SDK for Go developer blog Couple areas for improvement:
|
Not so much a performance improvement, but an opportunity for API cleanup that I hope will be considered before v2 is frozen and unable to make breaking changes: aws/aws-sdk-go#2500 (Should I make a separate issue re that and v2?) |
@jasdel so does that mean you would potentially be adding support for aws-chunked? This seems like one obvious way to reduce the amount of data that needs to be kept in memory (instead of needing an entire part in memory for calculating the signature; you just need a (much smaller) chunk). To clarify, my main two problems with the v1 sdk are.
|
@eriksw This issue should definitely be fixed in the V2 SDK. We also need to investigate if this issue can be addressed in the v1 SDK without a breaking change. @Rsm10 Our plans with performance improvements mostly focused on generated API serializes for marshaling and unmarshaling requests removing the need for reflection in that path. The s3 transfer manager performance will be a target for v2. With regard to |
@jasdel Ok, I guess a sort of high level question I'd ask you to think about is: "How would I use this SDK to upload a 5 TB object to AWS?"
|
While reviewing https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html |
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
I intend to give the new developer preview of the go sdk a try, but am wondering if there is a high level description of some of the more fundamental changes you have made (I already looked at the suggestions mentioned in #187). For instance, have their been any improvements with memory consumption? Such as solutions to the problems mentioned here:
aws/aws-sdk-go#2036.
Will there be a way to directly stream data instead of buffering it in memory? Just trying to get an idea of what things I should monitor when giving it a try. Currently the memory issues with the UploadManager in V1 are my biggest problems so hoping something will address that.
Cheers
Robin
The text was updated successfully, but these errors were encountered: