Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Add additional checksum algorithms in aws_s3_object resource and data source #27370

Conversation

mediba-h-sugawara
Copy link

Description

Add additional checksum algorithms (CRC32, CRC32C, SHA1, SHA256) to S3 object resource and data source.

Relations

Related #23901
Related #25647

References

Output from Acceptance Testing

$ make testacc TESTS=TestAccS3Object_checksum\|TestAccS3Object_source\|TestAccS3ObjectDataSource_checksum\|TestAccS3ObjectDataSource_allParams PKG=s3
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/s3/... -v -count 1 -parallel 20 -run='TestAccS3Object_checksum|TestAccS3Object_source|TestAccS3ObjectDataSource_checksum|TestAccS3ObjectDataSource_allParams'  -timeout 180m
=== RUN   TestAccS3ObjectDataSource_allParams
=== PAUSE TestAccS3ObjectDataSource_allParams
=== RUN   TestAccS3ObjectDataSource_checksumCRC32
=== PAUSE TestAccS3ObjectDataSource_checksumCRC32
=== RUN   TestAccS3ObjectDataSource_checksumCRC32C
=== PAUSE TestAccS3ObjectDataSource_checksumCRC32C
=== RUN   TestAccS3ObjectDataSource_checksumSHA1
=== PAUSE TestAccS3ObjectDataSource_checksumSHA1
=== RUN   TestAccS3ObjectDataSource_checksumSHA256
=== PAUSE TestAccS3ObjectDataSource_checksumSHA256
=== RUN   TestAccS3Object_source
=== PAUSE TestAccS3Object_source
=== RUN   TestAccS3Object_sourceHashTrigger
=== PAUSE TestAccS3Object_sourceHashTrigger
=== RUN   TestAccS3Object_checksumCRC32
=== PAUSE TestAccS3Object_checksumCRC32
=== RUN   TestAccS3Object_checksumCRC32FromContent
=== PAUSE TestAccS3Object_checksumCRC32FromContent
=== RUN   TestAccS3Object_checksumCRC32C
=== PAUSE TestAccS3Object_checksumCRC32C
=== RUN   TestAccS3Object_checksumCRC32CFromContent
=== PAUSE TestAccS3Object_checksumCRC32CFromContent
=== RUN   TestAccS3Object_checksumSHA1
=== PAUSE TestAccS3Object_checksumSHA1
=== RUN   TestAccS3Object_checksumSHA1FromContent
=== PAUSE TestAccS3Object_checksumSHA1FromContent
=== RUN   TestAccS3Object_checksumSHA256
=== PAUSE TestAccS3Object_checksumSHA256
=== RUN   TestAccS3Object_checksumSHA256FromContent
=== PAUSE TestAccS3Object_checksumSHA256FromContent
=== CONT  TestAccS3ObjectDataSource_allParams
=== CONT  TestAccS3Object_checksumCRC32FromContent
=== CONT  TestAccS3Object_checksumSHA1FromContent
=== CONT  TestAccS3Object_checksumSHA256FromContent
=== CONT  TestAccS3Object_checksumSHA1
=== CONT  TestAccS3Object_checksumSHA256
=== CONT  TestAccS3Object_checksumCRC32C
=== CONT  TestAccS3ObjectDataSource_checksumSHA256
=== CONT  TestAccS3Object_checksumCRC32CFromContent
=== CONT  TestAccS3ObjectDataSource_checksumCRC32C
=== CONT  TestAccS3Object_checksumCRC32
=== CONT  TestAccS3ObjectDataSource_checksumSHA1
=== CONT  TestAccS3Object_sourceHashTrigger
=== CONT  TestAccS3Object_source
=== CONT  TestAccS3ObjectDataSource_checksumCRC32
--- PASS: TestAccS3ObjectDataSource_checksumSHA256 (202.11s)
--- PASS: TestAccS3ObjectDataSource_checksumCRC32C (202.27s)
--- PASS: TestAccS3ObjectDataSource_checksumCRC32 (208.73s)
--- PASS: TestAccS3ObjectDataSource_checksumSHA1 (208.74s)
--- PASS: TestAccS3ObjectDataSource_allParams (213.26s)
--- PASS: TestAccS3Object_checksumSHA256FromContent (224.77s)
--- PASS: TestAccS3Object_checksumSHA1FromContent (234.12s)
--- PASS: TestAccS3Object_checksumCRC32FromContent (234.42s)
--- PASS: TestAccS3Object_checksumCRC32CFromContent (234.71s)
--- PASS: TestAccS3Object_source (235.57s)
--- PASS: TestAccS3Object_sourceHashTrigger (293.14s)
--- PASS: TestAccS3Object_checksumCRC32C (294.43s)
--- PASS: TestAccS3Object_checksumSHA1 (295.33s)
--- PASS: TestAccS3Object_checksumCRC32 (296.25s)
--- PASS: TestAccS3Object_checksumSHA256 (297.32s)
PASS
ok      github.com/hashicorp/terraform-provider-aws/internal/service/s3 300.476s

1. Add arguments checksum_algorithm to aws_s3_object resource.
2. Add attributes checksum_crc32, checksum_crc32c, checksum_sha1 and checksum_sha256 to aws_s3_object resource.
3. Calculate checksum from value of argument checksum_algorithm.
4. Enable ChecksumMode at s3.HeadObjectInput in order to retrieve checksum.
@github-actions
Copy link

Community Note

Voting for Prioritization

  • Please vote on this pull request by adding a 👍 reaction to the original post to help the community and maintainers prioritize this pull request.
  • Please see our prioritization guide for information on how we prioritize.
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

For Submitters

  • Review the contribution guide relating to the type of change you are making to ensure all of the necessary steps have been taken.
  • For new resources and data sources, use skaff to generate scaffolding with comments detailing common expectations.
  • Whether or not the branch has been rebased will not impact prioritization, but doing so is always a welcome surprise.

@github-actions github-actions bot added documentation Introduces or discusses updates to documentation. service/s3 Issues and PRs that pertain to the s3 service. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. needs-triage Waiting for first response or review from a maintainer. size/XL Managed by automation to categorize the size of a PR. labels Oct 21, 2022
Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Welcome @mediba-h-sugawara 👋

It looks like this is your first Pull Request submission to the Terraform AWS Provider! If you haven’t already done so please make sure you have checked out our CONTRIBUTOR guide and FAQ to make sure your contribution is adhering to best practice and has all the necessary elements in place for a successful approval.

Also take a look at our FAQ which details how we prioritize Pull Requests for inclusion.

Thanks again, and welcome to the community! 😃

@justinretzolk justinretzolk added enhancement Requests to existing resources that expand the functionality or scope. and removed needs-triage Waiting for first response or review from a maintainer. labels Oct 21, 2022
@md5
Copy link
Contributor

md5 commented Oct 30, 2022

Speaking as a user of this provider and an interested third party, this looks like a nice addition. That being said, there were a couple of things that surprised me about this PR.

The first thing was the addition of code in the provider itself to calculate the checksums for the new algorithms. Based on the linked blog post, it looks like this functionality was added to the SDKs. Indeed, it looks like it was added to the AWS SDK for Go V2 in aws/aws-sdk-go-v2#1600, but it does not seem to have been added to the V1 SDK. Since the aws_s3_object resource and data source are using the V1 SDK, I understand why the choice to add the checksum code here was made, but perhaps it would be better to move to the V2 SDK?

The other thing that I noticed is that this PR does not pass along user-provided values for the checksum_* fields other than checksum_algorithm. The linked blog post shows an example of passing on a precomputed checksum using the Python SDK and I think this would be helpful for the use case where a user has already stored the checksum alongside the file to be uploaded, particularly in the case of large objects.

I think if these files were updated to use the V2 SDK, the resourceObjectUpload function could be simplified to just pass all the checksum_* fields to the SDK without looking at their values at all. If you support user-provided values for the checksum fields, hasObjectContentChanges will also need to be updated, since right now it is only checking for changes to checksum_algorithm.

@mediba-h-sugawara
Copy link
Author

Thanks for your comment! 👍
I refactor from v1 sdk to v2 sdk now. 😎

@mediba-h-sugawara mediba-h-sugawara changed the title Add additional checksum algorithms in aws_s3_object resource and data source [WIP] Add additional checksum algorithms in aws_s3_object resource and data source Oct 31, 2022
@md5
Copy link
Contributor

md5 commented Oct 31, 2022

Hopefully that dependency update is a straightforward change to make with no unforseen downsides. The maintainers would know best if there's a specific reason that change has not been made previously.

It looks like you may also want to add ConflictsWith for all the new checksum_* properties based on that blog post. It seems to only make sense to specify checksum_algorithm when you do not provide a precomputed value or to only specify the precomputed checksum for a single algorithm, but that may be an incorrect assumption on my part.

@adamMo
Copy link

adamMo commented Nov 26, 2022

hey @mediba-h-sugawara do you have some news?

@github-actions
Copy link

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 13, 2023
@mediba-h-sugawara mediba-h-sugawara deleted the f-aws_s3_object-additional_checksum_algorithms branch October 13, 2023 08:40
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
documentation Introduces or discusses updates to documentation. enhancement Requests to existing resources that expand the functionality or scope. service/s3 Issues and PRs that pertain to the s3 service. size/XL Managed by automation to categorize the size of a PR. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants