d/aws_s3_bucket_object: Don't crash when S3 HeadObject returns empty response #14154

ewbankkit · 2020-07-12T22:38:51Z

Community Note

Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Closes #14151.
Relates #12992.
Relates #7584.

Release note for CHANGELOG:

data-source/aws_s3_bucket_object: Don't crash when S3 `HeadObject` API returns empty response.

Output from acceptance testing:

$ make testacc TEST=./aws/ TESTARGS='-run=TestAccDataSourceAWSS3BucketObject_' 
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws/ -v -count 1 -parallel 20 -run=TestAccDataSourceAWSS3BucketObject_ -timeout 120m
=== RUN   TestAccDataSourceAWSS3BucketObject_basic
=== PAUSE TestAccDataSourceAWSS3BucketObject_basic
=== RUN   TestAccDataSourceAWSS3BucketObject_basicViaAccessPoint
=== PAUSE TestAccDataSourceAWSS3BucketObject_basicViaAccessPoint
=== RUN   TestAccDataSourceAWSS3BucketObject_readableBody
=== PAUSE TestAccDataSourceAWSS3BucketObject_readableBody
=== RUN   TestAccDataSourceAWSS3BucketObject_kmsEncrypted
=== PAUSE TestAccDataSourceAWSS3BucketObject_kmsEncrypted
=== RUN   TestAccDataSourceAWSS3BucketObject_allParams
=== PAUSE TestAccDataSourceAWSS3BucketObject_allParams
=== RUN   TestAccDataSourceAWSS3BucketObject_ObjectLockLegalHoldOff
=== PAUSE TestAccDataSourceAWSS3BucketObject_ObjectLockLegalHoldOff
=== RUN   TestAccDataSourceAWSS3BucketObject_ObjectLockLegalHoldOn
=== PAUSE TestAccDataSourceAWSS3BucketObject_ObjectLockLegalHoldOn
=== RUN   TestAccDataSourceAWSS3BucketObject_LeadingSlash
=== PAUSE TestAccDataSourceAWSS3BucketObject_LeadingSlash
=== RUN   TestAccDataSourceAWSS3BucketObject_MultipleSlashes
=== PAUSE TestAccDataSourceAWSS3BucketObject_MultipleSlashes
=== RUN   TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey
=== PAUSE TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey
=== CONT  TestAccDataSourceAWSS3BucketObject_basic
=== CONT  TestAccDataSourceAWSS3BucketObject_ObjectLockLegalHoldOn
=== CONT  TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey
=== CONT  TestAccDataSourceAWSS3BucketObject_ObjectLockLegalHoldOff
=== CONT  TestAccDataSourceAWSS3BucketObject_MultipleSlashes
=== CONT  TestAccDataSourceAWSS3BucketObject_allParams
=== CONT  TestAccDataSourceAWSS3BucketObject_LeadingSlash
=== CONT  TestAccDataSourceAWSS3BucketObject_kmsEncrypted
=== CONT  TestAccDataSourceAWSS3BucketObject_readableBody
=== CONT  TestAccDataSourceAWSS3BucketObject_basicViaAccessPoint
--- PASS: TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey (42.12s)
--- PASS: TestAccDataSourceAWSS3BucketObject_basic (45.22s)
--- PASS: TestAccDataSourceAWSS3BucketObject_readableBody (47.39s)
--- PASS: TestAccDataSourceAWSS3BucketObject_allParams (47.41s)
--- PASS: TestAccDataSourceAWSS3BucketObject_ObjectLockLegalHoldOff (48.20s)
--- PASS: TestAccDataSourceAWSS3BucketObject_kmsEncrypted (48.49s)
--- PASS: TestAccDataSourceAWSS3BucketObject_basicViaAccessPoint (48.81s)
--- PASS: TestAccDataSourceAWSS3BucketObject_ObjectLockLegalHoldOn (49.41s)
--- PASS: TestAccDataSourceAWSS3BucketObject_LeadingSlash (67.87s)
--- PASS: TestAccDataSourceAWSS3BucketObject_MultipleSlashes (68.27s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	68.374s

aws/internal/keyvaluetags/s3_tags.go

DrFaust92

LGTM

…eSlashAsKey'. Acceptance test output: $ make testacc TEST=./aws/ TESTARGS='-run=TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey' ==> Checking that code complies with gofmt requirements... TF_ACC=1 go test ./aws/ -v -count 1 -parallel 20 -run=TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey -timeout 120m === RUN TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey === PAUSE TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey === CONT TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x3d83617] goroutine 867 [running]: github.com/terraform-providers/terraform-provider-aws/aws.dataSourceAwsS3BucketObjectRead(0xc00040b9d0, 0x5e9e540, 0xc0003fca00, 0xc00040b9d0, 0x0) /home/kit/wrk/src/github.com/terraform-providers/terraform-provider-aws/aws/data_source_aws_s3_bucket_object.go:168 +0x887 github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Resource).ReadDataApply(0xc0001478c0, 0xc000effb80, 0x5e9e540, 0xc0003fca00, 0xc00003b920, 0x1, 0x0) /home/kit/wrk/pkg/mod/github.com/hashicorp/terraform-plugin-sdk@v1.14.0/helper/schema/resource.go:403 +0x88 github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Provider).ReadDataApply(0xc00012a200, 0xc000cdba88, 0xc000effb80, 0xc000effb80, 0x0, 0x0) /home/kit/wrk/pkg/mod/github.com/hashicorp/terraform-plugin-sdk@v1.14.0/helper/schema/provider.go:451 +0x8f github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin.(*GRPCProviderServer).ReadDataSource(0xc0017fd758, 0x796f360, 0xc0018d4960, 0xc00162f500, 0xc0017fd758, 0xc0018d4960, 0xc00008cb30) /home/kit/wrk/pkg/mod/github.com/hashicorp/terraform-plugin-sdk@v1.14.0/internal/helper/plugin/grpc_provider.go:1036 +0x44e github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5._Provider_ReadDataSource_Handler(0x68e0540, 0xc0017fd758, 0x796f360, 0xc0018d4960, 0xc002672240, 0x0, 0x796f360, 0xc0018d4960, 0xc0005b2e00, 0x1ab) /home/kit/wrk/pkg/mod/github.com/hashicorp/terraform-plugin-sdk@v1.14.0/internal/tfplugin5/tfplugin5.pb.go:3341 +0x217 google.golang.org/grpc.(*Server).processUnaryRPC(0xc0020ae780, 0x799ef80, 0xc0020ae900, 0xc000f63c00, 0xc0020be1e0, 0xb6896f0, 0x0, 0x0, 0x0) /home/kit/wrk/pkg/mod/google.golang.org/grpc@v1.27.1/server.go:1024 +0x4f4 google.golang.org/grpc.(*Server).handleStream(0xc0020ae780, 0x799ef80, 0xc0020ae900, 0xc000f63c00, 0x0) /home/kit/wrk/pkg/mod/google.golang.org/grpc@v1.27.1/server.go:1313 +0xd97 google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc001f8ee70, 0xc0020ae780, 0x799ef80, 0xc0020ae900, 0xc000f63c00) /home/kit/wrk/pkg/mod/google.golang.org/grpc@v1.27.1/server.go:722 +0xbb created by google.golang.org/grpc.(*Server).serveStreams.func1 /home/kit/wrk/pkg/mod/google.golang.org/grpc@v1.27.1/server.go:720 +0xa1 FAIL github.com/terraform-providers/terraform-provider-aws/aws 24.265s FAIL GNUmakefile:26: recipe for target 'testacc' failed make: *** [testacc] Error 1

Failing acceptance test: $ make testacc TEST=./aws/ TESTARGS='-run=TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey' ==> Checking that code complies with gofmt requirements... TF_ACC=1 go test ./aws/ -v -count 1 -parallel 20 -run=TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey -timeout 120m === RUN TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey === PAUSE TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey === CONT TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey --- FAIL: TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey (24.58s) testing.go:684: Step 0 error: errors during apply: Error: error listing tags for S3 Bucket (tf-acc-test-2642544354113174819) Object (/): NoSuchTagSet: The TagSet does not exist status code: 404, request id: 3ADF2BE8D6DAD57E, host id: zIz72pHcPivqgl0t79J/9kF02E5N7ibZy3Rof+ggH2+Y3sffOQsjRQVkP5B/PinBmLJnI1nfvCk= on /tmp/tf-test734233549/main.tf line 6: (source code not available) testing.go:745: Error destroying resource! WARNING: Dangling resources may exist. The full state and error is shown below. Error: errors during refresh: error listing tags for S3 Bucket (tf-acc-test-2642544354113174819) Object (/): NoSuchTagSet: The TagSet does not exist status code: 404, request id: 9746D2A0387F2F59, host id: tCpkS+zPf8miGLqN8BaDPGbPBs03PZoC3iJHWvhd3SeN7k/Y1BD1AKJmujxgfBqGshau1OP9Pu4= State: <nil> FAIL FAIL github.com/terraform-providers/terraform-provider-aws/aws 24.654s FAIL GNUmakefile:26: recipe for target 'testacc' failed make: *** [testacc] Error 1

…ponse. Acceptance test output: $ make testacc TEST=./aws/ TESTARGS='-run=TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey' ==> Checking that code complies with gofmt requirements... TF_ACC=1 go test ./aws/ -v -count 1 -parallel 20 -run=TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey -timeout 120m === RUN TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey === PAUSE TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey === CONT TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey --- PASS: TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey (40.69s) PASS ok github.com/terraform-providers/terraform-provider-aws/aws 40.749s

…eSlashAsKey'. Acceptance test output: $ make testacc TEST=./aws/ TESTARGS='-run=TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey' ==> Checking that code complies with gofmt requirements... TF_ACC=1 go test ./aws/ -v -count 1 -parallel 20 -run=TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey -timeout 120m === RUN TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey === PAUSE TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey === CONT TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x3d83617] goroutine 867 [running]: github.com/terraform-providers/terraform-provider-aws/aws.dataSourceAwsS3BucketObjectRead(0xc00040b9d0, 0x5e9e540, 0xc0003fca00, 0xc00040b9d0, 0x0) /home/kit/wrk/src/github.com/terraform-providers/terraform-provider-aws/aws/data_source_aws_s3_bucket_object.go:168 +0x887 github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Resource).ReadDataApply(0xc0001478c0, 0xc000effb80, 0x5e9e540, 0xc0003fca00, 0xc00003b920, 0x1, 0x0) /home/kit/wrk/pkg/mod/github.com/hashicorp/terraform-plugin-sdk@v1.14.0/helper/schema/resource.go:403 +0x88 github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Provider).ReadDataApply(0xc00012a200, 0xc000cdba88, 0xc000effb80, 0xc000effb80, 0x0, 0x0) /home/kit/wrk/pkg/mod/github.com/hashicorp/terraform-plugin-sdk@v1.14.0/helper/schema/provider.go:451 +0x8f github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin.(*GRPCProviderServer).ReadDataSource(0xc0017fd758, 0x796f360, 0xc0018d4960, 0xc00162f500, 0xc0017fd758, 0xc0018d4960, 0xc00008cb30) /home/kit/wrk/pkg/mod/github.com/hashicorp/terraform-plugin-sdk@v1.14.0/internal/helper/plugin/grpc_provider.go:1036 +0x44e github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5._Provider_ReadDataSource_Handler(0x68e0540, 0xc0017fd758, 0x796f360, 0xc0018d4960, 0xc002672240, 0x0, 0x796f360, 0xc0018d4960, 0xc0005b2e00, 0x1ab) /home/kit/wrk/pkg/mod/github.com/hashicorp/terraform-plugin-sdk@v1.14.0/internal/tfplugin5/tfplugin5.pb.go:3341 +0x217 google.golang.org/grpc.(*Server).processUnaryRPC(0xc0020ae780, 0x799ef80, 0xc0020ae900, 0xc000f63c00, 0xc0020be1e0, 0xb6896f0, 0x0, 0x0, 0x0) /home/kit/wrk/pkg/mod/google.golang.org/grpc@v1.27.1/server.go:1024 +0x4f4 google.golang.org/grpc.(*Server).handleStream(0xc0020ae780, 0x799ef80, 0xc0020ae900, 0xc000f63c00, 0x0) /home/kit/wrk/pkg/mod/google.golang.org/grpc@v1.27.1/server.go:1313 +0xd97 google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc001f8ee70, 0xc0020ae780, 0x799ef80, 0xc0020ae900, 0xc000f63c00) /home/kit/wrk/pkg/mod/google.golang.org/grpc@v1.27.1/server.go:722 +0xbb created by google.golang.org/grpc.(*Server).serveStreams.func1 /home/kit/wrk/pkg/mod/google.golang.org/grpc@v1.27.1/server.go:720 +0xa1 FAIL github.com/terraform-providers/terraform-provider-aws/aws 24.265s FAIL GNUmakefile:26: recipe for target 'testacc' failed make: *** [testacc] Error 1

…ponse. Acceptance test output: $ make testacc TEST=./aws/ TESTARGS='-run=TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey' ==> Checking that code complies with gofmt requirements... TF_ACC=1 go test ./aws/ -v -count 1 -parallel 20 -run=TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey -timeout 120m === RUN TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey === PAUSE TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey === CONT TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey --- PASS: TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey (40.69s) PASS ok github.com/terraform-providers/terraform-provider-aws/aws 40.749s

Co-authored-by: Brian Flad <bflad417@gmail.com>

ewbankkit · 2020-10-06T15:24:57Z

Not sure why the PR blew up in size there but I have rebased and now it's back down to size/S.
Re-ran acceptance tests:

$ make testacc TEST=./aws TESTARGS='-run=TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey -timeout 120m
=== RUN   TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey
=== PAUSE TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey
=== CONT  TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey
--- PASS: TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey (18.81s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	18.888s

bflad

Looks good to me 🚀

Output from acceptance testing:

--- PASS: TestAccDataSourceAWSS3BucketObject_kmsEncrypted (53.21s)
--- PASS: TestAccDataSourceAWSS3BucketObject_readableBody (53.78s)
--- PASS: TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey (54.16s)
--- PASS: TestAccDataSourceAWSS3BucketObject_basic (55.33s)
--- PASS: TestAccDataSourceAWSS3BucketObject_ObjectLockLegalHoldOff (55.08s)
--- PASS: TestAccDataSourceAWSS3BucketObject_ObjectLockLegalHoldOn (56.63s)
--- PASS: TestAccDataSourceAWSS3BucketObject_allParams (57.27s)
--- PASS: TestAccDataSourceAWSS3BucketObject_basicViaAccessPoint (59.73s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_startAfter (74.51s)
--- PASS: TestAccDataSourceAWSS3BucketObject_MultipleSlashes (76.46s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_fetchOwner (76.46s)
--- PASS: TestAccDataSourceAWSS3BucketObject_LeadingSlash (78.07s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_maxKeys (78.09s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_basicViaAccessPoint (77.91s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_basic (78.31s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_all (79.05s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_prefixes (78.72s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_encoded (78.60s)

ghost · 2020-10-09T16:18:32Z

This has been released in version 3.10.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

ghost · 2020-11-06T17:10:07Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

ewbankkit requested a review from a team July 12, 2020 22:38

ewbankkit commented Jul 12, 2020

View reviewed changes

aws/internal/keyvaluetags/s3_tags.go Outdated Show resolved Hide resolved

ewbankkit added bug Addresses a defect in current functionality. crash Results from or addresses a Terraform crash or kernel panic. and removed provider Pertains to the provider itself, rather than any interaction with AWS. labels Jul 12, 2020

anGie44 force-pushed the master branch from 54db70d to 48ba2f7 Compare July 21, 2020 18:26

DrFaust92 requested review from DrFaust92 and removed request for a team July 31, 2020 21:09

DrFaust92 approved these changes Aug 24, 2020

View reviewed changes

ewbankkit added 3 commits October 6, 2020 09:45

ghost added the provider Pertains to the provider itself, rather than any interaction with AWS. label Oct 6, 2020

bflad self-assigned this Oct 6, 2020

ghost added size/XXL Managed by automation to categorize the size of a PR. and removed size/S Managed by automation to categorize the size of a PR. labels Oct 6, 2020

ewbankkit and others added 3 commits October 6, 2020 11:19

Update aws/internal/keyvaluetags/s3_tags.go

7123ffd

Co-authored-by: Brian Flad <bflad417@gmail.com>

ewbankkit force-pushed the b-aws_s3_bucket_object-data-source-crash-on-nil-etag branch from c9bbf98 to 7123ffd Compare October 6, 2020 15:21

ghost added size/S Managed by automation to categorize the size of a PR. and removed size/XXL Managed by automation to categorize the size of a PR. labels Oct 6, 2020

Bump CI.

ee23f2a

bflad added this to the v3.10.0 milestone Oct 6, 2020

bflad approved these changes Oct 6, 2020

View reviewed changes

bflad merged commit a6a0a96 into hashicorp:master Oct 6, 2020

bflad added a commit that referenced this pull request Oct 6, 2020

Update CHANGELOG for #14154

85d8cda

ewbankkit deleted the b-aws_s3_bucket_object-data-source-crash-on-nil-etag branch October 6, 2020 18:48

ghost locked as resolved and limited conversation to collaborators Nov 6, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

d/aws_s3_bucket_object: Don't crash when S3 HeadObject returns empty response #14154

d/aws_s3_bucket_object: Don't crash when S3 HeadObject returns empty response #14154

ewbankkit commented Jul 12, 2020

DrFaust92 left a comment

ewbankkit commented Oct 6, 2020

bflad left a comment

ghost commented Oct 9, 2020

ghost commented Nov 6, 2020

d/aws_s3_bucket_object: Don't crash when S3 HeadObject returns empty response #14154

d/aws_s3_bucket_object: Don't crash when S3 HeadObject returns empty response #14154

Conversation

ewbankkit commented Jul 12, 2020

Community Note

DrFaust92 left a comment

Choose a reason for hiding this comment

ewbankkit commented Oct 6, 2020

bflad left a comment

Choose a reason for hiding this comment

ghost commented Oct 9, 2020

ghost commented Nov 6, 2020