feat: add additional s3_settings attributes to DMS endpoint for S3

[bryantbiggs]

Community Note

• Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Closes #13491
Closes #22377

Output from acceptance testing:

$ make testacc TESTARGS='-run=TestAccXXX'
=== RUN   TestAccDMSEndpoint_s3
=== PAUSE TestAccDMSEndpoint_s3
=== CONT  TestAccDMSEndpoint_s3
--- PASS: TestAccDMSEndpoint_s3 (235.01s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/dms	235.224s
...

[zhelding]

Pull request #21306 has significantly refactored the AWS Provider codebase. As a result, most PRs opened prior to the refactor now have merge conflicts that must be resolved before proceeding.

Specifically, PR #21306 relocated the code for all AWS resources and data sources from a single aws directory to a large number of separate directories in internal/service, each corresponding to a particular AWS service. This separation of code has also allowed for us to simplify the names of underlying functions -- while still avoiding namespace collisions.

We recognize that many pull requests have been open for some time without yet being addressed by our maintainers. Therefore, we want to make it clear that resolving these conflicts in no way affects the prioritization of a particular pull request. Once a pull request has been prioritized for review, the necessary changes will be made by a maintainer -- either directly or in collaboration with the pull request author.

For a more complete description of this refactor, including examples of how old filepaths and function names correspond to their new counterparts: please refer to issue #20000.

For a quick guide on how to amend your pull request to resolve the merge conflicts resulting from this refactor and bring it in line with our new code patterns: please refer to our Service Package Refactor Pull Request Guide.

[beau-witter]

Is this a priority at all to get merged? I am running into an issue where a DMS resource that previously was created fine is now failing due to the cdcPath not being set, and I don't see a way to set it through this provider.

[bryantbiggs]

@ewbankkit any tips on how to handle diffs between the AWS Go SDK and the AWS API?

The SDK is setting a const of DatePartitionDelimiterValueSlash = "SLASH" which we use for the default value (likewise with DatePartitionSequenceValueYyyymmdd), but it looks like its creating a dirty diff and failing the tests. I assume this probably happens elsewhere but I couldn't find any examples on how the provider handles this case

=== RUN   TestAccDMSEndpoint_s3
=== PAUSE TestAccDMSEndpoint_s3
=== CONT  TestAccDMSEndpoint_s3
    endpoint_test.go:62: Step 1/3 error: After applying this test step, the plan was not empty.
        stdout:
        
        
        Terraform used the selected providers to generate the following execution
        plan. Resource actions are indicated with the following symbols:
          ~ update in-place
        
        Terraform will perform the following actions:
        
          # aws_dms_endpoint.dms_endpoint will be updated in-place
          ~ resource "aws_dms_endpoint" "dms_endpoint" {
                id                          = "tf-test-dms-endpoint-8mbgk0a1-s3"
                tags                        = {
                    "Name"   = "tf-test-s3-endpoint-8mbgk0a1-s3"
                    "Remove" = "to-remove"
                    "Update" = "to-update"
                }
                # (7 unchanged attributes hidden)
        
              ~ s3_settings {
                  ~ date_partition_delimiter         = "slash" -> "SLASH"
                  ~ date_partition_sequence          = "yyyymmdd" -> "YYYYMMDD"
                    # (27 unchanged attributes hidden)
                }
            }
        
        Plan: 0 to add, 1 to change, 0 to destroy.
--- FAIL: TestAccDMSEndpoint_s3 (203.29s)

[bryantbiggs]

I ended up doing this which seems to jive with existing patterns. doesn't seem "right" but let me know if there is a more preferred way of handling a case like this

[YakDriver]

Looks great 🎉

Output from acceptance tests (us-west-2):

% make testacc TESTS=TestAccDMSEndpoint_ PKG=dms
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/dms/... -v -count 1 -parallel 20 -run='TestAccDMSEndpoint_'  -timeout 180m
--- PASS: TestAccDMSEndpoint_MongoDB_basic (40.78s)
--- PASS: TestAccDMSEndpoint_Oracle_secretID (43.26s)
--- PASS: TestAccDMSEndpoint_PostgreSQL_secretID (43.41s)
--- PASS: TestAccDMSEndpoint_PostgreSQL_update (46.68s)
--- PASS: TestAccDMSEndpoint_OpenSearch_basic (46.71s)
--- PASS: TestAccDMSEndpoint_Oracle_update (47.23s)
--- PASS: TestAccDMSEndpoint_MongoDB_update (47.25s)
--- PASS: TestAccDMSEndpoint_db2 (48.76s)
--- PASS: TestAccDMSEndpoint_docDB (49.58s)
--- PASS: TestAccDMSEndpoint_kafka (49.73s)
--- PASS: TestAccDMSEndpoint_basic (49.73s)
--- PASS: TestAccDMSEndpoint_OpenSearch_errorRetryDuration (50.12s)
--- PASS: TestAccDMSEndpoint_Oracle_basic (50.68s)
--- PASS: TestAccDMSEndpoint_PostgreSQL_basic (50.91s)
--- PASS: TestAccDMSEndpoint_S3_extraConnectionAttributes (52.70s)
--- PASS: TestAccDMSEndpoint_OpenSearch_extraConnectionAttributes (53.42s)
--- PASS: TestAccDMSEndpoint_OpenSearch_fullLoadErrorPercentage (53.57s)
--- PASS: TestAccDMSEndpoint_dynamoDB (58.84s)
--- PASS: TestAccDMSEndpoint_S3_basic (59.53s)
--- PASS: TestAccDMSEndpoint_kinesis (84.54s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/dms	85.951s

Output from acceptance tests (GovCloud):

% make testacc TESTS=TestAccDMSEndpoint_ PKG=dms
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/dms/... -v -count 1 -parallel 20 -run='TestAccDMSEndpoint_'  -timeout 180m
--- PASS: TestAccDMSEndpoint_PostgreSQL_basic (121.37s)
--- PASS: TestAccDMSEndpoint_Oracle_basic (123.93s)
--- PASS: TestAccDMSEndpoint_MongoDB_basic (124.91s)
--- PASS: TestAccDMSEndpoint_S3_extraConnectionAttributes (136.23s)
--- PASS: TestAccDMSEndpoint_PostgreSQL_secretID (136.89s)
--- PASS: TestAccDMSEndpoint_Oracle_secretID (137.78s)
--- PASS: TestAccDMSEndpoint_OpenSearch_extraConnectionAttributes (140.97s)
--- PASS: TestAccDMSEndpoint_OpenSearch_basic (147.02s)
--- PASS: TestAccDMSEndpoint_OpenSearch_errorRetryDuration (147.36s)
--- PASS: TestAccDMSEndpoint_OpenSearch_fullLoadErrorPercentage (147.56s)
--- PASS: TestAccDMSEndpoint_PostgreSQL_update (147.64s)
--- PASS: TestAccDMSEndpoint_kafka (149.32s)
--- PASS: TestAccDMSEndpoint_basic (150.06s)
--- PASS: TestAccDMSEndpoint_Oracle_update (150.08s)
--- PASS: TestAccDMSEndpoint_db2 (150.32s)
--- PASS: TestAccDMSEndpoint_MongoDB_update (151.37s)
--- PASS: TestAccDMSEndpoint_dynamoDB (159.35s)
--- PASS: TestAccDMSEndpoint_docDB (160.74s)
--- PASS: TestAccDMSEndpoint_S3_basic (161.01s)
--- PASS: TestAccDMSEndpoint_kinesis (178.65s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/dms	180.284s

[github-actions]

This functionality has been released in v4.2.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.