[Docs]: aws_dms_endpoint.cdc_min_file_size should be in Megabytes and stated default should be 32,000

[alordthorsen]

Documentation Link

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/dms_endpoint#cdc_max_batch_interval

Description

Affected Pages

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/dms_endpoint#cdc_max_batch_interval

What is the docs issue?

cdc_min_file_size is listed in megabytes with a default of 32. My team set the value to 400 (in an attempt to get 400 MB files) and we ended up with 400 KB file sizes. If you look at the AWS Docs for cdcMinFileSize under the configuration page you'll see

Minimum file size condition as defined in kilobytes to output a file to Amazon S3.

Default Value: 32000 KB

When cdcMinFileSize is specified and cdcMaxBatchInterval is specified, the file write is triggered by whichever parameter condition is met first.

Which is a strong contradiction from the Terraform docs on this option.

Proposal

Change kilobytes to megabytes and 32 to 32000.

Specifically

[cdc_min_file_size](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/dms_endpoint#cdc_min_file_size) - (Optional) Minimum file size, defined in kilobytes, to reach for a file output. Default is 32000.

References

https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.S3.html#CHAP_Target.S3.Configuring

Would you like to implement a fix?

Yes

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[justinretzolk]

This is an interesting issue that I think might warrant more than a documentation change, so I'm glad you brought it up! In looking into this I found some interesting discrepancies; I think something may have changed with the API at some point.

The aws_dms_endpoint resource sets a default setting of 32 for the the cdc_min_file_size argument, and the documentation calls out that this is in megabytes. It does this because the Go SDK documents the same value as the default:

    // Minimum file size, defined in megabytes, to reach for a file output to Amazon
    // S3.
    //
    // The default value is 32 MB.
    CdcMinFileSize *int64 `type:"integer"`

Looking at the API documentation is what clued me in that this may be a situation where the API has changed. It states:

CdcMinFileSize
Minimum file size, defined in kilobytes, to reach for a file output to Amazon S3.

The default value is 32 MB.

Note that it says "kilobytes" in the first line, but still shows a default value of 32 MB. Moving on to the reference you linked, we see what you mentioned:

cdcMinFileSize
Minimum file size condition as defined in kilobytes to output a file to Amazon S3.

Default Value: 32000 KB

I suspect that what will need to happen is that the default value (for the Terraform resource) should be updated to 32000 and the documentation should be updated to say kilobytes rather than megabytes. I'd like to leave that determination to whomever picks this up to investigate it.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.