Add option to configure secrets manager secret for aws_dms_endpoint

[jan-ludvik-deltatre]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

The aws_dms_endpoint allows to configure AWS Secrets manager secret with database details like host, user, password. Please add this to the resource.

New or Affected Resource(s)

• aws_dms_endpoint

Potential Terraform Configuration

References

[nkchegg]

Yes, please.

We should NOT store the username/password for our databases in the plain text, even if it's a read-only access to get the data out.

[mwarkentin]

References:

• Announcement
• Documentation

[antonakv]

Your question is explained in the documentation:

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/secretsmanager_secret_rotation
https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets.html

More specific topic:
https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets-built-in.html

You can follow To turn on rotation (console) section of the article.

Or you can follow AWS SDK and AWS CLI section and using AWS CLI examples you can create equivalent ones in terraform.

It's not a trivial procedure and requires understanding how it works:

Secrets Manager can automatically rotate them on a schedule. When it rotates a secret, Secrets Manager updates the credentials in both the secret and the database or service so that you don't have to manually change the credentials. Secrets Manager uses a Lambda rotation function to communicate with both Secrets Manager and the database or service. The rotation function:

Calls the Secrets Manager API to retrieve and update secrets.

Sends requests to the database or service to update the user password.
So, To enable automatic secret rotation, the Secrets Manager service requires usage of a Lambda function. The Rotate Secrets section in the Secrets Manager User Guide provides additional information about deploying a prebuilt Lambda functions for supported credential rotation (e.g. RDS) or deploying a custom Lambda function.

NOTE:
Configuring rotation causes the secret to rotate once as soon as you enable rotation. Before you do this, you must ensure that all of your applications that use the credentials stored in the secret are updated to retrieve the secret from AWS Secrets Manager. The old credentials might no longer be usable after the initial rotation and any applications that you fail to update will break as soon as the old credentials are no longer valid.
NOTE:
If you cancel a rotation that is in progress (by removing the rotation configuration), it can leave the VersionStage labels in an unexpected state. Depending on what step of the rotation was in progress, you might need to remove the staging label AWSPENDING from the partially created version, specified by the SecretVersionId response value. You should also evaluate the partially rotated new version to see if it should be deleted, which you can do by removing all staging labels from the new version's VersionStage field.
Please take into account extra information about Lambda function which you should create for password rotation:

If your Lambda function runs in a VPC, then to allow it to communicate with Secrets Manager, you have two options:

You can enable your Lambda function to access the public Secrets Manager endpoint by adding a NAT gateway to your VPC, which allows traffic from your VPC to reach the public endpoint. This exposes your VPC to a level of risk because an IP address for the gateway can be attacked from the public Internet.

You can configure Secrets Manager service endpoints within your VPC. This allows your VPC to intercept any request addressed to the public regional endpoint and redirect the VPC to the private service endpoint running within your VPC. For more information, see VPC endpoints.

[jan-ludvik-deltatre]

@antonakv I think there has been a misunderstanding and I don't think the issue deserves a thumbs down - this is not about rotating Secrets manager secrets but adding an option to reference a secrets manager secret id in AWS DMS endpoint in Terraform.

I understand we can use Secrets Manager to also rotate the secrets but that's not what this issue is about - I wanted to just reference the SM secret ID in the DMS endpoint configuration in TF so DMS can get details about source/target from there.

That AWS announcement does talk about rotating secrets in DMS endpoints, but you don't really need to do that, I used this feature (manually in GUI) without any kind of rotation, just to store source and target credentials and not have them in committed in plain text in the resource

[sbrandtb]

We use to "manage" those endpoints using following code:

resource "aws_dms_endpoint" "this" {
  endpoint_id   = var.name
  endpoint_type = var.type
  engine_name   = var.engine

  database_name = var.database_name

  //  password = "unused"
  //  username = "unused"
  //  server_name = "unused"
  //  port = 1234

  extra_connection_attributes = join(";", [
    "secretsManagerAccessRoleArn=${module.credentials.access_role_arn}",
    "secretsManagerEndpointOverride=${var.secretsmanager_vpc_endpoint_dns}",
    "secretsManagerSecretId=${module.credentials.secret_id}",
    "", // for trailing ;
  ])

  lifecycle {
    ignore_changes = [password]
  }
}

(module.credentials has a data resource for the secret, creates a role that can access it and exposes both secret ID and role ARN)

Process is:

• Uncomment the four username/password/server_name/port fields
• Create the resource using Terraform
• Manually set up secretsmanager for the connection, i.e. in the AWS console
• Comment in the four fields. Terraform should not report any changes since the extra connection attributes are those you just implicitly set up.

[breathingdust]

Hi all 👋 Just letting you know that this is issue is featured on this quarters roadmap. If a PR exists to close the issue a maintainer will review and either make changes directly, or work with the original author to get the contribution merged. If you have written a PR to resolve the issue please ensure the "Allow edits from maintainers" box is checked. Thanks for your patience and we are looking forward to getting this merged soon!

[github-actions]

This functionality has been released in v3.70.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[blanco750]

I am trying to use secrets manager to store credentials .

  secrets_manager_arn         = aws_secretsmanager_secret.source_endpoint.arn
  ssl_mode                    = "none"
  #Below are all commented out . All are defined in secrets manager
  //password = "password"
  //username = "user"
  //server_name = "server"
  //port = "3306"

But it errors out for me

Error: Error creating DMS endpoint: InvalidParameterValueException: The parameter Password must be provided and must not be blank.
│       status code: 400, request id: d6019eac
│ 
│   with module.aws_dms.aws_dms_endpoint.source_endpoint,
│   on modules/aws_dms/endpoints.tf line 17, in resource "aws_dms_endpoint" "source_endpoint":
│   17: resource "aws_dms_endpoint" "source_endpoint" {

[sbrandtb]

@blanco750 which DBMS are you using? The feature has only been implemented for Oracle and Postgres so far, afaik, in #19040

[blanco750]

Hello @sbrandtb thanks for the reply. It is for MySql. Probably thats why it is not working.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.