[Bug]: aws_quicksight_data_source ResourceExistsException for same data_source_id but different name and credentials

[kajohnson-77]

Terraform Core Version

1.3.3

AWS Provider Version

4.67.0

Affected Resource(s)

aws_quicksight_data_source
Arguments:
data_source_id
name
credentials
type = REDSHIFT

Expected Behavior

Terraform AWS provider for aws_quicksight_data_source will create a new Quicksight data source, with the given name, connecting to data_source_id with the provided credentials.
Together the data_source_id and name should be unique. This is due to different credentials being provided to connect to the data source, so that underlying grants in redshift can be used to scope a users ability to use and select from specific schemas and tables.
Note we havew in place a cloudformation_stack definition for a quicksight data source with the same data_source (redshift), and it is able to exist along with the terraform provider resource. We are able to leverage both data sources in Quicksight with correct access to the redshift schemas and tables per connection credentials used.

Actual Behavior

An error is reported when attempting to create the resource if the data_source_id is already a terraformed aws_quicksight_data_source.

Relevant Error/Panic Output Snippet

Error: error creating QuickSight Data Source: ResourceExistsException: DataSource datasource/tf-redshift-cluster already exists
{
  RespMetadata: {
    StatusCode: 409,
    RequestID: "0bea6073-cd84-44d6-9208-2cef7942f825"
  },
  Message_: "DataSource datasource/tf-redshift-cluster already exists"
}

  with aws_quicksight_data_source.quicksight_redshift_source[0],
  on quicksight.tf line 69, in resource "aws_quicksight_data_source" "quicksight_redshift_source":
  69: resource "aws_quicksight_data_source" "quicksight_redshift_source" {

Terraform Configuration Files

resource "aws_quicksight_data_source" "quicksight_redshift_source" {
  count = (
    contains(flatten([var.quicksight_account_ids]), data.aws_caller_identity.current.account_id)
  ) ? 1 : 0

  data_source_id = module.tf_redshift_cluster[0].redshift_cluster.cluster_identifier
  name           = "quicksight-redshift-source"
  aws_account_id = data.aws_caller_identity.current.account_id

  credentials {
    credential_pair {
      password = jsondecode(data.aws_secretsmanager_secret_version.redshift_quicksight_credentials[0].secret_string)["password"]
      username = jsondecode(data.aws_secretsmanager_secret_version.redshift_quicksight_credentials[0].secret_string)["username"]
    }
  }

  parameters {
    redshift {
      cluster_id = module.tf_redshift_cluster[0].redshift_cluster.cluster_identifier
      database   = module.tf_redshift_cluster[0].redshift_cluster.database_name
      host       = module.tf_redshift_cluster[0].redshift_cluster.dns_name
      port       = module.tf_redshift_cluster[0].redshift_cluster.port
    }
  }

  permission {
    actions = [
      "quicksight:DescribeDataSource",
      "quicksight:DescribeDataSourcePermissions",
      "quicksight:PassDataSource",
    ]
    principal = aws_quicksight_group.quicksight_users[0].arn
  }

  ssl_properties {
    disable_ssl = false
  }

  vpc_connection_properties {
    vpc_connection_arn = var.quicksight_vpc_connection_arn
  }

  type = "REDSHIFT"

  tags = merge(
    local.tags_default,
    local.tags_data_classification,
    {
      Name = "quicksight-redshift-source"
    }
  )
}

Steps to Reproduce

1. Create a aws_quicksight_data_source with the given parameters above.
2. Create another aws_quicksight_data_source with the given parameters above, but change the name and credentials arguments.

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[justinretzolk]

Hey @kajohnson-77 👋 Thank you for taking the time to raise this! The error message looks to be coming from the AWS API itself, so I took a look over the API reference for CreateDataSource, which indicates that the data source ID:

An ID for the data source. This ID is unique per AWS Region for each AWS account.

Based on my read, it seems like you can connect to the same data source with multiple different aws_quicksight_data_source resources, but the data_source_id must differ.

[kajohnson-77]

Thank you for following this into the AWS API. Since this is the intended behavior from that API, I will follow-up with AWS. I believe we can close out this issue/bug.

[justinretzolk]

Hey @kajohnson-77 👋 Before we close this out, I'd be remiss if I didn't mention that you can almost certainly get around this by not using the module.tf_redshift_cluster[0].redshift_cluster.cluster_identifier as the data_source_id, or by prepending or appending that value with an additional string in order to make it unique. The data_source_id argument appears to just be a string identifier for the Quicksight data source, while the parameters (and some of the other arguments) actually specify the connection. I also presume the name needs to be unique as well. If that doesn't work, it may be worthwhile to follow up with AWS for additional clarification.

[kajohnson-77]

Hi @justinretzolk , thank you for the update and recommendations to try these params with unique values, we'll try it out and see what we get. I did find another issue though, and will open a new bug. We rotated the secret that is part of the credentials parameter, and it seems terraform is only sending that 'change' (along with name and id) to the update-data-source api, and its not enough, all required params need to be sent.
Anyway, thanks again.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.