azurerm_data_factory changes and corrupts customer_managed_key_id implicitly

[pw595]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave comments along the lines of "+1", "me too" or "any updates", they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

1.3.9

AzureRM Provider Version

4.6.0

Affected Resource(s)/Data Source(s)

azurerm_data_factory

Terraform Configuration Files

resource "azurerm_data_factory" "adf" {
  name                = var.adf_name
  location            = var.location
  resource_group_name = var.rgname
  tags                = var.tags
  lifecycle {
    ignore_changes = [customer_managed_key_id, public_network_enabled, github_configuration, global_parameter]
  }
  identity {
    type = "SystemAssigned"
  }
}

Debug Output/Panic Output

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.adf_ams["bdaa-poc-ha-adf-dev-ams"].azurerm_data_factory.adf will be updated in-place
  ~ resource "azurerm_data_factory" "adf" {
        id                              = "/subscriptions/76f909fb-a085-4e06-9837-20b44085be49/resourceGroups/rg-bdaa-poc-ha-dev-ams/providers/Microsoft.DataFactory/factories/bdaa-poc-ha-adf-dev-ams"
        name                            = "bdaa-poc-ha-adf-dev-ams"
      ~ tags                            = {
          + "extratag"       = "extra"
            # (10 unchanged elements hidden)
        }
        # (5 unchanged attributes hidden)

        # (9 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

2024-10-21T17:20:08.968+0200 [WARN]  Provider "registry.terraform.io/hashicorp/azurerm" produced an invalid plan for module.adf_ams["bdaa-poc-ha-adf-dev-ams"].azurerm_data_factory.adf, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .customer_managed_key_identity_id: planned value cty.StringVal("") for a non-computed attribute
      - .managed_virtual_network_enabled: planned value cty.False for a non-computed attribute
      - .purview_id: planned value cty.StringVal("") for a non-computed attribute
      - .identity[0].identity_ids: planned value cty.SetValEmpty(cty.String) for a non-computed attribute
module.adf_ams["bdaa-poc-ha-adf-dev-ams"].azurerm_data_factory.adf: Modifying... [id=/subscriptions/76f909fb-a085-4e06-9837-20b44085be49/resourceGroups/rg-bdaa-poc-ha-dev-ams/providers/Microsoft.DataFactory/factories/bdaa-poc-ha-adf-dev-ams]
2024-10-21T17:20:12.797+0200 [ERROR] provider.terraform-provider-azurerm_v4.6.0_x5.exe: Response contains error diagnostic: @caller=github.com/hashicorp/terraform-plugin-go@v0.23.0/tfprotov5/internal/diag/diagnostics.go:58 diagnostic_detail= tf_req_id=8aa0a672-bb11-253c-6a39-0
818aa034a45 tf_resource_type=azurerm_data_factory tf_rpc=ApplyResourceChange @module=sdk.proto diagnostic_severity=ERROR diagnostic_summary="parsing "": parsing "": cannot parse an empty string" tf_proto_version=5.6 tf_provider_addr=registry.terraform.io/hashicorp/azurerm time
stamp=2024-10-21T17:20:12.797+0200
2024-10-21T17:20:12.800+0200 [ERROR] vertex "module.adf_ams[\"bdaa-poc-ha-adf-dev-ams\"].azurerm_data_factory.adf" error: parsing "": parsing "": cannot parse an empty string
╷
│ Error: parsing "": parsing "": cannot parse an empty string
│
│   with module.adf_ams["bdaa-poc-ha-adf-dev-ams"].azurerm_data_factory.adf,
│   on ..\..\..\modules\datafactory\ha-adf\adf.tf line 2, in resource "azurerm_data_factory" "adf":
│    2: resource "azurerm_data_factory" "adf" {

Expected Behaviour

Changing tags in datafactory resource should not corrupt and break terraform code

Actual Behaviour

After tag changes, customer_managed_key_id is changed in the Datafactory resource itself where an extra slash is added. This non intended and non-authorized change corrupts the whole deployment

Steps to Reproduce

In a Datafactory resource that has Key Encription enabled just change the tags (add a new tag) and then terraform apply.

Important Factoids

nothing special, tested with West Eruope and North Europe

References

Not that I have found

[gerrytan]

Hi @pw595 can you please provide more detailed instruction to reproduce the problem. I've tried the following:

1. Deploy a terraform config you pasted above
2. Create a "Vault access policy" key vault from portal, generate a new key
3. Assign permission for the data factory to be able to read keys from the key vault
4. Via portal -> data factory -> add a Customer managed key URL

I'm stuck at the last step above, I got "Operation failed. Data Factory Managed Identity doesn't have access to customer managed key vault." error despite having assigned the required permission in keyvault

[pw595]

@gerrytan thank you for looking at this issue. Can it be that you have created a key vault with RBAC access policy set up and not access policies (legacy). Did you assign permissions using RBAC (not access policies)?
MS docu https://learn.microsoft.com/en-us/azure/key-vault/general/rbac-access-policy

[maxonchikbk]

@pw595 I think we have the same issue, but with "global_parameter".

The only thing we changing is tag.

Plan: 0 to add, 1 to change, 0 to destroy.

Changes to Outputs:
  ~ datafactory = {
      ~ shared_adf_in = {
          ~ tags                             = {
              ~ "Broken"           = "true" -> "false"
                # (19 unchanged elements hidden)
            }
            # (14 unchanged elements hidden)
        }
        # (1 unchanged element hidden)
    }

We are using 3.89.0 version of azurerm provider and 1.0.0 version of terraform

[gerrytan]

@pw595 I can reproduce the problem now after referring to https://learn.microsoft.com/en-us/azure/data-factory/enable-customer-managed-key.

I confirm this is a bug in azurerm_data_factory resource where customer_managed_key_identity_id property is required when customer_managed_key_id is provided. This contradicts the REST API doc where identity is clearly marked as optional:

User assigned identity to use to authenticate to customer's key vault. If not provided Managed Service Identity will be used.

https://learn.microsoft.com/en-us/rest/api/datafactory/factories/create-or-update?view=rest-datafactory-2018-06-01&tabs=HTTP#encryptionconfiguration

We will need to create a bugfix PR for this, but meanwhile can you confirm if supplying customer_managed_key_identity_id is possible for your situation?

[pw595]

Hello @gerrytan, thank you for looking at this issue again. Since our architecture uses System Assigned Managed Identity as the Identity for our ADF and Access Management, using customer_managed_key_identity_id is not possible for us since UMI does not exist for our legacy ADF instances. When we use a ADF with UMI and Custom Encryption Key then this bug will not show up for us. As you mentioned, using APIs, or Azure Portal this issue does not come up, it is a pure terraform issue.