Terraform fails when removing already destroyed kms key.

[natedogith1]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

v1.2.3 on linux_amd64

Affected Resource(s)

• google_kms_crypto_key
• google_kms_crypto_key_version

Panic Output

Error: googleapi: Error 400: The request cannot be fulfilled. Resource projects/<project>/locations/europe/keyRings/key-ring/cryptoKeys/key/cryptoKeyVersions/1 has value DESTROYED in field crypto_key_version.state.
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.PreconditionFailure",
    "violations": [
      {
        "subject": "projects/<project>/locations/europe/keyRings/key-ring/cryptoKeys/key/cryptoKeyVersions/1",
        "type": "INVALID_ENUM_VALUE_CHOICE"
      }
    ]
  }
]
, failedPrecondition

Expected Behavior

Terraform continues, since key is already destroyed

Actual Behavior

Terraform apply fails.

Steps to Reproduce

1. create key with terraform
2. delete key without terraform (gcloud kms keys versions destroy)
3. wait for key to be destroyed (google documentation says 24 hours)
4. remove key from terraform

References

• source key/cryptoKeyVersions/1 has value DESTROYED in field crypto_key_version.state., failedPrecondition #7044

[edwardmedia]

@natedogith1 I think this is the pattern for most of the resources. When the provider sends the DELETE request to the API, this is the expected response.

If you delete the resources outside of the terraform, you may call terraform rm to delete them in the state.

Does this make sense?

[natedogith1]

That's not the behavior I see for other resources. If I delete a VM in the GCP console and then try to remove it from terraform, terraform apply runs without issues.

[edwardmedia]

@natedogith1 I need more info for its repo. What did you see from below command before you execute step #4? For step #4, did you call terraform apply or terraform destroy?

gcloud kms keys versions list

[natedogith1]

I used terraform apply after removing it from terraform.

I only checked after the failure, and the project has since been deleted, but I believe it said the state was DESTROYED for version 1. I don't remember what it said for versions 2-6, but a different key/key-ring showed something to the effect of DESTROY_SCHEDULED for its versions when it was removed from terraform in the same apply.

[edwardmedia]

@natedogith1 I can't repro the issue.

Waited for a day, I deleted the key on a specific version. Then the version shows DESTROY_SCHEDULED. After a certain time, it turns DESTROYED. At this time, With terraform apply, a new version is created. No error is discovered.

Without your debug log, it's hard to imagine what happened to yours.

[natedogith1]

@edwardmedia terraform shouldn't have created a new version. Did you remove the key/key-ring from the tf file before running the apply?

[eraac]

I got the same issue, but on key with rotation

The last key is active, but the first is deleted, and Terraform try to delete the first version (despite new versions exist)

I guess the issue is here

terraform-provider-google/google/kms_utils.go

Line 177 in b2abd5d

func clearCryptoKeyVersions(cryptoKeyId *kmsCryptoKeyId, userAgent string, config *Config) error {

, there are no filter for disable/deleted keys

[slevenick]

Hmmmm, I thought this was fixed recently but I can't seem to find it. I think we need to check if the key is in the destroyed state already and remove it from state if it is. This should happen when the resource refreshes, so it should be fairly easy to handle in the read method. @edwardmedia can you look into adding this?

[eraac]

I'm not sure google_kms_crypto_key resource hold the version. I don't think the resource has a bug, is more the utils method in charge of deleting a key, because we can't really delete a key, Terraform (like the console does) must remove all the versions of the key + disable the auto-rotating, but it's seems to have no exception for non enable versions, which cause the error (the call to destroy a version on a already destroyed version).

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.