Synthetics Canary update not working

[aws-user-2021]

Terraform CLI and Terraform AWS Provider Version

Terraform v0.14.6

• provider registry.terraform.io/hashicorp/archive v2.0.0
• provider registry.terraform.io/hashicorp/aws v3.28.0
• provider registry.terraform.io/hashicorp/null v3.0.0

Affected Resource(s)

aws_synthetics_canary

Terraform Configuration Files

  ~ resource "aws_synthetics_canary" "synthetics" {
        id                       = "cm-dev-testui"
        name                     = "cm-dev-testui"
      ~ zip_file                 = "files/testui-canary-298c366a80006e33f3b97d2dbc9a4c8f.zip" -> "files/testui-canary-082d98273331513b8d3c4ae575c3fbb2.zip"
        # (12 unchanged attributes hidden)


        # (2 unchanged blocks hidden)
    }

Debug Output

2021-02-16T13:27:43.115-0500 [INFO] plugin.terraform-provider-aws_v3.28.0_x5: 2021/02/16 13:27:43 [DEBUG] [aws-sdk-go] DEBUG: Request synthetics/GetCanary Details:

State is RUNNING

2021-02-16T13:27:47.372-0500 [INFO] plugin.terraform-provider-aws_v3.28.0_x5: 2021/02/16 13:27:47 [DEBUG] [aws-sdk-go] DEBUG: Request synthetics/StopCanary Details:

2021-02-16T13:27:47.590-0500 [INFO] plugin.terraform-provider-aws_v3.28.0_x5: 2021/02/16 13:27:47 [DEBUG] [aws-sdk-go] DEBUG: Request synthetics/GetCanary Details:

State is STOPPING

2021-02-16T13:27:48.610-0500 [INFO] plugin.terraform-provider-aws_v3.28.0_x5: 2021/02/16 13:27:48 [DEBUG] [aws-sdk-go] DEBUG: Request synthetics/GetCanary Details:

State is STOPPED

2021-02-16T13:27:48.795-0500 [INFO] plugin.terraform-provider-aws_v3.28.0_x5: 2021/02/16 13:27:48 [DEBUG] [aws-sdk-go] DEBUG: Request synthetics/UpdateCanary Details:

State is UPDATING

2021-02-16T13:28:13.377-0500 [INFO] plugin.terraform-provider-aws_v3.28.0_x5: 2021/02/16 13:28:13 [DEBUG] [aws-sdk-go] DEBUG: Request synthetics/GetCanary Details:

State is STOPPED

2021/02/16 13:28:13 [DEBUG] aws_synthetics_canary.synthetics[0]: apply errored, but we're indicating that via the Error pointer rather than returning it: error waiting for Synthetics Canary (cm-dev-testui) updating: unexpected state 'STOPPED', wanted target 'READY'. last error: %!s()

Error: error waiting for Synthetics Canary (cm-dev-testui) updating: unexpected state 'STOPPED', wanted target 'READY'. last error: %!s()

Expected Behavior

canary should have been restarted but instead terraform update fails.

Actual Behavior

looks like the code is expecting the canary to be in READY state after update but CloudWatch Synthetics is putting the canary backed to STOPPED state since it was in STOPPED state before UpdateCanary API was called. It is likely that AWS updated behavior after the developer had tested updating canary scenario.

Steps to Reproduce

Update the canary code and run terraform apply

[aws-user-2021]

Here is some more info. When a canary is first created and not started, it goes into READY state (corresponding AWS Console state is "Not started"). If the canary code is updated while in READY state, the state changes to UPDATING while update is in progress and then changes back to READY state once update is done.

However, once the canary has been started, the update process stops the canary first (state changes to STOPPING and then to STOPPED) before updating the code. The state change in this case goes from STOPPED to UPDATING and then back to STOPPED and not READY. Current implementation is expecting the canary state to go to READY after the code update completes and hence canary update fails in this scenario, which is the most common use case.

[aws-user-2021]

One more issue
canary resource replacement works but when you try to delete a running canary, it fails. AWS requires that canary be stopped before it is deleted. Looks like the code to stop the canary before deletion is not working.

[aws-user-2021]

Primary issue mentioned regarding update failing when code is updated also happens when tags are updated likely because of the same reason.

[bill-rich]

Hi @aws-user-2021! Can you please provide your Terraform configuration to help reproduce the issue?

[DrFaust92]

Im able to reproduce the issue, ill take a crack at fixing it.

[aws-user-2021]

@DrFaust92 thank you. Deleting canary is also failing and I am getting by right now using a destroy provisioner to stop the canary. Not sure if you were able to reproduce this issue.

[ghost]

This has been released in version 3.30.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!