IAM eventual consistency race condition for aws_iam_role leaves lambda function in defunct state

[samjgalbraith]

To ensure continued functioning of my system through changes which require role replacement, I have used create_before_destroy = true with my lambda function lifecycles. However, I have found that in general, changes which require the role to be replaced fail on terraform apply with the error

InvalidParameterValueException: The role defined for the function cannot be assumed by Lambda

A second run a short time later it succeeds. When I look in the AWS console, between usages of apply, it appears to leave my Lambda in an indeterminate state with no execution role. When I look at Terraform's output, what has happened is:

• The new role is successfully created to replace the old one.
• The old role has been destroyed.
• The change of role for the Lambda function fails, citing "InvalidParameterValueException: The role defined for the function cannot be assumed by Lambda"

It looks like this might be because AWS returns from the role creation call before the new role is completely available to trusted AWS services such as Lambda.

I have been able to work around this issue by adding a sleep provisioner to every one of my roles, but this is not a tidy workaround:

resource "aws_iam_role" "lambda_execution_role" {
  name_prefix = "${var.role_name_prefix}"
  assume_role_policy = "${data.aws_iam_policy_document.entrust_lambda_to_assume_function_role.json}"
  description = "${var.role_description}"
  path = "${var.path}"
  lifecycle {
    create_before_destroy = true
  }
  # AWS returns success for IAM change but change is not yet available for a few seconds. Sleep to miss the race condition failure.
  provisioner "local-exec" {
    interpreter = ["bash", "-c"]
    command = "sleep 30"
  }
}

Terraform Version

Terraform v0.11.5

• provider.archive v1.0.3
• provider.aws v1.11.0
• provider.local v1.1.0
• provider.null v1.0.0
• provider.template v1.0.0

Affected Resource(s)

• aws_iam_role

Expected Behavior

IAM role replacement with create_before_destroy = true provides zero downtime for AWS Lambda.

Actual Behavior

Lambda function left in indeterminate state, unable to execute.

Steps to Reproduce

data "aws_iam_policy_document" "entrust_lambda_to_assume_function_role" {
  statement {
    effect = "Allow"
    principals {
      identifiers = ["lambda.amazonaws.com"]
      type        = "Service"
    }
    actions = ["sts:AssumeRole"]
  }
}

resource "aws_iam_role" "lambda_execution_role" {
  name_prefix = "testing"
  assume_role_policy = "${data.aws_iam_policy_document.entrust_lambda_to_assume_function_role.json}"
  lifecycle {
    create_before_destroy = true
  }
}

Refer to this IAM role resource with a aws_lambda_function resource. Create and apply this infrastructure. Now change the name_prefix field in aws_iam_role.

[bflad]

PR submitted to retry on this condition during update (was only present during creation): #3988

[bflad]

The fix has been merged into master and will release with v1.14.0 of the AWS provider, in a week most likely.

[samjgalbraith]

Thanks for that @bflad

[samjgalbraith]

It turns out my problem was actually more general than this and related to create_before_destroy not working as expected. It was creating a new dependency (the role), destroying the old one, THEN updating Lambda's reference to it last. This meant that for a while the Lambda function was still referring to an already deleted role. The failure of Lambda to update due to this IAM race condition caused infrastructure changes to halt at this temporarily broken state. This applies more generally than Lambda or even AWS and is an issue with the core engine, that I've lodged as hashicorp/terraform#17735

I agree that the change you have made is a good one and will help in some cases, it just turns out my problem was more general.

[bflad]

This has been released in version 1.14.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!