terraform apply times out but route is still created and terraform does not track this

[JoeyG1973]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform v0.12.12

Affected Resource(s)

• aws_route

Terraform Configuration Files

data "aws_subnet" "bakery" {
  id = "${var.build_subnet_id}"
  provider = "aws.bakery"
}

data "aws_route_table" "bakery" {
  provider = "aws.bakery"
  subnet_id = "${var.build_subnet_id}"
}
resource "aws_route" "temp_route" {
  provider = "aws.bakery"
  route_table_id            = "${data.aws_route_table.bakery.route_table_id}"
  destination_cidr_block    = "${aws_vpc.vpc.cidr_block}"
  vpc_peering_connection_id = "${aws_vpc_peering_connection_accepter.peer.id }"
}

Debug Output

See comments below

Panic Output

N/A

Expected Behavior

Terrraform apply creates the route and doesn't time out

Actual Behavior

terraform apply creates the route and times out not recording the created route state. Running terraform apply again fails because it says the route already exists. Terrafrom destroy does not remove the route because it isn't in the terraform state file.

Steps to Reproduce

1. terraform apply
2. terraform destroy
3. Check to see if route is still in route table, if so you found the intermittent issue, else goto 1

Important Factoids

route is pointing to a peering connection

• #0000

[JoeyG1973]

I figured out what is going on.

This is happening during apply.

aws_route.temp_route: Still creating... [10s elapsed]
aws_route.temp_route: Still creating... [20s elapsed]
aws_route.temp_route: Still creating... [30s elapsed]
aws_route.temp_route: Still creating... [40s elapsed]
aws_route.temp_route: Still creating... [50s elapsed]
aws_route.temp_route: Still creating... [1m0s elapsed]
aws_route.temp_route: Still creating... [1m10s elapsed]
aws_route.temp_route: Still creating... [1m20s elapsed]
aws_route.temp_route: Still creating... [1m30s elapsed]
aws_route.temp_route: Still creating... [1m40s elapsed]
aws_route.temp_route: Still creating... [1m50s elapsed]
aws_route.temp_route: Still creating... [2m0s elapsed]

Error: Error finding route after creating it: Unable to find matching route for Route Table (rtb-8a423bf0) and destination CIDR block (10.95.0.0/16).

  on main.tf line 243, in resource "aws_route" "temp_route":
 243: resource "aws_route" "temp_route" {

Running terraform apply after that throws this errror:

aws_route.temp_route: Creating...

Error: Error creating route: RouteAlreadyExists: The route identified by 10.95.0.0/16 already exists.
	status code: 400, request id: cf3f7ea1-1e80-4af8-a130-5dca16e8e5fe

  on main.tf line 243, in resource "aws_route" "temp_route":
 243: resource "aws_route" "temp_route" {

Running terraform destroy after this cleans up everything but the route that the first apply created. Hence any terraform apply after that first timeout failure bomb as well.

There is a timing issue here and it looks like the route timeout wait needs to be increased.

[JoeyG1973]

This is a horrible hack but it works:

function terraform_apply() {
    $TERRAFORM_PATH init
    $TERRAFORM_PATH apply -auto-approve 2>&1 |tee apply.log
    ROUTE_ALREADY_EXISTS_ERROR=$(cat apply.log | sed 's/\x1b\[[0-9;]*m//g' | grep --color=never 'Error: Error creating route: RouteAlreadyExists: The route identified by ')
    echo ${ROUTE_ALREADY_EXISTS_ERROR}
    if [[ -n "${ROUTE_ALREADY_EXISTS_ERROR}" ]] ;then
        echo "Found RouteAlreadyExists error"
        $TERRAFORM_PATH refresh
        $TERRAFORM_PATH output
        ROUTE_TABLE_ID=$($TERRAFORM_PATH output bakery_route_table_id)
        ROUTE_IDENTIFIER=$(echo ${ROUTE_ALREADY_EXISTS_ERROR} | grep --color=never -o -E '(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(3[0-2]|[1-2][0-9]|[0-9]))')
        echo ${ROUTE_IDENTIFIER}
        echo ${ROUTE_TABLE_ID}
        aws ec2 delete-route --destination-cidr-block ${ROUTE_IDENTIFIER} --route-table-id ${ROUTE_TABLE_ID}
        $TERRAFORM_PATH apply -auto-approve
    fi

[JoeyG1973]

More comprehensive horrible hack that works around this issue:

function terraform_apply() {
    $TERRAFORM_PATH init
    $TERRAFORM_PATH apply -auto-approve 2>&1 |tee apply.log
    FINDING_ROUTE_ERROR=$(cat apply.log | sed 's/\x1b\[[0-9;]*m//g' | grep --color=never 'Error: Error finding route after creating it: Unable to find matching route for Route Table ')
    ROUTE_ALREADY_EXISTS_ERROR=$(cat apply.log | sed 's/\x1b\[[0-9;]*m//g' | grep --color=never 'Error: Error creating route: RouteAlreadyExists: The route identified by ')
    GENERIC_ERROR=$(cat apply.log | sed 's/\x1b\[[0-9;]*m//g' | grep -E --color=never '^Error: ')
    if [[ -n "${ROUTE_ALREADY_EXISTS_ERROR}" ]] || [[ -n "${FINDING_ROUTE_ERROR}" ]];then
        echo "Found route creation error"
        $TERRAFORM_PATH refresh
        $TERRAFORM_PATH output
        ROUTE_TABLE_ID=$($TERRAFORM_PATH output bakery_route_table_id)
        ROUTE_IDENTIFIER=$($TERRAFORM_PATH output vpc_cidr_block)
        echo ${ROUTE_IDENTIFIER}
        echo ${ROUTE_TABLE_ID}
        aws ec2 delete-route --destination-cidr-block ${ROUTE_IDENTIFIER} --route-table-id ${ROUTE_TABLE_ID}
        $TERRAFORM_PATH apply -auto-approve
    elif [[ -n "${GENERIC_ERROR}" ]]; then
        echo "Some other error occured with Terraform"
        exit 1
    fi
}

[d11-acummins]

This happens for me when the IP in destination_cidr_block is more specific than mask accompanying it. For example, supplying 172.1.2.3/16 will cause it to hang upon creation (and then error saying it can't be found), even though it is created successfully as 172.1.0.0. I suspect terraform is looking for the exact CIDR block specified, whereas AWS is removing the unnecessary extra detail.

[bflad]

Additional validation to prevent the CIDR misalignment has been merged and will release with version 2.68.0 of the Terraform AWS Provider, likely tomorrow. Thanks to @ewbankkit for the implementation. 👍

If you are still having issues after upgrading to 2.68.0 when its released, please file a new issue.

[ghost]

This has been released in version 2.68.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!