Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

resource/aws_ecs_service: Retry DescribeServices after creation #3387

Merged
merged 1 commit into from
Feb 15, 2018
Merged

resource/aws_ecs_service: Retry DescribeServices after creation #3387

merged 1 commit into from
Feb 15, 2018

Conversation

radeksimko
Copy link
Member

This is to address YAEC (Yet Another Eventual Consistency), as shown in this test failure:

=== RUN   TestAccAWSEcsService_withLaunchTypeEC2AndNetworkConfiguration
--- FAIL: TestAccAWSEcsService_withLaunchTypeEC2AndNetworkConfiguration (610.25s)
    testing.go:513: Step 0 error: Check failed: Check 1/4 error: Not found: aws_ecs_service.main
    testing.go:573: Error destroying resource! WARNING: Dangling resources
        may exist. The full state and error is shown below.
        
        Error: Error applying: 1 error(s) occurred:
        
        * aws_ecs_cluster.main (destroy): 1 error(s) occurred:
        
        * aws_ecs_cluster.main: ClusterContainsServicesException: The Cluster cannot be deleted while Services are active.
            status code: 400, request id: c145171f-114b-11e8-a7a8-47bba0340bdc
        
        State: aws_ecs_cluster.main:
          ID = arn:aws:ecs:us-west-2:*******:cluster/tf-acc-cluster-svc-w-nc-39kv6nox
          provider = provider.aws
          arn = arn:aws:ecs:us-west-2:*******:cluster/tf-acc-cluster-svc-w-nc-39kv6nox
          name = tf-acc-cluster-svc-w-nc-39kv6nox

Snippet from debug log to prove it is eventual consistency:

2018/02/14 05:46:15 [DEBUG] [aws-sdk-go] DEBUG: Request ecs/CreateService Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: ecs.us-west-2.amazonaws.com
User-Agent: aws-sdk-go/1.12.75 (go1.9.4; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.3
Content-Length: 568
Authorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20180214/us-west-2/ecs/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-target, Signature=*REDACTED*
Content-Type: application/x-amz-json-1.1
X-Amz-Date: 20180214T054615Z
X-Amz-Target: AmazonEC2ContainerServiceV20141113.CreateService
Accept-Encoding: gzip

{"clientToken":"terraform-20180214054615299400000001","cluster":"arn:aws:ecs:us-west-2:*REDACTED*:cluster/tf-acc-cluster-svc-w-nc-39kv6nox","deploymentConfiguration":{"maximumPercent":200,"minimumHealthyPercent":100},"desiredCount":1,"launchType":"EC2","networkConfiguration":{"awsvpcConfiguration":{"assignPublicIp":"DISABLED","securityGroups":["sg-929717ed","sg-e498189b"],"subnets":["subnet-a5a84dee","subnet-2c738855"]}},"serviceName":"tf-acc-svc-w-nc-39kv6nox","taskDefinition":"arn:aws:ecs:us-west-2:*REDACTED*:task-definition/tf-acc-td-svc-w-nc-39kv6nox:1"}
-----------------------------------------------------
2018/02/14 05:46:15 [DEBUG] [aws-sdk-go] DEBUG: Response ecs/CreateService Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Connection: close
Content-Length: 1345
Content-Type: application/x-amz-json-1.1
Date: Wed, 14 Feb 2018 05:46:15 GMT
Server: Server
X-Amzn-Requestid: 5f03b2dc-114a-11e8-b9a2-15992728e1bf


-----------------------------------------------------
2018/02/14 05:46:15 [DEBUG] [aws-sdk-go] {"service":{"clusterArn":"arn:aws:ecs:us-west-2:*REDACTED*:cluster/tf-acc-cluster-svc-w-nc-39kv6nox","createdAt":1.518587175873E9,"deploymentConfiguration":{"maximumPercent":200,"minimumHealthyPercent":100},"deployments":[{"createdAt":1.518587175873E9,"desiredCount":1,"id":"ecs-svc/9223370518267599934","launchType":"EC2","networkConfiguration":{"awsvpcConfiguration":{"assignPublicIp":"DISABLED","securityGroups":["sg-929717ed","sg-e498189b"],"subnets":["subnet-a5a84dee","subnet-2c738855"]}},"pendingCount":0,"runningCount":0,"status":"PRIMARY","taskDefinition":"arn:aws:ecs:us-west-2:*REDACTED*:task-definition/tf-acc-td-svc-w-nc-39kv6nox:1","updatedAt":1.518587175873E9}],"desiredCount":1,"events":[],"launchType":"EC2","loadBalancers":[],"networkConfiguration":{"awsvpcConfiguration":{"assignPublicIp":"DISABLED","securityGroups":["sg-929717ed","sg-e498189b"],"subnets":["subnet-a5a84dee","subnet-2c738855"]}},"pendingCount":0,"placementConstraints":[],"placementStrategy":[],"roleArn":"arn:aws:iam::*REDACTED*:role/aws-service-role/ecs.amazonaws.com/AWSServiceRoleForECS","runningCount":0,"serviceArn":"arn:aws:ecs:us-west-2:*REDACTED*:service/tf-acc-svc-w-nc-39kv6nox","serviceName":"tf-acc-svc-w-nc-39kv6nox","status":"ACTIVE","taskDefinition":"arn:aws:ecs:us-west-2:*REDACTED*:task-definition/tf-acc-td-svc-w-nc-39kv6nox:1"}}
2018/02/14 05:46:15 [DEBUG] ECS service created: arn:aws:ecs:us-west-2:*REDACTED*:service/tf-acc-svc-w-nc-39kv6nox
2018/02/14 05:46:15 [DEBUG] Reading ECS service arn:aws:ecs:us-west-2:*REDACTED*:service/tf-acc-svc-w-nc-39kv6nox
2018/02/14 05:46:15 [DEBUG] Waiting for state to become: [success]
2018/02/14 05:46:15 [DEBUG] [aws-sdk-go] DEBUG: Request ecs/DescribeServices Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: ecs.us-west-2.amazonaws.com
User-Agent: aws-sdk-go/1.12.75 (go1.9.4; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.3
Content-Length: 172
Authorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20180214/us-west-2/ecs/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-target, Signature=*REDACTED*
Content-Type: application/x-amz-json-1.1
X-Amz-Date: 20180214T054615Z
X-Amz-Target: AmazonEC2ContainerServiceV20141113.DescribeServices
Accept-Encoding: gzip

{"cluster":"arn:aws:ecs:us-west-2:*REDACTED*:cluster/tf-acc-cluster-svc-w-nc-39kv6nox","services":["arn:aws:ecs:us-west-2:*REDACTED*:service/tf-acc-svc-w-nc-39kv6nox"]}
-----------------------------------------------------
2018/02/14 05:46:15 [DEBUG] [aws-sdk-go] DEBUG: Response ecs/DescribeServices Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Connection: close
Content-Length: 125
Content-Type: application/x-amz-json-1.1
Date: Wed, 14 Feb 2018 05:46:15 GMT
Server: Server
X-Amzn-Requestid: 5f4d65ee-114a-11e8-b3a0-39e0ea505d2c


-----------------------------------------------------
2018/02/14 05:46:15 [DEBUG] [aws-sdk-go] {"failures":[{"arn":"arn:aws:ecs:us-west-2:*REDACTED*:service/tf-acc-svc-w-nc-39kv6nox","reason":"MISSING"}],"services":[]}
2018/02/14 05:46:15 [DEBUG] Removing ECS service tf-acc-svc-w-nc-39kv6nox (arn:aws:ecs:us-west-2:*REDACTED*:service/tf-acc-svc-w-nc-39kv6nox) because it's gone

and from another run:

2018/02/14 18:09:45 [DEBUG] [aws-sdk-go] DEBUG: Request ecs/CreateService Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: ecs.us-west-2.amazonaws.com
User-Agent: aws-sdk-go/1.12.75 (go1.9.4; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.3
Content-Length: 376
Authorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20180214/us-west-2/ecs/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-target, Signature=*REDACTED*
Content-Type: application/x-amz-json-1.1
X-Amz-Date: 20180214T180945Z
X-Amz-Target: AmazonEC2ContainerServiceV20141113.CreateService
Accept-Encoding: gzip

{"clientToken":"terraform-20180214180945931300000001","cluster":"arn:aws:ecs:us-west-2:*REDACTED*:cluster/terraform-acc-test","deploymentConfiguration":{"maximumPercent":200,"minimumHealthyPercent":100},"desiredCount":1,"launchType":"EC2","serviceName":"tf-acc-ecs-svc","taskDefinition":"arn:aws:ecs:us-west-2:*REDACTED*:task-definition/terraform-acc-sc-volume-test:3307"}
-----------------------------------------------------
2018/02/14 18:09:46 [DEBUG] [aws-sdk-go] DEBUG: Response ecs/CreateService Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Connection: close
Content-Length: 876
Content-Type: application/x-amz-json-1.1
Date: Wed, 14 Feb 2018 18:09:46 GMT
Server: Server
X-Amzn-Requestid: 3d1dbd8f-11b2-11e8-b6ac-ef5162fb4b37


-----------------------------------------------------
2018/02/14 18:09:46 [DEBUG] [aws-sdk-go] {"service":{"clusterArn":"arn:aws:ecs:us-west-2:*REDACTED*:cluster/terraform-acc-test","createdAt":1.518631786247E9,"deploymentConfiguration":{"maximumPercent":200,"minimumHealthyPercent":100},"deployments":[{"createdAt":1.518631786247E9,"desiredCount":1,"id":"ecs-svc/9223370518222989414","launchType":"EC2","pendingCount":0,"runningCount":0,"status":"PRIMARY","taskDefinition":"arn:aws:ecs:us-west-2:*REDACTED*:task-definition/terraform-acc-sc-volume-test:3307","updatedAt":1.518631786247E9}],"desiredCount":1,"events":[],"launchType":"EC2","loadBalancers":[],"pendingCount":0,"placementConstraints":[],"placementStrategy":[],"runningCount":0,"serviceArn":"arn:aws:ecs:us-west-2:*REDACTED*:service/tf-acc-ecs-svc","serviceName":"tf-acc-ecs-svc","status":"ACTIVE","taskDefinition":"arn:aws:ecs:us-west-2:*REDACTED*:task-definition/terraform-acc-sc-volume-test:3307"}}
2018/02/14 18:09:46 [DEBUG] ECS service created: arn:aws:ecs:us-west-2:*REDACTED*:service/tf-acc-ecs-svc
2018/02/14 18:09:46 [DEBUG] Reading ECS service arn:aws:ecs:us-west-2:*REDACTED*:service/tf-acc-ecs-svc
2018/02/14 18:09:46 [DEBUG] Waiting for state to become: [success]
2018/02/14 18:09:46 [DEBUG] [aws-sdk-go] DEBUG: Request ecs/DescribeServices Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: ecs.us-west-2.amazonaws.com
User-Agent: aws-sdk-go/1.12.75 (go1.9.4; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.3
Content-Length: 148
Authorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20180214/us-west-2/ecs/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-target, Signature=*REDACTED*
Content-Type: application/x-amz-json-1.1
X-Amz-Date: 20180214T180946Z
X-Amz-Target: AmazonEC2ContainerServiceV20141113.DescribeServices
Accept-Encoding: gzip

{"cluster":"arn:aws:ecs:us-west-2:*REDACTED*:cluster/terraform-acc-test","services":["arn:aws:ecs:us-west-2:*REDACTED*:service/tf-acc-ecs-svc"]}
-----------------------------------------------------
2018/02/14 18:09:46 [DEBUG] [aws-sdk-go] DEBUG: Response ecs/DescribeServices Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Connection: close
Content-Length: 895
Content-Type: application/x-amz-json-1.1
Date: Wed, 14 Feb 2018 18:09:46 GMT
Server: Server
X-Amzn-Requestid: 3d3f759c-11b2-11e8-92d7-31faf84d925e


-----------------------------------------------------
2018/02/14 18:09:46 [DEBUG] [aws-sdk-go] {"failures":[],"services":[{"clusterArn":"arn:aws:ecs:us-west-2:*REDACTED*:cluster/terraform-acc-test","createdAt":1.518587188556E9,"deploymentConfiguration":{"maximumPercent":200,"minimumHealthyPercent":100},"deployments":[{"createdAt":1.518587193291E9,"desiredCount":0,"id":"ecs-svc/9223370518267582515","launchType":"EC2","pendingCount":0,"runningCount":0,"status":"PRIMARY","taskDefinition":"arn:aws:ecs:us-west-2:*REDACTED*:task-definition/terraform-acc-sc-volume-test:3305","updatedAt":1.518587198693E9}],"desiredCount":0,"events":[],"launchType":"EC2","loadBalancers":[],"pendingCount":0,"placementConstraints":[],"placementStrategy":[],"runningCount":0,"serviceArn":"arn:aws:ecs:us-west-2:*REDACTED*:service/tf-acc-ecs-svc","serviceName":"tf-acc-ecs-svc","status":"INACTIVE","taskDefinition":"arn:aws:ecs:us-west-2:*REDACTED*:task-definition/terraform-acc-sc-volume-test:3305"}]}
2018/02/14 18:09:46 [DEBUG] Removing ECS service "arn:aws:ecs:us-west-2:*REDACTED*:service/tf-acc-ecs-svc" because it's INACTIVE

Test plan

=== RUN   TestAccAWSEcsService_withPlacementConstraints
--- PASS: TestAccAWSEcsService_withPlacementConstraints (11.34s)
=== RUN   TestAccAWSEcsService_withEcsClusterName
--- PASS: TestAccAWSEcsService_withEcsClusterName (16.51s)
=== RUN   TestAccAWSEcsDataSource_ecsTaskDefinition
--- PASS: TestAccAWSEcsDataSource_ecsTaskDefinition (16.68s)
=== RUN   TestAccAWSEcsCluster_basic
--- PASS: TestAccAWSEcsCluster_basic (17.63s)
=== RUN   TestAccAWSEcsDataSource_ecsCluster
--- FAIL: TestAccAWSEcsDataSource_ecsCluster (20.37s)
	testing.go:513: Step 0 error: Error applying: 1 error(s) occurred:
		
		* data.aws_ecs_cluster.default: data.aws_ecs_cluster.default: cluster with name "default-2179568069295143950" not found
FAIL
=== RUN   TestAccAWSEcsCluster_importBasic
--- PASS: TestAccAWSEcsCluster_importBasic (20.41s)
=== RUN   TestAccAWSEcsService_withUnnormalizedPlacementStrategy
--- PASS: TestAccAWSEcsService_withUnnormalizedPlacementStrategy (24.20s)
=== RUN   TestAccAWSEcsService_withARN
--- PASS: TestAccAWSEcsService_withARN (37.56s)
=== RUN   TestAccAWSEcsTaskDefinition_withScratchVolume
--- PASS: TestAccAWSEcsTaskDefinition_withScratchVolume (21.45s)
=== RUN   TestAccAWSEcsTaskDefinition_basic
--- PASS: TestAccAWSEcsTaskDefinition_basic (22.30s)
=== RUN   TestAccAWSEcsTaskDefinition_constraint
--- PASS: TestAccAWSEcsTaskDefinition_constraint (17.73s)
=== RUN   TestAccAWSEcsService_withRenamedCluster
--- PASS: TestAccAWSEcsService_withRenamedCluster (42.71s)
=== RUN   TestAccAWSEcsService_withPlacementConstraints_emptyExpression
--- PASS: TestAccAWSEcsService_withPlacementConstraints_emptyExpression (43.95s)
=== RUN   TestAccAWSEcsTaskDefinition_withNetworkMode
--- PASS: TestAccAWSEcsTaskDefinition_withNetworkMode (27.94s)
=== RUN   TestAccAWSEcsTaskDefinition_Fargate
--- PASS: TestAccAWSEcsTaskDefinition_Fargate (12.77s)
=== RUN   TestAccAWSEcsTaskDefinition_arrays
--- PASS: TestAccAWSEcsTaskDefinition_arrays (18.24s)
=== RUN   TestAccAWSEcsTaskDefinition_ExecutionRole
--- PASS: TestAccAWSEcsTaskDefinition_ExecutionRole (17.41s)
=== RUN   TestAccAWSEcsService_withDeploymentValues
--- PASS: TestAccAWSEcsService_withDeploymentValues (69.68s)
=== RUN   TestAccAWSEcsTaskDefinition_changeVolumesForcesNewResource
--- PASS: TestAccAWSEcsTaskDefinition_changeVolumesForcesNewResource (33.42s)
=== RUN   TestAccAWSEcsService_withFamilyAndRevision
--- PASS: TestAccAWSEcsService_withFamilyAndRevision (72.60s)
=== RUN   TestAccAWSEcsTaskDefinition_withTaskRoleArn
--- PASS: TestAccAWSEcsTaskDefinition_withTaskRoleArn (56.56s)
=== RUN   TestAccAWSEcsDataSource_ecsContainerDefinition
--- PASS: TestAccAWSEcsDataSource_ecsContainerDefinition (77.76s)
=== RUN   TestAccAWSEcsTaskDefinition_withEcsService
--- PASS: TestAccAWSEcsTaskDefinition_withEcsService (67.94s)
=== RUN   TestAccAWSEcsService_withPlacementStrategy
--- PASS: TestAccAWSEcsService_withPlacementStrategy (90.93s)
=== RUN   TestAccAWSEcsService_basicImport
--- PASS: TestAccAWSEcsService_basicImport (109.45s)
=== RUN   TestAccAWSEcsService_withLaunchTypeEC2AndNetworkConfiguration
--- PASS: TestAccAWSEcsService_withLaunchTypeEC2AndNetworkConfiguration (112.98s)
=== RUN   TestAccAWSEcsService_withIamRole
--- PASS: TestAccAWSEcsService_withIamRole (127.24s)
=== RUN   TestAccAWSEcsService_withLbChanges
--- PASS: TestAccAWSEcsService_withLbChanges (140.82s)
=== RUN   TestAccAWSEcsService_healthCheckGracePeriodSeconds
--- PASS: TestAccAWSEcsService_healthCheckGracePeriodSeconds (191.43s)
=== RUN   TestAccAWSEcsService_withAlb
--- PASS: TestAccAWSEcsService_withAlb (203.16s)
=== RUN   TestAccAWSEcsService_withLaunchTypeFargate
--- PASS: TestAccAWSEcsService_withLaunchTypeFargate (240.11s)

@radeksimko radeksimko added bug Addresses a defect in current functionality. service/ecs Issues and PRs that pertain to the ecs service. labels Feb 15, 2018
@ghost ghost added the size/S Managed by automation to categorize the size of a PR. label Feb 15, 2018
@radeksimko radeksimko mentioned this pull request Feb 15, 2018
Merged
bflad
bflad approved these changes Feb 15, 2018
View reviewed changes
Copy link
Contributor

@bflad bflad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM -- thanks for fixing this!

@radeksimko radeksimko merged commit 40b6110 into master Feb 15, 2018
@radeksimko radeksimko deleted the b-ecs-retry-read-on-creation branch February 15, 2018 16:02
@bflad bflad added this to the v1.10.0 milestone Feb 27, 2018
@ghost
Copy link

ghost commented Apr 7, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked and limited conversation to collaborators Apr 7, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@bflad bflad bflad approved these changes

Assignees
No one assigned
Labels
bug Addresses a defect in current functionality. service/ecs Issues and PRs that pertain to the ecs service. size/S Managed by automation to categorize the size of a PR.
Projects
None yet
Milestone
v1.10.0
Development

Successfully merging this pull request may close these issues.
2 participants
@radeksimko @bflad