aws_ecs_service.my-ecs-service->task_definition" is always shows up as modified

[pjain00]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.11.7

Affected Resource(s)

• aws_ecs_service

Terraform Configuration Files

#Define ECS Task Definition
resource "aws_ecs_task_definition" "my-task-def" {
  family = "my-task-def"
  container_definitions = "${file("${var.task-definition-json}")}"
}

#ECS Service Configuration
resource "aws_ecs_service" "my-ecs-service" {
  	name            = "my-ecs-service"
  	cluster         = "my-cluster"
  	task_definition = "my-task-def"
  	desired_count   = 2
	deployment_minimum_healthy_percent="50"
	deployment_maximum_percent="100"
  	load_balancer {
    	target_group_arn  = "${aws_alb_target_group.ecs-target-group.arn}"
    	container_port    = 8080
    	container_name    = "myserver"
	}
	depends_on = ["aws_alb_target_group.ecs-target-group"]
}

Debug Output

Panic Output

Expected Behavior

As I did not make any change in the task definition, so I was expecting that there should not be any change in my ECS object and there should not be any new version created for task definition.

Actual Behavior

Everytime I run my terraform script it modifies the task definition of my ECS object as below.
task_definition: "my-task-def:14" => "my-task-def"

Steps to Reproduce

1. terraform apply

Important Factoids

References

• #0000

[Etiene]

Duplicate of #5323
The problem is on the container definition

[sebastianfuss]

Hi there,

@Etiene
I am not sure, if this an duplicate of #5323
#5323 was solved with #5833 and it is only helps in case, if you didn't set the container host port in container definition.
@pjain00 Please check, if your problem is solved or paste your container definition here. Otherwise I will open a new ticket for my problem.

For me #5833 does not solve my problem. In my case container_definitions forces all the time a new resource. But I noticed same as in #5323 described. The tf plan out is different to tf apply related to environment inside of container_definitions. I have 6 services and only one service has this problem. This service has a bit more environment variables configured.

Thanks a lot! I a very appreciate your work for the whole terraform projects. Cheers Sebastian

[oavdonin]

Hello all,
I think I came across the same bug, for me terraform shows "destroy and then recreate" is required on each execution for aws_ecs_task_definition and aws_ecs_service resources.

I even compared the .rendered values of templates which are being passed as an arguments to container_definitions among different executions and they were absolutely the same.

But the most confusing part for me is that it is not 100% reproducable. I have shed load of services and for some of them it behaves expectedly (without changes).

└──╼ terraform version
Terraform v0.11.8
+ provider.aws v1.40.0
+ provider.template v1.0.0

Here is my task definition for the service with a problem

[
      {
    "name": "lb-nginx",
    "image": "path_to_ecr_repo:image1tag",
    "environment": [
        {
    "name" : "ACCESS_KEY",
    "value" : "secret"
  },
  {
   "name" : "SECRET_KEY",
   "value" : "secret"
  },
  {
   "name" : "ENVIRONMENT",
   "value" : "quality"
  },
  {
   "name" : "spring_profiles_active",
   "value" : "aws_config"
  },
  {
   "name" : "JAVA_OPTS",
   "value" : "-Xmx3072M -Xms3072M"
  }
   ],
    "memoryReservation": 1024,
    "healthCheck": {
      "retries": 3,
      "command": [
        "CMD-SHELL",
        "/bin/true"
      ],
      "interval": 10,
      "timeout": 9,
      "startPeriod": 15
    },
        "portMappings": [
      {
        "hostPort": 80,
        "containerPort": 80,
        "protocol": "tcp"
      }
    ],
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-group": "/ecs/fargate-testing/abc",
        "awslogs-region": "eu-central-1",
        "awslogs-stream-prefix": "ecs"
      }
    },
    "dockerLabels": {
      "environment": "quality"
    }
  }
,  {
    "name": "tomcat",
    "image": "path_to_ecr_repo:image2tag",
    "environment": [
              {
            "name" : "Environment",
            "value" : "quality"
        }, {
            "name" : "APP_PROFILE",
            "value" : "test_value"
        }
    ],
    "memoryReservation": 1024,
    "healthCheck": {
      "retries": 3,
      "command": [
        "CMD-SHELL",
        "/bin/true"
      ],
      "interval": 10,
      "timeout": 9,
      "startPeriod": 15
    },
      
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-group": "/ecs/fargate-testing/abc",
        "awslogs-region": "eu-central-1",
        "awslogs-stream-prefix": "ecs"
      }
    },
    "dockerLabels": {
      "environment": "quality"
    }
  }
  ]

Best Regards,
Oleg

[bhoormeena]

facing the same issue like @oavdonin and same goes to the most confusing part that it is not 100% reproducable.

Kindly share if this issue is fixed by some other ticker or any info regarding the same will also help.

regards,
Bhoor

[patrykk2252]

I have the same issue. In my case the modification of the service and restart of the service every time I run 'terraform apply' happens only for services with task definition which includes mountPoints

 [
  {
    "name": "${name}",
    "hostname": "${hostname}",
    "image": "${image}",
    "essential": true,
    "portMappings": [
      {
        "containerPort": 9092,
        "hostPort": 9092
      },
      {
        "containerPort": 9094,
        "hostPort": 9094
      }
    ],
    "environment": [
      {
        "name": "KAFKA_ZOOKEEPER_CONNECT",
        "value": "${kafka_zookeeper_connect}"
      },
      {
        "name": "HOSTNAME_COMMAND",
        "value": "curl http://169.254.169.254/latest/meta-data/hostname"
      },
      {
        "name": "KAFKA_ADVERTISED_LISTENERS",
        "value": "INSIDE://:9092,OUTSIDE://_{HOSTNAME_COMMAND}:9094"
      },
      {
        "name": "KAFKA_LISTENERS",
        "value": "INSIDE://:9092,OUTSIDE://:9094"
      },
      {
        "name": "KAFKA_LISTENER_SECURITY_PROTOCOL_MAP",
        "value": "INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT"
      },
      {
        "name": "KAFKA_LISTENERS",
        "value": "INSIDE://:9092,OUTSIDE://:9094"
      },  
      {
        "name": "KAFKA_INTER_BROKER_LISTENER_NAME",
        "value": "INSIDE"
      },
      {
        "name": "KAFKA_BROKER_ID",
        "value": "${kafka_broker_id}"
      },
      {
        "name": "HOSTNAME",
        "value": "${hostname}"
      }  
    ],
    "mountPoints": [ 
      {
        "sourceVolume": "kafka-data",
        "containerPath": "/kafka",
        "readOnly": false
      } 
    ]    
  }
]

If I run the same service with the same task def. but without the mountPoints everything works fine.
Any ideas?

[rafaelss]

same issue here. my container definition have a lot of environment variables. with 41 variables it works fine, when I add one more, terraplan plan forces a new resource. I'm using a template_file to pass variables and render the template. it could be the size of the template file, maybe?

[wr0ngway]

I see this when I have duplicate environment variables with the same name and value at different locations within the set in the container definition. That is, I noticed the problem when I started using a formatlist to generate the environment json from a map, but forgot to take out one of the variables that were in the map from the hardcoded portion of the environment json in the container definition json. I discovered this by setting "environment": [], applied a few times . to verify it was no longer dirty, then starting to add back things in till it broke. Not sure if this is something terraform can/should fix, or maybe warn about.

[bflad]

Hi folks 👋 It appears this issue has diverged from the original bug report. The later comments, while partially fitting the issue title, are ultimately are due to different circumstances in the aws_ecs_task_definition resource and not in the aws_ecs_service resource itself. I'll try to step through the various scenarios described here, but please note I'll be closing this issue since the original post is answered below. Using Terraform 0.12 with its richer JSON plan output might also be very helpful.

Please note that we use GitHub issues for tracking bugs and enhancements rather than for questions. While we may be able to help with certain simple problems here it's generally better to use the community forums where there are far more people ready to help, whereas the GitHub issues here are generally monitored only by the small set of code maintainers.

If you feel like you have a valid bug report still when using recent versions of Terraform 0.12 and the Terraform AWS Provider, please feel free to submit a new issue filling out all the details requested by the Bug Report issue template. Thanks. 👍

---

For the original report (@pjain00), a perpetual difference of the following in a aws_ecs_service resource:

~ task_definition                    = "bflad-testing:1" -> "bflad-testing"

This can be reproduced with the following simplified, but full configuration:

terraform {
  required_providers {
    aws = "2.34.0"
  }

  required_version = "0.12.13"
}

provider "aws" {
  region = "us-east-2"
}

resource "aws_ecs_cluster" "test" {
  name = "bflad-testing"
}

resource "aws_ecs_task_definition" "test" {
  family = "bflad-testing"

  container_definitions = <<DEFINITION
[
  {
    "cpu": 128,
    "essential": true,
    "image": "busybox:latest",
    "memory": 128,
    "name": "busybox"
  }
]
DEFINITION
}

resource "aws_ecs_service" "test" {
  cluster         = aws_ecs_cluster.test.id
  desired_count   = 0
  name            = "bflad-testing"
  task_definition = aws_ecs_task_definition.test.family
}

This difference is due to how the ECS API works, while it allows submitting just the ECS Task Definition family name, it will always return the family name and active revision number separated with a colon (:).

To fix this in your configuration, you can either concatenate the family name and revision as the API (and therefore Terraform) expects, e.g.

resource "aws_ecs_service" "test" {
  # ... other settings omitted for clarity ...
  task_definition = "${aws_ecs_task_definition.test.family}:${aws_ecs_task_definition.test.revision}"
}

Or use the ECS Task Definition ARN (which will show a difference the first apply but be stable afterwards), e.g.

resource "aws_ecs_service" "test" {
  # ... other settings omitted for clarity ...
  task_definition = aws_ecs_task_definition.test.arn
}

If an external system (such as a Continuous Deployment system) is updating the ECS Task Definition revisions external to Terraform, you will need to use ignore_changes to allow Terraform to ignore the attribute value difference:

resource "aws_ecs_service" "test" {
  # ... other settings omitted for clarity ...
  task_definition = "${aws_ecs_task_definition.test.family}:${aws_ecs_task_definition.test.revision}"

  lifecycle {
    ignore_changes = [task_definition]
  }
}

It is also worth mentioning that the above ignore_changes addition could also help the original situation, however we generally encourage fixing configurations to match the expected values instead of introducing potentially unnecessary workarounds.

Please see the aws_ecs_service resource documentation examples and task_definition argument for more information.

---

For this report (@oavdonin), I'm not able to reproduce any issue with the following configuration (no difference on second plan):

terraform {
  required_providers {
    aws = "2.34.0"
  }

  required_version = "0.12.13"
}

provider "aws" {
  region = "us-east-2"
}

resource "aws_ecs_cluster" "test" {
  name = "bflad-testing"
}

resource "aws_ecs_task_definition" "test" {
  family = "bflad-testing"

  container_definitions = <<DEFINITION
[
      {
    "name": "lb-nginx",
    "image": "busybox:latest",
    "environment": [
        {
    "name" : "ACCESS_KEY",
    "value" : "secret"
  },
  {
   "name" : "SECRET_KEY",
   "value" : "secret"
  },
  {
   "name" : "ENVIRONMENT",
   "value" : "quality"
  },
  {
   "name" : "spring_profiles_active",
   "value" : "aws_config"
  },
  {
   "name" : "JAVA_OPTS",
   "value" : "-Xmx3072M -Xms3072M"
  }
   ],
    "memoryReservation": 1024,
    "healthCheck": {
      "retries": 3,
      "command": [
        "CMD-SHELL",
        "/bin/true"
      ],
      "interval": 10,
      "timeout": 9,
      "startPeriod": 15
    },
        "portMappings": [
      {
        "hostPort": 80,
        "containerPort": 80,
        "protocol": "tcp"
      }
    ],
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-group": "/ecs/fargate-testing/abc",
        "awslogs-region": "eu-central-1",
        "awslogs-stream-prefix": "ecs"
      }
    },
    "dockerLabels": {
      "environment": "quality"
    }
  }
,  {
    "name": "tomcat",
    "image": "busybox:latest",
    "environment": [
              {
            "name" : "Environment",
            "value" : "quality"
        }, {
            "name" : "APP_PROFILE",
            "value" : "test_value"
        }
    ],
    "memoryReservation": 1024,
    "healthCheck": {
      "retries": 3,
      "command": [
        "CMD-SHELL",
        "/bin/true"
      ],
      "interval": 10,
      "timeout": 9,
      "startPeriod": 15
    },
      
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-group": "/ecs/fargate-testing/abc",
        "awslogs-region": "eu-central-1",
        "awslogs-stream-prefix": "ecs"
      }
    },
    "dockerLabels": {
      "environment": "quality"
    }
  }
  ]
DEFINITION
}

resource "aws_ecs_service" "test" {
  name            = "bflad-testing"
  cluster         = aws_ecs_cluster.test.id
  task_definition = "${aws_ecs_task_definition.test.family}:${aws_ecs_task_definition.test.revision}"
  desired_count   = 0
}

$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

aws_ecs_cluster.test: Refreshing state... [id=arn:aws:ecs:us-east-2:--OMITTED--:cluster/bflad-testing]
aws_ecs_task_definition.test: Refreshing state... [id=bflad-testing]
aws_ecs_service.test: Refreshing state... [id=arn:aws:ecs:us-east-2:--OMITTED--:service/bflad-testing]

------------------------------------------------------------------------

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.

---

For the rest of the comments, we would need to see a fully reproducible configuration to potentially triage these. As mentioned above, if you feel like you have a valid bug report still when using recent versions of Terraform 0.12 and the Terraform AWS Provider, please feel free to submit a new issue filling out all the details requested by the Bug Report issue template. Thanks. 👍

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!