Apply twice keeps on showing changes for code_engine_app

[l2fprod]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform IBM Provider Version

Terraform v1.5.3
on darwin_arm64
+ provider registry.terraform.io/ibm-cloud/ibm v1.55.0

Affected Resource(s)

• ibm_code_engine_app

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

resource "ibm_code_engine_project" "project" {
  name              = "${var.basename}-project"
  resource_group_id = ibm_resource_group.group.id
}

resource "ibm_code_engine_app" "app" {
  project_id          = ibm_code_engine_project.project.project_id
  name                = "${var.basename}-app"
  image_reference     = "icr.io/codeengine/helloworld"
  scale_cpu_limit     = "0.25"
  scale_memory_limit  = "0.5G"
  scale_min_instances = 0
  scale_max_instances = 1
  scale_concurrency   = 1
}

Expected Behavior

Two subsequent calls to apply should not show any changes on the second call

Actual Behavior

Calling apply and apply again show changes everytime, even though I did not change anything

Steps to Reproduce

1. terraform apply -> OK. Project and app are created
2. terraform apply again
   -> terraform detect changes, although I did change nothing

ibm_resource_group.group: Refreshing state... [id=57dbe218112b43d2bc7023b34d85ddc8]
ibm_code_engine_project.project: Refreshing state... [id=93530e14-f951-4b87-bb08-0f1e9eadd8f1]
ibm_code_engine_app.app: Refreshing state... [id=93530e14-f951-4b87-bb08-0f1e9eadd8f1/xxx-demo-app]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # ibm_code_engine_app.app will be updated in-place
  ~ resource "ibm_code_engine_app" "app" {
        id                            = "93530e14-f951-4b87-bb08-0f1e9eadd8f1/xxx-demo-app"
        name                          = "xxx-demo-app"
      ~ scale_memory_limit            = "500M" -> "0.5G"
        # (24 unchanged attributes hidden)

      - run_env_variables {
          - name  = "CE_SUBDOMAIN" -> null
          - type  = "literal" -> null
          - value = "15cgvt6p2451" -> null
        }
      - run_env_variables {
          - name  = "CE_APP" -> null
          - type  = "literal" -> null
          - value = "xxx-demo-app" -> null
        }
      - run_env_variables {
          - name  = "CE_DOMAIN" -> null
          - type  = "literal" -> null
          - value = "us-east.codeengine.appdomain.cloud" -> null
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

• this should not happen, there should be no change
• notice how the memory limit is being rewritten, although it is the same
• notice how the env variable are being removed

Important Factoids

References

• #0000

[michael-magrian]

Thanks for reporting this @l2fprod.
I'll bring this up in the team and discuss potential approaches.

Regards, Michael

[SRodi]

@michael-magrian is there any progress made on solving this issue? I have opened #4852 as I can see no activity for the past few months on this. Please feel free to close #4852 as duplicate if you are actively working on this.

[michael-magrian]

I'll bring this topic up again internally. Thanks for the reminder.

[akocbek]

+1 we are seeing the same behavior with env variables.

tip: In contrast ibm_code_engine_job, which also supports env variables block usage, doesn't have that problems. (maybe similar logic can be used for apps)

[michael-magrian]

Thanks for the message. There is actually a simple explanation why jobs don't cause the same issue. They don't receive "default" environment variables during creation, which is why it doesn't pop up during a terraform plan or terraform apply. Still, the issue remains and we need to address it.

[akocbek]

fyi, the same behavior can be seen for ibm_code_engine_job resource now

~ resource "ibm_code_engine_job" "ce_job" {
        id                            = "5e7a7942-774b-4413-a83a-2aa8432976bb/ce-jobs-test-job-2"
        name                          = "ce-jobs-test-job-2"
        # (19 unchanged attributes hidden)

      - run_env_variables {
          - name  = "CE_REGION" -> null
          - type  = "literal" -> null
          - value = "us-south" -> null
        }
      - run_env_variables {
          - name  = "CE_PROJECT_ID" -> null
          - type  = "literal" -> null
          - value = "5e7a7942-774b-4413-a83a-2aa8432976bb" -> null
        }
      - run_env_variables {
          - name  = "CE_API_BASE_URL" -> null
          - type  = "literal" -> null
          - value = "https://api.us-south.codeengine.cloud.ibm.com/" -> null
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

[nthomsencph]

What is the status on this issue, @michael-magrian and team?

[michael-magrian]

Hello @nthomsencph,
we are currently working on a guide to work around this issue. There is no easy technical solution we can implement right now, but you can avoid this by setting some definitions. I'll update the issue when that is public.
Thanks for your patience.

[saevarb]

@michael-magrian Is it possible to get some sort of ETA?

This issue was created in july last year, so it's almost a year old, and nothing has happened since.

This issue is particularly bothersome when combined with #5232. This effectively means that 1) it's not possible to actually use service bindings via terraform and 2) workarounds where you create everything yourself are extra annoying to implement due to this issue.

There is actually a simple explanation why jobs don't cause the same issue. They don't receive "default" environment variables during creation, which is why it doesn't pop up during a terraform plan or terraform apply. Still, the issue remains and we need to address it.

This really begs the question: when implementing this part of the provider, was it not tested? I've been using Code Engine since before there was any real terraform support for it, and even back then, there were default environment variables that were added on creation, so this behavior can't have come as a surprise. Should I take this explanation to mean that when this feature was implemented, no one actually tried running terraform apply twice in sequence when testing it?

EDIT:

Another thing which might be pertinent to this issue: It also seems that the order of run_env_variables matters, which I think might be considered a bug. The order is not important inside code engine, is it? It means that even if you have all the right environment variables configured in terraform as is running live, if your order is wrong, tf will want to update your service. Combined with secret or config references, this means you will regularly have to fix the order because those are moved to the top when you create them.

[nthomsencph]

@michael-magrian would you be able to answer @saevarb s question some time soon? This is an urgent issue for us

[reggeenr]

Hi @nthomsencph , hi @saevarb , thanks for your patience.

The workaround for this issue is to keep the list of environment variables, defined through terraform, in sync with the actual state of the application to avoid unnecessary updates. Recently, we published a document in the troubleshooting section of the Code Engine documentation that explains the workaround in detail https://cloud.ibm.com/docs/codeengine?topic=codeengine-ts-terraform-environment-variables

We are aware that this workaround isn't what one would refer to as an ideal solution. Having said that, we are working on a permanent solution that will split the user-defined env vars and those that are computed by the system apart. At this point, I cannot share an ETA for this fix.

Furthermore, as this issue is urgent for you, I suggest that you open a support case https://cloud.ibm.com/docs/codeengine?topic=codeengine-get-support to allow our team of engineers to work more closely with you on a temporary solution.

Another thing which might be pertinent to this issue: It also seems that the order of run_env_variables matters, which I think might be considered a bug. The order is not important inside code engine, is it?

Actually, the order matters. Assuming multiple entries are defined for the same variable name, the last one that is defined will override the previous one(s). The reason why full refs are listed first is particularly important to enable app owners to override the env vars that are defined through full references by specifying specific literal and/or key-ref values.

[nthomsencph]

Hi @reggeenr, thank you for the thorough answer. We are looking forward to a solve for this.