Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle Jobs with ttl_seconds_after_finished = 0 correctly #2596

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

JaylonmcShan03
Copy link
Contributor

@JaylonmcShan03 JaylonmcShan03 commented Oct 1, 2024

Description

Fixes #2531
When a Kubernetes Job has ttl_seconds_after_finished set to 0, Kubernetes deletes the Job immediately after it completes. This can cause Terraform to plan the recreation of the Job in subsequent runs, which is sort of undesirable behavior. The expected behavior is for Terraform to recognize that the Job was deleted intentionally and not attempt to recreate it or show any diffs.

Changes made
- Read function:
- When the read crud func encounters a NotFound error for a job, it now checks if ttl is set to 0, if so it keeps the
resource in the state without marking it as destroyed, this helps prevent the recreation of the job. If it is not 0, it
behaves as before by removing the resource from the state.

CustomizeDiff function introduced:
- Supressing diffs when the job has been deleted due to due to ttl = 0

Acceptance tests

  • Have you added an acceptance test for the functionality being added?
  • Have you run the acceptance tests on this branch?

Output from acceptance testing:

└─(10:13:50 on fix-job-ttl-zero-handling)──> make test TESTARGS="-run TestAccKubernetesJobV1_customizeDiff_ttlZero"
==> Checking that code complies with gofmt requirements...
go vet ./...
go test "/Users/mau/Dev/terraform-provider-kubernetes/kubernetes" -vet=off -run TestAccKubernetesJobV1_customizeDiff_ttlZero -parallel 8 -timeout=30s
ok      github.com/hashicorp/terraform-provider-kubernetes/kubernetes   0.924s
...

Release Note

Release note for CHANGELOG:

Properly handle Kubernetes Jobs with ttl_seconds_after_finished = 0 to prevent unnecessary recreation.

References

Fixes #2531

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

@@ -516,3 +550,28 @@ func testAccKubernetesJobV1Config_modified(name, imageName string) string {
wait_for_completion = false
}`, name, imageName)
}

func testAccKubernetesJobV1Config_customizeDiff_ttlZero(name, imageName string) string {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When applying the tfconfig manually it works as expected. The apply goes through with no diff occurring.

I did however attempt to update ttl_seconds_after_finished from 0 to 5 and got the following error:

(base) ┌─(~/Dev/Scratch/ttl_test)────────────────────────────────────(mau@mau-JKDT676NCP:s017)─┐
└─(10:16:32)──> tfa                                                       ──(Thu,Oct10)─┘
kubernetes_job_v1.test: Refreshing state... [id=default/ttl-test]

Terraform used the selected providers to generate the following execution plan. Resource
actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # kubernetes_job_v1.test will be updated in-place
  ~ resource "kubernetes_job_v1" "test" {
        id                  = "default/ttl-test"
        # (1 unchanged attribute hidden)

      ~ spec {
          ~ ttl_seconds_after_finished = "0" -> "5"
            # (8 unchanged attributes hidden)

            # (2 unchanged blocks hidden)
        }

        # (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.
kubernetes_job_v1.test: Modifying... [id=default/ttl-test]
╷
│ Error: Failed to update Job! API error: jobs.batch "ttl-test" not found
│ 
│   with kubernetes_job_v1.test,
│   on main.tf line 1, in resource "kubernetes_job_v1" "test":
│    1: resource "kubernetes_job_v1" "test" {

From my understanding we're wanting to prevent this error from happening when set to 0 which has been achieved. But in doing so we are now unable to update the existing job that's in tfstate assuming we want to update the already existing job that's part of state despite it having a ttl of 0.

We'll want to consider how to solve this since if this is left users will need to destroy every job that has ttl_seconds_after_finished = 0 if wanting to apply an update to the job already existing in state.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, initially the issue was when ttl_seconds_after_finished = 0, the job deletes, and we don't want terraform to recreate the job in the next apply.

I modified the update function, first I attempt to get the job, if a NotFound error occurs, we proceed to check the previous ttl_seconds_after_finished, I then check if ttl = 0. If so set the resource id to "" to remove it from the state, and with my understanding I thought terraform would recreate it.

But when attempting that solution, I get this error produced an unexpected new value: Root object was present, but now absent.
And I believe that's due to, during an update terraform expects the resource to remain in the state unless explicitly destroyed by the config.

I will give it some more thought today, with that being said do you have another idea in mind? Is this ttl solution viable, meaning with us solving one issue, another edge case arises.

@github-actions github-actions bot added size/L and removed size/M labels Oct 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Terraform lifecycle of a job that has ttl_seconds_after_finished set
2 participants