Healthcheck on Job ignores job status

[kcighon]

Healthcheck of Kind: job on Kustomization ignores job status and moves to Ready True Applied.

► checking prerequisites
✗ flux 0.31.1 <0.31.2 (new version is available, please upgrade)
✔ Kubernetes 1.23.6+k3s1 >=1.20.6-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.22.1
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.26.1
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.24.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.25.5
✔ all checks passed

• test job status

Every 2.0s: kubectl get job -n flux-system                   MYSERVER: Tue Jun 28 09:53:24 2022

NAME      COMPLETIONS   DURATION   AGE
testjob   0/1           5m53s      5m53s

• flux get ks testks

NAMESPACE       NAME                                    REVISION                        SUSPENDED       READY   MESSAGE
flux-system     kustomization/testks                  main/6335077                  False           True    Applied revision: main/6335077 

• however test job still erroring

Every 2.0s: kubectl get pods -n flux-system                  MYSERVER: Tue Jun 28 09:52:36 2022

NAME                                      READY   STATUS    RESTARTS   AGE
helm-controller-9bb4968f9-zbq5f           1/1     Running   0          7m42s
kustomize-controller-655c6969df-wrdrq     1/1     Running   0          7m42s
source-controller-66d8b45dfb-nqftz        1/1     Running   0          7m42s
notification-controller-b8cd574d6-wg579   1/1     Running   0          7m42s
testjob-pgcnj                             0/1     Error     0          5m5s
testjob-vkcpv                             0/1     Error     0          4m54s
testjob-nm7wt                             0/1     Error     0          4m43s
testjob-2nrfp                             0/1     Error     0          4m32s
testjob-kj22j                             0/1     Error     0          4m20s

• testks.yaml

---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: testks
  namespace: flux-system
spec:
  force: true
  interval: 60m0s
  healthChecks:
    - apiVersion: batch/v1
      kind: Job
      name: testjob
      namespace: flux-system
  path: ./tenant/kustomize
  prune: true
  retryInterval: 2m0s
  serviceAccountName: kustomize-controller
  sourceRef:
    kind: GitRepository
    name: flux-system
    namespace: flux-system
  targetNamespace: flux-system
  timeout: 3m0s
  # wait: true

• ./tenant/kustomize/test-job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: testjob
  namespace: flux-system
spec:
  ttlSecondsAfterFinished: 300
  backoffLimit: 180
  template:
    spec:
      serviceAccountName: kustomize-controller
      containers:
        - name: kubectl
          image: "bitnami/kubectl"
          command:
            - /bin/sh
            - -c
            - |
              sleep 1; if [ "$(kubectl get cm test -n default -o jsonpath={.data.dummy})" = "True" ]; then exit 0; fi; sleep 5; exit 1;
      restartPolicy: Never