ImageUpdateAutomation commits the same changes over and over on .spec.PushBranch

[klainte]

Describe the bug

Since v0.37.0, ImageUpdateAutomation with a .spec.PushBranch different from its spec.GitCheckoutRefBranch continuously commits modifications picked up by ImagePolicy in the .spec.PushBranch until the modifications are merged on the main branch.

We have notifications set up to MS Teams when a commit is performed on the repo.
with v0.36.0, the changes induced by ImagePolicy + ImageUpdateAutomation were only committed once to the target branch.
Now it is committed over and over, every 'interval` delay (5 min in our case).

Screenshot and logs available below.

I saw the change introduced by the v0.37.0 regarding "Automatic force-push of ImageUpdateAutomation". Maybe that's related, but I do not think the observed behaviour is expected.

Steps to reproduce

• Flux v0.37.0
• Self-hosted Gitlab as GitRepo
  To reproduce, you need a similar configuration

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 1m0s
  ref:
    branch: master
  secretRef:
    name: flux-system
  url: _redacted_
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
  name: git-updater-staging
  namespace: flux-system
spec:
  git:
    checkout:
      ref:
        branch: master
    commit:
      author:
        email: _redacted_
        name: fluxcdbot
      messageTemplate: '{{range .Updated.Images}}{{println .}}{{end}}'
    push:
      branch: staging
  interval: 5m0s
  sourceRef:
    kind: GitRepository
    name: flux-system
  update:
    path: ./apps/staging
    strategy: Setters
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImagePolicy
metadata:
  name: my-policy
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: my-repo-ref
    namespace: flux-system
  filterTags:
    pattern: '^release-(?P<rv>.*)$'
    extract: '$rv'
  policy:
    semver:
      range: '>0.0.0-0'
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: my-hr
spec:
  chart:
    spec:
      version: "1.3.x"
  values:
    replicaCount: 2
    servicemonitor:
      enabled: false
    image:
      tag: "release-4.3.0-rc.1" # {"$imagepolicy": "flux-system:my-policy:tag"}

Expected behavior

Same as v0.36: changes are committed only once.

Screenshots and recordings

Log from the ImageAutomationController

{"level":"info","ts":"2022-12-08T08:32:17.154Z","msg":"pushed commit to origin","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"git-updater-staging","namespace":"flux-system"},"namespace":"flux-system","name":"git-updater-staging","reconcileID":"a33e726e-9151-4c39-96b7-7cafa8866561","revision":"7030d70f5e14659de06dd606be4b978d2dcadec7","branch":"staging"}
{"level":"info","ts":"2022-12-08T08:37:19.525Z","msg":"pushed commit to origin","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"git-updater-staging","namespace":"flux-system"},"namespace":"flux-system","name":"git-updater-staging","reconcileID":"d1cbbf5d-15f8-4de3-b159-ca63c6596310","revision":"a01c76d335c89d2c1bc94f2a59e8d4d2a1e54e70","branch":"staging"}
{"level":"info","ts":"2022-12-08T08:42:22.077Z","msg":"pushed commit to origin","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"git-updater-staging","namespace":"flux-system"},"namespace":"flux-system","name":"git-updater-staging","reconcileID":"91a79988-f449-40b6-a92a-610c227985cb","revision":"4aadf4d779834b0f69bb06de69ac7607d782c7a4","branch":"staging"}

Corresponding notifications

OS / Distro

Ubuntu 18.04.6 LTS

Flux version

v0.37.0

Flux check

flux check
► checking prerequisites
✔ Kubernetes 1.24.8+IKS >=1.20.6-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.27.0
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.27.0
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.23.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.31.0
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.29.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.32.1
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta1
✔ buckets.source.toolkit.fluxcd.io/v1beta1
✔ gitrepositories.source.toolkit.fluxcd.io/v1beta1
✔ helmcharts.source.toolkit.fluxcd.io/v1beta1
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta1
✔ imagepolicies.image.toolkit.fluxcd.io/v1beta1
✔ imagerepositories.image.toolkit.fluxcd.io/v1beta1
✔ imageupdateautomations.image.toolkit.fluxcd.io/v1beta1
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1beta2
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta1
✔ receivers.notification.toolkit.fluxcd.io/v1beta1
✔ all checks passed

Git provider

Gitlab CE self-hosted

Container Registry provider

No response

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[stefanprodan]

Duplicate of: fluxcd/image-automation-controller#466

[klainte]

Sorry about that, I forgot to look into the controller specific github repo, closing!

[pjbgf]

Reopening as this is orthogonal to fluxcd/image-automation-controller#466. This issue ignores changes already made and pushed into a push branch, and do them again, however the commits are not empty. I got this reproduced and am working on a fix.

The other issue is not related to push branches, the controller keeps on writing empty commits.

[pjbgf]

@klainte can you please try the image below and check whether the error persists?
ghcr.io/fluxcd/image-automation-controller:rc-92f0c052

[klainte]

Ok so I was able to test the rc-92f0c052.

On the one hand, it seems to have fixed the issue (no more repeating commit), on the other hand it introduced a strange behaviour, which might also be due my config.

I have 2 ImageUpdateAutomation using the same target branch (I'm planning to change that but it's been working well so far).

---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
  name: git-updater-production-1
  namespace: flux-system
spec:
  git:
    checkout:
      ref:
        branch: master
    commit:
      author:
        email: _redacted_
        name: fluxcdbot
      messageTemplate: '{{range .Updated.Images}}{{println .}}{{end}}'
    push:
      branch: production
  interval: 30m0s
  sourceRef:
    kind: GitRepository
    name: flux-system
  update:
    path: ./apps/production-1
    strategy: Setters
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
  name: git-updater-production-2
  namespace: flux-system
spec:
  git:
    checkout:
      ref:
        branch: master
    commit:
      author:
        email: _redacted_
        name: fluxcdbot
      messageTemplate: '{{range .Updated.Images}}{{println .}}{{end}}'
    push:
      branch: production
  interval: 30m0s
  sourceRef:
    kind: GitRepository
    name: flux-system
  update:
    path: ./apps/production-2
    strategy: Setters

while this used to work with ghcr.io/fluxcd/image-automation-controller:v0.27.0 and before (changes were committed on top of each other's in the same target branch as stated in the documentation), it's not with rc-92f0c052.

To trigger the issue, I pushed a new version of a docker image for the corresponding ImageUpdateAutomation, and here is the result:

{"level":"error","ts":"2022-12-14T19:40:55.141Z","msg":"Reconciler error","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"git-updater-production-1","namespace":"flux-system"},"namespace":"flux-system","name":"git-updater-production-1","reconcileID":"4dedd675-72f1-4832-9095-4782c8a09fbf","error":"command error on refs/heads/production: failed to update ref"}
{"level":"info","ts":"2022-12-14T19:40:55.162Z","msg":"pushed commit to origin","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"git-updater-production-2","namespace":"flux-system"},"namespace":"flux-system","name":"git-updater-production-2","reconcileID":"d823cba8-bd48-4894-9f71-e745048f4cd7","revision":"bd7e745395605dbf64cfed1f6c2bf5d1da084c06","branch":"production"}
{"level":"info","ts":"2022-12-14T19:40:57.160Z","msg":"pushed commit to origin","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"git-updater-production-1","namespace":"flux-system"},"namespace":"flux-system","name":"git-updater-production-1","reconcileID":"dd4af75d-2a16-4738-b682-682acb490ef7","revision":"07d2539d2adf3fd8086e3551e221fed8f4e35cb5","branch":"production"}

Instead of pushing to the expected target branch (production), the commit were pushed directly in the master. Oopsy.

You can see the 2 commit short-SHAs matching the IUA controller logs ones.

This only occurs when having 2 ImageUpdateAutomation using the same PushBranch. It may also be linked to the fact that both are using the same interval, and the controller is having trouble resolving them at the same time?

I have 2 other ImageUpdateAutomation using their own branch, and I had no issue with rc-92f0c052.

Maybe my config is just inappropriate?

[pjbgf]

@klainte Thank you for coming back so quickly. Would you mind testing this again with this image instead: ghcr.io/fluxcd/image-automation-controller:rc-982331e6

It should fix the branch issue you reported.

As for your config I believe it is fine, and the controller should be able to support it. The only thing I would suggest is that you probably want to have force push disabled, so one automation does not overwrite the other. To do that, you can try to start the controller with:

--feature-gates=GitForcePushBranch=false

For more information, please refer to the v0.27.0 change log.

[klainte]

@pjbgf Tests are conclusive with rc-982331e6. No more commit spam, changes committed properly and only once in the proper target branch.

I saw the v0.27.0 change but decided to give it a try without the forcePush disabled.
As both my policies target different folder, everything is running smoothly.

Thanks :)

[pjbgf]

That's awesome, thank you @klainte for confirming. 🙇 We should release a new official version with the fix soon.

[pjbgf]

The new official image automation controller image contains the fix: ghcr.io/fluxcd/image-automation-controller:v0.28.0. We are in the process of releasing Flux v0.38.0 which will include that image.