-
Notifications
You must be signed in to change notification settings - Fork 262
chart is upgraded to out-of-sync git repo #562
Comments
Hello folks, our repo structure:
in order to update a chart we create a feature branch
this correctly installs thee new chart based on feat-232 on the destination cluster. When we merge this last PR we noticed that on some clusters (not all of them) a re-deploy happens when it shouldn’t.
looking at the diff between the releases we see: We believe that there is a sync problem between the event of CRD update and the branch used to calculate the diff. |
Given we are in maintenance mode, it would be great to know if this issue is reproducible in the latest Helm Operator version ( |
hello @hiddeco, thank you for the reply. With that being said, updating atm doesn't guarantee that this "global" deploy problem goes away... and we would like to fix this problem we are having in the short term. We would love and input from one of you folks on what can it be (because on a first pass on the codebase we didn't find anything) and we will be happy to dig further and open a PR. |
I can guess at what's happening here: the code mirroring the git repository does so asynchronously, and when you update the custom resource to switch branches, there's a race between the upgrade running and the mirror having fetched the merge commit. The relevant code is around here: https://github.com/fluxcd/helm-operator/blob/master/pkg/chartsync/git.go. This matches the git repos used in HelmRelease resources with those currently being mirrored; when there's a lack, a new mirror is set up. But if a mirror already exists, it's assumed that it is usable. I don't think this design accounts for the use you are putting it to. When you switch branches, there is nothing that makes sure it has the current head of the updated branch. (Why doesn't this cause a problem when you switch to the feature branch? If the ref is missing entirely, it error-loops until it's mirrored that ref.) To work around it, you could update the git ref used in the HelmRelease to the merge commit, rather than master; and once that is released, you know it has that particular commit and it's safe to switch to master branch. |
@squaremo thanks for the detailed explanation. |
We don't use Helm charts in Weave Cloud -- it's all driven by updating images automatically, and occasionally adapting config by hand. We sync the main branch, and rely on mistakes showing up in dev before the same change is made for production. I would not hold this up as the ideal process though. If you can manage the workaround (instead of reverting the chart ref, change it to a specific revision before proceeding), that's what I'd do. As a more future-proof alternative, consider porting your system to use the |
@squaremo thank you for responding. could you confirm if this issue wont be there if we port our system to the new helm-controller ? thanks |
I expect that when you changed the branch back to But don't rely on my head-computer -- I recommend making a throwaway environment to persuade yourselves it will work with your workflow. Getting all the bits running is easier than with helm-operator -- see https://toolkit.fluxcd.io/get-started/. |
@squaremo we have analyzed the code. there will always be a possibility that whenever we switch the helmrelease's gitrepo branch (feature-branch -> master, eg), that mirror ll be out of sync. We have created a PR to detect the switch and avoid the incorrect helm reconciliation by trigger the mirror sync upfront. (in the above PR) |
Describe the bug
helmrelease X is upgraded to an out-of-sync chart sources(git repo) before being correctly upgraded to in-sync chart sources (after 10 mins).
To Reproduce
We were not able to reproduce it. looks like a corner case. we dived into the code to see if we could detect something, however, not much progress there either.
timeline of events:
we have two branches master and feature-branch. both contain identical chart sources.
timeline:
t0: X's helmrelease.spec.chart.ref is updated from feature-branch -> master
t0+2mins: X is upgraded to out-of-sync chart sources
t0+12mins: X is upgraded to in-sync chart sources
Steps to reproduce the behaviour:
kubectl describe helmrelease <name>
Expected behavior
the helm upgrade should not have been to out-of-sync chart sources.
Logs
dont have the logs related to that timeline.
Additional context
The text was updated successfully, but these errors were encountered: