Rancher suggests users upgrade the k3s-packaged Traefik chart, breaking ingress

[brandond]

Copied from rancher/partner-charts#220 for visibility.

A recent PR added a Traefik chart to partner-charts:

• Traefik chart partner-charts#205

K3s also packages Traefik. If you go into Installed Apps in Apps and Marketplace, there is an option to update it:

If the user clicks through the new version, the chart upgrade will succeed, but the traefik pod will go into ImagePullBackoff because the tag used by version 10.6.0 of the chart (v2.5.3) is not mirrored in the rancher Docker Hub org. This will leave the ingress broken and (if it is being used to access Rancher) lock the user out of Rancher. At this point the user would have to either restart k3s (so that the packaged chart is reinstalled), use kubectl to fix the deployment image tag, or helm to roll back the chart update.

I believe that the Rancher UI should not offer upgrades to K3s or RKE2 packaged components. Especially not on the local cluster.

Originally reported on Users Slack at https://rancher-users.slack.com/archives/C3ASABBD1/p1634871651031000

[vincent99]

The UI has no way to know which helm releases you want to consider special. We can add a label or something for just the UI, but... don't produce & expose Helm metadata (secrets) if you don't want it managed like one?

[brandond]

RKE2 does the same thing, is there no protection for the packaged charts on that side either? With RKE2 we moved more towards using Helm for all the packaged components, but they're not intended to be managed directly by end users, and will in fact get actively reverted to the packaged versions during server node startup.

The difference with the RKE2 charts may be that they're all uniquely named with an rke2- prefix, whereas the k3s traefik chart matches the partner-charts name and is therefore matched for upgrading.

[vincent99]

The UI shows the same thing helm ls would show and helm upgrade would let you do. There's no unique attribution to say "this release came from this chart in this repo" so the upgrade check is just looking for matches by name*, and there is now another set of possible versions matching traefik so you get offered it. RKE2 I assume doesn't have any common name conflicts.

*: with an extra hint for the repo name that the UI sets if you deploy something through it, which won't apply here

[brandond]

Could we have K3s set a bogus or protected value for that repo hint to prevent the UI from matching the one from partner-charts?

[vincent99]

You could add some annotations that the UI could recognize to do that... but that doesn't help any existing installs.

[brandond]

What's the annotation, and on what resource does it need to be set? We could try to roll it into the next round of K3s releases.

cc @cwayne18

[vincent99]

Naming them more uniquely as rke2 does seems like a better idea to me, but if you want the UI to block it I guess something like catalog.cattle.io/ui-upgradable: false. There's a list of others we use somewhere (@cbron?).

They go into the helm release itself, which eventually becomes data.release = base64(base64(gzip({chart: {metadata: {annotations: { your annotations here, ...}, ...}, ...}))) in the secret it generates. Presumably you just add them to the request to install the chart and don't have to go secret-diving, though I don't see an option in helm install.

[brandond]

Hmm, so it's got to be in the release data within the helm secret, not just an annotation on the secret itself... and it can't be set via the helm CLI. That does make things a bit more difficult.

[brandond]

We wrote a Helm plugin to poke at the release metadata to un-stick failed chart operations, I guess we could do another one to set arbitrary annotations, and bake that into our helm-controller image. Does that feel like an appropriate thing for K3s/RKE2 to be doing if we want to keep the UI from allowing changes? Is there anything additional or more correct that we could do?

Tracking this on the K3s side as: k3s-io/k3s#4315

[cbron]

Link sent. Should we make it more generic (ex ui-managed) to cover other cases ? Or is the only thing we are blocking here upgrade and not edit-in-place/introspect etc.. If so then catalog.cattle.io/ui-upgradable: boolean works.

[brandond]

Since catalog.cattle.io/ui-upgradable is still just a proposed annotation, I think I might just go with setting catalog.cattle.io/hidden or catalog.cattle.io/ui-source-repo/catalog.cattle.io/ui-source-repo-type on our packaged helm chart for the time being. We're already preprocessing the upstream chart to split out the CRDs, so I can just add that in the Chart.yaml without having to fuss about with a plugin to set it on existing releases.

It looks like there's a fleet annotation that actually does what I want in terms of disabling upgrades; unfortunately setting this is probably a hack?

dashboard/models/catalog.cattle.io.app.js

Lines 74 to 76 in a17c7de

	if ( this.spec?.chart?.metadata?.annotations?.[FLEET.BUNDLE_ID] ) {
	// Things managed by fleet shouldn't show ugrade available even if there might be.
	return false;

[brandond]

I added a bunch of annotations to the chart - including hidden! - and it still shows up in the UI and offers an upgrade. Is there something else I'm missing to get dashboard to pick up on these annotations? Are there annotations that must be set in order for the ones I want to use to be honored?

apiVersion: catalog.cattle.io/v1
kind: App
metadata:
  annotations:
    objectset.rio.cattle.io/applied: ""
    objectset.rio.cattle.io/id: helm-app
    objectset.rio.cattle.io/owner-gvk: /v1, Kind=Secret
    objectset.rio.cattle.io/owner-name: sh.helm.release.v1.traefik.v1
    objectset.rio.cattle.io/owner-namespace: kube-system
  creationTimestamp: "2021-10-26T00:41:07Z"
  generation: 1
  labels:
    objectset.rio.cattle.io/hash: 299159b444ad8f1fbc1df8e2303e40b8c8c8e226
  managedFields: {}
  name: traefik
  namespace: kube-system
  ownerReferences:
  - apiVersion: v1
    blockOwnerDeletion: false
    controller: true
    kind: Secret
    name: sh.helm.release.v1.traefik.v1
    uid: 3f87f1b6-2983-4d77-8509-6fffff0b6e00
  resourceVersion: "2291"
  uid: 57710c0d-7ebc-4171-898b-db4ca55ecd10
spec:
  chart:
    metadata:
      annotations:
        catalog.cattle.io/certified: rancher
        catalog.cattle.io/hidden: "true"
        catalog.cattle.io/ui-component: k3s
        catalog.cattle.io/ui-source-repo: k3s-charts
        catalog.cattle.io/ui-source-repo-type: cluster
      apiVersion: v2
      appVersion: 2.4.8
      description: A Traefik based Kubernetes ingress controller
      home: https://traefik.io/
      icon: https://raw.githubusercontent.com/traefik/traefik/v2.3/docs/content/assets/img/traefik.logo.png
      keywords:
      - traefik
      - ingress
      maintainers:
      - email: emile@vauge.com
        name: emilevauge
      - email: daniel.tomcej@gmail.com
        name: dtomcej
      - email: ldez@traefik.io
        name: ldez
      name: traefik
      sources:
      - https://github.com/traefik/traefik
      - https://github.com/traefik/traefik-helm-chart
      type: application
      version: 9.18.2

[brandond]

I added a couple more annotations and that didn't seem to make any difference:

spec:
  chart:
    metadata:
      annotations:
        catalog.cattle.io/certified: rancher
        catalog.cattle.io/display-name: Traefik
        catalog.cattle.io/hidden: "true"
        catalog.cattle.io/namespace: kube-system
        catalog.cattle.io/release-name: traefik
        catalog.cattle.io/ui-component: k3s
        catalog.cattle.io/ui-source-repo: k3s-charts
        catalog.cattle.io/ui-source-repo-type: cluster

[brandond]

It looks like the fleet bundle-id annotation is the only one that'll work. It doesn't prevent upgrades, but it does hide the prompt in the UI with a fixed "Managed" label.

[cbron]

Going with a fresh annotation would be better than shoehorning the fleet one.

[brandond]

Yeah but that won't help anyone on existing versions of Rancher. We were already setting the hidden annotation, not knowing that this only worked for charts in the repo, not ones installed to the cluster. I think I'm going to use the fleet annotation for now; we can migrate to a new more specific annotation when it is available.

[rak-phillip]

@brandond do you have any repro steps for getting to a state where kube-system traefik will prompt for update? I'm having issues replicating, but I suspect that I'm missing something in my setup

[brandond]

Shouldn't take anything more than simply installing Rancher 2.6 on K3s.

[brandond]

This has been fixed for the single chart that K3s packages, but on the RKE2 side our chart building process is a bit more complex and it hasn't been addressed yet.

[gaktive]

Backend work moved over to Team 3.

[gaktive]

Based on discussion, we should expect 2 new tickets: one for k3s to create a new annotation (catalog.cattle.io.managed=true or something of that ilk), which should unblock UI work, and another for RKE2 to take that annotation to then edit the bundled charts.

@pennyscissors will do the k3s ticket, which would then spawn the rke2 work via @brandond

[markusewalker]

Verified on v2.6-head rancher/rancher@a478def.

ENVIRONMENT DETAILS

• Linode Ubuntu 20.04 VM
• Docker install: 20.10
• Rancher version: v2.6-head rancher/rancher@a478def
• K3s version: v1.22.8+k3s1
• RKE2 version: v1.22.8+rke2r1

TEST RESULT
PASS

VERIFICATION STEPS

1. Setup Rancher v2.6-head and navigated to Rancher UI home page > Create.
2. Toggled RKE1 to RKE2/K3s and chose Linode node driver to create a K3s cluster.
3. Navigated to top-left dropdown menu > newly created K3s cluster > Apps > Installed Apps.
4. Verified that Rancher is not prompting upgrades for any of the installed K3s apps:
   
5. Repeated steps 2-5, but for a Rancher-created RKE2 cluster. I see the below:
   

As noted in the earlier comments, there will be separate issues to address adding annotations to further enhance this experience, but the original issue has been addressed and verified.