feat(on-premises): node labels and annotations

[ralgozino]

Summary

• Added the possibility to set labels and annotations for both the control-plane nodes and the regular nodes.
• Added new (optional) fields to the on-prem schema, with pattern validation.
• Added logic to the Ansible playbook to set the labels and annotations on the nodes, and also remove them when deleted from the furyctl.yaml file, checking against the saved state.

Description

This PR extends the logic we had for setting the nodes roles and adds the possibility to set custom labels and annotations to both control-plane nodes and regular nodes for on-premises clusters.

We use kubectl commands instead of kubeadm/kubelet configuration because:

• kubeadm/kubelet configuration is set when joining the node.
• kubeadm/kubelet configuration does not support setting some special labels, like the one that sets the node's role, and this is a use case that we wan't to cover.

Labels and annotations are defined at node group level (control-plane, workers, infra, etc. ) instead of at single node level, for simplicity and I believe most of the time this is what the user would want to do.

Notice that furyctl still adds the role control-plane for the control plane nodes and the node group name as a role for the rest of the nodes. If a user adds manually the same label and then deletes it in a second apply, the label with the default role won't be deleted.

Labeling (or annotating) single nodes is not supported by this feature.

Example:

...
spec:
  kubernetes:
    masters:
      hosts:
        - name: master1
          ip: 192.168.66.31
      labels:
        node-role.kubernetes.io/dungeon-master: ""
        dnd-enabled: "true"
      annotations:
        level: "100"
    nodes:
      - name: infra
        hosts:
          - name: infra1
            ip: 192.168.66.32
          - name: infra2
            ip: 192.168.66.33
          - name: infra3
            ip: 192.168.66.34
        taints:
          - effect: NoSchedule
            key: node.kubernetes.io/role
            value: infra
        labels:
          a-label: with-content
          empty-label: ""
          label/sighup: "with-slashes"
          node-role.kubernetes.io/wizard: ""
          dnd-enabled: "true"
        annotations:
          with-spaces: "annotation with spaces"
          without-spaces: annotation-without-spaces
          level: "20"
      - name: worker
        hosts:
          - name: worker1
            ip: 192.168.66.35
        taints: []
        labels:
          node-role.kubernetes.io/barbarian: ""
          dnd-enabled: "true"
          label-custom: "with-value"
        annotations:
          level: "10"
      - name: empty-labels-and-annotations
        hosts:
          - name: empty1
            ip: 192.168.66.50
        taints: []
        labels:
        annotations:
      - name: undefined-labels-and-annotations
        hosts:
          - name: undefined1
            ip: 192.168.66.51
        taints: []
...

Tests

The following manual tests were performed:

• Adding labels and annotations that do not exist
• Overwriting labels and annotations that already exist with a different value.
• Adding labels and annotations with an empty value
• Overwriting labels and annotations with an empty value
• Node groups with no labels: propery defined.
• Node groups with labels: propery defined but empty.
• Annotations with spaces in the value.
• Annotations and label keys with spaces are not allowed.
• Setting the label that defines the node role works.
• Create a cluster from scratch without custom labels and annotations, add some after creation, then delete them. All using the furyctl.yaml file.
• Create a cluster from scratch with custom labels and annotations. Checked that they are present after creation.
• Removing a label from the furyctl.yaml that has been manually deleted before from the cluster does not produce an error.
• Removing an annotation from the furyctl.yaml that has been manually deleted before from the cluster does not produce an error.
• Multi master cluster: labels and annotations are set on all nodes.
• Multi node groups: labels and annotatoins are properly set to the nodes of the node group.
• Execution of furyctl apply does not fail when the furyctl-config secret is manually deleted from the cluster.

Future work

One possible improvement for the future is that furyctl already has downloaded and checked the previous cluster state (the furyctl-config secret). So it could make this information available to the template engine instead of downloading it again like we do now.

[speziato]

seems legit overall! I pointed out just two typos, when I'm back home I will also try this locally

[speziato]

LGTM! 🚀

[ralgozino]

I added an example to the description and an unrelased.md file with the new feature.

should we point this PR to feat/release-v1.31.0 instead of main @nutellinoit ? done

[stefanoghinelli]

The changes are OK.
In a future iteration, we could integrate a check based on Kubernetes'official well-known labels and annotations.

[nutellinoit]

LGTM for me!