Add support for Kustomize components

[klausenbusk]

We are using Flux's Kustomization for its dependsOn functionality to install the workloads in a specific order, ex: Kyverno first, then CRDs (ServiceMonitors, ExternalSecrets etc.) and last team applications. It works very well.

Kustomize supports components and part of the motivation is (described in the KEP):

The problem is that modular applications cannot always be expressed in a tall hierarchy while preserving all combinations of available features. Doing so would require putting each feature in an overlay, and making overlays for independent features inherit from each other. However, this is semantically incorrect, cannot not scale as the number of features grows, and soon results in duplicate manifests and kustomizations.

Instead, such applications are much better expressed as a collection of components, i.e., reusable pieces of configuration logic that are defined in a common place and that distinct overlays can then mix-and-match. This approach abides by the DRY principle and increases ease of maintenance.

This is a very useful feature and what we need for our use-case. Unfortunately, it is bit hard to use with Flux's Kustomization.

In our case we have a few environments (clusters):

.
├── environments
│   ├── base
│   │   ├── flux-system
│   │   │   ├── gotk-components.yaml
│   │   │   ├── gotk-sync.yaml
│   │   │   └── kustomization.yaml
│   │   ├── infrastructure-sources.yaml
│   │   ├── kustomization.yaml
│   │   └── monitoring-kustomization.yaml
│   ├── dev
│   │   ├── flux-system
│   │   │   └── gotk-sync.yaml
│   │   └── kustomization.yaml
│   └── prod
│       ├── flux-system
│       │   └── gotk-sync.yaml
│       └── kustomization.yaml

The rest of the structure

└── infrastructure
    ├── monitoring
    │   ├── cortex
    │   │   ├── bkt-claim.yaml
    │   │   ├── cortex-hr.yaml
    │   │   └── kustomization.yaml
    │   ├── grafana
    │   │   ├── grafana-hr.yaml
    │   │   └── kustomization.yaml
    │   ├── kustomization.yaml
    │   ├── kube-prometheus-stack
    │   │   ├── kube-prometheus-stack-hr.yaml
    │   │   ├── kustomization.yaml
    │   │   └── namespace.yaml
    │   └── loki
    │       ├── bkt-claim.yaml
    │       ├── kustomization.yaml
    │       ├── loki-hr.yaml
    │       └── promtail-hr.yaml
    └── sources
        ├── cortex-project.yaml
        ├── grafana-helm-charts.yaml
        ├── kustomization.yaml
        ├── prometheus-community.yaml

We want to enable some optional features for Grafana in the dev environment, let's say OAuth and add a ingress.

If Flux's Kustomization supported components that would be very easy to implement, I suppose it could look like this:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: monitoring
  namespace: flux-system
spec:
  components:
    - ../../components/grafana-oauth
    - ../../components/grafana-ingress

(preferable with x-kubernetes-patch-merge-key and x-kubernetes-patch-strategy for the components so it can be strategic merged)

Today the only option is patching the Flux Kustomization and creating a new overlay:

environments/dev/monitoring-kustomization.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: monitoring
  namespace: flux-system
spec:
  path: ./components-hack/monitoring-dev

components-hack/monitoring-dev/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../infrastructure/monitoring
components:
    - ../../components/grafana-oauth
    - ../../components/grafana-ingress

This is a bit cumbersome and adds unnecessary complexity. If Flux's Kustomization supported components we could avoid all this.