Getting gardener-extension-registry-cache production ready

[oliver-goetz]

How to categorize this issue?

/area quality cost
/kind task

What would you like to be added:
At the end of our Hackathon we had some tasks on your list in order to make the registry-cache production ready.

Why is this needed:
Getting the registry-cache extension production ready.

Steps

• Refactor current approach to change containerd config files
  • Use the new way of configuring mirrors for containerd. The new way is https://github.com/containerd/containerd/blob/main/docs/hosts.md. The old (current) way is https://github.com/containerd/containerd/blob/main/docs/cri/registry.md#configure-registry-credentials: Refactor the extension using a webhook for OperatingSystemConfigs to add containerd hosts.toml files #27
  • Replace the DaemonSet approach by OperatingSystemConfig mutating webhook. The webhook adds new containerd configuration files under /etc/containerd/cert.d/: Refactor the extension using a webhook for OperatingSystemConfigs to add containerd hosts.toml files #27
  • Introduce a dependency in the Shoot reconciliation flow such as - the OperatingSystemConfig deploy should be after the Extension resources are ready. In this way an extension such as registry-cache can first deploy the registries to the Shoot, then mutate the OperatingSystemConfig with information gathered after the Extension is ready (the Service IP of the registry Services). Deploy OperatingSystemConfig after Extension resources gardener#8232
  • Adapt the containerd-initializer unit in gardener/gardener to configure set the config_path: Introduce the ContainerdRegistryHostsDir feature gate gardener#8094
    Modify the default containerd config in gardener to specify

    [plugins."io.containerd.grpc.v1.cri".registry]
       config_path = "/etc/containerd/certs.d"

    • Make sure that there are no side effects of configuring empty hosts dir for containerd (ref https://github.com/containerd/containerd/blob/main/docs/hosts.md)
    • Graduate the ContainerdRegistryHostsDir feature gate to GA. Clean up the feature gate
      • Promote ContainerdRegistryHostsDir feature gate to beta gardener#8873
      • Promote ContainerdRegistryHostsDir feature gate to GA gardener#8979
      • Remove the GA-ed ContainerdRegistryHostsDir feature gate gardener#9058
    • Clean up migration code for provider-local in gardener/gardener:
      • provider-local: Drop the migration code related to the ContainerdRegistryHostsDir feature gate (part 1) gardener#8441
      • provider-local: Drop the migration code related to the ContainerdRegistryHostsDir feature gate (part 2) gardener#8442
  • Implement orphan file deletion in the cloud-config executor. When the registry-cache extension stops mutating the OperatingSystemConfig, the registry config files will leak on the host. With such orphan file deletion mechanism, cloud-config executor will take care to delete files which are no longer present in the OperatingSystemConfig. According to the docs, updates under the hosts config dir (for example /etc/containerd/cert.d/) do not require restart of containerd.
    • gardener/gardener: Implement orphan file deletion in the cloud-config executor script gardener#8227
    • Vendor gardener/gardener@v1.76.0 in gardener-extension-os-gardenlinux - Bump github.com/gardener/gardener from 1.75.1 to 1.76.0 gardener-extension-os-gardenlinux#118, released with v0.21.0
    • Vendor gardener/gardener@v1.76.0 in gardener-extension-os-suse-chost - Bump github.com/gardener/gardener from 1.75.1 to 1.76.0 gardener-extension-os-suse-chost#101, released with v1.23.0
    • Vendor gardener/gardener@v1.76.0 in gardener-extension-os-ubuntu - Bump github.com/gardener/gardener from 1.75.1 to 1.76.0 gardener-extension-os-ubuntu#88, released with v1.23.0
    • Vendor gardener/gardener@v1.76.0 in gardener-extension-os-coreos and do the required adaptation - Bump github.com/gardener/gardener from 1.75.1 to 1.76.0 gardener-extension-os-coreos#70, released with v1.18.0
  • Using the /etc/containerd/conf.d/ imported files is flawed and doesn't work for all cases: https://github.com/gardener/gardener/blob/master/docs/usage/custom-containerd-config.md / Local Provider: Fix containerd configuration for registry mirrors gardener#7316 / config merge via imports overwrites the wrong part of the config containerd/containerd#5837. That's why we cannot use the custom containerd config import functionality for now.
  • Node bootstrap problem: Fix the Node bootstrap problem #68
• Node machines cannot resolve the DNS names of the registry cache services yet. Currently, containerd config includes IP addresses of the registry cache pods/services. Find a way to make this DNS resolution work.
• Support container registries which require credentials: Add support for private image access #94
  • Add e2e/testmachinery tests: Add e2e for upstream registries that require credentials #184
  • Document the limitation why we have single credentials per pull through cache instance: Improve the upstream credentials docs #152
• Tests
  • Add unit tests: Extract the registry deployment into component. Add missing unit tests #36
  • Add testmachinery tests: Add testmachinery integration tests #38
  • Add e2e tests
    • Add e2e tests #30
    • Add e2e tests for registry-cache pull requests ci-infra#847
  • Fix make extension-down and rework skaffold handling to also deploy the registry-cache admission during make extension-up: Improve the skaffold setup of the extension. Fix make extension-down #40
• Add documentation
  • Add documentation for end users: Add docs for end users #44
  • Add documentation for developers (architectural design, internal workings): Add docs for developers #81
  • Add validation for immutability of the disk size. Currently the corresponding field in the StatefulSet is immutable. There is a dedicate task for the resize but we need to agree how we want to support it: Make the cache size field immutable #42
  • Add validation to prevent adding multiple registry-cache extensions with the same upstream: gardener-apiserver: Consider forbidding specifying more than 1 extension from the same type for a Shoot gardener#8446
  • Forbid caches with the same upstream: Do not allow caches with duplicate upstream #45
• Double-check what happens if disk size of registry is insufficient (consider activating garbage collection or similar solution)
  • The garbage collection working is described in the docs PR (Add docs for end users #44)
• Redesign the garbage collection API: Introduce v1alpha3 and make the garbage collection TTL configurable #144
  • Expose ttl and remove the existing option to enable/disable the garbage collection. Consider forbidding ttl < 24h as ttl=0 can mean that image never gets garbage collected (double check this).
  • Drop the existing garbageCollection.enabled field in favor of the garbageCollection.ttl field.
  • Change the garbageCollectionEnabled field to garbageCollection.enabled. This is to make possible adding new fields related to the garbage collection in future (the ttl field): Change the garbageCollectionEnabled field to garbageCollection.enabled #53
• Add monitoring (https://docs.docker.com/registry/configuration/)
  • (upstream contribution) Make available the registry-cache metrics in the /metrics endpoint that is prometheus native format
    • Fix proxy statistics distribution/distribution#4045
    • Add prometheus proxy related metrics distribution/distribution#4047, [cherry pick] Automated cherry pick of #4047 origin release 2.8 distribution/distribution#4156
    • (upstream contribution) Init upstream metrics with 0 vector: Initialize proxy prometheus counters values to 0 distribution/distribution#4283
  • Decide on which namespace the extension will be deployed - registry-cache or kube-system. The control plane monitoring currently does not monitor workload from the registry-cache namespace (i.e. we don't have metrics for Pod cpu/memory usage; logs are not collected): Use kube-system namespace to deploy registry cache extension #52
  • Add dashboard with:
    • registry cache panels: Add scrape config and monitoring dashboard #110
      initialize counters: Initialize proxy prometheus counters values to 0 distribution/distribution#4283
    • volume panels: Add monitoring dashboard for registry cache persistent volumes #112
  • Check for kubelet metrics to show a panel about image pull times: Add plutono dashboard for image pull duration gardener#9422
  • Aggregate the registry-cache metrics: Add recording rules that will be federated by aggregate prometheus and garden-prometheus #169
• Add alerting for the registry-cache volume size: Add alerts for registry caches' volumes #96
  • Scrape the required kubelet metrics in gardener/gardener: Scrape kubelet volume metrics gardener#8798
• Allow configuring containerd mirrors without deploying the caches to the shoot (add option to providerConfig and corresponding handling in controller): Support providing containerd registry mirror configuration #134
  • Update docs in gardener/gardener: Update containerd registry configuration docs gardener#9165
• Switch the registry-cache extension to do not use deprecated containerd configs
  • [prerequisite] Adapt the kind clusters in gardener/gardener to do not use the deprecated keys: Adapt kind cluster's registry configuration to the new way of configuring registry mirrors in containerd gardener#8047
  • [prerequisite] Adapt the Shoot cluster in provider-local to do not use the deprecated keys: Introduce the ContainerdRegistryHostsDir feature gate gardener#8094
• Check and if possible reduce the resources requests and limits for the prow job definition that builds the registry-cache container image (gardener-extension-registry-cache-test-builds and gardener-extension-registry-cache-build-images): Reduce the memory resource requests for the registry-cache builds ci-infra#795
• cri-config-ensurer and registry images are wrong: Fix cri-config-ensurer and registry images #23
• Deletion of a Shoot cluster with enabled registry-cache doesn't work. gardenlet in the Cleaning Kubernetes resources deletes the registry-cache resources which are recreated by GRM in few moments: Prevent gardenlet to delete registry-cache resources on Shoot deletion #25
• PVC and StorageClass creation is racy. It can be the case that the PVC is created first before the StorageClass. Then the PVC hangs in Pending state with reason no persistent volumes available for this claim and no storage class is set: Explicitly set the PVC's storageClassName to default #51
• Fix the --version command: Fix the --version flag #98
• Switch to the REUSE license format: Switch to use REUSE license format #146
• Support make extension-up in provider-extensions setup @dimitar-kostadinov Introduce remote-extension-up make target #193
• skaffold does not build and update on changes to the images.yaml file: skaffold: Rebuild when there are changes to go:embed files #57
• Allow reuse of the logic in gardener/gardener that checks for skaffold ko dependencies: Allow reuse of the functionality that checks for skaffold dependencies gardener#8630
• Go through the component-checklist in gardener/gardener and double-check every item: Align the extension according to Gardener's component checklist #85
  • Add readiness and liveness probe to the extension: Sync charts with latest changes #31
  • Add readiness and liveness probe to the registry Pods: Add readiness and liveness probe to the registry Pods #55
  • Drop the healthcheck controller as after Consider all MR in shoot namespace in seed for health check except one having class seed gardener#7462 gardenlet checks for all ManagedResources in the Shoot control plane. We don't need anymore the healthcheck controller: Drop the healthcheck controller #32
  • Add PriorityClass for the registry-cache Pod: Set priorityClassName for the registry cache Pods #47
  • Provide resource requirements: Define VPA managed resources requests for registry pull through caches #71
  • Define a VerticalPodAutoscaler: Define VPA managed resources requests for registry pull through caches #71
  • Define RBAC roles with minimal privileges: Align the extension according to Gardener's component checklist #85
  • Use NetworkPolicys to restrict network traffic: Align the extension according to Gardener's component checklist #85
  • Do not run components in privileged mode: Align the extension according to Gardener's component checklist #85
  • Choose the proper Seccomp profile: Align the extension according to Gardener's component checklist #85
• Review all registry config options before we move to production (ref https://distribution.github.io/distribution/about/configuration/):
  • proxy: Misleading warning for missing HTTP secret distribution/distribution#4304: proxy: Do not configure HTTP secret for proxy registry distribution/distribution#4305
• Check Pulls take a long time to complete when the repository mirror endpoint is timing out containerd/containerd#4861 and see whether with docker there are timeouts of 30 seconds
• Add support for /test pull-gardener-publish-test-images command for the registry-cache repo so that it is possible to publish images from PRs: Add publish-test-images job to the registry-cache extension ci-infra#970
• Beautify the imageVectorOverwrite so that it is possible to pass it as YAML, not as a single line string: Change the imageVectorOverwrite value type from string to object #80
• Make the storageClassName configurable: Introduce v1alpha2 and make the StorageClass name configurable in v1alpha2 #101
• Remove the deprecated v1alpha1 API version: Remove the v1alpha1 API #141
  • Adapt the e2e and testmachinery tests to use v1alpha2 API version only: Clean up v1alpha1 usages from tests #140
• Fix CPM for a Shoot with registry-cache extension enabled: Fix CPM for Shoot with registry-cache extension enabled #114
• Fix/mitigate proxy: TTLExpirationScheduler removes blobs that are still in use by other images distribution/distribution#4191: Mitigate distribution/distribution#4191 by disabling blobdescriptor cache #128
• Adjust the StatefulSet naming convention shorter to accommodate longer registry name: Adjust the statefulset naming convention shorter to accommodate longer registry name #120: Fix StatefulSet Pod creation for caches with a long upstream #129
• Think about mitigation for proxy: Switch from proxy.ttl=0 to proxy.ttl > 0 leaks blobs that never get garbage collected distribution/distribution#4249: Forbid GC enablement once it is disabled #131
• Check if we can speed up the configure-containerd-registries.service unit by using another retry strategy in the unit other than exponential backoff: Improve configure-containerd-registries unit retry strategy #137
• Upgrade distribution/distribution to v3.0.0-alpha.1: Update registry to 3.0.0-alpha.1 #138
  • Copy the DockerHub image to AR: Copy registry:3.0.0-alpha.1 ci-infra#1128
  • [v3.0.0-alpha.1] Disable exporting traces by default distribution/distribution#4270
• Drop vendoring from the registry-cache repo: Drop vendoring #147
• Fix e2e tests flakes: Rework image pull check in E2E tests #196
• Investigate how to reproduce and fix panic: unexpected end of JSON input distribution/distribution#2374: Stop proxy scheduler on system exit distribution/distribution#4293, Rework image pull check in E2E tests #196
  • Activate registry graceful shutdown: Activate registry graceful shutdown #162
• Remove the deprecated v1alpha2 API version: Remove the v1alpha2 API #165
  • Adapt the e2e and testmachinery tests to use v1alpha3 API version only: Clean up v1alpha2 usages #151
• Support upstreams with http:// and port (optional): Introduce remoteURL field in RegistryCache API #183
• Long upstreams will be rejected due to the label restriction of 63 chars: Limit upstream-host label value to 43 chars #186
• (upstream contribution) Open issue to the upstream to change the base image of the registry image to distroless: Registry image with distroless base distribution/distribution-library-image#168
  • (prerequisite) Rework the e2e tests to use the registry API instead of the scheduler-state.json file Rework image pull check in E2E tests #196
• Prepare a blog: Add "Gardener's Registry Cache Extension: Another Cost Saving Win and More" blog post documentation#495

---

• Evaluate dragonfly: https://github.com/dimitar-kostadinov/gardener-extension-registry-cache/tree/dragonfly-eval

[JensAc]

Seems to be a reasonable list. However, I am wondering whether we could just release a v0.0.1 before. Then we (23T) could easily include the extension in our public gardener installation and just see how the current state and further developments behave in a running environment. Any comments on that?

[JensAc]

Would you like to release the v0.0.1 @oliver-goetz? Actually, I am not entirely sure how what kind of release workflow is intended for this repo.

[oliver-goetz]

@JensAc sorry I missed your question 😅
There is no release workflow yet. I'll think about it.
However, we are already building dev images for each commit on main. You can find the images names + tags in the jobs log.

[rfranzke]

gardener/gardener#6999 is merged now which (I think) was the prerequisite to follow the "webhook approach" :)

[oliver-goetz]

@JensAc finally, there is the first release 😄

[JensAc]

Nice! Many thanks 👍

[timebertt]

Considering the points in gardener/gardener#7316, this extension should not make use of the containerd imports feature.
Otherwise it might overwrite important containerd configuration.

[dimitar-kostadinov]

All sub-tasks are completed.

/close

[gardener-prow]

@dimitar-kostadinov: Closing this issue.

In response to this:

All sub-tasks are completed.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.