Update rendering containerd config.toml to support insecure registries

[pbchekin]

kubespray 2.19.0
containerd github.com/containerd/containerd v1.6.4 212e8b6fa2f44b9c21b2798135fc6fb7c53efc16

Currently Kubespray allows specifying insecure registries for containerd with parameter containerd_insecure_registries, for example:

    containerd_insecure_registries:
      docker.io:
        - dockerhubcache.example.com

The rendered /etc/containerd/config.toml contains:

    [plugins."io.containerd.grpc.v1.cri".registry]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
          endpoint = ["dockerhubcache.example.com"]
        [plugins."io.containerd.grpc.v1.cri".registry.configs."docker.io".tls]
          insecure_skip_verify = true

Containerd reports the following error:

level=info msg="PullImage \"docker.io/library/busybox:1.35.0\""
level=info msg="trying next host" error="failed to do request: Head \"https://dockerhubcache.example.com/v2/library/busybox/manifests/1.35.0?ns=docker.io\": x509: certificate signed by unknown authority" host=dockerhubcache.example.com

Per https://github.com/containerd/containerd/blob/main/docs/cri/registry.md this configuration is deprecated and per https://github.com/containerd/containerd/blob/2b661b890f689716b2928408e386e656d8dd7118/pkg/cri/config/config.go#L212 it will be removed in containerd 1.7. Looks like insecure_skip_verify is already ignored in containerd 1.6.4.

The updated configuration should look like:

/etc/containerd/config.toml

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = "/etc/containerd/certs.d"

/etc/containerd/certs.d/docker.io/hosts.toml

server = "https://registry-1.docker.io"

[host."https://dockerhubcache.example.com"]
  capabilities = ["pull", "resolve", "push"]
  skip_verify = true

[pbchekin]

I took another look to the issue and found that the existing configuration can work with containerd 1.6.4, however there is a bug with rendering registry.configs key: instead of using a registry name

        # incorrect
        [plugins."io.containerd.grpc.v1.cri".registry.configs."docker.io".tls]
          insecure_skip_verify = true

the mirror endpoint should be used:

        # correct
        [plugins."io.containerd.grpc.v1.cri".registry.configs."dockerhubcache.example.com".tls]
          insecure_skip_verify = true

I am tagging this issue as "bug". UPD: looks like I cannot edit labels for the issue, please consider converting this to a bug on your side or let me know if I need to file another issue as a bug.

[VincentDu2021]

I tried this settings with 1.6.4, it does not work, the regsitry.mirror endpoint is skipped.
only way for nerdctl to use registry.mirror is the 1.7 way by setting /etc/containerd/certs.d/docker.io/hosts.toml

[pbchekin]

I tried this settings with 1.6.4, it does not work, the regsitry.mirror endpoint is skipped. only way for nerdctl to use registry.mirror is the 1.7 way by setting /etc/containerd/certs.d/docker.io/hosts.toml

@VincentDu2021, works for me for pre-1.7 containerd. Note that nerdctl does not use config.toml from containerd, as far as i understand. This is my my Kubespray configuration snippet that works with insecure registry:

# Required by Kubespray to download images during Kubernetes deployment
docker_image_repo: dockerhubcache.example.com
nerdctl_extra_flags: " --insecure-registry"

# Required by Kubernetes to download images
containerd_insecure_registries:
  docker.io:
    - dockerhubcache.example.com
containerd_registries: {}

What is your configuration that does not work?