Failed to deploy with containerd because could not pull image from an insecure image registry

[1eedaegon]

here is the config about insecure registries for containerd.

containerd_insecure_registries:
  "192.168.x.x:8080":
    - "http://192.168.x.x:8080"

the rendered /etc/containerd/config.toml contains:

[plugins."io.containerd.grpc.v1.cri".registry.configs."http://192.168.x.x:8080".tls]
  insecure_skip_verify = true

but it needs to be:

[plugins."io.containerd.grpc.v1.cri".registry.configs."192.168.x.x:8080".tls]
  insecure_skip_verify = true

because insecure_registries template has registry address instead of registry name

diff --git a/roles/container-engine/containerd/templates/config.toml.j2 b/roles/container-engine/containerd/templates/config.toml.j2
index c1bda12b..096dea40 100644
--- a/roles/container-engine/containerd/templates/config.toml.j2
+++ b/roles/container-engine/containerd/templates/config.toml.j2
@@ -57,8 +57,8 @@ oom_score = {{ containerd_oom_score }}
         [plugins."io.containerd.grpc.v1.cri".registry.mirrors."{{ registry }}"]
           endpoint = ["{{ ([ addr ] | flatten ) | join('","') }}"]
 {% endfor %}
-{% for addr in containerd_insecure_registries.values() | flatten | unique %}
-        [plugins."io.containerd.grpc.v1.cri".registry.configs."{{ addr }}".tls]
+{% for registry in containerd_insecure_registries.keys() %}
+        [plugins."io.containerd.grpc.v1.cri".registry.configs."{{ registry }}".tls]
           insecure_skip_verify = true
 {% endfor %}
 {% endif %}

recently issues: #9371 #9207

Environment:

• Cloud provider or hardware configuration:
  bare-metal

• OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"):
  Linux 4.18.0-372.9.1.el8.x86_64 x86_64
  Red Hat Enterprise Linux 8.6 (Ootpa)

• Version of Ansible (ansible --version):
  ansible [core 2.12.5]

• Version of Python (python --version):
  Python 3.9.13

Kubespray version (commit) (git rev-parse --short HEAD):
release-2.20

Network plugin used:
Calico CNI

Full inventory with variables (ansible -i inventory/sample/inventory.ini all -m debug -a "var=hostvars[inventory_hostname]"):

Command used to invoke ansible:

CLUSTER_NAME=sample
ansible-playbook -i inventory/"$CLUSTER_NAME"/hosts.yml cluster.yml --become

Output of ansible run:

Anything else do we need to know:

[1eedaegon]

hmm...
I have changed from containerd_insecure_registries.values() to containerd_insecure_registries.keys()
in roles/container-engine/containerd/templates/config.toml.j2 and then installed manually from offline nexus registry
Now the image pulling on containerd looks good as expected and everything is working fine.

Tested: RHEL8.6, Ubuntu20.04
PR: #9651

[1eedaegon]

fixed: #9729, close