Format of containerd_insecure_registries returning key error after upgrade to v1.21.0

[supertetelman]

I have a repo that does a full deployment of Kubernetes with a few various add-ons that has been running great for some time. I just went to bump to the Kubespray version we are consuming from v 1.19.0 to v1.21.0. It appears that either the format expected of containerd_insecure_registries has changed or there is a bug that has been introduced for this variable. This deployment was working fine with the group_vars as-is for the previous version, but when I try to deploy with the latest code it is unable parse the variable.

I looked through the example configuraiton provided here and it does not appear as if the format there has changed https://github.com/kubernetes-sigs/kubespray/blob/master/inventory/sample/group_vars/all/containerd.yml.

Any help on what I might have wrong with my configuration and what this variable is now expected to look like?

Version Info:

$ansible --version
ansible [core 2.13.7]
  config file = None
  configured module search path = ['/home/versa/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt//env/lib/python3.8/site-packages/ansible
  ansible collection location = /home/versa/.ansible/collections:/usr/share/ansible/collections
  executable location = /opt/env/bin/ansible
  python version = 3.8.10 (default, Nov 14 2022, 12:59:47) [GCC 9.4.0]
  jinja version = 3.1.2
  libyaml = True

$python --version
Python 3.8.10

$git branch
  master
* release-2.21
$git log -n 1
commit 813576efeb6e32e09b1d11fe9ed7be880dd7df79 (HEAD -> release-2.21, origin/release-2.21)
Author: Tristan <tde@hey.com>
Date:   Wed Jan 25 00:02:12 2023 +0000

Relevant configuration in group_vars:

containerd_insecure_registries:
  "registry.local:31500": "http://registry.local:31500"

I also tried it with an alternate format as shown in this code example with the same error https://github.com/kubernetes-sigs/kubespray/blob/34d0451585eab0dc2c0ba99f02341fe2d25ca300/roles/kubespray-defaults/defaults/main.yaml:

containerd_insecure_registries:
  - registry.local:31500
  - http://registry.local:31500

Error Message:

    - name: containerd ｜ Create registry directories
      ^ here
fatal: [virtual-01]: FAILED! => 
  msg: |-
    The task includes an option with an undefined variable. The error was: 'ansible.utils.unsafe_proxy.AnsibleUnsafeText object' has no attribute 'key'
  
    The error appears to be in '/var/lib/jenkins/workspace/PR-123/submodules/kubespray/roles/container-engine/containerd/tasks/main.yml': line 114, column 3, but may
    be elsewhere in the file depending on the exact syntax problem.
  
    The offending line appears to be:
  
  
    - name: containerd ｜ Create registry directories
      ^ here

Example config:

## Port number is also needed if the default HTTPS port is not used.
# containerd_insecure_registries:
#   "localhost": "http://127.0.0.1"
#   "172.19.16.11:5000": "http://172.19.16.11:5000"

Alternate example config:

## Port number is also needed if the default HTTPS port is not used.
# containerd_insecure_registries:
#   - mirror.registry.io
#   - 172.19.16.11:5000

Failing code block (https://github.com/kubernetes-sigs/kubespray/blob/8a03bb1bb4f6768c84a76571a75d9066b1087c1d/roles/container-engine/containerd/tasks/main.yml):

- name: containerd ｜ Create registry directories
  file:
    path: "{{ containerd_cfg_dir }}/certs.d/{{ item.key }}"
    state: directory
    mode: 0755
    recurse: true
  with_items: "{{ containerd_insecure_registries }}"
  when: containerd_insecure_registries is defined

[floryut]

Did you try this format ?

containerd_insecure_registries:
  docker.io:
    - dockerhubcache.example.com

[bmartron]

Hello,
i try
containerd_insecure_registries:
192.168.22.54:
- http://192.168.22.54
In inventory/bmacluster/group_vars/all/containerd.yml
and this is the same

[teriyakichild]

here is the solution:

diff --git a/roles/container-engine/containerd/tasks/main.yml b/roles/container-engine/containerd/tasks/main.yml
index ecea9dd75..c057a08e2 100644
--- a/roles/container-engine/containerd/tasks/main.yml
+++ b/roles/container-engine/containerd/tasks/main.yml
@@ -117,7 +117,7 @@
     state: directory
     mode: 0755
     recurse: true
-  with_items: "{{ containerd_insecure_registries }}"
+  with_dict: "{{ containerd_insecure_registries }}"
   when: containerd_insecure_registries is defined
 
 - name: containerd ｜ Write hosts.toml file
@@ -131,7 +131,7 @@
       [host."{{ item.value }}"]
         capabilities = ["pull", "resolve", "push"]
         skip_verify = true
-  with_items: "{{ containerd_insecure_registries }}"
+  with_dict: "{{ containerd_insecure_registries }}"
   when: containerd_insecure_registries is defined
 
 # you can sometimes end up in a state where everything is installed

[bmartron]

Hello,
Thanks for the tips, work for me
Changing with_items by with_dict in roles/container-engine/containerd/tasks/main.yml
This is solve the Ansible issue we have also an issue witch the new configuration file in cert.d in containerd

[bmartron]

Hello,
To solve the issue i also have to remove mirror line in /etc/containerd/config.toml
To remove mirror

version = 2
root = "/var/lib/containerd"
state = "/run/containerd"
oom_score = 0

[grpc]
max_recv_message_size = 16777216
max_send_message_size = 16777216

[debug]
level = "info"

[metrics]
address = ""
grpc_histogram = false

[plugins]
[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "registry.k8s.io/pause:3.8"
max_container_log_line_size = -1
enable_unprivileged_ports = false
enable_unprivileged_icmp = false
[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "runc"
snapshotter = "overlayfs"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"
runtime_engine = ""
runtime_root = ""
base_runtime_spec = "/etc/containerd/cri-base.json"

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
        systemdCgroup = true
[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/etc/containerd/certs.d"

~
With this config the ansible task is ok and pulling from unsecure registry is ok

[supertetelman]

So it looks like this is in fact a bug in the main.yml and not a misunderstanding on intended variable format. @teriyakichild, will you be submitting your fix as a PR?

[teriyakichild]

here is the PR with the fix: #9729

[oomichi]

here is the PR with the fix: #9729

Thanks for pointing this out.
We can close this issue with #9729

/close

[k8s-ci-robot]

@oomichi: Closing this issue.

In response to this:

here is the PR with the fix: #9729

Thanks for pointing this out.
We can close this issue with #9729

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.