kube-ovn is failing after enabling "interconnection to an existing IC database server". #11356

sanshah1211 · 2024-07-06T13:03:26Z

What happened?

Failing "TASK [kubernetes-apps/network_plugin/kube-ovn : Kube-OVN | Start Resources]" with error "err='error: error parsing /etc/kubernetes/cni-kube-ovn.yml: error converting YAML to JSON: yaml: line 15: could not find expected ':'\n'"}"

What did you expect to happen?

Execution should happen successfully.

How can we reproduce it (as minimally and precisely as possible)?

By enabling "kube_ovn_ic_enable" option in "inventory/k8s/group_vars/k8s_cluster/k8s-net-kube-ovn.yml" file.

OS

rocky 9

Version of Ansible

ansible [core 2.16.8]
  config file = /home/stack/kubespray/ansible.cfg
  configured module search path = ['/home/stack/kubespray/library']
  ansible python module location = /home/stack/.local/lib/python3.11/site-packages/ansible
  ansible collection location = /home/stack/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/stack/.local/bin/ansible
  python version = 3.11.5 (main, Sep  7 2023, 00:00:00) [GCC 11.4.1 20230605 (Red Hat 11.4.1-2)] (/usr/bin/python3.11)
  jinja version = 3.1.4
  libyaml = True

Version of Python

Python 3.9.18

Version of Kubespray (commit)

35e904d

Network plugin used

kube-ovn

Full inventory with variables

https://gist.github.com/sanshah1211/f779b995276163d16a884c5db6fa14a0

Command used to invoke ansible

ansible-playbook -i inventory/k8s/inventory.ini -b cluster.yml

Output of ansible run

PLAY [Install Calico Route Reflector] **********************************************************************************************************************************************************
skipping: no hosts matched

PLAY [Patch Kubernetes for Windows] ************************************************************************************************************************************************************
Saturday 06 July 2024  12:44:52 +0000 (0:00:05.909)       0:03:55.273 *********
Saturday 06 July 2024  12:44:54 +0000 (0:00:01.601)       0:03:56.875 *********
Saturday 06 July 2024  12:44:54 +0000 (0:00:00.248)       0:03:57.123 *********
Saturday 06 July 2024  12:44:54 +0000 (0:00:00.240)       0:03:57.364 *********
Saturday 06 July 2024  12:44:54 +0000 (0:00:00.244)       0:03:57.608 *********
Saturday 06 July 2024  12:44:55 +0000 (0:00:00.244)       0:03:57.853 *********

PLAY [Install Kubernetes apps] *****************************************************************************************************************************************************************
Saturday 06 July 2024  12:44:57 +0000 (0:00:02.402)       0:04:00.255 *********
Saturday 06 July 2024  12:44:58 +0000 (0:00:01.380)       0:04:01.636 *********
Saturday 06 July 2024  12:44:59 +0000 (0:00:00.256)       0:04:01.892 *********
Saturday 06 July 2024  12:45:00 +0000 (0:00:00.868)       0:04:02.761 *********
Saturday 06 July 2024  12:45:00 +0000 (0:00:00.270)       0:04:03.031 *********
Saturday 06 July 2024  12:45:00 +0000 (0:00:00.579)       0:04:03.611 *********
Saturday 06 July 2024  12:45:04 +0000 (0:00:03.783)       0:04:07.394 *********
Saturday 06 July 2024  12:45:05 +0000 (0:00:00.593)       0:04:07.988 *********
Saturday 06 July 2024  12:45:06 +0000 (0:00:01.235)       0:04:09.223 *********

TASK [kubernetes-apps/network_plugin/kube-ovn : Kube-OVN | Start Resources] ********************************************************************************************************************
ok: [k8s-master-1] => (item={'diff': {'before': {'path': '/etc/kubernetes/cni-kube-ovn-crd.yml'}, 'after': {'path': '/etc/kubernetes/cni-kube-ovn-crd.yml'}}, 'path': '/etc/kubernetes/cni-kube-ovn-crd.yml', 'changed': False, 'uid': 0, 'gid': 0, 'owner': 'root', 'group': 'root', 'mode': '0644', 'state': 'file', 'secontext': 'system_u:object_r:kubernetes_file_t:s0', 'size': 40828, 'invocation': {'module_args': {'mode': 420, 'dest': '/etc/kubernetes/cni-kube-ovn-crd.yml', '_original_basename': 'cni-kube-ovn-crd.yml.j2', 'recurse': False, 'state': 'file', 'path': '/etc/kubernetes/cni-kube-ovn-crd.yml', 'force': False, 'follow': True, 'modification_time_format': '%Y%m%d%H%M.%S', 'access_time_format': '%Y%m%d%H%M.%S', 'unsafe_writes': False, '_diff_peek': None, 'src': None, 'modification_time': None, 'access_time': None, 'owner': None, 'group': None, 'seuser': None, 'serole': None, 'selevel': None, 'setype': None, 'attributes': None}}, 'checksum': '3f3094b2ce5170cec991b7afe74dc5393fbdc209', 'dest': '/etc/kubernetes/cni-kube-ovn-crd.yml', 'failed': False, 'item': {'name': 'kube-ovn-crd', 'file': 'cni-kube-ovn-crd.yml'}, 'ansible_loop_var': 'item'})
ok: [k8s-master-1] => (item={'diff': {'before': {'path': '/etc/kubernetes/cni-ovn.yml'}, 'after': {'path': '/etc/kubernetes/cni-ovn.yml'}}, 'path': '/etc/kubernetes/cni-ovn.yml', 'changed': False, 'uid': 0, 'gid': 0, 'owner': 'root', 'group': 'root', 'mode': '0644', 'state': 'file', 'secontext': 'system_u:object_r:kubernetes_file_t:s0', 'size': 11449, 'invocation': {'module_args': {'mode': 420, 'dest': '/etc/kubernetes/cni-ovn.yml', '_original_basename': 'cni-ovn.yml.j2', 'recurse': False, 'state': 'file', 'path': '/etc/kubernetes/cni-ovn.yml', 'force': False, 'follow': True, 'modification_time_format': '%Y%m%d%H%M.%S', 'access_time_format': '%Y%m%d%H%M.%S', 'unsafe_writes': False, '_diff_peek': None, 'src': None, 'modification_time': None, 'access_time': None, 'owner': None, 'group': None, 'seuser': None, 'serole': None, 'selevel': None, 'setype': None, 'attributes': None}}, 'checksum': '6190ec4e8920abda3a0c6694da60655d89eb1a11', 'dest': '/etc/kubernetes/cni-ovn.yml', 'failed': False, 'item': {'name': 'ovn', 'file': 'cni-ovn.yml'}, 'ansible_loop_var': 'item'})
failed: [k8s-master-1] (item={'diff': [], 'dest': '/etc/kubernetes/cni-kube-ovn.yml', 'src': '/home/rocky/.ansible/tmp/ansible-tmp-1720269839.6702313-2232930-163883104078346/source', 'md5sum': '45b3d18abb5aa60fa2bd879e649d9534', 'checksum': 'a08b11137acc1f22a3bdb433198752102f9139c3', 'changed': True, 'uid': 0, 'gid': 0, 'owner': 'root', 'group': 'root', 'mode': '0644', 'state': 'file', 'secontext': 'system_u:object_r:kubernetes_file_t:s0', 'size': 18020, 'invocation': {'module_args': {'src': '/home/rocky/.ansible/tmp/ansible-tmp-1720269839.6702313-2232930-163883104078346/source', 'dest': '/etc/kubernetes/cni-kube-ovn.yml', 'mode': 420, 'follow': False, '_original_basename': 'cni-kube-ovn.yml.j2', 'checksum': 'a08b11137acc1f22a3bdb433198752102f9139c3', 'backup': False, 'force': True, 'unsafe_writes': False, 'content': None, 'validate': None, 'directory_mode': None, 'remote_src': None, 'local_follow': None, 'owner': None, 'group': None, 'seuser': None, 'serole': None, 'selevel': None, 'setype': None, 'attributes': None}}, 'failed': False, 'item': {'name': 'kube-ovn', 'file': 'cni-kube-ovn.yml'}, 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": true, "checksum": "a08b11137acc1f22a3bdb433198752102f9139c3", "dest": "/etc/kubernetes/cni-kube-ovn.yml", "diff": [], "failed": false, "gid": 0, "group": "root", "invocation": {"module_args": {"_original_basename": "cni-kube-ovn.yml.j2", "attributes": null, "backup": false, "checksum": "a08b11137acc1f22a3bdb433198752102f9139c3", "content": null, "dest": "/etc/kubernetes/cni-kube-ovn.yml", "directory_mode": null, "follow": false, "force": true, "group": null, "local_follow": null, "mode": 420, "owner": null, "remote_src": null, "selevel": null, "serole": null, "setype": null, "seuser": null, "src": "/home/rocky/.ansible/tmp/ansible-tmp-1720269839.6702313-2232930-163883104078346/source", "unsafe_writes": false, "validate": null}}, "item": {"file": "cni-kube-ovn.yml", "name": "kube-ovn"}, "md5sum": "45b3d18abb5aa60fa2bd879e649d9534", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:kubernetes_file_t:s0", "size": 18020, "src": "/home/rocky/.ansible/tmp/ansible-tmp-1720269839.6702313-2232930-163883104078346/source", "state": "file", "uid": 0}, "msg": "error running kubectl (/usr/local/bin/kubectl apply --force --filename=/etc/kubernetes/cni-kube-ovn.yml) command (rc=1), out='deployment.apps/kube-ovn-controller configured\ndaemonset.apps/kube-ovn-cni configured\ndaemonset.apps/kube-ovn-pinger unchanged\ndeployment.apps/kube-ovn-monitor unchanged\nservice/kube-ovn-monitor unchanged\nservice/kube-ovn-pinger unchanged\nservice/kube-ovn-controller unchanged\n', err='error: error parsing /etc/kubernetes/cni-kube-ovn.yml: error converting YAML to JSON: yaml: line 15: could not find expected ':'\n'"}

NO MORE HOSTS LEFT *****************************************************************************************************************************************************************************

PLAY RECAP *************************************************************************************************************************************************************************************
k8s-master-1               : ok=18   changed=4    unreachable=0    failed=1    skipped=113  rescued=0    ignored=0
k8s-master-2               : ok=11   changed=3    unreachable=0    failed=0    skipped=88   rescued=0    ignored=0
k8s-master-3               : ok=11   changed=3    unreachable=0    failed=0    skipped=88   rescued=0    ignored=0
k8s-worker-1               : ok=11   changed=3    unreachable=0    failed=0    skipped=80   rescued=0    ignored=0
k8s-worker-2               : ok=11   changed=3    unreachable=0    failed=0    skipped=77   rescued=0    ignored=0

Saturday 06 July 2024  12:45:11 +0000 (0:00:05.375)       0:04:14.599 *********
===============================================================================
kubernetes/preinstall : Create calico cni directories ---------------------------------------------------------------------------------------------------------------------------------- 31.51s
network_plugin/cni : CNI | Copy cni plugins --------------------------------------------------------------------------------------------------------------------------------------------- 8.84s
network_plugin/kube-ovn : Kube-OVN | Create Kube-OVN manifests -------------------------------------------------------------------------------------------------------------------------- 7.20s
kubespray-defaults : Set `etcd_deployment_type` to "kubeadm" if `etcd_kubeadm_enabled` is true ------------------------------------------------------------------------------------------ 7.15s
network_plugin/multus : Multus | Copy manifest templates -------------------------------------------------------------------------------------------------------------------------------- 6.98s
kubespray-defaults : Set `etcd_deployment_type` to "kubeadm" if `etcd_kubeadm_enabled` is true ------------------------------------------------------------------------------------------ 6.90s
Gather necessary facts (hardware) ------------------------------------------------------------------------------------------------------------------------------------------------------- 6.76s
kubespray-defaults : Set `etcd_deployment_type` to "kubeadm" if `etcd_kubeadm_enabled` is true ------------------------------------------------------------------------------------------ 6.32s
helm-apps : Validating arguments against arg spec 'main' - Install a list of Helm charts. ----------------------------------------------------------------------------------------------- 5.91s
network_plugin/multus : Multus | Copy manifest files ------------------------------------------------------------------------------------------------------------------------------------ 5.70s
kubernetes-apps/network_plugin/kube-ovn : Kube-OVN | Start Resources -------------------------------------------------------------------------------------------------------------------- 5.38s
etcd : Set etcd_client_cert_serial ------------------------------------------------------------------------------------------------------------------------------------------------------ 4.70s
Check that jinja is not too old (install via pip) --------------------------------------------------------------------------------------------------------------------------------------- 4.43s
kubespray-defaults : Set `etcd_deployment_type` to "kubeadm" if `etcd_kubeadm_enabled` is true ------------------------------------------------------------------------------------------ 4.42s
kubespray-defaults : Set `etcd_deployment_type` to "kubeadm" if `etcd_kubeadm_enabled` is true ------------------------------------------------------------------------------------------ 3.78s
network_plugin/multus : Multus | Check container engine type ---------------------------------------------------------------------------------------------------------------------------- 3.10s
network_plugin/kube-ovn : Kube-OVN | Label ovn-db node ---------------------------------------------------------------------------------------------------------------------------------- 2.87s
kubespray-defaults : Set `etcd_deployment_type` to "kubeadm" if `etcd_kubeadm_enabled` is true ------------------------------------------------------------------------------------------ 2.75s
kubernetes/preinstall : Create kubernetes directories ----------------------------------------------------------------------------------------------------------------------------------- 2.73s
kubespray-defaults : Set `etcd_deployment_type` to "kubeadm" if `etcd_kubeadm_enabled` is true ------------------------------------------------------------------------------------------ 2.65s

Anything else we need to know

After troubleshooting an error found that something is wrong with task execution.

Indentation should not be like this (Line 643)

---
kind: Service
apiVersion: v1
metadata:
  name: kube-ovn-cni
  namespace: kube-system
  labels:
    app: kube-ovn-cni
spec:
  selector:
    app: kube-ovn-cni
  ports:
    - port: 10665
      name: metrics
      ---
kind: ConfigMap
apiVersion: v1
metadata:
  name: ovn-ic-config
  namespace: kube-system
data:
  enable-ic: "true"
  az-name: "kubernetes"
  ic-db-host: "10.10.0.145"
  ic-nb-port: "6645"
  ic-sb-port: "6646"
  gw-nodes: "k8s-master-1,k8s-master-2,k8s-master-3"
  auto-route: "true"

Strange thing is this is only happening when I am enabling interconnection option, otherwise execution goes smoother.

The text was updated successfully, but these errors were encountered:

Indentation was wrong (L658) which caused kubernetes-sigs#11356, confirmed at my side after making it correct kubespray playbook executed successfully.

Made corrections in Indentation at L658 which causes kubespray execution failed with YAML to Json conversion. kubernetes-sigs#11356

bobz965 · 2024-07-07T02:54:27Z

about

failed: [k8s-master-1] (item={'diff': [], 'dest': '/etc/kubernetes/cni-kube-ovn.yml', 'src': '/home/rocky/.ansible/tmp/ansible-tmp-1720269839.6702313-2232930-163883104078346/source', 'md5sum': '45b3d18abb5aa60fa2bd879e649d9534', 'checksum': 'a08b11137acc1f22a3bdb433198752102f9139c3', 'changed': True, 'uid': 0, 'gid': 0, 'owner': 'root', 'group': 'root', 'mode': '0644', 'state': 'file', 'secontext': 'system_u:object_r:kubernetes_file_t:s0', 'size': 18020, 'invocation': {'module_args': {'src': '/home/rocky/.ansible/tmp/ansible-tmp-1720269839.6702313-2232930-163883104078346/source', 'dest': '/etc/kubernetes/cni-kube-ovn.yml', 'mode': 420, 'follow': False, '_original_basename': 'cni-kube-ovn.yml.j2', 'checksum': 'a08b11137acc1f22a3bdb433198752102f9139c3', 'backup': False, 'force': True, 'unsafe_writes': False, 'content': None, 'validate': None, 'directory_mode': None, 'remote_src': None, 'local_follow': None, 'owner': None, 'group': None, 'seuser': None, 'serole': None, 'selevel': None, 'setype': None, 'attributes': None}}, 'failed': False, 'item': {'name': 'kube-ovn', 'file': 'cni-kube-ovn.yml'}, 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": true, "checksum": "a08b11137acc1f22a3bdb433198752102f9139c3", "dest": "/etc/kubernetes/cni-kube-ovn.yml", "diff": [], "failed": false, "gid": 0, "group": "root", "invocation": {"module_args": {"_original_basename": "cni-kube-ovn.yml.j2", "attributes": null, "backup": false, "checksum": "a08b11137acc1f22a3bdb433198752102f9139c3", "content": null, "dest": "/etc/kubernetes/cni-kube-ovn.yml", "directory_mode": null, "follow": false, "force": true, "group": null, "local_follow": null, "mode": 420, "owner": null, "remote_src": null, "selevel": null, "serole": null, "setype": null, "seuser": null, "src": "/home/rocky/.ansible/tmp/ansible-tmp-1720269839.6702313-2232930-163883104078346/source", "unsafe_writes": false, "validate": null}}, "item": {"file": "cni-kube-ovn.yml", "name": "kube-ovn"}, "md5sum": "45b3d18abb5aa60fa2bd879e649d9534", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:kubernetes_file_t:s0", "size": 18020, "src": "/home/rocky/.ansible/tmp/ansible-tmp-1720269839.6702313-2232930-163883104078346/source", "state": "file", "uid": 0}, "msg": "error running kubectl (/usr/local/bin/kubectl apply --force --filename=/etc/kubernetes/cni-kube-ovn.yml) command (rc=1), out='deployment.apps/kube-ovn-controller configured\ndaemonset.apps/kube-ovn-cni configured\ndaemonset.apps/kube-ovn-pinger unchanged\ndeployment.apps/kube-ovn-monitor unchanged\nservice/kube-ovn-monitor unchanged\nservice/kube-ovn-pinger unchanged\nservice/kube-ovn-controller unchanged\n', err='error: error parsing /etc/kubernetes/cni-kube-ovn.yml: error converting YAML to JSON: yaml: line 15: could not find expected ':'\n'"}

bobz965 · 2024-07-07T02:56:06Z

error parsing /etc/kubernetes/cni-kube-ovn.yml: error converting YAML to JSON: yaml: line 15: could not find expected ':'\n'

please show /etc/kubernetes/cni-kube-ovn.yml

sanshah1211 · 2024-07-07T05:35:01Z

@bobz965 found error and manage to resolve it. It happened because of the indentation error in Template.
#11357

After making correction issue resolved.

mzaian · 2024-07-26T08:47:10Z

/assign @sanshah1211

Made corrections in Indentation at L658 which causes kubespray execution failed with YAML to Json conversion. #11356

sanshah1211 added the kind/bug Categorizes issue or PR as related to a bug. label Jul 6, 2024

sanshah1211 added a commit to sanshah1211/kubespray-1 that referenced this issue Jul 6, 2024

Update cni-kube-ovn.yml.j2

465e527

Indentation was wrong (L658) which caused kubernetes-sigs#11356, confirmed at my side after making it correct kubespray playbook executed successfully.

sanshah1211 added a commit to sanshah1211/kubespray-1 that referenced this issue Jul 6, 2024

Update cni-kube-ovn.yml.j2

d504b2a

Made corrections in Indentation at L658 which causes kubespray execution failed with YAML to Json conversion. kubernetes-sigs#11356

sanshah1211 mentioned this issue Jul 6, 2024

Update cni-kube-ovn.yml.j2 #11357

Merged

k8s-ci-robot assigned sanshah1211 Jul 26, 2024

k8s-ci-robot pushed a commit that referenced this issue Aug 1, 2024

Update cni-kube-ovn.yml.j2 (#11357)

54ac5a6

Made corrections in Indentation at L658 which causes kubespray execution failed with YAML to Json conversion. #11356

k8s-ci-robot closed this as completed in #11357 Aug 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kube-ovn is failing after enabling "interconnection to an existing IC database server". #11356

kube-ovn is failing after enabling "interconnection to an existing IC database server". #11356

sanshah1211 commented Jul 6, 2024

bobz965 commented Jul 7, 2024

bobz965 commented Jul 7, 2024

sanshah1211 commented Jul 7, 2024

mzaian commented Jul 26, 2024

kube-ovn is failing after enabling "interconnection to an existing IC database server". #11356

kube-ovn is failing after enabling "interconnection to an existing IC database server". #11356

Comments

sanshah1211 commented Jul 6, 2024

What happened?

What did you expect to happen?

How can we reproduce it (as minimally and precisely as possible)?

OS

Version of Ansible

Version of Python

Version of Kubespray (commit)

Network plugin used

Full inventory with variables

Command used to invoke ansible

Output of ansible run

Anything else we need to know

bobz965 commented Jul 7, 2024

bobz965 commented Jul 7, 2024

sanshah1211 commented Jul 7, 2024

mzaian commented Jul 26, 2024