Cannot do Inventory Sync due to binascii.Error: Incorrect padding

[Syed-ghaffar93]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.

Bug Summary

We were running 21.10.2 , we had to update our helm chart yesterday to start using 21.11.0 , we started getting this error

Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/jobs.py", line 593, in run
    res = receptor_job.run()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/receptor.py", line 319, in run
    res = self._run_internal(receptor_ctl)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/receptor.py", line 394, in _run_internal
    res = processor_future.result()
  File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 446, in result
    return self.__get_result()
  File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/usr/lib64/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/utils/common.py", line 1214, in wrapper_cleanup_new_process
    return func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/receptor.py", line 462, in processor
    return ansible_runner.interface.run(
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/ansible_runner/interface.py", line 211, in run
    r.run()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/ansible_runner/streaming.py", line 254, in run
    self.artifacts_callback(data)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/ansible_runner/streaming.py", line 233, in artifacts_callback
    unstream_dir(self._input, length, self.artifact_dir)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/ansible_runner/utils/streaming.py", line 71, in unstream_dir
    data = source.read(chunk_size)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/ansible_runner/utils/base64io.py", line 282, in read
    results.write(base64.b64decode(data))
  File "/usr/lib64/python3.9/base64.py", line 87, in b64decode
    return binascii.a2b_base64(s)
binascii.Error: Incorrect padding
  config file = None
  configured module search path = ['/home/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.8/site-packages/ansible
  executable location = /usr/local/bin/ansible-inventory
  python version = 3.8.13 (default, Jun 24 2022, 15:27:57) [GCC 8.5.0 20210514 (Red Hat 8.5.0-13)]
No config file found; using defaults
setting up inventory plugins`

all the host file seems to be same like yesterday , it just looks like it stopped working all of the sudden making it fail everytime with inventory Sync

AWX version

21.11.0

Select the relevant components

• UI
• API
• Docs
• Collection
• CLI
• Other

Installation method

kubernetes

Modifications

no

Ansible version

2.10.0

Operating system

Linux

Web browser

Chrome

Steps to reproduce

Well it's happening with all of our playbooks which were running for last couple of years. It started from yesterday after i pulled the latest helm chart of aws operator

Expected results

N/A

Actual results

posted the error output in above message

Additional information

No response

[TheRealHaoLiu]

potentially a dup of #13469 and ansible/awx-operator#1203

in recent receptor changes we leverage a fix that was delivered in k8s to allow receptor to reconnect if disconnected

it require specific kubelet version and we decided (incorrectly) to use k8s apiserver version to detect if the fix exist

in case when kubelet version and k8s apiserver are not the same (and we recently found bug in version comparison logic too)

please add the following to awx resource

  ee_extra_env: |
    - name: RECEPTOR_KUBE_SUPPORT_RECONNECT
      value: disabled

[Syed-ghaffar93]

Thanks @TheRealHaoLiu, It worked :)

[bandwiches]

Just wanted to stop by and say this persisted on AWX 21.12.0 and the fix still works. Thanks!