Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Continue waiting when an exception is raised #408

Merged

Conversation

abikouo
Copy link
Contributor

@abikouo abikouo commented Mar 16, 2022

SUMMARY

When an exception is raised and the wait_timeout is not reached, we should continue waiting as this may occurs due to temporary issue on cluster

Fixes #407

ISSUE TYPE
  • Bugfix Pull Request
COMPONENT NAME
ADDITIONAL INFORMATION

@softwarefactory-project-zuul
Copy link

Build succeeded.

Copy link
Member

@gravesm gravesm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add a changelog fragment? Otherwise, LGTM

@abikouo
Copy link
Contributor Author

abikouo commented Mar 17, 2022

Could you add a changelog fragment? Otherwise, LGTM

done

@abikouo abikouo requested a review from gravesm March 17, 2022 11:41
@sugitk
Copy link

sugitk commented Mar 17, 2022

I am testing your PR on my side, but unfortunately the Error was not eliminated. This is the result on my testing environment (sorry not RHEL, but macOS). I tested by just rebooting the target OCP4 node.

TASK [Get clusterversion resource] *****************************************************************************************************************************************************
task path: /Users/sugimura/work/xxxxxxxx/playbook.yml:7
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: sugimura
<127.0.0.1> EXEC /bin/sh -c 'echo ~sugimura && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/sugimura/.ansible/tmp `"&& mkdir "` echo /Users/sugimura/.ansible/tmp/ansible-tmp-1647520940.7230232-60303-161261133254949 `" && echo ansible-tmp-1647520940.7230232-60303-161261133254949="` echo /Users/sugimura/.ansible/tmp/ansible-tmp-1647520940.7230232-60303-161261133254949 `" ) && sleep 0'
redirecting (type: modules) ansible.builtin.k8s_info to kubernetes.core.k8s_info
Loading collection cloud.common from /Users/sugimura/.ansible/collections/ansible_collections/cloud/common
Using module file /Users/sugimura/.ansible/collections/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py
<127.0.0.1> PUT /Users/sugimura/.ansible/tmp/ansible-local-60300mmlq7imj/tmp50w84m_x TO /Users/sugimura/.ansible/tmp/ansible-tmp-1647520940.7230232-60303-161261133254949/AnsiballZ_k8s_info.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /Users/sugimura/.ansible/tmp/ansible-tmp-1647520940.7230232-60303-161261133254949/ /Users/sugimura/.ansible/tmp/ansible-tmp-1647520940.7230232-60303-161261133254949/AnsiballZ_k8s_info.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/Users/sugimura/venv/2.12/bin/python3.9 /Users/sugimura/.ansible/tmp/ansible-tmp-1647520940.7230232-60303-161261133254949/AnsiballZ_k8s_info.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /Users/sugimura/.ansible/tmp/ansible-tmp-1647520940.7230232-60303-161261133254949/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
    conn.connect()
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/connection.py", line 358, in connect
    conn = self._new_conn()
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x107263910>: Failed to establish a new connection: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sugimura/.ansible/tmp/ansible-tmp-1647520940.7230232-60303-161261133254949/AnsiballZ_k8s_info.py", line 107, in <module>
    _ansiballz_main()
  File "/Users/sugimura/.ansible/tmp/ansible-tmp-1647520940.7230232-60303-161261133254949/AnsiballZ_k8s_info.py", line 99, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/Users/sugimura/.ansible/tmp/ansible-tmp-1647520940.7230232-60303-161261133254949/AnsiballZ_k8s_info.py", line 47, in invoke_module
    runpy.run_module(mod_name='ansible_collections.kubernetes.core.plugins.modules.k8s_info', init_globals=dict(_module_fqn='ansible_collections.kubernetes.core.plugins.modules.k8s_info', _modlib_path=modlib_path),
  File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 210, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/var/folders/nr/nvs29cdn3lv18czlnzl99kh80000gn/T/ansible_k8s_info_payload_n1643dd1/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py", line 202, in <module>
  File "/var/folders/nr/nvs29cdn3lv18czlnzl99kh80000gn/T/ansible_k8s_info_payload_n1643dd1/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py", line 198, in main
  File "/var/folders/nr/nvs29cdn3lv18czlnzl99kh80000gn/T/ansible_k8s_info_payload_n1643dd1/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py", line 156, in execute_module
  File "/var/folders/nr/nvs29cdn3lv18czlnzl99kh80000gn/T/ansible_k8s_info_payload_n1643dd1/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/common.py", line 262, in kubernetes_facts
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/kubernetes/dynamic/client.py", line 112, in get
    return self.request('get', path, **kwargs)
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/kubernetes/dynamic/client.py", line 55, in inner
    resp = func(self, *args, **kwargs)
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/kubernetes/dynamic/client.py", line 270, in request
    return self.client.call_api(
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/kubernetes/client/rest.py", line 240, in GET
    return self.request("GET", url,
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/kubernetes/client/rest.py", line 213, in request
    r = self.pool_manager.request(method, url,
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/request.py", line 74, in request
    return self.request_encode_url(
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/request.py", line 96, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/poolmanager.py", line 375, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/connectionpool.py", line 813, in urlopen
    return self.urlopen(
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/connectionpool.py", line 813, in urlopen
    return self.urlopen(
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/connectionpool.py", line 813, in urlopen
    return self.urlopen(
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.ocp.sugimura.home', port=6443): Max retries exceeded with url: /apis/config.openshift.io/v1/clusterversions/version?fieldSelector=&labelSelector= (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x107263910>: Failed to establish a new connection: [Errno 61] Connection refused'))

I suppose this block caused the above error.

try:
result = resource.get(
name=name,
namespace=namespace,
label_selector=",".join(label_selectors),
field_selector=",".join(field_selectors),
)
except BadRequestError:
return dict(resources=[], api_found=True)
except NotFoundError:
if not wait or name is None:
return dict(resources=[], api_found=True)

@softwarefactory-project-zuul
Copy link

Build failed.

@abikouo
Copy link
Contributor Author

abikouo commented Mar 17, 2022

I am testing your PR on my side, but unfortunately the Error was not eliminated. This is the result on my testing environment (sorry not RHEL, but macOS). I tested by just rebooting the target OCP4 node.

TASK [Get clusterversion resource] *****************************************************************************************************************************************************
task path: /Users/sugimura/work/xxxxxxxx/playbook.yml:7
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: sugimura
<127.0.0.1> EXEC /bin/sh -c 'echo ~sugimura && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/sugimura/.ansible/tmp `"&& mkdir "` echo /Users/sugimura/.ansible/tmp/ansible-tmp-1647520940.7230232-60303-161261133254949 `" && echo ansible-tmp-1647520940.7230232-60303-161261133254949="` echo /Users/sugimura/.ansible/tmp/ansible-tmp-1647520940.7230232-60303-161261133254949 `" ) && sleep 0'
redirecting (type: modules) ansible.builtin.k8s_info to kubernetes.core.k8s_info
Loading collection cloud.common from /Users/sugimura/.ansible/collections/ansible_collections/cloud/common
Using module file /Users/sugimura/.ansible/collections/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py
<127.0.0.1> PUT /Users/sugimura/.ansible/tmp/ansible-local-60300mmlq7imj/tmp50w84m_x TO /Users/sugimura/.ansible/tmp/ansible-tmp-1647520940.7230232-60303-161261133254949/AnsiballZ_k8s_info.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /Users/sugimura/.ansible/tmp/ansible-tmp-1647520940.7230232-60303-161261133254949/ /Users/sugimura/.ansible/tmp/ansible-tmp-1647520940.7230232-60303-161261133254949/AnsiballZ_k8s_info.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/Users/sugimura/venv/2.12/bin/python3.9 /Users/sugimura/.ansible/tmp/ansible-tmp-1647520940.7230232-60303-161261133254949/AnsiballZ_k8s_info.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /Users/sugimura/.ansible/tmp/ansible-tmp-1647520940.7230232-60303-161261133254949/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
    conn.connect()
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/connection.py", line 358, in connect
    conn = self._new_conn()
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x107263910>: Failed to establish a new connection: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sugimura/.ansible/tmp/ansible-tmp-1647520940.7230232-60303-161261133254949/AnsiballZ_k8s_info.py", line 107, in <module>
    _ansiballz_main()
  File "/Users/sugimura/.ansible/tmp/ansible-tmp-1647520940.7230232-60303-161261133254949/AnsiballZ_k8s_info.py", line 99, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/Users/sugimura/.ansible/tmp/ansible-tmp-1647520940.7230232-60303-161261133254949/AnsiballZ_k8s_info.py", line 47, in invoke_module
    runpy.run_module(mod_name='ansible_collections.kubernetes.core.plugins.modules.k8s_info', init_globals=dict(_module_fqn='ansible_collections.kubernetes.core.plugins.modules.k8s_info', _modlib_path=modlib_path),
  File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 210, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/var/folders/nr/nvs29cdn3lv18czlnzl99kh80000gn/T/ansible_k8s_info_payload_n1643dd1/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py", line 202, in <module>
  File "/var/folders/nr/nvs29cdn3lv18czlnzl99kh80000gn/T/ansible_k8s_info_payload_n1643dd1/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py", line 198, in main
  File "/var/folders/nr/nvs29cdn3lv18czlnzl99kh80000gn/T/ansible_k8s_info_payload_n1643dd1/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py", line 156, in execute_module
  File "/var/folders/nr/nvs29cdn3lv18czlnzl99kh80000gn/T/ansible_k8s_info_payload_n1643dd1/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/common.py", line 262, in kubernetes_facts
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/kubernetes/dynamic/client.py", line 112, in get
    return self.request('get', path, **kwargs)
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/kubernetes/dynamic/client.py", line 55, in inner
    resp = func(self, *args, **kwargs)
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/kubernetes/dynamic/client.py", line 270, in request
    return self.client.call_api(
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/kubernetes/client/rest.py", line 240, in GET
    return self.request("GET", url,
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/kubernetes/client/rest.py", line 213, in request
    r = self.pool_manager.request(method, url,
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/request.py", line 74, in request
    return self.request_encode_url(
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/request.py", line 96, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/poolmanager.py", line 375, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/connectionpool.py", line 813, in urlopen
    return self.urlopen(
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/connectionpool.py", line 813, in urlopen
    return self.urlopen(
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/connectionpool.py", line 813, in urlopen
    return self.urlopen(
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.ocp.sugimura.home', port=6443): Max retries exceeded with url: /apis/config.openshift.io/v1/clusterversions/version?fieldSelector=&labelSelector= (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x107263910>: Failed to establish a new connection: [Errno 61] Connection refused'))

I suppose this block caused the above error.

try:
result = resource.get(
name=name,
namespace=namespace,
label_selector=",".join(label_selectors),
field_selector=",".join(field_selectors),
)
except BadRequestError:
return dict(resources=[], api_found=True)
except NotFoundError:
if not wait or name is None:
return dict(resources=[], api_found=True)

Indeed this piece of code was also part of the fix but forgot to commit it.

I am testing your PR on my side, but unfortunately the Error was not eliminated. This is the result on my testing environment (sorry not RHEL, but macOS). I tested by just rebooting the target OCP4 node.

TASK [Get clusterversion resource] *****************************************************************************************************************************************************
task path: /Users/sugimura/work/xxxxxxxx/playbook.yml:7
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: sugimura
<127.0.0.1> EXEC /bin/sh -c 'echo ~sugimura && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/sugimura/.ansible/tmp `"&& mkdir "` echo /Users/sugimura/.ansible/tmp/ansible-tmp-1647520940.7230232-60303-161261133254949 `" && echo ansible-tmp-1647520940.7230232-60303-161261133254949="` echo /Users/sugimura/.ansible/tmp/ansible-tmp-1647520940.7230232-60303-161261133254949 `" ) && sleep 0'
redirecting (type: modules) ansible.builtin.k8s_info to kubernetes.core.k8s_info
Loading collection cloud.common from /Users/sugimura/.ansible/collections/ansible_collections/cloud/common
Using module file /Users/sugimura/.ansible/collections/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py
<127.0.0.1> PUT /Users/sugimura/.ansible/tmp/ansible-local-60300mmlq7imj/tmp50w84m_x TO /Users/sugimura/.ansible/tmp/ansible-tmp-1647520940.7230232-60303-161261133254949/AnsiballZ_k8s_info.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /Users/sugimura/.ansible/tmp/ansible-tmp-1647520940.7230232-60303-161261133254949/ /Users/sugimura/.ansible/tmp/ansible-tmp-1647520940.7230232-60303-161261133254949/AnsiballZ_k8s_info.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/Users/sugimura/venv/2.12/bin/python3.9 /Users/sugimura/.ansible/tmp/ansible-tmp-1647520940.7230232-60303-161261133254949/AnsiballZ_k8s_info.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /Users/sugimura/.ansible/tmp/ansible-tmp-1647520940.7230232-60303-161261133254949/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
    conn.connect()
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/connection.py", line 358, in connect
    conn = self._new_conn()
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x107263910>: Failed to establish a new connection: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sugimura/.ansible/tmp/ansible-tmp-1647520940.7230232-60303-161261133254949/AnsiballZ_k8s_info.py", line 107, in <module>
    _ansiballz_main()
  File "/Users/sugimura/.ansible/tmp/ansible-tmp-1647520940.7230232-60303-161261133254949/AnsiballZ_k8s_info.py", line 99, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/Users/sugimura/.ansible/tmp/ansible-tmp-1647520940.7230232-60303-161261133254949/AnsiballZ_k8s_info.py", line 47, in invoke_module
    runpy.run_module(mod_name='ansible_collections.kubernetes.core.plugins.modules.k8s_info', init_globals=dict(_module_fqn='ansible_collections.kubernetes.core.plugins.modules.k8s_info', _modlib_path=modlib_path),
  File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 210, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/var/folders/nr/nvs29cdn3lv18czlnzl99kh80000gn/T/ansible_k8s_info_payload_n1643dd1/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py", line 202, in <module>
  File "/var/folders/nr/nvs29cdn3lv18czlnzl99kh80000gn/T/ansible_k8s_info_payload_n1643dd1/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py", line 198, in main
  File "/var/folders/nr/nvs29cdn3lv18czlnzl99kh80000gn/T/ansible_k8s_info_payload_n1643dd1/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py", line 156, in execute_module
  File "/var/folders/nr/nvs29cdn3lv18czlnzl99kh80000gn/T/ansible_k8s_info_payload_n1643dd1/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/common.py", line 262, in kubernetes_facts
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/kubernetes/dynamic/client.py", line 112, in get
    return self.request('get', path, **kwargs)
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/kubernetes/dynamic/client.py", line 55, in inner
    resp = func(self, *args, **kwargs)
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/kubernetes/dynamic/client.py", line 270, in request
    return self.client.call_api(
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/kubernetes/client/rest.py", line 240, in GET
    return self.request("GET", url,
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/kubernetes/client/rest.py", line 213, in request
    r = self.pool_manager.request(method, url,
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/request.py", line 74, in request
    return self.request_encode_url(
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/request.py", line 96, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/poolmanager.py", line 375, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/connectionpool.py", line 813, in urlopen
    return self.urlopen(
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/connectionpool.py", line 813, in urlopen
    return self.urlopen(
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/connectionpool.py", line 813, in urlopen
    return self.urlopen(
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/Users/sugimura/venv/2.12/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.ocp.sugimura.home', port=6443): Max retries exceeded with url: /apis/config.openshift.io/v1/clusterversions/version?fieldSelector=&labelSelector= (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x107263910>: Failed to establish a new connection: [Errno 61] Connection refused'))

I suppose this block caused the above error.

try:
result = resource.get(
name=name,
namespace=namespace,
label_selector=",".join(label_selectors),
field_selector=",".join(field_selectors),
)
except BadRequestError:
return dict(resources=[], api_found=True)
except NotFoundError:
if not wait or name is None:
return dict(resources=[], api_found=True)

@sugitk thanks for the feedback
The backtrace on the issue was confusing, seems not aligned with the playbook, I tried to fix now the impacted function, could you please give another try ? Just to make sure you are testing with the right version, your backtrace above is showing common.py, line 262, in kubernetes_facts, but this is not good, could you please ensure that you are using my branch for the test ?

@sugitk
Copy link

sugitk commented Mar 17, 2022

@abikouo Thank you very much for your quick review, I will try your branch tomorrow. (now it's 11pm here..)

@sugitk
Copy link

sugitk commented Mar 17, 2022

@abikouo Good morning. I have tested with your branch (ex_wait), it worked as expected!
Thank you very much again for your prompt implementation 😊

@abikouo
Copy link
Contributor Author

abikouo commented Mar 18, 2022

@abikouo Good morning. I have tested with your branch (ex_wait), it worked as expected! Thank you very much again for your prompt implementation blush

Thanks for the feedback!!

@abikouo
Copy link
Contributor Author

abikouo commented Mar 18, 2022

recheck

@softwarefactory-project-zuul
Copy link

Build succeeded.

@sugitk
Copy link

sugitk commented Mar 23, 2022

@gravesm Do you have time to review this PR to merge? The customer is waiting for the fix..

@abikouo
Copy link
Contributor Author

abikouo commented Mar 23, 2022

@Akasurde could you please add your review here ? Thanks

@abikouo abikouo requested a review from Akasurde March 23, 2022 09:50
@abikouo abikouo requested a review from Akasurde March 23, 2022 09:55
@softwarefactory-project-zuul
Copy link

Build failed.

@abikouo
Copy link
Contributor Author

abikouo commented Mar 23, 2022

recheck

@softwarefactory-project-zuul
Copy link

Build succeeded.

@abikouo abikouo removed the request for review from Akasurde March 23, 2022 13:09
@softwarefactory-project-zuul
Copy link

Build succeeded (gate pipeline).

@softwarefactory-project-zuul softwarefactory-project-zuul bot merged commit f418353 into ansible-collections:main Mar 23, 2022
gravesm pushed a commit to gravesm/kubernetes.core that referenced this pull request Apr 28, 2022
Continue waiting when an exception is raised

SUMMARY
When an exception is raised and the wait_timeout is not reached, we should continue waiting as this may occurs due to temporary issue on cluster

Fixes ansible-collections#407

ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME

ADDITIONAL INFORMATION

Reviewed-by: Mike Graves <mgraves@redhat.com>
Reviewed-by: Abhijeet Kasurde <None>
(cherry picked from commit f418353)
gravesm pushed a commit to gravesm/kubernetes.core that referenced this pull request May 2, 2022
Continue waiting when an exception is raised

SUMMARY
When an exception is raised and the wait_timeout is not reached, we should continue waiting as this may occurs due to temporary issue on cluster

Fixes ansible-collections#407

ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME

ADDITIONAL INFORMATION

Reviewed-by: Mike Graves <mgraves@redhat.com>
Reviewed-by: Abhijeet Kasurde <None>
(cherry picked from commit f418353)
softwarefactory-project-zuul bot pushed a commit that referenced this pull request May 2, 2022
)

[backport/2.3] continue waiting when an exception is raised (#408)

Depends-On: #446
Continue waiting when an exception is raised
SUMMARY
When an exception is raised and the wait_timeout is not reached, we should continue waiting as this may occurs due to temporary issue on cluster
Fixes #407
ISSUE TYPE
Bugfix Pull Request
COMPONENT NAME
ADDITIONAL INFORMATION
Reviewed-by: Mike Graves mgraves@redhat.com
Reviewed-by: Abhijeet Kasurde 
(cherry picked from commit f418353)
gravesm added a commit to gravesm/community.okd that referenced this pull request May 4, 2022
Previous to
ansible-collections/kubernetes.core#408 an
attempt to list resources for which the user was not authorized resulted
in an unhandled exception. The task now does not fail, but instead
reports the error message.
gravesm added a commit to openshift/community.okd that referenced this pull request May 5, 2022
* Fix integration tests for k8s 2.3

Previous to
ansible-collections/kubernetes.core#408 an
attempt to list resources for which the user was not authorized resulted
in an unhandled exception. The task now does not fail, but instead
reports the error message.

* try to fix sanity

* Fix tests

* fix tests

* fix tests

* Revert change to downstream build
gravesm added a commit to gravesm/kubernetes.core that referenced this pull request Jun 8, 2022
Port changes from commit f418353 to the refactored code.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

k8s_info causes an unavoidable error when the API cannot accept the connection
4 participants