Error: Credentials cache I/O operation failed - on remote login to multiple windows hosts using same credentials

[lastlad]

Hello - We have a requirement where we want to login to multiple (4+) windows hosts to deploy our application code. We are using Rundeck PyWinRM plugin to do the same. Everything works well when we try to log onto a single node or two nodes. When we increase the nodes to more than 3 to run our deploy script, we are running into what we believe is a concurrency issue. We are able to easily reproduce this error even with a simple script that just tries to login and do a pwd. It happens with both NodeFirst strategy as well as Parallel execution strategy.

Here's an excerpt from the error that's thrown on execution.

[ERROR  ]  generate_request_header(): authGSSClientStep() failed: (kerberos_.py:258)[winrm.vendor.requests_kerberos.kerberos_]
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/winrm/vendor/requests_kerberos/kerberos_.py", line 246, in generate_request_header
    negotiate_resp_value)
GSSError: (('Invalid credential was supplied', 655360), ('Credentials cache I/O operation failed', -1765328191))
[ERROR  ]  (('Invalid credential was supplied', 655360), ('Credentials cache I/O operation failed', -1765328191)) (kerberos_.py:259)[winrm.vendor.requests_kerberos.kerberos_]
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/winrm/vendor/requests_kerberos/kerberos_.py", line 246, in generate_request_header
    negotiate_resp_value)
GSSError: (('Invalid credential was supplied', 655360), ('Credentials cache I/O operation failed', -1765328191))
[ERROR  ]  Execution finished with the following error (winrm-exec.py:303)[root]
[ERROR  ]  authGSSClientStep() failed: (('Invalid credential was supplied', 655360), ('Credentials cache I/O operation failed', -1765328191)) (winrm-exec.py:304)[root]

Any help is appreciated.

Thanks!

[eagle-rr]

We are seeing similar issue. In running Rundeck 4.7 (Community version) on Oracle Linux 7; we are seeing where the python winrm plugin is returning the following errors when connecting to windows targets:

[ERROR ] generate_request_header(): authGSSClientStep() failed: (kerberos_.py:258)[winrm.vendor.requests_kerberos.kerberos_]
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/winrm/vendor/requests_kerberos/kerberos_.py", line 246, in generate_request_header
negotiate_resp_value)
kerberos.GSSError: (('Invalid credential was supplied', 655360), ('Bad format in credentials cache (filename: /tmp/krb5cc_40000)', -1765328185))
[ERROR ] (('Invalid credential was supplied', 655360), ('Bad format in credentials cache (filename: /tmp/krb5cc_40000)', -1765328185)) (kerberos_.py:259)[winrm.vendor.requests_kerberos.kerberos_]
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/winrm/vendor/requests_kerberos/kerberos_.py", line 246, in generate_request_header
negotiate_resp_value)
kerberos.GSSError: (('Invalid credential was supplied', 655360), ('Bad format in credentials cache (filename: /tmp/krb5cc_40000)', -1765328185))
[ERROR ] Execution finished with the following exit code: -1 (winrm-exec.py:339)[root]
[ERROR ] -1 (winrm-exec.py:340)[root]
[ERROR ] authGSSClientStep() failed: (('Invalid credential was supplied', 655360), ('Bad format in credentials cache (filename: /tmp/krb5cc_40000)', -1765328185)) (winrm-exec.py:341)[root]
Failed: NonZeroResultCode: [WinRMPython] Result code: 255

We have tried using the latest python winrm plugin (py-winrm-plugin-2.1.2); but the above error still happens randomly. For example of we run on 20+ windows servers; target server 10 and 15 might fail. Then if we re-run the same job; target server 8 and 12 might fail.

Our pip3 installation list contains:

certifi (2020.12.5)
cffi (1.14.4)
chardet (4.0.0)
circuitbreaker (1.3.1)
cryptography (2.9.2)
deepdiff (5.7.0)
docutils (0.14)
idna (2.10)
Jinja2 (2.11.1)
lockfile (0.11.0)
MarkupSafe (0.23)
netaddr (0.7.18)
ntlm-auth (1.5.0)
oci (2.93.1)
oci-utils (0.14.0)
ordered-set (4.0.2)
pathspec (0.6.0)
pexpect (4.8.0)
pip (9.0.3)
ply (3.9)
ptyprocess (0.7.0)
pycparser (2.20)
pykerberos (1.2.1)
PySocks (1.6.8)
python-daemon (2.1.2)
pytz (2017.2)
pywinrm (0.4.1)
PyYAML (5.4.1)
requests (2.25.1)
requests-kerberos (0.12.0)
requests-ntlm (1.1.0)
sdnotify (0.3.2)
selenium (3.9.0)
setuptools (39.2.0)
six (1.15.0)
urllib3 (1.26.2)
xmltodict (0.12.0)
yamllint (1.26.0)

[kevinpemonon]

Hello,

We're experiencing a similar problem with our rundeck jobs.

Rundeck is installed in version 5.4.0 on a Debian 10 server.

We use the py-winrm-plugin-2.1.3 python module to target machines running Windows.

When executing a job, we randomly get this execution error. When we rerun the job on the same host, sometimes the job works, sometimes it doesn't, and we don't know why.

This is the error we get:

kerberos.GSSError: (('Invalid credential was supplied', 655360), ('No credentials cache found (filename: /tmp/krb5cc_112)

Do you have any idea of the cause of the error and how to resolve it?

Thank you for your help,

Kevin Pemonon