.dbtoken can not be updated on OSG computing node

[dachengx]

When set the XENON_CONFIG as:

[RunDB]
rundb_api_url = http://xenon-runsdb.grid.uchicago.edu:80
tk_rundb_api_url = http://xenon-runsdb.grid.uchicago.edu:5000

and with PR #135, the .dbtoken is still not updated on OSG computing node:

Database version: '5.0.9dev' (sqlite:////scratch/xudc/workflows/el9.2024.09.1-xenonnt_offline-global_v16-053018-202410071959/runs/outsource_workflow-0.stampede.db)
Database version: '5.0.9dev' (sqlite:////scratch/xudc/workflows/el9.2024.09.1-xenonnt_offline-global_v16-053018-202410071959/runs/outsource_workflow-0.stampede.db)
Database version: '5.0.9dev' (sqlite:////scratch/xudc/workflows/el9.2024.09.1-xenonnt_offline-global_v16-053018-202410071959/runs/outsource_workflow-0.stampede.db)

************************************Summary*************************************

 Submit Directory   : /scratch/xudc/workflows/el9.2024.09.1-xenonnt_offline-global_v16-053018-202410071959/runs
 Workflow Status    : failure
 Total jobs         :     23 (100.00%)
 # jobs succeeded   :      9 (39.13%)
 # jobs failed      :      1 (4.35%)
 # jobs held        :      0 (0.00%)
 # jobs unsubmitted :     13 (56.52%)

******************************Failed jobs' details******************************

===============================combine_ID0000003================================

 last state: POST_SCRIPT_FAILED
       site: condorpool
submit file: 00/00/combine_ID0000003.sub
output file: 00/00/combine_ID0000003.out.001
 error file: 00/00/combine_ID0000003.err.001

-------------------------------Task #1 - Summary--------------------------------

site        : condorpool
hostname    : cn429.storrs.hpc.uconn.edu
executable  : /srv/pegasus.QHnLu1ip6/combine
arguments   : -
exitcode    : 1
working dir : /srv/pegasus.QHnLu1ip6

----------------Task #1 - combine - ID0000003 - Kickstart stderr----------------

 WARNING: X509_CERT_DIR is set set and could lead to problems when using this environment
2024-10-07 21:40:56,473 - utilix - DEBUG - Token exists at /srv/pegasus.QHnLu1ip6/.dbtoken
2024-10-07 21:40:56,473 - utilix - DEBUG - Creating a new token: doing API call now
2024-10-07 21:43:08,326 - admix - WARNING - Initializing utilix DB failed. You cannot do database operations
2024-10-07 21:43:17,656 - utilix - DEBUG - Token exists at /srv/pegasus.QHnLu1ip6/.dbtoken
2024-10-07 21:43:17,656 - utilix - DEBUG - Creating a new token: doing API call now
2024-10-07 21:45:27,590 - admix - WARNING - Initializing utilix DB failed. You cannot do database operations
2024-10-07 21:45:50,178 - utilix - DEBUG - Token exists at /srv/pegasus.QHnLu1ip6/.dbtoken
2024-10-07 21:45:50,178 - utilix - DEBUG - Creating a new token: doing API call now
2024-10-07 21:47:59,142 - admix - WARNING - Initializing utilix DB failed. You cannot do database operations
2024-10-07 21:48:05,134 - utilix - DEBUG - Token exists at /srv/pegasus.QHnLu1ip6/.dbtoken
2024-10-07 21:48:05,134 - utilix - DEBUG - Creating a new token: doing API call now
Traceback (most recent call last):
  File "/opt/XENONnT/anaconda/envs/XENONnT_el9.2024.09.1/lib/python3.9/site-packages/urllib3/connection.py", line 199, in _new_conn
    sock = connection.create_connection(
  File "/opt/XENONnT/anaconda/envs/XENONnT_el9.2024.09.1/lib/python3.9/site-packages/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/opt/XENONnT/anaconda/envs/XENONnT_el9.2024.09.1/lib/python3.9/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/XENONnT/anaconda/envs/XENONnT_el9.2024.09.1/lib/python3.9/site-packages/urllib3/connectionpool.py", line 789, in urlopen
    response = self._make_request(
  File "/opt/XENONnT/anaconda/envs/XENONnT_el9.2024.09.1/lib/python3.9/site-packages/urllib3/connectionpool.py", line 495, in _make_request
    conn.request(
  File "/opt/XENONnT/anaconda/envs/XENONnT_el9.2024.09.1/lib/python3.9/site-packages/urllib3/connection.py", line 441, in request
    self.endheaders()
  File "/opt/XENONnT/anaconda/envs/XENONnT_el9.2024.09.1/lib/python3.9/http/client.py", line 1280, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/opt/XENONnT/anaconda/envs/XENONnT_el9.2024.09.1/lib/python3.9/http/client.py", line 1040, in _send_output
    self.send(msg)
  File "/opt/XENONnT/anaconda/envs/XENONnT_el9.2024.09.1/lib/python3.9/http/client.py", line 980, in send
    self.connect()
  File "/opt/XENONnT/anaconda/envs/XENONnT_el9.2024.09.1/lib/python3.9/site-packages/urllib3/connection.py", line 279, in connect
    self.sock = self._new_conn()
  File "/opt/XENONnT/anaconda/envs/XENONnT_el9.2024.09.1/lib/python3.9/site-packages/urllib3/connection.py", line 214, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x153d7a937c40>: Failed to establish a new connection: [Errno 110] Connection timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/XENONnT/anaconda/envs/XENONnT_el9.2024.09.1/lib/python3.9/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/opt/XENONnT/anaconda/envs/XENONnT_el9.2024.09.1/lib/python3.9/site-packages/urllib3/connectionpool.py", line 843, in urlopen
    retries = retries.increment(
  File "/opt/XENONnT/anaconda/envs/XENONnT_el9.2024.09.1/lib/python3.9/site-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='xenon-runsdb.grid.uchicago.edu', port=5000): Max retries exceeded with url: /login (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x153d7a937c40>: Failed to establish a new connection: [Errno 110] Connection timed out'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/pegasus.QHnLu1ip6/combine.py", line 8, in <module>
    from outsource.upload import upload_to_rucio
  File "/srv/pegasus.QHnLu1ip6/.local/lib/python3.9/site-packages/outsource/upload.py", line 13, in <module>
    db = DB()
  File "/srv/pegasus.QHnLu1ip6/.local/lib/python3.9/site-packages/utilix/rundb.py", line 200, in __init__
    self._instances[key].initialize(*args, **kwargs)
  File "/srv/pegasus.QHnLu1ip6/.local/lib/python3.9/site-packages/utilix/rundb.py", line 214, in initialize
    token = Token(token_path)
  File "/srv/pegasus.QHnLu1ip6/.local/lib/python3.9/site-packages/utilix/rundb.py", line 96, in __init__
    self.new_token()
  File "/srv/pegasus.QHnLu1ip6/.local/lib/python3.9/site-packages/utilix/rundb.py", line 120, in new_token
    response = requests.post(path, data=data, headers=BASE_HEADERS)
  File "/opt/XENONnT/anaconda/envs/XENONnT_el9.2024.09.1/lib/python3.9/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
  File "/opt/XENONnT/anaconda/envs/XENONnT_el9.2024.09.1/lib/python3.9/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/XENONnT/anaconda/envs/XENONnT_el9.2024.09.1/lib/python3.9/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/XENONnT/anaconda/envs/XENONnT_el9.2024.09.1/lib/python3.9/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/opt/XENONnT/anaconda/envs/XENONnT_el9.2024.09.1/lib/python3.9/site-packages/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='xenon-runsdb.grid.uchicago.edu', port=5000): Max retries exceeded with url: /login (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x153d7a937c40>: Failed to establish a new connection: [Errno 110] Connection timed out'))

real	4m26.201s
user	0m6.045s
sys	0m0.877s