Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/koning/info conn timeout #226

Merged
merged 20 commits into from
Jun 2, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
a120c3a
Merge pull request #1 from LLNL/develop
koning Dec 4, 2019
d76d67f
Merge remote-tracking branch 'upstream/develop' into develop
koning Mar 6, 2020
c18866b
Merge branch 'develop' of github.com:llnl/merlin into develop
koning Mar 9, 2020
3a7d035
Merge branch 'develop' of github.com:llnl/merlin into develop
koning Mar 9, 2020
43c8e49
Merge branch 'develop' of github.com:llnl/merlin into develop
koning Mar 11, 2020
3bb9184
Merge branch 'develop' of github.com:llnl/merlin into develop
koning Mar 12, 2020
9776a3e
Merge branch 'develop' of github.com:llnl/merlin into develop
koning Mar 13, 2020
461aad3
Merge branch 'develop' of github.com:llnl/merlin into develop
koning Mar 18, 2020
0bad54c
Merge branch 'develop' of github.com:llnl/merlin into develop
koning Mar 23, 2020
3b8b4a4
Merge branch 'develop' of github.com:llnl/merlin into develop
koning Mar 25, 2020
7ee915b
Merge branch 'develop' of github.com:llnl/merlin into develop
koning Mar 25, 2020
3de4524
Merge branch 'develop' of github.com:llnl/merlin into develop
koning Mar 30, 2020
b3b3d81
Merge branch 'develop' of github.com:llnl/merlin into develop
koning Mar 31, 2020
58bedd7
Merge branch 'develop' of github.com:llnl/merlin into develop
koning Apr 2, 2020
40865c1
Merge branch 'develop' of github.com:koning/merlin into develop
koning Apr 27, 2020
273b02d
Merge branch 'develop' of github.com:llnl/merlin into develop
koning Apr 27, 2020
a3e21ca
Merge branch 'develop' of github.com:llnl/merlin into develop
koning May 11, 2020
034004d
Merge remote-tracking branch 'upstream/develop' into develop
koning Jun 2, 2020
c0dc398
Add a timeout check for the merlin info kombu connection test. The redis
koning Jun 2, 2020
ac98c34
Update docs.
koning Jun 2, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- The broker name can now be amqps (with ssl) or amqp (without ssl).
- The encryption key will now be created when running merlin config.
- The merlin info connection check will now enforce a minute timeout
check for the server connections.

### Fixed
- Added a check for initial running workers when using merlin monitor to
Expand Down
4 changes: 3 additions & 1 deletion docs/source/merlin_commands.rst
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,9 @@ Information (``merlin info``)
-----------------------------

Information about your merlin and python configuration can be printed out by using the
``info`` command. This is helpful for debugging.
``info`` command. This is helpful for debugging. Included in this command
is a server check which will check for server connections. The connection
check will timeout after 60 seconds.

.. code:: bash
Expand Down
15 changes: 14 additions & 1 deletion merlin/display.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,8 @@
"""
import pprint
import subprocess
import time
from multiprocessing import Process

from kombu import Connection
from tabulate import tabulate
Expand All @@ -53,11 +55,22 @@ def check_server_access(sconf):
print("-" * 28)

excpts = {}
connect_timeout = 60
for s in servers:
if s in sconf:
try:
conn = Connection(sconf[s])
conn.connect()
conn_check = Process(target=conn.connect)
conn_check.start()
counter = 0
while conn_check.is_alive():
time.sleep(1)
counter += 1
if counter > connect_timeout:
conn_check.kill()
raise Exception(
f"Connection was killed due to timeout ({connect_timeout}s)"
)
conn.release()
print(f"{s} connection: OK")
except Exception as e:
Expand Down