v1.5.4 (#228)

* locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. Co-authored-by: Joe Koning <koning@users.noreply.github.com>
LLNL · Jun 2, 2020 · a05115f · a05115f
1 parent e624d7b
commit a05115f
Show file tree

Hide file tree

Showing 51 changed files with 326 additions and 111 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,6 +4,28 @@ All notable changes to Merlin will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.5.4]
+
+### Added
+- The broker name can now be amqps (with ssl) or amqp (without ssl). 
+- The encryption key will now be created when running merlin config.
+- The merlin info connection check will now enforce a minute timeout 
+  check for the server connections.
+
+### Fixed
+- Added a check for initial running workers when using merlin monitor to
+  eliminate race condition.
+- A bug that did not change the filename of the output workspace nor of the provenance spec
+  when a user variable was included in the `description.name` field.
+- Temporarily locked Celery version at 4.4.2 to avoid fatal bug.
+
+### Changed
+- The default rabbitmq vhost is now <user> instead of /<user>.
+- Changed default visibility timeout from 2 hours to 24 hours. Exposed this in the config
+  file.
+- The monitor function will now check the queues to determine when
+  to exit.
+
 ## [1.5.3]
 
 ### Fixed

diff --git a/docs/source/merlin_commands.rst b/docs/source/merlin_commands.rst
@@ -96,7 +96,9 @@ Information (``merlin info``)
 -----------------------------
 
 Information about your merlin and python configuration can be printed out by using the 
-``info`` command. This is helpful for debugging.
+``info`` command. This is helpful for debugging. Included in this command
+is a server check which will check for server connections. The connection
+check will timeout after 60 seconds.
 
 .. code:: bash
 
@@ -108,21 +110,31 @@ Monitor (``merlin monitor``)
 Batch submission scripts may not keep the batch allocation alive
 if there is not a blocking process in the submission script. The
 ``merlin monitor`` command addresses this by providing a blocking process that
-checks for running workers every 60 seconds. If the 
-``$(MERLIN_STOP_WORKERS)`` option is used or when workers are stopped
-through some other mechanism, then the blocking process
-will exit and allow the allocation to end. The ``monitor`` command
-will not exit until all celery workers on the allocation cease.
+checks for tasks in the queues every (sleep) seconds. When the queues are empty, the 
+blocking process will exit and allow the allocation to end.
 
 .. code:: bash
 
-    $ merlin monitor [--task_server celery] [--sleep <duration>]
+    $ merlin monitor <input.yaml> [--steps <steps>] [--vars <VARIABLES=<VARIABLES>>] [--sleep <duration>][--task_server celery]
+
+Use the ``--steps`` option to identify specific steps in the specification that you want to query.
+
+The ``--vars`` option will specify desired Merlin variable values to override
+those found in the specification. The list is space-delimited and should be given after
+the input yaml file.
+``Example: --vars LEARN=path/to/new_learn.py EPOCHS=3``
 
 The ``--sleep`` argument is the duration in seconds between checks
 for workers. The default is 60 seconds.
 
 The only currently available option for ``--task_server`` is celery, which is the default when this flag is excluded.
 
+The ``monitor`` function will check for celery workers for up to
+10*(sleep) seconds before monitoring begins. The loop happens when the 
+queue(s) in the spec contain tasks, but no running workers are detected.
+This is to protect against a failed worker launch.
+
+
 Purging Tasks (``merlin purge``)
 --------------------------------
 

diff --git a/docs/source/merlin_config.rst b/docs/source/merlin_config.rst
@@ -30,16 +30,20 @@ app.yaml file, then the user's ``~/.merlin`` directory is checked.
 using a socket. 
 
 
-Broker: ``rabbitmq``
---------------------
+Broker: ``rabbitmq``, ``amqps``, ``amqp``
+-----------------------------------------
 Merlin constructs the following connection string from the relevant options in the 
 ``broker`` section of the app.yaml file. If the ``port`` argument is not defined,
 the default rabbitmq TLS port, 5671,  will be used. See the :ref:`broker_rabbitmq_ssl`
-section for more info about security with this broker.
+section for more info about security with this broker. When the ``broker``
+is ``amqp``, the default port will be 5672.
 
 
 | The prototype url for this configuration is:
-| ``amqps://{username}:{password}@{server}:{port}//{vhost}``
+| ``{conn}://{username}:{password}@{server}:{port}/{vhost}``
+
+Here ``conn`` is ``amqps`` (with ssl) when ``name`` is ``rabbitmq`` or ``amqps`` and
+``amqp`` (without ssl) when name is ``amqp``.
 
 ::
 

diff --git a/merlin/__init__.py b/merlin/__init__.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.5.3.
+# This file is part of Merlin, Version: 1.5.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #
@@ -38,7 +38,7 @@
 import sys
 
 
-__version__ = "1.5.3"
+__version__ = "1.5.4"
 VERSION = __version__
 PATH_TO_PROJ = os.path.join(os.path.dirname(__file__), "")
 

diff --git a/merlin/ascii_art.py b/merlin/ascii_art.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.5.3.
+# This file is part of Merlin, Version: 1.5.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/celery.py b/merlin/celery.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.5.3.
+# This file is part of Merlin, Version: 1.5.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #
@@ -47,6 +47,7 @@
     broker,
     results_backend,
 )
+from merlin.config.configfile import CONFIG
 from merlin.log_formatter import FORMATS
 from merlin.router import route_for_task
 
@@ -100,9 +101,12 @@
     redis_max_connections=100000,
 )
 
-# Set a one hour timeout to acknowledge a task before it's available to grab
-# again.
-app.conf.broker_transport_options = {"visibility_timeout": 7200, "max_connections": 100}
+# Set a timeout to acknowledge a task before it's available to grab
+# again (default 24 hours).
+app.conf.broker_transport_options = {
+    "visibility_timeout_seconds": CONFIG.celery.visibility_timeout_seconds,
+    "max_connections": 100,
+}
 
 app.conf.update(broker_pool_limit=0)
 

diff --git a/merlin/common/__init__.py b/merlin/common/__init__.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.5.3.
+# This file is part of Merlin, Version: 1.5.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/abstracts/__init__.py b/merlin/common/abstracts/__init__.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.5.3.
+# This file is part of Merlin, Version: 1.5.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/abstracts/enums/__init__.py b/merlin/common/abstracts/enums/__init__.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.5.3.
+# This file is part of Merlin, Version: 1.5.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/openfilelist.py b/merlin/common/openfilelist.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.5.3.
+# This file is part of Merlin, Version: 1.5.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/opennpylib.py b/merlin/common/opennpylib.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.5.3.
+# This file is part of Merlin, Version: 1.5.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/sample_index.py b/merlin/common/sample_index.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.5.3.
+# This file is part of Merlin, Version: 1.5.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/sample_index_factory.py b/merlin/common/sample_index_factory.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.5.3.
+# This file is part of Merlin, Version: 1.5.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/security/__init__.py b/merlin/common/security/__init__.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.5.3.
+# This file is part of Merlin, Version: 1.5.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/security/encrypt.py b/merlin/common/security/encrypt.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.5.3.
+# This file is part of Merlin, Version: 1.5.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/security/encrypt_backend_traffic.py b/merlin/common/security/encrypt_backend_traffic.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.5.3.
+# This file is part of Merlin, Version: 1.5.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/tasks.py b/merlin/common/tasks.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.5.3.
+# This file is part of Merlin, Version: 1.5.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/util_sampling.py b/merlin/common/util_sampling.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.5.3.
+# This file is part of Merlin, Version: 1.5.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/config/__init__.py b/merlin/config/__init__.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.5.3.
+# This file is part of Merlin, Version: 1.5.4.
 #
 # For details, see https://github.com/LLNL/merlin.
 #