1.7.0 (#257)

* locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (#229) * 1.6.1 * bugfix (#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (#236) will not alter the lsf launch. * Bugfix for new celery versions (#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (#251) * Feature/ben/pgen (#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (#254) * Update CHANGELOG.md * pretty yaml (#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (#256) * expose celery (#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com>
LLNL · Jul 23, 2020 · ee8cb8b · ee8cb8b
1 parent 3f5ad6d
commit ee8cb8b
Show file tree

Hide file tree

Showing 65 changed files with 611 additions and 187 deletions.
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
@@ -0,0 +1,19 @@
+image: python:3.8-slim-buster
+
+job1:
+    script:
+        - python3 -m venv venv
+        - source venv/bin/activate
+        - pip3 install --upgrade pip
+        - pip3 install -r requirements.txt
+        - pip3 install -r requirements/dev.txt
+        - pip3 install -r merlin/examples/workflows/feature_demo/requirements.txt
+        - pip3 install -e .
+        - pip3 install --upgrade sphinx
+        - merlin config
+
+        - merlin stop-workers
+
+        - python3 -m pytest tests/
+        - python3 tests/integration/run_tests.py --verbose --local
+
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,6 +4,28 @@ All notable changes to Merlin will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.7.0]
+
+### Added
+- The ability to override any value of the celery configuration thru `app.yaml` in `celery.override`.
+- Support and faq entry for `pgen` with `merlin run --pgen` and optional `--parg`.
+- Documentation on `level_max_dirs`.
+- Easier-to-read provenance specs.
+- Documentation on the new 3 types of provenance spec.
+
+### Fixed
+- Flux test example data collection for new versions of flux.
+- Fixed Docker ubuntu version.
+- Removed expansion of env variables in shell sections (`cmd` and `restart`) of provenance
+  specs. This allows the shell command itself to expand environment variables, and gives
+  users greater flexibility.
+- Allowed environment variables to be properly expanded in study `description.name`.
+- Tilde (~) now properly expands as part of a path in non-shell sections.
+- The rediss cert_reqs keyword was changed to ssl_cert_reqs.
+
+### Changed
+- Updated tutorial redis version to 6.0.5.
+
 ## [1.6.2]
 
 ### Added

diff --git a/Dockerfile b/Dockerfile
@@ -1,4 +1,4 @@
-FROM ubuntu
+FROM ubuntu:18.04
 LABEL maintainer="Joe Koning koning1@llnl.gov"
 
 ARG USER=merlinu

diff --git a/Makefile b/Makefile
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.6.2.
+# This file is part of Merlin, Version: 1.7.0.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/docs/source/faq.rst b/docs/source/faq.rst
@@ -377,6 +377,24 @@ num resource set: Number of resource sets
 launch_distribution : The distribution of resources (default: plane:{procs/nodes})
 lsf: Verbatim flags only for the lsf parallel launch (jsrun ... <lsf>)
 
+What is level_max_dirs?
+~~~~~~~~~~~~~~~~~~~~~~~
+``level_max_dirs`` is an optional field that goes under the ``merlin.samples`` section
+of a yaml spec. It caps the number of sample directories that can be generated
+at a single level of a study's sample hierarchy. This is useful for getting around
+filesystem constraints when working with massive amounts of data.
+
+Defaults to 25.
+
+What is pgen?
+~~~~~~~~~~~~~
+``pgen`` stands for "parameter generator". It's a way to override the parameters in the
+``global.parameters`` spec section, instead generating them programatically with a python script.
+Merlin offers the same pgen functionality as Maestro.
+
+See `this guide <https://maestrowf.readthedocs.io/en/latest/parameters.html#parameter-generator-pgen>`_ for details on using ``pgen``.
+It's a Maestro doc, but the exact same flags can be used in conjunction with ``merlin run``.
+
 Where can I learn more about merlin?
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 Check out `our paper <https://arxiv.org/abs/1912.02892>`_ on arXiv.
diff --git a/docs/source/merlin_specification.rst b/docs/source/merlin_specification.rst
@@ -290,3 +290,4 @@ see :doc:`./merlin_variables`.
       generate:
         cmd: |
         python $(SPECROOT)/make_samples.py -dims 2 -n 10 -outfile=$(INPUT_PATH)/samples.npy "[(1.3, 1.3, 'linear'), (3.3, 3.3, 'linear')]"
+      level_max_dirs: 25
diff --git a/docs/source/merlin_variables.rst b/docs/source/merlin_variables.rst
@@ -21,8 +21,10 @@ The directory structure of merlin output looks like this:
     OUTPUT_PATH
         MERLIN_WORKSPACE
             MERLIN_INFO
-                <provenance_spec.yaml>
-            <other_step_name>.workspace
+                <name>.orig.yaml
+                <name>.partial.yaml
+                <name>.expanded.yaml
+            <step_name>.workspace
             WORKSPACE
 
 
@@ -52,7 +54,7 @@ Reserved variables
     - The workspace directory for a single step.
     - ``$(OUTPUT_PATH)/ensemble_name_$(MERLIN_TIMESTAMP)/step_name/``
   * - ``$(MERLIN_INFO)``
-    - Directory within ``MERLIN_WORKSPACE`` that holds a provenance spec.
+    - Directory within ``MERLIN_WORKSPACE`` that holds the provenance specs and sample generation results.
       Commonly used to hold ``samples.npy``.
     - ``$(MERLIN_WORKSPACE)/merlin_info/``
   * - ``$(MERLIN_SAMPLE_ID)``
@@ -91,17 +93,68 @@ Variables defined by a specification file in the ``env`` section, as in this exa
 
     env:
         variables:
+            ID: 42
             EXAMPLE_VAR:    hello
 
+As long as they're defined in order, you can nest user variables like this:
 
-Like all other Merlin variables, these may be used within specification steps as below:
+.. code-block:: yaml
+
+    env:
+        variables:
+            EXAMPLE_VAR:    hello
+            WORKER_NAME: $(EXAMPLE_VAR)_worker
+
+Like all other Merlin variables, user variables may be used anywhere (as a yaml key or value) within a specification as below:
 
 .. code-block:: yaml
 
     cmd: echo "$(EXAMPLE_VAR), world!"
+    ...
+    $(WORKER_NAME):
+        args: ...
+
+If you want to programmatically define the study name, you can include variables
+in the ``description.name`` field as long as it makes a valid filename:
+
+.. code-block:: yaml
+
+    description:
+        name: my_$(EXAMPLE_VAR)_study_$(ID)
+        description: example of programmatic study name
+
+The above would produce a study called ``my_hello_study_42``.
 
+Environment variables
+---------------------
+Merlin expands Unix environment variables for you. The values of the user variables below would be expanded:
 
+.. code-block:: yaml
+
+    env:
+        variables:
+            MY_HOME: ~/
+            MY_PATH: $PATH
+            USERNAME: ${USER}
+
+However, Merlin leaves environment variables found in shell scripts (think ``cmd`` and ``restart``) alone.
+So this step:
+
+.. code-block:: yaml
+
+    - name: step1
+      description: an example
+      run:
+        cmd: echo $PATH ; echo $(MY_PATH)
+
+...would be expanded as:
+
+.. code-block:: yaml
 
+    - name: step1
+      description: an example
+      run:
+        cmd: echo $PATH ; echo /an/example/:/path/string/
 
 Step return variables
 -----------------------------------

diff --git a/docs/source/modules/hello_world/hello_world.rst b/docs/source/modules/hello_world/hello_world.rst
@@ -188,7 +188,7 @@ The whole file tree looks like this:
 
 A lot of stuff, right? Here's what it means:
 
-* The yaml file inside ``merlin_info/`` is called the provenance spec. It's a copy of the original spec that was run.
+* The 3 yaml files inside ``merlin_info/`` are called the provenance specs. They are copies of the original spec that was run, some showing under-the-hood variable expansions.
 
 * ``MERLIN_FINISHED`` files indicate that the step ran successfully.
 

diff --git a/docs/source/modules/installation/installation.rst b/docs/source/modules/installation/installation.rst
@@ -94,7 +94,7 @@ can also be used for the celery broker. This method will be called local-redis.
 .. code-block:: bash
 
   # Download redis
-  wget http://download.redis.io/releases/redis-5.0.7.tar.gz
+  wget http://download.redis.io/releases/redis-6.0.5.tar.gz
 
   # Untar
   tar xvf redis*.tar.gz
@@ -361,8 +361,8 @@ server and python.
      mkdir -p ${HOME}/merlinu/cert_redis
      cp result/* ${HOME}/merlinu/cert_redis
 
-The ``redis:6.0-rc2`` docker service is exchanged for the previous
-``redis:latest`` service. The configuration below does not use client
+
+The configuration below does not use client
 verification ``--tls-auth-clients no`` so the ssl files do not need to 
 be defined as shown in the :ref:`broker_redis_ssl` section.
 

diff --git a/merlin/__init__.py b/merlin/__init__.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.6.2.
+# This file is part of Merlin, Version: 1.7.0.
 #
 # For details, see https://github.com/LLNL/merlin.
 #
@@ -38,7 +38,7 @@
 import sys
 
 
-__version__ = "1.6.2"
+__version__ = "1.7.0"
 VERSION = __version__
 PATH_TO_PROJ = os.path.join(os.path.dirname(__file__), "")
 

diff --git a/merlin/ascii_art.py b/merlin/ascii_art.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.6.2.
+# This file is part of Merlin, Version: 1.7.0.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/celery.py b/merlin/celery.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.6.2.
+# This file is part of Merlin, Version: 1.7.0.
 #
 # For details, see https://github.com/LLNL/merlin.
 #
@@ -40,10 +40,10 @@
 from celery.signals import worker_process_init
 
 import merlin.common.security.encrypt_backend_traffic
-from merlin.config import broker, results_backend
+from merlin.config import broker, celeryconfig, results_backend
 from merlin.config.configfile import CONFIG
-from merlin.log_formatter import FORMATS
 from merlin.router import route_for_task
+from merlin.utils import nested_namespace_to_dicts
 
 
 LOG = logging.getLogger(__name__)
@@ -54,65 +54,59 @@
 results_ssl = False
 try:
     BROKER_URI = broker.get_connection_string()
-    LOG.info(f"broker: {broker.get_connection_string(include_password=False)}")
+    LOG.debug(f"broker: {broker.get_connection_string(include_password=False)}")
     broker_ssl = broker.get_ssl_config()
-    LOG.info(f"broker_ssl = {broker_ssl}")
+    LOG.debug(f"broker_ssl = {broker_ssl}")
     RESULTS_BACKEND_URI = results_backend.get_connection_string()
     results_ssl = results_backend.get_ssl_config(celery_check=True)
-    LOG.info(
+    LOG.debug(
         f"results: {results_backend.get_connection_string(include_password=False)}"
     )
-    LOG.info(f"results: redis_backed_use_ssl = {results_ssl}")
+    LOG.debug(f"results: redis_backed_use_ssl = {results_ssl}")
 except ValueError:
     # These variables won't be set if running with '--local'.
     BROKER_URI = None
     RESULTS_BACKEND_URI = None
 
-
+# initialize app with essential properties
 app = Celery(
     "merlin",
     broker=BROKER_URI,
     backend=RESULTS_BACKEND_URI,
     broker_use_ssl=broker_ssl,
     redis_backend_use_ssl=results_ssl,
+    task_routes=(route_for_task,),
 )
 
+# load merlin config defaults
+app.conf.update(**celeryconfig.DICT)
+
+# load config overrides from app.yaml
+if (
+    (not hasattr(CONFIG.celery, "override"))
+    or (CONFIG.celery.override is None)
+    or (len(nested_namespace_to_dicts(CONFIG.celery.override)) == 0)
+):
+    LOG.debug("Skipping celery config override; 'celery.override' field is empty.")
+else:
+    override_dict = nested_namespace_to_dicts(CONFIG.celery.override)
+    override_str = ""
+    i = 0
+    for k, v in override_dict.items():
+        if k not in str(app.conf.__dict__):
+            raise ValueError(f"'{k}' is not a celery configuration.")
+        override_str += f"\t{k}:\t{v}"
+        if i != len(override_dict) - 1:
+            override_str += "\n"
+        i += 1
+    LOG.info(
+        f"Overriding default celery config with 'celery.override' in 'app.yaml':\n{override_str}"
+    )
+    app.conf.update(**override_dict)
 
-app.conf.update(
-    task_serializer="pickle", accept_content=["pickle"], result_serializer="pickle"
-)
-
+# auto-discover tasks
 app.autodiscover_tasks(["merlin.common"])
 
-app.conf.update(
-    task_acks_late=True,
-    task_reject_on_worker_lost=True,
-    task_publish_retry_policy={
-        "interval_start": 10,
-        "interval_step": 10,
-        "interval_max": 60,
-    },
-    redis_max_connections=100000,
-)
-
-# Set a timeout to acknowledge a task before it's available to grab
-# again (default 24 hours).
-app.conf.broker_transport_options = {
-    "visibility_timeout_seconds": CONFIG.celery.visibility_timeout_seconds,
-    "max_connections": 100,
-}
-
-app.conf.update(broker_pool_limit=0)
-
-# Task routing: call our default queue merlin
-app.conf.task_routes = (route_for_task,)
-app.conf.task_default_queue = "merlin"
-
-# Log formatting
-app.conf.worker_log_color = True
-app.conf.worker_log_format = FORMATS["DEFAULT"]
-app.conf.worker_task_log_format = FORMATS["WORKER"]
-
 
 @worker_process_init.connect()
 def setup(**kwargs):

diff --git a/merlin/common/__init__.py b/merlin/common/__init__.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.6.2.
+# This file is part of Merlin, Version: 1.7.0.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/abstracts/__init__.py b/merlin/common/abstracts/__init__.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.6.2.
+# This file is part of Merlin, Version: 1.7.0.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/abstracts/enums/__init__.py b/merlin/common/abstracts/enums/__init__.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.6.2.
+# This file is part of Merlin, Version: 1.7.0.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/openfilelist.py b/merlin/common/openfilelist.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.6.2.
+# This file is part of Merlin, Version: 1.7.0.
 #
 # For details, see https://github.com/LLNL/merlin.
 #