1.7.3 #266
Merged
1.7.3 #266
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc.
* updated default task timeout * Added to CHANGELOG
* Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption.
* Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style.
* working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info
* exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG
* Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs.
* sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style
* working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema
…236) will not alter the lsf launch.
* Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com>
* added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG
* now using regex to find variable tokens * added shell ref function
* Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment.
* Update redis to new version with TLS support. * Add changelog.
* added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden
* copy samplesfile to merlin_info * added to CHANGELOG
* Add bugfix log. * Add an exception catcher for the multiprocessing Process class.
lucpeterson
added a commit
to lucpeterson/merlin
that referenced
this pull request
Aug 19, 2020
* 1.7.0 (LLNL#257) * locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (LLNL#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (LLNL#218) race conditions. * updated timeout (LLNL#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (LLNL#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (LLNL#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (LLNL#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (LLNL#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (LLNL#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (LLNL#229) * 1.6.1 * bugfix (LLNL#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (LLNL#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (LLNL#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (LLNL#236) will not alter the lsf launch. * Bugfix for new celery versions (LLNL#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (LLNL#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (LLNL#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (LLNL#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (LLNL#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (LLNL#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (LLNL#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (LLNL#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (LLNL#251) * Feature/ben/pgen (LLNL#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (LLNL#254) * Update CHANGELOG.md * pretty yaml (LLNL#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (LLNL#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (LLNL#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (LLNL#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (LLNL#256) * expose celery (LLNL#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (LLNL#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> * 1.7.1 hotfix (LLNL#261) * locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (LLNL#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (LLNL#218) race conditions. * updated timeout (LLNL#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (LLNL#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (LLNL#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (LLNL#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (LLNL#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (LLNL#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (LLNL#229) * 1.6.1 * bugfix (LLNL#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (LLNL#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (LLNL#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (LLNL#236) will not alter the lsf launch. * Bugfix for new celery versions (LLNL#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (LLNL#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (LLNL#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (LLNL#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (LLNL#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (LLNL#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (LLNL#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (LLNL#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (LLNL#251) * Feature/ben/pgen (LLNL#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (LLNL#254) * Update CHANGELOG.md * pretty yaml (LLNL#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (LLNL#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (LLNL#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (LLNL#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (LLNL#256) * expose celery (LLNL#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (LLNL#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (LLNL#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (LLNL#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> * 1.7.2 (LLNL#264) * locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (LLNL#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (LLNL#218) race conditions. * updated timeout (LLNL#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (LLNL#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (LLNL#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (LLNL#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (LLNL#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (LLNL#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (LLNL#229) * 1.6.1 * bugfix (LLNL#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (LLNL#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (LLNL#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (LLNL#236) will not alter the lsf launch. * Bugfix for new celery versions (LLNL#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (LLNL#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (LLNL#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (LLNL#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (LLNL#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (LLNL#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (LLNL#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (LLNL#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (LLNL#251) * Feature/ben/pgen (LLNL#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (LLNL#254) * Update CHANGELOG.md * pretty yaml (LLNL#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (LLNL#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (LLNL#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (LLNL#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (LLNL#256) * expose celery (LLNL#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (LLNL#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (LLNL#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (LLNL#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 * hotfix * fix * 1.7.2 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> * 1.7.2 (LLNL#265) * locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (LLNL#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (LLNL#218) race conditions. * updated timeout (LLNL#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (LLNL#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (LLNL#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (LLNL#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (LLNL#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (LLNL#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (LLNL#229) * 1.6.1 * bugfix (LLNL#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (LLNL#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (LLNL#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (LLNL#236) will not alter the lsf launch. * Bugfix for new celery versions (LLNL#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (LLNL#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (LLNL#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (LLNL#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (LLNL#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (LLNL#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (LLNL#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (LLNL#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (LLNL#251) * Feature/ben/pgen (LLNL#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (LLNL#254) * Update CHANGELOG.md * pretty yaml (LLNL#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (LLNL#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (LLNL#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (LLNL#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (LLNL#256) * expose celery (LLNL#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (LLNL#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (LLNL#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (LLNL#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 * hotfix * fix * 1.7.2 * Update CHANGELOG.md * Update CHANGELOG.md Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> * 1.7.3 (LLNL#266) * locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (LLNL#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (LLNL#218) race conditions. * updated timeout (LLNL#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (LLNL#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (LLNL#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (LLNL#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (LLNL#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (LLNL#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (LLNL#229) * 1.6.1 * bugfix (LLNL#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (LLNL#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (LLNL#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (LLNL#236) will not alter the lsf launch. * Bugfix for new celery versions (LLNL#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (LLNL#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (LLNL#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (LLNL#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (LLNL#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (LLNL#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (LLNL#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (LLNL#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (LLNL#251) * Feature/ben/pgen (LLNL#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (LLNL#254) * Update CHANGELOG.md * pretty yaml (LLNL#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (LLNL#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (LLNL#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (LLNL#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (LLNL#256) * expose celery (LLNL#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (LLNL#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (LLNL#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (LLNL#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 * hotfix * fix * 1.7.2 * Update CHANGELOG.md * Update CHANGELOG.md * 1.7.3 * 1.7.3 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com>
AlexanderWinterLLNL
added a commit
that referenced
this pull request
Jul 6, 2021
* Squash and merge develop to Main * Master (#2) * 1.7.0 (#257) * locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (#229) * 1.6.1 * bugfix (#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (#236) will not alter the lsf launch. * Bugfix for new celery versions (#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (#251) * Feature/ben/pgen (#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (#254) * Update CHANGELOG.md * pretty yaml (#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (#256) * expose celery (#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> * 1.7.1 hotfix (#261) * locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (#229) * 1.6.1 * bugfix (#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (#236) will not alter the lsf launch. * Bugfix for new celery versions (#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (#251) * Feature/ben/pgen (#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (#254) * Update CHANGELOG.md * pretty yaml (#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (#256) * expose celery (#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> * 1.7.2 (#264) * locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (#229) * 1.6.1 * bugfix (#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (#236) will not alter the lsf launch. * Bugfix for new celery versions (#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (#251) * Feature/ben/pgen (#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (#254) * Update CHANGELOG.md * pretty yaml (#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (#256) * expose celery (#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 * hotfix * fix * 1.7.2 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> * 1.7.2 (#265) * locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (#229) * 1.6.1 * bugfix (#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (#236) will not alter the lsf launch. * Bugfix for new celery versions (#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (#251) * Feature/ben/pgen (#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (#254) * Update CHANGELOG.md * pretty yaml (#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (#256) * expose celery (#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 * hotfix * fix * 1.7.2 * Update CHANGELOG.md * Update CHANGELOG.md Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> * 1.7.3 (#266) * locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (#229) * 1.6.1 * bugfix (#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (#236) will not alter the lsf launch. * Bugfix for new celery versions (#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (#251) * Feature/ben/pgen (#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (#254) * Update CHANGELOG.md * pretty yaml (#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (#256) * expose celery (#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 * hotfix * fix * 1.7.2 * Update CHANGELOG.md * Update CHANGELOG.md * 1.7.3 * 1.7.3 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> * Revert "Master (#2)" This reverts commit 1d7495b. * [Flake8 src refactor] display.py and broker.py refactored. * [Standardize CI linting and Makefile] Updated the Makefile and CI to hold one standard for linting from setup.cfg. * Add ability to enter step walltime in HH:MM:SS format. With HH and MM optional. Previously, script adapters had assumed a format that depends on the batch type. Now it'll get converted internally to the right format. * Updated CHANGELOG and docs * Add testing for integers too * An explicit conversion test. Making mypy and black happy. * Move to 1.8.0 * Explictly wrap queue name string in quotes to make it more robust (#320) * Explictly wrap queue name string in quotes to make it more robust * Move CHANGELOG note to v1.8.0 * Update CHANGELOG.md * Update test_time_formats.py Utilized typing for all variables, utilizing PyTest parametrized marks to allow testing to flow regardless of individual fail states of test data. * Update test_time_formats.py Added test to verify some exception handling, compressed tests to not have as much indirection. * Typo in variable name change * Fix style before merging, update to Makefile so fix-style upgrades all dev packages before style check. (#321) Co-authored-by: Alexander Cameron Winter <winter27@quartz1148.llnl.gov> * Revert "Merge branch 'main' into develop" using -m option. This reverts commit a9276b8, reversing changes made to 63a094b. Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> Co-authored-by: Alexander Cameron Winter <winter27@quartz1148.llnl.gov> Co-authored-by: Alexander Winter <80929291+AlexanderWinterLLNL@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.