App.yaml for merlin server #369
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…rlin server configuration
…w app.yaml to use merlin server
lucpeterson
reviewed
Jul 21, 2022
lucpeterson
approved these changes
Jul 26, 2022
ryannova
added a commit
that referenced
this pull request
Dec 21, 2022
* Create python-publish.yml (#353) * Adding GitHub action for publishing to pypi * Workflows Community Initiative Metadata (#355) * Added Workflows Community Initiative metadata info; fixed some old links * Run black * Add updates for lgtm CI security site (#357) * Update code to remove LGTM Errors and Warnings and implement Recommendations. * Change BaseException to Exception. * Add lgtm config file. * Changes for flake8. * Add TypeError yo yam read. * Add TypeError to yaml read. * Just return when successful on the yaml read. * Fix typo. * Add merlin/examples to lgtm exclude list as well. * Add ssl comment. * Fix typo. * Update version to 1.8.5. * Update conf.py for new sphinx versions. * Added Server Command Feature to Merlin (#360) * Added merlin server capability to initalize, monitor and stop redis containers * Added configuration for singularity, docker, and podman in merlin/server/ * Created documentation for "merlin server" command * Added tests to initialize, start and stop a singularity+redis server to the local test suite. (Future: add to the "distributed tests" connecting to that server and running merlin) Co-authored-by: Ryan Lee <ryannova@users.noreply.github.com> & Joe Koning <koning@users.noreply.github.com> * Fix lgtm returns (#363) * Changed script sys.exit commands to use try/catch, per lgtm recommendation * Allow for flux exec arguments to limit the number of celery workers. (#366) * Added the flux_exec batch argument to allow for flux exec arguments, e.g. flux_exec: flux exec -r "0-1" to run celery workers only on ranks 0 and 1 of a multi-rank allocation. * Remove period. * Merlin server configuration through commands (#365) * Reorganized functions within server_setup and server_config * Rename server_setup file to server_commands * Added password generation for redis container in merlin server * Changed redis configuration to require password authentication * Added merlin config flags ipaddress, port, password, directory, snapshot_seconds, snapshot_changes, snapshot_file, append_mode, append_file * Added server_util.py * Added merlin user file into merlin server config * Added RedisConfig class to interact and change config values within the redis config file * Added merlin server restart * Updated info messages * Added function to add/remove users and store info to user file * Update running container with new users and removed users * Added ServerConfig, ProcessConfig, ContainerConfig, and ContainerFormatConfig classes to interact with configuration files * Adjusted adding user and password to use values in config files * Updated host in redis.conf * Updated pull_server_image step * Moved creation of local merlin config to create_server_config() * Added placeholder for documentation of restart and config commands for merlin server * Bugfix/changelog ci (#370) * remove deprecated gitlab ci file * Change CHANGELOG test to work for PRs other than to main * App.yaml for merlin server (#369) * Added AppYaml class to pull app.yaml and make changes required for merlin server configuration * Applied AppYaml class and added log message to inform users to use new app.yaml to use merlin server * Update LOG messages to inform users regarding local runs and instruct users of how to use app.yaml for local configuration * Changed type to image type in ContainerConfig * Shorten CHANGELOG.md for merlin server changes * Updated read in AppYaml to utilize merlin.util.load_yaml * Updated merlin server unit testing (#372) * Added additional tests for merlin server in test definitions * Fixed directory change to create a new directory if one doesn't exist * Updated redis version to provide acl user channel support * Addition of new shortcuts in specification file (#375) * Added five shortcuts to the specification definition MERLIN_SAMPLE_VECTOR, MERLIN_SAMPLE_NAMES, MERLIN_SPEC_ORIGINAL_TEMPLATE, MERLIN_SPEC_EXECUTED_RUN, MERLIN_SPEC_ARCHIVED_COPY * Added documentation for the above shortcuts. Co-authored-by: Jim Gaffney <gaffney3@llnl.gov> * Remove emoji from issue templates (#377) * Update bug_report.md remove "buggy" emoji * Update feature_request.md * Update question.md * Update CHANGELOG.md * Update CHANGELOG.md typo fix Co-authored-by: Ryan Lee <lee1098@llnl.gov> * Update contribute.rst Remove more emoji from docs that are breaking pdf builds * Update cert_req to cert_regs in the docs. (#379) Co-authored-by: Ryan Lee <44886374+ryannova@users.noreply.github.com> * Ssl server check fixes (#380) * Add ssl to the Connection object for checking broker and results server acess. * Update CHANGELOG * Update documentation in tutorial and merlin server (#378) * Updated installation in instroduction and removed redis requirements * Removed pip headers and added commands for merlin server into installation * Removed additional references to old redis way and update description of merlin server * Remove more emoji from docs that are breaking pdf builds * Updated CHANGELOG to reflect changes to documentation Co-authored-by: Luc Peterson <peterson76@llnl.gov> * Update MANIFEST.in (#381) * Update MANIFEST.in Add .temp to examples in MANIFEST, so that they get bundled with pypi releases * Update CHANGELOG.md * Add support for non-merlin blocks in specification file (#376) * Adding support for "user" block in _dict_to_string method * Updated CHANGELOG * Updated Merlin Spec docs * Added user block in feature_demo.yaml example Co-authored-by: Jim Gaffney <gaffney3@llnl.gov> * Update Merlin Server (#385) * Added condition for fatal error from redis server * Update default value for config_dir * Updated fix-style target in Makefile to be consistent with other style related targets * Update default password to use generated password * Updated run user to be default rather than created user * Updated singularity command to specify configuration directory as home directory to solve unaccessible directory issue * Update merlin to use app.yaml configuration rather than its own configuration file * Docs/install changes (#383) Many modifications to documentation, including installation instructions and formatting fixes. * Maestro v 1.1.9dev1 Compatibility (#388) Maestro up to date compatibility Also unpacked maestro DAG to just use what we need, which should help reduce task message size and perhaps allow us to use other serializers in the future. * Bump certifi from 2022.9.24 to 2022.12.7 in /docs (#387) Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.9.24 to 2022.12.7. - [Release notes](https://github.com/certifi/python-certifi/releases) - [Commits](certifi/python-certifi@2022.09.24...2022.12.07) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Release 1.9.0 (#390) * Make CHANGELOG more concise * Updated Merlin version and added License to files missing it * Incremented python version for workflow test * fix merlinspec not being installed with pip and python 3.7 issues with celery * update changelog and version to 1.9.1 * fix a numpy issue on new numpy version release * modify changelog to show numpy fix * add version change to all files * re-add fix for numpy since it got removed in the last commit by accident * revert utils.py back to previous implementation * change dtype to python str type * Hotfix for merlin server unable to write config files. (#394) * Hotfix for merlin server unable to write config files. Change files to modules and copy files from new file modules Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Brian Gunnarson <49216024+bgunnar5@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Brian Gunnarson <brianfunnarson14@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added a new capability to interface with the main app.yaml, and create a new app.yaml containing new configurations specific to merlin server. This should allow users to be able to use the new app.yaml to connect to the merlin server container without having to edit the main configuration in ~/.merlin/. To use a local copy of app.yaml instead of the main configuration there should an app.yaml file in the current run directory. The app.yaml for merlin server is stored in ./merlin_server/ subdirectory, so users will need to move the app.yaml from ./merlin_server/ to the current running directory.