Hotfix for merlin server unable to write config files. #394
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…o modules and copy files from modules
|
@ryannova I hate to be a pain but is there a way to write a test for this? E.g some test that would break before but passes now. I would like every PR to include a test. |
|
@doutriaux1 There are ways to test for this but not with the current testing structure. All of the current tests are done by cloning the merlin repo and building a virtual environment where tests are then ran using python's subprocess library. To test this you would have to reconfigure github workflow files to build the wheel and install it through that. |
|
ok that sounds overkill 😉 let's skip it for this case. |
|
😄 lol okay sounds good. Normally I do try to write a test for each PR, but this was more of a hotfix for features that were working in development environment, but a small discrepancy in the develop and release environment caused it to error during release. |
|
@bgunnar5 and I brainstormed a way to modify our testing to test building against the wheel. it should probably be a separate PR tho, since it's a little more involved |
|
@lucpeterson @bgunnar5 I would like to move us to gitlab, could that "future" PR be done on gitlab? |
lucpeterson
approved these changes
Dec 21, 2022
ryannova
added a commit
to ryannova/merlin
that referenced
this pull request
Dec 21, 2022
* Hotfix for merlin server unable to write config files. Change files to modules and copy files from new file modules
ryannova
added a commit
to ryannova/merlin
that referenced
this pull request
Dec 21, 2022
* Hotfix for merlin server unable to write config files. Change files to modules and copy files from new file modules
ryannova
added a commit
to ryannova/merlin
that referenced
this pull request
Dec 21, 2022
* Hotfix for merlin server unable to write config files. Change files to modules and copy files from new file modules
ryannova
added a commit
that referenced
this pull request
Dec 21, 2022
* Create python-publish.yml (#353) * Adding GitHub action for publishing to pypi * Workflows Community Initiative Metadata (#355) * Added Workflows Community Initiative metadata info; fixed some old links * Run black * Add updates for lgtm CI security site (#357) * Update code to remove LGTM Errors and Warnings and implement Recommendations. * Change BaseException to Exception. * Add lgtm config file. * Changes for flake8. * Add TypeError yo yam read. * Add TypeError to yaml read. * Just return when successful on the yaml read. * Fix typo. * Add merlin/examples to lgtm exclude list as well. * Add ssl comment. * Fix typo. * Update version to 1.8.5. * Update conf.py for new sphinx versions. * Added Server Command Feature to Merlin (#360) * Added merlin server capability to initalize, monitor and stop redis containers * Added configuration for singularity, docker, and podman in merlin/server/ * Created documentation for "merlin server" command * Added tests to initialize, start and stop a singularity+redis server to the local test suite. (Future: add to the "distributed tests" connecting to that server and running merlin) Co-authored-by: Ryan Lee <ryannova@users.noreply.github.com> & Joe Koning <koning@users.noreply.github.com> * Fix lgtm returns (#363) * Changed script sys.exit commands to use try/catch, per lgtm recommendation * Allow for flux exec arguments to limit the number of celery workers. (#366) * Added the flux_exec batch argument to allow for flux exec arguments, e.g. flux_exec: flux exec -r "0-1" to run celery workers only on ranks 0 and 1 of a multi-rank allocation. * Remove period. * Merlin server configuration through commands (#365) * Reorganized functions within server_setup and server_config * Rename server_setup file to server_commands * Added password generation for redis container in merlin server * Changed redis configuration to require password authentication * Added merlin config flags ipaddress, port, password, directory, snapshot_seconds, snapshot_changes, snapshot_file, append_mode, append_file * Added server_util.py * Added merlin user file into merlin server config * Added RedisConfig class to interact and change config values within the redis config file * Added merlin server restart * Updated info messages * Added function to add/remove users and store info to user file * Update running container with new users and removed users * Added ServerConfig, ProcessConfig, ContainerConfig, and ContainerFormatConfig classes to interact with configuration files * Adjusted adding user and password to use values in config files * Updated host in redis.conf * Updated pull_server_image step * Moved creation of local merlin config to create_server_config() * Added placeholder for documentation of restart and config commands for merlin server * Bugfix/changelog ci (#370) * remove deprecated gitlab ci file * Change CHANGELOG test to work for PRs other than to main * App.yaml for merlin server (#369) * Added AppYaml class to pull app.yaml and make changes required for merlin server configuration * Applied AppYaml class and added log message to inform users to use new app.yaml to use merlin server * Update LOG messages to inform users regarding local runs and instruct users of how to use app.yaml for local configuration * Changed type to image type in ContainerConfig * Shorten CHANGELOG.md for merlin server changes * Updated read in AppYaml to utilize merlin.util.load_yaml * Updated merlin server unit testing (#372) * Added additional tests for merlin server in test definitions * Fixed directory change to create a new directory if one doesn't exist * Updated redis version to provide acl user channel support * Addition of new shortcuts in specification file (#375) * Added five shortcuts to the specification definition MERLIN_SAMPLE_VECTOR, MERLIN_SAMPLE_NAMES, MERLIN_SPEC_ORIGINAL_TEMPLATE, MERLIN_SPEC_EXECUTED_RUN, MERLIN_SPEC_ARCHIVED_COPY * Added documentation for the above shortcuts. Co-authored-by: Jim Gaffney <gaffney3@llnl.gov> * Remove emoji from issue templates (#377) * Update bug_report.md remove "buggy" emoji * Update feature_request.md * Update question.md * Update CHANGELOG.md * Update CHANGELOG.md typo fix Co-authored-by: Ryan Lee <lee1098@llnl.gov> * Update contribute.rst Remove more emoji from docs that are breaking pdf builds * Update cert_req to cert_regs in the docs. (#379) Co-authored-by: Ryan Lee <44886374+ryannova@users.noreply.github.com> * Ssl server check fixes (#380) * Add ssl to the Connection object for checking broker and results server acess. * Update CHANGELOG * Update documentation in tutorial and merlin server (#378) * Updated installation in instroduction and removed redis requirements * Removed pip headers and added commands for merlin server into installation * Removed additional references to old redis way and update description of merlin server * Remove more emoji from docs that are breaking pdf builds * Updated CHANGELOG to reflect changes to documentation Co-authored-by: Luc Peterson <peterson76@llnl.gov> * Update MANIFEST.in (#381) * Update MANIFEST.in Add .temp to examples in MANIFEST, so that they get bundled with pypi releases * Update CHANGELOG.md * Add support for non-merlin blocks in specification file (#376) * Adding support for "user" block in _dict_to_string method * Updated CHANGELOG * Updated Merlin Spec docs * Added user block in feature_demo.yaml example Co-authored-by: Jim Gaffney <gaffney3@llnl.gov> * Update Merlin Server (#385) * Added condition for fatal error from redis server * Update default value for config_dir * Updated fix-style target in Makefile to be consistent with other style related targets * Update default password to use generated password * Updated run user to be default rather than created user * Updated singularity command to specify configuration directory as home directory to solve unaccessible directory issue * Update merlin to use app.yaml configuration rather than its own configuration file * Docs/install changes (#383) Many modifications to documentation, including installation instructions and formatting fixes. * Maestro v 1.1.9dev1 Compatibility (#388) Maestro up to date compatibility Also unpacked maestro DAG to just use what we need, which should help reduce task message size and perhaps allow us to use other serializers in the future. * Bump certifi from 2022.9.24 to 2022.12.7 in /docs (#387) Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.9.24 to 2022.12.7. - [Release notes](https://github.com/certifi/python-certifi/releases) - [Commits](certifi/python-certifi@2022.09.24...2022.12.07) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Release 1.9.0 (#390) * Make CHANGELOG more concise * Updated Merlin version and added License to files missing it * Incremented python version for workflow test * fix merlinspec not being installed with pip and python 3.7 issues with celery * update changelog and version to 1.9.1 * fix a numpy issue on new numpy version release * modify changelog to show numpy fix * add version change to all files * re-add fix for numpy since it got removed in the last commit by accident * revert utils.py back to previous implementation * change dtype to python str type * Hotfix for merlin server unable to write config files. (#394) * Hotfix for merlin server unable to write config files. Change files to modules and copy files from new file modules Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Brian Gunnarson <49216024+bgunnar5@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Brian Gunnarson <brianfunnarson14@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
After building the merlin wheel and installing it, users are unable to initiate merlin server.
The cause was due to the fact that the configuration files that lived within the repository couldn't be access and copied over to the users local directory. To circumvent this issue, the problem was fixed by performing the same actions as how the information the "app.yaml" is copied over to the user.
The configuration files are added to merlin as a module and then modules are then loaded, read, and finally written to the user's local directory.