Merlin server configuration through commands

[ryannova]

A command interface to change/add/remove/edit the merlin server configuration, which can be utilized using the 'merlin server config' sub-command. Users should be encouraged to edit configurations through the command interface rather than through editing configuration files. This helps to maintain format consistency and schema of configuration files for merlin server. Note: Users might still need to changes configuration files if there are more fine tune adjustments needed for specific situations, or when options are not available within the 'merlin server config' command. Edits to configurations are only made to the configuration file located either in the main or local configuration directory. In order for these new changes to take effect(with some notable exceptions such as --add-user and --remove-user), merlin server must be restarted. A new command 'merlin server restart' has been added to provide convenience for users when needed to restart the container. A list of available configuration can be seen in the changelog below.

Added layer of security which requires users to authenticate before they are allowed access data within the merlin server containers. Authentication utilizes ACL(Access Control List) already implemented within redis to authenicate users. On merlin server initialization a "default" user with password listed in the redis.conf file is created.(This can be changed using the -p,--password, but defaults to 'merlin_password') A user with the current username of the user on the machine will also be generated. The user's password is generated and stored in the "redis.pass" file within the main merlin config directory. Note: This main password is only generated once and will not generated every time merlin server is initialized. If a new password is needed simply remove the "redis.pass" file and run 'merlin server init' again. Additional users can be added with 'merlin server config' command. A local configuration file called 'redis.users', which lists all users information and permissions are stored in the local configuration directory. Users added or removed using 'merlin server config' are applied to the current running container if there is one. If there are no running containers, changes will only apply the next time merlin server is started..

A refactoring of the code was also done in hopes to provide better clarity, transparency, readability of the code base. Notable changes is the addition of the server_utils.py file which stores classes and functions that are frequently used by other scripts. Constants are also stored in this file, which were previously located in server_config.py. A new server_commands.py script which contains function calls for each of the corresponding merlin server commands has also been created. Lastly a few minor edits and changes to configuration files were made. More detail can be seen in the Changelog below and details within this merge request.

Changelog:

• Create functional calls for each merlin server command
• Create server_command.py to store command calls.
• Added config subcommand to merlin server
• Added following flags to config subcommand
  • ipaddress (Set the binded ip address of the container)
  • port (Set the binded port of the container)
  • user (Set the main user file for container)
  • password (Set the main user password file for container)
  • add-user (Add a user to the container image [outputs an associated password file for user]) (ONLY CHANGE IN CONFIG FILE)
  • remote-user (Remove user from list of added users) (ONLY CHANGE IN CONFIG FILE)
  • directory (Set the directory of the merlin server container files)
  • snapshot-seconds (Set the number of seconds elapsed before snapshot change condition is checked)
  • snapshot-changes (Set snapshot change condition for a snapshot to be made)
  • snapshot-file (Set the database file that the snapshot will be written to)
  • append-mode (Set the append mode for redis)
  • append-file (Set the name of the append only file for redis)
• Added user_file to merlin server config
• Added pass_file to merlin server config
• add_user function add user to exisiting merlin server instance if one is running
• remove_user function remove user from merlin server instance if one is running
• Enable masteruser in redis config
• Enable requirepass in redis config
• Moved default constants from server_setup.py to server_config.py
• Added server_util.py file to store utility functions.
• Created RedisConfig class to interface with redis.conf file
• Created RedisUsers class to interface with redis.user file
• Better interface (ServerConfig, ContainerConfig, ContainerFormatConfig, and ProcessConfig) with getting configuration values from merlin server config file, with classes.
• Changed "default" user password to be "merlin_password" as default.
• Merlin server reapply users based on the saved redis.users config file.
• Default user can use the password stored in redis.pass file in main merlin configuration.

Test functionality:

• Authentication through redis-cli by providing file with password.
• Authentication through redis python library.

Things needed to test:

• Test on LC machines.
• Test using app.yaml main configuration files
• Test with merlin workflows and workers.

[koning]

What is the process to test this on LC?

[koning]

(merlin3_8) [host]:merlin server init  
[2022-06-09 15:02:13: INFO] Unable to find exisiting server configuration.
[2022-06-09 15:02:13: INFO] Creating default configuration in /<path>/.merlin/server/
[2022-06-09 15:02:13: INFO] Copying file singularity.yaml to configuration directory.
[2022-06-09 15:02:13: INFO] Copying file docker.yaml to configuration directory.
[2022-06-09 15:02:13: INFO] Copying file podman.yaml to configuration directory.
[2022-06-09 15:02:13: INFO] Copying file merlin_server.yaml to configuration directory.
[2022-06-09 15:02:13: INFO] Creating merlin server directory.
[2022-06-09 15:02:13: INFO] Creating password file for merlin server container.
[2022-06-09 15:02:13: INFO] User <user> created in user file for merlin server container
[2022-06-09 15:02:13: INFO] Fetching redis image from docker://redis
[2022-06-09 15:02:26: INFO] Copying default redis configuration file.
[2022-06-09 15:02:26: INFO] Merlin server initialization successful.
(merlin3_8) [host]:merlin server start
[2022-06-09 15:02:37: ERROR] cannot unpack non-iterable NoneType object

[ryannova]

I made some slight changes to the repo when I was testing this on the LC. Try to pull the lastest version and let me know if it works.

As for testing on the LC here are some tests that you can try.

Merlin Server basic functionality test

1. merlin server init
2. merlin server start
3. merlin server info
4. redis-cli
5. AUTH merlin_password
6. ACL list
7. exit
8. merlin server stop

Merlin Server restart functionality test

1. merlin server init
2. merlin server start
3. merlin server restart
4. merlin server info
5. redis-cli
6. AUTH merlin_password
7. ACL list
8. exit
9. merlin server stop

Note here that you can AUTH as your current user by using your current username on the machine and the password located in "merlin_server/redis.pass". You will need to open the file can copy the random string and use it in step 6 (AUTH )

Merlin workflow test

1. merlin server init
2. merlin server start
3. merlin server info
4. merlin example hello_samples
5. merlin run hello/hello_samples.yaml
6. merlin server stop

Somethings to note about your merlin config file in "~/.merlin/app.yaml". Make sure that you have either "localhost" or "127.0.0.1" set as the server and the user set to "default" and password set to "merlin_password" (For some reason its not able to read a redis.pass file which contains the password for the current user when placed in the app.yaml file. I'm still looking into it.)

I'm also working on putting something together that I can show working on LC and hopefully in another pull request I can make some changes to the current testing framework to allow for automating some of these tests.

[ryannova]

I've updated the changelogs to reflect the changes made in this pull request. @lucpeterson @koning Please take a look at it and if everything looks good, it is ready to be merged into develop.

[lucpeterson]

a few pending comments. but otherwise looks good!