diff --git a/CHANGELOG.md b/CHANGELOG.md index c5afb8a2a..dbae21b54 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - New shortcuts in specification file (sample_vector, sample_names, spec_original_template, spec_executed_run, spec_archived_copy) - Changed "default" user password to be "merlin_password" as default. - Update requirements to require redis 4.3.4 for acl user channel support +- Updated tutorial documentation to use merlin server over manual redis installation. - Added ssl to the broker and results backend server checks when "merlin info" is called ### Fixed diff --git a/docs/source/_static/custom.css b/docs/source/_static/custom.css index 367d8e1f2..72d82b073 100644 --- a/docs/source/_static/custom.css +++ b/docs/source/_static/custom.css @@ -18,3 +18,7 @@ div.highlight .copybtn:hover { div.highlight { position: relative; } +div.sphinxsidebar { + max-height: 100%; + overflow-y: auto; +} diff --git a/docs/source/conf.py b/docs/source/conf.py index d5cef3d64..2905856e8 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -149,7 +149,7 @@ # author, documentclass [howto, manual, or own class]). latex_documents = [ (master_doc, 'Merlin.tex', u'Merlin Documentation', - u'MLSI', 'manual'), + u'The Merlin Development Team', 'manual'), ] diff --git a/docs/source/index.rst b/docs/source/index.rst index 72469f968..3776466d3 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -81,4 +81,3 @@ Need help? `merlin@llnl.gov `_ spack merlin_developer docker - diff --git a/docs/source/merlin_commands.rst b/docs/source/merlin_commands.rst index 0a1a9f2bc..c15adb6d6 100644 --- a/docs/source/merlin_commands.rst +++ b/docs/source/merlin_commands.rst @@ -406,4 +406,36 @@ The only currently available option for ``--task_server`` is celery, which is th only one might get the signal. In this case, you can send it again. +Hosting Local Server (``merlin server``) +---------------------------------------- + +To create a local server for merlin to connect to. Merlin server creates and configures a server on the current directory. +This allows multiple instances of merlin server to exist for different studies or uses. + +The ``init`` subcommand initalizes a new instance of merlin server. + +The ``status`` subcommand checks to the status of the merlin server. + +The ``start`` subcommand starts the merlin server. + +The ``stop`` subcommand stops the merlin server. + +The ``restart`` subcommand performs stop command followed by a start command on the merlin server. + +The ``config`` subcommand edits configurations for the merlin server. There are multiple flags to allow for different configurations. + +- The ``-ip IPADDRESS, --ipaddress IPADDRESS`` option set the binded IP address for merlin server. +- The ``-p PORT, --port PORT`` option set the binded port for merlin server. +- The ``-pwd PASSWORD, --password PASSWORD`` option set the password file for merlin server. +- The ``--add-user USER PASSWORD`` option add a new user for merlin server. +- The ``--remove-user REMOVE_USER`` option remove an exisiting user from merlin server. +- The ``-d DIRECTORY, --directory DIRECTORY`` option set the working directory for merlin server. +- The ``-ss SNAPSHOT_SECONDS, --snapshot-seconds SNAPSHOT_SECONDS`` option set the number of seconds before each snapshot. +- The ``-sc SNAPSHOT_CHANGES, --snapshot-changes SNAPSHOT_CHANGES`` option set the number of database changes before each snapshot. +- The ``-sf SNAPSHOT_FILE, --snapshot-file SNAPSHOT_FILE`` option set the name of snapshots. +- The ``-am APPEND_MODE, --append-mode APPEND_MODE`` option set the appendonly mode. Options are always, everysec, no. +- The ``-af APPEND_FILE, --append-file APPEND_FILE`` option set the filename for server append/change file. + +More information can be found on :doc:`Merlin Server <./merlin_server>` + diff --git a/docs/source/modules/installation/installation.rst b/docs/source/modules/installation/installation.rst index 2eb1ac95d..c4d58192b 100644 --- a/docs/source/modules/installation/installation.rst +++ b/docs/source/modules/installation/installation.rst @@ -6,7 +6,7 @@ Installation * python3 >= python3.6 * pip3 * wget - * build tools (make, C/C++ compiler for local-redis) + * build tools (make, C/C++ compiler) * docker (required for :doc:`Module 4: Run a Real Simulation<../run_simulation/run_simulation>`) * file editor for docker config file editing @@ -17,9 +17,7 @@ Installation .. admonition:: You will learn * How to install merlin in a virtual environment using pip. - * How to install a local redis server. - * How to install merlin using docker (optional). - * How to start the docker containers, including redis (optional). + * How to install a container platform eg. singularity, docker, or podman. * How to configure merlin. * How to test/verify the installation. @@ -27,27 +25,21 @@ Installation :local: This section details the steps necessary to install merlin and its dependencies. -Merlin will then be configured and this configuration checked to ensure a proper installation. +Merlin will then be configured for the local machine and the configuration +will be checked to ensure a proper installation. Installing merlin ----------------- -A merlin installation is required for the subsequent modules of this tutorial. You can choose between the pip method or the docker method. Choose one or the other but -do not use both unless you are familiar with redis servers run locally and through docker. -**The pip method is recommended.** +A merlin installation is required for the subsequent modules of this tutorial. -Once merlin is installed, it requires servers to operate. -The pip section will inform you how to setup a -local redis server to use in merlin. An alternative method for setting up a -redis server can be found in the docker section. Only setup one redis server either -local-redis or docker-redis. -Your computer/organization may already have a redis server available, please check +Once merlin is installed, it requires servers to operate. While you are able to host your own servers, +we will use merlin's containerized servers in this tutorial. However, if you prefer to host your own servers +you can host a redis server that is accessible to your current machine. +Your computer/organization may already have a redis server available you can use, please check with your local system administrator. -Pip (recommended) -+++++++++++++++++ - Create a virtualenv using python3 to install merlin. .. code-block:: bash @@ -85,168 +77,111 @@ but leave the virtualenv activated for the subsequent steps. deactivate -redis local server -^^^^^^^^^^^^^^^^^^ +redis server +++++++++++++ A redis server is required for the celery results backend server, this same server -can also be used for the celery broker. This method will be called local-redis. +can also be used for the celery broker. We will be using merlin's containerized server +however we will need to download one of the supported container platforms avaliable. For +the purpose of this tutorial we will be using singularity. .. code-block:: bash + # Update and install singularity dependencies + apt-get update && apt-get install -y \ + build-essential \ + libssl-dev \ + uuid-dev \ + libgpgme11-dev \ + squashfs-tools \ + libseccomp-dev \ + pkg-config + + # Download dependency go + wget https://go.dev/dl/go1.18.1.linux-amd64.tar.gz + + # Extract go into local + tar -C /usr/local -xzf go1.18.1.linux-amd64.tar.gz + + # Remove go tar file + rm go1.18.1.linux-amd64.tar.gz + + # Update PATH to include go + export PATH=$PATH:/usr/local/go/bin + + # Download singularity + wget https://github.com/sylabs/singularity/releases/download/v3.9.9/singularity-ce-3.9.9.tar.gz + + # Extract singularity + tar -xzf singularity-ce-3.9.9.tar.gz + + # Configure and install singularity + cd singularity-ce-3.9.9 + ./mconfig && \ + make -C ./builddir && \ + sudo make -C ./builddir install - # Download redis - wget http://download.redis.io/releases/redis-6.0.5.tar.gz - - # Untar - tar xvf redis*.tar.gz - - # cd into redis dir - cd redis*/ - - # make redis - make - - # make test (~3.5 minutes) - make test - - -The redis server is started by calling the ``redis-server`` command located in -the src directory. -This should be run in a separate terminal in the top-level source -directory so the output can be examined. -The redis server will use the default ``redis.conf`` file in the top-level -redis directory. - -.. code:: bash - - # run redis with default config, server is at localhost port 6379 - ./src/redis-server & - -You can shutdown the local-redis server by using the ``redis-cli shutdown`` command -when you are done with the tutorial. +Configuring merlin +------------------ +Merlin requires a configuration script for the celery interface. +Run this configuration method to create the ``app.yaml`` +configuration file. .. code-block:: bash - #cd to redis directory - cd /redis*/ - ./src/redis-cli shutdown - - -Docker -++++++ - -Merlin and the servers required by merlin are all available as docker containers on dockerhub. Do not use this method if you have already set up a virtualenv through -the pip installation method. - -.. note:: - - When using the docker method the celery workers will run inside the - merlin container. This - means that any workflow tools that are also from docker containers must - be installed in, or - otherwise made available to, the merlin container. - + merlin config --broker redis -To run a merlin docker container with a docker redis server, cut -and paste the commands below into a new file called ``docker-compose.yml``. -This file can be placed anywhere in your filesystem but you may want to put it in -a directory ``merlin_docker_redis``. +The ``merlin config`` command above will create a file called ``app.yaml`` +in the ``~/.merlin`` directory. +If you are running a redis server locally then you are all set, look in the ``~/.merlin/app.yaml`` file +to see the configuration, it should look like the configuration below. -.. literalinclude:: ./docker-compose.yml +.. literalinclude:: ./app_local_redis.yaml :language: yaml -This file can then be run with the ``docker-compose`` command in same directory -as the ``docker-compose.yml`` file. - -.. code-block:: bash - - docker-compose up -d - -The ``volume`` option in the ``docker-compose.yml`` file -will link the local ``$HOME/merlinu`` directory to the ``/home/merlinu`` -directory in the container. - -Some aliases can be defined for convenience. -.. code-block:: bash +.. _Verifying installation: - # define some aliases for the merlin and celery commands (assuming Bourne shell) - alias merlin="docker exec my-merlin merlin" - alias celery="docker exec my-merlin celery" - alias python3="docker exec my-merlin python3" +Checking/Verifying installation +------------------------------- -When you are done with the containers you can stop them using ``docker-compose down``. -We will be using the containers in the subsequent modules so leave them running. +First launch the merlin server containers by using the ``merlin server`` commands .. code-block:: bash - docker-compose down + merlin server init + merlin server start -Any required python modules can be installed in the running ``my-merlin`` container -through ``docker exec``. When using docker-compose, these changes will persist -if you stop the containers with ``docker-compose down`` and restart them with -``docker-compose up -d``. +A subdirectory called ``merlin_server/`` will have been created in the current run directory. +This contains all of the proper configuration for the server containers merlin creates. +Configuration can be done through the ``merlin server config`` command, however users have +the flexibility to edit the files directly in the directory. Additionally an preconfigured ``app.yaml`` +file has been created in the ``merlin_server/`` subdirectory to utilize the merlin server +containers . To use it locally simply copy it to the run directory with a cp command. .. code-block:: bash - docker exec my-merlin pip3 install pandas faker - -Configuring merlin ------------------- - -Merlin configuration is slightly different between the pip and docker methods. -The fundamental differences include the app.yaml file location and the server name. + cp ./merlin_server/app.yaml . -Merlin requires a configuration script for the celery interface and optional -passwords for the redis server and encryption. Run this configuration method -to create the ``app.yaml`` configuration file. +You can also make this server container your main server configuration by replacing the one located in your home +directory. Make sure you make back-ups of your current app.yaml file in case you want to use your previous +configurations. Note: since merlin servers are created locally on your run directory you are allowed to create +multiple instances of merlin server with their unique configurations for different studies. Simply create different +directories for each study and run ``merlin server init`` in each directory to create an instance for each. .. code-block:: bash - merlin config --broker redis - -Pip -+++ - -The ``merlin config`` command above will create a file called ``app.yaml`` -in the ``~/.merlin`` directory. -If you are using local-redis then you are all set, look in the ``~/.merlin/app.yaml`` file -to see the configuration, it should look like the configuration below. - -.. literalinclude:: ./app_local_redis.yaml - :language: yaml - -Docker -++++++ - -If you are using the docker merlin with docker-redis server then the -``~/merlinu/.merlin/app.yaml`` will be created by the ``merlin config`` -command above. -This file must be edited to -add the server from the redis docker container my-redis. Change the ``server: localhost``, in both the -broker and backend config definitions, to ``server: my-redis``, the port will remain the same. - -.. note:: - You can use the docker redis server, instead of the local-redis server, - with the virtualenv installed merlin by using the local-redis - ``app.yaml`` file above. - -.. literalinclude:: ./app_docker_redis.yaml - :language: yaml - -.. _Verifying installation: - -Checking/Verifying installation -------------------------------- + mv ~/.merlin/app.yaml ~/.merlin/app.yaml.bak + cp ./merlin_server/app.yaml ~/.merlin/ The ``merlin info`` command will check that the configuration file is installed correctly, display the server configuration strings, and check server -access. This command works for both the pip and docker installed merlin. +access. .. code-block:: bash merlin info -If everything is set up correctly, you should see (assuming local-redis servers): +If everything is set up correctly, you should see: .. code-block:: bash diff --git a/docs/source/server/commands.rst b/docs/source/server/commands.rst index af3d2e419..dd8ca1b02 100644 --- a/docs/source/server/commands.rst +++ b/docs/source/server/commands.rst @@ -40,16 +40,18 @@ Stopping an exisiting Merlin Server (``merlin server stop``) Stop any exisiting container being managed and monitored by merlin server. Restarting a Merlin Server instance (``merlin server restart``) ------------------------------------------------------------- +--------------------------------------------------------------- Restarting an existing container that is being managed and monitored by merlin server. Configurating Merlin Server instance (``merlin server config``) ------------------------------------------------------------- +--------------------------------------------------------------- Place holder for information regarding merlin server config command Possible Flags -.. code:: none + +.. code-block:: none + -ip IPADDRESS, --ipaddress IPADDRESS Set the binded IP address for the merlin server container. (default: None) @@ -82,3 +84,4 @@ Possible Flags -af APPEND_FILE, --append-file APPEND_FILE Set append only filename for merlin server container. (default: None) +