LLNL · ryannova · Oct 28, 2022 · Oct 10, 2022 · Oct 10, 2022 · Oct 10, 2022
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -21,6 +21,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - New shortcuts in specification file (sample_vector, sample_names, spec_original_template, spec_executed_run, spec_archived_copy)
 - Changed "default" user password to be "merlin_password" as default.
 - Update requirements to require redis 4.3.4 for acl user channel support
+- Updated tutorial documentation to use merlin server over manual redis installation.
 - Added ssl to the broker and results backend server checks when "merlin info" is called
 
 ### Fixed

diff --git a/docs/source/_static/custom.css b/docs/source/_static/custom.css
@@ -18,3 +18,7 @@ div.highlight .copybtn:hover {
 div.highlight {
     position: relative;
 }
+div.sphinxsidebar {
+    max-height: 100%;
+    overflow-y: auto;
+}
diff --git a/docs/source/conf.py b/docs/source/conf.py
@@ -149,7 +149,7 @@
 #  author, documentclass [howto, manual, or own class]).
 latex_documents = [
     (master_doc, 'Merlin.tex', u'Merlin Documentation',
-     u'MLSI', 'manual'),
+     u'The Merlin Development Team', 'manual'),
 ]
 
 

diff --git a/docs/source/index.rst b/docs/source/index.rst
@@ -81,4 +81,3 @@ Need help? `merlin@llnl.gov <merlin@llnl.gov>`_
    spack
    merlin_developer
    docker
-
diff --git a/docs/source/merlin_commands.rst b/docs/source/merlin_commands.rst
@@ -406,4 +406,36 @@ The only currently available option for ``--task_server`` is celery, which is th
    only one might get the signal. In this case, you can send it
    again.
 
+Hosting Local Server (``merlin server``)
+----------------------------------------
+
+To create a local server for merlin to connect to. Merlin server creates and configures a server on the current directory.
+This allows multiple instances of merlin server to exist for different studies or uses.
+
+The ``init`` subcommand initalizes a new instance of merlin server.
+
+The ``status`` subcommand checks to the status of the merlin server.
+
+The ``start`` subcommand starts the merlin server.
+
+The ``stop`` subcommand stops the merlin server.
+
+The ``restart`` subcommand performs stop command followed by a start command on the merlin server.
+
+The ``config`` subcommand edits configurations for the merlin server. There are multiple flags to allow for different configurations.
+
+- The ``-ip IPADDRESS, --ipaddress IPADDRESS`` option set the binded IP address for merlin server.
+- The ``-p PORT, --port PORT`` option set the binded port for merlin server.
+- The ``-pwd PASSWORD, --password PASSWORD`` option set the password file for merlin server.
+- The ``--add-user USER PASSWORD`` option add a new user for merlin server.
+- The ``--remove-user REMOVE_USER`` option remove an exisiting user from merlin server.
+- The ``-d DIRECTORY, --directory DIRECTORY`` option set the working directory for merlin server.
+- The ``-ss SNAPSHOT_SECONDS, --snapshot-seconds SNAPSHOT_SECONDS`` option set the number of seconds before each snapshot.
+- The ``-sc SNAPSHOT_CHANGES, --snapshot-changes SNAPSHOT_CHANGES`` option set the number of database changes before each snapshot.
+- The ``-sf SNAPSHOT_FILE, --snapshot-file SNAPSHOT_FILE`` option set the name of snapshots.
+- The ``-am APPEND_MODE, --append-mode APPEND_MODE`` option set the appendonly mode. Options are always, everysec, no.
+- The ``-af APPEND_FILE, --append-file APPEND_FILE`` option set the filename for server append/change file.
+
+More information can be found on :doc:`Merlin Server <./merlin_server>`
+
 
diff --git a/docs/source/modules/installation/installation.rst b/docs/source/modules/installation/installation.rst
@@ -6,7 +6,7 @@ Installation
   * python3 >= python3.6
   * pip3
   * wget
-  * build tools (make, C/C++ compiler for local-redis)
+  * build tools (make, C/C++ compiler)
   * docker (required for :doc:`Module 4: Run a Real Simulation<../run_simulation/run_simulation>`)
   * file editor for docker config file editing
 
@@ -17,37 +17,29 @@ Installation
 .. admonition:: You will learn
 
   * How to install merlin in a virtual environment using pip.
-  * How to install a local redis server.
-  * How to install merlin using docker (optional).
-  * How to start the docker containers, including redis (optional).
+  * How to install a container platform eg. singularity, docker, or podman.
   * How to configure merlin.
   * How to test/verify the installation.
 
 .. contents:: Table of Contents:
   :local:
 
 This section details the steps necessary to install merlin and its dependencies.
-Merlin will then be configured and this configuration checked to ensure a proper installation.
+Merlin will then be configured for the local machine and the configuration 
+will be checked to ensure a proper installation.
 
 
 Installing merlin
 -----------------
 
-A merlin installation is required for the subsequent modules of this tutorial. You can choose between the pip method or the docker method. Choose one or the other but
-do not use both unless you are familiar with redis servers run locally and through docker.
-**The pip method is recommended.**
+A merlin installation is required for the subsequent modules of this tutorial.
 
-Once merlin is installed, it requires servers to operate.
-The pip section will inform you how to setup a
-local redis server to use in merlin.  An alternative method for setting up a
-redis server can be found in the docker section. Only setup one redis server either
-local-redis or docker-redis.
-Your computer/organization  may already have a redis server available, please check
+Once merlin is installed, it requires servers to operate. While you are able to host your own servers,
+we will use merlin's containerized servers in this tutorial. However, if you prefer to host your own servers
+you can host a redis server that is accessible to your current machine.
+Your computer/organization may already have a redis server available you can use, please check
 with your local system administrator.
 
-Pip (recommended)
-+++++++++++++++++
-
 Create a virtualenv using python3 to install merlin.
 
 .. code-block:: bash
@@ -85,168 +77,111 @@ but leave the virtualenv activated for the subsequent steps.
   deactivate
 
 
-redis local server
-^^^^^^^^^^^^^^^^^^
+redis server
+++++++++++++
 
 A redis server is required for the celery results backend server, this same server
-can also be used for the celery broker. This method will be called local-redis.
+can also be used for the celery broker. We will be using merlin's containerized server
+however we will need to download one of the supported container platforms avaliable. For 
+the purpose of this tutorial we will be using singularity.
 
 .. code-block:: bash
+  # Update and install singularity dependencies
+  apt-get update && apt-get install -y \
+    build-essential \
+    libssl-dev \
+    uuid-dev \
+    libgpgme11-dev \
+    squashfs-tools \
+    libseccomp-dev \
+    pkg-config
+
+  # Download dependency go
+  wget https://go.dev/dl/go1.18.1.linux-amd64.tar.gz
+
+  # Extract go into local
+  tar -C /usr/local -xzf go1.18.1.linux-amd64.tar.gz
+
+  # Remove go tar file
+  rm go1.18.1.linux-amd64.tar.gz
+
+  # Update PATH to include go
+  export PATH=$PATH:/usr/local/go/bin
+
+  # Download singularity
+  wget https://github.com/sylabs/singularity/releases/download/v3.9.9/singularity-ce-3.9.9.tar.gz
+
+  # Extract singularity
+  tar -xzf singularity-ce-3.9.9.tar.gz
+
+  # Configure and install singularity
+  cd singularity-ce-3.9.9
+  ./mconfig && \
+    make -C ./builddir && \
+    sudo make -C ./builddir install
 
-  # Download redis
-  wget http://download.redis.io/releases/redis-6.0.5.tar.gz
-
-  # Untar
-  tar xvf redis*.tar.gz
-
-  # cd into redis dir
-  cd redis*/
-
-  # make redis
-  make
-
-  # make test (~3.5 minutes)
-  make test
-
-
-The redis server is started by calling the ``redis-server`` command located in
-the src directory.
-This should be run in a separate terminal in the top-level source
-directory so the output can be examined.
-The redis server will use the default ``redis.conf`` file in the top-level
-redis directory.
-
-.. code:: bash
-
-  # run redis with default config, server is at localhost port 6379
-  ./src/redis-server &
-
-You can shutdown the local-redis server by using the ``redis-cli shutdown`` command
-when you are done with the tutorial.
+Configuring merlin
+------------------
+Merlin requires a configuration script for the celery interface. 
+Run this configuration method to create the ``app.yaml`` 
+configuration file.
 
 .. code-block:: bash
 
-  #cd to redis directory
-  cd <path to>/redis*/
-  ./src/redis-cli shutdown
-
-
-Docker
-++++++
-
-Merlin and the servers required by merlin are all available as docker containers on dockerhub. Do not use this method if you have already set up a virtualenv through
-the pip installation method.
-
-.. note::
-
-  When using the docker method the celery workers will run inside the
-  merlin container. This
-  means that any workflow tools that are also from docker containers must
-  be installed in, or
-  otherwise made available to, the merlin container.
-
+  merlin config --broker redis
 
-To run a merlin docker container with a docker redis server, cut
-and paste the commands below into a new file called ``docker-compose.yml``.
-This file can be placed anywhere in your filesystem but you may want to put it in
-a directory ``merlin_docker_redis``.
+The ``merlin config`` command above will create a file called ``app.yaml``
+in the ``~/.merlin`` directory.
+If you are running a redis server locally then you are all set, look in the ``~/.merlin/app.yaml`` file
+to see the configuration, it should look like the configuration below.
 
-.. literalinclude:: ./docker-compose.yml
+.. literalinclude:: ./app_local_redis.yaml
    :language: yaml
 
-This file can then be run with the ``docker-compose`` command in same directory
-as the ``docker-compose.yml`` file.
-
-.. code-block:: bash
-
-  docker-compose up -d
-
-The ``volume`` option in the ``docker-compose.yml`` file
-will link the local ``$HOME/merlinu`` directory to the ``/home/merlinu``
-directory in the container.
-
-Some aliases can be defined for convenience.
 
-.. code-block:: bash
+.. _Verifying installation:
 
-  # define some aliases for the merlin and celery commands (assuming Bourne shell)
-  alias merlin="docker exec my-merlin merlin"
-  alias celery="docker exec my-merlin celery"
-  alias python3="docker exec my-merlin python3"
+Checking/Verifying installation
+-------------------------------
 
-When you are done with the containers you can stop them using ``docker-compose down``.
-We will be using the containers in the subsequent modules so leave them running.
+First launch the merlin server containers by using the ``merlin server`` commands
 
 .. code-block:: bash
 
-  docker-compose down
+  merlin server init
+  merlin server start
 
-Any required python modules can be installed in the running ``my-merlin`` container
-through ``docker exec``. When using docker-compose, these changes will persist
-if you stop the containers with ``docker-compose down`` and restart them with 
-``docker-compose up -d``.
+A subdirectory called ``merlin_server/`` will have been created in the current run directory.
+This contains all of the proper configuration for the server containers merlin creates.
+Configuration can be done through the ``merlin server config`` command, however users have
+the flexibility to edit the files directly in the directory. Additionally an preconfigured ``app.yaml``
+file has been created in the ``merlin_server/`` subdirectory to utilize the merlin server 
+containers . To use it locally simply copy it to the run directory with a cp command.
 
 .. code-block:: bash
 
-     docker exec my-merlin pip3 install pandas faker
-
-Configuring merlin
-------------------
-
-Merlin configuration is slightly different between the pip and docker methods.
-The fundamental differences include the app.yaml file location and the server name.
+  cp ./merlin_server/app.yaml .
 
-Merlin requires a configuration script for the celery interface and optional
-passwords for the redis server and encryption. Run this configuration method
-to create the ``app.yaml`` configuration file.
+You can also make this server container your main server configuration by replacing the one located in your home 
+directory. Make sure you make back-ups of your current app.yaml file in case you want to use your previous
+configurations. Note: since merlin servers are created locally on your run directory you are allowed to create 
+multiple instances of merlin server with their unique configurations for different studies. Simply create different
+directories for each study and run ``merlin server init`` in each directory to create an instance for each.
 
 .. code-block:: bash
 
-  merlin config --broker redis
-
-Pip
-+++
-
-The ``merlin config`` command above will create a file called ``app.yaml``
-in the ``~/.merlin`` directory.
-If you are using local-redis then you are all set, look in the ``~/.merlin/app.yaml`` file
-to see the configuration, it should look like the configuration below.
-
-.. literalinclude:: ./app_local_redis.yaml
-   :language: yaml
-
-Docker
-++++++
-
-If you are using the docker merlin with docker-redis server then the
-``~/merlinu/.merlin/app.yaml`` will be created by the ``merlin config``
-command above.
-This file must be edited to
-add the server from the redis docker container my-redis. Change the ``server: localhost``, in both the
-broker and backend config definitions, to ``server: my-redis``, the port will remain the same.
-
-.. note::
-  You can use the docker redis server, instead of the local-redis server,
-  with the virtualenv installed merlin by using the local-redis
-  ``app.yaml`` file above.
-
-.. literalinclude:: ./app_docker_redis.yaml
-   :language: yaml
-
-.. _Verifying installation:
-
-Checking/Verifying installation
--------------------------------
+  mv ~/.merlin/app.yaml ~/.merlin/app.yaml.bak
+  cp ./merlin_server/app.yaml ~/.merlin/
 
 The ``merlin info`` command will check that the configuration file is
 installed correctly, display the server configuration strings, and check server
-access. This command works for both the pip and docker installed merlin.
+access.
 
 .. code-block:: bash
 
   merlin info
 
-If everything is set up correctly, you should see (assuming local-redis servers):
+If everything is set up correctly, you should see:
 
 .. code-block:: bash
 

diff --git a/docs/source/server/commands.rst b/docs/source/server/commands.rst
@@ -40,16 +40,18 @@ Stopping an exisiting Merlin Server (``merlin server stop``)
 Stop any exisiting container being managed and monitored by merlin server.
 
 Restarting a Merlin Server instance (``merlin server restart``)
-------------------------------------------------------------
+---------------------------------------------------------------
 
 Restarting an existing container that is being managed and monitored by merlin server.
 
 Configurating Merlin Server instance (``merlin server config``)
-------------------------------------------------------------
+---------------------------------------------------------------
 Place holder for information regarding merlin server config command
 
 Possible Flags
-.. code:: none
+
+.. code-block:: none
+
     -ip IPADDRESS, --ipaddress IPADDRESS
                             Set the binded IP address for the merlin server
                             container. (default: None)
@@ -82,3 +84,4 @@ Possible Flags
     -af APPEND_FILE, --append-file APPEND_FILE
                             Set append only filename for merlin server container.
                             (default: None)
+
Original file line number	Diff line number	Diff line change
Expand Up		@@ -81,4 +81,3 @@ Need help? `merlin@llnl.gov <merlin@llnl.gov>`_
		spack
		merlin_developer
		docker