From 609876d205c98bb76733c0e7a3118a95940b64a5 Mon Sep 17 00:00:00 2001 From: Carl Simon Adorf Date: Tue, 20 Oct 2020 09:35:17 +0200 Subject: [PATCH 01/13] Revise "Get started" page. - Overhaul of page structure and installation flow. - Incorporate content from detailed installation page. --- docs/source/conf.py | 2 +- docs/source/intro/get_started.rst | 517 +++++++++++++++++++++++------- 2 files changed, 399 insertions(+), 120 deletions(-) diff --git a/docs/source/conf.py b/docs/source/conf.py index f9c15cdd80..633078f48f 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -47,7 +47,7 @@ ] ipython_mplbackend = '' copybutton_selector = 'div:not(.no-copy)>div.highlight pre' -copybutton_prompt_text = r'>>> |\.\.\. |\$ |In \[\d*\]: | {2,5}\.\.\.: | {5,8}: ' +copybutton_prompt_text = r'>>> |\.\.\. |(?:\(.*\) )?\$ |In \[\d*\]: | {2,5}\.\.\.: | {5,8}: ' copybutton_prompt_is_regexp = True todo_include_todos = False diff --git a/docs/source/intro/get_started.rst b/docs/source/intro/get_started.rst index 0bf71e3386..f895796999 100644 --- a/docs/source/intro/get_started.rst +++ b/docs/source/intro/get_started.rst @@ -1,7 +1,7 @@ .. _intro:get_started: **************** -Getting Started +Getting started **************** An AiiDA installation consists of three core components (plus any external codes you wish to run): @@ -10,213 +10,492 @@ An AiiDA installation consists of three core components (plus any external codes * |PostgreSQL|: The service that manages the database that AiiDA uses to store data. * |RabbitMQ|: The message broker used for communication within AiiDA. +Setup +===== + There are multiple routes to setting up a working AiiDA environment. -These are shown below, followed by a recommended "quick-install" route on your local computer. +Which of those is optimal depends on your environment and use case. +If you are unsure, use the :ref:`system-wide installation ` method. .. panels:: :body: bg-light :footer: bg-light border-0 - :fa:`desktop,mr-1` **Direct (Bare Metal)** + :fa:`desktop,mr-1` **System-wide installation** - *Install software directly into your local root directory.* + .. link-button:: intro:get_started:system-wide-install + :type: ref + :text: Install all software directly on your workstation or laptop. + :classes: stretched-link btn-link - The prerequisite software can be installed using most package managers, including: apt, Homebrew, MacPorts, Gentoo and Windows Subsystem for Linux. + Install the prerequisite services using standard package managers (apt, homebrew, etc.) with administrative privileges. - +++ + --------------- - :link-badge:`intro:install:prerequisites,Prerequisites install,ref,badge-primary text-white` - :link-badge:`intro:install:aiida-core,aiida-core install,ref,badge-primary text-white` + :fa:`folder,mr-1` **Installation into Conda environment** - --------------- + .. link-button:: intro:get_started:conda-install + :type: ref + :text: Install all software into an isolated conda environment. + :classes: stretched-link btn-link - :fa:`folder,mr-1` **Virtual Environment** + This method does not require administrative privileges, but involves manual management of start-up and shut-down of services. - *Install software into an isolated directory on your machine.* + --------------- - Environment managers such as `Conda `__, `pipenv `__ and `venv `__ create isolated environments, allowing for installation of multiple versions of software on the same machine. - It is advised that you install ``aiida-core`` into one of these managed environments. + :fa:`cube,mr-1` **Run via docker container** - +++ + .. link-button:: intro:get_started:docker + :type: ref + :text: Run AiiDA and prerequisite services as a single docker container. + :classes: stretched-link btn-link - :link-badge:`intro:install:virtual_environments,Environments Tutorial,ref,badge-primary text-white` - :link-badge:`https://anaconda.org/conda-forge/aiida-core,aiida-core on Conda,url,badge-primary text-white` + Does not require the separate installation of prerequisite services. + Especially well-suited to get directly started on the **tutorials**. --------------- - :fa:`cube,mr-1` **Containers** + :fa:`cloud,mr-1` **Run via virtual machine** - *Use a pre-made image of all the required software.* + .. link-button:: intro:get_started:virtual-machine + :type: ref + :text: Use a virtual machine with all the required software pre-installed. + :classes: stretched-link btn-link - AiiDA maintains a `Docker `__ image, which is particularly useful for learning and testing purposes. - It is a great way to quickly get started on the tutorials. + `Materials Cloud `__ provides both downloadable and web based VMs, + also incorporating pre-installed Materials Science codes. - +++ - :link-badge:`intro:install:docker,Docker Tutorial,ref,badge-primary text-white` - :link-badge:`https://hub.docker.com/r/aiidateam/aiida-core,aiida-core on DockerHub,url,badge-primary text-white` +.. _intro:get_started:system-wide-install: - --------------- +System-wide installation +------------------------ - :fa:`cloud,mr-1` **Virtual Machines** +The system-wide installation will install the prerequisite services (PostgreSQL and RabbitMQ) via standard package managers such that their startup and shut-down is largely managed by the operating system. +The AiiDA (core) Python package is then installed either with Conda or pip. - *Use a pre-made machine with all the required software.* +This is the *recommended* installation method to setup AiiDA on a personal laptop or workstation for the majority of users. - `Materials Cloud `__ provides both downloadable and web based VMs, - also incorporating pre-installed Materials Science codes. +.. panels:: + :container: container-lg pb-3 + :column: col-lg-12 p-2 + + **Install prerequisite services** + + AiiDA is designed to run on `Unix `_ operating systems and requires a `bash `_ or `zsh `_ shell, and Python >= 3.6. + + .. tabbed:: Ubuntu + + *AiiDA is tested on Ubuntu versions 14.04, 16.04, and 18.04.* + + Open a terminal and execute: + + .. code-block:: console + + $ sudo apt install \ + git python3-dev python3-pip \ + postgresql postgresql-server-dev-all postgresql-client rabbitmq-server + + .. tabbed:: MacOS X (Homebrew) + + The recommended installation method for Mac OS X is to use `Homebrew `__. + + #. Follow `this guide `__ to install Homebrew on your system if not installed yet. + + #. Open a terminal and execute: + + .. code-block:: console + + $ brew install postgresql rabbitmq git python + $ brew services start postgresql + $ brew services start rabbitmq + + .. tabbed:: Windows Subsystem for Linux + + *The following instructions are for setting up AiiDA on WSL 2 in combination with Ubuntu.* + + #. The `Windows native RabbitMQ `_ should be installed and started. + + #. Install Python and PostgreSQL: + + .. code-block:: console + + $ sudo apt-get install \ + postgresql postgresql-server-dev-all postgresql-client \ + git python3-dev python-pip + $ sudo service postgresql start + + .. dropdown:: How to setup WSL to automatically start services after system boot. + + Create a ``.sh`` file with the lines above, but *without* ``sudo``. + Make the file executeable, i.e., type: + + .. code-block:: console + + $ chmod +x /path/to/file.sh + $ sudo visudo + + And add the line: + + .. code-block:: sh + + ALL=(root) NOPASSWD: /path/to/file.sh + + Replacing ```` with your Ubuntu username. + This will allow you to run *only* this specific ``.sh`` file with ``root`` access (without password), without lowering security on the rest of your system. + + .. dropdown:: :fa:`wrench` How to resolve a timezone issue on Ubuntu 18.04. + + There is a `known issue `_ in WSL Ubuntu 18.04 where the timezone is not configured correctly out-of-the-box, which may cause a problem for the database. + The following command can be used to re-configure the time zone: + + .. code-block:: console + + $ sudo dpkg-reconfigure tzdata + + .. tabbed:: Other + + #. Install RabbitMQ following the `instructions applicable to your system `__. + #. Install PostgreSQL following the `instructions applicable to your system `__. + + .. hint:: + + Alternatively use the :ref:`pure conda installation method `. + + --- + + **Install AiiDA (core)** + + .. tabbed:: Conda + + *Install the aiida-core package in a Conda environment.* + + #. Make sure that conda is installed, e.g., by following `the instructions on installing Miniconda `__. + + #. Open a terminal and execute: + + .. code-block:: console + + $ conda create -n aiida -c conda-forge aiida-core + $ conda activate aiida + (aiida) $ reentry scan + + .. tabbed:: pip + venv + + *Install the aiida-core package from PyPI into a virtual environment.* + + Open a terminal and execute: + + .. code-block:: console + + $ python -m pip venv ~/envs/aiida + $ source ~/envs/aiida/bin/activate + (aiida) $ pip install aiida-core + (aiida) $ reentry scan + + .. dropdown:: :fa:`plus-circle` Installation extras + + There are additional optional packages that you may want to install, which are grouped in the following categories: + + * ``atomic_tools``: packages that allow importing and manipulating crystal structure from various formats + * ``ssh_kerberos``: adds support for ssh transport authentication through Kerberos + * ``REST``: allows a REST server to be ran locally to serve AiiDA data + * ``docs``: tools to build the documentation + * ``notebook``: jupyter notebook - to allow it to import AiiDA modules + * ``tests``: python modules required to run the automatic unit tests + * ``pre-commit``: pre-commit tools required for developers to enable automatic code linting and formatting - +++ + In order to install any of these package groups, simply append them as a comma separated list in the ``pip`` install command, for example: - :link-badge:`https://materialscloud.org/quantum-mobile,Quantum Mobile,url,badge-primary text-white` - :link-badge:`https://aiidalab.materialscloud.org,AiiDA lab,url,badge-primary text-white` + .. code-block:: console -.. _intro:quick_start: + (aiida) $ pip install aiida-core[atomic_tools,docs] -Quick Start -=========== + .. dropdown:: :fa:`wrench` Kerberos on Ubuntu -Here we first provide a simple approach for installation and setup on your local computer. + If you are installing the optional ``ssh_kerberos`` and you are on Ubuntu you might encounter an error related to the ``gss`` package. + To fix this you need to install the ``libffi-dev`` and ``libkrb5-dev`` packages: -Install Software ----------------- + .. code-block:: console + + $ sudo apt-get install libffi-dev libkrb5-dev + + + + .. tabbed:: From source + + *Install the aiida-core package directly from the cloned repository.* + + Open a terminal and execute: + + .. code-block:: console + + $ git clone https://github.com/aiidateam/aiida-core.git + $ cd aiida-core/ + $ python -m pip venv ~/envs/aiida + $ source ~/envs/aiida/bin/activate + (aiida) $ pip install . + (aiida) $ reentry scan + + --- + + **Start verdi daemons** + + Start the verdi daemon(s) that are used to run AiiDA workflows. + + .. code-block:: console + + (aiida) $ verdi daemon start 2 + + .. important:: + + The verdi daemon(s) must be restarted after a system reboot. + + .. hint:: + + Do not start more daemons then there are physical processors on your system. + + --- + + **Setup profile** + + Next, set up an AiiDA configuration profile and related data storage, with the ``verdi quicksetup`` command. + + .. code-block:: console + + (aiida) $ verdi quicksetup + Info: enter "?" for help + Info: enter "!" to ignore the default and set no value + Profile name: me + Email Address (for sharing data): me@user.com + First name: my + Last name: name + Institution: where-i-work + + --- + + **Check setup** + + To check that everything is set up correctly, execute: + + .. code-block:: console + + (aiida) $ verdi status + ✓ config dir: /home/ubuntu/.aiida + ✓ profile: On profile me + ✓ repository: /home/ubuntu/.aiida/repository/me + ✓ postgres: Connected as aiida_qs_ubuntu_c6a4f69d255fbe9cdb7385dcdcf3c050@localhost:5432 + ✓ rabbitmq: Connected as amqp://127.0.0.1?heartbeat=600 + ✓ daemon: Daemon is running as PID 16430 since 2020-04-29 12:17:31 + + At this point you should now have a working AiiDA environment, from which you can add and retrieve data. + + .. admonition:: Missing a checkmark or ecountered some other issue? + :class: attention title-icon-troubleshoot + + :ref:`See the troubleshooting section `. + + .. link-button:: intro:get_started:next + :type: ref + :text: What's next? + :classes: btn-outline-primary btn-block font-weight-bold + +.. _intro:get_started:conda-install: + +Installation into Conda environment +----------------------------------- + +This installation route installs all necessary software -- including the prerequisite services PostgreSQL and RabbitMQ -- into a Conda environment. +This is the recommended method for users on shared systems and systems where the user has no administrative privileges. + +.. important:: + + This installation method installs **all** software into a conda environment, including PostgreSQL and RabbitMQ. + See the :ref:`system-wide installation ` to use Conda only to install the AiiDA (core) Python package. .. panels:: - :column: col-lg-6 col-md-6 col-sm-12 col-xs-12 p-2 + :container: container-lg pb-3 + :column: col-lg-12 p-2 - :fa:`download,mr-1` **Install with conda** + **Install prerequisite services + AiiDA (core)** - .. parsed-literal:: + .. code-block:: console - conda create -n aiida -c conda-forge aiida-core=\ |release|\ aiida-core.services=\ |release| - conda activate aiida - reentry scan + $ conda create -n aiida -c conda-forge aiida-core aiida-core.services + $ conda activate aiida + (aiida) $ reentry scan - `Conda `__ provides a cross-platform package management system, from which we can install all the basic components of the AiiDA infrastructure in an isolated environment. + --- - ---------------------------------------------- + **Start-up services and initialize data storage** - :fa:`download,mr-1` **Install with pip** + Before working with AiiDA, you must first initialize a database storage area on disk. - .. parsed-literal:: + .. code-block:: console - pip install aiida-core - reentry scan + (aiida) $ initdb -D mylocal_db - ``aiida-core`` can be installed from `PyPi `__. - It is strongly recommended that you install into a :ref:`virtual environment `. - You will then need to install |PostgreSQL| and |RabbitMQ| depending on your operating system. + This *database cluster* may contain a collection of databases (one per profile) that is managed by a single running server process. + We start this process with: - :link-badge:`intro:install:prerequisites,Install prerequisites,ref,badge-primary text-white` + .. code-block:: console -Initialise Data Storage ------------------------- + (aiida) $ pg_ctl -D mylocal_db -l logfile start -Before working with AiiDA, you must first initialize a database storage area on disk. + .. admonition:: Further Reading + :class: seealso title-icon-read-more -.. code-block:: console + - `Creating a Database Cluster `__. + - `Starting the Database Server `__. - $ initdb -D mylocal_db + Then, start the RabbitMQ server: + .. code-block:: console -This *database cluster* may contain a collection of databases (one per profile) that is managed by a single running server process. -We start this process with: + (aiida) $ rabbitmq-server -detached -.. code-block:: console + Finally, start the AiiDA daemon(s): - $ pg_ctl -D mylocal_db -l logfile start + .. code-block:: console -.. admonition:: Further Reading - :class: seealso title-icon-read-more + (aiida) $ verdi daemon start 2 - - `Creating a Database Cluster `__. - - `Starting the Database Server `__. + .. important:: -Next, we set up an AiiDA configuration profile and related data storage, with the `quicksetup` command. + The verdi daemon(s) must be restarted after a system reboot. -.. code-block:: console + .. hint:: - $ verdi quicksetup - Info: enter "?" for help - Info: enter "!" to ignore the default and set no value - Profile name: me - Email Address (for sharing data): me@user.com - First name: my - Last name: name - Institution: where-i-work + Do not start more daemons then there are physical processors on your system. -At this point you now have a working AiiDA environment, from which you can add and retrieve data. + --- -.. admonition:: Tab Completion - :class: tip title-icon-lightbulb + **Setup profile** - Enable tab completion of ``verdi`` commands in the terminal with: + Next, set up an AiiDA configuration profile and related data storage, with the ``verdi quicksetup`` command. - .. code-block:: console + .. code-block:: console - $ eval "$(_VERDI_COMPLETE=source verdi)" + (aiida) $ verdi quicksetup + Info: enter "?" for help + Info: enter "!" to ignore the default and set no value + Profile name: me + Email Address (for sharing data): me@user.com + First name: my + Last name: name + Institution: where-i-work - :link-badge:`how-to:installation:configure:tab-completion,Read More,ref,badge-primary text-white` + --- -Start Computation Services --------------------------- + **Check setup** -In order to run computations, some additional steps are required to start the services that manage these background processes. -The |RabbitMQ| service is used, to manage communication between processes and retain process states, even after restarting your computer: + To check that everything is set up correctly, execute: -.. code-block:: console + .. code-block:: console - $ rabbitmq-server -detached + (aiida) $ verdi status + ✓ config dir: /home/ubuntu/.aiida + ✓ profile: On profile me + ✓ repository: /home/ubuntu/.aiida/repository/me + ✓ postgres: Connected as aiida_qs_ubuntu_c6a4f69d255fbe9cdb7385dcdcf3c050@localhost:5432 + ✓ rabbitmq: Connected as amqp://127.0.0.1?heartbeat=600 + ✓ daemon: Daemon is running as PID 16430 since 2020-04-29 12:17:31 -We then start one or more "daemon" processes, which handle the execution and monitoring of all submitted computations. + At this point you now have a working AiiDA environment, from which you can add and retrieve data. -.. code-block:: console + .. admonition:: Missing a checkmark or ecountered some other issue? + :class: attention title-icon-troubleshoot - $ verdi daemon start 2 + :ref:`See the troubleshooting section `. -Finally, to check that all services are running as expected use: + .. link-button:: intro:get_started:next + :type: ref + :text: What's next? + :classes: btn-outline-primary btn-block font-weight-bold -.. code-block:: console + --- - $ verdi status - ✓ config dir: /home/ubuntu/.aiida - ✓ profile: On profile me - ✓ repository: /home/ubuntu/.aiida/repository/me - ✓ postgres: Connected as aiida_qs_ubuntu_c6a4f69d255fbe9cdb7385dcdcf3c050@localhost:5432 - ✓ rabbitmq: Connected as amqp://127.0.0.1?heartbeat=600 - ✓ daemon: Daemon is running as PID 16430 since 2020-04-29 12:17:31 + **Shut-down services** -Awesome! You now have a fully operational installation from which to take the next steps! + After finishing with your aiida session, particularly if switching between profiles, you may wish to power down the services: -Stopping Services ------------------ + .. code-block:: console -After finishing with your aiida session, particularly if switching between profiles, you may wish to power down the services: + (aiida) $ verdi daemon stop + (aiida) $ pg_ctl stop -.. code-block:: console +.. _intro:get_started:docker: - $ verdi daemon stop - $ pg_ctl stop +Run AiiDA via a Docker image +---------------------------- -Any computations that are still running at this point, will be picked up next time the services are started. +.. _intro:get_started:virtual-machine: +The AiiDA team maintains a `Docker `__ image on `Docker Hub `__. +This image contains a fully pre-configured AiiDA environment which makes it particularly useful for learning and testing purposes. -.. admonition:: Having problems? - :class: attention title-icon-troubleshoot +.. panels:: + :container: container-lg pb-3 + :column: col-lg-12 p-2 - :ref:`See the troubleshooting section `. + **Start container** -.. admonition:: In-depth instructions - :class: seealso title-icon-read-more + First, pull the image: - For more ways to install AiiDA, :ref:`check the detailed installation section `. + .. parsed-literal:: - For more detailed instructions on configuring AiiDA, :ref:`see the configuration how-to `. + $ docker pull aiidateam/aiida-core:\ |release|\ + + Then start the container with: + + .. parsed-literal:: + + $ docker run -d --name aiida-container aiidateam/aiida-core:\ |release|\ + + --- -What Next? ----------- + **Check setup** -If you are new to AiiDA, go through the :ref:`Basic Tutorial `, + The default profile is created under the ``aiida`` user, so to execute commands you must add the ``--user aiida`` option. + + For example, to check the verdi status, execute: + + .. code-block:: console + + $ docker exec -t --user aiida aiida-container /bin/bash -l -c 'verdi status' + ✓ config dir: /home/aiida/.aiida + ✓ profile: On profile default + ✓ repository: /home/aiida/.aiida/repository/default + ✓ postgres: Connected as aiida_qs_aiida_477d3dfc78a2042156110cb00ae3618f@localhost:5432 + ✓ rabbitmq: Connected as amqp://127.0.0.1?heartbeat=600 + ✓ daemon: Daemon is running as PID 1795 since 2020-05-20 02:54:00 + + .. link-button:: intro:get_started:next + :type: ref + :text: What's next? + :classes: btn-outline-primary btn-block font-weight-bold + +Run AiiDA via a Virtual Machine +------------------------------- + +The `Materials Cloud `__ provides both downloadable and web based VMs, also incorporating additional pre-installed Materials Science codes. +See `here `__ for information on how to download and use the virtual machine. + +.. _intro:get_started:next: + +What's next? +============ + +If you are new to AiiDA, we recommed you go through the :ref:`Basic Tutorial `, or see our :ref:`Next steps guide `. +If however, you encountered some issues, proceed to the :ref:`troubleshooting section `. + +.. admonition:: In-depth instructions + :class: seealso title-icon-read-more + + For more detailed instructions on configuring AiiDA, :ref:`see the configuration how-to `. + .. |PostgreSQL| replace:: `PostgreSQL `__ .. |RabbitMQ| replace:: `RabbitMQ `__ +.. |Homebrew| replace:: `Homebrew `__ From fc3309529701d66eb90b167c304b9af242a874ae Mon Sep 17 00:00:00 2001 From: Carl Simon Adorf Date: Tue, 20 Oct 2020 18:59:36 +0200 Subject: [PATCH 02/13] Rename 'Detailed installation' page to 'Avanced configuration'. And remove all content covered by the get started page. --- docs/source/intro/get_started.rst | 15 + docs/source/intro/installation.rst | 672 +---------------------------- 2 files changed, 28 insertions(+), 659 deletions(-) diff --git a/docs/source/intro/get_started.rst b/docs/source/intro/get_started.rst index f895796999..adb01b515b 100644 --- a/docs/source/intro/get_started.rst +++ b/docs/source/intro/get_started.rst @@ -10,6 +10,8 @@ An AiiDA installation consists of three core components (plus any external codes * |PostgreSQL|: The service that manages the database that AiiDA uses to store data. * |RabbitMQ|: The message broker used for communication within AiiDA. +.. _intro:get_started:setup: + Setup ===== @@ -425,6 +427,7 @@ This is the recommended method for users on shared systems and systems where the (aiida) $ pg_ctl stop .. _intro:get_started:docker: +.. _intro:install:docker: Run AiiDA via a Docker image ---------------------------- @@ -452,6 +455,12 @@ This image contains a fully pre-configured AiiDA environment which makes it part $ docker run -d --name aiida-container aiidateam/aiida-core:\ |release|\ + You can use the following command to block until all services have started up: + + .. code-block:: console + + $ docker exec -t aiida-container wait-for-services + --- **Check setup** @@ -475,6 +484,12 @@ This image contains a fully pre-configured AiiDA environment which makes it part :text: What's next? :classes: btn-outline-primary btn-block font-weight-bold +.. caution:: + + All data stored in the container will persist as long as you restart the same container, e.g., with (``docker start aiida-container``), however if you remove the container, all data will be lost. + Use `volumes `__ to share data between containers and ensure its persistency on the host machine. + + Run AiiDA via a Virtual Machine ------------------------------- diff --git a/docs/source/intro/installation.rst b/docs/source/intro/installation.rst index 832cf71cb4..bdcb8d87eb 100644 --- a/docs/source/intro/installation.rst +++ b/docs/source/intro/installation.rst @@ -1,539 +1,18 @@ .. _intro:install: +.. _intro:advanced-config: -********************* -Detailed Installation -********************* +********************** +Advanced configuration +********************** -.. _intro:install:prerequisites: - -Installing Prerequisites -======================== - -AiiDA is designed to run on `Unix `_ operating systems and requires the following software: - -* `bash `_ or - `zsh `_ (The shell) -* `python`_ >= 3.6 (The programming language used by AiiDA) -* `python3-pip`_ (Python 3 package manager) -* `postgresql`_ (Database software, version 9.4 or higher) -* `RabbitMQ`_ (A message broker necessary for AiiDA to communicate between processes) - -.. admonition:: PostgreSQL and RabbitMQ as services - :class: tip title-icon-tip - - PostgreSQL and RabbitMQ can also be configured to run as services that run on other machines. - When setting up a profile, after having installed AiiDA, you can specify how to connect to these machines. - With this setup, it is not necesseray to install PostgreSQL nor RabbitMQ on the machine where AiiDA is installed. - -Depending on your set up, there are a few optional dependencies: - -* `git`_ (Version control system used for AiiDA development) -* `graphviz`_ (For plotting AiiDA provenance graphs) - -.. _graphviz: https://www.graphviz.org/download -.. _git: https://git-scm.com/downloads -.. _python: https://www.python.org/downloads -.. _python3-pip: https://packaging.python.org/installing/#requirements-for-installing-packages -.. _virtualenv: https://packages.ubuntu.com/bionic/virtualenv -.. _virtualenvwrapper: https://packages.ubuntu.com/bionic/virtualenvwrapper -.. _postgresql: https://www.postgresql.org/downloads -.. _RabbitMQ: https://www.rabbitmq.com/ - -AiiDA has been tested on the following platforms: - -* Ubuntu 14.04, 16.04, 18.04 -* Mac OS X - -We expect AiiDA to also run on: - -* Older and newer Ubuntu versions -* Other Linux distributions -* Windows Subsystem for Linux (WSL) - -Below, we provide installation instructions for a number of operating systems. - -.. div:: dropdown-group - - .. dropdown:: Ubuntu - - .. code-block:: console - - $ sudo apt-get install \ - postgresql postgresql-server-dev-all postgresql-client rabbitmq-server \ - git python3-dev python3-pip - $ pip install aiida-core - - See :ref:`Ubuntu instructions` for details. - - .. dropdown:: MacOS X (Homebrew) - - .. code-block:: console - - $ brew install postgresql rabbitmq git python - $ brew services start postgresql - $ brew services start rabbitmq - $ curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py - $ python get-pip.py - $ pip install aiida-core - - See :ref:`MacOS X (Homebrew) instructions` for details. - - .. dropdown:: MacOS X (MacPorts) - - .. code-block:: console - - $ sudo port install postgresql postgresql-server rabbitmq-server - $ pg_ctl -D /usr/local/var/postgres start - $ curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py - $ python get-pip.py - $ pip install aiida-core - - See :ref:`MacOS X (MacPorts) instructions` for details. - - .. dropdown:: Gentoo Linux - - .. code-block:: console - - $ emerge -av git postgresql rabbitmq-server - $ rc-update add rabbitmq - $ emerge --ask dev-lang/python:3.7 - $ emerge --ask dev-python/pip - $ pip install aiida-core - - See :ref:`Gentoo Linux instructions` for details. - - .. dropdown:: Windows Subsystem for Linux - - .. code-block:: console - - $ sudo apt-get install \ - postgresql postgresql-server-dev-all postgresql-client \ - git python3-dev python-pip - $ sudo service postgresql start - $ pip install aiida-core - - Then RabbitMQ must be `installed directly on Windows `__. - See :ref:`WSL instructions` for details. - -.. _intro:install:prerequisites:ubuntu: - -Ubuntu ------- - -To install the prerequisites on Ubuntu and any other Debian derived distribution, you can use the *apt* package manager. -The following will install the basic Python requirements and the *git* source control manager: - -.. code-block:: console - - $ sudo apt-get install git python3-dev python3-pip - -To install the requirements for the ``postgres`` database run the following: - -.. code-block:: console - - $ sudo apt-get install postgresql postgresql-server-dev-all postgresql-client - -Install the RabbitMQ message broker: - -.. code-block:: console - - $ sudo apt-get install rabbitmq-server - -This installs and adds RabbitMQ as a system service. To check whether it is running: - -.. code-block:: console - - $ sudo rabbitmqctl status - -If it is not running already, it should after a reboot. - -Finally install the aiida-core Python environment: - -.. code-block:: console - - $ sudo git python3-dev python3-pip - $ pip install aiida-core - -.. admonition:: Further Reading - :class: seealso title-icon-read-more - - - For a more detailed description of database requirements and usage see the :ref:`database topic`. - - For problems with installing RabbitMQ, refer to the detailed instructions provided on the `RabbitMQ website for Debian based distributions `_. - -.. _intro:install:prerequisites:brew: - -Mac OS X (homebrew) -------------------- - -For Mac OS we recommend using the `Homebrew`_ package manager. -If you have not installed Homebrew yet, see their `installation guide `__. - -.. _Homebrew: https://brew.sh/ - -After you have installed Homebrew, you can install the basic requirements as follows: - -.. code-block:: console - - $ brew install postgresql rabbitmq git python - -To start the ``postgres`` database server and ``rabbitmq`` service, execute: - -.. code-block:: console - - $ brew services start postgresql - $ brew services start rabbitmq - -You can check whether it is running by checking the status through the command: - -.. code-block:: console - - $ /usr/local/sbin/rabbitmqctl status - -Finally install the aiida-core python environment: - -.. code-block:: console - - $ curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py - $ python get-pip.py - $ pip install aiida-core - -.. admonition:: Further Reading - :class: seealso title-icon-read-more - - - For a more detailed description of database requirements and usage see the :ref:`database topic`. - - For problems with installing RabbitMQ, refer to the detailed instructions provided on the `RabbitMQ website for Homebrew based distributions `_. - - For details on the installation of the ``pip`` package manager, refer to `their documentation `_ - -.. _intro:install:prerequisites:macports: - -Mac OS X (MacPorts) -------------------- - -.. _macports: https://www.macports.org/ - -Another package manager for MacOS is `macports`_. - -.. code-block:: console - - $ sudo port install postgresql postgresql-server rabbitmq-server git python - -To start the ``postgres`` database server, run: - -.. code-block:: console - - $ sudo su postgres - $ pg_ctl -D /opt/local/var/db/postgresql96/defaultdb start - -To start the ``rabbitmq`` server, run: - -.. code-block:: console - - $ sudo launchctl load -w /Library/LaunchDaemons/org.macports.rabbitmq-server.plist - -You can check whether it is running as follows: - -.. code-block:: console - - $ sudo rabbitmqctl status - # this starts ``rabbitmq`` at system startup: - $ sudo port load rabbitmq-server - -Finally install the aiida-core python environment: - -.. code-block:: console - - $ curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py - $ python get-pip.py - $ pip install aiida-core - -.. admonition:: Trouble Installing RabbitMQ? - :class: attention title-icon-troubleshoot - - Be sure to install ``rabbitmq-server 3.7.9`` or later. - If ``rabbitmqctl status`` returns an error "Hostname mismatch", the easiest solution can be to simply ``sudo port uninstall`` the package and install it again. - -.. admonition:: Further Reading - :class: seealso title-icon-read-more - - - For a more detailed description of database requirements and usage see the :ref:`database topic`. - - For details on the installation of the ``pip`` package manager, refer to `their documentation `_ - -.. _intro:install:prerequisites:gentoo: - -Gentoo Linux ------------- - -To install RabbitMQ on a Gentoo distribution through the ``portage`` package manager run the following command: - -.. code-block:: console - - $ emerge -av rabbitmq-server - -To make sure that RabbitMQ is started at system boot, execute: - -.. code-block:: console - - rc-update add rabbitmq - -If you want to manually start the RabbitMQ server you can use: - -.. code-block:: console - - /etc/init.d/rabbitmq start - -Make sure that RabbitMQ is running with: - -.. code-block:: console - - rabbitmqctl status - -.. admonition:: Trouble Installing RabbitMQ? - :class: attention title-icon-troubleshoot - - If you have encounter the following error - - .. code-block:: console - - Argument '-smp enable' not supported." - - Remove the mentioned option from the file ``/usr/libexec/rabbitmq/rabbitmq-env`` and restart the server. - If you still have trouble getting RabbitMQ to run, please refer to the detailed instructions provided on the `website of RabbitMQ itself for generic Unix systems `_. - - -.. _intro:install:prerequisites:wsl: - -Windows Subsystem for Linux (Ubuntu) ------------------------------------- - -The guide for Ubuntu above can generally be followed, but there are a few things to note: - -.. admonition:: Tip - :class: tip title-icon-tip - - Installing `Ubuntu `_ instead of the version specific applications, will let you have the latest LTS version. - -#. The `Windows native RabbitMQ `_ should be installed and started. - (For WSL 2, this should not be necessary.) - -#. Linux services under WSL are not started automatically. - To start the PostgreSQL and RabbitMQ-server services, type the commands below in the terminal:: - - sudo service postgresql start - sudo service rabbitmq-server start - - .. admonition:: Tip - :class: tip title-icon-tip - - These services may be run at startup *without* passing a password in the following way: - - Create a ``.sh`` file with the lines above, but *without* ``sudo``. - Make the file executeable, i.e., type: - - .. code-block:: console - - $ chmod +x /path/to/file.sh - $ sudo visudo - - And add the line: - - .. code-block:: sh - - ALL=(root) NOPASSWD: /path/to/file.sh - - Replacing ```` with your Ubuntu username. - This will allow you to run *only* this specific ``.sh`` file with ``root`` access (without password), without lowering security on the rest of your system. - -#. There is a `known issue `_ in WSL Ubuntu 18.04 where the timezone is not configured correctly out-of-the-box, which may cause problem for the database. - The following command can be used to re-configure the time zone: - - .. code-block:: console - - $ sudo dpkg-reconfigure tzdata - -#. The file open limit may need to be raised using ``ulimit -n 2048`` (default is 1024), when running tests. - You can check the limit by using ``ulimit -n``. - - .. hint:: This may need to be run every time the system starts up. - -It may be worth considering adding some of these commands to your ``~/.bashrc`` file, since some of these settings may reset upon reboot. - -.. admonition:: Further Reading - :class: seealso title-icon-read-more - - For using WSL as a developer, please see the considerations made in our `wiki-page for developers `_. - -.. _intro:install:virtual_environments: - -Using virtual environments -========================== - -AiiDA depends on a number of third party python packages, and usually on specific versions of those packages. -In order to not interfere with third party packages needed by other software on your system, we **strongly** recommend isolating AiiDA in a virtual Python environment, for example, by means of one of the methods described below. - -.. admonition:: Additional Information - :class: seealso title-icon-read-more - - A very good tutorial on Python environments is provided by `realpython.com `__. - -venv ----- - -The `venv `__ module for creating virtual environments ships directly with Python. -To create a virtual environment, in a given directory, run: - -.. code-block:: console - - $ python3 -m venv /path/to/new/virtual/environment/aiida - -The command to activate the environment is shell specific (see `the documentation `__). -With bash the following command is used: - -.. code-block:: console - - $ source /path/to/new/virtual/environment/aiida/bin/activate - -To leave or deactivate the environment, simply run: - -.. code-block:: console - - (aiida) $ deactivate - -.. admonition:: Update install tools - :class: tip title-icon-tip - - You may need to update ``pip`` and ``setuptools`` in your virtual environment, in case the system or user version of these tools is old. - - .. code-block:: console - - (aiida) $ pip install -U setuptools pip - -Conda ------ - -If you have `Conda`_ installed then you can directly create a new environment with ``aiida-core`` and (optionally) the Postgres and RabbitMQ services installed. - -.. code-block:: console - - $ conda create -n aiida -c conda-forge python=3.7 aiida-core aiida-core.services pip - $ conda activate aiida - $ conda deactivate - - -.. _intro:install:aiida-core: - -Installing the aiida-core package -================================= - -.. _Conda: https://anaconda.org/conda-forge/aiida-core -.. _Anaconda Cloud: https://anaconda.org/conda-forge/aiida-core -.. _PyPI: https://pypi.python.org/pypi/aiida-core -.. _github repository: https://github.com/aiidateam/aiida-core - -AiiDA can be installed either from the python package index `PyPI`_, `Anaconda Cloud`_ (both good for general use) or directly from the aiida-core `github repository`_ (good for developers). - -With your virtual environment active (see above), install the ``aiida-core`` python package from `PyPI`_ using: - -.. code-block:: console - - $ pip install aiida-core - -.. admonition:: Installing AiiDA in your system environment - :class: tip title-icon-tip - - Consider adding the ``--user`` flag to avoid the need for administrator privileges. - -Or, if using `Conda`_, from the `Anaconda Cloud `__: - -.. code-block:: console - - $ conda install -c conda-forge aiida-core aiida-core.services - -.. note:: - - `aiida-core.services `__ is an optional package to also install PostgreSQL and RabbitMQ. - -Alternatively, you can create a directory where to clone the AiiDA source code and install AiiDA from source: - -.. code-block:: console - - $ mkdir - $ cd - $ git clone https://github.com/aiidateam/aiida-core - $ pip install -e aiida-core - -There are additional optional packages that you may want to install, which are grouped in the following categories: - - * ``atomic_tools``: packages that allow importing and manipulating crystal structure from various formats - * ``ssh_kerberos``: adds support for ssh transport authentication through Kerberos - * ``REST``: allows a REST server to be ran locally to serve AiiDA data - * ``docs``: tools to build the documentation - * ``notebook``: jupyter notebook - to allow it to import AiiDA modules - * ``tests``: python modules required to run the automatic unit tests - * ``pre-commit``: pre-commit tools required for developers to enable automatic code linting and formatting - -In order to install any of these package groups, simply append them as a comma separated list in the ``pip`` install command - -.. code-block:: console - - $ pip install -e "aiida-core[atomic_tools,docs]" - -.. admonition:: Kerberos on Ubuntu - :class: note title-icon-troubleshoot - - If you are installing the optional ``ssh_kerberos`` and you are on Ubuntu you might encounter an error related to the ``gss`` package. - To fix this you need to install the ``libffi-dev`` and ``libkrb5-dev`` packages: - -.. code-block:: console - - $ sudo apt-get install libffi-dev libkrb5-dev - -AiiDA uses the `reentry `_ package to keep a fast cache of entry points for a snappy ``verdi`` cli. -After installing AiiDA packages, always remember to update the reentry cache using: - -.. code-block:: console - - $ reentry scan - -.. _intro:install:setup: - -Setting up the installation -=========================== - -.. _intro:install:quicksetup: - -AiiDA profile quicksetup ------------------------- - -After successful installation, you need to create an AiiDA profile via AiiDA's command line interface ``verdi``. - -Most users should use the interactive quicksetup: - -.. code-block:: console - - $ verdi quicksetup - -which leads through the installation process and takes care of creating the corresponding AiiDA database. - -For maximum customizability, one can use ``verdi setup``, that provides fine-grained control in configuring how AiiDA should connect to the required services. -This is useful, for example, if PostgreSQL and or RabbitMQ are not installed and configured with default settings, or are run on a different machine from AiiDA itself. - -.. admonition:: Don't forget to backup your data! - :class: tip title-icon-tip - - See the :ref:`installation backup how-to `. - -.. admonition:: Want to manage multiple profiles? - :class: seealso title-icon-read-more - - See the :ref:`managing profiles how-to `. +This chapter covers topics that go beyond the :ref:`standard setup of AiiDA `. +If you are new to AiiDA, we recommed you first go through the :ref:`Basic Tutorial `, +or see our :ref:`Next steps guide `. .. _intro:install:database: -AiiDA profile custom setup --------------------------- - Creating the database -..................... +--------------------- AiiDA uses a database to store the nodes, node attributes and other information, allowing the end user to perform fast queries of the results. Currently, the highly performant `PostgreSQL`_ database is supported as a database backend. @@ -615,7 +94,7 @@ If you use the same names as in the example commands above, then during the ``ve See the :ref:`Database backup how-to `), and :ref:`how to move your database `. Database setup using 'peer' authentication -.......................................... +------------------------------------------ On Ubuntu Linux, the default PostgreSQL setup is configured to use ``peer`` authentication, which allows password-less login via local Unix sockets. In this mode, PostgreSQL compares the Unix user connecting to the socket with its own database of users and allows a connection if a matching user exists. @@ -666,7 +145,7 @@ During the ``verdi setup`` phase, use ``!`` to leave host empty and specify your RabbitMQ configuration -...................... +---------------------- In most normal setups, RabbitMQ will be installed and run as a service on the same machine that hosts AiiDA itself. In that case, using the default configuration proposed during a profile setup will work just fine. @@ -692,7 +171,7 @@ The following parameters can be configured: verdi setup -........... +----------- After the database has been created, do: @@ -739,7 +218,7 @@ Then, the following prompts will help you configure the database. Typical settin .. _intro:install:start_daemon: -Starting the daemon +Managing the daemon ------------------- The AiiDA daemon process runs in the background and takes care of processing your submitted calculations and workflows, checking their status, retrieving their results once they are finished and storing them in the AiiDA database. @@ -755,53 +234,10 @@ The AiiDA daemon is controlled using three simple commands: While operational, the daemon logs its activity to a file in ``~/.aiida/daemon/log/`` (or, more generally, ``$AIIDA_PATH/.aiida/daemon/log``). Get the latest log messages via ``verdi daemon logshow``. -.. _intro:install:final_checks: - -Final checks ------------- - -Use the ``verdi status`` command to check that all services are up and running: - -.. code-block:: console - - $ verdi status - - ✓ profile: On profile quicksetup - ✓ repository: /repo/aiida_dev/quicksetup - ✓ postgres: Connected as aiida@localhost:5432 - ✓ rabbitmq: Connected as amqp://127.0.0.1?heartbeat=600 - ✓ daemon: Daemon is running as PID 2809 since 2019-03-15 16:27:52 - -In the example output, all service have a green check mark and so should be running as expected. - -.. admonition:: Are you in your virtual environment? - :class: note title-icon-troubleshoot - - Remember that in order to work with AiiDA through for example the ``verdi`` command, you need to be in your virtual environment. - If you open a new terminal for example, be sure to activate it first, e.g. for venv: - - .. code-block:: console - - $ source ~/.virtualenvs/aiida/bin/activate - -.. admonition:: Want to manage your install? - :class: seealso title-icon-read-more - - See the :ref:`Installation how-to ` - - :link-badge:`how-to:installation:configure,Configuring,ref,badge-primary text-white` - :link-badge:`how-to:installation:update,Updating,ref,badge-primary text-white` - :link-badge:`how-to:installation:backup:software,Backing-up,ref,badge-primary text-white` - -.. admonition:: What next? - :class: seealso title-icon-read-more - - Try out the :ref:`Basic Tutorial `. - .. _intro:install:jupyter: Using AiiDA in Jupyter -====================== +---------------------- `Jupyter `_ is an open-source web application that allows you to create in-browser notebooks containing live code, visualizations and formatted text. @@ -857,85 +293,3 @@ After this, if you open a Jupyter notebook as explained above and type in a cell followed by ``Shift-Enter``. You should receive the message "Loaded AiiDA DB environment." This line magic should also be enabled in standard ipython shells. - - -.. _intro:install:docker: - -Using the Docker image -====================== - -AiiDA maintains a `Docker `__ image on `Docker Hub `__. -This image contains a pre-configured AiiDA environment, and so is particularly useful for learning and testing purposes. -It is a great way to quickly get started on the tutorials! - -Follow Docker's `install guide `__ to download Docker and start its daemon. -Now you can pull the aiida-core image straight from Docker Hub, for a specific version. - -.. code-block:: console - - $ docker pull aiidateam/aiida-core:1.2.1 - -We can start a container running by: - -.. code-block:: console - - $ docker run -d --name aiida-container aiidateam/aiida-core:1.2.1 - -The container comes installed with all required services and, on start-up, will automatically start them and create an AiiDA profile (plus a localhost computer). -To (optionally) wait for the services to start and inspect the start-up process, we can run: - -.. code-block:: console - - $ docker exec -t aiida-container wait-for-services - $ docker logs aiida-container - -The profile is created under the ``aiida`` username, so to execute commands use: - -.. code-block:: console - - $ docker exec -t --user aiida aiida-container /bin/bash -l -c 'verdi status' - ✓ config dir: /home/aiida/.aiida - ✓ profile: On profile default - ✓ repository: /home/aiida/.aiida/repository/default - ✓ postgres: Connected as aiida_qs_aiida_477d3dfc78a2042156110cb00ae3618f@localhost:5432 - ✓ rabbitmq: Connected as amqp://127.0.0.1?heartbeat=600 - ✓ daemon: Daemon is running as PID 1795 since 2020-05-20 02:54:00 - -Or to enter into the container interactively: - -.. code-block:: console - - $ docker exec -it --user aiida aiida-container /bin/bash - -If you stop the container and start it again, any data you created will persist. - -.. code-block:: console - - $ docker stop aiida-container - $ docker start aiida-container - -But if you remove the container all data will be removed. - -.. code-block:: console - - $ docker stop aiida-container - $ docker rm aiida-container - -To store data and even share it between containers, you may wish to `add a volume `__: - -.. code-block:: console - - $ docker run -d --name aiida-container --mount source=my_data,target=/tmp/my_data aiidateam/aiida-core:1.2.1 - -Now anything that you save to the ``/tmp/my_data`` folder will be saved to the volume persistently. -You can even add files directly to the folder outside of the container, by finding its mounting point: - -.. code-block:: console - - $ docker volume inspect my_data - $ echo "hallo" | sudo tee -a /var/lib/docker/volumes/my_data/_data/hallo.txt > /dev/null - -.. admonition:: What next? - :class: seealso title-icon-read-more - - Try out the :ref:`Basic Tutorial `. From 3011bc73ca1b3778b01b794e243b7d1ef007baad Mon Sep 17 00:00:00 2001 From: Carl Simon Adorf Date: Wed, 21 Oct 2020 11:24:06 +0200 Subject: [PATCH 03/13] Fix references. --- docs/source/howto/installation.rst | 2 +- docs/source/howto/ssh.rst | 2 +- docs/source/intro/get_started.rst | 1 + 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/source/howto/installation.rst b/docs/source/howto/installation.rst index 04a05c5d4f..f741ff6f16 100644 --- a/docs/source/howto/installation.rst +++ b/docs/source/howto/installation.rst @@ -157,7 +157,7 @@ Isolating multiple instances An AiiDA instance is defined as the installed source code plus the configuration folder that stores the configuration files with all the configured profiles. It is possible to run multiple AiiDA instances on a single machine, simply by isolating the code and configuration in a virtual environment. -To isolate the code, simply create a virtual environment, e.g., with conda or venv, and then follow the instructions for :ref:`installation` after activation. +To isolate the code, make sure to install AiiDA into a virtual environment, e.g., with conda or venv, as described :ref:`here `. Whenever you activate this particular environment, you will be running the particular version of AiiDA (and all the plugins) that you installed specifically for it. This is separate from the configuration of AiiDA, which is stored in the configuration directory which is always named ``.aiida`` and by default is stored in the home directory. diff --git a/docs/source/howto/ssh.rst b/docs/source/howto/ssh.rst index bf2f4cc65e..ad590e53c1 100644 --- a/docs/source/howto/ssh.rst +++ b/docs/source/howto/ssh.rst @@ -268,6 +268,6 @@ Using kerberos tokens If the remote machine requires authentication through a Kerberos token (that you need to obtain before using ssh), you typically need to * install ``libffi`` (``sudo apt-get install libffi-dev`` under Ubuntu) - * install the ``ssh_kerberos`` extra during the installation of aiida-core (see :ref:`intro:install:aiida-core`). + * install the ``ssh_kerberos`` extra during the installation of aiida-core (see :ref:`intro:install:setup`). If you provide all necessary ``GSSAPI`` options in your ``~/.ssh/config`` file, ``verdi computer configure`` should already pick up the appropriate values for all the gss-related options. diff --git a/docs/source/intro/get_started.rst b/docs/source/intro/get_started.rst index adb01b515b..4da4bfd652 100644 --- a/docs/source/intro/get_started.rst +++ b/docs/source/intro/get_started.rst @@ -10,6 +10,7 @@ An AiiDA installation consists of three core components (plus any external codes * |PostgreSQL|: The service that manages the database that AiiDA uses to store data. * |RabbitMQ|: The message broker used for communication within AiiDA. +.. _intro:install:setup: .. _intro:get_started:setup: Setup From 9c6ff076678a4ed90ca1422504a6f37f522b2a68 Mon Sep 17 00:00:00 2001 From: Carl Simon Adorf Date: Wed, 21 Oct 2020 17:29:06 +0200 Subject: [PATCH 04/13] Directly link to Quantum Mobile docs for VM route. --- docs/source/intro/get_started.rst | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-) diff --git a/docs/source/intro/get_started.rst b/docs/source/intro/get_started.rst index 4da4bfd652..4be21edfec 100644 --- a/docs/source/intro/get_started.rst +++ b/docs/source/intro/get_started.rst @@ -60,8 +60,7 @@ If you are unsure, use the :ref:`system-wide installation `__ image on `Docker Hub `__. This image contains a fully pre-configured AiiDA environment which makes it particularly useful for learning and testing purposes. @@ -491,12 +488,6 @@ This image contains a fully pre-configured AiiDA environment which makes it part Use `volumes `__ to share data between containers and ensure its persistency on the host machine. -Run AiiDA via a Virtual Machine -------------------------------- - -The `Materials Cloud `__ provides both downloadable and web based VMs, also incorporating additional pre-installed Materials Science codes. -See `here `__ for information on how to download and use the virtual machine. - .. _intro:get_started:next: What's next? From a7140eed4caa7186a8c6d5533ce9ee9f4aba81e5 Mon Sep 17 00:00:00 2001 From: Carl Simon Adorf Date: Wed, 21 Oct 2020 17:39:29 +0200 Subject: [PATCH 05/13] Adjust the list of tested Ubuntu versions. --- docs/source/intro/get_started.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/intro/get_started.rst b/docs/source/intro/get_started.rst index 4be21edfec..eb174ee3aa 100644 --- a/docs/source/intro/get_started.rst +++ b/docs/source/intro/get_started.rst @@ -88,7 +88,7 @@ This is the *recommended* installation method to setup AiiDA on a personal lapto .. tabbed:: Ubuntu - *AiiDA is tested on Ubuntu versions 14.04, 16.04, and 18.04.* + *AiiDA is tested on Ubuntu versions 16.04, 18.04, and 20.04.* Open a terminal and execute: From 0bed035aef324504babbda7662c6937321f30e8c Mon Sep 17 00:00:00 2001 From: Carl Simon Adorf Date: Wed, 21 Oct 2020 17:56:43 +0200 Subject: [PATCH 06/13] Add hint for differing commands for activation of virtual environments. --- docs/source/intro/get_started.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/source/intro/get_started.rst b/docs/source/intro/get_started.rst index eb174ee3aa..c465e8526f 100644 --- a/docs/source/intro/get_started.rst +++ b/docs/source/intro/get_started.rst @@ -195,6 +195,11 @@ This is the *recommended* installation method to setup AiiDA on a personal lapto (aiida) $ pip install aiida-core (aiida) $ reentry scan + .. hint:: + + See the `venv documentation `__ if the activation command fails. + The exact command for activating a virtual environment differs slightly based on the used shell. + .. dropdown:: :fa:`plus-circle` Installation extras There are additional optional packages that you may want to install, which are grouped in the following categories: From b97c47968d65203d7d2b767d6d89f7a73e9a2ddc Mon Sep 17 00:00:00 2001 From: Carl Simon Adorf Date: Wed, 21 Oct 2020 18:25:23 +0200 Subject: [PATCH 07/13] Place individual setup routes on separate pages. --- docs/source/intro/get_started.rst | 436 +-------------------------- docs/source/intro/install_conda.rst | 124 ++++++++ docs/source/intro/install_system.rst | 239 +++++++++++++++ docs/source/intro/run_docker.rst | 61 ++++ 4 files changed, 433 insertions(+), 427 deletions(-) create mode 100644 docs/source/intro/install_conda.rst create mode 100644 docs/source/intro/install_system.rst create mode 100644 docs/source/intro/run_docker.rst diff --git a/docs/source/intro/get_started.rst b/docs/source/intro/get_started.rst index c465e8526f..cc14b7f67a 100644 --- a/docs/source/intro/get_started.rst +++ b/docs/source/intro/get_started.rst @@ -10,6 +10,14 @@ An AiiDA installation consists of three core components (plus any external codes * |PostgreSQL|: The service that manages the database that AiiDA uses to store data. * |RabbitMQ|: The message broker used for communication within AiiDA. +.. toctree:: + :maxdepth: 1 + :hidden: + + install_system + install_conda + run_docker + .. _intro:install:setup: .. _intro:get_started:setup: @@ -67,438 +75,12 @@ If you are unsure, use the :ref:`system-wide installation `__ provides both downloadable and web based VMs, also incorporating pre-installed Materials Science codes. - -.. _intro:get_started:system-wide-install: - -System-wide installation ------------------------- - -The system-wide installation will install the prerequisite services (PostgreSQL and RabbitMQ) via standard package managers such that their startup and shut-down is largely managed by the operating system. -The AiiDA (core) Python package is then installed either with Conda or pip. - -This is the *recommended* installation method to setup AiiDA on a personal laptop or workstation for the majority of users. - -.. panels:: - :container: container-lg pb-3 - :column: col-lg-12 p-2 - - **Install prerequisite services** - - AiiDA is designed to run on `Unix `_ operating systems and requires a `bash `_ or `zsh `_ shell, and Python >= 3.6. - - .. tabbed:: Ubuntu - - *AiiDA is tested on Ubuntu versions 16.04, 18.04, and 20.04.* - - Open a terminal and execute: - - .. code-block:: console - - $ sudo apt install \ - git python3-dev python3-pip \ - postgresql postgresql-server-dev-all postgresql-client rabbitmq-server - - .. tabbed:: MacOS X (Homebrew) - - The recommended installation method for Mac OS X is to use `Homebrew `__. - - #. Follow `this guide `__ to install Homebrew on your system if not installed yet. - - #. Open a terminal and execute: - - .. code-block:: console - - $ brew install postgresql rabbitmq git python - $ brew services start postgresql - $ brew services start rabbitmq - - .. tabbed:: Windows Subsystem for Linux - - *The following instructions are for setting up AiiDA on WSL 2 in combination with Ubuntu.* - - #. The `Windows native RabbitMQ `_ should be installed and started. - - #. Install Python and PostgreSQL: - - .. code-block:: console - - $ sudo apt-get install \ - postgresql postgresql-server-dev-all postgresql-client \ - git python3-dev python-pip - $ sudo service postgresql start - - .. dropdown:: How to setup WSL to automatically start services after system boot. - - Create a ``.sh`` file with the lines above, but *without* ``sudo``. - Make the file executeable, i.e., type: - - .. code-block:: console - - $ chmod +x /path/to/file.sh - $ sudo visudo - - And add the line: - - .. code-block:: sh - - ALL=(root) NOPASSWD: /path/to/file.sh - - Replacing ```` with your Ubuntu username. - This will allow you to run *only* this specific ``.sh`` file with ``root`` access (without password), without lowering security on the rest of your system. - - .. dropdown:: :fa:`wrench` How to resolve a timezone issue on Ubuntu 18.04. - - There is a `known issue `_ in WSL Ubuntu 18.04 where the timezone is not configured correctly out-of-the-box, which may cause a problem for the database. - The following command can be used to re-configure the time zone: - - .. code-block:: console - - $ sudo dpkg-reconfigure tzdata - - .. tabbed:: Other - - #. Install RabbitMQ following the `instructions applicable to your system `__. - #. Install PostgreSQL following the `instructions applicable to your system `__. - - .. hint:: - - Alternatively use the :ref:`pure conda installation method `. - - --- - - **Install AiiDA (core)** - - .. tabbed:: Conda - - *Install the aiida-core package in a Conda environment.* - - #. Make sure that conda is installed, e.g., by following `the instructions on installing Miniconda `__. - - #. Open a terminal and execute: - - .. code-block:: console - - $ conda create -n aiida -c conda-forge aiida-core - $ conda activate aiida - (aiida) $ reentry scan - - .. tabbed:: pip + venv - - *Install the aiida-core package from PyPI into a virtual environment.* - - Open a terminal and execute: - - .. code-block:: console - - $ python -m pip venv ~/envs/aiida - $ source ~/envs/aiida/bin/activate - (aiida) $ pip install aiida-core - (aiida) $ reentry scan - - .. hint:: - - See the `venv documentation `__ if the activation command fails. - The exact command for activating a virtual environment differs slightly based on the used shell. - - .. dropdown:: :fa:`plus-circle` Installation extras - - There are additional optional packages that you may want to install, which are grouped in the following categories: - - * ``atomic_tools``: packages that allow importing and manipulating crystal structure from various formats - * ``ssh_kerberos``: adds support for ssh transport authentication through Kerberos - * ``REST``: allows a REST server to be ran locally to serve AiiDA data - * ``docs``: tools to build the documentation - * ``notebook``: jupyter notebook - to allow it to import AiiDA modules - * ``tests``: python modules required to run the automatic unit tests - * ``pre-commit``: pre-commit tools required for developers to enable automatic code linting and formatting - - In order to install any of these package groups, simply append them as a comma separated list in the ``pip`` install command, for example: - - .. code-block:: console - - (aiida) $ pip install aiida-core[atomic_tools,docs] - - .. dropdown:: :fa:`wrench` Kerberos on Ubuntu - - If you are installing the optional ``ssh_kerberos`` and you are on Ubuntu you might encounter an error related to the ``gss`` package. - To fix this you need to install the ``libffi-dev`` and ``libkrb5-dev`` packages: - - .. code-block:: console - - $ sudo apt-get install libffi-dev libkrb5-dev - - - - .. tabbed:: From source - - *Install the aiida-core package directly from the cloned repository.* - - Open a terminal and execute: - - .. code-block:: console - - $ git clone https://github.com/aiidateam/aiida-core.git - $ cd aiida-core/ - $ python -m pip venv ~/envs/aiida - $ source ~/envs/aiida/bin/activate - (aiida) $ pip install . - (aiida) $ reentry scan - - --- - - **Start verdi daemons** - - Start the verdi daemon(s) that are used to run AiiDA workflows. - - .. code-block:: console - - (aiida) $ verdi daemon start 2 - - .. important:: - - The verdi daemon(s) must be restarted after a system reboot. - - .. hint:: - - Do not start more daemons then there are physical processors on your system. - - --- - - **Setup profile** - - Next, set up an AiiDA configuration profile and related data storage, with the ``verdi quicksetup`` command. - - .. code-block:: console - - (aiida) $ verdi quicksetup - Info: enter "?" for help - Info: enter "!" to ignore the default and set no value - Profile name: me - Email Address (for sharing data): me@user.com - First name: my - Last name: name - Institution: where-i-work - - --- - - **Check setup** - - To check that everything is set up correctly, execute: - - .. code-block:: console - - (aiida) $ verdi status - ✓ config dir: /home/ubuntu/.aiida - ✓ profile: On profile me - ✓ repository: /home/ubuntu/.aiida/repository/me - ✓ postgres: Connected as aiida_qs_ubuntu_c6a4f69d255fbe9cdb7385dcdcf3c050@localhost:5432 - ✓ rabbitmq: Connected as amqp://127.0.0.1?heartbeat=600 - ✓ daemon: Daemon is running as PID 16430 since 2020-04-29 12:17:31 - - At this point you should now have a working AiiDA environment, from which you can add and retrieve data. - - .. admonition:: Missing a checkmark or ecountered some other issue? - :class: attention title-icon-troubleshoot - - :ref:`See the troubleshooting section `. - - .. link-button:: intro:get_started:next - :type: ref - :text: What's next? - :classes: btn-outline-primary btn-block font-weight-bold - -.. _intro:get_started:conda-install: - -Installation into Conda environment ------------------------------------ - -This installation route installs all necessary software -- including the prerequisite services PostgreSQL and RabbitMQ -- into a Conda environment. -This is the recommended method for users on shared systems and systems where the user has no administrative privileges. - -.. important:: - - This installation method installs **all** software into a conda environment, including PostgreSQL and RabbitMQ. - See the :ref:`system-wide installation ` to use Conda only to install the AiiDA (core) Python package. - -.. panels:: - :container: container-lg pb-3 - :column: col-lg-12 p-2 - - **Install prerequisite services + AiiDA (core)** - - .. code-block:: console - - $ conda create -n aiida -c conda-forge aiida-core aiida-core.services - $ conda activate aiida - (aiida) $ reentry scan - - --- - - **Start-up services and initialize data storage** - - Before working with AiiDA, you must first initialize a database storage area on disk. - - .. code-block:: console - - (aiida) $ initdb -D mylocal_db - - This *database cluster* may contain a collection of databases (one per profile) that is managed by a single running server process. - We start this process with: - - .. code-block:: console - - (aiida) $ pg_ctl -D mylocal_db -l logfile start - - .. admonition:: Further Reading - :class: seealso title-icon-read-more - - - `Creating a Database Cluster `__. - - `Starting the Database Server `__. - - Then, start the RabbitMQ server: - - .. code-block:: console - - (aiida) $ rabbitmq-server -detached - - Finally, start the AiiDA daemon(s): - - .. code-block:: console - - (aiida) $ verdi daemon start 2 - - .. important:: - - The verdi daemon(s) must be restarted after a system reboot. - - .. hint:: - - Do not start more daemons then there are physical processors on your system. - - --- - - **Setup profile** - - Next, set up an AiiDA configuration profile and related data storage, with the ``verdi quicksetup`` command. - - .. code-block:: console - - (aiida) $ verdi quicksetup - Info: enter "?" for help - Info: enter "!" to ignore the default and set no value - Profile name: me - Email Address (for sharing data): me@user.com - First name: my - Last name: name - Institution: where-i-work - - --- - - **Check setup** - - To check that everything is set up correctly, execute: - - .. code-block:: console - - (aiida) $ verdi status - ✓ config dir: /home/ubuntu/.aiida - ✓ profile: On profile me - ✓ repository: /home/ubuntu/.aiida/repository/me - ✓ postgres: Connected as aiida_qs_ubuntu_c6a4f69d255fbe9cdb7385dcdcf3c050@localhost:5432 - ✓ rabbitmq: Connected as amqp://127.0.0.1?heartbeat=600 - ✓ daemon: Daemon is running as PID 16430 since 2020-04-29 12:17:31 - - At this point you now have a working AiiDA environment, from which you can add and retrieve data. - - .. admonition:: Missing a checkmark or ecountered some other issue? - :class: attention title-icon-troubleshoot - - :ref:`See the troubleshooting section `. - - .. link-button:: intro:get_started:next - :type: ref - :text: What's next? - :classes: btn-outline-primary btn-block font-weight-bold - - --- - - **Shut-down services** - - After finishing with your aiida session, particularly if switching between profiles, you may wish to power down the services: - - .. code-block:: console - - (aiida) $ verdi daemon stop - (aiida) $ pg_ctl stop - -.. _intro:get_started:docker: -.. _intro:install:docker: - -Run AiiDA via a Docker image ----------------------------- - -The AiiDA team maintains a `Docker `__ image on `Docker Hub `__. -This image contains a fully pre-configured AiiDA environment which makes it particularly useful for learning and testing purposes. - -.. panels:: - :container: container-lg pb-3 - :column: col-lg-12 p-2 - - **Start container** - - First, pull the image: - - .. parsed-literal:: - - $ docker pull aiidateam/aiida-core:\ |release|\ - - Then start the container with: - - .. parsed-literal:: - - $ docker run -d --name aiida-container aiidateam/aiida-core:\ |release|\ - - You can use the following command to block until all services have started up: - - .. code-block:: console - - $ docker exec -t aiida-container wait-for-services - - --- - - **Check setup** - - The default profile is created under the ``aiida`` user, so to execute commands you must add the ``--user aiida`` option. - - For example, to check the verdi status, execute: - - .. code-block:: console - - $ docker exec -t --user aiida aiida-container /bin/bash -l -c 'verdi status' - ✓ config dir: /home/aiida/.aiida - ✓ profile: On profile default - ✓ repository: /home/aiida/.aiida/repository/default - ✓ postgres: Connected as aiida_qs_aiida_477d3dfc78a2042156110cb00ae3618f@localhost:5432 - ✓ rabbitmq: Connected as amqp://127.0.0.1?heartbeat=600 - ✓ daemon: Daemon is running as PID 1795 since 2020-05-20 02:54:00 - - .. link-button:: intro:get_started:next - :type: ref - :text: What's next? - :classes: btn-outline-primary btn-block font-weight-bold - -.. caution:: - - All data stored in the container will persist as long as you restart the same container, e.g., with (``docker start aiida-container``), however if you remove the container, all data will be lost. - Use `volumes `__ to share data between containers and ensure its persistency on the host machine. - - .. _intro:get_started:next: What's next? ============ -If you are new to AiiDA, we recommed you go through the :ref:`Basic Tutorial `, +After successfully completing one of the above outlined setup routes, if you are new to AiiDA, we recommed you go through the :ref:`Basic Tutorial `, or see our :ref:`Next steps guide `. If however, you encountered some issues, proceed to the :ref:`troubleshooting section `. diff --git a/docs/source/intro/install_conda.rst b/docs/source/intro/install_conda.rst new file mode 100644 index 0000000000..56144727f3 --- /dev/null +++ b/docs/source/intro/install_conda.rst @@ -0,0 +1,124 @@ +.. _intro:get_started:conda-install: + +*********************************** +Installation into Conda environment +*********************************** + +This installation route installs all necessary software -- including the prerequisite services PostgreSQL and RabbitMQ -- into a Conda environment. +This is the recommended method for users on shared systems and systems where the user has no administrative privileges. + +.. important:: + + This installation method installs **all** software into a conda environment, including PostgreSQL and RabbitMQ. + See the :ref:`system-wide installation ` to use Conda only to install the AiiDA (core) Python package. + +.. panels:: + :container: container-lg pb-3 + :column: col-lg-12 p-2 + + **Install prerequisite services + AiiDA (core)** + + .. code-block:: console + + $ conda create -n aiida -c conda-forge aiida-core aiida-core.services + $ conda activate aiida + (aiida) $ reentry scan + + --- + + **Start-up services and initialize data storage** + + Before working with AiiDA, you must first initialize a database storage area on disk. + + .. code-block:: console + + (aiida) $ initdb -D mylocal_db + + This *database cluster* may contain a collection of databases (one per profile) that is managed by a single running server process. + We start this process with: + + .. code-block:: console + + (aiida) $ pg_ctl -D mylocal_db -l logfile start + + .. admonition:: Further Reading + :class: seealso title-icon-read-more + + - `Creating a Database Cluster `__. + - `Starting the Database Server `__. + + Then, start the RabbitMQ server: + + .. code-block:: console + + (aiida) $ rabbitmq-server -detached + + Finally, start the AiiDA daemon(s): + + .. code-block:: console + + (aiida) $ verdi daemon start 2 + + .. important:: + + The verdi daemon(s) must be restarted after a system reboot. + + .. hint:: + + Do not start more daemons then there are physical processors on your system. + + --- + + **Setup profile** + + Next, set up an AiiDA configuration profile and related data storage, with the ``verdi quicksetup`` command. + + .. code-block:: console + + (aiida) $ verdi quicksetup + Info: enter "?" for help + Info: enter "!" to ignore the default and set no value + Profile name: me + Email Address (for sharing data): me@user.com + First name: my + Last name: name + Institution: where-i-work + + --- + + **Check setup** + + To check that everything is set up correctly, execute: + + .. code-block:: console + + (aiida) $ verdi status + ✓ config dir: /home/ubuntu/.aiida + ✓ profile: On profile me + ✓ repository: /home/ubuntu/.aiida/repository/me + ✓ postgres: Connected as aiida_qs_ubuntu_c6a4f69d255fbe9cdb7385dcdcf3c050@localhost:5432 + ✓ rabbitmq: Connected as amqp://127.0.0.1?heartbeat=600 + ✓ daemon: Daemon is running as PID 16430 since 2020-04-29 12:17:31 + + At this point you now have a working AiiDA environment, from which you can add and retrieve data. + + .. admonition:: Missing a checkmark or ecountered some other issue? + :class: attention title-icon-troubleshoot + + :ref:`See the troubleshooting section `. + + .. link-button:: intro:get_started:next + :type: ref + :text: What's next? + :classes: btn-outline-primary btn-block font-weight-bold + + --- + + **Shut-down services** + + After finishing with your aiida session, particularly if switching between profiles, you may wish to power down the services: + + .. code-block:: console + + (aiida) $ verdi daemon stop + (aiida) $ pg_ctl stop diff --git a/docs/source/intro/install_system.rst b/docs/source/intro/install_system.rst new file mode 100644 index 0000000000..17fdfa5580 --- /dev/null +++ b/docs/source/intro/install_system.rst @@ -0,0 +1,239 @@ +.. _intro:get_started:system-wide-install: + +************************ +System-wide installation +************************ + +The system-wide installation will install the prerequisite services (PostgreSQL and RabbitMQ) via standard package managers such that their startup and shut-down is largely managed by the operating system. +The AiiDA (core) Python package is then installed either with Conda or pip. + +This is the *recommended* installation method to setup AiiDA on a personal laptop or workstation for the majority of users. + +.. panels:: + :container: container-lg pb-3 + :column: col-lg-12 p-2 + + **Install prerequisite services** + + AiiDA is designed to run on `Unix `_ operating systems and requires a `bash `_ or `zsh `_ shell, and Python >= 3.6. + + .. tabbed:: Ubuntu + + *AiiDA is tested on Ubuntu versions 16.04, 18.04, and 20.04.* + + Open a terminal and execute: + + .. code-block:: console + + $ sudo apt install \ + git python3-dev python3-pip \ + postgresql postgresql-server-dev-all postgresql-client rabbitmq-server + + .. tabbed:: MacOS X (Homebrew) + + The recommended installation method for Mac OS X is to use `Homebrew `__. + + #. Follow `this guide `__ to install Homebrew on your system if not installed yet. + + #. Open a terminal and execute: + + .. code-block:: console + + $ brew install postgresql rabbitmq git python + $ brew services start postgresql + $ brew services start rabbitmq + + .. tabbed:: Windows Subsystem for Linux + + *The following instructions are for setting up AiiDA on WSL 2 in combination with Ubuntu.* + + #. The `Windows native RabbitMQ `_ should be installed and started. + + #. Install Python and PostgreSQL: + + .. code-block:: console + + $ sudo apt-get install \ + postgresql postgresql-server-dev-all postgresql-client \ + git python3-dev python-pip + $ sudo service postgresql start + + .. dropdown:: How to setup WSL to automatically start services after system boot. + + Create a ``.sh`` file with the lines above, but *without* ``sudo``. + Make the file executeable, i.e., type: + + .. code-block:: console + + $ chmod +x /path/to/file.sh + $ sudo visudo + + And add the line: + + .. code-block:: sh + + ALL=(root) NOPASSWD: /path/to/file.sh + + Replacing ```` with your Ubuntu username. + This will allow you to run *only* this specific ``.sh`` file with ``root`` access (without password), without lowering security on the rest of your system. + + .. dropdown:: :fa:`wrench` How to resolve a timezone issue on Ubuntu 18.04. + + There is a `known issue `_ in WSL Ubuntu 18.04 where the timezone is not configured correctly out-of-the-box, which may cause a problem for the database. + The following command can be used to re-configure the time zone: + + .. code-block:: console + + $ sudo dpkg-reconfigure tzdata + + .. tabbed:: Other + + #. Install RabbitMQ following the `instructions applicable to your system `__. + #. Install PostgreSQL following the `instructions applicable to your system `__. + + .. hint:: + + Alternatively use the :ref:`pure conda installation method `. + + --- + + **Install AiiDA (core)** + + .. tabbed:: Conda + + *Install the aiida-core package in a Conda environment.* + + #. Make sure that conda is installed, e.g., by following `the instructions on installing Miniconda `__. + + #. Open a terminal and execute: + + .. code-block:: console + + $ conda create -n aiida -c conda-forge aiida-core + $ conda activate aiida + (aiida) $ reentry scan + + .. tabbed:: pip + venv + + *Install the aiida-core package from PyPI into a virtual environment.* + + Open a terminal and execute: + + .. code-block:: console + + $ python -m pip venv ~/envs/aiida + $ source ~/envs/aiida/bin/activate + (aiida) $ pip install aiida-core + (aiida) $ reentry scan + + .. hint:: + + See the `venv documentation `__ if the activation command fails. + The exact command for activating a virtual environment differs slightly based on the used shell. + + .. dropdown:: :fa:`plus-circle` Installation extras + + There are additional optional packages that you may want to install, which are grouped in the following categories: + + * ``atomic_tools``: packages that allow importing and manipulating crystal structure from various formats + * ``ssh_kerberos``: adds support for ssh transport authentication through Kerberos + * ``REST``: allows a REST server to be ran locally to serve AiiDA data + * ``docs``: tools to build the documentation + * ``notebook``: jupyter notebook - to allow it to import AiiDA modules + * ``tests``: python modules required to run the automatic unit tests + * ``pre-commit``: pre-commit tools required for developers to enable automatic code linting and formatting + + In order to install any of these package groups, simply append them as a comma separated list in the ``pip`` install command, for example: + + .. code-block:: console + + (aiida) $ pip install aiida-core[atomic_tools,docs] + + .. dropdown:: :fa:`wrench` Kerberos on Ubuntu + + If you are installing the optional ``ssh_kerberos`` and you are on Ubuntu you might encounter an error related to the ``gss`` package. + To fix this you need to install the ``libffi-dev`` and ``libkrb5-dev`` packages: + + .. code-block:: console + + $ sudo apt-get install libffi-dev libkrb5-dev + + + + .. tabbed:: From source + + *Install the aiida-core package directly from the cloned repository.* + + Open a terminal and execute: + + .. code-block:: console + + $ git clone https://github.com/aiidateam/aiida-core.git + $ cd aiida-core/ + $ python -m pip venv ~/envs/aiida + $ source ~/envs/aiida/bin/activate + (aiida) $ pip install . + (aiida) $ reentry scan + + --- + + **Start verdi daemons** + + Start the verdi daemon(s) that are used to run AiiDA workflows. + + .. code-block:: console + + (aiida) $ verdi daemon start 2 + + .. important:: + + The verdi daemon(s) must be restarted after a system reboot. + + .. hint:: + + Do not start more daemons then there are physical processors on your system. + + --- + + **Setup profile** + + Next, set up an AiiDA configuration profile and related data storage, with the ``verdi quicksetup`` command. + + .. code-block:: console + + (aiida) $ verdi quicksetup + Info: enter "?" for help + Info: enter "!" to ignore the default and set no value + Profile name: me + Email Address (for sharing data): me@user.com + First name: my + Last name: name + Institution: where-i-work + + --- + + **Check setup** + + To check that everything is set up correctly, execute: + + .. code-block:: console + + (aiida) $ verdi status + ✓ config dir: /home/ubuntu/.aiida + ✓ profile: On profile me + ✓ repository: /home/ubuntu/.aiida/repository/me + ✓ postgres: Connected as aiida_qs_ubuntu_c6a4f69d255fbe9cdb7385dcdcf3c050@localhost:5432 + ✓ rabbitmq: Connected as amqp://127.0.0.1?heartbeat=600 + ✓ daemon: Daemon is running as PID 16430 since 2020-04-29 12:17:31 + + At this point you should now have a working AiiDA environment, from which you can add and retrieve data. + + .. admonition:: Missing a checkmark or ecountered some other issue? + :class: attention title-icon-troubleshoot + + :ref:`See the troubleshooting section `. + + .. link-button:: intro:get_started:next + :type: ref + :text: What's next? + :classes: btn-outline-primary btn-block font-weight-bold diff --git a/docs/source/intro/run_docker.rst b/docs/source/intro/run_docker.rst new file mode 100644 index 0000000000..6229ae1e4b --- /dev/null +++ b/docs/source/intro/run_docker.rst @@ -0,0 +1,61 @@ +.. _intro:get_started:docker: +.. _intro:install:docker: + +**************************** +Run AiiDA via a Docker image +**************************** + +The AiiDA team maintains a `Docker `__ image on `Docker Hub `__. +This image contains a fully pre-configured AiiDA environment which makes it particularly useful for learning and testing purposes. + +.. panels:: + :container: container-lg pb-3 + :column: col-lg-12 p-2 + + **Start container** + + First, pull the image: + + .. parsed-literal:: + + $ docker pull aiidateam/aiida-core:\ |release|\ + + Then start the container with: + + .. parsed-literal:: + + $ docker run -d --name aiida-container aiidateam/aiida-core:\ |release|\ + + You can use the following command to block until all services have started up: + + .. code-block:: console + + $ docker exec -t aiida-container wait-for-services + + --- + + **Check setup** + + The default profile is created under the ``aiida`` user, so to execute commands you must add the ``--user aiida`` option. + + For example, to check the verdi status, execute: + + .. code-block:: console + + $ docker exec -t --user aiida aiida-container /bin/bash -l -c 'verdi status' + ✓ config dir: /home/aiida/.aiida + ✓ profile: On profile default + ✓ repository: /home/aiida/.aiida/repository/default + ✓ postgres: Connected as aiida_qs_aiida_477d3dfc78a2042156110cb00ae3618f@localhost:5432 + ✓ rabbitmq: Connected as amqp://127.0.0.1?heartbeat=600 + ✓ daemon: Daemon is running as PID 1795 since 2020-05-20 02:54:00 + + .. link-button:: intro:get_started:next + :type: ref + :text: What's next? + :classes: btn-outline-primary btn-block font-weight-bold + +.. caution:: + + All data stored in the container will persist as long as you restart the same container, e.g., with (``docker start aiida-container``), however if you remove the container, all data will be lost. + Use `volumes `__ to share data between containers and ensure its persistency on the host machine. From bd934139d7295e1dae4a0ecfa5914386abc7d914 Mon Sep 17 00:00:00 2001 From: Carl Simon Adorf Date: Thu, 22 Oct 2020 09:10:45 +0200 Subject: [PATCH 08/13] Swap 'pip + venv' and 'Conda' tabs in system-wide install. That makes it more likely for users to choose that method which is overall still better supported than Conda; the latter has no provision to install extras. --- docs/source/intro/install_system.rst | 26 ++++++++++++-------------- 1 file changed, 12 insertions(+), 14 deletions(-) diff --git a/docs/source/intro/install_system.rst b/docs/source/intro/install_system.rst index 17fdfa5580..e7bcab5d0c 100644 --- a/docs/source/intro/install_system.rst +++ b/docs/source/intro/install_system.rst @@ -99,20 +99,6 @@ This is the *recommended* installation method to setup AiiDA on a personal lapto **Install AiiDA (core)** - .. tabbed:: Conda - - *Install the aiida-core package in a Conda environment.* - - #. Make sure that conda is installed, e.g., by following `the instructions on installing Miniconda `__. - - #. Open a terminal and execute: - - .. code-block:: console - - $ conda create -n aiida -c conda-forge aiida-core - $ conda activate aiida - (aiida) $ reentry scan - .. tabbed:: pip + venv *Install the aiida-core package from PyPI into a virtual environment.* @@ -158,7 +144,19 @@ This is the *recommended* installation method to setup AiiDA on a personal lapto $ sudo apt-get install libffi-dev libkrb5-dev + .. tabbed:: Conda + + *Install the aiida-core package in a Conda environment.* + #. Make sure that conda is installed, e.g., by following `the instructions on installing Miniconda `__. + + #. Open a terminal and execute: + + .. code-block:: console + + $ conda create -n aiida -c conda-forge aiida-core + $ conda activate aiida + (aiida) $ reentry scan .. tabbed:: From source From 0867e09ee112d903d8d649ca4452abc65821164b Mon Sep 17 00:00:00 2001 From: Dominik Gresch Date: Thu, 22 Oct 2020 12:23:57 +0200 Subject: [PATCH 09/13] Update WSL 1+2 installation instructions - Distinguishes between WSL 1 / 2 by how RabbitMQ is installed only - Adds instructions for setting up a task in Task Scheduler for starting the services. - Removes the warning about the timezone problem, as it seems to be fixed. --- docs/source/intro/install_system.rst | 65 +++++++++++++++++++++++----- 1 file changed, 53 insertions(+), 12 deletions(-) diff --git a/docs/source/intro/install_system.rst b/docs/source/intro/install_system.rst index e7bcab5d0c..c1b0027f7e 100644 --- a/docs/source/intro/install_system.rst +++ b/docs/source/intro/install_system.rst @@ -47,44 +47,85 @@ This is the *recommended* installation method to setup AiiDA on a personal lapto *The following instructions are for setting up AiiDA on WSL 2 in combination with Ubuntu.* - #. The `Windows native RabbitMQ `_ should be installed and started. + #. Installing RabbitMQ: + + #. On WSL 1, the `Windows native RabbitMQ `_ should be installed and started. + + #. On WSL 2, RabbitMQ can be installed inside WSL: + + .. code-block:: console + + $ sudo apt install rabbitmq-server + $ sudo service rabbitmq-server start #. Install Python and PostgreSQL: .. code-block:: console - $ sudo apt-get install \ + $ sudo apt install \ postgresql postgresql-server-dev-all postgresql-client \ git python3-dev python-pip $ sudo service postgresql start .. dropdown:: How to setup WSL to automatically start services after system boot. - Create a ``.sh`` file with the lines above, but *without* ``sudo``. - Make the file executeable, i.e., type: + Create a ``.sh`` file containing the following lines: + + .. code-block:: console + + service postgresql start + service rabbitmq-server start # Only for WSL 2! + + + Make the file executeable, and make it editable only by root users by typing + + .. code-block:: console + + $ chmod a+x,go-w /path/to/file.sh + $ sudo chown root:root /path/to/file.sh + + To allow running this file as sudo without requiring a password, run .. code-block:: console - $ chmod +x /path/to/file.sh $ sudo visudo - And add the line: + and add the line .. code-block:: sh ALL=(root) NOPASSWD: /path/to/file.sh - Replacing ```` with your Ubuntu username. + replacing ```` with your Ubuntu username. This will allow you to run *only* this specific ``.sh`` file with ``root`` access (without password), without lowering security on the rest of your system. - .. dropdown:: :fa:`wrench` How to resolve a timezone issue on Ubuntu 18.04. + Now you can use the Windows Task Scheduler to automatically execute this file on startup: - There is a `known issue `_ in WSL Ubuntu 18.04 where the timezone is not configured correctly out-of-the-box, which may cause a problem for the database. - The following command can be used to re-configure the time zone: + #. Open Task Scheduler - .. code-block:: console + #. In the "Actions" menu, click "Create Task" + + #. In "General/Security options", select "Run whether user is logged on or not". + + #. In the "Triggers" tab, click "New..." + + #. In the "Begin the task:" dropdown, select "At startup". + + #. Click "OK" to confirm. + + #. In the "Actions" tab, click "New..." + + #. In the "Action" dropdown, select "Start a program". + + #. In the "Program/script" text field, add ``C:\Windows\System32\bash.exe`` + + #. In the "Add arguments (optional)" text field, add ``-c "sudo /path/to/file.sh"`` + + #. Click "OK" to confirm. + + #. Click "OK" to confirm the task. - $ sudo dpkg-reconfigure tzdata + The details of this task can be tweaked to fit your needs - use your own judgement as to when you would like the services to be started. .. tabbed:: Other From b4113ca6e027d2bcac6115d6708a46b6cf7706a5 Mon Sep 17 00:00:00 2001 From: Dominik Gresch Date: Thu, 22 Oct 2020 13:42:25 +0200 Subject: [PATCH 10/13] Fix formatting of WSL install instructions. --- docs/source/intro/install_system.rst | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/source/intro/install_system.rst b/docs/source/intro/install_system.rst index c1b0027f7e..f89dfe4a6b 100644 --- a/docs/source/intro/install_system.rst +++ b/docs/source/intro/install_system.rst @@ -49,14 +49,14 @@ This is the *recommended* installation method to setup AiiDA on a personal lapto #. Installing RabbitMQ: - #. On WSL 1, the `Windows native RabbitMQ `_ should be installed and started. + #. On WSL 1, the `Windows native RabbitMQ `_ should be installed and started. - #. On WSL 2, RabbitMQ can be installed inside WSL: + #. On WSL 2, RabbitMQ can be installed inside WSL: - .. code-block:: console + .. code-block:: console - $ sudo apt install rabbitmq-server - $ sudo service rabbitmq-server start + $ sudo apt install rabbitmq-server + $ sudo service rabbitmq-server start #. Install Python and PostgreSQL: @@ -84,7 +84,7 @@ This is the *recommended* installation method to setup AiiDA on a personal lapto $ chmod a+x,go-w /path/to/file.sh $ sudo chown root:root /path/to/file.sh - To allow running this file as sudo without requiring a password, run + Next, run .. code-block:: console @@ -101,25 +101,25 @@ This is the *recommended* installation method to setup AiiDA on a personal lapto Now you can use the Windows Task Scheduler to automatically execute this file on startup: - #. Open Task Scheduler + #. Open Task Scheduler. - #. In the "Actions" menu, click "Create Task" + #. In the "Actions" menu, click "Create Task". #. In "General/Security options", select "Run whether user is logged on or not". - #. In the "Triggers" tab, click "New..." + #. In the "Triggers" tab, click "New...". #. In the "Begin the task:" dropdown, select "At startup". #. Click "OK" to confirm. - #. In the "Actions" tab, click "New..." + #. In the "Actions" tab, click "New...". #. In the "Action" dropdown, select "Start a program". - #. In the "Program/script" text field, add ``C:\Windows\System32\bash.exe`` + #. In the "Program/script" text field, add ``C:\Windows\System32\bash.exe``. - #. In the "Add arguments (optional)" text field, add ``-c "sudo /path/to/file.sh"`` + #. In the "Add arguments (optional)" text field, add ``-c "sudo /path/to/file.sh"``. #. Click "OK" to confirm. From f90328570f7adc49673573a090aa80ab66dba465 Mon Sep 17 00:00:00 2001 From: Carl Simon Adorf Date: Mon, 2 Nov 2020 10:40:38 +0100 Subject: [PATCH 11/13] Tweak the WSL instructions. --- docs/source/intro/install_system.rst | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/source/intro/install_system.rst b/docs/source/intro/install_system.rst index f89dfe4a6b..d74caf5de6 100644 --- a/docs/source/intro/install_system.rst +++ b/docs/source/intro/install_system.rst @@ -45,13 +45,13 @@ This is the *recommended* installation method to setup AiiDA on a personal lapto .. tabbed:: Windows Subsystem for Linux - *The following instructions are for setting up AiiDA on WSL 2 in combination with Ubuntu.* + *The following instructions are for setting up AiiDA on WSL 1/2 in combination with Ubuntu.* #. Installing RabbitMQ: - #. On WSL 1, the `Windows native RabbitMQ `_ should be installed and started. + * (WSL 1) Install and start the `Windows native RabbitMQ `_. - #. On WSL 2, RabbitMQ can be installed inside WSL: + * (WSL 2) Install RabbitMQ inside the the WSL: .. code-block:: console @@ -69,20 +69,20 @@ This is the *recommended* installation method to setup AiiDA on a personal lapto .. dropdown:: How to setup WSL to automatically start services after system boot. - Create a ``.sh`` file containing the following lines: + Create a file ``start_aiida_services.sh`` containing the following lines: .. code-block:: console service postgresql start service rabbitmq-server start # Only for WSL 2! - - Make the file executeable, and make it editable only by root users by typing + and store it in your preferred location, e.g., the home directory. + Then make the file executeable, and editable only by root users with: .. code-block:: console - $ chmod a+x,go-w /path/to/file.sh - $ sudo chown root:root /path/to/file.sh + $ chmod a+x,go-w /path/to/start_aiida_services.sh + $ sudo chown root:root /path/to/start_aiida_services.sh Next, run @@ -94,7 +94,7 @@ This is the *recommended* installation method to setup AiiDA on a personal lapto .. code-block:: sh - ALL=(root) NOPASSWD: /path/to/file.sh + ALL=(root) NOPASSWD: /path/to/start_aiida_services.sh replacing ```` with your Ubuntu username. This will allow you to run *only* this specific ``.sh`` file with ``root`` access (without password), without lowering security on the rest of your system. @@ -119,13 +119,13 @@ This is the *recommended* installation method to setup AiiDA on a personal lapto #. In the "Program/script" text field, add ``C:\Windows\System32\bash.exe``. - #. In the "Add arguments (optional)" text field, add ``-c "sudo /path/to/file.sh"``. + #. In the "Add arguments (optional)" text field, add ``-c "sudo /path/to/start_aiida_services.sh"``. #. Click "OK" to confirm. #. Click "OK" to confirm the task. - The details of this task can be tweaked to fit your needs - use your own judgement as to when you would like the services to be started. + You can tweak other details of this task to fit your needs. .. tabbed:: Other From 778e31ae9dece1293969b13b566cf1dc9050e6b2 Mon Sep 17 00:00:00 2001 From: Carl Simon Adorf Date: Mon, 2 Nov 2020 13:56:44 +0100 Subject: [PATCH 12/13] Extend docker instructions. To retain more from the previous advanced installation instructions. --- docs/source/intro/run_docker.rst | 54 +++++++++++++++++++++++++++++--- 1 file changed, 49 insertions(+), 5 deletions(-) diff --git a/docs/source/intro/run_docker.rst b/docs/source/intro/run_docker.rst index 6229ae1e4b..8083572ce9 100644 --- a/docs/source/intro/run_docker.rst +++ b/docs/source/intro/run_docker.rst @@ -8,6 +8,10 @@ Run AiiDA via a Docker image The AiiDA team maintains a `Docker `__ image on `Docker Hub `__. This image contains a fully pre-configured AiiDA environment which makes it particularly useful for learning and testing purposes. +.. caution:: + + All data stored in a container will persist only over the lifetime of that particular container unless you use volumes (see instructions below). + .. panels:: :container: container-lg pb-3 :column: col-lg-12 p-2 @@ -50,12 +54,52 @@ This image contains a fully pre-configured AiiDA environment which makes it part ✓ rabbitmq: Connected as amqp://127.0.0.1?heartbeat=600 ✓ daemon: Daemon is running as PID 1795 since 2020-05-20 02:54:00 + --- + + **Use container interactively** + + To "enter" the container and run commands directly in the shell, use: + + .. code-block:: console + + $ docker exec -it --user aiida aiida-container /bin/bash + + This will drop you into the shell within the container as the user "aiida". + + --- + + **Persist data across different containers** + + If you stop the container and start it again, any data you created will persist. + + .. code-block:: console + + $ docker stop aiida-container + $ docker start aiida-container + + However, if you remove the container, **all data will be removed as well**. + + .. code-block:: console + + $ docker stop aiida-container + $ docker rm aiida-container + + The preferred way to persistently store data is to `create a volume `__. + To create a simple volume, run: + + .. code-block:: console + + $ docker volume create my-data + + Then make sure to mount that volume when running the aiida container: + + .. parsed-literal:: + + $ docker run -d --name aiida-container --mount source=my-data,target=/tmp/my_data aiidateam/aiida-core:\ |release|\ + + Starting the container with the above command, ensures that any data stored in the ``/tmp/my_data`` path within the container is stored in the ``my-data`` volume and therefore persists even if the container is removed. + .. link-button:: intro:get_started:next :type: ref :text: What's next? :classes: btn-outline-primary btn-block font-weight-bold - -.. caution:: - - All data stored in the container will persist as long as you restart the same container, e.g., with (``docker start aiida-container``), however if you remove the container, all data will be lost. - Use `volumes `__ to share data between containers and ensure its persistency on the host machine. From a9c57c6195c843c7f77d179e3f28dbd1deb3aa7f Mon Sep 17 00:00:00 2001 From: Simon Adorf Date: Thu, 5 Nov 2020 13:52:39 +0100 Subject: [PATCH 13/13] Replace all 'hint' admonitions with 'tip'. --- docs/source/intro/install_conda.rst | 2 +- docs/source/intro/install_system.rst | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/source/intro/install_conda.rst b/docs/source/intro/install_conda.rst index 56144727f3..fb0bc101f2 100644 --- a/docs/source/intro/install_conda.rst +++ b/docs/source/intro/install_conda.rst @@ -63,7 +63,7 @@ This is the recommended method for users on shared systems and systems where the The verdi daemon(s) must be restarted after a system reboot. - .. hint:: + .. tip:: Do not start more daemons then there are physical processors on your system. diff --git a/docs/source/intro/install_system.rst b/docs/source/intro/install_system.rst index d74caf5de6..7d11b8e6d8 100644 --- a/docs/source/intro/install_system.rst +++ b/docs/source/intro/install_system.rst @@ -132,7 +132,7 @@ This is the *recommended* installation method to setup AiiDA on a personal lapto #. Install RabbitMQ following the `instructions applicable to your system `__. #. Install PostgreSQL following the `instructions applicable to your system `__. - .. hint:: + .. tip:: Alternatively use the :ref:`pure conda installation method `. @@ -153,7 +153,7 @@ This is the *recommended* installation method to setup AiiDA on a personal lapto (aiida) $ pip install aiida-core (aiida) $ reentry scan - .. hint:: + .. tip:: See the `venv documentation `__ if the activation command fails. The exact command for activating a virtual environment differs slightly based on the used shell. @@ -228,7 +228,7 @@ This is the *recommended* installation method to setup AiiDA on a personal lapto The verdi daemon(s) must be restarted after a system reboot. - .. hint:: + .. tip:: Do not start more daemons then there are physical processors on your system.