add instructions for cloud deployment

This marks the transition of this repository from being merely an
ansible playbook for deployment of cloud services to becoming an entry
point for users who simply want to run pre-built images on cloud
infrastructures.

Besides migrating updates from the Quantum Mobile playbook for
virtualbox, this commit mainly updates the README.

README.md

@@ -1,45 +1,90 @@
-# Quantum Mobile Workhorse
+# Quantum Mobile Workhorse Server
 
-This ansible playbook installs a 
-[Quantum Mobile](https://github.com/marvel-nccr/quantum-mobile)
-workhorse server on a remote virtual machine (tested on OpenStack, Amazon Web Services and Huawei Cloud).
+This repository contains instructions for deploying [Quantum Mobile](www.materialscloud.org/quantum-mobile) on a remote server (for running Quantum Mobile directly on your laptop or workstation, see [here](https://github.com/marvel-nccr/quantum-mobile) instead).
+
+Quantum Mobile has been deployed successfully on Amazon Web Services, Google Compute Cloud, Huawei Cloud, Openstack as well as on bare metal servers.
+
+## Launching a pre-built image
+
+For some platforms, we offer pre-built images on the [release page](https://github.com/marvel-nccr/ansible-playbook-workhorse/releases) that allow you to launch a new Quantum Mobile VM in a few clicks.
+
+--Video Placeholder--
+
+## Building the image yourself
+
+If your platform of choice is not listed above or if you would like to customize your deployment of Quantum Mobile, you'll need to build the image yourself.
+
+This procedure is automated entirely using the [ansible playbook](https://docs.ansible.com/ansible/latest/user_guide/playbooks.html) in this repository.
 
 ### Prerequisites
 
 #### Server
-- A server running Ubuntu 18.04 
+- A server running Ubuntu 18.04 LTS
   Can be hardware or virtual machine (tested on OpenStack, Amazon Web Services and Huawei Cloud).
-- Access to server via SSH key
+- At least 12GB disk size (including Ubuntu); better 15GB or more
+- Access to server via SSH key as user with sudo rights
 
-Note on security rules:
-- SSH access requires port 22 to be open
-- You may want to open further port for other servers:
+Security rules:
+- Port 22 open (for SSH access)
+- You may want to open further ports (optional):
   - port 8888 to connect to Jupyter Notebook Servers (AiiDA lab)
   - port 5000 to connect to the AiiDA REST API
 
 #### Client
 - [python](https://www.python.org/)
 - [git](https://git-scm.com)
 
-To get set up, run the following on your client:
+To get set up, run the following on your client (e.g. your laptop -- *not* on the server itself):
 ```
 git clone https://github.com/marvel-nccr/ansible-playbook-workhorse.git
 cd ansible-playbook-workhorse
 pip install -r requirements.txt  # installs python requirements
 ansible-galaxy install -r requirements.yml  # installs ansible roles
 ```
 
-### Set up Virtual Machine
+### Provisioning the server
 
-1. select aws/os host in `./hosts` file
-1. adapt path to your ssh key (to connect as admin to the VM)
-   in corresponding `./group_vars/*.yml` file
-1. Tune the `globalconfig.yml` file (in particular `vm_user`, `vm_memory`, `vm_cpus`)
-1. edit the `playbook.yml` file replacing, in the role `"add user {{ vm_user }} with key" (marvel-nccr.add_user`),
-   under `add_user_public_key`, the correct path to the public key the students will have: 
-1. Tune what you want to have in the machine in the `playbook.yml` file
+1. Add your server to the [ansible inventory](https://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html) in the `hosts` file, e.g.
+   ```
+   [myplatform]
+   qm ansible_host=1.1.1.1
+   ```
+1. Adapt the corresponding `./group_vars/*.yml` file (or create your own), with
+   * the path to your private SSH key for connecting to the server
+   * the user to connect as via SSH
+1. Tune the `globalconfig.yml` file, in particular:
+   * `vm_user`: the user for which to install the simulation environment (usually *not* the admin user you are connecting as)
+   * `vm_memory`, `vm_cpus`
+1. (optional) adaptation of the ansible `playbook.yml`
+   * You want to preload a SSH public key for the `vm_user`?  
+   Then uncomment the `"add user {{ vm_user }} with key"` role and adjust the path to the public key in the lookup for the `add_user_public_key` variable
+   * Add/remove further roles depending on what you want to have in the image
 1. run `ansible-playbook playbook.yml`
 
+Your server should now be fully deployed and operational.
+
+You can log in to the server as the `vm_user` via the public SSH key you provided.
+
+### Saving the image
+
+Before creating an image from the server, here are a few tricks to reduce image size:
+
+1. Remove unnecessary files:  
+   `ansible-playbook playbook.yml --extra-vars "clean=true" --tags qm_customizations,simulationbase`
+1. Follow instructions to remove SSH key for image publication  
+   AWS: `sudo shred -u /etc/ssh/*_key /etc/ssh/*_key.pub`  
+
+1. Clear bash history: `cat /dev/null > ~/.bash_history && history -c && exit`
+
+1. Shut down instance (note: this will also clear temporary data in `/tmp`)
+
+1. Create image (following instructions for your platform)
+
+1. Publish image
+   [GCP](https://cloud.google.com/compute/docs/images/managing-access-custom-images#share-images-publicly)
+   [AWS]()
+
+
 ### Tweaks
 
 #### Slow network connection

globalconfig.yml

@@ -1,18 +1,22 @@
 ---
 # Caution: This file is read by vagrant, ansible and bash
 vm_name: "Quantum Mobile"
-vm_version: "19.09.0-tutorial"
+vm_version: "20.05.0"
 vm_description: "A Virtual Machine for Computational Materials Science"
 vm_url: "https://github.com/marvel-nccr/marvel-virtualmachine"
 vm_author: "MARVEL NCCR and MaX CoE"
 vm_author_url: "http://nccr-marvel.ch"
+vm_release_notes_file: "${HOME}/Desktop/release_notes.txt"
+vm_readme_file: "${HOME}/Desktop/README.md"
 
 # VM configuration
 vm_hostname: "workhorse"
 vm_user: "max"
-vm_memory: 32768
-vm_cpus: 8
+vm_memory: 4000
+vm_cpus: 2
 vm_codes_folder: "${HOME}/codes"
+vm_data_folder: "/usr/local/share"
 vm_examples_folder: "${HOME}/examples"
 vm_headless: True
 vm_browser: "chromium"
+vm_scheduler: slurm # 'slurm' or 'direct'

group_vars/aws.yml

@@ -1,6 +1,6 @@
 ansible_user: ubuntu
-ansible_ssh_common_args: -i ./keys/aiida-tutorial.pem -o StrictHostKeyChecking=no
+ansible_ssh_common_args: -i ./keys/quantum-mobile.pem -o StrictHostKeyChecking=no
 cloud_platform: aws
 
-# modify this line to switch to pyhon2
+# modify this line to switch to python2
 ansible_python_interpreter: /usr/bin/python3

group_vars/gcp.yml

@@ -0,0 +1,6 @@
+ansible_user: max
+ansible_ssh_common_args: -i ./keys/gcp-max.pem -o StrictHostKeyChecking=no
+cloud_platform: gcp
+
+# modify this line to switch to pyhon2
+ansible_python_interpreter: /usr/bin/python3

group_vars/hwcloud.yml

@@ -2,5 +2,5 @@ ansible_user: root
 ansible_ssh_common_args: -i ./keys/aiida-tutorial-master -o StrictHostKeyChecking=no
 cloud_platform: hwcloud
 
-# modify this line to switch to pyhon2
+# modify this line to switch to python2
 ansible_python_interpreter: /usr/bin/python3

hosts

@@ -1,10 +1,11 @@
 [aws]
-aws-vm1 ansible_host=18.203.221.13
-#aws-vm3 ansible_host=176.34.156.97
+quantum-mobile ansible_host=34.250.68.129
 
-[hwcloud]
+#[hwcloud]
 #hw-vm2 ansible_host=114.116.226.33
 
-[openstack]
+#[openstack]
 #openstack-vm1 ansible_host=148.187.96.53
 
+#[gcp]
+#quantum-mobile ansible_host=34.65.240.28 ansible_user=leopold

playbook.yml

@@ -23,31 +23,27 @@
 - name: set up the VM
   hosts: all
   vars:
+    - release_notes: false
+    - release_notes_vm_user: "{{ vm_user }}"
     - run_tests: False
     - clean: False
     - root_user: root
   roles: 
-    - name: add user {{ vm_user }} with key
-      role: marvel-nccr.add_user
-      vars:
-        add_user_name: "{{ vm_user }}"
-        add_user_public_key: "{{ lookup('file', './keys/aiida-tutorial-max.pub') }}"
-        add_user_groups:
-          - "{{ vm_user }}"
-          - sudo
-        add_user_passwordless_sudo: true
+  # - name: add user {{ vm_user }} with key
+  #   role: marvel-nccr.add_user
+  #   vars:
+  #     add_user_name: "{{ vm_user }}"
+  #     add_user_public_key: "{{ lookup('file', './keys/aiida-tutorial-max.pub') }}"
+  #     add_user_groups:
+  #       - "{{ vm_user }}"
+  #       - sudo
+  #     add_user_passwordless_sudo: true
     - role: marvel-nccr.simulationbase
       tags: simulationbase
       vars:
         simulationbase_vm_user: "{{ vm_user }}"
         simulationbase_hostname: "{{ vm_hostname }}"
         simulationbase_codes_folder: "{{ vm_codes_folder }}"
-    - role: marvel-nccr.ubuntu_desktop
-      tags: ubuntu_desktop
-      vars:
-        ubuntu_desktop_browser: "{{ vm_browser }}"
-        ubuntu_desktop_vm_user: "{{ vm_user }}"
-      when: not vm_headless
     - role: marvel-nccr.quantum_mobile_customizations
       tags: qm_customizations
       vars:
@@ -56,9 +52,10 @@
         qm_customizations_vm_name: "{{ vm_name }}"
         qm_customizations_vm_version: "{{ vm_version }}"
         qm_customizations_vm_author: "{{ vm_author }}"
+        qm_customizations_vm_cpus: "{{ vm_cpus }}"
         qm_customizations_headless: "{{ vm_headless }}"
-        qm_customizations_codes_folder: "{{ vm_codes_folder }}"
         qm_customizations_examples_folder: "{{ vm_examples_folder }}"
+        qm_customizations_data_folder: "{{ vm_data_folder }}"
     - role: marvel-nccr.editors
       tags: editors
       vars:
@@ -72,6 +69,8 @@
         slurm_hostname_service: true
     - role: marvel-nccr.quantum_espresso
       tags: quantum_espresso
+      vars:
+        quantum_espresso_data_folder: "{{ vm_data_folder }}"
     - role: marvel-nccr.yambo
       tags: yambo
     - role: marvel-nccr.fleur
@@ -82,19 +81,32 @@
       tags: cp2k
       #vars:
       #  cp2k_installation_route: "apt"
+    # Wannier90 3.0 bundled with QE
     - role: marvel-nccr.wannier90
       tags: wannier90
+    - role: marvel-nccr.wannier_tools
+      tags: wannier_tools
+      #- role: marvel-nccr.bigdft
+      #  tags: bigdft 
     - role: marvel-nccr.aiidalab
       become: true
       become_user: "{{ vm_user }}"
       tags: aiidalab
       vars:
         aiidalab_headless: "{{ vm_headless }}"
+    # note: installing aiida role after aiidalab role, since aiida role has latest AiiDA version
     - role: marvel-nccr.aiida
       tags: aiida 
       become: true
       become_user: "{{ vm_user }}"
       vars:
-        aiida_code_folder: "{{ vm_codes_folder }}"
-        aiida_localhost_cpus: "{{ vm_cpus }}"
+        aiida_source_folder: "{{ vm_codes_folder }}"
+        aiida_data_folder: "{{ vm_data_folder }}/aiida"
+        aiida_computer_cpus: "{{ vm_cpus }}"
+        aiida_computer_scheduler: "{{ vm_scheduler }}"
         aiida_examples_folder: "{{ vm_examples_folder }}"
+        aiida_components:
+          - computers
+          - plugins
+          - pseudopotentials
+          - examples

requirements.yml

@@ -1,36 +1,37 @@
 ---
+- src: gantsign.default-web-browser
+  version: 3.2.0
 - src: marvel-nccr.ansible_prerequisites
   version: v0.1.0
 - src: marvel-nccr.current_user
   version: v0.1.0
 - src: marvel-nccr.add_user
   version: v0.2.1
 - src: marvel-nccr.simulationbase
-  version: v1.0.1
-- src: marvel-nccr.ubuntu_desktop
-  version: v1.0.0
+  version: v1.0.7
 - src: marvel-nccr.quantum_mobile_customizations
-  version: v1.0.0
+  version: v1.0.6
 - src: marvel-nccr.editors
-  version: v1.0.1
+  version: v1.0.2
 - src: marvel-nccr.slurm
-  version: v1.0.0
+  version: v1.1.0
 - src: marvel-nccr.quantum_espresso
-  version: v1.0.1
+  version: v1.1.3
 - src: marvel-nccr.yambo
-  version: v1.0.1
+  version: v1.1.0
 - src: marvel-nccr.fleur
-  version: v1.0.1
+  version: v1.0.2
 - src: marvel-nccr.siesta
-  version: v1.0.1
+  version: v1.0.4
 - src: marvel-nccr.cp2k
-  version: v1.0.1
+  version: v1.1.1
+- src: marvel-nccr.bigdft
+  version: v1.0.0
 - src: marvel-nccr.wannier90
+  version: v1.1.2
+- src: marvel-nccr.wannier_tools
   version: v1.0.0
 - src: marvel-nccr.aiida
-  version: v1.0.0
-#- src: https://github.com/marvel-nccr/ansible-role-aiida.git
-#  version: develop
-#  name: marvel-nccr.aiida
+  version: v2.1.8
 - src: marvel-nccr.aiidalab
-  version: v1.0.0
+  version: v1.1.3

roles/.gitignore

@@ -1 +1,2 @@
 marvel-nccr.*
+gantsign.*