Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/cocktail #693

Merged
merged 30 commits into from
Jan 4, 2023
Merged
Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
4f84931
Refactored creation of OpenStack components and updated group_vars + …
pneerincx Dec 23, 2022
1a22577
Bugfixes after refactoring.
pneerincx Dec 23, 2022
1415a12
Fixed merge conflict
pneerincx Dec 23, 2022
5dd7eac
Bugfixes after refactoring and added IP addresses for new bb-porch an…
pneerincx Dec 23, 2022
7738799
Fixed bug in static_hostname_lookup role and added missing values for…
pneerincx Dec 23, 2022
2091df8
Removed deprecated plays to create OpenStack components.
pneerincx Dec 27, 2022
4511296
Removed redundant template var for CIFS credentials from group_vars/w…
pneerincx Dec 27, 2022
d0c80b9
Bugfix for online_docs role when stack does not use a repo_manager.
pneerincx Dec 27, 2022
0e17532
Bugfixes for working with new OpenStack roles.
pneerincx Dec 27, 2022
ecfaa01
Fixed wrong gateway IP for nb_internal_management_13 network.
pneerincx Dec 27, 2022
70abd68
Fixed wrong value for local_volume_size_extra on nb-dai.
pneerincx Dec 27, 2022
c87bdd0
Wrapped long line.
pneerincx Dec 27, 2022
7c137ce
Bugfix: geerlingguy.security fix sshd_config 2FA on tunnel jumphost.
pneerincx Dec 27, 2022
30e70a4
Ignore dir and link timestamps when rsyncing figlet fonts.
pneerincx Dec 28, 2022
9a2f162
Relocated some vars to host_vars, which can be used without jumphost …
pneerincx Dec 28, 2022
2495f91
Added new openstack.yml wrapper playbook for both openstack_networkin…
pneerincx Dec 28, 2022
9fe1428
Fixed linter issues in new roles.
pneerincx Dec 28, 2022
c375c08
Updated README.md
pneerincx Dec 28, 2022
cb4c482
Updated README.md
pneerincx Dec 28, 2022
76c382c
Removed "assign_floating_ip: true" from hc-repo (copy-paste error).
pneerincx Dec 28, 2022
607e989
Make sure only hosts with a cloud_flavor attribute are added to ip_ad…
pneerincx Dec 28, 2022
746b312
Fixed linter issue.
pneerincx Dec 30, 2022
f348f69
Improved hack to decide when to create routers in management networks.
pneerincx Dec 30, 2022
4c585d1
Added missing VLAN990 IPs to ip_addresses.yml for betabarrel and copp…
pneerincx Dec 30, 2022
d962d0b
Fixed wrong task order in smb_server role.
pneerincx Dec 30, 2022
f407415
Update host_vars/README.txt
pneerincx Jan 4, 2023
e089f77
Fixed typo in README.
pneerincx Jan 4, 2023
36eb727
Merge branch 'feature/cocktail' of https://github.com/pneerincx/leagu…
pneerincx Jan 4, 2023
8dfa303
Update roles/openstack_networking/defaults/main.yml
pneerincx Jan 4, 2023
69c9413
Merge branch 'develop' into feature/cocktail
pneerincx Jan 4, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 18 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ This repo currently contains code and configs for the following clusters:

* Talos: Development cluster hosted by the [Center for Information Technology (CIT) at the University of Groningen](https://www.rug.nl/society-business/centre-for-information-technology/).
* Gearshift: [UMCG](https://www.umcg.nl) Research IT production cluster hosted by the [Center for Information Technology (CIT) at the University of Groningen](https://www.rug.nl/society-business/centre-for-information-technology/).
* Nibbler: [UMCG](https://www.umcg.nl) Research IT production cluster hosted by the [Center for Information Technology (CIT) at the University of Groningen](https://www.rug.nl/society-business/centre-for-information-technology/).
* Hyperchicken: Development cluster hosted by [The European Bioinformatics Institute (EMBL-EBI)](https://www.ebi.ac.uk/) in the [Embassy Cloud](https://www.embassycloud.org/).
* Fender: [Solve-RD](solve-rd.eu/) production cluster hosted by [The European Bioinformatics Institute (EMBL-EBI)](https://www.ebi.ac.uk/) in the [Embassy Cloud](https://www.embassycloud.org/).

Expand Down Expand Up @@ -79,7 +80,8 @@ The clusters use the following types of storage systems / folders:

Some other stacks of related machines are:

* docs_library: web servers hosting documentation.
* ```docs_library```: web servers hosting documentation.
* ```jenkins_server```: Continues Integration testing server.
* ...: iRODS machines

## Deployment phases
Expand All @@ -93,8 +95,8 @@ Deploying a fully functional stack of virtual machines from scratch involves the
* For the _Shikra_ cloud, which hosts the _Talos_ and _Gearshift_ HPC clusters
we use the ansible playbooks from the [hpc-cloud](https://git.webhosting.rug.nl/HPC/hpc-cloud) repository
to create the OpenStack cluster.
* For other HPC clusters we use OpenStack clouds from other service providers as is.
3. Create, start and configure virtual machines on an OpenStack cluster to create a Slurm HPC cluster.
* For other HPC clusters we use OpenStack clouds from other service providers _as is_.
3. Create, start and configure virtual networks and machines on an OpenStack cluster.
* This repo.
4. Deploy bioinformatics software and reference datasets.
* Off topic for this repo.
Expand All @@ -103,7 +105,7 @@ Deploying a fully functional stack of virtual machines from scratch involves the

---

## Details for phase 3. Create, start and configure virtual machines on an OpenStack cluster to create a Slurm HPC cluster.
## Details for phase 3. Create, start and configure virtual machines on an OpenStack cluster.

#### 0. Clone this repo and configure Python virtual environment.

Expand Down Expand Up @@ -355,18 +357,23 @@ Use the **_entire_** strings as the ```hash``` values in ```group_vars/[stack_na

#### 8. Running playbooks.

There are two playbooks:
There are two _wrapper playbooks_:

1. `deploy-os_servers.yml`:
* Creates virtual resources in OpenStack: networks, subnets, routers, volumes and finally the virtual machines.
1. `openstack.yml`:
* Creates virtual resources in OpenStack: networks, subnets, routers, ports, volumes and finally the virtual machines.
* Interacts with the OpenstackSDK / API on localhost.
* Uses a static inventory from `static_inventories/*.yaml` parsed with our custom inventory plugin `inventory_plugins/yaml_with_jumphost.py`
1. `cluster.yml`:
* Configures the virtual machines created with the `deploy-os_servers.yml` playbook.
* Configures the virtual machines created with the `openstack.yml` playbook.
* Has no dependency on the OpenstackSDK / API.
* Uses a static inventory from `static_inventories/*.yaml` parsed with our custom inventory plugin `inventory_plugins/yaml_with_jumphost.py`

##### deploy-os_servers.yml
The _wrapper playbooks_ execute several _roles_ in the right order to create the complete `stack`.
_Playbooks_ from the `single_role_playbooks/` or `single_group_playbooks/` sub directories can be used to
(re)deploy individual roles or all roles for only a certain type of machine (inventory group), respectively.
These shorter subset _playbooks_ can a lot of time during development, testing or regular maintenance.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can a lot
something is missing here


##### openstack.yml

* Login to the OpenStack web interface -> _Identity_ -> _Application Credentials_ -> click the _Create Application Credential_ button.
This will result in a popup window: specify _Name_, _Expiration Date_, _Expiration Time_, leave the rest empty / use defaults
Expand All @@ -383,11 +390,11 @@ There are two playbooks:
#
source ./[Application_Credential_Name]-openrc.sh
#
# Configure this repo for deployment of a specifc stack.
# Configure this repo for deployment of a specific stack.
#
source ./lor-init
lor-config [stack_prefix]
ansible-playbook deploy-os_servers.yml
ansible-playbook openstack.yml
```

##### cluster.yml
Expand Down
252 changes: 0 additions & 252 deletions create-docs-server.yml

This file was deleted.

Loading