Aperture is Redbrick's fleet of hardware that was installed in May 2022 by distro, pints, skins, cawnj, ymacomp
+and arkues.
It consists of:
+3x Dell R6515
+| CPU | +RAM | +Storage | +
|---|---|---|
| AMD 7302P 3GHz, 16C/32T, 128M, 155W, 3200 | +2x 16GB RDIMM, 3200MT/s Dual Rank | +4x 2TB SATA HDDs (hardware RAID) | +
2x Ubiquiti USW Pro
+The three servers are named glados , wheatley and chell.
The firewall is called mordor, and the two 24-port switches are called rivendell and isengard.
The IP address range for the aperture subnet is 10.10.0.0/24, with 10.10.0.0/16 being used for user VMs.
| Hostname | +Internal Address | +External Address | +Purpose | +
|---|---|---|---|
mordor |
+10.10.0.1 | +N/A | +Firewall | +
rivendell |
+10.10.0.2 | +N/A | +Switch | +
isengard |
+10.10.0.3 | +N/A | +Switch | +
glados |
+10.10.0.4 | +136.206.16.4 | +Server | +
wheatley |
+10.10.0.5 | +136.206.16.5 | +Server | +
chell |
+10.10.0.6 | +136.206.16.6 | +Server | +
Note
+Blue cables are used for production network.
+nexus is the name of the KVM switch. It's internal IP address is 10.10.0.10.
glados is connected on port 1, wheatley on port 2, and chell on port 3.
Note
+Yellow cables are used for KVM network.
+The new servers are all equipped with IDRACs. These still need to be configured.
+Note
+Red cables are used for IDRAC network.
+We have two address ranges that come in on a single redundant link, so we're exchanging that redundant link for two
+separate links, each taking responsibility for an address range (136.26.15.0/24 and 136.206.16.0/24). So we're surrendering
+redundancy to gain uptime/connectivity during the switchover only. Once the new servers are production ready, we can
+recombine the link to regain the redundancy.
Redbrick uses ansible to manage its infrastructure. This document describes the procedures and some tips to get the most
+out of it.
Ansible is a python package, so you'll need to install python first. On Debian/Ubuntu, you can do this with:
+pip install ansible
+Ansible uses ssh to connect to the remote hosts. You'll need to set up your ssh key so that you can connect to the hosts
+without constant prompts for passwords.
This is used a phonebook of sorts for ansible. It tells ansible which hosts to connect to, and what user to use.
+[aperture]
+glados
+wheatley
+chell
+
+[aperture:vars]
+ansible_user= <your username>
+++Contact @distro for a fully populated file.
+
ansible all -m ping
+This should connect to all the hosts in the aperture group, and run the ping module. If it works, you're good to go!
Ansible playbooks are a set of instructions for ansible to run. They're written in YAML, and are usually stored in a file
+called playbook.yml.
Ansible playbooks are written in YAML. The basic structure is:
+- hosts: <group name>
+ tasks:
+ - name: <task name>
+ <module name>:
+ <module options>
+- hosts: aperture
+ tasks:
+ - name: Install curl
+ apt:
+ name: curl
+ state: present
+This playbook will connect to all the hosts in the aperture group, and run the apt module with the name and state
+options.
ansible-playbook playbook.yml -i hosts
+Redbrick's ansible configuration is stored in the ansible repository. There's
+some more documentation there on each playbook.
Ansible's documentation is available here.
+Sometimes, when running a playbook, you'll get an error like this:
+TASK [apt : apt update packages to their latest version and autoclean] ***************************************************************************************************
+fatal: [wheatley]: FAILED! => {"changed": false, "msg": "Failed to update apt cache: unknown reason"}
+fatal: [chell]: FAILED! => {"changed": false, "msg": "Failed to update apt cache: unknown reason"}
+fatal: [glados]: FAILED! => {"changed": false, "msg": "Failed to update apt cache: unknown reason"}
+This is because the Hashicorp apt key has expired. To fix this, uncomment the hashicorp-apt task in the playbook.
This VM is an ephemeral machine that can be placed on any nomad client that has the qemu driver enabled.
+It acts as the point of ingress for Aperture, with ISS and our firewall allowing traffic to reach it's IP address externally. The VM is configured as a Nomad client itself, in the ingress node pool to ensure that only ingress-type allocations are placed there (like traefik). Those services can proxy requests from the Bastion VM to internal services using consul's service DNS resolution, it's service mesh, or by plain IP and port.
cloud-init is given a static address during the initialisation phase to configure the interface. This ensures that, even if it is replanned, it will be able to accept traffic.
The base image that the VM uses is a Debian 12 qcow file. After all configuration was done, the size of the image is ~3.2GB. The image can be used to create replicas of the ingress on other external IP addresses, creating more availability if needed.
You'll need to ensure the hosts have a bridge device configured to ensure that the networking aspect of the VM can function. See theredbrick/nomad repo for more information about the steps needed for that.
You'll need a webserver to serve the cloud-init configs. There may be another solution to this in the near future, but for now, wheatley:/home/mojito/tmp/serve contains the configurations.
Plan the Nomad job and wait for the allocation to be created. If you used the correct image (for example a backup of the qcow file) the virtual machine should be configured and should connect as normal to the Consul and Nomad clusters and become eligible for allocations. If you started from scratch, then use the ansible/redbrick-ansible.yml playbook in the redbrick/nomad repo and ensure that the hosts file is up to date.
For security's sake, there is no root login and no user accounts on the bastion VM. This is an attempt to make the node more secure. If you need to make changes, you should change the base image and apply that. The less vulnerabilities that are discovered on the bastion VM, the happier we can keep ISS and the safer Redbrick will be.
+ + + + + + + + + + + + + + + + + + + + +The firewall is set up using the personal setup type, using the elected-admins@redbrick.dcu.ie account (stored in pwsafe
+2FA is stored on the same device as the Github 2FA code.
The UDM Pro is not set up for automatic updates for reliability reasons.
+We have a 10 GB/s link to DCU's core.
+The current elected admins should all have access to the rbadmin account on the firewall. Rootholders should not have
+access to the firewall unless they are explicity granted access.
The owner account of the unifi equipment is rbadmins (email: elected-admins@redbrick.dcu.ie) with the password stored
+in pwsafe under unifi.
There is a "super admin" account that can be used for local access only, details are stored in pwsafe under
+udmpro-super-admin.
The UDM Pro should be kept up to date at all times using the web interface. Please ensure there are no breaking changes before
+updating.
Error
+This is to prevent a bad update from breaking the UDM Pro and thus the entire network.
+If you are confident that Unifi can produce stable updates, you may turn it on, however please let the next admins
+know that you have done this (and update these docs with a comment!).
SSH is enabled to allow for rollbacks in case of a bad update (I warned you!).
+Remote access is disabled as it should not be needed, the admin vpn should provide enough access for you.
+If it is enabled in future, please update these docs with your reasons.
Backups are configured to run every week at 1am on a Sunday. 20 backups are stored at a time, therefore storing 20 weeks
+of configuration. This should be plenty of time to recover from a bad configuration change.
Mordor is natted when it accesses the Internet. This is because the link address between it and DCU is on a private address.
+This natting is used only for the UDM pro device itself, not for the 136.206.16.0/24 network, and is to allow the UDM
+box itself to access the Internet.
The 136.206.16.0/24 network is routed down to the UDM pro box, within the DCU network. Essentially there is a route in
+DCU's network that says "if you want to access 136.206.16.0/24 go to mordor".
Icecast is a streaming server that we currently host on aperture.
+We stream DCUFm's Broadcasts to their apps via a stream presented on dcufm.redbrick.dcu.ie.
The configuration file for icecast is located in the nomad config repo.
+It should just be a case of running nomad job plan clubs-socs/dcufm.hcl to plan and run the job.
Note
+The job may bind to either the internal or external address. Ensure that if you make a change to the config, you
+inform DCUfm that they may need to switch which server they use.
DCUfm use butt on a desktop in their studio to stream to Icecast.
+The desktop must be connected to the VPN to ensure the stream stays up, and traefik doesn't reset the connection every
+10 seconds. The current icecast configuration for the server is 10.10.0.5:2333 or 136.206.16.5:2333 (see above note).
+Read more about it in this issue.
A shortcut to the VPN is available on the desktop (change a shortcut to the binary to include --connect profile.ovpn.
+See here).
This is a cheat sheet for DCUfm to help them stream to icecast.
+You'll need to connect to the Redbrick VPN to stream to icecast. You can do this by double clicking the shortcut on the desktop.
+You'll then need to go to bottom right corner of the screen and right click this icon:
+
A popup will appear, click connect. This will connect you to the VPN. It may take a second, but a window will pop up with
+a lot of text. The VPN will connect and then it'll close.
+
You should end up with an icon like this:
+
You're now connected to the VPN.
+You'll need to connect to icecast to stream to it. BUTT is the software we use to stream to icecast. You'll also find this
+on the desktop. Once its open, (and you're connected to the VPN), press the small "play" button in the top left corner. This
+will start your stream to the server.
The username and password should already be configured in the software. If not, ask a redbrick sysadmin
+for the login details.
Warning
+If you find that butt is not connecting, then you may need to switch which server you're connecting to. To do this,
+go to settings, and then the "Main" tab. In the dropdown, select either DCUfm 1 or DCUfm 2 (try both,
+one will definitely work).
Your stream will be saved automatically onto the desktop into a folder called Recordings YYYY (where YYYY is the
+current year), with the date and time of the recording, and the format .mp3. Take this file with you (via a USB or similar)
+if you want to keep it for later, it will not be kept on the desktop for long!
If you have any questions, please ask a redbrick sysadmin.
+ + + + + + + + + + + + + + + + + + + + +
+
+
+
+
+
+
It's nothing to do with cameras. See about for more information on the hardware.
+If you're a new admin, this is a cheat sheet for you. In order to get broadly up to speed and understand the content of
+these pages, I suggest you read the following:
So, you've hit a problem. Here's some quicklinks to some common problems:
+ + + + + + + + + + + + + + + + + + + + + +distro, wizzdom++Adapted from the redbrick/nomad repo's README
+
Good question!
+++Nomad is a simple and flexible scheduler and orchestrator to deploy and manage
+
+containers and non-containerized applications
+- Nomad Docs
All Nomad job related configurations are stored in the nomad directory.
The terminology used here is explained here. This is required reading.
+All of the job files are stored in the nomad directory. To deploy a Nomad job manually, connect to a host and run
nomad job plan path/to/job/file.hcl
+This will plan the allocations and ensure that what is deployed is the correct version.
+If you are happy with the deployment, run
+nomad job run -check-index [id from last command] path/to/job/file.hcl
+This will deploy the planned allocations, and will error if the file changed on disk between the plan and the run.
+You can shorten this command to just
+nomad job plan path/to/file.hcl | grep path/to/file.hcl | bash
+This will plan and run the job file without the need for you to copy and paste the check index id. Only use this once you are comfortable with how Nomad places allocations.
+nomad job stop -purge name-of-running-job
+nomad system gc
+nomad system reconcile summaries
+nomad system gc # (yes, again)
+nomad job plan path/to/job/file.hcl
+DCUfm - Icecast
+ + + + + + + + + + + + + + + + + + + + +The admin VPN is set up to allow admins to access the network from outside of DCU, giving them an IP address on the
+internal network for troubleshooting, testing and integrating.
If you just want to create a new client configuration, go here: adding a new client
+Installed OpenVPN using this script on Glados.
+To add a new client, run the following command (as root) on Glados:
+bash /root/ovpn/openvpn-install.sh
+You will be prompted to add a new client, enter a name for the client and then the script will generate a new client.
+It will be saved in /root/[client name].ovpn.
To revoke a client, run the following command (as root) on Glados:
+bash /root/ovpn/openvpn-install.sh
+You will be prompted to revoke a client, enter the name of the client you want to revoke.
+To connect to the VPN, you will need to download the client configuration file from glados and then import it into your
+OpenVPN client.