update local (#1)

* Bugfix: Install with RTC Failing (PlexTrac#139) * Reworked mod_start to separate migration call * Updated RTC env vars for migration in podman * Reworked timer, podman sections, and reordered install for both * Bunch of podman related changes to get it to work with RTC * Added method to auth to CKE Registry specifically * Fixed minor issue with permissions on backups * Updated documentation * Version bump * var fix * Removed need for hard-coded plextrac username * Updated a chown * Add override for group creation * Correct position of flag -f * Bugfix: `PLEXTRAC_HOME` and Migration Container Names (PlexTrac#142) * Fixed containner naming on pulling logs * Bump version to 0.7.1 * Bugfix: Messaging Clarity (PlexTrac#146) * Bad wording in plextrac stop command * Updated migration container message to be more clear * Missing space * Bugfix: Removal of deprecated SMTP values (PlexTrac#145) * Removal of deprecated env values * Version bump * Bugfix: Check for migration container (PlexTrac#143) * Added actual container check * added podman support * Updated to validate against podman / docker correctly * missed a docker call * add a mac file for examples on running this on apple silicon (PlexTrac#112) Co-authored-by: Michael Burke <72173919+Michael-Burke@users.noreply.github.com> * sp\ Adds MinIO ENV Support For Future State * sp\ENV Var Fix For MinIO * Version bump * Updated method to be variable for future use * OPS-350: PGVector Enable (PlexTrac#144) * OPS-350: PGVector Enable * OPS-350: condition execution pg:superuser:bootstrap on script existence * Updated docker-compose.yml to add the new plextrac postgres image * Corrected new postgres image name * Reflecting the image change for Podman users * Version bump --------- Co-authored-by: Michael Burke <michael@plextrac.com> Co-authored-by: Michael Burke <72173919+Michael-Burke@users.noreply.github.com> * Bugfix: Suppress warning on normal updates (PlexTrac#149) * Fixed error message showing up on utility update when SKIP_SELF_UPGRADE wasn't set * Version bump to v0.7.5 * Bugfix: Backup of Uploads with unintended dependency (PlexTrac#150) Added --no-deps to the backing up of the uploads container * version bump to 0.7.6 * Bugfix: Migration logs, `plextrac info` postgres bug, better messaging (PlexTrac#152) * Fixed warning about missing postgres container due to hardcoded image name * Fixed printf usage on migration container log printing to console and exiting script * Fixed messaging around starting update so it doesn't look like it freezes after acknowledging warning * Updated some Podman documentation * Iterated version to 0.7.7 * sp\logix for 2.10 Contiguous Updates * Change var name in if logic * Bump version to 0.7.8 * Update src/_update.sh Co-authored-by: mmorgan-pt <99763568+mmorgan-pt@users.noreply.github.com> * sp\Minio Object Sotrage Service Declaration * Forgot to add a few other vars * Push version updates * Mm/semver bugfix (PlexTrac#155) * This seems to be working * some tweaks for more testing * cleanup of comments and another sort for the latest tag --------- Co-authored-by: ali <amyahya@gmail.com> * feat: add new env vars relating to cteam-56 * docs: update comment * fix: update ctem api url value and add it to notification services * fix: fix whitespace in docker compose * fix: another whitespace fix * sp\bump version * Ensure correct uploads volume ownership * podman postgres tweaks * Changes Podman PG startup image * missed one * 2.11 Bump Version * Update airgapped.md * remove duplicate podman pulls * bumped version * add podman remove for containers * close if statement * add entrypoint override for migrations container * put the entrypoint into the entrypoint line rather than array * sp\minio-image-update * minio service definiions updated * Mm/stop and start tweak (PlexTrac#169) * add checks for the stop process * function, not variable * docker not compose * bumpversion and update more comments --------- Co-authored-by: Michael Burke <72173919+Michael-Burke@users.noreply.github.com> Co-authored-by: mmorgan-pt <99763568+mmorgan-pt@users.noreply.github.com> Co-authored-by: Steven Paugh <spaugh@plextrac.com> Co-authored-by: Steven Paugh <122935851+StevenPaugh@users.noreply.github.com> Co-authored-by: Michael Burke <michael@plextrac.com> Co-authored-by: JJ Zabkar <148792556+jjzabkar-plextrac@users.noreply.github.com> Co-authored-by: ali <amyahya@gmail.com> Co-authored-by: Nate Pratt <npratt@plextrac.com> Co-authored-by: Darius Houle <126509519+darius-plextrac@users.noreply.github.com> Co-authored-by: Tim Daly <155962442+EramusBlack1@users.noreply.github.com> Co-authored-by: Mike Morgan <mmorgan@plextrac.com>
Erasmus-Black · Jan 9, 2025 · 2784aae · 2784aae
1 parent ed72e7c
commit 2784aae
Show file tree

Hide file tree

Showing 20 changed files with 499 additions and 154 deletions.
diff --git a/.bumpversion.cfg b/.bumpversion.cfg
@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.6.11
+current_version = 0.7.14
 
 commit = True
 tag = True

diff --git a/docs/airgapped.md b/docs/airgapped.md
@@ -20,9 +20,9 @@ docker pull plextrac/plextracnginx:<NEXT_VERSION>
 # The plextracdb shouldn't ever get updated so this will be a one time pull and can be omited from process / automation
 docker pull plextrac/plextracdb:7.2.0
 docker pull redis:6.2-alpine
-docker pull postgres:14-alpine
+docker pull plextrac/plextracpostgres:stable
 # Save the images into a TAR(s)
-docker save -o plextrac_images.tar plextrac/plextracapi:<NEXT_VERSION> plextrac/plextracnginx:<NEXT_VERSION> plextrac/plextracdb:7.2.0 redis:6.2-alpine postgres:14-alpine
+docker save -o plextrac_images.tar plextrac/plextracapi:<NEXT_VERSION> plextrac/plextracnginx:<NEXT_VERSION> plextrac/plextracdb:7.2.0 redis:6.2-alpine  plextrac/plextracpostgres:stable
 ```
 
 > Note you'll want to specify the image's platform if there are differences between where you're pulling the image (e.g., linux/arm64) and the VM (linux/x86_64)

diff --git a/docs/podman.md b/docs/podman.md
@@ -1,6 +1,24 @@
+# Podman
+
+This is a very basic guide to using Podman and explaining how it works a bit
+
+## How To's
+
+### Using Custom SSL Certificates and Custom Logos
+
+The Custom SSL Certificates and Custom Logos are mounted at the following locations:
+
+```shell
+"${PLEXTRAC_HOME:-.}/volumes/nginx_ssl_certs"
+"${PLEXTRAC_HOME:-.}/volumes/nginx_logos"
+```
+
+To use a Custom SSL Certificate or Logo, simply navigate to this location on the HOST OS, replace the files present there with the appropriate replacements, and then restart the NGINX container `podman restart plextracnginx`
+
 ## Additional Package Requirements
 
 podman | >=v4.6 (RHEL 8/9 only)
+`jq`, `bc`, `bash v5+`, and `wget`
 
 ## Podman support
 
@@ -9,12 +27,12 @@ We've expanded the capabilities to support podman in specific circumstances.
 *OS:* RHEL 8/9+
 *Podman Compose:* No (currently)
 
-> Note: the module for podman was written with RHEL 9 specifically in mind. It is not officially supported at this time to use the container runtime set to Podman on Debian, Ubuntu, or CentOS.
-
-> Note: All testing has been done on BASE images without hardening with a security profile or SELinux or anything -- its just a stock operating system
+> Note: the module for podman was written with RHEL 8/9 specifically in mind. It is not officially supported at this time to use the container runtime set to Podman on Debian, Ubuntu, or CentOS.
+> Note: All testing has been done on BASE images without hardening with a security profile or SELinux or anything -- its just a stock operating system.
 
 ---
 
+
 ### Podman Troubleshooting
 
 Depending on your configuration, you may need to solve the following issues:
@@ -56,7 +74,7 @@ The following will need to be done before running any PlexTrac specific commands
     ```bash
     vim /etc/default/grub
 
-    # Add the following line and then save
+    # Add the following line to the `GRUB_CMDLINE_LINUX` key and then save
     systemd.unified_cgroup_hierarchy=1
 
     # From CLI, run:

diff --git a/mac.Vagrantfile b/mac.Vagrantfile
@@ -0,0 +1,173 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+supportedBoxes = [
+  {
+    :name     => "amzn",
+    :box      => "bento/amazonlinux-2",
+    :default  => false,
+  },
+  {
+    :name     => "debian11",
+    :box      => "bento/debian-11",
+    :default  => false,
+  },
+  {
+    :name     => "debian12",
+    :box      => "bento/debian-12",
+    :default  => false,
+  },
+  {
+    :name     => "centos7",
+    :box      => "bento/centos-7",
+    :default  => true,
+  },
+  {
+    :name     => "centos8",
+    :box      => "bento/centos-stream-8",
+    :default  => false,
+  },
+  {
+    :name     => "centos9",
+    :box      => "bento/centos-stream-9",
+    :default  => false,
+  },
+  {
+    :name     => "rockylinux8",
+    :box      => "bento/rockylinux-8",
+    :default  => false,
+  },
+  {
+    :name     => "rockylinux9",
+    :box      => "bento/rockylinux-9-x86_64",
+    :default  => false,
+  },
+  {
+    :name     => "ubuntu",
+    :box      => "bento/ubuntu-20.04",
+    :default  => false,
+  },
+  {
+    :name     => "ubuntu2204",
+    :box      => "bento/ubuntu-22.04",
+    :default  => false,
+  },
+  {
+    :name     => "ubuntu2304-ARM",
+    :box      => "bento/ubuntu-23.04-arm64",
+    :default  => false,
+  },
+]
+
+# All Vagrant configuration is done below. The "2" in Vagrant.configure
+# configures the configuration version (we support older styles for
+# backwards compatibility). Please don't change it unless you know what
+# you're doing.
+Vagrant.configure("2") do |config|
+  if Vagrant.has_plugin?("vagrant-hostmanager")
+    # Manage hosts file entries
+    # Do `vagrant plugin install vagrant-hostmanager` if you want this
+    config.hostmanager.enabled = true
+    config.hostmanager.manage_host = true
+    config.hostmanager.ignore_private_ip = false
+    config.hostmanager.ip_resolver = proc do |vm, resolving_vm|
+      if hostname = (vm.ssh_info && vm.ssh_info[:host])
+        `vagrant ssh -c "hostname -I"`.split()[1]
+      end
+    end
+  end
+  # The most common configuration options are documented and commented below.
+  # For a complete reference, please see the online documentation at
+  # https://docs.vagrantup.com.
+
+  # Every Vagrant development environment requires a box. You can search for
+  # boxes at https://vagrantcloud.com/search.
+  supportedBoxes.each do |boxConfig|
+    hostname = "test-instance-#{boxConfig[:name]}.plextrac.local"
+    isDefault = boxConfig[:default] ? true : false 
+    config.vm.define hostname, primary: isDefault, autostart: isDefault do |host|
+      host.vm.box = boxConfig[:box]
+      host.vm.box_check_update = true # disable this to skip box updates, but remember to run `vagrant box outdated`
+    end
+  end
+
+  # Create a forwarded port mapping which allows access to a specific port
+  # within the machine from a port on the host machine. In the example below,
+  # accessing "localhost:8080" will access port 80 on the guest machine.
+  # NOTE: This will enable public access to the opened port
+  # config.vm.network "forwarded_port", guest: 80, host: 8080
+
+  # Create a forwarded port mapping which allows access to a specific port
+  # within the machine from a port on the host machine and only allow access
+  # via 127.0.0.1 to disable public access
+  # config.vm.network "forwarded_port", guest: 80, host: 8080, host_ip: "127.0.0.1"
+
+  # Create a private network, which allows host-only access to the machine
+  # using a specific IP.
+  config.vm.network "private_network", type: "dhcp"
+
+  # Create a public network, which generally matched to bridged network.
+  # Bridged networks make the machine appear as another physical device on
+  # your network.
+  # config.vm.network "public_network"
+
+  # Share an additional folder to the guest VM. The first argument is
+  # the path on the host to the actual folder. The second argument is
+  # the path on the guest to mount the folder. And the optional third
+  # argument is a set of non-required options.
+  config.vm.synced_folder ".", "/vagrant", type: "rsync"
+
+  # Provider-specific configuration so you can fine-tune various
+  # backing providers for Vagrant. These expose provider-specific options.
+  # Example for VirtualBox:
+  #
+  config.vm.provider "qemu" do |qe|
+    # Customize the amount of memory on the VM:
+    #vb.memory = "3072"
+    #vb.cpus = 2
+    #vb.customize ["modifyvm", :id, "--cpuexecutioncap", "50"]
+    qe.arch = "x86_64"
+    qe.machine = "q35"
+    qe.cpu = "max"
+    qe.memory = "16G"
+    qe.smp = "cpus=8,sockets=1,cores=8,threads=1"
+    qe.net_device = "virtio-net-pci"
+  end
+  #
+  # View the documentation for the provider you are using for more
+  # information on available options.
+
+  # Enable provisioning with a shell script. Additional provisioners such as
+  # Ansible, Chef, Docker, Puppet and Salt are also available. Please see the
+  # documentation for more information about their specific syntax and use.
+  # U291bmR0cmFjayBmb3IgdGVzdGluZzogaHR0cHM6Ly93d3cueW91dHViZS5jb20vd2F0Y2g/dj1FbDkwT0JJTEZCdwo=
+
+  config.vm.provision "shell", inline: <<-SHELL
+  echo "Generating plextrac CLI dist"
+  /vagrant/src/plextrac dist > plextrac && chmod +x plextrac
+
+  echo ""
+  echo "# Example customized deployment directory and domain name:"
+  echo "#   PLEXTRAC_HOME=/var/apps/plextrac-demo CLIENT_DOMAIN_NAME=192.168.56.37 ./plextrac initialize"
+  echo ""
+
+  echo "Initializing PlexTrac at default location..."
+  echo ""
+  ./plextrac initialize -v 2>&1
+
+  echo "You need to provide a valid DOCKER_HUB_KEY to configure PlexTrac"
+  echo "On Linux, this can be retrieved using the following command:"
+  echo ""
+  echo -n 'RE9DS0VSX0hVQl9LRVk9JChqcSAnLmF1dGhzLiJodHRwczovL2luZGV4LmRvY2tlci5pby92MS8iLmF1dGgnIH4vLmRvY2tlci9jb25maWcuanNvbiAtciB8IGJhc2U2NCAtZCB8IGN1dCAtZCc6JyAtZjIpOwo=' | base64 -d
+  echo ""
+  echo "On MacOS, this can be retrieved using the following command (enter login passphrase in the prompt(s):"
+  echo ""
+  echo -n 'RE9DS0VSX0hVQl9LRVk9JChzZWN1cml0eSBmaW5kLWludGVybmV0LXBhc3N3b3JkIC1hIHBsZXh0cmFjdXNlcnMgLXMgaW5kZXguZG9ja2VyLmlvIC13KTsK' | base64 -d
+  echo ""
+  echo "If on Windows, please figure out where that is stored and issue a PR to add support here :)"
+  echo ""
+  echo "One-liner configuration for Linux users:"
+  echo ""
+  echo -n 'RE9DS0VSX0hVQl9LRVk9JChqcSAnLmF1dGhzLiJodHRwczovL2luZGV4LmRvY2tlci5pby92MS8iLmF1dGgnIH4vLmRvY2tlci9jb25maWcuanNvbiAtciB8IGJhc2U2NCAtZCB8IGN1dCAtZCc6JyAtZjIpOyB2YWdyYW50IHNzaCAtYyAic3VkbyAtaSAtdSBwbGV4dHJhYyBET0NLRVJfSFVCX0tFWT0ke0RPQ0tFUl9IVUJfS0VZfSBwbGV4dHJhYyBjb25maWd1cmU7IHN1ZG8gLWkgLXUgcGxleHRyYWMgcGxleHRyYWMgdXBkYXRlOyBzdWRvIC1pIC11IHBsZXh0cmFjIHBsZXh0cmFjIHN0YXJ0OyBzdWRvIC1pIC11IHBsZXh0cmFjIGRvY2tlciBsb2dzIC1mIHBsZXh0cmFjYXBpIgo=' | base64 -d
+  SHELL
+end
diff --git a/src/_backup.sh b/src/_backup.sh
@@ -34,7 +34,7 @@ function backup_fullUploadsBackup() {
     podman exec --workdir="/usr/src/plextrac-api/uploads" plextracapi rm $current_date.tar.gz
     debug "Cleaned Archive from container"
   else
-    debug "`compose_client run --user $(id -u) -v ${uploadsBackupDir}:/backups \
+    debug "`compose_client run --user $(id -u) --no-deps -v ${uploadsBackupDir}:/backups \
       --workdir /usr/src/plextrac-api --rm --entrypoint='' -T  $coreBackendComposeService \
       tar -czf /backups/$(date -u "+%Y-%m-%dT%H%M%Sz").tar.gz uploads`"
   fi
@@ -43,7 +43,7 @@ function backup_fullUploadsBackup() {
 
 function backup_fullCouchbaseBackup() {
   info "$couchbaseComposeService: Performing backup of couchbase database"
-  local user_id=$(id -u plextrac)
+  local user_id=$(id -u ${PLEXTRAC_USER_NAME:-plextrac})
   local cmd="compose_client exec -T"
   if [ "$CONTAINER_RUNTIME" == "podman" ]; then
     cmd='podman exec'
@@ -67,7 +67,7 @@ function backup_fullCouchbaseBackup() {
 
 function backup_fullPostgresBackup() {
   info "$postgresComposeService: Performing backup of postgres database"
-  local user_id=$(id -u plextrac)
+  local user_id=$(id -u ${PLEXTRAC_USER_NAME:-plextrac})
   local cmd="compose_client exec -T --user $user_id"
   if [ "$CONTAINER_RUNTIME" == "podman" ]; then
     cmd='podman exec'

diff --git a/src/_check.sh b/src/_check.sh
@@ -22,6 +22,7 @@ function mod_check() {
       fi
     fi
     mod_etl_fix
+    mod_uploads_vol_fix
     VALIDATION_ONLY=1 configure_couchbase_users
     postgres_metrics_validation
     check_for_maintenance_mode
@@ -44,24 +45,41 @@ function mod_etl_fix() {
     local dir=`compose_client exec plextracapi find -type d -name etl-logs`
     if [ -n "$dir" ]; then
       local owner=`compose_client exec plextracapi stat -c '%U' uploads/etl-logs`
-      info "Checking volume permissions"
-      if [ "$owner" != "plextrac" ]
+      info "Checking ETL log destination permissions"
+      if [ "$owner" != "${PLEXTRAC_USER_NAME:-plextrac}" ]
         then
-          local user_id=$(id -u plextrac)
-          info "Volume permissions are wrong; initiating fix"
+          local user_id=$(id -u ${PLEXTRAC_USER_NAME:-plextrac})
+          info "ETL log destination permissions are wrong; initiating fix"
           compose_client exec -u 0 plextracapi chown -R $user_id:$user_id uploads/etl-logs
       else
-        info "Volume permissions are correct"
+        info "ETL log destination permissions are correct"
       fi
     else
       info "Fixing ETL Folder creation"
       compose_client exec plextracapi mkdir uploads/etl-logs
-      local user_id=$(id -u plextrac)
+      local user_id=$(id -u ${PLEXTRAC_USER_NAME:-plextrac})
       compose_client exec plextracapi chown -R $user_id:$user_id uploads/etl-logs
     fi
   fi
 }
 
+function mod_uploads_vol_fix() {
+  if [ "$CONTAINER_RUNTIME" == "podman" ]; then
+    error "Uploads volume ownership checks are not supported with Podman. Skipping"
+    return
+  else
+    info "Checking uploads volume ownership"
+    local user=`compose_client exec plextracapi whoami`
+    local dotfile_exist=`compose_client exec plextracapi find uploads -type f -name .vol-chown-pt`
+    if [ "$user" != "root"  ] && [ "$dotfile_exist" = "" ]; then
+      # this uid:gid is hardcoded in the base image and expected by the backend, do NOT change this chown
+      info "Ensuring upload volume ownership is 1337:1337, this may take awhile..."
+      compose_client exec -u 0 plextracapi chown -R 1337:1337 uploads/
+      compose_client exec plextracapi touch uploads/.vol-chown-pt
+    fi
+  fi
+}
+
 # Check for an existing installation
 function _check_no_existing_installation() {
   if [ ${IGNORE_EXISTING_INSTALLATION:-0} -eq 1 ]; then

diff --git a/src/_clean.sh b/src/_clean.sh
@@ -54,7 +54,7 @@ function clean_compressCouchbaseBackups() {
     -exec tar --remove-files -czvf /backups/{}.tar.gz {} \;
     2>&1`"
   debug "Fixing permissions on backups"
-  local user_id=$(id -u plextrac)
+  local user_id=$(id -u ${PLEXTRAC_USER_NAME:-plextrac})
   debug "`$cmd --entrypoint= --workdir /backups $image \
     find . -maxdepth 1 -type f -name '*.tar.gz' \
     -exec chown $user_id:$user_id {} \;

diff --git a/src/_cli_common_utilities.sh b/src/_cli_common_utilities.sh
@@ -31,8 +31,8 @@ function requires_user_root() {
 }
 
 function requires_user_plextrac {
-  if [ "$EUID" -ne $(id -u plextrac) ]; then
-    die "${RED}Please run as plextrac user${RESET}"
+  if [ "$EUID" -ne $(id -u ${PLEXTRAC_USER_NAME:-plextrac}) ]; then
+    die "${RED}Please run as ${PLEXTRAC_USER_NAME:-plextrac} user${RESET}"
   fi
 }