Permission denied: /var/lib/awx/projects/

[chuegel]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.

Bug Summary

AWX can't write into /var/lib/awx/projects

Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/jobs.py", line 491, in run
    self.pre_run_hook(self.instance, private_data_dir)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/jobs.py", line 1322, in pre_run_hook
    self.acquire_lock(instance.project, instance.id)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/jobs.py", line 388, in acquire_lock
    self.lock_fd = os.open(lock_path, os.O_RDWR | os.O_CREAT)
PermissionError: [Errno 13] Permission denied: '/var/lib/awx/projects/_14__librenms_automation.lock'

AWX Operator version

2.6.0

AWX version

23.2.0

Kubernetes platform

kubernetes

Kubernetes/Platform version

v1.27.3+rke2r1

Modifications

no

Steps to reproduce

AWX deployment:

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx-production
  namespace: apps
spec:
  replicas: 1
  ingress_type: ingress
  hostname: awx.example.tech 
  ingress_annotations: |
     cert-manager.io/cluster-issuer: letsencrypt-production
     kubernetes.io/ingress.class: nginx
     nginx.ingress.kubernetes.io/affinity: cookie
     nginx.ingress.kubernetes.io/session-cookie-name: "AWX-SESSION-COOKIE-PROD-TECH"
     nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
     nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"
     nginx.ingress.kubernetes.io/affinity-mode: persistent
     nginx.ingress.kubernetes.io/session-cookie-hash: sha1
  ingress_tls_secret: awx-cert
  projects_persistence: true
  projects_storage_class: kadalu.kadula-pool-1
  projects_storage_size: 20Gi
  postgres_configuration_secret: awx-production-postgres-configuration
  web_resource_requirements:
    requests:
      cpu: 1000m
      memory: 2Gi 
    limits:
      cpu: 2000m
      memory: 4Gi
  task_resource_requirements:
    requests:
      cpu: 500m
      memory: 1Gi
    limits:
      cpu: 1000m
      memory: 2Gi
  ee_resource_requirements:
    requests:
      cpu: 500m
      memory: 1Gi
    limits:
      cpu: 1000m
      memory: 2Gi


### Expected results

To be able to write into AWX PROJECTS_ROOT folder


### Actual results

Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/jobs.py", line 491, in run
self.pre_run_hook(self.instance, private_data_dir)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/jobs.py", line 1322, in pre_run_hook
self.acquire_lock(instance.project, instance.id)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/jobs.py", line 388, in acquire_lock
self.lock_fd = os.open(lock_path, os.O_RDWR | os.O_CREAT)
PermissionError: [Errno 13] Permission denied: '/var/lib/awx/projects/_14__librenms_automation.lock'

### Additional information

Inside the web container:

bash-5.1$ df -h
Filesystem Size Used Avail Use% Mounted on
overlay 30G 25G 5.6G 82% /
tmpfs 64M 0 64M 0% /dev
/dev/mapper/vg1-root 30G 25G 5.6G 82% /etc/hosts
shm 64M 0 64M 0% /dev/shm
tmpfs 32G 4.0K 32G 1% /etc/tower/SECRET_KEY
tmpfs 32G 8.0K 32G 1% /etc/receptor/work_public_key.pem
tmpfs 32G 12K 32G 1% /etc/tower/conf.d/ldap.py
kadalu:kadula-pool-1 21G 201M 20G 1% /var/lib/awx/projects
tmpfs 32G 12K 32G 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 32G 8.0K 32G 1% /etc/receptor/tls/ca/mesh-CA.key
tmpfs 16G 0 16G 0% /proc/acpi
tmpfs 16G 0 16G 0% /proc/scsi
tmpfs 16G 0 16G 0% /sys/firmware

bash-5.1$ ls -la /var/lib/awx/
total 5
drwxrwxr-x 1 root root 58 Nov 21 09:43 .
drwxr-xr-x 1 root root 17 Sep 26 17:25 ..
prw------- 1 awx root 0 Nov 21 09:41 awxfifo
-rw------- 1 awx root 129 Nov 21 09:49 .bash_history
drwxr-xr-x 3 root root 19 Sep 26 17:25 .local
drwxrwxr-x 4 root 1000 107 Oct 2 14:37 projects
drwxr-xr-x 3 root root 20 Sep 26 17:25 public
drwxrwxr-x 1 root root 40 Sep 26 17:25 rsyslog
drwxr-xr-x 3 root root 17 Sep 26 17:14 venv

bash-5.1$ touch test /var/lib/awx/projects/
touch: cannot touch 'test': Permission denied
touch: setting times of '/var/lib/awx/projects/': Permission denied

bash-5.1$ whoami
awx

bash-5.1$ cat /etc/passwd
awx:x:1000:0:,,,:/var/lib/awx:/bin/bash

### Operator Logs

_No response_