Volume creation is failing after successful deployment

[auniyal61]

$ oc logs cinder-scheduler-0 -f


2024-07-15 09:51:36.759 1 ERROR cinder.scheduler.flows.create_volume [None req-e4c1db97-7d84-42ec-82b7-88bbdda63fa2 5fd3485dc42d43da9005f43a01a6a741 b0ab007b59994f0ea98a56460fddfe32 - - - -] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. No weighed backends available: cinder.exception.NoValidBackend: No valid backend was found. No weighed backends available
2024-07-15 09:51:50.453 1 INFO cinder.scheduler.base_filter [None req-fae6f759-bbe9-48c3-8ce3-5ead4d4efb34 5fd3485dc42d43da9005f43a01a6a741 b0ab007b59994f0ea98a56460fddfe32 - - - -] Filtering removed all hosts for the request with volume ID '923d2182-9953-4300-8392-ae9494430b49'. Filter results: AvailabilityZoneFilter: (start: 0, end: 0), CapacityFilter: (start: 0, end: 0), CapabilitiesFilter: (start: 0, end: 0)
2024-07-15 09:51:50.453 1 WARNING cinder.scheduler.filter_scheduler [None req-fae6f759-bbe9-48c3-8ce3-5ead4d4efb34 5fd3485dc42d43da9005f43a01a6a741 b0ab007b59994f0ea98a56460fddfe32 - - - -] No weighed backend found for volume with properties: {'id': 'b65d234d-2743-45fd-9e30-5caf32e5bb3b', 'name': '__DEFAULT__', 'description': 'Default Volume Type', 'is_public': True, 'projects': [], 'extra_specs': {}, 'qos_specs_id': None, 'created_at': '2024-07-15T08:50:47.000000', 'updated_at': '2024-07-15T08:50:47.000000', 'deleted_at': None, 'deleted': False}
2024-07-15 09:51:50.454 1 INFO cinder.message.api [None req-fae6f759-bbe9-48c3-8ce3-5ead4d4efb34 5fd3485dc42d43da9005f43a01a6a741 b0ab007b59994f0ea98a56460fddfe32 - - - -] Creating message record for request_id = req-fae6f759-bbe9-48c3-8ce3-5ead4d4efb34
2024-07-15 09:51:50.459 1 ERROR cinder.scheduler.flows.create_volume [None req-fae6f759-bbe9-48c3-8ce3-5ead4d4efb34 5fd3485dc42d43da9005f43a01a6a741 b0ab007b59994f0ea98a56460fddfe32 - - - -] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. No weighed backends available: cinder.exception.NoValidBackend: No valid backend was found. No weighed backends available

sh-5.1$ openstack volume type list
+--------------------------------------+-------------+-----------+
| ID                                   | Name        | Is Public |
+--------------------------------------+-------------+-----------+
| b65d234d-2743-45fd-9e30-5caf32e5bb3b | __DEFAULT__ | True      |
+--------------------------------------+-------------+-----------+

$ oc get cinder
NAME     STATUS   MESSAGE
cinder   True     Setup complete

$ oc get pods | grep cinder
cinder-api-0                                                    2/2     Running     0          66m
cinder-scheduler-0                                              2/2     Running     0          66m

$ oc get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                                                 STORAGECLASS                   REASON   AGE
local-storage01-crc                        10Gi       RWO,ROX,RWX    Delete           Bound       openstack/mysql-db-openstack-galera-0                 local-storage                           87m
local-storage02-crc                        10Gi       RWO,ROX,RWX    Delete           Bound       openstack/swift-swift-storage-0                       local-storage                           87m
local-storage03-crc                        10Gi       RWO,ROX,RWX    Delete           Available                                                         local-storage                           87m
local-storage04-crc                        10Gi       RWO,ROX,RWX    Delete           Bound       openstack/ovndbcluster-sb-etc-ovn-ovsdbserver-sb-0    local-storage                           87m
local-storage05-crc                        10Gi       RWO,ROX,RWX    Delete           Bound       openstack/ovndbcluster-nb-etc-ovn-ovsdbserver-nb-0    local-storage                           87m
local-storage06-crc                        10Gi       RWO,ROX,RWX    Delete           Bound       openstack/glance-glance-default-single-0              local-storage                           87m
local-storage07-crc                        10Gi       RWO,ROX,RWX    Delete           Bound       openstack/ansible-ee-logs                             local-storage                           87m
local-storage08-crc                        10Gi       RWO,ROX,RWX    Delete           Bound       openstack/persistence-rabbitmq-cell1-server-0         local-storage                           87m
local-storage09-crc                        10Gi       RWO,ROX,RWX    Delete           Available                                                         local-storage                           87m
local-storage10-crc                        10Gi       RWO,ROX,RWX    Delete           Bound       openstack/mysql-db-openstack-cell1-galera-0           local-storage                           87m
local-storage11-crc                        10Gi       RWO,ROX,RWX    Delete           Available                                                         local-storage                           87m
local-storage12-crc                        10Gi       RWO,ROX,RWX    Delete           Bound       openstack/persistence-rabbitmq-server-0               local-storage                           87m
pvc-d170c8f3-f380-40f6-8855-30661cd4aa27   30Gi       RWX            Retain           Bound       openshift-image-registry/crc-image-registry-storage   crc-csi-hostpath-provisioner            26d

$ oc describe cinder  | grep "Cinder Volume" -A 5
  Cinder Volumes:
    volume1:
      Container Image:  quay.io/podified-antelope-centos9/openstack-cinder-volume@sha256:5b96ac3775cc478580a3f3a87ceacdff38884142fc0290f4e979502f95d9319f
      Network Attachments:
        storage
      Replicas:  0

$ oc describe cinder  | grep "Cinder Backup" -A 5
  Cinder Backup:
    Container Image:  quay.io/podified-antelope-centos9/openstack-cinder-backup@sha256:08165b7fc8d0949db38a5339ed775a1a5eb2b40f155ae338bb2289af7967ef2b
    Network Attachments:
      storage
    Replicas:  0
    Resources: