Metrics for caikit, tgi and istio were not observed in the deployed namespace

[manosnoam]

After following instructions to deploy and access Metrics on RHODS 1.32 v2 RC7 (brew.registry.redhat.io/rh-osbs/iib:568805), according to:
https://github.com/opendatahub-io/caikit-tgis-serving/blob/main/demo/kserve/metrics.md
which involves applying 2 configmaps into a test namespace (for example TEST_NS=watsonx)

• https://github.com/opendatahub-io/caikit-tgis-serving/blob/main/demo/kserve/custom-manifests/metrics/uwm-cm-enable.yaml
• https://github.com/opendatahub-io/caikit-tgis-serving/blob/main/demo/kserve/custom-manifests/metrics/uwm-cm-conf.yaml

The configMaps were created in the test namespace:

$▶ oc describe configmap/cluster-monitoring-config -n ${TEST_NS}
Name:         cluster-monitoring-config
Namespace:    watsonx
Labels:       <none>
Annotations:  <none>

Data
====
config.yaml:
----
enableUserWorkload: true


BinaryData
====

Events:  <none>

$▶ oc describe configmap/user-workload-monitoring-config -n ${TEST_NS}
Name:         user-workload-monitoring-config
Namespace:    watsonx
Labels:       <none>
Annotations:  <none>

Data
====
config.yaml:
----
prometheus:
  logLevel: debug 
  retention: 15d #Change as needed


BinaryData
====

Events:  <none>

But the expected metrics for caikit, tgi or istio were not observed as expected:

Looking at the default namespace openshift-monitoring - we can see that the original configmap was not changed:

Apparently, if updating the default configmaps in openshift-user-workload-monitoring and in cluster-monitoring-config then the expected metrics show up.

Related feature: opendatahub-io/caikit#3

[VedantMahabaleshwarkar]

@manosnoam Hey, i updated the instructions to remove the -n ${TEST_NS} from the oc apply commands for the UWM configmaps. This should result in them getting applied in the correct namespace