[target-allocator] TA pod doesn't restart when changing config settings

[moh-osman3]

Trying to debug some issues I have in my statefulset cluster, and attempted to disable prometheusCR, to stop discovering serviceMonitor jobs. Ran into an issue when setting prometheusCR: false, where the target-allocator does not restart after applying these changes.

In my collector config

spec:
  mode: {{ .Values.collector.mode }}
  image: {{ .Values.collector.image }}
  replicas: 3
  targetAllocator:
    enabled: true
    image: ghcr.io/open-telemetry/opentelemetry-operator/target-allocator:latest
    serviceAccount: {{ .Release.Name }}-collector-targetallocator
    prometheusCR:
      enabled: false

What I expected

I expected the target-allocator pod to restart and the output of curl http://lightstep-collector-targetallocator:80/jobs to exclude any jobs that were discovered by serviceMonitors.

What actually happened

No restart occurred of my target-allocator pod and even after deleting the target-allocator pod (kubectl delete po target-allocator), the output of curl http://lightstep-collector-targetallocator:80/jobs was not updated to reflect disabling prometheusCR

In fact the only way to get the update I expected was to completely delete my opentelemetry namespace (collectors, target-allocator, and otel-operator) and reinstall the pods.

In case its helpful I attached the configs of the collector and target-allocator

k get OpenTelemetryCollector lightstep-collector -o yaml: bug_attachment_otc.txt

k get deployment lightstep-collector-targetallocator -o yaml: bug_attachment_ta.txt

[jaronoff97]

I'm gonna look in to the fix for this