Create Prometheus ServiceMonitor/PodMonitor for collector

[pavolloffay]

The operator could create Prometheus ServiceMonitor/PodMonitor which would enable scraping of collector metrics.

This functionality could be conditionally enabled in CR along when the p8s CRDs are present in the cluster.

We should also investigate sending collector internal metrics via {OTLP} exporter configured in the collector.

cc) @kevinearls

[jaronoff97]

iirc collector internal metrics are currently done only through prometheus? I would love to open an issue in the collector project about emitting OTLP, however.

For the operator creating a CRD, I think making a service monitor would make sense. In the manually created one we generated, we had to specify some extra labels so we didn't double scrape targets between the -collector and -collector-headless (which share the same port configuration.) In retrospect, I should have gone with the -collector-monitoring service which would be the most accurate and wouldn't have a chance for duplicated metric settings.

I can take on this work as well if that would be helpful.

[alexdresko]

I'm not the smartest kid here, but I wonder if this is what I'm looking for. I've got the kube prometheus stack helm chart, and the otel collector chart running in my cluster. I need prometheus to scrape otel col.

Currentlly, I've configured the prometheus exporter on port 8889. I also created a service to expose this port in the cluster. And now I'm trying to create a ServiceMonitor, but it's not quite working. I'm sure I'll figure it out.. but I landed here because I was looking to see if anyone else had done this before. Surely I'm not the first person.

[kristinapathak]

Hi @alexdresko, creating a ServiceMonitor for the collector may not work until the ports are included for the container (#1011). I included my ServiceMonitor file as a part of the .zip in this comment, but it will only work once that issue is fixed.

[iblancasa]

@pavolloffay is this the same as #1768?

[jaronoff97]

This is closed by #1963