Add the ability to expose a port on a collector container

[kristinapathak]

collector.Container() returns a Container that doesn't include Port information.

opentelemetry-operator/pkg/collector/container.go

Lines 88 to 98 in 7ce297d

	return corev1.Container{
	Name: naming.Container(),
	Image: image,
	ImagePullPolicy: otelcol.Spec.ImagePullPolicy,
	VolumeMounts: volumeMounts,
	Args: args,
	Env: envVars,
	EnvFrom: otelcol.Spec.EnvFrom,
	Resources: otelcol.Spec.Resources,
	SecurityContext: otelcol.Spec.SecurityContext,
	LivenessProbe: livenessProbe,

Will this prevent the Prometheus PodMonitor from finding the collector? How can we add the port information? Are the container ports the same as otelcol.Spec.Ports?

opentelemetry-operator/apis/v1alpha1/opentelemetrycollector_types.go

Line 98 in 7ce297d

Ports []v1.ServicePort `json:"ports,omitempty"`

[pavolloffay]

Will this prevent the Prometheus PodMonitor from finding the collector?

I don't know, perhaps if the pod monitor uses port names. Could you please report back if you played with this? I think it's worth fixing this.

[kristinapathak]

I investigated this more - I set up a pod monitor for the collector while running the collecter as a statefulset. The ports for the operator could be found in service discovery labels, but there weren't any ports with the collectors' labels. I tried with port and targetPort (deprecated), but neither worked.

[jpkrohling]

Stupid question, but can't you use a ServiceMonitor instead? From what I remember, we create a service explicitly meant to be used with the ServiceMonitor.

[kristinapathak]

I believe this is a problem for both ServiceMonitor and PodMonitor but will try a ServiceMonitor scenario. 🙂

[pavolloffay]

@kevinearls will be working on this sortly.

[kevinearls]

Hi @kristinapathak Do you have an example CR I can use while working on this? I don't have much experience working with prometheus and none with a collector instance that is using it. Thanks.

[kristinapathak]

Hi @kevinearls,

My setup is not public, but below is information on how I verified that the lack of ports is a problem. I've attached my servicemonitor and podmonitor configurations as well as the helm chart and configuration files for prometheus.
files.zip

I am running the collector as a statefulset, using the following images.

• collector: otel/opentelemetry-collector-contrib:0.58.0
• operator: ghcr.io/open-telemetry/opentelemetry-operator/opentelemetry-operator:0.56.0
• targetallocator: ghcr.io/open-telemetry/opentelemetry-operator/target-allocator:latest

As long as prometheus and a collector are running, the pod monitor and service monitor can be applied.

Once everything is up, I port forward the prometheus UI (hosted at 9090) and view the monitors at http://localhost:9091/service-discovery. There, I can see that targets were found and dropped for my pod monitor and service monitor. The targets for the collectors in the statefulset are missing the labels __meta_kubernetes_pod_container_port_name and __meta_kubernetes_pod_container_port_number. These labels should be there and contain the metric port information (8888).

This can be easily seen using prometheus, but the targetallocator has the same issue when prometheusCR is enabled. I find it easier to troubleshoot with the prometheus UI.

I can work on making a kubernetes setup that I can share, but at least this will hopefully give you a better idea of what I'm doing.

[kristinapathak]

Hi @kevinearls, were you able to get prometheus and a pod/service monitor set up? If not, would you rather I work on this issue?

[kevinearls]

Hi @kristinapathak Sorry, I have been tied up with other things this week. Yes, please go ahead and work on it. Thanks.