Adding a service to expose Operator SDK metrics

[DharmitD]

Description

Adding a service to expose Operator SDK metrics port

How Has This Been Tested?

On a test cluster:

• Added this service config
• Port forwarded to the localhost port via the service:
  oc port-forward svc/data-science-pipelines-operator 8080:8080 -n data-science-pipelines-operator
• Curled to the localhost to retrieve a list of Operator SDK metrics:
  curl -v http://localhost:8080/metrics

Find a list of metrics returned by the curl command here.

Merge criteria:

• The commits are squashed in a cohesive manner and have meaningful messages.
• Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
• The developer has manually tested the changes and verified that the changes work

[gregsheremeta]

content looks good to me. The only question I have is whether we'd separate out the service into its own config file. The other config directories seem to have resources be mostly separate (for example: https://github.com/opendatahub-io/data-science-pipelines-operator/tree/main/config/rbac), and I'm not familiar enough with the conventions. Perhaps another reviewer can make a suggestion.

[DharmitD]

content looks good to me. The only question I have is whether we'd separate out the service into its own config file. The other config directories seem to have resources be mostly separate (for example: https://github.com/opendatahub-io/data-science-pipelines-operator/tree/main/config/rbac), and I'm not familiar enough with the conventions. Perhaps another reviewer can make a suggestion.

Yes, that makes sense. Updated to have the service in a separate config file, and called it in the kustomization.yaml, to maintain consistency across all configs/directories.

[HumairAK]

Can you confirm that the servicemonitor works? Confirm this by checking if the metrics show up in ocp monitoring stack. My guess is it won't work because it's missing the labels for the selector here: https://github.com/opendatahub-io/data-science-pipelines-operator/blob/main/config/prometheus/monitor.yaml

The port will also need to be updated to match the exposed port in the service.

[gregsheremeta]

My guess is it won't work because it's missing the labels for the selector here

By "it", did you mean the service? The service has to have the same labels as the servicemonitor?

[HumairAK]

By "it", did you mean the service? The service has to have the same labels as the servicemonitor?

it here refers to the servicemonitor, which will not work if it cannot select on the labels for the service

[HumairAK]

@DharmitD you should also test by port-forwarding the service, not the pod oc port-forward service/$service_name since that is the change you are making

[DharmitD]

@DharmitD you should also test by port-forwarding the service, not the pod oc port-forward service/$service_name since that is the change you are making

I updated the service to add the same label as the service monitor. Tried port forwarding through the service:
oc port-forward svc/data-science-pipelines-operator 8080:8080 -n data-science-pipelines-operator

The port forward worked, further queried metrics using curl:
curl http://localhost:8080/metrics
Curl returned the same set of metrics.

[DharmitD]

Can you confirm that the servicemonitor works? Confirm this by checking if the metrics show up in ocp monitoring stack. My guess is it won't work because it's missing the labels for the selector here: https://github.com/opendatahub-io/data-science-pipelines-operator/blob/main/config/prometheus/monitor.yaml

The port will also need to be updated to match the exposed port in the service.

Added a ServiceMonitor config in #24 with labels matching to the above service config.
The ServiceMonitor works and metrics can be queried.

[gregsheremeta]

/lgtm

[HumairAK]

/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: DharmitD, HumairAK

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [HumairAK]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment