feat: add metrics for jobset

[googs1025]

Fix: #613 (comment)

only add FailedTotal CompletedTotal, two metrics, If more metrics are needed, I will add them

var (
	FailedTotal = prometheus.NewCounterVec(
		prometheus.CounterOpts{
			Subsystem: constants.JobSetName,
			Name:      "jobset_failed_total",
			Help:      `The total number of jobset failed case`,
		}, []string{"jobsetName"},
	)

	CompletedTotal = prometheus.NewCounterVec(
		prometheus.CounterOpts{
			Subsystem: constants.JobSetName,
			Name:      "jobset_completed_total",
			Help:      `The total number of jobset completed case`,
		}, []string{"jobsetName"},
	)
)

[netlify]

✅ Deploy Preview for kubernetes-sigs-jobset ready!

Name	Link
🔨 Latest commit	7ca93cb
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-jobset/deploys/66a2f15bacee500008b135c3
😎 Deploy Preview	https://deploy-preview-614--kubernetes-sigs-jobset.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[googs1025]

The following test: simulates the experiment of whether the indicators are still correct after the controller is restarted

# Installing the prometheus operator
root@VM-0-5-ubuntu:/home/ubuntu/jobset/examples/simple# kubectl get pods
NAME                                   READY   STATUS    RESTARTS   AGE
prometheus-operator-76469b7f8c-5wb8x   1/1     Running   0          12h

# install ServiceMonitor, So we can see prometheus-prometheus-0 in the jobset-system namespace
root@VM-0-5-ubuntu:/home/ubuntu/jobset/examples/simple# kubectl get pods -njobset-system
NAME                                         READY   STATUS    RESTARTS   AGE
jobset-controller-manager-76767b599b-c49wz   2/2     Running   0          14m
prometheus-prometheus-0                      2/2     Running   0          11h

# run paralleljobs-before jobset
root@VM-0-5-ubuntu:/home/ubuntu/jobset/examples/simple# kubectl apply -f paralleljobs.yaml
jobset.jobset.x-k8s.io/paralleljobs-before created

We can use prometheus to view metrics

# Next, we delete the controller to simulate restarting
root@VM-0-5-ubuntu:/home/ubuntu/jobset/examples/simple# kubectl get pods -njobset-system
NAME                                         READY   STATUS    RESTARTS   AGE
jobset-controller-manager-76767b599b-4rk4h   2/2     Running   0          2m13s
prometheus-prometheus-0                      2/2     Running   0          11h
root@VM-0-5-ubuntu:/home/ubuntu/jobset/examples/simple# kubectl get jobset
NAME                  TERMINALSTATE   RESTARTS   COMPLETED   SUSPENDED   AGE
paralleljobs-before   Completed                  True                    35s

root@VM-0-5-ubuntu:/home/ubuntu/jobset/examples/simple# kubectl get pods -njobset-system
NAME                                         READY   STATUS    RESTARTS   AGE
jobset-controller-manager-76767b599b-c49wz   2/2     Running   0          16m
prometheus-prometheus-0                      2/2     Running   0          11h
root@VM-0-5-ubuntu:/home/ubuntu/jobset/examples/simple# kubectl delete pod jobset-controller-manager-76767b599b-c49wz -njobset-system
pod "jobset-controller-manager-76767b599b-c49wz" deleted
root@VM-0-5-ubuntu:/home/ubuntu/jobset/examples/simple# kubectl get pods -njobset-system
NAME                                         READY   STATUS    RESTARTS   AGE
jobset-controller-manager-76767b599b-4rk4h   1/2     Running   0          4s
prometheus-prometheus-0                      2/2     Running   0          11h
root@VM-0-5-ubuntu:/home/ubuntu/jobset/examples/simple# kubectl get pods -njobset-system
NAME                                         READY   STATUS    RESTARTS   AGE
jobset-controller-manager-76767b599b-4rk4h   2/2     Running   0          22s
prometheus-prometheus-0                      2/2     Running   0          11h

# run paralleljobs-after jobset
root@VM-0-5-ubuntu:/home/ubuntu/jobset/examples/simple# ls
exclusive-placement.yaml  jobset-with-network.yaml  max-restarts.yaml  paralleljobs.yaml  success-policy.yaml  ttl-after-finished.yaml

root@VM-0-5-ubuntu:/home/ubuntu/jobset/examples/simple# kubectl get jobset
NAME                  TERMINALSTATE   RESTARTS   COMPLETED   SUSPENDED   AGE
paralleljobs-after                                                       8s
paralleljobs-before   Completed                  True                    2m57s
root@VM-0-5-ubuntu:/home/ubuntu/jobset/examples/simple# kubectl get jobset
NAME                  TERMINALSTATE   RESTARTS   COMPLETED   SUSPENDED   AGE
paralleljobs-after                                                       14s
paralleljobs-before   Completed                  True                    3m3s
root@VM-0-5-ubuntu:/home/ubuntu/jobset/examples/simple# kubectl get jobset
NAME                  TERMINALSTATE   RESTARTS   COMPLETED   SUSPENDED   AGE
paralleljobs-after    Completed                  True                    20s
paralleljobs-before   Completed                  True                    3m9s


root@VM-0-5-ubuntu:/home/ubuntu/jobset/examples/simple# kubectl get jobset
NAME                  TERMINALSTATE   RESTARTS   COMPLETED   SUSPENDED   AGE
paralleljobs-after    Completed                  True                    20s
paralleljobs-before   Completed                  True                    3m9s

It can be seen from the indicators that when the simulation controller is restarted, the indicators still exist.

ps: if using kind, we can use port-forward, kubectl port-forward services/prometheus 39090:9090 --address 0.0.0.0
This allows us to access prometheus using a browser: http://<ecs public IP>:39090/graph

[googs1025]

The yaml file used is as follows:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  namespace: jobset-system
spec:
  serviceAccountName: prometheus1
  # ServiceMonitor 
  serviceMonitorSelector:
    #  label
    matchLabels:
      control-plane: controller-manager
  resources:
    requests:
      memory: 400Mi
  enableAdminAPI: false
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: jobset-system
spec:
  type: NodePort
  # kubectl port-forward services/prometheus  39090:9090 --address 0.0.0.0
  ports:
    - name: web
      nodePort: 30900
      port: 9090
      protocol: TCP
      targetPort: web
  selector:
    prometheus: prometheus

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus1
  namespace: jobset-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus1
rules:
  - apiGroups: [""]
    resources:
      - nodes
      - nodes/metrics
      - services
      - endpoints
      - pods
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources:
      - configmaps
    verbs: ["get"]
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingresses
    verbs: ["get", "list", "watch"]
  - nonResourceURLs: ["/metrics"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus1
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus1
subjects:
  - kind: ServiceAccount
    name: prometheus1
    namespace: jobset-system

[googs1025]

@danielvegamyhre /PTAL thanks!

[googs1025]

In addition, I found that the docs does not have very clear steps for combining with prometheus operator. I will submit a PR to supplement it.

[danielvegamyhre]

In addition, I found that the docs does not have very clear steps for combining with prometheus operator. I will submit a PR to supplement it.

This would be great, thanks!

[danielvegamyhre]

@googs1025 in your example/test above, I see your query is for jobset_jobset_completed_total so I assume the format is {$SUBSYSTEM_NAME}_${METRIC_NAME}. I think maybe we should change the metric names to completed_total and failed_total, so the full metric is jobset_completed_total and jobset_failed_total, and we don't have a jobset_jobset_ prefix with the duplication. What do you think?

[googs1025]

@googs1025 in your example/test above, I see your query is for jobset_jobset_completed_total so I assume the format is {$SUBSYSTEM_NAME}_${METRIC_NAME}. I think maybe we should change the metric names to completed_total and failed_total, so the full metric is jobset_completed_total and jobset_failed_total, and we don't have a jobset_jobset_ prefix with the duplication. What do you think?

done

[googs1025]

ping @danielvegamyhre :)

[danielvegamyhre]

Looks good to me. @kannon92 want to do a pass on this as well?

[kannon92]

/lgtm

[danielvegamyhre]

/approve

Thanks for the great work @googs1025!

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danielvegamyhre, googs1025

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [danielvegamyhre]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment