add orion config for cluster density report mode

[qiliRedHat]

To address #83
Test job https://jenkins-csb-openshift-qe-mastern.dno.corp.redhat.com/job/scale-ci/job/paige-e2e-multibranch/job/orion/133/console

03-03 19:14:08.264  small-scale-cluster-density-v2-report
03-03 19:14:08.264  =====================================
03-03 19:14:08.264  time                       uuid                                  buildUrl                                                                                                                                                                                                                                 timestamp    podReadyLatency_P99    CPU_usage_masters_avg    Maximum_aggregated_memory_usage_masters_avg    CPU_Usage_kube-apiserver_avg    Max_Aggregated_RSS_Usage_kube-apiserver_avg    CPU_Usage_etcd_avg    Max_Aggregated_RSS_Usage_etcd_avg    Read_Only_API_request_P99_latency_namespace_avg    Read_Only_API_request_P99_latency_cluster_avg    Read_Only_API_request_P99_latency_avg_avg    99th_WAL_fsync_avg    99th_Backend_I/O_avg    99th_Roundtrip_avg
03-03 19:14:08.265  -------------------------  ------------------------------------  -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------  -----------  ---------------------  -----------------------  ---------------------------------------------  ------------------------------  ---------------------------------------------  --------------------  -----------------------------------  -------------------------------------------------  -----------------------------------------------  -------------------------------------------  --------------------  ----------------------  --------------------
03-03 19:14:08.265  2024-09-06 20:34:03 +0000  99ea94f1-dfe4-48b1-8e48-1e78e210300d  https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_release/56105/rehearse-56105-pull-ci-openshift-qe-ocp-qe-perfscale-ci-main-nutanix-4.17-nightly-x86-cluster-density-v2-24nodes/1832130110669459456  1.72565e+09                  13000                  2.80263                                    3.05709e+10                        0.66581                                     9.50154e+09              0.305839                          4.14487e+08                                           0.131548                                         0.466944                                     0.117232             0.0257487               0.0402395             0.0386308
03-03 19:14:08.265                                                                                                                                                                                                                                                                                                                       ·····················  ·······················  ·············································  ······························  ·············································  ····················  ···································  ·················································  ···············································  ···········································  ····················  ······················  ····················  
03-03 19:14:08.265                                                                                                                                                                                                                                                                                                                                      -15.4%                   +15.1%                                          -8.0%                          +48.3%                                         -10.1%                 +4.3%                                -5.6%                                             +46.7%                                           -28.7%                                      +135.4%                -23.2%                  -23.4%                -43.4%  
03-03 19:14:08.266                                                                                                                                                                                                                                                                                                                       ·····················  ·······················  ·············································  ······························  ·············································  ····················  ···································  ·················································  ···············································  ···········································  ····················  ······················  ····················  
03-03 19:14:08.266  2024-11-20 17:08:17 +0000  6b802f13-0bb9-47e8-aa0d-c43291578980  https://mastern-jenkins-csb-openshift-qe.apps.ocp-c1.prod.psi.redhat.com/job/scale-ci/job/e2e-benchmarking-multibranch-pipeline/job/kube-burner-ocp/1981/                                                                              1.73212e+09                  11000                  3.22653                                    2.81197e+10                        0.987562                                    8.54581e+09              0.319047                          3.91352e+08                                           0.192979                                         0.333053                                     0.27595              0.0197785               0.0308124             0.0218743

[paigerube14]

might be good to add a metric to capture ovn as we have seen bugs in this area:

similar to

orion/examples/small-scale-cluster-density.yaml

Lines 34 to 40 in 3514633

	- name: ovnCPU
	metricName : containerCPU
	labels.namespace.keyword: openshift-ovn-kubernetes
	metric_of_interest: value
	agg:
	value: cpu
	agg_type: avg

could use these metrics:
https://github.com/kube-burner/kube-burner-ocp/blob/303796a9288a31b6c4e317243e009a6168a25a9a/cmd/config/metrics-report.yml#L229-L231

https://github.com/kube-burner/kube-burner-ocp/blob/303796a9288a31b6c4e317243e009a6168a25a9a/cmd/config/metrics-report.yml#L245-L247

[qiliRedHat]

@paigerube14 I added the ovn metrics as you suggested. The new test job passed.

03-04 18:24:57.677  small-scale-cluster-density-v2-report
03-04 18:24:57.677  =====================================
03-04 18:24:57.677  time                       uuid                                  buildUrl                                                                                                                                                                                                                                 timestamp    podReadyLatency_P99    CPU_usage_masters_avg    Maximum_aggregated_memory_usage_masters_avg    CPU_Usage_kube-apiserver_avg    Max_Aggregated_RSS_Usage_kube-apiserver_avg    CPU_Usage_etcd_avg    Max_Aggregated_RSS_Usage_etcd_avg    Read_Only_API_request_P99_latency_namespace_avg    Read_Only_API_request_P99_latency_cluster_avg    Read_Only_API_request_P99_latency_avg_avg    99th_WAL_fsync_avg    99th_Backend_I/O_avg    99th_Roundtrip_avg    cpu_ovn_control_plane_avg    cpu_ovnkube_node_avg
03-04 18:24:57.678  -------------------------  ------------------------------------  -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------  -----------  ---------------------  -----------------------  ---------------------------------------------  ------------------------------  ---------------------------------------------  --------------------  -----------------------------------  -------------------------------------------------  -----------------------------------------------  -------------------------------------------  --------------------  ----------------------  --------------------  ---------------------------  ----------------------
03-04 18:24:57.678  2024-09-06 20:34:03 +0000  99ea94f1-dfe4-48b1-8e48-1e78e210300d  https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_release/56105/rehearse-56105-pull-ci-openshift-qe-ocp-qe-perfscale-ci-main-nutanix-4.17-nightly-x86-cluster-density-v2-24nodes/1832130110669459456  1.72565e+09                  13000                  2.80263                                    3.05709e+10                        0.66581                                     9.50154e+09              0.305839                          4.14487e+08                                           0.131548                                         0.466944                                     0.117232             0.0257487               0.0402395             0.0386308                   0.00329786               0.0443364
03-04 18:24:57.679                                                                                                                                                                                                                                                                                                                       ·····················  ·······················  ·············································  ······························  ·············································  ····················  ···································  ·················································  ···············································  ···········································  ····················  ······················  ····················  ···························  ······················  
03-04 18:24:57.679                                                                                                                                                                                                                                                                                                                                      -15.4%                   +15.1%                                          -8.0%                          +48.3%                                         -10.1%                 +4.3%                                -5.6%                                             +46.7%                                           -28.7%                                      +135.4%                -23.2%                  -23.4%                -43.4%                       +84.5%                   +5.4%  
03-04 18:24:57.680                                                                                                                                                                                                                                                                                                                       ·····················  ·······················  ·············································  ······························  ·············································  ····················  ···································  ·················································  ···············································  ···········································  ····················  ······················  ····················  ···························  ······················  
03-04 18:24:57.680  2024-11-20 17:08:17 +0000  6b802f13-0bb9-47e8-aa0d-c43291578980  https://mastern-jenkins-csb-openshift-qe.apps.ocp-c1.prod.psi.redhat.com/job/scale-ci/job/e2e-benchmarking-multibranch-pipeline/job/kube-burner-ocp/1981/                                                                              1.73212e+09                  11000                  3.22653                                    2.81197e+10                        0.987562                                    8.54581e+09              0.319047                          3.91352e+08                                           0.192979                                         0.333053                                     0.27595              0.0197785               0.0308124             0.0218743                   0.00608538               0.0467234

However, I saw a problem, when I click on the report mode and non-report mode grafana link in this job, except for the Pod latency metrics, I only see the data of 4.18. Any idea?

03-04 18:24:26.286  grafana url https://grafana.rdu2.scalelab.redhat.com:3000/d/g4dJlkBnz3/kube-burner-compare?orgId=1&var-Datasource=ab3f14e6-a50f-4d52-93fa-a5076794f864&var-sdn=OVNKubernetes&var-workload=cluster-density-v2&var-latencyPercentile=P99&var-condition=Ready&var-component=kube-apiserver&var-uuid=6b802f13-0bb9-47e8-aa0d-c43291578980&var-uuid=99ea94f1-dfe4-48b1-8e48-1e78e210300d&var-workerNodesCount=24&from=now-1y&to=now&var-platform=AWS&var-platform=Azure&var-platform=GCP&var-platform=IBMCloud&var-platform=AlibabaCloud&var-platform=VSphere&var-platform=rosa&var-clusterType=rosa&var-clusterType=self-managed
03-04 18:24:26.286  grafana report mode link: https://grafana.rdu2.scalelab.redhat.com:3000/d/D5E8c5XVz/kube-burner-report-mode?orgId=1&var-Datasource=ab3f14e6-a50f-4d52-93fa-a5076794f864&var-sdn=OVNKubernetes&var-clusterType=rosa&var-clusterType=self-managed&var-job=cluster-density-v2&var-ocpMajorVersion=4.18&var-ocpMajorVersion=4.17&var-compare_by=metadata.ocpMajorVersion&var-component=kube-apiserver&var-component=kube-controller-manager&var-node_roles=masters&var-node_roles=workers&var-node_roles=infra&to=now&var-uuid=6b802f13-0bb9-47e8-aa0d-c43291578980&var-uuid=99ea94f1-dfe4-48b1-8e48-1e78e210300d&var-workerNodesCount=24&from=now-1y&to=now&var-platform=AWS&var-platform=Azure&var-platform=GCP&var-platform=IBMCloud&var-platform=AlibabaCloud&var-platform=VSphere&var-platform=rosa&var-clusterType=rosa&var-clusterType=self-managed

[paigerube14]

@qiliRedHat I see both 4.17 and 4.18 in the grafana, maybe try reloading or putting in incognito. That link I generate is not part of what is output from this repo

[qiliRedHat]

@paigerube14 ok, the grafana is not related to this pr, let me take it offline.

[qiliRedHat]

@paigerube14 Are you able to approve the pr?

[paigerube14]

/lgtm

[qiliRedHat]

@vishnuchalla I need a reviewer with write access. PTAL.