MON-3775: add tests for the `MetricsCollectionProfiles` feature-gate #28889

rexagod · 2024-06-18T08:21:28Z

openshift-ci-robot · 2024-06-18T08:21:31Z

@rexagod: This pull request references MON-3775 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.17.0" version, but no target version was set.

In response to this:

OEP: openshift/enhancements#1298

/cc @simonpasquier

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

rexagod · 2024-06-18T08:21:43Z

/test e2e-gcp-ovn-techpreview

rexagod · 2024-06-18T11:42:29Z

@deads2k Following up on #28768 (comment), I dropped g.Context but /verify still seems to fail. I believe the root cause is make update being non-idempotent for local and CI environments.

I'm able to include my tests in the generated manifests using update, but doing so makes verify fail. Similarly, if I leave them out, and don't push the generated manifests that include the tests, verify passes but the tests don't run (as there's nothing to run). For some reason, CI's verify run doesn't seem to "discover" the added tests (as you mentioned), and generates artifacts that leave those tests out, making the tree dirty, causing the job to fail.

test/extended/prometheus/collection_profiles.go

simonpasquier · 2024-06-19T15:35:17Z

test/extended/prometheus/collection_profiles.go

+		}
+	})
+	g.It("should hide a default-only metric when minimal collection profile is enabled", func() {
+		defaultOnlyMetric := "kube_deployment_status_replicas"


can we pick a Prometheus metric which we know for sure will (likely) never be present in the minimal profile (for instance prometheus_engine_query_log_enabled).

test/extended/prometheus/collection_profiles.go

rexagod · 2024-07-09T08:05:12Z

/test e2e-gcp-ovn-techpreview

openshift-trt-bot · 2024-07-09T12:07:24Z

Job Failure Risk Analysis for sha: 9c556df

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade	High [sig-arch] events should not repeat pathologically for ns/openshift-kube-apiserver-operator This test has passed 99.92% of 5096 runs on release 4.17 [Overall] in the last week.
pull-ci-openshift-origin-master-e2e-aws-ovn-ipsec-serial	High operator conditions monitoring This test has passed 100.00% of 34 runs on jobs ['periodic-ci-openshift-release-master-nightly-4.17-e2e-aws-ovn-serial' 'periodic-ci-openshift-release-master-ci-4.17-e2e-aws-ovn-serial'] in the last 14 days.

rexagod · 2024-07-09T15:10:52Z

/test e2e-gcp-ovn-techpreview

Updated manifests.

rexagod · 2024-07-09T20:06:25Z

/test e2e-gcp-ovn-techpreview

pkg/clioptions/suiteselection/feature_filter.go

test/extended/prometheus/collection_profiles.go

openshift-trt-bot · 2024-07-23T21:05:55Z

Job Failure Risk Analysis for sha: 49dd188

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-ipsec-serial	IncompleteTests Tests for this run (27) are below the historical average (423): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

jan--f · 2024-07-29T14:52:32Z

test/extended/prometheus/collection_profiles.go

+		})
+	})
+
+	var _ = g.Context("in an environment that has the minimal profile pre-set", func() {


I have to again point out I'm not a ginko expert, but shouldn't this context have a BeforeAll node to apply the minimal profile? Iiuc the intention is that ginko can reorder context nodes as it sees fit, so I'm not sure this will succeed in all cases. This currently relies on the previous context to apply the minimal profile.

The documentation mentions that Context (alias for Describe) supports decorators (i.e., the Ordered parameter being passed here) while nesting more containers internally. The containers seem to execute as they appear (among other "weights" that work towards the same affect, and IDs, in case of Its).

I believe that as long as the structure is maintained, things should happen as expected. But LMK if this seems prone to manual oversight and I'll add those in.

Ahh I missed that. Should we maybe have a comment preceding the context so future readers don't miss this like I did?

test/extended/prometheus/collection_profiles.go

pkg/clioptions/suiteselection/feature_filter.go

rexagod · 2024-07-30T07:01:50Z

/test e2e-gcp-ovn-techpreview

rexagod · 2024-07-30T07:16:22Z

/retest-required

pkg/clioptions/suiteselection/feature_filter.go

test/extended/prometheus/collection_profiles.go

jan--f · 2024-07-30T08:42:33Z

test/extended/prometheus/collection_profiles.go

+		})
+	})
+
+	var _ = g.Context("in an environment that has the minimal profile pre-set", func() {


Ahh I missed that. Should we maybe have a comment preceding the context so future readers don't miss this like I did?

rexagod · 2024-07-30T10:25:06Z

Testcases breaking after rebase, in addition to the Ginkgo's earlier Interrupted by user for two of them. I'll take a look.

openshift-trt-bot · 2024-07-30T11:19:02Z

Job Failure Risk Analysis for sha: a86de08

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-ipsec-serial	IncompleteTests Tests for this run (27) are below the historical average (511): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

rexagod · 2024-07-30T16:51:10Z

/test e2e-gcp-ovn-techpreview

openshift-trt-bot · 2024-07-30T20:09:57Z

Job Failure Risk Analysis for sha: 889ca0f

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-ipsec-serial	IncompleteTests Tests for this run (27) are below the historical average (523): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

rexagod · 2024-07-30T22:21:56Z

/test e2e-gcp-ovn-techpreview

rexagod · 2024-09-18T07:50:50Z

/test e2e-aws-ovn-ipsec-serial

None of the tests affect e2e-aws-ovn-ipsec-serial, which is also a non-required job. I believe the job is flaky and may be skipped.

cc @jan--f for lgtm, will post this in the API review forum once we have that on the PR.

openshift-trt-bot · 2024-09-18T12:11:40Z

Job Failure Risk Analysis for sha: da1d01a

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-ipsec-serial	High [sig-architecture] platform pods should not exit more than once with a non-zero exit code This test has passed 100.00% of 30 runs on jobs ['periodic-ci-openshift-release-master-ci-4.18-e2e-aws-ovn-serial' 'periodic-ci-openshift-release-master-nightly-4.18-e2e-aws-ovn-serial'] in the last 14 days. Open Bugs Excessive Restarts on ingress operator
pull-ci-openshift-origin-master-e2e-gcp-ovn-techpreview-serial	IncompleteTests Tests for this run (106) are below the historical average (743): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

jan--f · 2024-09-23T08:47:50Z

/lgtm

neisw · 2024-09-25T13:25:03Z

Anyone reviewing the failures in e2e-gcp-ovn-techpreview-serial?

  • [FAILED] [902.571 seconds]
  [sig-instrumentation][OCPFeatureGate:MetricsCollectionProfiles][Serial][Conformance] The collection profiles feature-set in an environment that has the minimal profile pre-set [It] should hide all default-only metrics when minimal collection profile is enabled [Suite:openshift/conformance/serial/minimal]
  github.com/openshift/origin/test/extended/prometheus/collection_profiles.go:148

    [FAILED] Timed out after 900.001s.
    Expected
        <*errors.errorString | 0xc001fcfc90>: 
        no monitors found with collection profile: "minimal" and "app.kubernetes.io/name"="kube-state-metrics-minimal"
        {
            s: "no monitors found with collection profile: \"minimal\" and \"app.kubernetes.io/name\"=\"kube-state-metrics-minimal\"",
        }
    to be nil
    In [It] at: github.com/openshift/origin/test/extended/prometheus/collection_profiles.go:172 @ 08/01/24 15:27:20.654

neisw · 2024-09-25T13:35:16Z

/test e2e-gcp-ovn-techpreview-serial
/test e2e-gcp-ovn-techpreview

neisw · 2024-09-25T13:43:47Z

/test e2e-aws-ovn-single-node-techpreview
/test e2e-aws-ovn-techpreview

rexagod · 2024-09-25T14:48:11Z

@neisw The test quoted here ran on Thu Aug 1 13:52:31 UTC 2024. The patch has seen multiple changes since then, in addition to the test not being serialized anymore, so any -serial jobs should exclude them.

AFAICT looking at the JUnit output, none of the logs in the currently failing jobs seem to be related to any of the tests introduced here. IIRC e2e-aws-ovn-ipsec-serial was the only failing job here most recently, but I'll trigger them all again and check for flakes stemming from here, just to be sure.

rexagod · 2024-09-25T14:53:31Z

/retest

neisw · 2024-09-25T15:47:23Z

Thanks @rexagod - Just seemed worth double checking since this is targeting techpreview. Did note that run was from a few commits back. Likely not related but appreciate the recheck.

openshift-trt-bot · 2024-09-25T19:09:33Z

Job Failure Risk Analysis for sha: da1d01a

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-ipsec-serial	High [sig-architecture] platform pods should not exit more than once with a non-zero exit code This test has passed 100.00% of 3 runs on jobs ['periodic-ci-openshift-release-master-ci-4.18-e2e-aws-ovn-serial' 'periodic-ci-openshift-release-master-nightly-4.18-e2e-aws-ovn-serial'] in the last 14 days. Open Bugs Excessive Restarts on ingress operator

neisw · 2024-09-25T20:25:27Z

/approve

openshift-ci · 2024-09-25T20:25:58Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jan--f, neisw, rexagod

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [neisw]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-trt-bot · 2024-09-26T03:37:31Z

Job Failure Risk Analysis for sha: da1d01a

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-ipsec-serial	High [sig-architecture] platform pods should not exit more than once with a non-zero exit code This test has passed 100.00% of 6 runs on jobs ['periodic-ci-openshift-release-master-ci-4.18-e2e-aws-ovn-serial' 'periodic-ci-openshift-release-master-nightly-4.18-e2e-aws-ovn-serial'] in the last 14 days. Open Bugs Excessive Restarts on ingress operator

openshift-trt-bot · 2024-10-01T21:44:14Z

Job Failure Risk Analysis for sha: da1d01a

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-ipsec-serial	High [sig-architecture] platform pods should not exit more than once with a non-zero exit code This test has passed 100.00% of 10 runs on jobs ['periodic-ci-openshift-release-master-ci-4.18-e2e-aws-ovn-serial' 'periodic-ci-openshift-release-master-nightly-4.18-e2e-aws-ovn-serial'] in the last 14 days. Open Bugs Excessive Restarts on ingress operator
pull-ci-openshift-origin-master-e2e-gcp-ovn-builds	IncompleteTests Tests for this run (2) are below the historical average (829): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

Refer: openshift/enhancements#1298 Signed-off-by: Pranshu Srivastava <rexagod@gmail.com>

openshift-ci · 2024-10-09T06:55:16Z

New changes are detected. LGTM label has been removed.

rexagod · 2024-10-09T07:00:32Z

test/extended/prometheus/collection_profiles.go

+	r := &runner{}
+
+	g.BeforeAll(func() {
+		if !exutil.IsTechPreviewNoUpgrade(tctx, oc.AdminConfigClient()) {


Signature changed during rebase, so modified this (from exutil.IsTechPreviewNoUpgrade(oc).

openshift-trt-bot · 2024-10-09T11:33:54Z

Job Failure Risk Analysis for sha: 0bd6a90

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-ipsec-serial	High
pull-ci-openshift-origin-master-e2e-gcp-ovn-builds	IncompleteTests Tests for this run (2) are below the historical average (1219): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

openshift-trt-bot · 2024-10-09T13:24:27Z

Job Failure Risk Analysis for sha: 0bd6a90

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-gcp-ovn-builds	IncompleteTests Tests for this run (2) are below the historical average (1219): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

rexagod · 2024-10-10T06:03:45Z

/retest-required

openshift-ci · 2024-10-10T08:19:47Z

@rexagod: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-gcp-ovn-techpreview-serial	`da1d01a`	link	false	`/test e2e-gcp-ovn-techpreview-serial`
ci/prow/e2e-aws-ovn-ipsec-serial	`0bd6a90`	link	false	`/test e2e-aws-ovn-ipsec-serial`
ci/prow/e2e-openstack-ovn	`0bd6a90`	link	false	`/test e2e-openstack-ovn`
ci/prow/e2e-aws-ovn-single-node-upgrade	`0bd6a90`	link	false	`/test e2e-aws-ovn-single-node-upgrade`
ci/prow/e2e-gcp-ovn-builds	`0bd6a90`	link	true	`/test e2e-gcp-ovn-builds`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-trt-bot · 2024-10-10T08:25:46Z

Job Failure Risk Analysis for sha: 0bd6a90

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-gcp-ovn-builds	IncompleteTests Tests for this run (15) are below the historical average (1242): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

openshift-trt-bot · 2024-10-10T08:35:34Z

Job Failure Risk Analysis for sha: 0bd6a90

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-gcp-ovn-builds	IncompleteTests Tests for this run (15) are below the historical average (1242): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

openshift-ci bot requested a review from simonpasquier June 18, 2024 08:21

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jun 18, 2024

openshift-ci bot added the vendor-update Touching vendor dir or related files label Jun 18, 2024

simonpasquier reviewed Jun 19, 2024

View reviewed changes

rexagod force-pushed the 3775 branch from 9c556df to 27aa6c4 Compare July 9, 2024 15:10

jan--f reviewed Jul 23, 2024

View reviewed changes

pkg/clioptions/suiteselection/feature_filter.go Outdated Show resolved Hide resolved

jan--f reviewed Jul 23, 2024

View reviewed changes

test/extended/prometheus/collection_profiles.go Outdated Show resolved Hide resolved

rexagod force-pushed the 3775 branch from 27aa6c4 to 49dd188 Compare July 23, 2024 17:54

jan--f reviewed Jul 29, 2024

View reviewed changes

test/extended/prometheus/collection_profiles.go Outdated Show resolved Hide resolved

jan--f reviewed Jul 29, 2024

View reviewed changes

test/extended/prometheus/collection_profiles.go Outdated Show resolved Hide resolved

rexagod force-pushed the 3775 branch from 49dd188 to a86de08 Compare July 29, 2024 22:46

rexagod commented Jul 29, 2024

View reviewed changes

pkg/clioptions/suiteselection/feature_filter.go Outdated Show resolved Hide resolved

jan--f suggested changes Jul 30, 2024

View reviewed changes

rexagod force-pushed the 3775 branch from a86de08 to 889ca0f Compare July 30, 2024 16:28

openshift-ci bot assigned jan--f Sep 23, 2024

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Sep 23, 2024

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 25, 2024

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 26, 2024

MON-3775: add tests for the MetricsCollectionProfiles feature-gate

0bd6a90

Refer: openshift/enhancements#1298 Signed-off-by: Pranshu Srivastava <rexagod@gmail.com>

rexagod force-pushed the 3775 branch from da1d01a to 0bd6a90 Compare October 9, 2024 06:55

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Oct 9, 2024

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 9, 2024

rexagod commented Oct 9, 2024

View reviewed changes

MON-3775: add tests for the MetricsCollectionProfiles feature-gate #28889

Are you sure you want to change the base?

MON-3775: add tests for the MetricsCollectionProfiles feature-gate #28889

Conversation

rexagod commented Jun 18, 2024

openshift-ci-robot commented Jun 18, 2024 • edited by openshift-ci bot Loading

rexagod commented Jun 18, 2024

rexagod commented Jun 18, 2024 • edited Loading

simonpasquier Jun 19, 2024

Choose a reason for hiding this comment

rexagod commented Jul 9, 2024

openshift-trt-bot commented Jul 9, 2024

rexagod commented Jul 9, 2024

rexagod commented Jul 9, 2024

openshift-trt-bot commented Jul 23, 2024

jan--f Jul 29, 2024

Choose a reason for hiding this comment

rexagod Jul 29, 2024

Choose a reason for hiding this comment

jan--f Jul 30, 2024

Choose a reason for hiding this comment

rexagod commented Jul 30, 2024

rexagod commented Jul 30, 2024

jan--f Jul 30, 2024

Choose a reason for hiding this comment

rexagod commented Jul 30, 2024

openshift-trt-bot commented Jul 30, 2024

rexagod commented Jul 30, 2024

openshift-trt-bot commented Jul 30, 2024

rexagod commented Jul 30, 2024

rexagod commented Sep 18, 2024 • edited Loading

openshift-trt-bot commented Sep 18, 2024

jan--f commented Sep 23, 2024

neisw commented Sep 25, 2024

neisw commented Sep 25, 2024

neisw commented Sep 25, 2024

rexagod commented Sep 25, 2024 • edited Loading

rexagod commented Sep 25, 2024

neisw commented Sep 25, 2024

openshift-trt-bot commented Sep 25, 2024

neisw commented Sep 25, 2024

openshift-ci bot commented Sep 25, 2024

openshift-trt-bot commented Sep 26, 2024

openshift-trt-bot commented Oct 1, 2024

openshift-ci bot commented Oct 9, 2024

rexagod Oct 9, 2024

Choose a reason for hiding this comment

openshift-trt-bot commented Oct 9, 2024

openshift-trt-bot commented Oct 9, 2024

rexagod commented Oct 10, 2024

openshift-ci bot commented Oct 10, 2024

openshift-trt-bot commented Oct 10, 2024

openshift-trt-bot commented Oct 10, 2024

MON-3775: add tests for the `MetricsCollectionProfiles` feature-gate #28889

MON-3775: add tests for the `MetricsCollectionProfiles` feature-gate #28889

openshift-ci-robot commented Jun 18, 2024 •

edited by openshift-ci bot

Loading

rexagod commented Jun 18, 2024 •

edited

Loading

rexagod commented Sep 18, 2024 •

edited

Loading

rexagod commented Sep 25, 2024 •

edited

Loading