Bug 2020107: Remove run-level label

[mcoops]

Given the original commit for this was in 2018, it might be possible to remove the label now entirely. Given #24 is specifically setting it a dependency on the openshift-apiserver, I doubt it,

Will use this PR for testing and further discussion.

[mcoops]

/retest

[mcoops]

/retest

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[mcoops]

/remove-lifecycle stale

[wking]

Background here. Install doesn't seem much slower in the e2e-agnostic presubmit that installs with the new code:

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_cluster-version-operator/623/pull-ci-openshift-cluster-version-operator-master-e2e-agnostic/1412624472652910592/artifacts/e2e-agnostic/ipi-install-install/artifacts/.openshift_install.log | tail
time="2021-07-07T04:54:29Z" level=info msg="Install complete!"
time="2021-07-07T04:54:29Z" level=info msg="To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/tmp/installer/auth/kubeconfig'"
time="2021-07-07T04:54:29Z" level=info msg="Access the OpenShift web-console here: https://console-openshift-console.apps.ci-op-jjir8t9y-3302f.ci.azure.devcluster.openshift.com"
time="2021-07-07T04:54:29Z" level=info msg="Login to the console with user: \"kubeadmin\", and password: REDACTED
time="2021-07-07T04:54:29Z" level=debug msg="Time elapsed per stage:"
time="2021-07-07T04:54:29Z" level=debug msg="                  : 16m45s"
time="2021-07-07T04:54:29Z" level=debug msg="Bootstrap Complete: 7m32s"
time="2021-07-07T04:54:29Z" level=debug msg=" Bootstrap Destroy: 4m49s"
time="2021-07-07T04:54:29Z" level=debug msg=" Cluster Operators: 11m31s"
time="2021-07-07T04:54:29Z" level=info msg="Time elapsed: 40m42s"

Although that's ~3m slower than the old code used for the e2e-agnostic-upgrade presubmit:

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_cluster-version-operator/623/pull-ci-openshift-cluster-version-operator-master-e2e-agnostic-upgrade/1412961787673841664/artifacts/e2e-agnostic-upgrade/ipi-install-install-stableinitial/artifacts/.openshift_install.log | tail
time="2021-07-08T03:21:40Z" level=info msg="Install complete!"
time="2021-07-08T03:21:40Z" level=info msg="To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/tmp/installer/auth/kubeconfig'"
time="2021-07-08T03:21:40Z" level=info msg="Access the OpenShift web-console here: https://console-openshift-console.apps.ci-op-26hc2j3r-7ee27.ci.azure.devcluster.openshift.com"
time="2021-07-08T03:21:40Z" level=info msg="Login to the console with user: \"kubeadmin\", and password: REDACTED
time="2021-07-08T03:21:40Z" level=debug msg="Time elapsed per stage:"
time="2021-07-08T03:21:40Z" level=debug msg="                  : 14m56s"
time="2021-07-08T03:21:40Z" level=debug msg="Bootstrap Complete: 6m59s"
time="2021-07-08T03:21:40Z" level=debug msg=" Bootstrap Destroy: 4m54s"
time="2021-07-08T03:21:40Z" level=debug msg=" Cluster Operators: 16m52s"
time="2021-07-08T03:21:40Z" level=info msg="Time elapsed: 43m47s"

I'm not sure if the slowdown is statistically significant or a fluke.

Also, the CVO requires labels from the manifest to exist in the in-cluster resource, but we do not clear unrecognised labels, so the update presubmit still has them after updating to the patched release:

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_cluster-version-operator/623/pull-ci-openshift-cluster-version-operator-master-e2e-agnostic-upgrade/1412961787673841664/artifacts/e2e-agnostic-upgrade/gather-extra/artifacts/namespaces.json | jq '.items[].metadata | select(.name == "openshift-cluster-version").labels'
{
  "kubernetes.io/metadata.name": "openshift-cluster-version",
  "name": "openshift-cluster-version",
  "olm.operatorgroup.uid/0e8650ee-d36e-47bf-bdb3-b48357056c6b": "",
  "openshift.io/cluster-monitoring": "true",
  "openshift.io/run-level": "1"
}

Two questions:

1. Is that label's presence a problem? Or is this label just a bootstrap-time thing?
2. If it's a problem, is there an explicit value we can set for the label that says "don't give me any run-level handling; I'm fine without it"?

[openshift-ci]

@mcoops: This pull request references Bugzilla bug 2020107, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target release (4.10.0) matches configured target release for branch (4.10.0)
• bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @jianlinliu

In response to this:

Bug 2020107: Remove run-level label

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mcoops]

Two questions:

1. Is that label's presence a problem? Or is this label just a bootstrap-time thing?
2. If it's a problem, is there an explicit value we can set for the label that says "don't give me any run-level handling; I'm fine without it"?

1. It would be good to remove the label entirely. Originally it was just a bootstrap time thing, but it also means no Security Context Constraint is applied at all. Even tho without the runlevel it gets admitted with hostaccess, that means it at least requires the pod to be run with a UID and gets a SELinux context. Better than nothing.

2. That's a good point, dang. In theory if we set it to anything but a 0 or 1, then the code should still apply an SCC: https://github.com/openshift/origin/blob/0104fb51cb31e1f5920b778b17eec8b3286eefee/vendor/k8s.io/kubernetes/openshift-kube-apiserver/admission/namespaceconditions/labelcondition.go#L26
   But then it's kind of messy. On the other hand, its no worse than the status quo at the moment, the issue is just existing customers who upgrade wont get the benefit of this change. What do you think @deads2k?

[wking]

openshift.io/run-level: ooh, yeah, gimme those security context constraints ;)

[deads2k]

I suggest building a way to clear the label or annotation. We do it in library-go using a trailing - as the signal to remove to match oc label and oc annotate.

[mcoops]

/retest

[mcoops]

/retest

[mcoops]

/retest e2e-agnostic

[mcoops]

/test e2e-agnostic

[mcoops]

/test e2e-agnostic-upgrade

[mcoops]

I had a quick look at implementing something similar to: openshift/library-go#727, however as we'd need to specify openshift.io/run-level-: that's actually an invalid label so it never gets created failing with an invalid label error. We also don't particularly want to code specifically for the upgrade or make the CVO aware of run-levels if we can avoid it.

So I think it might be just easier at the moment to set it to an empty string with a comment. That then ensures that on a new install it doesn't install with a run-level set, and then on update it also unsets it. Otherwise if we just remove it from the manifest it only removes it from a new new cluster install.

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_cluster-version-operator/623/pull-ci-openshift-cluster-version-operator-master-e2e-agnostic/1464219753190002688/artifacts/e2e-agnostic/gather-extra/artifacts/namespaces.json | jq '.items[].metadata | select(.name == "openshift-cluster-version").labels'
{
  "kubernetes.io/metadata.name": "openshift-cluster-version",
  "name": "openshift-cluster-version",
  "olm.operatorgroup.uid/b0aeeb7a-918d-4b76-893d-856f61f4bac9": "",
  "openshift.io/cluster-monitoring": "true",
  "openshift.io/run-level": "",
  "pod-security.kubernetes.io/audit": "privileged",
  "pod-security.kubernetes.io/enforce": "privileged",
  "pod-security.kubernetes.io/warn": "privileged"
}

Then on upgrade:

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_cluster-version-operator/623/pull-ci-openshift-cluster-version-operator-master-e2e-agnostic-upgrade/1465099749160914944/artifacts/e2e-agnostic-upgrade/gather-extra/artifacts/namespaces.json | jq '.items[].metadata | select(.name == "openshift-cluster-version").labels'
{
  "kubernetes.io/metadata.name": "openshift-cluster-version",
  "name": "openshift-cluster-version",
  "olm.operatorgroup.uid/0d609421-4ec9-4b0c-b3ee-37ef0c060fca": "",
  "openshift.io/cluster-monitoring": "true",
  "openshift.io/run-level": "",
  "pod-security.kubernetes.io/audit": "privileged",
  "pod-security.kubernetes.io/enforce": "privileged",
  "pod-security.kubernetes.io/warn": "privileged"
}

And admits with an SCC assigned:

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_cluster-version-operator/623/pull-ci-openshift-cluster-version-operator-master-e2e-agnostic-upgrade/1465099749160914944/artifacts/e2e-agnostic-upgrade/gather-extra/artifacts/pods.json | jq '.items[].metadata | select(.name | startswith("cluster-version-operator-")).annotations'
{
  "openshift.io/scc": "hostaccess"
}

Also matches what the MCO is now doing: openshift/machine-config-operator#2655

Although I got no idea what's wrong with the tests @wking?

[wking]

/lgtm

We can come back and check install durations in a week once we have a larger sample size, to gauge any slowdowns.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mcoops, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [wking]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-ci]

@mcoops: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/golangci-lint	74d8547	link	true	/test golangci-lint
ci/prow/gofmt	74d8547	link	true	/test gofmt

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-ci]

@mcoops: All pull requests linked via external trackers have merged:

• openshift/cluster-version-operator#623

Bugzilla bug 2020107 has been moved to the MODIFIED state.

In response to this:

Bug 2020107: Remove run-level label

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.