The centos image streams need an imagepullsecret now too

[brenton]

No description provided.

[openshift-ci]

@brenton: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

Restoring original timeout for clusterup.sh flake

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci]

@brenton: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

The centos image streams need an imagepullsecret now too

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[bparees]

W0629 23:40:32.260068    1013 dockerimagelookup.go:233] Docker registry lookup failed: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

i'm honestly not sure we can get some of this 3.11 CI off the ground again w/o major rework to move away from docker.io repositories in the test code and the sample imagestreams.

[brenton]

@bparees, which test is that error from? To be clear, the only job I'm fixing with this PR is the extended_clusterup one.

[openshift-ci]

@brenton: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

The centos image streams need an imagepullsecret now too

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[brenton]

It's worth noting, this file is only used by 'oc cluster up' so I don't think the other test jobs are relevant.

[bparees]

@bparees, which test is that error from? To be clear, the only job I'm fixing with this PR is the extended_clusterup one.

one of the jenkins ones

[openshift-ci]

@brenton: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/openshift-jenkins/cmd	449ebee	link	/test cmd

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[bparees]

@brenton this job shows the docker rate limiting issue:
https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/26277/test_pull_request_origin_cmd/25182/

if it's not a concern to you for now that's fine but if we ever want to get to the point where people are able to merge to 3.11 w/o someone overriding, we either need to cleanup that job (likely to be hard) or possibly just get rid of it (not great, but we can discuss with @gabemontero what coverage we lose if we drop the jenkins jobs).

that said, i'm surprised more of our 3.11 e2e jobs are not hitting docker.io rate limit issues due to use of the various docker.io centos imagestreams and such.

edit: whoops, that's not even a jenkins specific one, that's the generic cmd test job, so that one is reasonably important.

[gabemontero]

@brenton this job shows the docker rate limiting issue:
https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/26277/test_pull_request_origin_cmd/25182/

if it's not a concern to you for now that's fine but if we ever want to get to the point where people are able to merge to 3.11 w/o someone overriding, we either need to cleanup that job (likely to be hard) or possibly just get rid of it (not great, but we can discuss with @gabemontero what coverage we lose if we drop the jenkins jobs).

@bparees I see the rate limiting occuring for the imagestream used in this cmd test link you noted ^^, but I'm missing where you want me to potentially chime in on wrt "drop the jenkins job".

By jenkins job, you mean the ci/openshift-jenkins/cmd job? .... lots of components have tests in the cmd job, rith?

Are you getting at installing (sample?) imagestreams that point to redhat registries vs. docker.io?

that said, i'm surprised more of our 3.11 e2e jobs are not hitting docker.io rate limit issues due to use of the various docker.io centos imagestreams and such.

edit: whoops, that's not even a jenkins specific one, that's the generic cmd test job, so that one is reasonably important.

[bparees]

By jenkins job, you mean the ci/openshift-jenkins/cmd job? .... lots of components have tests in the cmd job, rith?
Are you getting at installing (sample?) imagestreams that point to redhat registries vs. docker.io?

so first off i was (most recently at least) looking at the cmd job and not the jenkins-cmd job, so my bad on that.

That said, the jenkins-cmd job has similar issues:
https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/26277/test_pull_request_origin_cmd/25182/

i'm not sure why we have a jenkins specific cmd job? So i'm not sure what it actually does in terms of which components it covers but it appears to be invoking oc new-app among other things (perhaps to setup for a jenkins pipeline job).

regardless, the main issue it(all of the failing jobs) seems to be hitting right now is that the imagestreams don't import successfully because of docker rate limiting, so to get it to passing we'd need to switch all those tests to use quay.io images or registry.redhat.io images(which in turn means ensuring we have the right pullsecrets on the cluster and i don't think imagestreams even use the node pullsecrets in 3.11 though i can't recall for sure, which makes getting those secrets setup even more complicated).

[gabemontero]

By jenkins job, you mean the ci/openshift-jenkins/cmd job? .... lots of components have tests in the cmd job, rith?
Are you getting at installing (sample?) imagestreams that point to redhat registries vs. docker.io?

so first off i was (most recently at least) looking at the cmd job and not the jenkins-cmd job, so my bad on that.

That said, the jenkins-cmd job has similar issues:
https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/26277/test_pull_request_origin_cmd/25182/

i'm not sure why we have a jenkins specific cmd job? So i'm not sure what it actually does in terms of which components it covers but it appears to be invoking oc new-app among other things (perhaps to setup for a jenkins pipeline job).

regardless, the main issue it(all of the failing jobs) seems to be hitting right now is that the imagestreams don't import successfully because of docker rate limiting, so to get it to passing we'd need to switch all those tests to use quay.io images or registry.redhat.io images(which in turn means ensuring we have the right pullsecrets on the cluster and i don't think imagestreams even use the node pullsecrets in 3.11 though i can't recall for sure, which makes getting those secrets setup even more complicated).

Yeah the immediate, only thinking about it a few minutes, the only "hollistic" view that comes to my mind is

• converting in some fashion the openshift namespaces sample imagestream 3.11 gets from openshift-ansible away from docker.io for the cases where any cmd/integration/e2e test uses any of those
• then an audit of the shell script and golang files for use in 3.11 tests for docker.io, converting those refs to openshift namespace imagestreams or red hat registry refs

i.e. the time consuming task we took on in 4.x when docker.io throttling arrived

Or we disable tests as you mentioned if we don't think 3.11 support and the high bar for changes there does not require those tests to still be used

[brenton]

@bparees We're talking about two different things. This PR is to fix the clusterup.sh job. You can see it's passing now: https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/26277/test_pull_request_origin_extended_clusterup-release-3.11/2383/ (and was failing before I started working on it). This PR does not affect the test_cmd job you mentioned in any way.

[brenton]

I have a different PR where I'm working to fix test_cmd. I'm only now picking it back up though (been a long day). This PR is a WIP: #26292

[brenton]

When I say test_cmd I mean the jenkins/cmd job. It's the same thing.

[bparees]

@bparees We're talking about two different things. This PR is to fix the clusterup.sh job.

yup, I understand and i apologize for not being clear. I was just commenting that in general getting 3.11 CI back to a state where it passes is going to take additional efforts.

to the extent that this PR gets one of the jobs green, that's great and we should merge it :)

[bparees]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bparees, brenton

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/oc/clusteradd/components/default-imagestreams/OWNERS [bparees]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment