Move Che release workflow from Centos CI #17857

mkuznyetsov · 2020-09-14T13:05:47Z

Is your task related to a problem? Please describe.

Centos CI was announced to be deprecated around the end of this autumn. Before that happens, we would want to move to the alternative options, that would allow us to perform release processes.

This issue is only about Che release related projects, for example, QE test jobs are not covered here.

List of projects, that require CI for release:

We should also ensure that CI builds (nightly, latest, next), tests, and Hosted Che jobs are also migrated away from Centos CI.

See https://gitlab.cee.redhat.com/service/app-interface/-/blob/master/resources/jenkins/ci-centos/defaults.yaml

Ideally, we would want to preserve a workflow, of having 1 primary workflow, that would run all the subsequent tasks in the correct order, considering that some builds must be triggered after others, some can be run in parallel etc..

One of the alternatives to be considered is Github Actions. Some of the projects, e.g. Che Dashboard already run daily builds, so we should try creating a similar job, that would cover full release workflow for it.

nickboldt · 2020-12-03T17:55:35Z

chectl moved to GH action in https://github.com/che-incubator/chectl/runs/1494300249?check_suite_focus=true

nickboldt · 2020-12-09T19:30:11Z

Initial attempt to (easily) move to a GH action to invoke che-release process has failed, see #18576

A day later, it's better... but we need to load some secrets to try it out properly.

nickboldt · 2020-12-14T14:48:54Z

plugin and devfile registry RELEASE moved to GH actions:

PR to remove plugin and devfile release actions from cico: https://gitlab.cee.redhat.com/service/app-interface/-/merge_requests/13061

NOTE: this does not affect the release-preview job that Hosted Che uses https://ci.centos.org/job/devtools-che-plugin-registry-release-preview/ which should ALSO be migrated away from CICO eventually.

nickboldt · 2020-12-15T15:19:55Z

Looks like the che-bot (or the GH action bot?) doesn't have permission to push to che-incubator/community-operators.

 create mode 100644 upstream-community-operators/eclipse-che/7.23.0/checlusters.org.eclipse.che.crd.yaml
 create mode 100644 upstream-community-operators/eclipse-che/7.23.0/eclipse-che.v7.23.0.clusterserviceversion.yaml

   - Push branch update-eclipse-che-upstream-operator-7.23.0 to the 'che-incubator/community-operators' GitHub repository
Warning: Permanently added the RSA host key for IP address '140.82.114.3' to the list of known hosts.
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Error: Process completed with exit code 128.

-- https://github.com/eclipse/che-operator/runs/1557932455?check_suite_focus=true

benoitf · 2020-12-15T15:26:31Z

I've checked that che-bot has write permissions to che-incubator/community-operators github repository

benoitf · 2020-12-15T15:42:39Z

Token needs to be given to git tool (as it's an external org/repository)

nickboldt · 2020-12-18T14:54:33Z

@benoitf has created a new bot: https://github.com/che-incubator-bot and shared its credentials with the che-incubator org.

Latest news on the ci centos shutdown can be seen in https://issues.redhat.com/browse/APPSRE-2140

nickboldt · 2021-02-11T04:13:53Z

Only a few weeks left (one sprint) before cico is decommissioned!

Do we have all the nightlies, tests, and releases moved over to GH action? cc: @rhopp @benoitf

Bumping from P1 to P0 so we make sure this is the highest priority in backlog.

nickboldt · 2021-02-12T17:12:39Z

@mkuznyetsov I've added a couple TODOs to this issue:

update che-release issue template: add che-docs step; refactor table to drop... #19063
update che-release docs: remove stuff that no longer applies, document new process triggers (GH actions) https://github.com/eclipse/che-release/blob/master/README.md#phase-2---manual-steps

Please review my PR for the first, and submit a PR for the second.

mkuznyetsov · 2021-02-15T16:17:08Z

and looks like all essential release jobs for upstream Che release are now finally done on GitHub Actions workflows. So it's time for old jobs & docs cleanup.

nickboldt · 2021-02-19T18:44:10Z

Still waiting on a review of #19063 so I can merge it... :( 7 days = sadness

Also waiting on a rebase / fix in https://gitlab.cee.redhat.com/service/app-interface/-/merge_requests/15206/

benoitf · 2021-02-19T20:12:46Z

you can ping me before if I forgot some reviews

nickboldt · 2021-02-22T14:17:05Z

getting closer to being done here - both PR and MR from my previous comment are merged.

nickboldt · 2021-02-22T14:27:47Z

We could consider removing these chunks of code:

but not sure if that will break anything that @ibuziuk (rh-che, rhche, osio) or @rhopp (pr checks, happy paths) still need to run there.

mkuznyetsov · 2021-02-23T14:43:32Z

I've removed credentials for che-operator, that I have myself added for operator release on CentOS CI, so I know that only I have used them

But as for remove cico che-* credentials? section, I'm not sure if I can do it, since those still may be used by some nigthly/Hosted Che related jobs

nickboldt · 2021-02-24T20:15:36Z

Seems we still have some actively running che jobs on CICO:

Also seeing lots of recently-run builds here:

https://ci.centos.org/view/Devtools/

ericwill · 2021-02-24T20:17:35Z

The release preview jobs are for Hosted Che, which will EOL in 4 days. After that we can remove them.

nickboldt · 2021-02-24T20:19:07Z

oh, good! what about ...

and

nickboldt · 2021-02-24T20:20:53Z

For example
https://ci.centos.org/view/Devtools/job/devtools-che-devfile-registry-build-master/261/console ->
https://quay.io/repository/eclipse/che-devfile-registry?tag=ecabf9b&tab=tags seems to be the current process for publishing nightlies.

but there's also https://github.com/eclipse/che-devfile-registry/blob/master/.github/workflows/nightly-build-publish.yml -> https://github.com/eclipse/che-devfile-registry/runs/1928193391?check_suite_focus=true which seems to have attempted to push the same tags (which already exist in quay due to the cico build?)

Feb 18 10:33 GMT-5 or 3:33pm (according to CICO) trigger:

GH action: finished in 20m59s -> 10:54 gmt-5 (pushed two tags nightly + ecabf9b)
CICO: finished in 2m43s (pushed only the ecabf9b tag)

ericwill · 2021-02-24T20:23:11Z

For example
https://ci.centos.org/view/Devtools/job/devtools-che-devfile-registry-build-master/261/console ->
https://quay.io/repository/eclipse/che-devfile-registry?tag=ecabf9b&tab=tags seems to be the current process for publishing nightlies.

but there's also https://github.com/eclipse/che-devfile-registry/blob/master/.github/workflows/nightly-build-publish.yml

It's a duplicate job, we've had successful nightly build jobs via GitHub action for ages now:
https://github.com/eclipse/che-devfile-registry/runs/1928193391?check_suite_focus=true

I'll remove it from the cico job list

ericwill · 2021-02-25T20:49:49Z

For example
https://ci.centos.org/view/Devtools/job/devtools-che-devfile-registry-build-master/261/console ->
https://quay.io/repository/eclipse/che-devfile-registry?tag=ecabf9b&tab=tags seems to be the current process for publishing nightlies.
but there's also https://github.com/eclipse/che-devfile-registry/blob/master/.github/workflows/nightly-build-publish.yml

It's a duplicate job, we've had successful nightly build jobs via GitHub action for ages now:
https://github.com/eclipse/che-devfile-registry/runs/1928193391?check_suite_focus=true

I'll remove it from the cico job list

It's done: https://gitlab.cee.redhat.com/service/app-interface/-/merge_requests/15557

nickboldt · 2021-03-12T20:21:28Z

any updates here re: doc or removal of unneeded credentials?

nickboldt · 2021-03-15T13:58:30Z

Re: remove cico che-* credentials
We won't do this - there are too many remaning jobs to migrate first - see #19287 for followup (which probably needs to be split into prod and qe sub tasks).

nickboldt · 2021-03-19T17:20:22Z

Doc update:

Update release docs che-release#32 (merged)

mkuznyetsov added the kind/epic A long-lived, PM-driven feature request. Must include a checklist of items that must be completed. label Sep 14, 2020

che-bot added the status/need-triage An issue that needs to be prioritized by the curator responsible for the triage. See https://github. label Sep 14, 2020

vparfonov added area/productization severity/P1 Has a major impact to usage or development of the system. and removed status/need-triage An issue that needs to be prioritized by the curator responsible for the triage. See https://github. labels Sep 14, 2020

flacatus mentioned this issue Oct 1, 2020

Migrate che-machine-exec tasks to the github actions #17875

Closed

mkuznyetsov mentioned this issue Nov 9, 2020

Move release to GH actions eclipse-che/che-workspace-loader#38

Merged

This was referenced Dec 3, 2020

Create release.yml for chectl release (#17857) che-incubator/chectl#1014

Merged

add docker and quay logins; add GH... che-incubator/chectl#1015

Merged

nickboldt modified the milestones: 7.23, 7.24 Dec 4, 2020

This was referenced Dec 7, 2020

Add a GitHub action based release job eclipse-che/che-devfile-registry#309

Merged

Migrate cico che-release script to run in GH action / ubuntu #18576

Closed

nickboldt mentioned this issue Dec 15, 2020

Enable GH action for generating community operator PRs #18627

Closed

nickboldt mentioned this issue Jan 5, 2021

update links from Centos CI -> GH... #18724

Merged

9 tasks

nickboldt modified the milestones: 7.24, 7.25 Jan 8, 2021

nickboldt assigned mkuznyetsov Jan 18, 2021

mbenitez1607 modified the milestones: 7.25, 7.26 Jan 19, 2021

nickboldt mentioned this issue Feb 8, 2021

Improve Che Release Process Part 4 #19007

Closed

6 tasks

nickboldt modified the milestones: 7.26, 7.27 Feb 11, 2021

nickboldt added severity/blocker Causes system to crash and be non-recoverable or prevents Che developers from working on Che code. and removed severity/P1 Has a major impact to usage or development of the system. labels Feb 11, 2021

nickboldt mentioned this issue Feb 24, 2021

replace cico registry builds w/ GH action calls eclipse-che/che-release#27

Merged

9 tasks

nickboldt modified the milestones: 7.27, 7.28 Mar 12, 2021

nickboldt mentioned this issue Mar 15, 2021

Remove obsolete nightly/RC jobs from ci centos #19287

Closed

nickboldt closed this as completed Mar 19, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move Che release workflow from Centos CI #17857

Move Che release workflow from Centos CI #17857

mkuznyetsov commented Sep 14, 2020 •

edited by nickboldt

Loading

nickboldt commented Dec 3, 2020

nickboldt commented Dec 9, 2020 •

edited

Loading

nickboldt commented Dec 14, 2020

nickboldt commented Dec 15, 2020

benoitf commented Dec 15, 2020

benoitf commented Dec 15, 2020 •

edited

Loading

nickboldt commented Dec 18, 2020

nickboldt commented Feb 11, 2021 •

edited

Loading

nickboldt commented Feb 12, 2021

mkuznyetsov commented Feb 15, 2021

nickboldt commented Feb 19, 2021

benoitf commented Feb 19, 2021

nickboldt commented Feb 22, 2021

nickboldt commented Feb 22, 2021 •

edited

Loading

mkuznyetsov commented Feb 23, 2021

nickboldt commented Feb 24, 2021

ericwill commented Feb 24, 2021

nickboldt commented Feb 24, 2021

nickboldt commented Feb 24, 2021 •

edited

Loading

ericwill commented Feb 24, 2021

ericwill commented Feb 25, 2021

nickboldt commented Mar 12, 2021 •

edited

Loading

nickboldt commented Mar 15, 2021

nickboldt commented Mar 19, 2021 •

edited

Loading

Move Che release workflow from Centos CI #17857

Move Che release workflow from Centos CI #17857

Comments

mkuznyetsov commented Sep 14, 2020 • edited by nickboldt Loading

Is your task related to a problem? Please describe.

nickboldt commented Dec 3, 2020

nickboldt commented Dec 9, 2020 • edited Loading

nickboldt commented Dec 14, 2020

nickboldt commented Dec 15, 2020

benoitf commented Dec 15, 2020

benoitf commented Dec 15, 2020 • edited Loading

nickboldt commented Dec 18, 2020

nickboldt commented Feb 11, 2021 • edited Loading

nickboldt commented Feb 12, 2021

mkuznyetsov commented Feb 15, 2021

nickboldt commented Feb 19, 2021

benoitf commented Feb 19, 2021

nickboldt commented Feb 22, 2021

nickboldt commented Feb 22, 2021 • edited Loading

mkuznyetsov commented Feb 23, 2021

nickboldt commented Feb 24, 2021

ericwill commented Feb 24, 2021

nickboldt commented Feb 24, 2021

nickboldt commented Feb 24, 2021 • edited Loading

ericwill commented Feb 24, 2021

ericwill commented Feb 25, 2021

nickboldt commented Mar 12, 2021 • edited Loading

nickboldt commented Mar 15, 2021

nickboldt commented Mar 19, 2021 • edited Loading

mkuznyetsov commented Sep 14, 2020 •

edited by nickboldt

Loading

nickboldt commented Dec 9, 2020 •

edited

Loading

benoitf commented Dec 15, 2020 •

edited

Loading

nickboldt commented Feb 11, 2021 •

edited

Loading

nickboldt commented Feb 22, 2021 •

edited

Loading

nickboldt commented Feb 24, 2021 •

edited

Loading

nickboldt commented Mar 12, 2021 •

edited

Loading

nickboldt commented Mar 19, 2021 •

edited

Loading