Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cleanup the old release infrastructure #573

Closed
jlewi opened this issue Jan 23, 2020 · 6 comments
Closed

Cleanup the old release infrastructure #573

jlewi opened this issue Jan 23, 2020 · 6 comments
Labels
area/engprod kind/process lifecycle/stale priority/p1

Comments

@jlewi
Copy link
Contributor

jlewi commented Jan 23, 2020

Follow on to #450

We should clean up the old release infrastructure.

e.g. we should have an old cron job for updating the jupyter web app that we can delete
@issue-label-bot
Copy link

Issue-Label Bot is automatically applying the labels:

Label Probability
feature 0.91

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

@jlewi jlewi mentioned this issue Jan 23, 2020
Merged
@jlewi
Copy link
Contributor Author

jlewi commented Jan 23, 2020

Project: kubeflow-releasing
Cluster: kf-releasing

It looks like #366 was the issue tracking the creation of this cluster.

Although it looks like its actually a 0.6 cluster; so maybe it got recreated since #366 was merged

It doesn't look like anything is running in that cluster

kubectl -n kubeflow-releasing get all
NAME                                                  READY   STATUS              RESTARTS   AGE
pod/tf-operator-release-d746bde9-kunming-1083330308   0/2     ContainerCreating   0          204d
pod/tf-operator-release-d746bde9-kunming-2198430506   0/2     ContainerCreating   0          204d
pod/tf-operator-release-d746bde9-kunming-2409835398   0/2     Completed           0          208d
pod/tf-operator-release-d746bde9-kunming-2450483485   0/2     ContainerCreating   0          204d
pod/tf-operator-release-d746bde9-kunming-3548394101   0/2     ContainerCreating   0          204d
pod/tf-operator-release-d746bde9-kunming-4273792944   0/2     ContainerCreating   0          204d

kubectl -n kubeflow-releasing get all
NAME                                                  READY   STATUS              RESTARTS   AGE
pod/tf-operator-release-d746bde9-kunming-1083330308   0/2     ContainerCreating   0          204d
pod/tf-operator-release-d746bde9-kunming-2198430506   0/2     ContainerCreating   0          204d
pod/tf-operator-release-d746bde9-kunming-2409835398   0/2     Completed           0          208d
pod/tf-operator-release-d746bde9-kunming-2450483485   0/2     ContainerCreating   0          204d
pod/tf-operator-release-d746bde9-kunming-3548394101   0/2     ContainerCreating   0          204d
pod/tf-operator-release-d746bde9-kunming-4273792944   0/2     ContainerCreating   0          204d
jlewi@jlewi-glaptop2:~/git_kubeflow-kubeflow$ kubectl -n kubeflow get all
NAME                                                         READY   STATUS             RESTARTS   AGE
pod/admission-webhook-bootstrap-stateful-set-0               1/1     Running            26         148d
pod/admission-webhook-deployment-57bf887886-xgcfd            1/1     Running            0          6d5h
pod/application-controller-stateful-set-0                    1/1     Running            0          148d
pod/argo-ui-67659f4795-hm82n                                 1/1     Running            0          148d
pod/centraldashboard-85dbd5d544-5hwjk                        1/1     Running            0          148d
pod/jupyter-web-app-deployment-5c8dc58f-7lzp4                1/1     Running            0          148d
pod/katib-controller-79578f6f89-wb26f                        1/1     Running            1          148d
pod/katib-db-7dcbc9c964-wfvsm                                1/1     Running            0          148d
pod/katib-manager-7b8f875b8f-hgmj6                           1/1     Running            2          148d
pod/katib-manager-rest-58867b555f-2jtpb                      1/1     Running            0          148d
pod/katib-suggestion-bayesianoptimization-5b86c549f9-x6kfc   1/1     Running            0          148d
pod/katib-suggestion-grid-db47c7869-ndszd                    1/1     Running            0          148d
pod/katib-suggestion-hyperband-749c777fc7-k2rdj              1/1     Running            0          148d
pod/katib-suggestion-nasrl-b89dfc475-j95pp                   1/1     Running            0          148d
pod/katib-suggestion-random-fdc77bfc9-tlvtc                  1/1     Running            0          148d
pod/katib-ui-6984cf5975-6clb2                                1/1     Running            0          148d
pod/metacontroller-0                                         1/1     Running            0          148d
pod/metadata-db-5f79f99bcc-57t77                             1/1     Running            0          148d
pod/metadata-deployment-75ccb97d7d-6rtm4                     1/1     Running            0          148d
pod/metadata-deployment-75ccb97d7d-b5pxz                     1/1     Running            1          148d
pod/metadata-deployment-75ccb97d7d-ljjtc                     1/1     Running            0          148d
pod/metadata-ui-ff59689b-l8vnw                               1/1     Running            0          148d
pod/minio-75b7c8f4cf-vkbvs                                   0/1     Pending            0          148d
pod/ml-pipeline-7c97955449-vjc6f                             0/1     CrashLoopBackOff   41384      148d
pod/notebook-controller-deployment-66b548f989-znm5b          1/1     Running            0          148d
pod/pytorch-operator-696749cff6-hnjsx                        1/1     Running            5          148d
pod/spartakus-volunteer-6f9755884c-zrhbr                     1/1     Running            0          148d
pod/tensorboard-86d54ccdb4-w6kqk                             1/1     Running            0          148d
pod/tf-job-dashboard-66cd957b8f-77d6s                        1/1     Running            0          148d
pod/tf-job-operator-5dcb4d8db6-76t2b                         1/1     Running            459        148d
pod/workflow-controller-68c5865896-r59qk                     1/1     Running            0          148d

NAME                                            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
service/admission-webhook-service               ClusterIP   10.23.253.183   <none>        443/TCP             148d
service/application-controller-service          ClusterIP   10.23.255.240   <none>        443/TCP             148d
service/argo-ui                                 NodePort    10.23.244.76    <none>        80:32070/TCP        148d
service/centraldashboard                        ClusterIP   10.23.250.39    <none>        80/TCP              148d
service/jupyter-web-app-service                 ClusterIP   10.23.244.40    <none>        80/TCP              148d
service/katib-controller                        ClusterIP   10.23.245.16    <none>        443/TCP             148d
service/katib-db                                ClusterIP   10.23.244.98    <none>        3306/TCP            148d
service/katib-manager                           ClusterIP   10.23.240.212   <none>        6789/TCP            148d
service/katib-manager-rest                      ClusterIP   10.23.242.141   <none>        80/TCP              148d
service/katib-suggestion-bayesianoptimization   ClusterIP   10.23.251.104   <none>        6789/TCP            148d
service/katib-suggestion-grid                   ClusterIP   10.23.250.165   <none>        6789/TCP            148d
service/katib-suggestion-hyperband              ClusterIP   10.23.242.86    <none>        6789/TCP            148d
service/katib-suggestion-nasrl                  ClusterIP   10.23.244.123   <none>        6789/TCP            148d
service/katib-suggestion-random                 ClusterIP   10.23.245.53    <none>        6789/TCP            148d
service/katib-ui                                ClusterIP   10.23.250.65    <none>        80/TCP              148d
service/metadata-db                             ClusterIP   10.23.245.189   <none>        3306/TCP            148d
service/metadata-service                        ClusterIP   10.23.242.73    <none>        8080/TCP            148d
service/metadata-ui                             ClusterIP   10.23.255.214   <none>        80/TCP              148d
service/minio-service                           ClusterIP   10.23.245.106   <none>        9000/TCP            148d
service/ml-pipeline                             ClusterIP   10.23.240.235   <none>        8888/TCP,8887/TCP   148d
service/notebook-controller-service             ClusterIP   10.23.254.223   <none>        443/TCP             148d
service/pytorch-operator                        ClusterIP   10.23.253.222   <none>        8443/TCP            148d
service/tensorboard                             ClusterIP   10.23.254.234   <none>        9000/TCP            148d
service/tf-job-dashboard                        ClusterIP   10.23.249.82    <none>        80/TCP              148d
service/tf-job-operator                         ClusterIP   10.23.246.86    <none>        8443/TCP            148d

NAME                                                    DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/admission-webhook-deployment            1         1         1            1           148d
deployment.apps/argo-ui                                 1         1         1            1           148d
deployment.apps/centraldashboard                        1         1         1            1           148d
deployment.apps/jupyter-web-app-deployment              1         1         1            1           148d
deployment.apps/katib-controller                        1         1         1            1           148d
deployment.apps/katib-db                                1         1         1            1           148d
deployment.apps/katib-manager                           1         1         1            1           148d
deployment.apps/katib-manager-rest                      1         1         1            1           148d
deployment.apps/katib-suggestion-bayesianoptimization   1         1         1            1           148d
deployment.apps/katib-suggestion-grid                   1         1         1            1           148d
deployment.apps/katib-suggestion-hyperband              1         1         1            1           148d
deployment.apps/katib-suggestion-nasrl                  1         1         1            1           148d
deployment.apps/katib-suggestion-random                 1         1         1            1           148d
deployment.apps/katib-ui                                1         1         1            1           148d
deployment.apps/metadata-db                             1         1         1            1           148d
deployment.apps/metadata-deployment                     3         3         3            3           148d
deployment.apps/metadata-ui                             1         1         1            1           148d
deployment.apps/minio                                   1         1         1            0           148d
deployment.apps/ml-pipeline                             1         1         1            0           148d
deployment.apps/notebook-controller-deployment          1         1         1            1           148d
deployment.apps/pytorch-operator                        1         1         1            1           148d
deployment.apps/spartakus-volunteer                     1         1         1            1           148d
deployment.apps/tensorboard                             1         1         1            1           148d
deployment.apps/tf-job-dashboard                        1         1         1            1           148d
deployment.apps/tf-job-operator                         1         1         1            1           148d
deployment.apps/workflow-controller                     1         1         1            1           148d

NAME                                                               DESIRED   CURRENT   READY   AGE
replicaset.apps/admission-webhook-deployment-57bf887886            1         1         1       148d
replicaset.apps/argo-ui-67659f4795                                 1         1         1       148d
replicaset.apps/centraldashboard-85dbd5d544                        1         1         1       148d
replicaset.apps/jupyter-web-app-deployment-5c8dc58f                1         1         1       148d
replicaset.apps/katib-controller-79578f6f89                        1         1         1       148d
replicaset.apps/katib-db-7dcbc9c964                                1         1         1       148d
replicaset.apps/katib-manager-7b8f875b8f                           1         1         1       148d
replicaset.apps/katib-manager-rest-58867b555f                      1         1         1       148d
replicaset.apps/katib-suggestion-bayesianoptimization-5b86c549f9   1         1         1       148d
replicaset.apps/katib-suggestion-grid-db47c7869                    1         1         1       148d
replicaset.apps/katib-suggestion-hyperband-749c777fc7              1         1         1       148d
replicaset.apps/katib-suggestion-nasrl-b89dfc475                   1         1         1       148d
replicaset.apps/katib-suggestion-random-fdc77bfc9                  1         1         1       148d
replicaset.apps/katib-ui-6984cf5975                                1         1         1       148d
replicaset.apps/metadata-db-5f79f99bcc                             1         1         1       148d
replicaset.apps/metadata-deployment-75ccb97d7d                     3         3         3       148d
replicaset.apps/metadata-ui-ff59689b                               1         1         1       148d
replicaset.apps/minio-75b7c8f4cf                                   1         1         0       148d
replicaset.apps/ml-pipeline-7c97955449                             1         1         0       148d
replicaset.apps/notebook-controller-deployment-66b548f989          1         1         1       148d
replicaset.apps/pytorch-operator-696749cff6                        1         1         1       148d
replicaset.apps/spartakus-volunteer-6f9755884c                     1         1         1       148d
replicaset.apps/tensorboard-86d54ccdb4                             1         1         1       148d
replicaset.apps/tf-job-dashboard-66cd957b8f                        1         1         1       148d
replicaset.apps/tf-job-operator-5dcb4d8db6                         1         1         1       148d
replicaset.apps/workflow-controller-68c5865896                     1         1         1       148d

NAME                                                        DESIRED   CURRENT   AGE
statefulset.apps/admission-webhook-bootstrap-stateful-set   1         1         148d
statefulset.apps/application-controller-stateful-set        1         1         148d
statefulset.apps/metacontroller                             1         1         148d

I'm going to delete this cluster and the associated deployments.

jlewi pushed a commit to jlewi/testing that referenced this issue Jan 23, 2020
Remove Kubeflow configs for the old kf-releasing cluster
03d6fc0 
* This cluster is no longer used (See kubeflow#573). These configs refer to
  a very old Kubeflow config; i.e. still using ksonnet.

* Related to kubeflow#573
@jlewi jlewi mentioned this issue Jan 23, 2020
Merged
@jlewi
Copy link
Contributor Author

jlewi commented Jan 23, 2020

project: kubeflow-releasing
cluster: kf-releasing-0-6-2

 kubectl get cronjobs
NAME              SCHEDULE      SUSPEND   ACTIVE   LAST SCHEDULE   AGE
jupyter-updater   */6 * * * *   False     0        113s            140d

This is the old cron job for updating jupyter. We don't use this anymore (see #450) so I'm going to delete it.

@jlewi
Copy link
Contributor Author

jlewi commented Jan 23, 2020

I think we can close this issue as soon as #574 is merged

k8s-ci-robot pushed a commit that referenced this issue Jan 27, 2020
@jlewi @k8s-ci-robot
Remove Kubeflow configs for the old kf-releasing cluster (#574)
bb549cb 
* This cluster is no longer used (See #573). These configs refer to
  a very old Kubeflow config; i.e. still using ksonnet.

* Related to #573
@stale
Copy link

stale bot commented Apr 23, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in one week if no further activity occurs. Thank you for your contributions.

@stale
Copy link

stale bot commented Apr 30, 2020

This issue has been closed due to inactivity.

@stale stale bot closed this as completed Apr 30, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/engprod kind/process lifecycle/stale priority/p1
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jlewi