Verify that podspec charms can be deployed using juju 3.5

[DnPlas]

Context

Due to #859, we must ensure that podspec charms can actually be deployed using juju 3.4 3.5 controllers and agents. In recent interactions we have made with that, the result is an error as mentioned here.

This task allow us to track the progress on that issue in the CKF team, but the actual fix may be given by juju.

What needs to get done

• File an issue in juju launchpad describing the behaviour and steps to reproduce
• Sync with the juju team to find the root cause
• If the issue is in the charms side, make the appropriate changes; otherwise make the changes required to fix this on the juju side

Definition of Done

podspec charms can be deployed from Charmhub using juju 3.4 3.5.

Update: see the comment below that justifies why we are now targeting with 3.5 instead of 3.4.

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5542.

This message was autogenerated

[DnPlas]

Juju issue: https://bugs.launchpad.net/juju/+bug/2060943

[DnPlas]

Fix is merged juju/juju#17224, we need to wait for a new version of 3.4 to be released to stable (3.4.3). According to the juju team, a candidate version will be published next week, potentially having a stable one on a close date.

EDIT: tried to use juju 3.4/edge to test the latest changes, but I couldn't bootstrap a controller, perhaps juju from that channel is broken. Will wait for an update from the team to proceed with testing.

[ca-scribner]

Juju 3.4 is delayed by a few weeks. We'll instead use 3.5, but 3.5 is currently delayed by a few days. 3.3.5 also received the bugfix we need, so as a check I'll try out the podspec charms with 3.3.5.

[ca-scribner]

Also, re the 3.4/edge channel, I tried using 3.5/edge just now and noticed the controller is pulling from a non-existent image:

  Normal  BackOff  2m31s (x21 over 7m32s)  kubelet  Back-off pulling image "docker.io/jujusolutions/jujud-operator:3.5.0"

which causes it to timeout. Seems like it wants to use the production image even though it doesn't exist yet?

[NohaIhab]

Due to delays in the release of Juju 3.4.3, which has the fix we require for this bug, we instead will bump everything to Juju 3.5. As a result, this task will now be to confirm podspec charms work for 3.5.

[NohaIhab]

Juju 3.5 was released as of yesterday,
I tested by deploying the 1.8/stable bundle on Juju 3.5, and I'm not seeing an error with podspec charms.

juju status
Model     Controller          Cloud/Region        Version  SLA          Timestamp
kubeflow  microk8s-localhost  microk8s/localhost  3.5.0    unsupported  09:53:25Z

App                        Version                  Status  Scale  Charm                    Channel          Rev  Address         Exposed  Message
admission-webhook                                   active      1  admission-webhook        1.8/stable       301  10.152.183.238  no       
argo-controller                                     active      1  argo-controller          3.3.10/stable    424  10.152.183.227  no       
dex-auth                                            active      1  dex-auth                 2.36/stable      422  10.152.183.114  no       
envoy                      res:oci-image@cc06b3e    active      1  envoy                    2.0/stable       101  10.152.183.122  no       
istio-ingressgateway                                active      1  istio-gateway            1.17/stable      723  10.152.183.146  no       
istio-pilot                                         active      1  istio-pilot              1.17/stable      827  10.152.183.111  no       
jupyter-controller                                  active      1  jupyter-controller       1.8/stable       849  10.152.183.147  no       
jupyter-ui                                          active      1  jupyter-ui               1.8/stable       858  10.152.183.85   no       
katib-controller           res:oci-image@b6a6100    active      1  katib-controller         0.16/stable      446  10.152.183.96   no       
katib-db                   8.0.35-0ubuntu0.22.04.1  active      1  mysql-k8s                8.0/stable       127  10.152.183.168  no       
katib-db-manager                                    active      1  katib-db-manager         0.16/stable      411  10.152.183.26   no       
katib-ui                                            active      1  katib-ui                 0.16/stable      422  10.152.183.103  no       
kfp-api                                             active      1  kfp-api                  2.0/stable      1035  10.152.183.230  no       
kfp-db                     8.0.35-0ubuntu0.22.04.1  active      1  mysql-k8s                8.0/stable       127  10.152.183.34   no       
kfp-metadata-writer                                 active      1  kfp-metadata-writer      2.0/stable       118  10.152.183.32   no       
kfp-persistence                                     active      1  kfp-persistence          2.0/stable      1039  10.152.183.59   no       
kfp-profile-controller                              active      1  kfp-profile-controller   2.0/stable       998  10.152.183.60   no       
kfp-schedwf                                         active      1  kfp-schedwf              2.0/stable      1052  10.152.183.36   no       
kfp-ui                                              active      1  kfp-ui                   2.0/stable      1034  10.152.183.210  no       
kfp-viewer                                          active      1  kfp-viewer               2.0/stable      1064  10.152.183.99   no       
kfp-viz                                             active      1  kfp-viz                  2.0/stable       985  10.152.183.145  no       
knative-eventing                                    active      1  knative-eventing         1.10/stable      353  10.152.183.73   no       
knative-operator                                    active      1  knative-operator         1.10/stable      328  10.152.183.126  no       
knative-serving                                     active      1  knative-serving          1.10/stable      354  10.152.183.27   no       
kserve-controller                                   active      1  kserve-controller        0.11/stable      523  10.152.183.71   no       
kubeflow-dashboard                                  active      1  kubeflow-dashboard       1.8/stable       454  10.152.183.239  no       
kubeflow-profiles                                   active      1  kubeflow-profiles        1.8/stable       355  10.152.183.249  no       
kubeflow-roles                                      active      1  kubeflow-roles           1.8/stable       187  10.152.183.88   no       
kubeflow-volumes           res:oci-image@2261827    active      1  kubeflow-volumes         1.8/stable       260  10.152.183.131  no       
metacontroller-operator                             active      1  metacontroller-operator  3.0/stable       252  10.152.183.244  no       
minio                      res:oci-image@1755999    active      1  minio                    ckf-1.8/stable   278  10.152.183.253  no       
mlmd                       res:oci-image@44abc5d    active      1  mlmd                     1.14/stable      127  10.152.183.187  no       
oidc-gatekeeper                                     active      1  oidc-gatekeeper          ckf-1.8/stable   350  10.152.183.17   no       
pvcviewer-operator                                  active      1  pvcviewer-operator       1.8/stable        30  10.152.183.223  no       
seldon-controller-manager                           active      1  seldon-core              1.17/stable      664  10.152.183.240  no       
tensorboard-controller                              active      1  tensorboard-controller   1.8/stable       257  10.152.183.204  no       
tensorboards-web-app                                active      1  tensorboards-web-app     1.8/stable       245  10.152.183.18   no       
training-operator                                   active      1  training-operator        1.7/stable       347  10.152.183.142  no       

Unit                          Workload  Agent  Address      Ports          Message
admission-webhook/0*          active    idle   10.1.61.202                 
argo-controller/0*            active    idle   10.1.61.204                 
dex-auth/0*                   active    idle   10.1.61.206                 
envoy/0*                      active    idle   10.1.61.246  9090,9901/TCP  
istio-ingressgateway/0*       active    idle   10.1.61.205                 
istio-pilot/0*                active    idle   10.1.61.207                 
jupyter-controller/0*         active    idle   10.1.61.208                 
jupyter-ui/0*                 active    idle   10.1.61.210                 
katib-controller/0*           active    idle   10.1.61.248  443,8080/TCP   
katib-db-manager/0*           active    idle   10.1.61.212                 
katib-db/0*                   active    idle   10.1.61.214                 Primary
katib-ui/0*                   active    idle   10.1.61.213                 
kfp-api/0*                    active    idle   10.1.61.215                 
kfp-db/0*                     active    idle   10.1.61.223                 Primary
kfp-metadata-writer/0*        active    idle   10.1.61.218                 
kfp-persistence/0*            active    idle   10.1.61.219                 
kfp-profile-controller/0*     active    idle   10.1.61.222                 
kfp-schedwf/0*                active    idle   10.1.61.224                 
kfp-ui/0*                     active    idle   10.1.61.225                 
kfp-viewer/0*                 active    idle   10.1.61.226                 
kfp-viz/0*                    active    idle   10.1.61.228                 
knative-eventing/0*           active    idle   10.1.61.217                 
knative-operator/0*           active    idle   10.1.61.231                 
knative-serving/0*            active    idle   10.1.61.221                 
kserve-controller/0*          active    idle   10.1.61.232                 
kubeflow-dashboard/0*         active    idle   10.1.61.230                 
kubeflow-profiles/0*          active    idle   10.1.61.234                 
kubeflow-roles/0*             active    idle   10.1.61.227                 
kubeflow-volumes/0*           active    idle   10.1.61.251  5000/TCP       
metacontroller-operator/0*    active    idle   10.1.61.229                 
minio/0*                      active    idle   10.1.61.250  9000-9001/TCP  
mlmd/0*                       active    idle   10.1.61.249  8080/TCP       
oidc-gatekeeper/0*            active    idle   10.1.61.233                 
pvcviewer-operator/0*         active    idle   10.1.61.239                 
seldon-controller-manager/0*  active    idle   10.1.61.235                 
tensorboard-controller/0*     active    idle   10.1.61.236                 
tensorboards-web-app/0*       active    idle   10.1.61.237                 
training-operator/0*          active    idle   10.1.61.238                 

we can proceed by bumping to 3.5 in our CI

[ca-scribner]

lgtm!