Pebble permission denied mkdir /etc/config - Juju 3.5.1

[gustavosr98]

Bug Description

Pebble permission denied mkdir /etc/config - Juju 3.5.1

To Reproduce

juju deploy kubeflow --trust 1.8/stable

Environment

Juju 3.5.1
Charmed Kubernetes 1.29
Kubeflow 1.8

jupyter-ui 1.8/stable (rev 858)

Relevant Log Output

# juju status | grep -vE 'active|started' | grep -v grafana | grep -v stable | grep -v 22.04
Model     Controller            Cloud/Region  Version  SLA          Timestamp
kubeflow  openstack-prodstack6  k8s/default   3.5.1    unsupported  17:42:33Z

SAAS                             Status   Store                 URL
scrape-interval-config-metrics   blocked  openstack-prodstack6  admin/cos.scrape-interval-config-metrics
scrape-interval-config-monitors  blocked  openstack-prodstack6  admin/cos.scrape-interval-config-monitors

App                        Version                  Status   Scale  Charm                    Channel          Rev  Address        Exposed  Message

Unit                          Workload  Agent  Address          Ports          Message
istio-pilot/0*                waiting   idle   192.168.102.83                  Execution handled 1 errors.  See logs for details.
jupyter-ui/0*                 error     idle   192.168.102.85                  hook failed: "leader-elected"
kfp-persistence/0*            waiting   idle   192.168.102.92                  [container:persistenceagent] Waiting for Pebble services (persistenceagent).  If this persists, it could be a blockin...
kfp-profile-controller/0*     waiting   idle   192.168.102.94                  [container:kfp-profile-controller] Waiting for Pebble services (kfp-profile-controller).  If this persists, it could ...

# juju debug-log --include jupyter-ui/0
unit-jupyter-ui-0: 17:37:19 INFO unit.jupyter-ui/0.juju-log Reconcile completed successfully
unit-jupyter-ui-0: 17:37:19 INFO unit.jupyter-ui/0.juju-log Pebble plan updated with new configuration, replaning
unit-jupyter-ui-0: 17:37:19 ERROR unit.jupyter-ui/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 484, in <module>
    main(JupyterUI)
  File "/var/lib/juju/agents/unit-jupyter-ui-0/charm/venv/ops/main.py", line 439, in main
    framework.reemit()
  File "/var/lib/juju/agents/unit-jupyter-ui-0/charm/venv/ops/framework.py", line 851, in reemit
    self._reemit()
  File "/var/lib/juju/agents/unit-jupyter-ui-0/charm/venv/ops/framework.py", line 930, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 465, in main
    self._update_spawner_ui_config()
  File "./src/charm.py", line 387, in _update_spawner_ui_config
    self._upload_jwa_file_to_container(jwa_content)
  File "./src/charm.py", line 358, in _upload_jwa_file_to_container
    self.container.push(
  File "/var/lib/juju/agents/unit-jupyter-ui-0/charm/venv/ops/model.py", line 2073, in push
    self._pebble.push(str(path), source, encoding=encoding,
  File "/var/lib/juju/agents/unit-jupyter-ui-0/charm/venv/ops/pebble.py", line 2012, in push
    self._raise_on_path_error(typing.cast('_FilesResponse', resp), path)
  File "/var/lib/juju/agents/unit-jupyter-ui-0/charm/venv/ops/pebble.py", line 1964, in _raise_on_path_error
    raise PathError(error['kind'], error['message'])
ops.pebble.PathError: permission-denied - cannot create directory: mkdir /etc/config: permission denied
unit-jupyter-ui-0: 17:37:19 ERROR juju.worker.uniter.operation hook "leader-elected" (via hook dispatching script: dispatch) failed: exit status 1

Additional Context

No response

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5905.

This message was autogenerated

[DnPlas]

Thanks @gustavosr98, this is already tracked in canonical/bundle-kubeflow#921. Please check canonical/bundle-kubeflow#940 to understand what the team has decided to do regarding the juju version and why. For now please just use 3.4/stable.

Closing this issue as there is nothing else to do.