Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added kubeflow pipelines driver rock image #159

Merged
merged 5 commits into from
Dec 3, 2024
Merged

Conversation

BON4
Copy link
Contributor

@BON4 BON4 commented Nov 30, 2024

Description


  • Add rockcraft.yaml
  • Add tests
  • Add tox.ini

This is re-open request for #109.


Logs

Using kfp-operators. Modified kfp-api/config.yaml. Replaced "gcr.io/ml-pipeline/kfp-driver" with "vyurchenko581/kfp-driver:2.2.0".

Additionally, the default port for KUBEFLOW_LOCAL_HOST has been changed from 8080 to 8081 due to its unavailability on my local machine, as it is already in use.

Juju status after all tests passed:

Model     Controller          Cloud/Region        Version  SLA          Timestamp
kubeflow  microk8s-localhost  microk8s/localhost  3.5.4    unsupported  14:32:11+02:00

App                      Version                  Status  Scale  Charm                    Channel       Rev  Address         Exposed  Message
argo-controller                                   active      1  argo-controller          latest/edge   615  10.152.183.134  no       
envoy                                             active      1  envoy                    latest/edge   307  10.152.183.51   no       
istio-ingressgateway                              active      1  istio-gateway            latest/edge  1287  10.152.183.229  no       
istio-pilot                                       active      1  istio-pilot              latest/edge  1235  10.152.183.61   no       
kfp-api                                           active      1  kfp-api                                  0  10.152.183.191  no       
kfp-db                   8.0.37-0ubuntu0.22.04.3  active      1  mysql-k8s                8.0/stable    180  10.152.183.188  no       
kfp-metadata-writer                               active      1  kfp-metadata-writer                      0  10.152.183.184  no       
kfp-persistence                                   active      1  kfp-persistence                          0  10.152.183.196  no       
kfp-profile-controller                            active      1  kfp-profile-controller                   0  10.152.183.246  no       
kfp-schedwf                                       active      1  kfp-schedwf                              0  10.152.183.91   no       
kfp-ui                                            active      1  kfp-ui                                   0  10.152.183.69   no       
kfp-viewer                                        active      1  kfp-viewer                               0  10.152.183.183  no       
kfp-viz                                           active      1  kfp-viz                                  0  10.152.183.252  no       
kubeflow-profiles                                 active      1  kubeflow-profiles        latest/edge   456  10.152.183.70   no       
kubeflow-roles                                    active      1  kubeflow-roles           latest/edge   264  10.152.183.62   no       
metacontroller-operator                           active      1  metacontroller-operator  latest/edge   349  10.152.183.107  no       
minio                    res:oci-image@220b31a    active      1  minio                    latest/edge   376  10.152.183.233  no       
mlmd                                              active      1  mlmd                     latest/edge   243  10.152.183.47   no       

Unit                        Workload  Agent  Address       Ports          Message
argo-controller/0*          active    idle   10.1.142.172                 
envoy/0*                    active    idle   10.1.142.158                 
istio-ingressgateway/0*     active    idle   10.1.142.176                 
istio-pilot/0*              active    idle   10.1.142.188                 
kfp-api/0*                  active    idle   10.1.142.140                 
kfp-db/0*                   active    idle   10.1.142.129                 Primary
kfp-metadata-writer/0*      active    idle   10.1.142.148                 
kfp-persistence/0*          active    idle   10.1.142.133                 
kfp-profile-controller/0*   active    idle   10.1.142.191                 
kfp-schedwf/0*              active    idle   10.1.142.130                 
kfp-ui/0*                   active    idle   10.1.142.157                 
kfp-viewer/0*               active    idle   10.1.142.159                 
kfp-viz/0*                  active    idle   10.1.142.136                 
kubeflow-profiles/0*        active    idle   10.1.142.170                 
kubeflow-roles/0*           active    idle   10.1.142.180                 
metacontroller-operator/0*  active    idle   10.1.142.173                 
minio/0*                    active    idle   10.1.142.163  9000-9001/TCP  
mlmd/0*                     active    idle   10.1.142.151                 

Logs of passed integration tests tox -vve bundle-integration-v2 -- --model kubeflow --bundle=./tests/integration/bundles/kfp_latest_edge.yaml.j2 --keep-models:

ROOT: 244 D setup logging to DEBUG on pid 2001545 [tox/report.py:222]
bundle-integration-v2: 330 D clear env temp folder /home/yurchenko/kfp-operators/.tox/bundle-integration-v2/tmp [tox/tox_env/api.py:313]
bundle-integration-v2: 339 I find interpreter for spec PythonSpec(path=/home/yurchenko/kfp-operators/env/bin/python) [virtualenv/discovery/builtin.py:74]
bundle-integration-v2: 340 D filesystem is case-sensitive [virtualenv/info.py:26]
bundle-integration-v2: 341 D got python info of %s from (PosixPath('/usr/bin/python3.12'), PosixPath('/home/yurchenko/.local/share/virtualenv/py_info/1/f0d7a494a3f776233427cb85a7e198c7cf4913b50a203c6febc678cc4f5bf265.json')) [virtualenv/app_data/via_disk_folder.py:133]
bundle-integration-v2: 341 I proposed PythonInfo(spec=CPython3.12.3.final.0-64, system=/usr/bin/python3.12, exe=/home/yurchenko/kfp-operators/env/bin/python, platform=linux, version='3.12.3 (main, Nov  6 2024, 18:32:19) [GCC 13.2.0]', encoding_fs_io=utf-8-utf-8) [virtualenv/discovery/builtin.py:81]
bundle-integration-v2: 341 D accepted PythonInfo(spec=CPython3.12.3.final.0-64, system=/usr/bin/python3.12, exe=/home/yurchenko/kfp-operators/env/bin/python, platform=linux, version='3.12.3 (main, Nov  6 2024, 18:32:19) [GCC 13.2.0]', encoding_fs_io=utf-8-utf-8) [virtualenv/discovery/builtin.py:83]
bundle-integration-v2: 392 W commands[0]> pytest -vv --tb=native -s --model kubeflow --bundle=./tests/integration/bundles/kfp_latest_edge.yaml.j2 --keep-models /home/yurchenko/kfp-operators/tests/integration/test_kfp_functional_v2.py [tox/tox_env/api.py:427]
/home/yurchenko/kfp-operators/.tox/bundle-integration-v2/lib/python3.12/site-packages/paramiko/pkey.py:100: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be removed from this module in 48.0.0.
  "cipher": algorithms.TripleDES,
/home/yurchenko/kfp-operators/.tox/bundle-integration-v2/lib/python3.12/site-packages/paramiko/transport.py:259: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be removed from this module in 48.0.0.
  "class": algorithms.TripleDES,
============================================================================================= test session starts ==============================================================================================
platform linux -- Python 3.12.3, pytest-8.3.2, pluggy-1.5.0 -- /home/yurchenko/kfp-operators/.tox/bundle-integration-v2/bin/python
cachedir: .tox/bundle-integration-v2/.pytest_cache
rootdir: /home/yurchenko/kfp-operators
configfile: pyproject.toml
plugins: operator-0.35.0, asyncio-0.21.2, anyio-4.4.0
asyncio: mode=Mode.STRICT
collected 6 items                                                                                                                                                                                              

tests/integration/test_kfp_functional_v2.py::test_build_and_deploy PASSED
tests/integration/test_kfp_functional_v2.py::test_upload_pipeline Forwarding from 127.0.0.1:8081 -> 3000
Forwarding from [::1]:8081 -> 3000
Handling connection for 8081
Handling connection for 8081
PASSED
tests/integration/test_kfp_functional_v2.py::test_create_and_monitor_run Handling connection for 8081
Handling connection for 8081
Experiment details: http://localhost:8081/#/experiments/details/0886d7d7-9fa4-43f4-ab66-45cc077bf044
Handling connection for 8081
Experiment details: http://localhost:8081/#/experiments/details/0886d7d7-9fa4-43f4-ab66-45cc077bf044
Handling connection for 8081
Run details: http://localhost:8081/#/runs/details/8c1bfee7-1f3f-4185-877e-dc8679f49c23
Handling connection for 8081
Handling connection for 8081
Handling connection for 8081
Handling connection for 8081
Handling connection for 8081
Handling connection for 8081
Handling connection for 8081
Handling connection for 8081
Handling connection for 8081
Handling connection for 8081
PASSEDHandling connection for 8081

tests/integration/test_kfp_functional_v2.py::test_create_and_monitor_recurring_run Handling connection for 8081
Handling connection for 8081
Handling connection for 8081
Handling connection for 8081
Experiment details: http://localhost:8081/#/experiments/details/08bd7e76-b6b8-4a0a-90a3-48bea3d9e57e
Handling connection for 8081
Handling connection for 8081
Handling connection for 8081
Handling connection for 8081
Handling connection for 8081
Handling connection for 8081
Handling connection for 8081
PASSEDHandling connection for 8081
Handling connection for 8081
Handling connection for 8081

tests/integration/test_kfp_functional_v2.py::test_apply_sample_viewer PASSED
tests/integration/test_kfp_functional_v2.py::test_viz_server_healthcheck PASSED

=============================================================================================== warnings summary ===============================================================================================
tests/integration/test_kfp_functional_v2.py::test_build_and_deploy
  /home/yurchenko/kfp-operators/.tox/bundle-integration-v2/lib/python3.12/site-packages/sh.py:1980: DeprecationWarning: This process (pid=2001570) is multi-threaded, use of fork() may lead to deadlocks in the child.
    self.pid = os.fork()

tests/integration/test_kfp_functional_v2.py::test_upload_pipeline
  /home/yurchenko/kfp-operators/.tox/bundle-integration-v2/lib/python3.12/site-packages/kfp/client/client.py:159: FutureWarning: This client only works with Kubeflow Pipeline v2.0.0-beta.2 and later versions.
    warnings.warn(

tests/integration/test_kfp_functional_v2.py: 31 warnings
  /home/yurchenko/kfp-operators/.tox/bundle-integration-v2/lib/python3.12/site-packages/kfp_server_api/rest.py:47: DeprecationWarning: HTTPResponse.getheader() is deprecated and will be removed in urllib3 v2.1.0. Instead use HTTPResponse.headers.get(name, default).
    return self.urllib3_response.getheader(name, default)

tests/integration/test_kfp_functional_v2.py::test_viz_server_healthcheck
  /home/yurchenko/kfp-operators/.tox/bundle-integration-v2/lib/python3.12/site-packages/_pytest/stash.py:108: RuntimeWarning: coroutine 'Connection.reconnect' was never awaited
    del self._storage[key]
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================================================== 6 passed, 34 warnings in 441.17s (0:07:21) ==================================================================================
Task exception was never retrieved
future: <Task finished name='Task-32' coro=<Model._watch.<locals>._all_watcher() done, defined at /home/yurchenko/kfp-operators/.tox/bundle-integration-v2/lib/python3.12/site-packages/juju/model.py:1194> exception=TypeError('Passing coroutines is forbidden, use tasks explicitly.')>
Traceback (most recent call last):
  File "/home/yurchenko/kfp-operators/.tox/bundle-integration-v2/lib/python3.12/site-packages/websockets/legacy/protocol.py", line 1301, in close_connection
    await self.transfer_data_task
  File "/home/yurchenko/kfp-operators/.tox/bundle-integration-v2/lib/python3.12/site-packages/websockets/legacy/protocol.py", line 963, in transfer_data
    message = await self.read_message()
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yurchenko/kfp-operators/.tox/bundle-integration-v2/lib/python3.12/site-packages/websockets/legacy/protocol.py", line 1033, in read_message
    frame = await self.read_data_frame(max_size=self.max_size)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yurchenko/kfp-operators/.tox/bundle-integration-v2/lib/python3.12/site-packages/websockets/legacy/protocol.py", line 1108, in read_data_frame
    frame = await self.read_frame(max_size)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yurchenko/kfp-operators/.tox/bundle-integration-v2/lib/python3.12/site-packages/websockets/legacy/protocol.py", line 1165, in read_frame
    frame = await Frame.read(
            ^^^^^^^^^^^^^^^^^
  File "/home/yurchenko/kfp-operators/.tox/bundle-integration-v2/lib/python3.12/site-packages/websockets/legacy/framing.py", line 68, in read
    data = await reader(2)
           ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/streams.py", line 752, in readexactly
    await self._wait_for_data('readexactly')
  File "/usr/lib/python3.12/asyncio/streams.py", line 545, in _wait_for_data
    await self._waiter
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/yurchenko/kfp-operators/.tox/bundle-integration-v2/lib/python3.12/site-packages/juju/client/connection.py", line 649, in rpc
    await self._ws.send(outgoing)
  File "/home/yurchenko/kfp-operators/.tox/bundle-integration-v2/lib/python3.12/site-packages/websockets/legacy/protocol.py", line 635, in send
    await self.ensure_open()
  File "/home/yurchenko/kfp-operators/.tox/bundle-integration-v2/lib/python3.12/site-packages/websockets/legacy/protocol.py", line 939, in ensure_open
    raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: sent 1011 (internal error) keepalive ping timeout; no close frame received

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/yurchenko/kfp-operators/.tox/bundle-integration-v2/lib/python3.12/site-packages/juju/model.py", line 1206, in _all_watcher
    results = await utils.run_with_interrupt(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yurchenko/kfp-operators/.tox/bundle-integration-v2/lib/python3.12/site-packages/juju/utils.py", line 191, in run_with_interrupt
    return task.result()  # may raise exception
           ^^^^^^^^^^^^^
TypeError: Passing coroutines is forbidden, use tasks explicitly.
bundle-integration-v2: 443121 I exit 0 (442.73 seconds) /home/yurchenko/kfp-operators> pytest -vv --tb=native -s --model kubeflow --bundle=./tests/integration/bundles/kfp_latest_edge.yaml.j2 --keep-models /home/yurchenko/kfp-operators/tests/integration/test_kfp_functional_v2.py pid=2001570 [tox/execute/api.py:286]
  bundle-integration-v2: OK (442.79=setup[0.07]+cmd[442.73] seconds)
  congratulations :) (442.88 seconds)

@BON4 BON4 requested a review from a team as a code owner November 30, 2024 06:20
Copy link
Contributor

@DnPlas DnPlas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @BON4 ! Just a tiny change in the version, other than that LGTM!

I have tested this rock by:

  1. Building the rock -> Success
  2. Running the image in a container and looking for the right files -> Success
$ docker run --rm -ti --entrypoint /bin/bash driver:0.1
_daemon_@f49948563229:/$ ls /bin/driver
/bin/driver
_daemon_@f49948563229:/$ ls /third_party/licenses.csv
/third_party/licenses.csv
_daemon_@f49948563229:/$ ls /third_party/NOTICES
cloud.google.com  github.com  go.opencensus.io	gocloud.dev  golang.org  google.golang.org  gopkg.in  k8s.io  sigs.k8s.io
  1. Executing the binary -> OK (the binary won't do anything w/o the proper args)
_daemon_@f49948563229:/$ /bin/driver
F1202 23:14:39.798672       9 main.go:79] KFP driver: argument --type must be specified
_daemon_@f49948563229:/$ /bin/driver --type CONTAINER
I1202 23:16:02.017136      20 main.go:108] input ComponentSpec:{}
I1202 23:16:02.017572      20 main.go:121] input ContainerSpec:{}
I1202 23:16:02.017661      20 main.go:136] input kubernetesConfig:{}
F1202 23:16:02.017926      20 main.go:79] KFP driver: failed to get namespace in Pod: open /var/run/secrets/kubernetes.io/serviceaccount/namespace: no such file or directory

driver/rockcraft.yaml Outdated Show resolved Hide resolved
@BON4 BON4 requested a review from DnPlas December 3, 2024 08:33
@DnPlas DnPlas merged commit 784baed into canonical:main Dec 3, 2024
7 checks passed
@DnPlas DnPlas mentioned this pull request Dec 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants