Authentication with Dex Static Credentials only works with kfp 1.0

[xwt-ml]

Checks

• I have searched the existing issues.
• This issue is NOT security related. (Please disclose responsibly)
• This issue is NOT specific to the CLI. (Please open a CLI issue)

deployKF Version

0.1.5

Kubernetes Distribution

EKS

Kubernetes Version

Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T17:44:59Z", GoVersion:"go1.19", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7

Description

Using the example code provided here https://www.deploykf.org/user-guides/access-kubeflow-pipelines-api/#reference-implementation_1, I can successfully authenticate with the dex static credential and get the list experiments with kfp 1.0. When upgrading to kfp 2.0, I receive the following error message:
kfp_server_api.exceptions.ApiException: (404)
Reason: Not Found

Additionally, there is currently no example showing how to attach shared memory to container operations with kfp 2.0, preventing users from upgrading the platform from v0.1.4 to v0.1.5, since the deploykf v0.1.5 only works together with kfp 2.0 as stated here https://www.deploykf.org/user-guides/access-kubeflow-pipelines-api/#overview.

Relevant Logs

No response

deployKF Values (Optional)

No response

[thesuperzapper]

@xwt-ml regarding your 404 error, can you confirm a few things:

1. What version of the kfp SDK are you using?

   import kfp
   print(kfp.__version__)

2. What commands are you trying to run after the initalized client which is resulting in kfp_server_api.exceptions.ApiException: (404)?

3. Are you trying to list objects which don't exist (hence the 404)?

---

Regarding your comment about "attach shared memory", I am not 100% sure what you are referring to here, but this sounds like a question for upstream Kubeflow Pipelines.

Also, you can continue to use Kubeflow Pipelines V1 with deployKF 0.1.5, just downgrade your kfp SDK to 1.8.22 and continue using it as before, it will still work.

[xwt-ml]

@thesuperzapper I found out this morning that the 404 error is caused by missing environment variables in my code. It has nothing to do with the kfp version. The authentication code works with both kfp 1.0 and kfp 2.0.

Regarding to "shared memory", attaching shared memory to Docker for machine learning training with PyTorch is necessary because PyTorch's DataLoader often uses multiple processes to load data efficiently. Increased shared memory prevents bottlenecks, crashes, and out-of-memory errors during data loading and augmentation. It also ensures better performance and utilization of CPU and GPU resources during training. My training job usually does not work without the shared memory.

kfp v1.0 supports this functionality through the dsl.PipelineVolume module.

volume = dsl.PipelineVolume(volume=k8s.client.V1Volume(name="shm", empty_dir=k8s.client.V1EmptyDirVolumeSource(medium='Memory')))

The shared memory can be attached to the ContainerOp using the following way:

container_op.add_pvolumes({'/dev/shm': volume})

In kfp 2.0, dsl.PipelineVolume module is not available anymore and I did not find any reasonable alternatives.

You are right, this is not related to this particular issue. I mentioned this because I thought deploykf v0.1.5 only works together with kfp 2.0 because of the information in this table

Therefore, I will close this issue. Thank you very much for the support.

[thesuperzapper]

@xwt-ml Thanks, I have updated that warning to be more correct (saying that 0.1.5 can use KFP SDK 1.8.22).

Regarding your question about emptyDir mounts in KFP v2, you are correct that this seems to be missing right now kubeflow/pipelines#10656. However, there is an upstream PR to add this functionality which I will chase up.