feat(Backend + SDK): Update kfp backend and kubernetes sdk to allow enabling shared memory

[hsteude]

This PR adds a method called 'enable_shared_memory' to the kubernetes_platform python sdk.

Why?
Without this, we can't train PyTorch models using dataloaders with multiple workers (see #9893).

Next Steps:
This PR has to be rebased once #10703 is merged. We'll also need to update the go.mod, go.sum, and the license CSV files.

Checklist:

• The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

[google-oss-prow]

Hi @hsteude. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[rimolive]

@hsteude Can you please add this PR as a topic to the next Pipelines WG meeting?

[hsteude]

@chensun: Thanks for merging #10703. Could you also take a look at this one, please?

[qadeem-qureshi]

Any update on this? Would be really nice to have.

[rimolive]

cc @chensun @zijianjoy @james-jwu

[hbelmiro]

/ok-to-test

[google-oss-prow]

@hsteude: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
kubeflow-pipeline-upgrade-test	0549ca3	link	false	/test kubeflow-pipeline-upgrade-test
kubeflow-pipelines-samples-v2	0549ca3	link	false	/test kubeflow-pipelines-samples-v2
kfp-kubernetes-execution-tests	0549ca3	link	false	/test kfp-kubernetes-execution-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[hsteude]

@rimolive @chensun @zijianjoy @james-jwu Would really appreciate a review on this.

[sanchesoon]

Hi all, any plans to merge it ?

[hbelmiro]

@hsteude can you please rebase so you get the latest changes on tests?

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign chensun for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• backend/OWNERS
• kubernetes_platform/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[hbelmiro]

/lgtm

[hbelmiro]

@HumairAK do you have this on your plate?

[hbelmiro]

@HumairAK @chensun
Can we merge this if it makes sense to you? Otherwise, could you please share your feedback so the author can work on it on time for KFP 2.4?