[sdk] KFP V2 pipeline fails with 'no module named ensurepip'

[ConverJens]

Environment

• How did you deploy Kubeflow Pipelines (KFP)?
  With KF 1.3 but KFP bumped to 1.7
  KFP version: 1.7
• KFP SDK version: 1.7.1

Steps to reproduce

Build simple kFP V2 one-component hello-world pipeline.

Expected result

Pipeline should run

Materials and Reference

Output logs:

I0915 11:43:08.161494 1 env.go:30] cannot find launcher configmap: name="kfp-launcher" namespace="observability"
I0915 11:43:08.161671 1 launcher.go:145] PipelineRoot defaults to "minio://mlpipeline/v2/artifacts".
I0915 11:43:08.161832 1 cache.go:143] Cannot detect ml-pipeline in the same namespace, default to ml-pipeline.kubeflow:8887 as KFP endpoint.
I0915 11:43:08.161839 1 cache.go:120] Connecting to cache endpoint ml-pipeline.kubeflow:8887
I0915 11:43:08.285842 1 launcher.go:194] enable caching
I0915 11:43:08.353559 1 object_store.go:285] Cannot detect minio-service in the same namespace, default to minio-service.kubeflow:9000 as MinIO endpoint.
/root/pyenv/bin/python3: No module named ensurepip
/root/pyenv/bin/python3: No module named ensurepip

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[zijianjoy]

cc @chensun

[jiasenwu]

👍 I meet the same issue. Running with the default image is ok, but the error pops up when I switch to my own image where there is no ensurepip (but pip is pre-installed).

using system python 3.6 on ubuntu 18.04. Then I installed python3.6-venv package and call python -m ensurepip but it complains

ensurepip is disabled in Debian/Ubuntu for the system python.

Python modules for the system python are usually handled by dpkg and apt-get.

    apt-get install python-<module name>

Install the python-pip package to use pip itself.  Using pip together
with the system python might have unexpected results for any system installed
module, so use it on your own risk, or make sure to only use it in virtual
environments.

[jiasenwu]

Another related issue is that in a different image I need to use a conda environment. I think a general solution can be a fully customizable command to install the packages.

[ConverJens]

Or the only reasonable solution is that the user should be able to choose which ever image they like.

[ConverJens]

@zijianjoy @chensun What has changed in this regard since KFP SDK V1 pipelines? The same image that works perfectly for V1 fails for V2.

[chensun]

@zijianjoy @chensun What has changed in this regard since KFP SDK V1 pipelines? The same image that works perfectly for V1 fails for V2.

@ConverJens, the change is we install kfp package into the component for the executor and build-in artifact types support. The alternative would be inline the code into the container command line as mentioned in #6116 (comment)

We will look into making kfp installation an option that you could opt-out. Before that is ready, can you use an alternative image that has pip installed?

[chensun]

👍 I meet the same issue. Running with the default image is ok, but the error pops up when I switch to my own image where there is no ensurepip (but pip is pre-installed).

using system python 3.6 on ubuntu 18.04. Then I installed python3.6-venv package and call python -m ensurepip but it complains

ensurepip is disabled in Debian/Ubuntu for the system python.

Python modules for the system python are usually handled by dpkg and apt-get.

    apt-get install python-<module name>

Install the python-pip package to use pip itself.  Using pip together
with the system python might have unexpected results for any system installed
module, so use it on your own risk, or make sure to only use it in virtual
environments.

@jiasenwu Did this fail your component execution? or was it only warnings?

[ConverJens]

@chensun Thanks for the response!

I have pip installed and I have already preinstalled both kfp and other pypi packages. So this is happening with an image that already has pip installed but I'm not using a virtual environment.

[jiasenwu]

+1 I meet the same issue. Running with the default image is ok, but the error pops up when I switch to my own image where there is no ensurepip (but pip is pre-installed).
using system python 3.6 on ubuntu 18.04. Then I installed python3.6-venv package and call python -m ensurepip but it complains

ensurepip is disabled in Debian/Ubuntu for the system python.

Python modules for the system python are usually handled by dpkg and apt-get.

    apt-get install python-<module name>

Install the python-pip package to use pip itself.  Using pip together
with the system python might have unexpected results for any system installed
module, so use it on your own risk, or make sure to only use it in virtual
environments.

@jiasenwu Did this fail your component execution? or was it only warnings?

This is an error. Ubuntu doesn't allow me to call ensurepip. I end up doing a monkey patch to this function kfp.v2.components.component_factory._get_packages_to_install_command

def patched_get_packages_to_install_command(package_list=None):
    result = []
    if package_list:
        install_packages_command = (
            "PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet \
                --no-warn-script-location {}"
        ).format(" ".join([repr(str(package)) for package in package_list]))
        result = [
            "sh",
            "-c",
            ' ({install_packages} || {install_packages} --user) && "$0" "$@"'.format(
                install_packages=install_packages_command
            ),
        ]
    return result

[ConverJens]

@chensun @jiasenwu
I feel that there must be something fundamentally broken about the way package installations is now handled in kfp v2.

Installing additional requirements worked great in kfp v1 using the same base image for the pipeline so it seems as some other change in how packages are now being installed is the culprit.

[chensun]

Closing the issue as this has been patricianly mitigated by the above linked PR.