Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]: integration install fails when using a uv-provisioned venv #2929

Closed
1 task done
mennoliefstingh opened this issue Aug 13, 2024 · 2 comments
Closed
1 task done

[BUG]: integration install fails when using a uv-provisioned venv #2929

mennoliefstingh opened this issue Aug 13, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@mennoliefstingh
Copy link
Contributor

mennoliefstingh commented Aug 13, 2024 •
edited
Loading

Contact Details [Optional]

[removed] because I get too much spam already. You know where to find me on slack ;)

System Information

ZENML_LOCAL_VERSION: 0.64.0
ZENML_SERVER_VERSION: 0.64.0
ZENML_SERVER_DATABASE: mysql
ZENML_SERVER_DEPLOYMENT_TYPE: cloud
ZENML_CONFIG_DIR: /Users/menno/Library/Application Support/zenml
ZENML_LOCAL_STORE_DIR: /Users/menno/Library/Application Support/zenml/local_stores
ZENML_SERVER_URL: https://564de462-zenml.cloudinfra.zenml.io
ZENML_ACTIVE_REPOSITORY_ROOT: /Users/menno/alliander/aix-use-case-facilities/zenml_starter
PYTHON_VERSION: 3.10.13
ENVIRONMENT: native
SYSTEM_INFO: {'os': 'mac', 'mac_version': '14.5'}
ACTIVE_WORKSPACE: default
ACTIVE_STACK: default
ACTIVE_USER: menno.liefstingh@alliander.com
TELEMETRY_STATUS: enabled
ANALYTICS_CLIENT_ID: 0c575a10-bad7-485a-95eb-4df4655ed703
ANALYTICS_USER_ID: fb49ba82-f5df-4ffd-8394-c1e5f1b1d030
ANALYTICS_SERVER_ID: 8d822bf3-e508-4376-81d5-b9de45f25eba
INTEGRATIONS: ['airflow', 'bitbucket', 'kaniko', 'pigeon', 'scipy', 'sklearn', 'xgboost']
PACKAGES: {'babel': '2.14.0', 'gitpython': '3.1.43', 'mako': '1.3.5', 'markupsafe': '2.1.5', 'pyjwt': '2.7.0', 'pymysql': '1.1.1',
'pyyaml': '6.0.1', 'qtpy': '2.4.1', 'sqlalchemy': '2.0.32', 'sqlalchemy-utils': '0.41.2', 'send2trash': '1.8.3', 'absl-py': '1.4.0',
'aiobotocore': '2.12.2', 'aiohttp': '3.9.3', 'aioitertools': '0.11.0', 'aiosignal': '1.3.1', 'alembic': '1.8.1', 'annotated-types':
'0.6.0', 'anyio': '3.7.1', 'appnope': '0.1.4', 'argon2-cffi': '23.1.0', 'argon2-cffi-bindings': '21.2.0', 'arrow': '1.3.0', 'asn1crypto':
'1.5.1', 'asttokens': '2.4.1', 'async-lru': '2.0.4', 'async-timeout': '4.0.3', 'atpublic': '4.1.0', 'attrs': '23.2.0', 'bcrypt': '4.0.1',
'beautifulsoup4': '4.12.3', 'bleach': '6.1.0', 'botocore': '1.34.51', 'cachetools': '5.3.3', 'certifi': '2024.2.2', 'cffi': '1.16.0',
'charset-normalizer': '3.3.2', 'click': '8.1.3', 'click-params': '0.3.0', 'cloudpickle': '2.2.1', 'colorama': '0.4.6', 'comm': '0.2.2',
'copier': '9.3.1', 'cryptography': '42.0.5', 'debugpy': '1.8.1', 'decorator': '5.1.1', 'defusedxml': '0.7.1', 'distro': '1.9.0', 'docker':
'7.1.0', 'dunamai': '1.22.0', 'exceptiongroup': '1.2.0', 'executing': '2.0.1', 'fastapi': '0.110.0', 'fastjsonschema': '2.19.1',
'filelock': '3.13.4', 'fqdn': '1.5.1', 'frozenlist': '1.4.1', 'fsspec': '2023.12.2', 'funcy': '2.0', 'gitdb': '4.0.11', 'h11': '0.14.0',
'httpcore': '1.0.5', 'httplib2': '0.19.1', 'httptools': '0.6.1', 'httpx': '0.27.0', 'idna': '3.7', 'importlib-resources': '6.4.0',
'ipinfo': '5.0.1', 'ipykernel': '6.29.4', 'ipython': '8.18.0', 'ipywidgets': '8.1.2', 'isoduration': '20.11.0', 'jedi': '0.19.1',
'jinja2': '3.1.4', 'jinja2-ansible-filters': '1.3.2', 'jinja2-time': '0.2.0', 'jmespath': '1.0.1', 'joblib': '1.4.0', 'json5': '0.9.24',
'jsonpointer': '2.4', 'jsonschema': '4.21.1', 'jsonschema-specifications': '2023.12.1', 'jupyter': '1.0.0', 'jupyter-client': '8.6.1',
'jupyter-console': '6.6.3', 'jupyter-core': '5.7.2', 'jupyter-events': '0.10.0', 'jupyter-lsp': '2.2.5', 'jupyter-server': '2.13.0',
'jupyter-server-terminals': '0.5.3', 'jupyterlab': '4.1.6', 'jupyterlab-pygments': '0.3.0', 'jupyterlab-server': '2.26.0',
'jupyterlab-widgets': '3.0.10', 'markdown-it-py': '3.0.0', 'matplotlib-inline': '0.1.6', 'mdurl': '0.1.2', 'mistune': '3.0.2',
'multidict': '6.0.5', 'nbclient': '0.10.0', 'nbconvert': '7.16.3', 'nbformat': '5.10.4', 'nest-asyncio': '1.6.0', 'notebook': '7.1.2',
'notebook-shim': '0.2.4', 'numpy': '1.26.4', 'orjson': '3.10.7', 'overrides': '7.7.0', 'packaging': '24.1', 'pandas': '1.5.3',
'pandocfilters': '1.5.1', 'parso': '0.8.4', 'passlib': '1.7.4', 'pathspec': '0.12.1', 'pexpect': '4.9.0', 'pip': '24.2', 'platformdirs':
'4.2.0', 'plumbum': '1.8.3', 'prometheus-client': '0.20.0', 'prompt-toolkit': '3.0.36', 'psutil': '5.9.8', 'ptyprocess': '0.7.0',
'pure-eval': '0.2.2', 'pyopenssl': '24.1.0', 'pyarrow': '15.0.2', 'pycparser': '2.22', 'pydantic': '2.7.4', 'pydantic-core': '2.18.4',
'pydantic-settings': '2.2.1', 'pygments': '2.17.2', 'pyparsing': '2.4.7', 'python-dateutil': '2.9.0.post0', 'python-dotenv': '1.0.1',
'python-json-logger': '2.0.7', 'python-multipart': '0.0.9', 'pytimeparse': '1.1.8', 'pytz': '2024.1', 'pyyaml-include': '1.4.1', 'pyzmq':
'25.1.2', 'qtconsole': '5.5.1', 'questionary': '2.0.1', 'referencing': '0.34.0', 'requests': '2.31.0', 'retrying': '1.3.4',
'rfc3339-validator': '0.1.4', 'rfc3986-validator': '0.1.1', 'rich': '13.7.1', 'rpds-py': '0.18.0', 'ruff': '0.5.7', 's3fs': '2023.12.2',
'scikit-learn': '1.3.2', 'scipy': '1.13.0', 'secure': '0.3.0', 'setuptools': '69.2.0', 'six': '1.16.0', 'smmap': '5.0.1', 'sniffio':
'1.3.1', 'snowflake': '0.7.0', 'snowflake-connector-python': '3.8.1', 'snowflake-core': '0.7.0', 'snowflake-legacy': '0.7.0',
'snowflake-ml-python': '1.4.0', 'snowflake-snowpark-python': '1.14.0', 'sortedcontainers': '2.4.0', 'soupsieve': '2.5', 'sqlmodel':
'0.0.18', 'sqlparse': '0.4.4', 'stack-data': '0.6.3', 'starlette': '0.36.3', 'terminado': '0.18.1', 'threadpoolctl': '3.4.0', 'tinycss2':
'1.2.1', 'tomli': '2.0.1', 'tomlkit': '0.12.4', 'tornado': '6.4', 'traitlets': '5.14.2', 'types-python-dateutil': '2.9.0.20240316',
'typing-extensions': '4.11.0', 'tzdata': '2024.1', 'uri-template': '1.3.0', 'urllib3': '2.0.7', 'uvicorn': '0.30.6', 'uvloop': '0.19.0',
'validators': '0.18.2', 'watchfiles': '0.23.0', 'wcwidth': '0.2.13', 'webcolors': '1.13', 'webencodings': '0.5.1', 'websocket-client':
'1.7.0', 'websockets': '12.0', 'wheel': '0.43.0', 'widgetsnbextension': '4.0.10', 'wrapt': '1.16.0', 'xgboost': '1.7.6', 'yarl': '1.9.4',
'zenml': '0.64.0'}

CURRENT STACK

Name: default
ID: d12077da-c36e-4c8f-8dee-9cbed28174de
Workspace: default / 4ef3a615-ed4b-4dbb-a2c9-dbbcbd58aefa

ORCHESTRATOR: default

Name: default
ID: 1acad1e9-82e0-41ac-9e46-cdb0451e5db9
Type: orchestrator
Flavor: local
Configuration: {}
Workspace: default / 4ef3a615-ed4b-4dbb-a2c9-dbbcbd58aefa

ARTIFACT_STORE: default

Name: default
ID: f9766ddc-03ff-45b2-8149-d4d8e8e37d40
Type: artifact_store
Flavor: local
Configuration: {'path': ''}
Workspace: default / 4ef3a615-ed4b-4dbb-a2c9-dbbcbd58aefa

What happened?

  • I created + activated a venv using uv:
#!/bin/bash

if [ ! -d ".venv" ]; then
    echo "Creating new virtual environment and activating it..."
    uv venv
else
    echo "Activating existing virtual environment..."
fi
  • Tried to run zenml integration install sklearn -y
  • Got the following error:
zenml integration install sklearn -y
/Users/menno/alliander/aix-use-case-facilities/.venv/bin/python3: No module named pip
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/menno/alliander/aix-use-case-facilities/.venv/bin/zenml:8 in <module>                     │
│                                                                                                  │
│   5 from zenml.cli.cli import cli                                                                │
│   6 if __name__ == "__main__":                                                                   │
│   7 │   sys.argv[0] = re.sub(r"(-script\.pyw|\.exe)?$", "", sys.argv[0])                         │
│ ❱ 8 │   sys.exit(cli())                                                                          │
│   9                                                                                              │
│                                                                                                  │
│ /Users/menno/alliander/aix-use-case-facilities/.venv/lib/python3.10/site-packages/click/core.py: │
│ 1130 in __call__                                                                                 │
│                                                                                                  │
│   1127 │                                                                                         │
│   1128 │   def __call__(self, *args: t.Any, **kwargs: t.Any) -> t.Any:                           │
│   1129 │   │   """Alias for :meth:`main`."""                                                     │
│ ❱ 1130 │   │   return self.main(*args, **kwargs)                                                 │
│   1131                                                                                           │
│   1132                                                                                           │
│   1133 class Command(BaseCommand):                                                               │
│                                                                                                  │
│ /Users/menno/alliander/aix-use-case-facilities/.venv/lib/python3.10/site-packages/click/core.py: │
│ 1055 in main                                                                                     │
│                                                                                                  │
│   1052 │   │   try:                                                                              │
│   1053 │   │   │   try:                                                                          │
│   1054 │   │   │   │   with self.make_context(prog_name, args, **extra) as ctx:                  │
│ ❱ 1055 │   │   │   │   │   rv = self.invoke(ctx)                                                 │
│   1056 │   │   │   │   │   if not standalone_mode:                                               │
│   1057 │   │   │   │   │   │   return rv                                                         │
│   1058 │   │   │   │   │   # it's not safe to `ctx.exit(rv)` here!                               │
│                                                                                                  │
│ /Users/menno/alliander/aix-use-case-facilities/.venv/lib/python3.10/site-packages/click/core.py: │
│ 1657 in invoke                                                                                   │
│                                                                                                  │
│   1654 │   │   │   │   super().invoke(ctx)                                                       │
│   1655 │   │   │   │   sub_ctx = cmd.make_context(cmd_name, args, parent=ctx)                    │
│   1656 │   │   │   │   with sub_ctx:                                                             │
│ ❱ 1657 │   │   │   │   │   return _process_result(sub_ctx.command.invoke(sub_ctx))               │
│   1658 │   │                                                                                     │
│   1659 │   │   # In chain mode we create the contexts step by step, but after the                │
│   1660 │   │   # base command has been invoked.  Because at that point we do not                 │
│                                                                                                  │
│ /Users/menno/alliander/aix-use-case-facilities/.venv/lib/python3.10/site-packages/click/core.py: │
│ 1657 in invoke                                                                                   │
│                                                                                                  │
│   1654 │   │   │   │   super().invoke(ctx)                                                       │
│   1655 │   │   │   │   sub_ctx = cmd.make_context(cmd_name, args, parent=ctx)                    │
│   1656 │   │   │   │   with sub_ctx:                                                             │
│ ❱ 1657 │   │   │   │   │   return _process_result(sub_ctx.command.invoke(sub_ctx))               │
│   1658 │   │                                                                                     │
│   1659 │   │   # In chain mode we create the contexts step by step, but after the                │
│   1660 │   │   # base command has been invoked.  Because at that point we do not                 │
│                                                                                                  │
│ /Users/menno/alliander/aix-use-case-facilities/.venv/lib/python3.10/site-packages/click/core.py: │
│ 1404 in invoke                                                                                   │
│                                                                                                  │
│   1401 │   │   │   echo(style(message, fg="red"), err=True)                                      │
│   1402 │   │                                                                                     │
│   1403 │   │   if self.callback is not None:                                                     │
│ ❱ 1404 │   │   │   return ctx.invoke(self.callback, **ctx.params)                                │
│   1405 │                                                                                         │
│   1406 │   def shell_complete(self, ctx: Context, incomplete: str) -> t.List["CompletionItem"]:  │
│   1407 │   │   """Return a list of completions for the incomplete value. Looks                   │
│                                                                                                  │
│ /Users/menno/alliander/aix-use-case-facilities/.venv/lib/python3.10/site-packages/click/core.py: │
│ 760 in invoke                                                                                    │
│                                                                                                  │
│    757 │   │                                                                                     │
│    758 │   │   with augment_usage_errors(__self):                                                │
│    759 │   │   │   with ctx:                                                                     │
│ ❱  760 │   │   │   │   return __callback(*args, **kwargs)                                        │
│    761 │                                                                                         │
│    762 │   def forward(                                                                          │
│    763 │   │   __self, __cmd: "Command", *args: t.Any, **kwargs: t.Any  # noqa: B902             │
│                                                                                                  │
│ /Users/menno/alliander/aix-use-case-facilities/.venv/lib/python3.10/site-packages/zenml/cli/inte │
│ gration.py:313 in install                                                                        │
│                                                                                                  │
│   310 │   │   )                                                                                  │
│   311 │   ):                                                                                     │
│   312 │   │   with console.status("Installing integrations..."):                                 │
│ ❱ 313 │   │   │   install_packages(requirements, use_uv=uv)                                      │
│   314                                                                                            │
│   315                                                                                            │
│   316 @integration.command(                                                                      │
│                                                                                                  │
│ /Users/menno/alliander/aix-use-case-facilities/.venv/lib/python3.10/site-packages/zenml/cli/util │
│ s.py:1117 in install_packages                                                                    │
│                                                                                                  │
│   1114 │   │   │   │   "Run `uv venv` to create a virtualenv and retry."                         │
│   1115 │   │   │   )                                                                             │
│   1116 │   │   else:                                                                             │
│ ❱ 1117 │   │   │   raise e                                                                       │
│   1118                                                                                           │
│   1119                                                                                           │
│   1120 def uninstall_package(package: str, use_uv: bool = False) -> None:                        │
│                                                                                                  │
│ /Users/menno/alliander/aix-use-case-facilities/.venv/lib/python3.10/site-packages/zenml/cli/util │
│ s.py:1105 in install_packages                                                                    │
│                                                                                                  │
│   1102 │   │   │   command.append("--no-warn-conflicts")                                         │
│   1103 │                                                                                         │
│   1104 │   try:                                                                                  │
│ ❱ 1105 │   │   subprocess.check_call(command)                                                    │
│   1106 │   except subprocess.CalledProcessError as e:                                            │
│   1107 │   │   if (                                                                              │
│   1108 │   │   │   use_uv                                                                        │
│                                                                                                  │
│ /opt/homebrew/Cellar/python@3.10/3.10.13_2/Frameworks/Python.framework/Versions/3.10/lib/python3 │
│ .10/subprocess.py:369 in check_call                                                              │
│                                                                                                  │
│    366 │   │   cmd = kwargs.get("args")                                                          │
│    367 │   │   if cmd is None:                                                                   │
│    368 │   │   │   cmd = popenargs[0]                                                            │
│ ❱  369 │   │   raise CalledProcessError(retcode, cmd)                                            │
│    370 │   return 0                                                                              │
│    371                                                                                           │
│    372                                                                                           │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
CalledProcessError: Command '['/Users/menno/alliander/aix-use-case-facilities/.venv/bin/python3', '-m', 'pip', 'install', 'scikit-learn',
'-qqq', '--no-warn-conflicts']' returned non-zero exit status 1.

This happens because pip isn't installed by default in venvs created using uv.

I mitigated this by installing pip in my venv (uv pip install pip).
To be fair I usually blame myself for using uv when I get these kinds of bugs, but as zenml supports uv (LOVE IT!) in so many places I think this warrants a fix of at least a warning of some sort.

Reproduction steps

See above (sorry)

Relevant log output

also see above

Code of Conduct

  • I agree to follow this project's Code of Conduct
@mennoliefstingh mennoliefstingh added the bug Something isn't working label Aug 13, 2024
@mennoliefstingh
Copy link
Contributor Author

mennoliefstingh commented Aug 13, 2024 •
edited
Loading

Reproduced the error in a 'clean' environment:

uv venv && source .venv/bin/activate
Using Python 3.11.4 interpreter at: /Library/Frameworks/Python.framework/Versions/3.11/bin/python3
Creating virtualenv at: .venv
Activate with: source .venv/bin/activate
python3
Python 3.11.4 (v3.11.4:d2340ef257, Jun  6 2023, 19:15:51) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.run(["python3", "-m", "pip", "install", "requests"])
/Users/menno/dev/python/test/.venv/bin/python3: No module named pip
CompletedProcess(args=['python3', '-m', 'pip', 'install', 'requests'], returncode=1)
>>> exit()
uv pip install pip
Resolved 1 package in 154ms
Installed 1 package in 17ms
 + pip==24.2
python3
Python 3.11.4 (v3.11.4:d2340ef257, Jun  6 2023, 19:15:51) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.run(["python3", "-m", "pip", "install", "requests"])
Looking in indexes: https://pypi.org/simple, https://al*****:****@alliander.jfrog.io/artifactory/api/pypi/pypi-all/simple
Collecting requests, etc etc it worked after this

mennoliefstingh added a commit to mennoliefstingh/zenml that referenced this issue Aug 13, 2024
@mennoliefstingh
Check if pip is installed when not using uv flag (zenml-io#2929)
5b87b9c
@mennoliefstingh mennoliefstingh mentioned this issue Aug 13, 2024
Merged
9 tasks
bcdurak added a commit that referenced this issue Aug 21, 2024
@mennoliefstingh @strickvl @bcdurak
Throw an error when running integration installs when uv == False but…
a496fff 
… pip is not installed (#2930)

* Check if pip is installed when not using uv flag (#2929)

* fix: Include --uv flag hint more explicitly in error message

Co-authored-by: Alex Strick van Linschoten <strickvl@users.noreply.github.com>

---------

Co-authored-by: Alex Strick van Linschoten <strickvl@users.noreply.github.com>
Co-authored-by: Barış Can Durak <36421093+bcdurak@users.noreply.github.com>
@htahir1
Copy link
Contributor

htahir1 commented Aug 22, 2024

Fixed with #2930

@htahir1 htahir1 closed this as completed Aug 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@htahir1 @mennoliefstingh