Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vendored poetry monkeypatches subprocess.run and Popen #400

Open
2 tasks done
AlbertDeFusco opened this issue Apr 21, 2023 · 14 comments
Open
2 tasks done

Vendored poetry monkeypatches subprocess.run and Popen #400

AlbertDeFusco opened this issue Apr 21, 2023 · 14 comments

Comments

@AlbertDeFusco
Copy link
Contributor

Checklist

  • I added a descriptive title
  • I searched open reports and couldn't find a duplicate

What happened?

I know that this is a strange bug report and that the problem is not conda-lock's fault. I wonder if a new version of poetry will help, even slightly. This compatibility code was removed for version 1.2.

I've noticed that when conda_lock is imported then subprocess.run no longer supports capture_ouptut=True. The error messages trace to the vendored version of poetry included with conda-lock.

In [1]: import subprocess

In [2]: subprocess.run(['/usr/bin/env', 'echo', '$SHELL'], capture_output=True).stdout
Out[2]: b'$SHELL\n'

In [3]: import conda_lock

In [4]: subprocess.run(['/usr/bin/env', 'echo', '$SHELL'], capture_output=True).stdout
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[4], line 1
----> 1 subprocess.run(['/usr/bin/env', 'echo', '$SHELL'], capture_output=True).stdout

File ~/cl/lib/python3.10/site-packages/conda_lock/_vendor/poetry/utils/_compat.py:200, in run(*popenargs, **kwargs)
    197         raise ValueError("stdin and input arguments may not both be used.")
    198     kwargs["stdin"] = PIPE
--> 200 process = Popen(*popenargs, **kwargs)
    201 try:
    202     process.__enter__()  # No-Op really... illustrate "with in 2.4"

TypeError: Popen.__init__() got an unexpected keyword argument 'capture_output'

Conda Info

active environment : /Users/adefusco/cl
    active env location : /Users/adefusco/cl
            shell level : 2
       user config file : /Users/adefusco/.condarc
 populated config files : /Users/adefusco/.condarc
          conda version : 23.3.1
    conda-build version : 3.23.3
         python version : 3.9.13.final.0
       virtual packages : __archspec=1=arm64
                          __osx=13.2.1=0
                          __unix=0=0
       base environment : /Users/adefusco/Applications/miniconda3  (writable)
      conda av data dir : /Users/adefusco/Applications/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/osx-arm64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-arm64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /Users/adefusco/Applications/miniconda3/pkgs
                          /Users/adefusco/.conda/pkgs
       envs directories : /Users/adefusco/Applications/miniconda3/envs
                          /Users/adefusco/.conda/envs
               platform : osx-arm64
             user-agent : conda/23.3.1 requests/2.28.1 CPython/3.9.13 Darwin/22.3.0 OSX/13.2.1
                UID:GID : 502:20
             netrc file : None
           offline mode : False

Conda Config

==> /Users/adefusco/.condarc <==
extra_safety_checks: False
ssl_verify: True

Conda list

# packages in environment at /Users/adefusco/cl:
#
# Name                    Version                   Build  Channel
appdirs                   1.4.4              pyhd3eb1b0_0    defaults
appnope                   0.1.2           py310hca03da5_1001    defaults
asttokens                 2.0.5              pyhd3eb1b0_0    defaults
backcall                  0.2.0              pyhd3eb1b0_0    defaults
brotlipy                  0.7.0           py310h1a28f6b_1002    defaults
bzip2                     1.0.8                h620ffc9_4    defaults
ca-certificates           2023.01.10           hca03da5_0    defaults
cachecontrol              0.12.11         py310hca03da5_1    defaults
cachecontrol-with-filecache 0.12.11         py310hca03da5_1    defaults
cachy                     0.3.0              pyhd3eb1b0_0    defaults
certifi                   2022.12.7       py310hca03da5_0    defaults
cffi                      1.15.1          py310h80987f9_3    defaults
charset-normalizer        2.0.4              pyhd3eb1b0_0    defaults
click                     8.0.4           py310hca03da5_0    defaults
click-default-group       1.2.2           py310hca03da5_0    defaults
clikit                    0.6.2                      py_0    defaults
conda-lock                1.4.0           py310hca03da5_0    defaults
crashtest                 0.3.1              pyhd3eb1b0_1    defaults
cryptography              39.0.1          py310h834c97f_0    defaults
decorator                 5.1.1              pyhd3eb1b0_0    defaults
distlib                   0.3.6           py310hca03da5_0    defaults
ensureconda               1.4.3           py310hca03da5_0    defaults
executing                 0.8.3              pyhd3eb1b0_0    defaults
filelock                  3.9.0           py310hca03da5_0    defaults
html5lib                  1.1                pyhd3eb1b0_0    defaults
idna                      3.4             py310hca03da5_0    defaults
importlib-metadata        6.0.0           py310hca03da5_0    defaults
importlib_metadata        6.0.0                hd3eb1b0_0    defaults
ipython                   8.12.0          py310hca03da5_0    defaults
jaraco.classes            3.2.1              pyhd3eb1b0_0    defaults
jedi                      0.18.1          py310hca03da5_1    defaults
jinja2                    3.1.2           py310hca03da5_0    defaults
keyring                   23.13.1         py310hca03da5_0    defaults
libcxx                    14.0.6               h848a8c0_0    defaults
libffi                    3.4.2                hca03da5_6    defaults
lockfile                  0.12.2          py310hca03da5_0    defaults
markupsafe                2.1.1           py310h1a28f6b_0    defaults
matplotlib-inline         0.1.6           py310hca03da5_0    defaults
more-itertools            8.12.0             pyhd3eb1b0_0    defaults
msgpack-python            1.0.3           py310h525c30c_0    defaults
ncurses                   6.4                  h313beb8_0    defaults
openssl                   1.1.1t               h1a28f6b_0    defaults
packaging                 23.0            py310hca03da5_0    defaults
parso                     0.8.3              pyhd3eb1b0_0    defaults
pastel                    0.2.1                      py_0    defaults
pexpect                   4.8.0              pyhd3eb1b0_3    defaults
pickleshare               0.7.5           pyhd3eb1b0_1003    defaults
pip                       23.0.1          py310hca03da5_0    defaults
pkginfo                   1.9.6           py310hca03da5_0    defaults
platformdirs              2.5.2           py310hca03da5_0    defaults
prompt-toolkit            3.0.36          py310hca03da5_0    defaults
ptyprocess                0.7.0              pyhd3eb1b0_2    defaults
pure_eval                 0.2.2              pyhd3eb1b0_0    defaults
pycparser                 2.21               pyhd3eb1b0_0    defaults
pydantic                  1.10.2          py310h1a28f6b_0    defaults
pygments                  2.11.2             pyhd3eb1b0_0    defaults
pylev                     1.3.0                      py_0    defaults
pyopenssl                 23.0.0          py310hca03da5_0    defaults
pysocks                   1.7.1           py310hca03da5_0    defaults
python                    3.10.10              hc0d8a6c_2    defaults
pyyaml                    6.0             py310h80987f9_1    defaults
readline                  8.2                  h1a28f6b_0    defaults
requests                  2.28.1          py310hca03da5_1    defaults
ruamel.yaml               0.17.21         py310h1a28f6b_0    defaults
ruamel.yaml.clib          0.2.6           py310h1a28f6b_1    defaults
setuptools                66.0.0          py310hca03da5_0    defaults
six                       1.16.0             pyhd3eb1b0_1    defaults
sqlite                    3.41.2               h80987f9_0    defaults
stack_data                0.2.0              pyhd3eb1b0_0    defaults
tk                        8.6.12               hb8d0fd4_0    defaults
tomli                     2.0.1           py310hca03da5_0    defaults
tomlkit                   0.11.1          py310hca03da5_0    defaults
toolz                     0.12.0          py310hca03da5_0    defaults
traitlets                 5.7.1           py310hca03da5_0    defaults
typing-extensions         4.5.0           py310hca03da5_0    defaults
typing_extensions         4.5.0           py310hca03da5_0    defaults
tzdata                    2023c                h04d1e81_0    defaults
urllib3                   1.26.15         py310hca03da5_0    defaults
virtualenv                20.17.1         py310hca03da5_0    defaults
wcwidth                   0.2.5              pyhd3eb1b0_0    defaults
webencodings              0.5.1           py310hca03da5_1    defaults
wheel                     0.38.4          py310hca03da5_0    defaults
xz                        5.2.10               h80987f9_1    defaults
yaml                      0.2.5                h1a28f6b_0    defaults
zipp                      3.11.0          py310hca03da5_0    defaults
zlib                      1.2.13               h5a0b063_0    defaults

Additional Context

I am a contributor to conda-project where conda-lock is utilized as a library. I noticed this error occurs when an environment containing conda-project also contains setupools-scm. Then when conda-project imports conda-lock something happens that setuptools-scm wants to do subprocess.run(capture_output=True) and this happens after the conda-lock-vendored poetry is imported.

@maresb
Copy link
Contributor

maresb commented Apr 21, 2023

Hi @AlbertDeFusco, thanks so much for the report!

I was confused at first since the monkeypatch you describe is within the if PY34: block. But the definition of PY34 is sys.version_info >= (3, 4), so that block is indeed running.

I agree that revendoring Poetry would help. I hope to get around to this soon.

@riccardoporreca
Copy link
Contributor

riccardoporreca commented Jul 5, 2023

@maresb, @AlbertDeFusco, I am experiencing the same problem, which is biting quite hard

The issue is indeed related to the vendored Poetry, which has dropped most of utils/_compat.py in https://github.com/python-poetry/poetry/pull/3405/files#diff-58970a6af112ab5dce31955e7d73936f8ad104ef7ae7d1ecf677f729e60206f2

Below a reproducible example of the behavior to help clarifying when this bites.

In a nutshell, on Python 3.9 and 3.10 the presence of setuptool-scm and corresponding configuration (even empty) in pyproject.toml causes the issue, triggered by the vendored poetry. And this is completely unrelated to running conda-lock and making explicit usage of pyproject.toml as a source of dependencies.

Any plan to update the vendored poetry? Or any quick workaround/fix to address the problem?

We are planning to transition a large set of projects to conda-lock 2x from conda-lock1.0.5 and this is a major blocker.
If there is anything I can help with, I will be happy to do so.

Python 3.9

mamba create -n conda-lock-test python=3.9 conda-lock setuptools-scm -c conda-forge --override-channels
conda activate conda-lock-test

mkdir conda-lock-test
cd conda-lock-test
git init
# all good so far
conda-lock --version
### conda-lock, version 2.1.0
echo "[tool.setuptools_scm]" > pyproject.toml
### TypeError: __init__() got an unexpected keyword argument 'capture_output'
# the actuall occurs at import time
python -c "import conda_lock"
### TypeError: Popen.__init__() got an unexpected keyword argument 'capture_output'

Note that an empty pyrpoject.toml or a pyproject.toml w/o [tool.setuptools_scm] would not harm

Python 3.10

Same behavior

Python 3.11

No issues

@riccardoporreca
Copy link
Contributor

If this helps: Poetry 1.2 should be enough for this, since python-poetry/poetry#3405 was included starting from 1.2.0a1

@maresb
Copy link
Contributor

maresb commented Jul 6, 2023

Yes, sorry about this. I have been meaning to do this for quite some time, and it's very high on my todo list. I'm hoping to get to it this weekend, but I can't promise anything.

@riccardoporreca
Copy link
Contributor

Thanks @maresb, much appreciated, and if there is anything I can help with, especially around testing, let me know!

@riccardoporreca
Copy link
Contributor

@maresb, would you have a realistic timeline for revendoring poetry and releasing a new conda-lock version?

Not trying to put pressure on you, but rather to manage internally expectations around a timeline for the adoption of conda-lock 2.x.

@maresb
Copy link
Contributor

maresb commented Jul 10, 2023

I'll try again this weekend, but since something might come up, better to assume that I'll get it by the subsequent weekend with about p=75%.

@robertodr
Copy link

I'm also getting hit by this, exactly as @riccardoporreca described.

@MarosMK
Copy link

MarosMK commented Nov 8, 2023

Hi @maresb, this still seems to be an issue. Are we planning to update the vendored Poetry anytime soon?

@maresb
Copy link
Contributor

maresb commented Nov 8, 2023

Thanks for the reminder. This is still on my radar, and I would like very much to do the update soon, but it requires a big block of time that's very difficult for me to allocate at the moment. 😞

@riccardoporreca
Copy link
Contributor

@maresb, let me know if there is anything I can help with: I understand this is quite delicate and time consuming, but if there is even some preparatory work we can help with, happy to support!

@MarosMK
Copy link

MarosMK commented Feb 2, 2024

Hi @maresb, just a reminder so it doesn't get forgotten.

@AlbertDeFusco
Copy link
Contributor Author

Just so I don't forget I found another edge case with the vendored poetry. Is there a safe way to manage the import of conda_lock so that these cases might be avoided?

❯ conda list "conda-lock|attrs"
# packages in environment at /Users/adefusco/Desktop/cl:
#
# Name                    Version                   Build  Channel
attrs                     23.1.0           py39hca03da5_0  
conda-lock                2.5.6            py39hca03da5_0

This version of attrs has AttrsInstance

python
Python 3.9.19 (main, May  6 2024, 14:39:30) 
[Clang 14.0.6 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from attrs import AttrsInstance
>>>

but the vendored version from conda-lock does not

python
Python 3.9.19 (main, May  6 2024, 14:39:30) 
[Clang 14.0.6 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import conda_lock
>>> from attrs import AttrsInstance
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/adefusco/Desktop/cl/lib/python3.9/site-packages/attrs/__init__.py", line 3, in <module>
    from attr import (
ImportError: cannot import name 'AttrsInstance' from 'attr' (/Users/adefusco/Desktop/cl/lib/python3.9/site-packages/conda_lock/_vendor/poetry/core/_vendor/attr/__init__.py)
>>> 

@maresb
Copy link
Contributor

maresb commented Jun 2, 2024

This will be closed by #637 once we fix the remaining failing test case.

@AlbertDeFusco, that branch also fixes your import issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants