[FAQ]: The tox answer should recommend poetry install --no-root instead of poetry install

[thejohnfreeman]

• I have searched the issues of this repo and believe that this is not a duplicate.

In the FAQ, there is a recommendation for how to use tox with Poetry. It suggests this tox.ini:

[tox]
isolated_build = true
envlist = py27, py36

[testenv]
whitelist_externals = poetry
commands =
    poetry install -v
    poetry run pytest tests/

With this configuration, when a user runs tox:

1. tox packages the project with Poetry.
2. tox creates a virtual environment.
3. tox installs the package it built, but not its dependencies (inst-nodeps).
4. tox calls poetry install, which installs the project dependencies, and then installs the project, as editable, overwriting what tox installed in step 3.
5. tox calls poetry run pytest tests which tests against the package as it exists in the working directory, not as it was packaged by Poetry.

To fix this, the recommended commands should be:

commands =
    poetry install --no-root -v
    poetry run pytest tests/

[sinoroc]

I believe Poetry should be left out of tox.ini entirely. This is the pattern I would recommend:

tox.ini

[tox]
envlist = py{36,37,38}-django{22,30}
isolated_build = True

[testenv]
# ...
deps =
    django22: Django==2.2
    django30: Django==3.0
extras =
    test
commands =
    pytest

pyproject.toml

[tool.poetry]
# ...

[tool.poetry.dependencies]
python = "^3.6"
django = "^2.2 || ^3.0"
#
pytest = { version = "^5.2", optional = true }

[tool.poetry.extras]
test = ["pytest"]

[build-system]
# ...

[Alexspayne]

I believe poetry should be left out of tox.ini entirely. This is the pattern I would recommend:

tox.ini

[tox]
envlist = py{36,37,38}-django{22,30}
isolated_build = True

[testenv]
# ...
deps =
    django22: Django==2.2
    django30: Django==3.0
extras =
    test
commands =
    pytest

pyproject.toml

[tool.poetry]
# ...

[tool.poetry.dependencies]
python = "^3.6"
django = "^2.2 || ^3.0"
#
pytest = { version = "^5.2", optional = true }

[tool.poetry.extras]
test = ["pytest"]

[build-system]
# ...

You will have maintain duplicate lists of your dependencies if you do that, and you might not get the same version of the dependencies in the tox environment as you would in production because it wouldn't be honoring the poetry.lock file.

[sinoroc]

In this workflow scenario the dev-dependencies section is explicitly left out in favor of the test extra in order to avoid having to maintain two lists of development dependencies. The run-time dependencies are listed as usual.

The tox tool is meant to test a library or application against multiple combinations of different versions of the Python interpreter and different versions of the dependencies. It is explicitly not intended to test just one combination. If it is your case, you probably don't need tox to begin with. Unless you are both the developer/maintainer and sole user of the project (application or library) and you can allow yourself to exactly pin the version numbers of all dependencies, there is little control over which versions of the other packages are installed in the environment your project runs in.

I believe it is the responsibility of the user (integrator, system administrator) to pin the versions of the dependencies and thoroughly test this exact combination. Not the responsibility of the maintainers of the project. This is the kind of workflow scenario this suggestion is targeted at.

A scenario where the solution currently presented in the FAQ could make sense (everything is pinned in poetry.lock) could be in the case of an application distributed as a whole with all dependencies included, such as pex, shiv, zipapp, or anything similar. But there is no need for tox to test such a static environment.

[thejohnfreeman]

I'm totally misunderstanding something then. How can I use tox or another tool to verify that poetry build packages my library correctly?

[sinoroc]

I'm totally misunderstanding something then. How can I use tox or another tool to verify that poetry build packages my library correctly?

Well, tox does exactly that. The first thing tox does is package a source distribution (sdist) of the project. Then it creates a virtual environment and installs the sdist in it. So tox executes the commands against the project as it is installed from a distribution of it, and not as an editable or development version of it (like poetry install does).

When isolated_build is set, tox knows to check for the build-system section of pyproject.toml in order to build the sdist. I believe it is safe to assume that executing the poetry.masonry.api function that is set as value for build-backend leads to the same result as executing poetry build.

To also run some tests against the project as installed from its wheel distribution, one could look into the tox-wheel plugin or something similar.

Additionally I would also recommend doing something like the following to check the distributions with twine:

[testenv:package]
skip_install = True
deps =
    twine
commands =
    python3 -m poetry build
    python3 -m twine check dist/*

And it also goes without tox:

$ poetry add --dev twine
$ poetry build
$ poetry run twine check dist/*

[igor47]

A scenario where the solution currently presented in the FAQ could make sense (everything is pinned in poetry.lock) could be in the case of an application distributed as a whole with all dependencies included, such as pex, shiv, zipapp, or anything similar. But there is no need for tox to test such a static environment.

i maintain some applications distributed as a whole, with all dependencies included (if by distributed, you mean packaged into docker containers to be served to users). i still like using tox, because you can specify multiple test commands. i tell other developers of the application to just run tox, and they get a run of pytest, flake8, mypy, black, etc... without tox, they would need to run each of these separately. by having developers as well as CI just invoke tox, i get consistency in testing between local and CI environments.

so, i believe that there is still utility in using tox, even when what i want tox to do is to install just the things in my poetry.lock. given that tox with poetry seems to ignore poetry.lock, i think this comment from @sinoroc might be incorrect at least in my use case:

I believe poetry should be left out of tox.ini entirely.

[sinoroc]

i believe that there is still utility in using tox

@igor47 My point is: Do use tox, but don't call poetry from inside tox.ini. The reason is that otherwise poetry cancels some of the work that tox does.

even when what i want tox to do is to install just the things in my poetry.lock. given that tox with poetry seems to ignore poetry.lock

The main reason for using tox is to test one's project against a wide range of combinations of dependencies and Python interpreters. So testing against the one and only combination registered in poetry.lock seems to me like it goes against this philosophy.

i maintain some applications distributed as a whole, with all dependencies included (if by distributed, you mean packaged into docker containers to be served to users). i still like using tox, because you can specify multiple test commands. i tell other developers of the application to just run tox, and they get a run of pytest, flake8, mypy, black, etc... without tox, they would need to run each of these separately. by having developers as well as CI just invoke tox, i get consistency in testing between local and CI environments.

Now I am open to admit that for some projects it might be the sensible thing to test only against one version of Python interpreter (in that case, the one in the Docker container) and the combination of dependencies registered in poetry.lock, and tox could help do such a thing, but I would still argue that in such a case it's not the right tool for the job. Maybe something like pyinvoke for example might be better suited to such a scenario (poetry run pyinvoke format lint test).

[schinckel]

Tox is not just useful for testing combinations of different versions of packages or python interpreters.

We use tox for automation: seperate tox environments are used to have things like Robot Framework tests run, tests of JS, pg_tap tests for testing database functions. Also, having tox environments for flake8, isort and friends (which can finish much faster than the full test suite) means that those can report failures much faster than waiting for all tests to run.

It's also very useful for managing dependencies between test environments (ie, runs the "clean" environment before anything else, and the "code-coverage" collector environment after all other environments).

[sinoroc]

Tox is not just useful for testing combinations of different versions of packages or python interpreters.

[...]

@schinckel Those are all legitimate use cases for tox indeed, but... What would that mean for the current topic? What would be your recommendation? What should be written in the FAQ? In short, which tool should handle the creation of the virtual environments, the installation of the project, of its dependencies? Should it be tox or poetry?

[jlumbroso]

Thanks to everyone for the conversation.

I agree with @sinoroc and others on the principle, that the whole goal of tox (and similar tools) is "What would happen in similar configurations I don't have exact control over?" In that sense, as package creators, we can choose to use poetry, but there is no guarantee that will be used by the end user.

However, the goal of poetry the way I understand it is precisely to abstract all these config tweaking/hacking problems away and to provide A Good Solution to all common package managing issues—in part just to make it easier for package creators to do the right thing and adhere to best practices. For instance, that's why poetry provides the pytest stub in new projects. Not everybody needs to have testing, or even use pytest ❤️, but it's a good choice and the default makes it easier.

In that sense, I agree with @thejohnfreeman, @Alexspayne and @schinckel, that there is something sub-optimal about the current answers.

In another issue, @wesleykendall provided an excellent alternate solution that is: Automatically export dependencies to a common API that tox understands without poetry's help (i.e., the requirements.txt format). I added a suggestion that this export happen at commit-time rather than run/compile/build-time so that poetry is not a dependency just to export the deps list.

Other issues re: tox/poetry: #848, #1745, #1941.

[thejohnfreeman]

I left a thumbs up on @sinoroc's explanation, but never explicitly said that he changed my opinion on this subject, after I copied his suggested pattern with success. I was unaware that tox would build the package with Poetry via the [build-system] section of pyproject.toml. I thought it wasn't, and that that was the reason it had to be hacked in with the pattern presently suggested in the documentation. Maybe that documentation was written at a time before tox was made aware of [build-system], when the pattern it recommends was necessary.

[sinoroc]

I was unaware that tox would build the package with Poetry via the [build-system] section of pyproject.toml. I thought it wasn't, and that that was the reason it had to be hacked in with the pattern presently suggested in the documentation. Maybe that documentation was written at a time before tox was made aware of [build-system], when the pattern it recommends was necessary.

This is enabled by the isolated_build = true setting. This lets tox know to look into pyproject.toml's build-system section, and delegates the build to the proper build backend, which in the case here is poetry obviously. This was always (as far as I can remember at least), clearly the case in both FAQ's (poetry and tox), but then poetry's FAQ muddies the water by recommending commands = poetry install, which is a step back: it destroys the work done by isolated_build.

This isolated build feature is the new standard (look up PEP 517 and PEP 518), works also with setuptools, flit, etc.

The missing element here is about the development dependencies. How to let tox know what the dev dependencies are?

This can be solved by using the extras (which is a standard, i.e. specified by a PEP), this pattern is quite common nowadays and is guaranteed to work across many different tools (not only tox, and poetry).

On the other hand, generating a requirements.txt file (not a standard, there is no PEP specification) dynamically, feels unnecessary to me. And poetry's own dev-dependencies field is not a standard either, so tox doesn't know how to handle it.

But again, if one wants to go all-in with poetry and use all its features (even the non standard ones), feel free. But it means that you will have to work harder to make it cooperate with other tools that do follow standards. Which could be fine, maybe there isn't even need for tox to begin with. Just running poetry run pytest could be perfectly good, no need for more.

So my point is: in my opinion poetry's FAQ on the topic of tox is misleading and shows a largely sub-optimal pattern.

---

I have been thinking for a while about writing a plugin for tox that would offer tox a way to automatically pick up poetry's dev dependencies. Maybe that would put this issue to rest. Or maybe there's already such a plugin? If you know of one, let us know.

[fredrikaverpil]

I'm hoping to be able to specify dependency groups in my CI once this is available: #1644 (planned for poetry 1.2)

But until then, what is the recommended usage of poetry with tox when you need to install dev dependencies for e.g. linting/pytest or the other usual CI suspects?

I've noticed that poetry's pyproject.toml itself differs from what the FAQ says on the build system declaration:
https://github.com/python-poetry/poetry/blob/master/pyproject.toml#L86
So right now I'm using that in conjunction with "poetry install" prior to my tox commands.

I'm also contemplating whether I should try to use tox-current-env right now too, as all my tox envs end up identical in terms of dependencies but takes ages to build. And no I cannot use poetry's extras...

Ping @abn @sdispater can someone please explain what is the current recommendation?

[jlumbroso]

@sinoroc

This can be solved by using the extras (which is a standard, i.e. specified by a PEP), this pattern is quite common nowadays and is guaranteed to work across many different tools (not only tox, and poetry).

On the other hand, generating a requirements.txt file (not a standard, there is no PEP specification) dynamically, feels unnecessary to me. And poetry's own dev-dependencies field is not a standard either, so tox doesn't know how to handle it.

Generating a requirements.txt feels unnecessary to you—and using an "extras" to specify dev dependencies when there is another more specific mechanism designed explicitly to do so (i.e., dev-dependencies, literally), that's what feels redundant/unnecessary to me.

I have been thinking for a while about writing a plugin for tox that would offer tox a way to automatically pick up poetry's dev dependencies. Maybe that would put this issue to rest. Or maybe there's already such a plugin? If you know of one, let us know.

I knew of no such plugin, however some due-diligence Googling led me to discover a very unusual effort:
https://github.com/tkukushkin/tox-poetry
https://pypi.org/project/tox-poetry/

There are some things that are unusual: The PyPi repo link is pointing to a different repo. The test coverage is 0%. The project seems to have been forked (a bit sloppily) from tox-pipenv-install (probably a good initial starter code).

I think it would be an excellent idea to have a reliable plugin.

[jlumbroso]

@fredrikaverpil

I'm hoping to be able to specify dependency groups in my CI once this is available: #1644 (planned for poetry 1.2)

But until then, what is the recommended usage of poetry with tox when you need to install dev dependencies for e.g. linting/pytest or the other usual CI suspects? [...] And no I cannot use poetry's extras...

I agree that using extras is not ideal.

So here's what I have done in my most recent package, comma (Python CSV for human!! 😇):

• I have my dev-dependencies specified in pyproject.toml
• I have my CI automatically update requirements.txt and requirements-dev.txt on any commit that changes dependencies
• [ Currently I use GitHub Actions as CI, and a big great shoutout to @abatilo for the super-convenient and foolproof actions-poetry ]
• Then I have configured my tox.ini to simply be:

  [tox]
  isolated_build = true
  envlist = py{36,37,38,39}
  
  [testenv]
  deps = -rrequirements-dev.txt
  commands =
      pytest

This works wonderfully and allowed me to expand the compatibility of my package to versions different than my own (Py 3.8 right now). What I like about this solution is that I still only specify everything in one place, pyproject.toml, the CI pipeline robustly takes care of everything else.

My next steps are:

• take advantage of the fact that you can embed the legacy tox.ini file description within pyproject.toml to reduce the number of configuration files;
• have tox be run on commit (right now, the regression tests built on commit only test Py 3.8).

[ If you find this useful, I would appreciate a ⭐ on my package 😅. ]

[thejohnfreeman]

One of the reasons I'm excited about PEP 518 (pyproject.toml) is that I don't have to keep a requirements.txt (much less two of them) in my codebase, confusing newcomers and adding an operational burden (CI job just to keep it synchronized).

Extras has good support across Python tools, so I have switched to using it instead of [dev-dependencies], a Poetry-specific extension not found in PEP 518 (or any other PEP to my knowledge).

[jlumbroso]

Good points @thejohnfreeman.

Although I agree with you, I think the practice of having requirements-dev.txt widespread in our community enough that GitHub automatically parses that file to provide metadata about a repository, which I think is important for many reasons—for instance it helps us track usage, it helps us track old dependencies, security vulnerabilities, etc..

But personally I agree it makes more sense to use an existing mechanism for specifying an additional subset of dependencies. It would be nice if there was some PEP just to say "the best way to provide development dependencies is an 'extra' called 'dev'" or something like that. I wonder what the process would be for that to be suggested. Any ideas?

[sinoroc]

It would be nice if there was some PEP just to say "the best way to provide development dependencies is an 'extra' called 'dev'" or something like that.

Agreed. From my point of view it should be a convention (a recommendation), not a specification (a rule).

I wonder what the process would be for that to be suggested. Any ideas?

Maybe. PyPA is in the process of writing new PEPs, one of them in particular PEP 621 proposes to specify the way project's metadata should be stored in pyproject.toml files. The discussion is a bit on pause since a few days, but still open, so it's probably the right time to make such suggestions.

You might want to read these two threads first:

• https://discuss.python.org/t/pep-621-storing-project-metadata-in-pyproject-toml/4513
• https://discuss.python.org/t/pep-621-how-to-specify-dependencies/4599

And then open a new thread on that forum to make your suggestion. But no guarantee: it might be considered out of scope for a PEP.

[sinoroc]

I have been thinking for a while about writing a plugin for tox that would offer tox a way to automatically pick up poetry's dev dependencies. Maybe that would put this issue to rest.

I published on PyPI a plugin for Tox to instruct it to install Poetry's development dependencies in the test environments: tox-poetry-dev-dependencies. It's just a proof of concept. I didn't test much, only with a very simple project.

[jlumbroso]

Very exciting @sinoroc — can't wait to try it out!!

[thejohnfreeman]

I'd rather move away from Poetry [tool.poetry.dev-dependencies] and toward extras (or dependency groups when they become available). Right now, I structure my projects with at least 3 extras:

• test: dependencies necessary to run the tests: pytest, hypothesis, ...
• docs: dependencies necessary to build the documentation: sphinx, sphinx_rtd_theme
• dev: dependencies used only when writing code: yapf, sphinx-autobuild, python-language-server, ...

Tox only needs to install test. Read The Docs only needs to install docs. My development environment installs everything. If Poetry is going to install any extra dependencies by default, it should let me pick a set of extras.

[sinoroc]

I'd rather move away from Poetry [tool.poetry.dev-dependencies] and toward extras (or dependency groups when they become available). Right now, I structure my projects with at least 3 extras [...]

I also prefer extras for my projects, (see the very code of tox-poetry-dev-dependencies for example). If I am not mistaken, in your case one of the reasons might be that you want to split your development dependencies over multiple categories (lint, test, and doc for example), while (for now) Poetry only offers one single group (dev-dependencies). But for some projects, one category might be enough. Poetry's dependency groups if they are released might indeed be a better fit for your use cases, but most likely that would still require a plugin to integrate them with Tox. So here we are, there is a plugin now for those projects that use Poetry's dev-dependencies, and when/if the time comes I guess it could be extended to support Poetry's dependency groups as well.

[layday]

Version A will be rejected by pip when installing, either from a wheel or from source, and by PyPI when uploading a wheel:

$ twine upload --repository testpypi dist/sampleproject_direct_url_reference-2.0.0-py3-none-any.whl
[...]
HTTPError: 400 Bad Request from https://test.pypi.org/legacy/
Invalid value for requires_dist. Error: Can't have direct dependency: "mypy @ git+https://github.com/python/mypy.git ; extra == 'dev_test'"

[sinoroc]

Version A will be rejected by pip when installing, either from a wheel or from source, and by PyPI when uploading a wheel:

Thanks for the confirmation that in version A the upload to PyPI fails.

Although, I must say, that for me pip was able to install version A from its wheel (local file system, not PyPI) and from its source directory (again local file system, obviously).

Would version B work for you? It requires specifying some of the development dependencies twice, but only those that require a direct URL. That would be the best compromise I can think of right now.

[layday]

I personally use Nox, but version B would work. Just a little gotcha to bear in mind. I can't imagine it crops up often.

[DustinMoriarty]

All of this seems like a work around to me. Is there something that needs to change on the tox side to make tox set up poetry environments correctly? In theory, it seems that the tox.ini should look the same for a poetry package as it would for a package using setup.py. I don't put "pip install" in a tox.ini for a setuptools project. If there are things that need to be fixed on the tox side, perhaps we can focus our efforts there.

Sometimes, I have been able to get this to work, but other times it has not. I have yet to chase down all the different system configuration effects to understand why it does not always work. It seems that the goal should be that this would be all that would be needed.

[tox]
envlist = py36,py37,py38
isolated_build = True

[testenv]
deps = pytest
commands = pytest

[hukkin]

The --dev-only flag was recently merged to Poetry master branch. Maybe that should be recommended in the docs instead of --no-root.

Also I think using poetry run before pytest might be needless, as Poetry will detect the venv created by tox and install the dev dependencies there (instead of creating yet another one), if I'm not entirely wrong?

[cglacet]

I don't know about you, but I also experienced issues with private package repository. I had strange issues on the 1st run only, I found that adding --skip-pkg-install to the tox command solved the issue.

The crash experienced during the first run looked like this:

$ tox 
.package create: /Users/xxx/.tox/.tmp/package
.package installdeps: poetry-core>=1.0.0
py38 create: /Users/xxx/.tox/py38
py38 inst: /Users/xxx/.tox/.tmp/package/1/notapackage-0.1.0.tar.gz
ERROR: invocation failed (exit code 1), logfile:/Users/xxx/.tox/py38/log/py38-1.log
======================= log start =======================
Processing ./.tox/.tmp/package/1/this_is_not_a_service-0.1.0.tar.gz
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Collecting graphene-sqlalchemy<3
  Using cached graphene_sqlalchemy-2.3.0-py2.py3-none-any.whl (38 kB)
...
ERROR: Could not find a version that satisfies the requirement some_private_package<0.10.0,>=0.9.1 
ERROR: No matching distribution found for some_private_package<0.10.0,>=0.9.1
WARNING: You are using pip version 21.1.1; however, version 21.1.2 is available.

Subsequent runs worked as expected and (py38 inst wasn't part of tox output). I can also force the error to show up again by running tox -r.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.