[virtualenvs] configuration not working with pyproject.toml

[lcbm]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Linux 5.8.9-arch2-1 x86_64
• Poetry version: 1.0.10
• Link of a Gist with the contents of your pyproject.toml file: 👇

[tool.poetry]
name = "name"
version = "0.1.0"
description = "description"
authors = ["my-name my-name@mail.com"]
license = "BSD-3-Clause"

[tool.poetry.dependencies]
python = "^3.8"

[tool.poetry.dev-dependencies]
pre-commit = "^2.7.1"
pylint = "^2.6.0"
pytest = "^6.0.2"
pytest-cov = "^2.10.1"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

[virtualenvs]
create = true
in-project = true

Issue

Hello, I've been trying to make Poetry create the virtual environment folder in the project's root but it kept installing in home/leleco/.cache/pypoetry/virtualenvs despite the configuration set in pyproject.toml file.

After a few tries, I ran poetry config --list and the output was this 👇

cache-dir = "/home/leleco/.cache/pypoetry"
virtualenvs.create = true
virtualenvs.in-project = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # /home/leleco/.cache/pypoetry/virtualenvs

This got me startled, so I tried poetry config virtualenvs.in-project true, then poetry install and it worked: a .venv was created inside my project's root directory. I then tried the same, except that with virtualenvs.create 👇

# pyproject.toml

[virtualenvs]
create = false
in-project = true

And the result from poetry config --list was this 👇

cache-dir = "/home/leleco/.cache/pypoetry"
virtualenvs.create = true
virtualenvs.in-project = true
virtualenvs.path = "{cache-dir}/virtualenvs"  # /home/leleco/.cache/pypoetry/virtualenvs

With this, I concluded that [virtualenvs] does not work with pyproject.toml (which was expected to me, as a user). Furthermore, poetry config virtualenvs.in-project true --local creates a poetry.toml instead of adding an entry in pyproject.toml. 👇

# poetry.toml

[virtualenvs]
in-project = true

Is this expected? If so, I couldn't find the documentation explaining this... also, is there a chance that we convert this issue from Bug to Feature Request in case the described behavior is the expected behavior? I'm more than glad to update this description and convert it to a feature request with proper use-case description.

Thanks in advance,
Hope I can be useful to resolve this, or to implement this feature, somehow 🥳

[itsthejoker]

To the best of my knowledge, using a key called [virtualenvs] has never been supported, nor has overriding config options in the pyproject.toml -- the poetry settings are used to handle processing for the pyproject.toml, so it makes sense that they are separate.

[lcbm]

I see your point and partially agree with the argument. Nevertheless, I believe there's at least one action item for this issue: better document poetry.toml. I read through the docs for a while and found nothing on this issue (pyproject.toml vs poetry.toml). Not even on stackoverflow I found much about it (the most I got was someone mentioning poetry.toml existed, but it was such an old post that I simply thought it was a deprecated Poetry configuration file).

For me, this is really a pain because VS Code doesn't work with the default poetry venv dir (there are issues open on this) and VS Code users HAVE to change venv's directory in order to use poetry venv. In order to avoid that my team has problems with poetry and VS Code, I looked for a way to enforce poetry to install venv in the project's root everytime (that way, VS Code would always find the venv folder). This being said, I can see that lots of developers will spend considerable time looking for an appropriate solution for their professional projects and not finding proper documentation and relying on trial and error or workarounds. I believe this justifies adding some documentation on poetry.toml and it's usage, what configs could go there and so on.

Lastly, an opinion on the separation of files: we have black, isort and many other tool configurations under pyproject.toml -> [tools.<tool-name>]. I don't see why we can't have a project setting (where to install venv, for this specific project) in the same file (pyproject.toml) under different keys in order to avoid increasing the number of files. 🤔

See this example of a all-in-one configuration file (there are many other examples).

Just to be clear, I really appreciate your help and opinion... this is a IMO response only, based on bad UX regarding the documentation (not Poetry itself, which is so awesome that I'm trying to change my peers mind to use it in our project) 😄

[finswimmer]

Hello @lcbm,

the reason why the local configs are not in pyproject.toml but in a separate poetry.toml is that one developer should not force another how poetry is configured. This would automatically the case if these local config would be in the pyproject.toml.

But it's a good idea to document the existens of this file and would fit at best in the Local Configuration section of the docs.

PR is welcome :)

fin swimmer

[max-sixty]

the reason why the local configs are not in pyproject.toml but in a separate poetry.toml is that one developer should not force another how poetry is configured. This would automatically the case if these local config would be in the pyproject.toml.

Could I ask what the difference between having the local config in poetry.toml and pyproject.toml is, on this dimension?

[finswimmer]

Could I ask what the difference between having the local config in poetry.toml and pyproject.toml is, on this dimension?

The pyproject.toml contains information about your package, like metadata and dependencies. Anything that is needed to build the package.

The poetry.toml contains the local configuration of the poetry application. One should not check-in this file into a repository, because these configuration are just personal flavors.

[max-sixty]

One should not check-in this file into a repository, because these configuration are just personal flavors.

I see, thanks @finswimmer

[0x2b3bfa0]

On some arcane setups where virtual environments need to be managed with another tool (see conda), it would be really helpful to have a way of overriding the virtual environment settings from the pyptoject.toml file, as disabling virtual environment creation is not a user preference but a must.

@finswimmer, would it be possible to preserve the current behavior and allow custom overrides? The priority order would look like this:

1. Global configuration.
2. Overrides from pyproject.toml.
3. Local configuration from poetry.toml.

[finswimmer]

Hello @0x2b3bfa0,

if poetry detects that it is already running in a virtual environment - also if it is created by conda - it will not create another venv. Does this fit to your use case?

fin swimmer

[0x2b3bfa0]

🙏 Thank you, @finswimmer! I wasn't able to trigger that behavior due to a misconfiguration, but it works perfectly now.

[kriansa]

@finswimmer I would like to add another point to maybe revisit this decision.

Recently Poetry implemented another config system-site-packages (#1393 and #3711), which not only allows for saving space, but also make currently incompatible packages (such as those not supporting PEP-517) to work seamlessly with Poetry as well (so we don't need to specify them on dependencies and still rely on them as a dependency on host).

This setting alone will turn the developer's environment consistent and therefore it should be set on a per-project basis IMHO. This was an example, but other settings might be introduced in the future that developers would like to "enforce" across their teams for sake of consistency/environment reproducibility.

Perhaps, as an option for future-proofing, it's worth considering allowing users to set these properties at the project-level as well (on pyproject.toml), being overridden by those in global, then by those in local scope (so we don't break compatibility). What do you think?

[maxsu]

Is there any patch or fork here yet?

[kriansa]

I assume you're talking about enabling poetry configuration at the project level. In that case, not that I'm aware of, but maybe due to the issue title and tags, this is not getting enough visibility.

To help the devs (and to better clarify the request) we can push this as a feature request in a separate issue.

[victorliun]

Hello @0x2b3bfa0,

if poetry detects that it is already running in a virtual environment - also if it is created by conda - it will not create another venv. Does this fit to your use case?

fin swimmer

Hi @finswimmer

I quite agree with @0x2b3bfa0 's suggestion. So I can have a default virtualenv setting in pyproject.toml(this makes perfect sense for the purpose of pyproject.toml), then if developer prefers a different venv setting then it can be overwritten by poetry.toml, this case poetry.toml can be excluded from version control.

[nacitar]

Hello @0x2b3bfa0,
if poetry detects that it is already running in a virtual environment - also if it is created by conda - it will not create another venv. Does this fit to your use case?
fin swimmer

Hi @finswimmer

I quite agree with @0x2b3bfa0 's suggestion. So I can have a default virtualenv setting in pyproject.toml(this makes perfect sense for the purpose of pyproject.toml), then if developer prefers a different venv setting then it can be overwritten by poetry.toml, this case poetry.toml can be excluded from version control.

This is precisely what needs to be done.

[Secrus]

poetry.toml is now mentioned in the documentation. There is no plan on adding poetry settings control to pyproject.toml file.

[woutervh]

The poetry.toml contains the local configuration of the poetry application. One should not check-in this file into a repository, because these configuration are just personal flavors.

It's just a personal opinion that these are personal opinions. Other teams can decide otherwise.

pyproject.toml contains the configuration for black,isort,mypy, pytest,...
It contains the information which python-version the virtualenv needs to be instantiated for, and what packages needs to be installed in the virtualenv, so why not support these extra settings?

We are happy to get rid of pytest.ini, .pylintrc,...
and a plentitude of other files that can be consolidated in pyproject.toml

Now we are forced to create a poetry.toml, add it to the repo,
just to have all our teammembers a consistent setup with an in-project virtualenv.

[Radiergummi]

Now we are forced to create a poetry.toml, add it to the repo, just to have all our teammembers a consistent setup with an in-project virtualenv.

To add to this, I can only assume you guys don't have a bunch of junior developers on your team. Having a consistent development setup is incredibly important to aid beginners and writing instructions. It would be great to ensure all of them have the venv folder at the same location.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.