Support Groups as Extras

[jaymegordo]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

There seems to be a few places where this issue is mentioned, but I don't see anything explicitly tracking it.

(Specifically @matthijskooijman's comment (point 2) 1644#issuecomment-895848818)

The new groups functionality is very helpful when managing packages installed while developing a project, but seems to miss a pretty good use case - basically using groups in place of extras when installing this package in another environment.

Instead of having to do:

# myproject_1

[tool.poetry.dependencies]
package_1 = "^0.0.1"

[tool.poetry.group.group_a]
optional = true

# this lets someone else develop this project with optional deps
[tool.poetry.group.group_a.dependencies]
package_2 = {version = "^0.0.2", optional = true}
package_3 = {version = "^0.0.3", optional = true}

# need to explicitly define "extras" for other packages install optional deps
[tool.poetry.extras]
group_a = ["package_2", "package_3"]

It would be cleaner (and make more sense I think, once groups is out of preview), if we could just do:

# myproject_1

[tool.poetry.dependencies]
package_1 = "^0.0.1"

[tool.poetry.group.group_a]
optional = true

[tool.poetry.group.group_a.dependencies]
package_2 = "^0.0.2"
package_3 = "^0.0.3"

# didn't need to explicitly define "extras", or individual packages as "optional"

And then in myproject_2:

# myproject_2

[tool.poetry.dependencies]
myproject_1 = {version = "^1.0.0", extras = ["group_a"]}

[jaymegordo]

Side note (not sure if this should be a separate issue), but it would also be helpful if poetry export supported groups as well.

My use case for this is that part of my code base is run as an azure functions app which has to be built with pip/requirements.txt. It would be helpful if I could do poetry export --with group_a,group_b.

[Conchylicultor]

From #4891,

Currently, using group & extras together will silently fail (package don't get installed but no error is raised)

[wk8]

+1, this would be awesome to have.

[AdamWRichardson]

A lot of other comments seem to say that groups and extras are different things and extras are for runtime dependencies and groups are for development dependencies. This is a fair distinction except some tools (such as tox) are well used in development and does separate installation of the package. A well known work around is to use the extras essentially as groups.

In short a big +1 for this feature as it allows grouping in toml but still outputting PEP compliant extras

[neersighted]

We have an FAQ entry about using tox -- note patterns 2 and 3 work with the use of groups, so what you want may already be possible in this case.

[ghost]

I agree, this seems like a huge missed opportunity, groups are much nicer to work with. Perhaps if it's deemed necessary groups could be annotated with category = "main" / category = "dev" (or dev = false / dev = true)

[neersighted]

You're describing optional and non-optional groups -- non-optional (default) groups label and sort your dependencies but are always in the tree. Optional groups are used as a replacement for dev dependencies.

Extras are inherently different as they can include the same package multiple time -- the current design of groups is not designed with similar functionality in mind. Given extras are a regular feature of the Python ecosystem, the current design for them is focused on familiarity and parity with the rest of the ecosystem.

It is certainly possible to add interactions between groups and extras, but no one on the core team is (to the best of my knowledge) yet thinking about or working on such a feature. A cohesive design proposal/MR would be welcome, but would likely have to go through a decent amount of iteration to get it mergable.

Also, any work on changing the project format is likely on hold until we can stabilize PEP 621 support.

[jacek-jablonski]

How can we elegantly solve this problem?

[tool.poetry.dependencies]
python = "~3.10"
asyncpg = { version = "*", optional = true }

[tool.poetry.group.dev.dependencies]
pytest = "^7.2.0"
pytest-asyncio = "^0.20.2"
pytest-cov = "^4.0.0"
pytest-mock = "^3.10.0"

asyncpg-stubs = { version = "*", optional = true }

[tool.poetry.extras]
asyncpg = ["asyncpg", "asyncpg-stubs"]

Command poetry install --all-extras --with dev doesn't install asyncpg-stubs.

[neersighted]

Optional dependencies outside the main group are meaningless in terms of extras. Move it to main and things will work as you expect.

[ghost]

i imagine once pep 621 support is added this will be solved as it has a much nicer syntax for extras:

[project]
dependencies = [
    "foo ~= 1.0",
]

[project.optional-dependencies]
dev = [
    "bar ~= 2.0",
]

[rino0601]

Let say I want to pip install myproject_1[group_a]
For now, it seems I have to write it this way:

# myproject_1
[tool.poetry.dependencies]
package_1 = "^0.0.1"

# annoying repetition 1
package_2 = {version = "^0.0.2", optional = true}
package_3 = {version = "^0.0.3", optional = true}

[tool.poetry.group.group_a]
optional = true

[tool.poetry.group.group_a.dependencies]
package_2 = "^0.0.2"
package_3 = "^0.0.3"

# need to explicitly define "extras" for other packages install optional deps
[tool.poetry.extras]
group_a = ["package_2", "package_3"] # annoying repetition 2

It would be nice if I could write it concisely like:

# myproject_1
[tool.poetry.dependencies]
package_1 = "^0.0.1"

[tool.poetry.group.group_a]
optional = true
extra = true    # just 1 flag to avoid repetition
[tool.poetry.group.group_a.dependencies]
package_2 = "^0.0.2"
package_3 = "^0.0.3"

There will be many cases in which you want to verify dependencies expressed in extras during development, so in many cases extras will soon become groups. Therefore, it would be very useful if we could express it as above.

[vwxyzjn]

Hello. I have another related bug #7743 to report. If I use the proposed solution from OP @jaymegordo, poetry export will export all dependencies, including optional ones, without distinction.

[BlackCatDevel0per]

Hi, I've created a simple plugin that wraps the build command and allows mixing dependencies from a TOML option into a specified group, like the "main" group:

toml:

[tool.poetry-plugin-deps-juice]
"poetry" = [  # mix into ("main" alias)
    # the groups list include from
     "base",
     "setup",
]

Poetry build:

poetry build && pkginfo -f requires_dist dist/*.whl

output:

Building poetry-plugin-deps-juice (0.0.2)
  - Building sdist
  - Built poetry_plugin_deps_juice-0.0.2.tar.gz
  - Building wheel
  - Built poetry_plugin_deps_juice-0.0.2-py3-none-any.whl
requires_dist: ['poetry (>=1.8.0,<2.0.0)']

Juice plugin build:

poetry jbuild && pkginfo -f requires_dist dist/*.whl

output:

Mixing juice..
base -> poetry
        {'poetry-core': '^1.7.0', 'wheel': '^0.42.0'}
setup -> poetry
        {'build': '^1.0.3', 'setuptools': '^69.0.3'}
Building poetry-plugin-deps-juice (0.0.2)
  - Building sdist
  - Built poetry_plugin_deps_juice-0.0.2.tar.gz
  - Building wheel
  - Built poetry_plugin_deps_juice-0.0.2-py3-none-any.whl
requires_dist: ['build (>=1.0.3,<2.0.0)', 'poetry (>=1.8.0,<2.0.0)', 'poetry-core (>=1.7.0,<2.0.0)', 'setuptools (>=69.0.3,<70.0.0)', 'wheel (>=0.42.0,<0.43.0)']

I think it's not the best solution, but it allows to put some dependency groups into the main (or else group) group without changing the TOML source for better readability.

[MichalDN]

Let say I want to pip install myproject_1[group_a] For now, it seems I have to write it this way:

# myproject_1
[tool.poetry.dependencies]
package_1 = "^0.0.1"

# annoying repetition 1
package_2 = {version = "^0.0.2", optional = true}
package_3 = {version = "^0.0.3", optional = true}

[tool.poetry.group.group_a]
optional = true

[tool.poetry.group.group_a.dependencies]
package_2 = "^0.0.2"
package_3 = "^0.0.3"

# need to explicitly define "extras" for other packages install optional deps
[tool.poetry.extras]
group_a = ["package_2", "package_3"] # annoying repetition 2

It would be nice if I could write it concisely like:

# myproject_1
[tool.poetry.dependencies]
package_1 = "^0.0.1"

[tool.poetry.group.group_a]
optional = true
extra = true    # just 1 flag to avoid repetition
[tool.poetry.group.group_a.dependencies]
package_2 = "^0.0.2"
package_3 = "^0.0.3"

There will be many cases in which you want to verify dependencies expressed in extras during development, so in many cases extras will soon become groups. Therefore, it would be very useful if we could express it as above.

If I may I would like to point out, that this solution would also be extremely for tools that rely on pip, like cibuildwheel.

Supplying test dependencies, without either code or data redundancy seems to be at the moment impossible

I can either
install the dependencies in some roundabout way like this

before-test = [
    "source $(dirname `which python`)/activate",
    "poetry install --no-root --with test"
  ]

which is hacky at best

or by extra/group duplication like this

[tool.poetry.dependencies]
python = ">=3.9,< 3.11"
pytest = {version="^7.2.1", optional=true}
pytest-benchmark = {version="^4.0.0", optional=true}
pytest-cov = {version="^4.0.0", optional=true}

[tool.poetry.group.test.dependencies]
pytest = {version="^7.2.1"}
pytest-benchmark = {version="^4.0.0"}
pytest-cov = {version="^4.0.0"}

[tool.poetry.extras]
test = ["pytest", "pytest-benchmark", "pytest-cov"]

the above suggested, or similarly

[tool.poetry.dependencies]
python = ">=3.9,< 3.11"

[tool.poetry.group.test.dependencies]
pytest = {version="^7.2.1", extra = true}
pytest-benchmark = {version="^4.0.0", extra = true}
pytest-cov = {version="^4.0.0", extra = true}

[tool.poetry.extras]
test = ["pytest", "pytest-benchmark", "pytest-cov"]

would be concise and maintainable.