Cross path dependencies cause SolverProblemError #3030

atugushev · 2020-10-01T18:21:23Z

I am on the latest Poetry version.
I have searched the issues of this repo and believe that this is not a duplicate.
If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

OS version and name: macOS 10.15.5
Poetry version: 1.1.0
Link of a Gist with the contents of your pyproject.toml file:

Issue

If a project (e.g., poetry-bug-with-cross-path-deps) has the following dependencies:

poetry-bug-with-cross-path-deps depends on package1
poetry-bug-with-cross-path-deps depends on package2
package1 depends on package2

and the package1 and package2 are path dependencies, then poetry lock fails on:

  SolverProblemError

  Because poetry-bug-with-cross-path-deps depends on package1 (0.1.0 deps/package1) which depends on package2 (0.2.0 deps/package2), package2 is required.
  So, because poetry-bug-with-cross-path-deps depends on package2 (0.2.0 deps/package2), version solving failed.

The text was updated successfully, but these errors were encountered:

finswimmer · 2020-10-01T20:01:23Z

Hello @atugushev,

the problem here is, that you the package2 dependency for poetry-bug-with-cross-path-deps is in a different path, than the package2 dependency for package2.

Your folder structure looks like this:

poetry-bug-with-cross-path-deps
├── deps
│   ├── package1
│   │   ├── deps
│   │   │   └── package2
│   │   │       └── pyproject.toml
│   │   └── pyproject.toml
│   └── package2
│       └── pyproject.toml
├── README.md
└── pyproject.toml

So you have poetry-bug-with-cross-path-deps/deps/package2 and poetry-bug-with-cross-path-deps/deps/package1/deps/package2. You have to rename one of these packages.

fin swimmer

atugushev · 2020-10-01T21:21:30Z

Hi @finswimmer,

Thanks for your suggestion. In the real-world package1 and package2 are submodules, so renaming can't help in that case.

finswimmer · 2020-10-02T04:20:57Z

Hello @atugushev,

can you add more detail what you mean by "submodules" here exactly?

At the end of the day, you cannot install two different packages with the same name. So I guess your task must be solved in another way. Happy to help here, but for that I need to understand what the task is :)

fin swimmer

benoit9126 · 2020-10-02T07:30:52Z

I have the same problem with git dependencies pointing on a specific branch.

  Because package1 (2.0.0 git rev develop) depends on package2 (0.1.0 git rev master)
   and poetry-bug-with-cross-path-deps depends on package2 (0.1.0 git branch master), package1 is forbidden.
  So, because poetry-bug-with-cross-path-deps depends on package1 (2.0.0 git branch develop), version solving failed.

atugushev · 2020-10-02T07:32:43Z

That's right. It's git submodules.

lig · 2020-10-02T07:57:07Z

@finswimmer actually, it doesn't matter here why the paths are like that. This is a real-world scenario that works in 1.0.

Basically, what happens here is that there are two packages specified via relative paths. In case one of the packages specifies the path which is already specified in the root package as its local dependency poetry fails to resolve the dependencies.

The poetry 1.0 behavior was to look for this path relative to the root package's dir. Whether this is good behavior or not is not relevant. That behavior worked and was intuitively expected.

Indeed, if package1 specifies package2 as its dependency and package2 could be found and installed according to the specification in package1 dependencies it is expected for this to work.

P.S.: This issue is not a feature request or proposal. This is a bug report regarding an expected behavior that is broken in poetry 1.1. I'd advise treating this one accordingly.

2m · 2020-10-06T13:59:25Z

We noticed a similar issue with poetry 1.1.0 and 1.1.1 (1.0.10 works fine).

We have a root-project that depends on helper and lib. lib also depends on helper. All of the dependencies are git="..." rev="..." pointing to the same revision.

The error we get is:

Resolving dependencies... (250.4s)
  SolverProblemError

  Because lib (0.1.0 git rev 7e8be74) depends on helper (0.2.0 git rev 833799f)
   and root-project depends on helper (0.2.0 git rev 833799f), lib is forbidden.
  So, because root-project depends on lib (0.1.0 git rev 7e8be74), version solving failed.

  at ~/.poetry/lib/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes

Note that poetry reports that both dependencies are the same:

... depends on helper (0.2.0 git rev 833799f) and root-project depends on helper (0.2.0 git rev 833799f) ...

But it still reports SolverProblemError.

abn · 2020-10-06T16:55:07Z

Basically, what happens here is that there are two packages specified via relative paths. In case one of the packages specifies the path which is already specified in the root package as its local dependency poetry fails to resolve the dependencies.

This is expected behaviour. In 1.0 the resolution was incorrectly performed for nested path dependencies. This meant that if you had ./deps/package2 and ./deps/package1/deps/package2, when ./deps/package1 metadata was inspected while resolving for your root project, build incorrectly selected ./deps/package2 as a dependency of package1 when it should have selected, ./deps/package1/deps/package2. This was resolved in 1.1

When correctly resolved, you will encounter a solving issue because effectively you have pinned two different versions of package2 - package2 @ ./deps/package2 and package2 @ ./deps/package1/deps/package2. Two pinned versions pointing to different paths are inherently incompatible.

abn · 2020-10-06T16:56:26Z

@2m is your scenario multiple copies of submodules as well? If not, could you please try the fix at #3105 ?

benoit9126 · 2020-10-06T17:44:08Z

@abn I agree with you with path dependencies: two different paths must be considered as two different dependencies.

Nevertheless, with the example of @2m, the dependency is the same git repository at the same revision... It must be considered as the same dependency and be valid. Idem with git dependencies pointing to the same branch on the same repository.

abn · 2020-10-06T17:49:48Z

It must be considered as the same dependency and be valid. Idem with git dependencies pointing to the same branch on the same repository.

The question is how they are included. Hence why my comment above asking for clarification. Submodules are path dependencies, not git dependencies from a poetry/pep-508 perspective.

I agree with you with path dependencies: two different paths must be considered as two different dependencies.

To be clear, they can be the same dependency, but are considered two specific versions of it.

benoit9126 · 2020-10-06T18:01:44Z

The question is how they are included. Hence why my comment above asking for clarification. Submodules are path dependencies, not git dependencies from a poetry/pep-508 perspective.

I can not answer for @2m but I have troubles with git dependencies of the form library = {git="git@github.com:organization/repository", branch="..."} and there are no submodules at all.

To be clear, they can be the same dependency, but are considered two specific versions of it.

Thanks!

2m · 2020-10-06T18:18:24Z

In my case all dependencies are provided as git dependencies, like so:

lib = { git = "git@github.com:org/repo.git", rev = "833799f" }

I will try the fix provided in #3105 a bit later and will report back.

2m · 2020-10-06T18:50:30Z

Tried poetry from #3105 and still got the same error as in #3030 (comment)

abn · 2020-10-06T19:02:11Z

@benoit9126 @2m can we capture the git dependency issue in another issue please? Would be great to have an example project as well.

2m · 2020-10-06T21:08:33Z

I tried to reproduce the problem I noticed in a minimized example but could not get the same error. Will try harder a bit later.

2m · 2020-10-07T09:58:44Z

Created a separate issue with a reproducer project: #3113

lig · 2020-10-08T16:55:09Z

@abn

This is expected behaviour. In 1.0 the resolution was incorrectly performed for nested path dependencies. This meant that if you had ./deps/package2 and ./deps/package1/deps/package2, when ./deps/package1 metadata was inspected while resolving for your root project, build incorrectly selected ./deps/package2 as a dependency of package1 when it should have selected, ./deps/package1/deps/package2. This was resolved in 1.1

I'm deeply sorry, but the 1.0 behavior is actually expected and logical behavior.

With 1.1 behavior it is impossible to install dependencies in this case.

The case is:

We have the current project which directly depends on ./deps/package2.
We have package1 which directly depends on ./deps/package2.
We add package1 as a direct dependency to the current project as ./deps/package1.
We want the current project to still depend directly on ./deps/package2 as we don't want to know what the underlying dependencies package1 has.
Now we have the current project which depends directly on ./deps/package1 and ./deps/package2 and package1 which depends on ./deps/package1/deps/package2 where ./deps/package2 and ./deps/package1/deps/package2 are different versions of package2.

Poetry 1.1 output is as follows

  SolverProblemError

  Because poetry-bug-with-cross-path-deps depends on package1 (0.1.0 deps/package1) which depends on package2 (0.2.0 deps/package2), package2 is required.
  So, because poetry-bug-with-cross-path-deps depends on package2 (0.2.0 deps/package2), version solving failed.

As far as I understand now this package2 (0.2.0 deps/package2) and package2 (0.2.0 deps/package2) refers to different absolute paths which is not clear btw.

Also, please notice that both packages have the same version defined in their pyproject.toml files. However, as those are different paths (which is unclear) poetry treats them as different versions.

Correct me if I'm wrong here.

If this is correct then:

I suggest displaying paths in a different way. Maybe, fallback to absolute paths if relative paths are the same or display them relative to the current project's base dir.
Will it help to add version = "*" to allow poetry to install at least one of those packages? Preferably it should be the direct dependency effectively falling back to poetry 1.0 behavior.
I still think that if the indirect requirement matches the version constraint of the direct one it's ok to just install the direct one.

Update:

Poetry version 1.1.2 displays absolute paths for everything except second-level dependency which is being displayed relative to its pyproject.toml
Adding version = "*" to path-dependency results in RuntimeError

  The Poetry configuration is invalid:
    - [dependencies.package2] {'path': 'deps/package2', 'version': '*'} is not valid under any of the given schemas

lig · 2020-10-08T17:07:33Z

Basically, Poetry 1.1.0-1.1.2 is not working at the moment for the case for which Poetry 1.0 was working at least in some way.

finswimmer · 2020-10-08T18:38:10Z

@lig am I right that your expectation about the dependency resolution is like that:

Whenever poetry finds a path dependency, it should read the name and the version from there. When its finished to collect all these information, it should decide based on that which path dependency it installs.

While this sounds reasonable on the first sight, this is not how it works or how it should work. A package from different sources must be treated as different version, regardless of which version number is defined. This is why the path is interpreted as a pinned version. And if you have a package with two different pinned version in your dependency graph, you cannot resolve it.

Why must we treat different sources as different version? Well, we cannot guarantee that they are the same package. Maybe it's a coincidence that the name is equal?

fin swimmer

lig · 2020-10-09T11:06:51Z

@finswimmer the reality is that whilst the software cannot guarantee something people are not dumb and could guarantee that via a policy on which they agreed on. In that case, the software needs to stand aside and allow people to handle that.

I want to state "I know what I'm doing" and software should be able to obey that.

i3v · 2024-11-05T13:45:31Z

Just in case... Relative paths themselves do work, and they are allowed to have .. in their path.

That is, if you replace
package2 = {path = "deps/package2"} with
package2 = {path = "../package2"}
in deps/package1/pyproject.toml, Poetry 1.8.4 works OK.

If all "dependency packages" are stored in the "deps" folder, any "dep" package may depend on any set of the other "dep" packages. This is entirely possible. In particular, in the following scenario, package1 may depend on package2.

poetry-bug-with-cross-path-deps
├── deps
│   ├── package1
│   │   └── pyproject.toml
│   └── package2
│       └── pyproject.toml
├── README.md
└── pyproject.toml

The only inconvenience with this approach is that you would no longer be able to work with "package1" as with a normal standalone self-sufficient project elsewhere. The most simple workaround is to create a dummy wrapper project:

package1-dev
├── deps
│   ├── package1
│   │   └── pyproject.toml
│   └── package2
│       └── pyproject.toml
├── README.md
└── pyproject.toml

Personally, I use this approach extensively, and it works fairly well for me.

github-actions · 2024-12-06T00:31:38Z

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

atugushev added kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels Oct 1, 2020

atugushev changed the title ~~Cross path dependencies causes SolverProblemError~~ Cross path dependencies cause SolverProblemError Oct 1, 2020

lig mentioned this issue Oct 6, 2020

Nested relative directory dependency failing to resolve. #3098

Closed

3 tasks

2m mentioned this issue Oct 7, 2020

Improve dependency source_url equality check to use normalised urls instead of given/derrived values #3113

Open

3 tasks

randallpittman mentioned this issue Oct 8, 2020

Fail resolving nested git dependencies 1.1.* #3102

Closed

3 tasks

finswimmer closed this as not planned Won't fix, can't repro, duplicate, stale Oct 29, 2024

github-actions bot locked as resolved and limited conversation to collaborators Dec 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cross path dependencies cause SolverProblemError #3030

Cross path dependencies cause SolverProblemError #3030

atugushev commented Oct 1, 2020 •

edited

Loading

finswimmer commented Oct 1, 2020

atugushev commented Oct 1, 2020 •

edited

Loading

finswimmer commented Oct 2, 2020

benoit9126 commented Oct 2, 2020

atugushev commented Oct 2, 2020

lig commented Oct 2, 2020 •

edited

Loading

2m commented Oct 6, 2020

abn commented Oct 6, 2020 •

edited

Loading

abn commented Oct 6, 2020

benoit9126 commented Oct 6, 2020

abn commented Oct 6, 2020 •

edited

Loading

benoit9126 commented Oct 6, 2020

2m commented Oct 6, 2020

2m commented Oct 6, 2020

abn commented Oct 6, 2020

2m commented Oct 6, 2020

2m commented Oct 7, 2020

lig commented Oct 8, 2020 •

edited

Loading

lig commented Oct 8, 2020

finswimmer commented Oct 8, 2020

lig commented Oct 9, 2020 •

edited

Loading

i3v commented Nov 5, 2024

github-actions bot commented Dec 6, 2024

Cross path dependencies cause SolverProblemError #3030

Cross path dependencies cause SolverProblemError #3030

Comments

atugushev commented Oct 1, 2020 • edited Loading

Issue

finswimmer commented Oct 1, 2020

atugushev commented Oct 1, 2020 • edited Loading

finswimmer commented Oct 2, 2020

benoit9126 commented Oct 2, 2020

atugushev commented Oct 2, 2020

lig commented Oct 2, 2020 • edited Loading

2m commented Oct 6, 2020

abn commented Oct 6, 2020 • edited Loading

abn commented Oct 6, 2020

benoit9126 commented Oct 6, 2020

abn commented Oct 6, 2020 • edited Loading

benoit9126 commented Oct 6, 2020

2m commented Oct 6, 2020

2m commented Oct 6, 2020

abn commented Oct 6, 2020

2m commented Oct 6, 2020

2m commented Oct 7, 2020

lig commented Oct 8, 2020 • edited Loading

lig commented Oct 8, 2020

finswimmer commented Oct 8, 2020

lig commented Oct 9, 2020 • edited Loading

i3v commented Nov 5, 2024

github-actions bot commented Dec 6, 2024

atugushev commented Oct 1, 2020 •

edited

Loading

atugushev commented Oct 1, 2020 •

edited

Loading

lig commented Oct 2, 2020 •

edited

Loading

abn commented Oct 6, 2020 •

edited

Loading

abn commented Oct 6, 2020 •

edited

Loading

lig commented Oct 8, 2020 •

edited

Loading

lig commented Oct 9, 2020 •

edited

Loading