[bugs] Workflow issues

[haf]

This is not a discussion, but a collection of bug reports.

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: macos 12.3.1

• Poetry version: 1.1.12

Background

Related to but not same as #1168 (comment)

I build whl files of my project as well as its libs that are in sibling folders in the source tree, outside of the Dockerfile, because it makes it faster to install, not to have to go through dependency resolution. The whl files, both of libs and the service itself, are copied into the dockerfile.

Issue 1

When you use Path dependencies like libpg = {path = "../../libs/libpg"}, the resulting PKG-INFO file has libpg @ ../../libs/libpg.

Expectation: poetry build, when building dependencies that are path-linked, would output not just service-X.Y.Z-...whl but also every dependency it was responsible for building as part of the build operation, and it would use version constraints in PKG-INFO, not path dependencies.

Issue 2

When you use dependencies like libpg = {path = "./dist/deps/libpg-0.4.0-py3-none-any.whl"}, again, the service-...whl file is built to depend on dist/deps/... instead of the versions.

Expectation: again, if I reference packages, it should build packages that depend on packages with versions and leave the indexing and source resolution up to the runtime (e.g. when I do pip install)

Issue 3

When you use dependencies like libpg = {path = "../../libs/libpg", develop = true}, see issue #1168

Expectation: #1168 and in particular allowing development dependencies to override this, and to have this override checked in (because: monorepo!) always.

Issue 4

If you use dependencies like libpg = "^0.4.0" in [tool.poetry.dependencies] but libpg = {path = "../../libs/libpg", develop = true} under [tool.poetry.dev-dependencies], poetry complains and won't install.

Expectation: since you can run poetry install --no-dev I would have expected dev-dependencies to be overriding non-dev dependencies, and build to ignore dev dependencies. Similar to how env files work in NextJS https://nextjs.org/docs/basic-features/environment-variables#default-environment-variables

Issue 5

When you create a new venv for developing, and you manually replace libpg = "^0.4.0" with libpg = {path = "../../libs/libpg", develop = true}, you have to run poetry update libpg. If this library now depend on say opentelemetry.exporters.jaeger.thrift which requires setuptools, the initialisation of your development environment fails, because poetry update crashes with an error that "setuptools is not available in your environment so I can't install the thrift package". You now have to use both pip and poetry.

Expectation: pip, setuptools and poetry are all locked when using poetry

Issue 6

A bit counter-intuitively when thinking of issue 4 above, if I:

• want to install with .whl packages to lower the size of the docker context transferred on build
• want to allow both whl in prod (as it's in git (gitops)), and manually hack around the limitations in poetry in dev (alike issue 5)

I have to:

• specify libpg = "^0.4.0" as a dependency

This means that if I don't first install libpg with pip (not with poetry because of issue 2), poetry will query pypi for this library, leaving me open to supply-chain attacks if someon guess this library name. Because the aim of poetry.lock is to avoid precisely this, and libpg is private, if issues 1..5 are unresolved (and even so), I should want to lock down the source — in this case "never install from any source if it's not already installed manually". If the above issues are fixed, I should want it to lock down that it's installed from a filesystem path and not from an index. Otherwise, I'm indeed open to supply-chain attacks if I forget to make packages before make image (as the packages are built from the outside of the Dockerfile).

---

In general, poetry should focus on maintaining, installing and building packages coherently. Not on trying to create a runtime for them. Only once package management is solved should the tool set up environments for packages to run in (source deps, linking to relative paths, shell).

[haf]

Issue 7

When using 1.1.13 instead of 1.1.12, it also deletes the setuptools installation, rendering it impossible to even install any packages similar to #3153

[abn]

Can you provide a bit more information on this one or an example? I suspect this is already handled. Poetry 1.2 does not remove reserved packages from the virtual envrionment unless it was in the lock file, and an operation removes it from the lockfile.

poetry/src/poetry/puzzle/transaction.py

Lines 81 to 88 in da9680b

	# We preserve pip/setuptools/wheel when not managed by poetry, this is
	# done to avoid externally managed virtual environments causing
	# unnecessary removals.
	preserved_package_names = {
	"pip",
	"setuptools",
	"wheel",
	} - current_package_names

[haf]

Because of issues 1-6 above, the only recourse is poetry update and that removes it from the lockfile / and / or the current installation when using 1.1.13. Perhaps you could detail what sort of information you'd like to get.

[abn]

An example would be helpful. Additionally, would be good to have this verified against the master branch.

[haf]

Would you comment on the above issues 1-6 first, before I spend some time creating repos of a minor issue? (Since I can enforce v1.1.12)

[haf]

Why do you convert this into a discussion? It's clearly an issue since there's no way to do monorepos right now with both dev+prod builds without hacking around poetry.

[justinTM]

the poetry maintainers are very stubborn and difficult to work with. just something you have to deal with