We can't install package include relative path dependency as VCS dependency

[ota42y]

Pull Request Check List

• Added tests for changed code.
• Updated documentation for changed code.

summary

• We can't install package include relative path dependency as VCS dependency
  • The package depend other package which in sub folder
  • The package is valid
    • We can install package as local package or use pip install
    • This is differnt issue than git subfolder
• This problem caused by removing tmp directory
  • poetry clone repository to tmp dir and remove it immedietly
  • poetry create directory dependency for sub folders package
  • When install process, poetry can't find sub folder
• We can install with small change but this change break lock file
  • We save lock file directory dependency for tmp dir
  • We can't install package from lock file
• We should add new semantics for lock file and I want to disscuss.

probllem

This package (poetry_package_with_subdir_package) is valid package and it have relative path dependency for sub directory (poetry_subdir_package).
https://github.com/ota42y/poetry_package_with_subdir_package

We can create a directory dependency for a locally cloned folder.
But we can't create a VCS dependency for this repository

Example is this.

mkdir /tmp/poetry-git-relative-bug && cd /tmp/poetry-git-relative-bug

# use vcs dependency
mkdir poetry_vcs_bug && cd poetry_vcs_bug

cat > pyproject.toml << EOF
[tool.poetry]
name = "poetry_vcs_bug"
version = "0.1.0"
description = ""
authors = ["ota42y <ota42y@gmail.com>"]

[tool.poetry.dependencies]
python = "^3.8"
poetry_package_with_subdir_package = {git = "git@github.com:ota42y/poetry_package_with_subdir_package.git"}
# poetry_package_with_subdir_package = {path = '../poetry_package_with_subdir_package'}

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
EOF

poetry install

# We got this error.
#  FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pypoetry-git-poetry_package_with_subdir_packagead_tu1nh/poetry_subdir_package'


# clone local folder and create directory dependency
cd ../
git clone git@github.com:ota42y/poetry_package_with_subdir_package.git
mkdir poetry_local_dependency && cd poetry_local_dependency

cat > pyproject.toml << EOF
[tool.poetry]
name = "poetry_local_dependency"
version = "0.1.0"
description = ""
authors = ["ota42y <ota42y@gmail.com>"]

[tool.poetry.dependencies]
python = "^3.8"
# poetry_package_with_subdir_package = {git = "git@github.com:ota42y/poetry_package_with_subdir_package.git"}
poetry_package_with_subdir_package = {path = '../poetry_package_with_subdir_package'}

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
EOF

poetry install
# succeeded

I create other package which have relative path dependency using setup.py but I got same error.
This mean we can't use VCS package which include relative path even if the package use pip or poetry.

Cause

temporary directory problem

Provider#get_package_from_vcs clone git repository to temporary directory.
poetry immediately creates Package object and remove temporary directory.

poetry/poetry/puzzle/provider.py

Line 227 in fcc2c83

safe_rmtree(str(tmp_dir))

The relative dependency of the package is then interpreted as a directory dependency on the folder in the temporary directory.

For example, in the previous example, the object would be like this:

pkg = Package(name='poetry_package_with_subdir_package", source_type='git')
pkg.requires = [
    DirectoryDependency(name='poetry_subdir_package' 
                        path='/tmp/pypoetry-git-poetry_package_with_subdir_packagead_tu1nh/poetry_subdir_package'
    )
]

poetry install original pakcage (VCS dependency package) from git repository so we can install this package correctory.
But the directory dependency package depende on temporary directory and which alerady removed so we can't install it.

long life temporary diectory

We can solve this issue by this PR's changes.
This changes

This change can be installed by extending the lifetime of the temporary directory.
But poetry write directory dependency which depend on temporary directory to lock file so we can't install package from lock file.

[[package]]
name = "poetry-subdir-package"
version = "0.1.0"
description = ""
category = "main"
optional = false
python-versions = "^3.8"
develop = false

[package.source]
type = "directory"
url = "../../../../tmp/pypoetry-git-poetry_package_with_subdir_packagead_tu1nh/poetry_subdir_package"

how to solve

I think these package is valid package because we can install from local folder and we can install by pip so we should fix this problem.

I've come up with two solutions.

• convert relative dependency in VCS dependency to git subfolder dependency
• craete new dependency type like VCS relative dependency

convert relative dependency in VCS dependency to git subfolder dependency

This problem cause by VCS dependency, and relative dependency in git repository is equal to git subfolder's dependency.
So we can convert git subfolder's dependency.

For example, if we have a Package that has a dependency on the ./poetry_subdir_package.

pkg = Package(name='demo", source_type='git',
	source_url="https://github.com/ota42y/poetry_git_with_local_package",
	source_reference="abc")

We can convert directory dependency (./poetry_subdir_package) to git subfolder dependency with same revision like this

pkg.requires = [
    VCSDependency(name='poetry_subdir_package' vcs='git',
		source="https://github.com/ota42y/poetry_git_with_local_package",
		rev="abc",
		subfolder="./poetry_subdir_package"
    )
]

We can convert in creating package from VCS

pros/cons

• 👍 small changes because we can use current lock file's semantics
• 👎 git subfolder doesn't released
• 👎 we should clone git repository for relative package so install process will be slow
• 👎 we can't save this package is relative dependency package on lock file
  • we should update relative dependency package when original package updated
  • poetry may be able to block the update with specific option because poetry treat it as an unrelated package.

craete new dependency type like VCS relative dependency

If we can create Dependency for relative dependency in VCS dependency, we can solve this problem by simple way.

For example, if we have a Package that has a dependency on the ./poetry_subdir_package.

pkg = Package(name='demo", source_type='git',
	source_url="https://github.com/ota42y/poetry_git_with_local_package",
	source_reference="abc")

We can create Dependency for relative dependency in VCS dependency and set pkg.requires.
When we install VCSDirectoryDependency, we should clone this repository to tmp dir and install like DirectoryDependency.

pkg.requires = [
    VCSDirectoryDependency(
		name='poetry_subdir_package',
		parent_name='demo', path="./poetry_subdir_package"
		source_type='git',
		source_url="https://github.com/ota42y/poetry_git_with_local_package",
		source_reference="abc",
		clone_dir='tmp/pypoetry-git-poetry_package_with_subdir_packagead_tu1nh'
    )
]

poetry write Package with VCSDirectoryDependency to lock file, we should add new propeties to source section and we can create VCSDirectoryDependency from lock file.

[[package]]
name = "poetry-subdir-package"
version = "0.1.0"
description = ""
category = "main"
optional = false
python-versions = "^3.8"
develop = false

[package.source]
type = "vcs_directory"
path = "./poetry_subdir_package"
name = "demo"
url = "git@github.com:ota42y/poetry_package_with_subdir_package.git"
reference = "master"
resolved_reference = "c3af6b7af157c0321a58c5e5c41dc929897af98a"

pros/cons

• 👍 only one clone
• 👍 we can save this package is relative dependency package and has relation to original package to lock file
• 👎 very big changes
  • We need to build dependency comparisons and conflict solver.
  • Keep backward compatibility

I want to solve this problem because we can't use valid package.
I think the latter is a simpler solution, but it may require a lot of change, and there may be a lot of things to solve because I don't know the full poetry spec.
If there are any other good solutions, I'd love to discuss them.

[sonarcloud]

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
2 Code Smells

No Coverage information
3.5% Duplication

[neersighted]

It was never really clear what this PR set out to address, and the change is quite invasive. I think that the intention was to allow a file:// style path to a VCS dep, but that seems partially useless as a VCS dep will be available in $VENV_ROOT/src and a regular path dependency works if you don't want to use a installed VCS dep (including in editable mode).

VCS deps are also meant to be reproducible anywhere (unlike local file deps) -- allowing them to be a path that is resolvable only on one machine breaks the existing semantics of the lock file and Poetry. I'm closing this for now, but if there's a compelling reason to consider this/continue to work on it I missed, please let me know.

[github-actions]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.