Dependency rooted in a subfolder of a GIT repository #755

sandrinr · 2018-12-20T08:59:58Z

I have searched the issues of this repo and believe that this is not a duplicate.

Feature request

Currently, as far as I can figure, it is impossible to use a GIT repository as a dependency source where the setup.py or pyproject.toml file is not in the root directory but in some sub-folder of the repository.

E.g. GIT repo structure:

GIT root
- pythonmodule
  - setup.py / pyproject.toml
  - module1.py
  - module2.py
- javamodule
  - pom.xml
- cmodule
  - Makefile

In this example I would like to be able to point poetry to the pythonmodule directory inside this repo, for example using poetry add mydep --git https://github.com/foo/bar.git --path path/to/mydep. I don't think this can be accomplished via a separate repository definition, however, the documentation is a bit scarce on that topic.

The text was updated successfully, but these errors were encountered:

bowendeng · 2019-01-02T22:06:56Z

Same here, i'm able to install a git dependency using pip with its VCS support

pip install "git+https://github.com/nmslib/hnswlib.git@bb2a1d1#egg=hnswlib&subdirectory=python_bindings"

It would be nice if we can do it using poetry add.

stale · 2019-11-13T17:45:13Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale · 2019-11-20T18:16:16Z

Closing this issue automatically because it has not had any activity since it has been marked as stale. If you think it is still relevant and should be addressed, feel free to open a new one.

zbarnett · 2020-01-31T22:17:40Z

I would love to switch to Poetry and it looks like this is the only thing preventing me. Very excited to see this feature is in progress!

finswimmer · 2020-02-02T09:28:08Z

Ready for testing :)

velovix · 2020-09-04T22:35:19Z

@finswimmer Thank you for your work on this! Is there any way I can help with getting your PRs ready for review?

TheButlah · 2020-12-18T19:40:32Z

Was this functionality implemented? My team relies on poetry and we just started breaking some of our code up into a separate git repo - we would need this to continue to use poetry

sinoroc · 2020-12-18T20:01:20Z

Was this functionality implemented? My team relies on poetry and we just started breaking some of our code up into a separate git repo - we would need this to continue to use poetry

As far as I can tell, it has been implemented, but not merged nor released. This still-in-progress implementation is here: #1822.

finswimmer · 2020-12-19T07:12:06Z

It isn't fully implemented until now. A lot has changed since I wrote #1822. This PR is now splitted in python-poetry/poetry-core#9 and #2242 But there is still a lot to do. I cannot tell when this will be ready.

Tinche · 2020-12-22T14:29:51Z

We are also looking into breaking out shared modules into a separate repo, and it would be awesome to be able to keep them all in one repository and reference them by sub-directories.

dvarrazzo · 2021-03-13T22:52:51Z

Do I understand correctly that it is not currently possible to install psycopg3 using poetry, owing to the lack of subdirectory support?

with pip or requirement it is possible to use:

pip install -e "git+https://github.com/psycopg/psycopg3.git@9688cd2#egg=psycopg3&subdirectory=psycopg3"
pip install -e "git+https://github.com/psycopg/psycopg3.git@9688cd2#egg=psycopg3-c&subdirectory=psycopg3_c"

Is there similar support for poetry?

golyalpha · 2021-05-18T14:03:15Z

Right now working around this via filesystem dependencies, so, clone repo first to a known location, (preferably ../), and define the dependency as lib_name = {path = '../lib-name/python-module'}

It does mean that I have to manually pull in every single repository that has the python module in a subdir, which is not ideal.

chr-wei · 2021-06-29T13:39:19Z

Anyone implementing this right now? Would be super helpful!

Datasets sometimes use libraries in their script. We install all of them (as of today). Note that https://github.com/TREMA-UNH/trec-car-tools could not be added using poetry since the package is in subdirectory. See python-poetry/poetry#755.

ashnair1 · 2022-01-16T16:01:18Z

You can use the latest version of poetry-core by specifying the following in your pyproject.toml

[build-system]
requires = ["poetry-core @ git+https://github.com/python-poetry/poetry-core.git"]
build-backend = "poetry.core.masonry.api"

mccolljr · 2022-01-16T17:18:56Z

You could use the latest version of poetry-core by specifying the following in your pyproject.toml
[build-system]
requires = ["poetry-core @ git+https://github.com/python-poetry/poetry-core.git"]
build-backend = "poetry.core.masonry.api"

Brilliant, thanks - this is exactly what I needed

falonso-alo · 2022-01-24T21:19:39Z

You can use the latest version of poetry-core by specifying the following in your pyproject.toml
[build-system]
requires = ["poetry-core @ git+https://github.com/python-poetry/poetry-core.git"]
build-backend = "poetry.core.masonry.api"

@ashnair1,

So would this allow me to use a dependency in subfolder? If so, do you happen to know what the synatx looks like? Something like this perhaps?

[tool.poetry.dependencies]
my_module = { git = "https://github.com/my_org/my_repo.git", directory = "src/my_module_folder" }

remram44 · 2022-01-24T21:33:08Z

@falonso-alo: it's subdirectory=

ashnair1 · 2022-01-25T06:04:09Z

@falonso-alo

It would be something like this

[tool.poetry.dependencies]
my_module = { git = "https://github.com/my_org/my_repo.git#egg=my_module_folder&subdirectory=src" }

which translates to install my_module_folder located in the subdirectory src of my_repo.git

Edit:

Using the subdirectory param as shown below does not work.

[tool.poetry.dependencies]
my_module = { git = "https://github.com/my_org/my_repo.git", subdirectory = "src/my_module_folder" }

The link gets resolved to the following incorrect git link and installation will fail since it won't be able to find setup.py or pyproject.toml

my_module@ git+https://github.com/my_org/my_repo.git#subdirectorysrc/my_module_folder

Ideally you should go with specifying poetry supported key word args as opposed to using the full link (as I mentioned above) as it creates other problems especially when you're distributing whls. But if you just want to install the dependency it should be fine. Truth be told, poetry is not yet in a state where you can reliably use and manage git dependencies rooted in subdirectories.

falonso-alo · 2022-01-25T17:07:59Z

Thank you @ashnair1 and @remram44! It does seem like this is a bit difficult right now. It may be too much of a curve ball for my team so I will probably have to wait for better support for subdirectories.

Thanks again!

ashnair1 · 2022-02-10T09:34:21Z

I've opened #5172 and python-poetry/poetry-core#288 to address this.

Felix-neko · 2022-02-14T19:57:17Z

my_module = { git = "https://github.com/my_org/my_repo.git", subdirectory = "src/my_module_folder" }

Hi folks! Have anybody already tested it? It should be extremely useful, but it looks i've installed this fork in wrong way and it doesn't work correctly on my machine = (

rickmark · 2022-03-04T07:13:57Z

Suggest edit of title to use subdirectory which is the term used in PIP to help it be found by search.

rickmark · 2022-03-18T18:57:20Z

my_module = { git = "https://github.com/my_org/my_repo.git", subdirectory = "src/my_module_folder" }
Hi folks! Have anybody already tested it? It should be extremely useful, but it looks i've installed this fork in wrong way and it doesn't work correctly on my machine = (

This doesn't seem to work even with poetry and poetry-core pre (as of today).

Our "problem dependency" is this:

unicoirn = { git = "https://github.com/unicorn-engine/unicorn.git", branch = "dev", subdirectory="bindings/python"}

It seems it does not enter the subdirectory when building the VCS package. With an error output of:

The following error occurred when trying to handle this error:


  PackageInfoError

  Unable to determine package info for path: /var/folders/0j/v1pqc5ks1v90nj3xf_h715sc0000gn/T/pypoetry-git-unicorntfo2cpdd
  
  Command ['/var/folders/0j/v1pqc5ks1v90nj3xf_h715sc0000gn/T/tmp1kapedvm/.venv/bin/python', '-'] errored with the following return code 1, and output: 
  Traceback (most recent call last):
    File "<stdin>", line 9, in <module>
    File "/var/folders/0j/v1pqc5ks1v90nj3xf_h715sc0000gn/T/tmp1kapedvm/.venv/lib/python3.10/site-packages/build/__init__.py", line 208, in __init__
      _validate_source_directory(srcdir)
    File "/var/folders/0j/v1pqc5ks1v90nj3xf_h715sc0000gn/T/tmp1kapedvm/.venv/lib/python3.10/site-packages/build/__init__.py", line 109, in _validate_source_directory
      raise BuildException(f'Source {srcdir} does not appear to be a Python project: no pyproject.toml or setup.py')
  build.BuildException: Source /var/folders/0j/v1pqc5ks1v90nj3xf_h715sc0000gn/T/pypoetry-git-unicorntfo2cpdd does not appear to be a Python project: no pyproject.toml or setup.py
  input was : import build
  import build.env
  import pep517
  
  source = '/var/folders/0j/v1pqc5ks1v90nj3xf_h715sc0000gn/T/pypoetry-git-unicorntfo2cpdd'
  dest = '/var/folders/0j/v1pqc5ks1v90nj3xf_h715sc0000gn/T/tmp1kapedvm/dist'
  
  with build.env.IsolatedEnvBuilder() as env:
      builder = build.ProjectBuilder(
          srcdir=source,
          scripts_dir=env.scripts_dir,
          python_executable=env.executable,
          runner=pep517.quiet_subprocess_runner,
      )
      env.install(builder.build_system_requires)
      env.install(builder.get_requires_for_build('wheel'))
      builder.metadata_path(dest)
  
  No fallback setup.py file was found to generate egg_info.

  at ~/.pyenv/versions/3.10.2/lib/python3.10/site-packages/poetry/inspection/info.py:497 in _pep517_metadata
      493│                 # fallback to egg_info if setup.py available
      494│                 cls._log(f"PEP517 build failed: {e}", level="debug")
      495│                 setup_py = path / "setup.py"
      496│                 if not setup_py.exists():
    → 497│                     raise PackageInfoError(
      498│                         path,
      499│                         e,
      500│                         "No fallback setup.py file was found to generate egg_info.",
      501│                     )```

StuartBertram · 2022-04-07T08:42:42Z

I've tried this but can't even get the subdirectory parameter to work.

I've got:

[build-system]
requires = ["poetry-core @ git+https://github.com/python-poetry/poetry-core.git"]
build-backend = "poetry.core.masonry.api"

in my pyproject.toml. I'm using a ssh URL for a private GitHub repo. But I get "[repo definition] is not valid under any of the given schemas".

Does this need the pre-release Poetry as well as the poetry-core settings?

- Remove depencies that are actually dependencies of discrete-optimization - Ideally should add discrete-optimization as a dependency but - for now only available through git, as a subdirectory - this is not yet implemented by poetry (python-poetry/poetry#755) - for testing purposes, we should use the version available on the same branch, as it is on the same repo for now - for building purpose and specify it as a dependency by the scikit-decide wheel available on pypi, we should point to an official release. Only possible when such a release exists - So "manual" install of discrete-optimization required for local testing and github workflow

danodonovan · 2022-07-08T13:30:36Z

So I think I've got poetry VCS installing to work with subdirectory kwarg - but I needed to update both poetry and poetry-core ~~to bleeding edge versions~~ first;

$ pip install git+https://github.com/python-poetry/poetry.git@1.2.0b2
$ pip install git+https://github.com/python-poetry/poetry-core.git@1.1.0b2

~~which is the commit where @ashnair1 s PR #5172 merged (using poetry-core HEAD isn't currently working for me)~~

After that, adding

[build-system]
requires = ["poetry-core @ git+https://github.com/python-poetry/poetry-core.git"]
build-backend = "poetry.core.masonry.api"

and the subdirectory kwarg

my_module = { git = "https://github.com/my_org/my_repo.git", subdirectory = "src/my_module_folder" }

seems to be working for me 🚀

~~NB This would be much nicer if there were some recent releases that contained these changes.~~

Edit. Sorry RE recent release comments maintainers - they do exist, and poetry is a fine tool 👍

b1r3k · 2022-07-29T15:55:38Z

seems to be working for me rocket

I think something is missing here since it does not work for me. Here is my environment:

$ pip freeze

poetry @ git+https://github.com/python-poetry/poetry.git@ccbeffc1f1a5809e999cc18c584ed2cee2a8f580
poetry-core==1.1.0b2

$ cat pyproject.toml

...
[tool.poetry.dependencies]
openapi-client = { git = "https://github.com/deribit/deribit-api-clients.git", subdirectory = "python" }

[build-system]
requires = ["poetry-core @ git+https://github.com/python-poetry/poetry-core.git"]
build-backend = "poetry.core.masonry.api"

Trying to install project with: poetry install ends up with:

  • Installing openapi-client (1.0.0 7ae49f6): Failed

  CalledProcessError

  Command '['ROOT/machine/.venv/bin/python', '/home/lukasz/.pyenv/versions/3.10.2/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/pip-22.0.4-py3-none-any.whl/pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', 'ROOT/machine/.venv', '--upgrade', '--no-deps', 'ROOT/machine/.venv/src/deribit-api-clients']' returned non-zero exit status 1.

  at ~/.pyenv/versions/3.10.2/lib/python3.10/subprocess.py:524 in run
       520│             # We don't call process.wait() as .__exit__ does that for us.
       521│             raise
       522│         retcode = process.poll()
       523│         if check and retcode:
    →  524│             raise CalledProcessError(retcode, process.args,
       525│                                      output=stdout, stderr=stderr)
       526│     return CompletedProcess(process.args, retcode, stdout, stderr)
       527│ 
       528│ 

The following error occurred when trying to handle this error:


  EnvCommandError

  Command ['ROOT/machine/.venv/bin/python', '/home/lukasz/.pyenv/versions/3.10.2/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/pip-22.0.4-py3-none-any.whl/pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', 'ROOT/machine/.venv', '--upgrade', '--no-deps', 'ROOT/machine/.venv/src/deribit-api-clients'] errored with the following return code 1, and output: 
  ERROR: Directory 'ROOT/machine/.venv/src/deribit-api-clients' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.

andersk · 2022-07-31T21:54:26Z

poetry @ git+https://github.com/python-poetry/poetry.git@ccbeffc1f1a5809e999cc18c584ed2cee2a8f580

@b1r3k ccbeffc is the 1.2.0b2 tag. You don’t have #5172, which has not yet been merged. You need to do:

pip install git+https://github.com/ashnair1/poetry.git@subdir-fix

nikowatari · 2023-03-01T03:22:54Z

this is again broken after new installer became the default (1.4.0). I was able to fix it locally by adding following:

if package.source_subdirectory is not None:
    archive = archive / Path(package.source_subdirectory)

after this line: https://github.com/python-poetry/poetry/blob/master/src/poetry/installation/executor.py#L571

Datasets sometimes use libraries in their script. We install all of them (as of today). Note that https://github.com/TREMA-UNH/trec-car-tools could not be added using poetry since the package is in subdirectory. See python-poetry/poetry#755.

github-actions · 2024-02-29T18:05:54Z

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

stale bot added the stale label Nov 13, 2019

stale bot closed this as completed Nov 20, 2019

finswimmer added kind/feature Feature requests/implementations and removed stale labels Jan 2, 2020

finswimmer mentioned this issue Jan 2, 2020

Subdirectory property in git links #961

Closed

2 tasks

finswimmer reopened this Jan 2, 2020

finswimmer self-assigned this Jan 3, 2020

finswimmer mentioned this issue Jan 3, 2020

support git subfolder (#755) #1822

Closed

2 tasks

finswimmer mentioned this issue Jan 18, 2020

Install package from Version Control System (git) where setup.py is in subdirectory #1915

Closed

ekcolysp mentioned this issue Mar 16, 2020

Ability to have poetry add git+https://... saves to current directory #2188

Open

This was referenced Mar 29, 2020

support git subfolder python-poetry/poetry-core#9

Closed

support git subfolder (#755) #2242

Closed

abn mentioned this issue Apr 13, 2020

Support for mutually exclusive dependencies across groups (e.g. path dependencies during development only) #1168

Open

2 tasks

alexlatchford mentioned this issue Oct 29, 2020

AIP-653: Sym-link python SDK components to avoid poetry lacking subdirectory install capabilities zillow/pipelines#1

Merged

frankier mentioned this issue Dec 27, 2020

requires_dist specifies numpy version <1.17 for Python 3 nmslib/nmslib#445

Closed

ryanlerch mentioned this issue Jul 30, 2021

Set up vagrant and a get it working fedora-infra/datagrepper#259

Merged

finswimmer mentioned this issue Feb 4, 2022

optional path for git repository specification #5144

Closed

2 tasks

dmvianna mentioned this issue Feb 4, 2022

Importing theta-python as dependency target/theta-idl#20

Closed

ashnair1 mentioned this issue Feb 9, 2022

Fix for git subdirectories #5172

Merged

1 task

yajo mentioned this issue Feb 15, 2022

Support PEP 517 config settings #5202

Closed

2 tasks

finswimmer mentioned this issue Mar 4, 2022

Add subdirectory to poetry VCS configs #5272

Closed

2 tasks

rht mentioned this issue Apr 30, 2022

poetry add of a GitHub repo with subdirectory fails #5521

Closed

3 tasks

grdddj mentioned this issue May 25, 2022

Support model R emulators trezor/trezor-user-env#178

Merged

frenzymadness mentioned this issue Jun 30, 2022

Micropipenv ignores subdirectory for VCS packages. thoth-station/micropipenv#236

Closed

radoering closed this as completed in #5172 Aug 10, 2022

radoering mentioned this issue Mar 1, 2023

"subdirectory" argument ignored #7575

Closed

4 tasks

harrymander mentioned this issue Mar 1, 2023

fix: append subdirectory to archive path (#7575) #7580

Merged

2 tasks

github-actions bot locked as resolved and limited conversation to collaborators Feb 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dependency rooted in a subfolder of a GIT repository #755

Dependency rooted in a subfolder of a GIT repository #755

sandrinr commented Dec 20, 2018 •

edited

Loading

bowendeng commented Jan 2, 2019

stale bot commented Nov 13, 2019

stale bot commented Nov 20, 2019

zbarnett commented Jan 31, 2020

finswimmer commented Feb 2, 2020

velovix commented Sep 4, 2020

TheButlah commented Dec 18, 2020

sinoroc commented Dec 18, 2020

finswimmer commented Dec 19, 2020

Tinche commented Dec 22, 2020

dvarrazzo commented Mar 13, 2021

golyalpha commented May 18, 2021

chr-wei commented Jun 29, 2021 •

edited

Loading

ashnair1 commented Jan 16, 2022 •

edited

Loading

mccolljr commented Jan 16, 2022

falonso-alo commented Jan 24, 2022

remram44 commented Jan 24, 2022

ashnair1 commented Jan 25, 2022 •

edited

Loading

falonso-alo commented Jan 25, 2022

ashnair1 commented Feb 10, 2022

Felix-neko commented Feb 14, 2022

rickmark commented Mar 4, 2022

rickmark commented Mar 18, 2022 •

edited

Loading

StuartBertram commented Apr 7, 2022

danodonovan commented Jul 8, 2022 •

edited

Loading

b1r3k commented Jul 29, 2022

andersk commented Jul 31, 2022

nikowatari commented Mar 1, 2023 •

edited

Loading

github-actions bot commented Feb 29, 2024

Dependency rooted in a subfolder of a GIT repository #755

Dependency rooted in a subfolder of a GIT repository #755

Comments

sandrinr commented Dec 20, 2018 • edited Loading

Feature request

bowendeng commented Jan 2, 2019

stale bot commented Nov 13, 2019

stale bot commented Nov 20, 2019

zbarnett commented Jan 31, 2020

finswimmer commented Feb 2, 2020

velovix commented Sep 4, 2020

TheButlah commented Dec 18, 2020

sinoroc commented Dec 18, 2020

finswimmer commented Dec 19, 2020

Tinche commented Dec 22, 2020

dvarrazzo commented Mar 13, 2021

golyalpha commented May 18, 2021

chr-wei commented Jun 29, 2021 • edited Loading

ashnair1 commented Jan 16, 2022 • edited Loading

mccolljr commented Jan 16, 2022

falonso-alo commented Jan 24, 2022

remram44 commented Jan 24, 2022

ashnair1 commented Jan 25, 2022 • edited Loading

falonso-alo commented Jan 25, 2022

ashnair1 commented Feb 10, 2022

Felix-neko commented Feb 14, 2022

rickmark commented Mar 4, 2022

rickmark commented Mar 18, 2022 • edited Loading

StuartBertram commented Apr 7, 2022

danodonovan commented Jul 8, 2022 • edited Loading

b1r3k commented Jul 29, 2022

andersk commented Jul 31, 2022

nikowatari commented Mar 1, 2023 • edited Loading

github-actions bot commented Feb 29, 2024

sandrinr commented Dec 20, 2018 •

edited

Loading

chr-wei commented Jun 29, 2021 •

edited

Loading

ashnair1 commented Jan 16, 2022 •

edited

Loading

ashnair1 commented Jan 25, 2022 •

edited

Loading

rickmark commented Mar 18, 2022 •

edited

Loading

danodonovan commented Jul 8, 2022 •

edited

Loading

nikowatari commented Mar 1, 2023 •

edited

Loading